How do I set up bias detection for AI agents?

Answer

Three-phase approach:

1. v1 — Keyword/lexicon analysis. Scan agent outputs for demographic term frequency across dimensions (gender, race, age). Score equals term density times 20, capped at 1.0. This catches obvious bias patterns with zero external dependencies.
2. v2 — Statistical analysis. Compare agent decision distributions across demographic groups. Measure disparate impact ratios. Requires structured decision data.
3. v3 — LLM-powered analysis. Use a separate evaluator model to assess outputs for subtle bias. Start with v1 (shipped today in AgentCompliant), add v2 with sufficient data, add v3 as a premium feature.