How do I detect drift in AI agent behavior?

Answer

Monitor three metrics:

1. Latency — agent response time. Establish baselines from 30+ day windows, compute mean and standard deviation, flag z-scores above thresholds (under 1 sigma is info, 1-2 sigma is warn, 2-3 sigma is critical, over 3 sigma is emergency).
2. Error rate — proportion of failed actions. Use Bernoulli standard deviation for baseline.
3. Compliance score — average score from evaluations. When baselines have zero standard deviation, fall back to relative drift with thresholds at 10%, 25%, 50%. Rebuild baselines daily.