Study finds AI auditing catches most agent anomalies but drowns reviewers in false positives

AI models can often spot problematic behavior by other AI agents in logs, but the harder they try to catch every case, the more harmless actions they flag as suspicious. This is the central finding of work by Andy Hall, a researcher at the Anthropic Institute, who reproduced and extended earlier research by METR and published his results in a Substack post covered by The Decoder. Hall tested whether frontier models can identify anomalous agent behavior from logs, running his audit experiments with GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8, and Gemini 3.8 Flash. The headline result is positive: GPT-5 found about 98 percent of reward hacking cases, which suggests that model-based auditing can surface most agent misbehavior without human reviewers examining every log entry. But the study highlights a serious trade-off. As the search is tuned more exhaustively to catch every individual incident, false alarms pile up, and harmless actions get flagged as suspicious. That flood of false positives overwhelms reviewers, and Hall concludes that the models are not yet sufficient for reliable safety checks that need to catch every single incident without fail. Instead, he argues, they are better suited to estimating how often things go wrong, using model audits as a statistical signal rather than a definitive gate. No model in the study managed to combine high coverage with low false positive rates, so the practical takeaway is that AI auditing is promising as a way to measure error frequency, but it cannot yet serve as a safety mechanism that guarantees no problematic behavior slips through.