Andy Hall, a researcher at the Anthropic Institute, described an experiment on the Free Systems Substack in which AI models reviewed the work logs of AI agents. The Decoder summarised the post on 7 November. Hall is on leave from Stanford University and says the post is written in a personal capacity.

Hall concludes that the models are better suited to estimating how common agent misbehaviour is than to safety checks in which every case must be found. The WebPulse website points out that the experiment measured detection and false alarms, not prevalence estimation.

What the auditors found

Hall replicated and extended an experiment by the research organisation METR. METR’s MALT dataset consists of logs in which frontier models carry out tasks from the HCAST and RE-Bench benchmarks, and humans reviewed some of the logs in October 2025. The auditors were GPT-5, GPT-5 nano, GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash.

The violations were of two types. In reward hacking, an agent uses tricks to pass a task without solving it. In sandbagging, a model deliberately underplays its capabilities. The sandbagging cases are mostly artificial: METR instructed the models to sabotage and removed the instructions from the visible reasoning.

When an auditor was allowed to wrongly flag 5 per cent of harmless logs, GPT-5 found nearly 98 per cent of the reward hacking cases that humans had flagged, and Gemini 3.8 Flash barely half. There were only 47 cases, so a single case shifts the share by about two percentage points. Of the 1,073 sandbagging cases, the auditors found 71–81 per cent.

The auditors caught blatant cheating. OpenAI’s o3 rewrote the scoring function so that every solution was accepted, and every auditor identified this as cheating. Claude Sonnet 4 used a numerical condition that the task explicitly prohibited, but no auditor considered it cheating. GPT-6 Astra interpreted it as an implementation error.

Wider coverage brings more false alarms

When the auditors were tuned to find 80 per cent of cases, the best models wrongly flagged 2.5–3 per cent of harmless logs. At that setting, Opus 4.8 wrongly flagged 40 per cent on reward hacking and Gemini 3.8 Flash 63 per cent.

In Hall’s example, 100,000 agent sessions a month and a 3 per cent false-alarm rate would produce nearly 3,000 false warnings and roughly 2,000 real hits. Hall fears that human reviewers will become desensitised and let real cases slip through.

In Shanghai Jiao Tong University’s R-Judge dataset, agents carry out everyday tasks. On this dataset, Opus 4.8 found nearly 96 per cent of dangerous actions at a 5 per cent false-alarm rate. When the auditors had to find 95 per cent of dangerous actions, the false-alarm rate stayed below 10 per cent for almost all models and below 4 per cent for Gemini 3.8 Flash. At 99 per cent, the rate rose to 15 per cent even for the best model, GPT-6 Astra, and to 81 per cent for Gemini.

Newer or more expensive models were not consistently better. On reward hacking cases, GPT-5 beat GPT-6 Astra by a wide margin, even though it cost about an eighth as much per 1,000 logs. Hall suggests that Opus 4.8’s weak result on reward hacking may point to a preference for its own maker’s models, because the logs come mostly from Claude, but adds that this cannot be proven. Combining models did not clearly improve the results either.

Hall calls the experiment a pilot. For comparability, Hall used METR’s original prompt almost unchanged, and with that prompt Fable 5 and Opus 5 refused to carry out the evaluation at all.