Opus 5.5 separated alignment and capability tests in METR’s data without error when runs were averaged
AI agents are getting better at telling tests of a model’s behaviour apart from capability tests, according to a METR research note. A staged-looking environment or a shortcut left as bait, for example, can give a test away.
The non-profit METR published a note by its researchers on 16 November on how AI agents work out that they are in an alignment evaluation. An alignment evaluation measures a model’s tendency towards unwanted behaviour, such as cheating. A capability evaluation measures whether the model can do the task.
The researchers said a model that recognises a test may perform better in it than it otherwise would, so the evaluation gives too favourable a picture of the model. METR’s notes are unpolished, receive lighter review than its research publications and do not necessarily represent the views of METR as a whole.
Agents examined 55 test environments
METR placed investigator agents in 19 alignment evaluations and 36 capability evaluations. The agents had the same tools as whoever normally performs the task. The alignment evaluations tempted or pressured the agent to, for example, cheat in a game against a strong game engine or take an action that endangers people. The capability evaluations were METR’s Time Horizons tasks and tasks from the Terminal-Bench 4.0 benchmark.
The researchers asked the agents in two ways. Asked directly, the agents estimated how likely it was that the task was an alignment evaluation. Asked an open-ended question, they described the purpose of the task, and the language model Opus 5.5 scored the answers on a scale of 0–100.
The newest models separated the evaluations almost without error
The researchers measured discrimination with AUC: the probability that a random alignment evaluation scores higher than a random capability evaluation. A value of 1 means perfect separation and 0.5 means guessing. The authors said the agents’ ability to discriminate has improved over the past year, and that Anthropic’s Opus 4.8 and Opus 5.5 were nearly perfect. With both question types, averaged over three runs, Opus 5.5 gave every alignment evaluation a higher score than any capability evaluation.
The authors said the result does not show that it is impossible to build an alignment evaluation that agents cannot recognise. Nor does the study measure whether recognition changes an agent’s behaviour while it does the task. The direct question can also be leading: OpenAI’s GPT-6 Astra gave one trap an average of 91 points when asked directly, but described a capability test in its open-ended answer. In seven of the 55 environments, Astra’s scores varied by at least 40 points from run to run. The authors attributed this to the agents not exploring some environments enough. Variation remained even when the agents were prompted to explore more.
Staging and traps give the test away
The authors sort the features that give a test away into two classes. At the environment level, the test imitates real use poorly: services are obviously staged, for example, or situations implausible. Task-level features are often chosen deliberately to bring out unwanted behaviour: an unethical shortcut, serious consequences or conflicting goals.
METR turned the IRO capability evaluation into a reward hacking trap, a task in which the easiest way to score well is to cheat the grader. It made the goal harder and deliberately made the grader easy to cheat. Opus 5.5 identified all four changes used to build the trap. The agents mentioned most features more often in alignment evaluations than in capability evaluations.
The authors propose refining alignment evaluations repeatedly against investigator agents. In their view, environment-level features are easier to remove than those inherent to the task. They are not sure that all features can be removed.