Google: monitoring tool flagged 87 per cent of executed harmful instructions in tests
Google said its Agent Anomaly Detection service flagged 87 per cent of the harmful instructions agents carried out across five benchmarks written by others, and raised false alarms in 2–16 per cent of benign sessions. When an instruction planted in memory made the agent act, the service caught 4 of 18 cases.
In a developer blog post on 12 November, Google described how its monitoring service for AI agents performed on public security benchmarks. Others wrote the benchmarks, but Google ran them itself.
Google introduced Agent Anomaly Detection for Gemini Enterprise Agent Platform in September. The service reads agents’ trace and log data after the fact, so it adds no latency. It does not block actions. Instead, it flags for review any session in which an agent acts outside the limits set for it.
The tests covered agent goal hijacking, which OWASP puts at the top of its risk list for agentic applications. An attacker hides an instruction for the agent in, for example, an email or a file that the agent reads with a tool.
Five benchmarks, one model
The benchmarks were AgentDojo, InjecAgent, AgentDyn, Agent Security Bench (ASB) and AgentHarm. In four of them the harmful instruction reaches the agent through a tool; in ASB it is planted in the agent’s long-term memory. In AgentHarm the user asks for the harmful action directly.
The agent used the gemini-2.5-flash model in the benchmarks’ simulated applications. Each benchmark’s own Python check classified a session by the end state of the applications, for example whether money left an account. Deterministic code, not a language model, made the judgement. There were 1,774 sessions. Each agent first ran at least 300 benign sessions so that the service could learn its normal behaviour.
The model did not carry out about half of the harmful instructions it read. It refused three in four direct requests, and ignored or failed to carry out about 30 per cent of the hidden instructions.
What the service detected
The service flagged 87 per cent of the harmful instructions that were carried out: 85 per cent in AgentDojo, 87 per cent in InjecAgent, 99 per cent in AgentDyn and 81 per cent in AgentHarm. Including sessions in which the agent did not read or follow the instruction, it flagged 45–62 per cent of harmful sessions, depending on the benchmark. The service returned no result for about 6 per cent of sessions. Counting these as missed lowers the shares by 1–2 percentage points.
The service performed worst in AgentDojo’s Slack application, where it flagged 63 per cent of executed instructions. Google said many instructions there asked for a single small action, such as opening a page.
In office tasks, false alarms occurred in 2–4 per cent of benign sessions. In AgentHarm the share was 16 per cent. Google said this is because the agent posts on social media, runs terminal commands and downloads files even in AgentHarm’s benign tasks. Google cautioned that the tests used one model and simulated applications, and that the benign tasks are likely simpler than production use, so the false alarm rate in production may differ.
Weaknesses: memory and new agents
When an instruction planted in long-term memory made the agent act, the service caught 4 of 18 cases. Some of its methods need a history of a few hundred sessions. In AgentDyn the service flagged 33 per cent of executed instructions after about 220 benign sessions, and 99 per cent after 360. Of the instructions hidden in tool output that the agent did not follow, it flagged 11–25 per cent.
Alongside the service, Google recommends measures that block harmful actions in advance, such as checking tool calls and requiring human approval for payments. Google said the service needs complete data, so users must switch on tracing and logging, including prompt and response content, in ADK (Agent Development Kit), its agent development tool.
Agent Anomaly Detection is in Private Preview, and users can request access on the service’s overview page.
Read more: How an agent works · Agent map