Researchers from Peking University and Meituan describe the MCPHallu benchmark in a preprint published on arXiv on 5 November. The benchmark breaks down the kinds of error that trip up AI agents. The agents use tools through MCP, a protocol that lets an agent find and call tools on different servers during a task. The study has not been peer reviewed.

Every task contains a trap

The benchmark has 358 tasks across five domains. Agents complete them in isolated Docker containers using 36 production MCP servers. Each task is built so that one of four error types is the most likely. In Branch Collapse tasks, the agent must choose its next step based on data returned by a tool. Unreachable Goal tasks cannot be completed with any of the available tools. Tool Misuse tasks offer similar tools built for different purposes. In Context Forgetting tasks, the agent must remember a value retrieved at the start across distracting steps.

The researchers carried out 5,012 runs with 14 models, allowing at most 20 tool calls per task. Scores (0–1) rely mainly on automated checks, such as the final state of the environment and which tools were called. A GPT-4o LLM judge accounts for 20–30 per cent. When the researchers used Claude Sonnet 4.5 as the judge instead, scores changed by at most 0.016 on average.

Agents fail to recognise impossibility

The models averaged 0.642, and 47.8 per cent of runs passed. Claude Opus 4.6 scored best (0.728). Human experts scored 0.88, but only on a sample of 40 tasks. The half-width of the confidence interval for overall scores is about 0.03–0.04, while adjacent models differ by 0.028 on average.

Unreachable Goal was the hardest error type for 12 of the 14 models. Of the runs in which the agent made no tool calls, 65 per cent passed. With 1–3 calls, 27.9 per cent passed. No run with four or more calls passed. The share of final answers claiming success kept rising up to runs with 8–15 calls. The authors interpret this to mean that extra calls amplify hallucination rather than helping the agent recover. The two hardest error types stayed the same even when the agent planned before acting or received a prompt that prepared it for impossible tasks.

In Tool Misuse tasks, 84.7 per cent of runs passed when the agent used only the correct tool. When the agent called a misleading tool as well as the correct one, the figure fell to 32.3 per cent.

On average, a model’s rank varied by 6.9 of 13 possible places between error types. Scores on Branch Collapse and Unreachable Goal tasks did not correlate (r = −0.12). The authors therefore recommend reporting results by error type.

How the tasks were generated and how accurately they were classified

Claude Sonnet 4.6 generated the tasks, and the same model filtered them in Claude Code. The authors plan to add models from other developers as task generators. The researchers examined 80 failed runs. In 67 of them, the error type designed into the task dominated. In most of the exceptions, an agent that had got stuck claimed that a feasible task was impossible.

MCPHallu is available on GitHub (J1zz/mcp_hallu).