Zihan Dong (Lenovo Group), Qishi Zhan (Marquette University) and Minxuan Hu (Cornell University) published their study on arXiv on 10 November. It is a preprint and has not been peer reviewed.

The researchers re-scored published runs of eight benchmarks. Most agent benchmarks check a run against a task-specific answer key, such as tests derived from a reference solution, and record a single bit: pass or fail. The authors said one bit is not enough to separate the leading agents and gives no extra credit for beating the reference solution.

Gaps are smaller than the errors in the tests

On SWE-bench Verified, the top ten agents’ results fall within 3.6 percentage points, and the largest gap between neighbouring agents is 1.4 percentage points. The ten form a single tier: a group the benchmark cannot statistically separate from the leader.

Stricter SWE-ABS tests rejected 17.7 per cent of the median agent’s accepted fixes. Among the top 30 agents, the stricter tests reversed the order in eight of the 18 neighbouring pairs with a non-zero gap. Removing the same share of passes at random reversed about as many. The order of neighbouring agents is therefore noise, the authors said. The analysis assumes the stricter tests are error-free, and the researchers assessed false rejections only through simulation.

Beating the reference goes unnoticed

In ALE-Bench’s 40 optimisation contests, the official contest score splits the top ten into three tiers. When the same runs were scored as a pass or fail against the median human, all ten fell into one tier, even though 47.8 per cent of runs beat the median human. When the researchers’ graded scoring gave extra credit for exceeding the reference, the top ten split into three tiers again.

Raising the threshold on the same ALE-Bench tasks did not solve the problem. Using the contest winner’s result as the threshold split the top ten of models released after 2025 into two tiers, but 89 per cent of earlier models scored zero on every task. The researchers estimated some of the later models’ release dates retrospectively. They did not simulate a new version that would also add new tasks.

Continuous scoring does not always help: on MLE-bench the answer key alone already separates the top, and on METR’s tasks few runs beat the reference.

The researchers re-ran 13 Terminal-Bench 2.0 tasks with four models, three times each. Three models tied on passes, but on the portfolio-optimisation task the median speedup of claude-opus-5’s solutions was 10.9-fold, against 1.3-fold for the other two. Because there were only three repetitions, the researchers consider the overall separation of the models borderline.

Proposal: acceptance, quality and cost reported separately

The researchers propose first accepting a run with a stricter check. A measured quantity, such as running time, is then scored as a log ratio: the baseline scores zero, the reference solution one, and the cap is three. The baseline and reference are locked and dated so that versions of the benchmark remain comparable.

Cost is reported separately. Four priced agents in the Terminal-Bench 2.0 top ten tied on passes, but their median cost ranged from $0.44 to $2.47 per run.

The scoring method suits only tasks with a measurable quantity. Of Terminal-Bench 2.0’s 89 tasks, 18 compare a number against a threshold. Small samples from SWE-bench Verified, OSWorld-Verified and τ²-bench contained no such tasks, but because the samples were small, the authors cannot rule out a share as high as 10–15 per cent.