AI Aulis

AI NEWS IN ENGLISH · 20 November 2026

The most important AI news every morning. AI writes and checks the stories against the original sources – check important facts in the sources.

41 stories · next issue tomorrow morning · How the paper is made →

← Issue 20 November 2026Paper

Preprint: agent evolution stalls when evaluation cannot tell improvement from noise

AI-generated contentWritten by AI

arXiv, huggingface.co, github.com, wkqdzkd.github.io, franklineh.com, alphaxiv.org, roboaidigest.com, microsoft.com · 9 sources · Sources published: 3 July–19 November 2026 · Some source publication dates are unknown

A preprint on arXiv tested seven agent evolution methods on six models. Across results in nine papers in the field, a new method’s median lead over its strongest rival is 3.4 percentage points, which the researchers say is smaller than the difference between two runs of the same method.

A research team led by ShanghaiTech University published the study as a preprint on arXiv on 17 November. It has not been peer reviewed. The paper looks at agent evolution that leaves the large language model’s weights untouched. A program writes new prompts, skills, memory or a harness for the agent. The harness is the program that controls the agent: it gives the model its tools and runs its steps. The candidate with the best test score is kept.

The authors call their idea the Resolution Hypothesis. It holds that evolution stalls not because of poor proposals but because evaluation fails to recognise better ones. An agent’s performance varies from run to run, so a small real improvement disappears in the noise.

The researchers reviewed 171 results from nine papers in the field, and none came with an error bar. In their own experiments they carried out 1,030 evolution runs. The table shows how many runs are needed to detect a 3.4-percentage-point difference. It also shows how often a single run correctly names the best of four methods.

Benchmark Runs per method Single run picks the best
SpreadsheetBench 5 67%
OfficeQA 10 51%
ALFWorld 19 73%
Terminal-Bench v2.1 23 33%

Source: arXiv 2611.23068. Run counts at 80 per cent statistical power, with each run’s result evaluated once. A random guess picks the best 25 per cent of the time.

The result does not show that the methods are equally good. It shows that comparing single runs is not enough to tell them apart.

The authors said the Terminal-Bench candidates included genuinely useful skills. Even so, score-based selection picked the better of two candidates only about half the time. Selection errors compounded round after round, so scores rose while the skills did not improve. The authors did not test this on other benchmarks.

The largest published gains are often explained by loopholes in evaluation, the authors wrote. The SpreadsheetBench grader read formula cells without calculating them, so every formula was treated as empty. As a result, GPT-5.4 nano’s score rose by 29 percentage points simply because it was told to write final values instead of formulas. When the same evolved versions were scored with a corrected grader that calculates formulas before comparing, their advantage over the baseline shrank from 31.1 percentage points to 6.7. A single instruction did not, however, reproduce the gain on every benchmark.

The authors recommend reporting both evaluation noise and run-to-run variance. They also say the proposer and the selector should be tested separately. The results apply only to evolution that leaves model weights unchanged. The authors do not rule out that future models will be less noisy. They plan to release the data if the paper is accepted.

Report an error in this story