Researchers from institutions including the University of Science and Technology of China and the University of California, Berkeley published a preprint on arXiv on 5 November. According to the preprint, benchmark scores can hide the tasks on which a self-evolving AI agent starts to fail. The study has not been peer-reviewed.

In self-evolution, the model weights stay unchanged. The changes go into the harness, the program that controls the agent, gives the model its tools and runs its steps. The evolution method picks an earlier version and runs tasks with it. It then looks for the causes of failures, edits the harness and decides whether to keep the change.

The researchers compared four methods: AHE, HyperAgents, A-Evolve and GEPA. In the MiniSWE harness, a method could change the source code. In Codex, it could change only outer parts such as prompts and skills. The benchmarks were τ³-Banking for banking tasks and Terminal-Bench 2 for coding. Every run used GPT-5.4. Each seed and final version was run for eight rounds.

Reliability is the share of successful attempts out of all attempts. Coverage is the share of tasks the agent solved at least once.

Scores rose, tasks dropped out

Reliability improved in 11 of 16 setups and coverage in 10. In four setups, the two metrics moved in opposite directions. All 16 versions solved tasks the seed version had never solved. At the same time, they failed tasks the seed version had solved. On τ³-Banking, the first and second halves of the rounds gave partly conflicting results. The authors do not interpret every change as skill gained or lost.

The table shows τ³-Banking results relative to the seed version. HyperAgents raised MiniSWE’s reliability by 5.2 percentage points even though it lost ten tasks the seed version had solved.

Source: Table 10 of the preprint, eight rounds. AGE is the authors’ own method.

The table does not support an overall ranking of the methods, because each combination was run as a single lineage.

Eight of the 11 versions that improved reliability cost more per attempt than the seed version. A version that costs more per attempt can still be cheaper overall. On τ³-Banking, AHE’s MiniSWE version solved 56.7 per cent of tasks in four rounds, while the seed version solved 54.6 per cent in eight. The AHE version cost less.

On other benchmarks, coverage fell for seven of eight versions on τ²-bench, which tests general tasks, but rose for seven of eight on the coding benchmark DeepSWE v1.1. Reliability rose for all four MiniSWE versions on τ²-bench but fell on DeepSWE.

The highest-scoring version did not produce the largest mean improvement in further evolution in any of the four experiments. A stricter admission rule kept the best candidate in 6 of 16 repeated experiments; the original rules kept it in 15.

Revised method lost fewer tasks

The authors turned their findings into a workflow called EvoAudit and added the four changes it proposed to A-Evolve. The resulting method, AGE, improved reliability and coverage over A-Evolve in all four setups. In three of them, it lost fewer tasks the seed version had solved. Neither benchmark scores nor expert review show that an agent is safe to use, the authors said.

The authors’ RSIHub code is available on GitHub.