Researchers from Chinese universities and a research institute, the University of California, Berkeley, and Simple Agent Lab published a preprint on arXiv on 5 November. It has not been peer reviewed. They examined whether higher test scores from automatic evolution mean a better AI agent.

The methods evolve the harness: the program that steers the agent, gives the model its tools and runs its steps. They modify the harness’s prompts, skills or code based on test results. The model weights stay unchanged.

Every version lost tasks

The researchers evolved two harnesses with four methods (AHE, HyperAgents, A-Evolve and GEPA). In MiniSWE a method could also change the source code; in Codex, only the outer layers. The tasks came from τ³-Banking, which mimics customer banking, and the coding benchmark Terminal-Bench 2. The researchers compared each of the 16 evolved versions with the seed version over eight runs.

Reliability, the average success rate, improved in 11 versions. Coverage, the share of tasks solved at least once, improved in ten. Yet every version failed entirely on some tasks that the seed version had solved. Versions lost 2–10 tasks and gained 1–15 new ones. Some of the τ³-Banking changes reversed direction between the first and second halves of the runs, the authors said.

In the table, the version evolved with HyperAgents raised reliability by 5.2 percentage points even though it lost ten tasks. Figures from a single setting cannot show which method is better in general.

τ³-Banking, MiniSWE. Cost is the estimated running cost per task across eight runs. AGE is the authors’ own method.

Of the 11 versions that improved reliability, eight cost more per run than the seed version. A more expensive version can still work out cheaper if it needs fewer attempts. The MiniSWE version evolved with AHE solved 56.7 per cent of τ³-Banking tasks in four runs; the seed version solved 54.6 per cent in eight runs, at a higher cost.

In the run logs, failures involved, for example, a wrong date, the agent stopping work midway, or a solution that worked only in the agent’s own environment.

The highest score did not predict later progress

The highest-scoring version did not yield the largest mean improvement in further evolution in any of the four comparison pools. Targeted changes improved reliability more than general reminders in 15 of 16 cases. When changes were combined, however, their benefit could change.

When the researchers tried the versions on other benchmarks without further modification, coverage usually fell on general tasks but rose on coding tasks with the Codex harness.

Building on their findings, the authors created the EvoAudit workflow and used it to build AGE, a variant of A-Evolve. In the authors’ own tests, AGE improved reliability and coverage over A-Evolve in all four settings. The authors stress that neither test scores nor expert support for EvoAudit’s suggestions show that an agent is safe to use.

The study’s code is in the RSIHub repository on GitHub.