Baidu’s ERNIE team described its automated research system, ERNIE Evolve, on its research blog on 9 November. The company calls the system its first step towards recursive self-improvement (RSI), in which AI improves the software used to develop AI.

Agents experiment, a separate evaluator decides

The system takes a repository, an optimisation target and evaluation criteria as input. A Director agent chooses the research direction and starting version based on earlier experiments and a knowledge base built from research papers and open-source code. In parallel research lines, the Selector picks the code version and hypothesis to test, and the Executor modifies the code in an isolated environment.

The Evaluator and the Reviewer decide whether a change is accepted. The Evaluator is an immutable measurement program outside the repository and is not visible to the agents. An improvement is accepted only if it exceeds measurement noise. The Reviewer rejects changes that exploit loopholes in the evaluation (reward hacking).

Small margins over the leaders on public tasks

Baidu tested the system on three public tasks: improving a training algorithm with a five-minute training budget on a single GPU (NanoChat AutoResearch), speeding up training (NanoGPT Speedrun) and optimising GPU kernels (SOL-ExecBench). The table shows the results Baidu reported.

Source: Baidu. The NanoChat comparison is the result reported by rekursiv.ai; the others come from official leaderboards.

MarkTechPost notes that the margins are small and that Baidu reported the results itself. On SOL-ExecBench, ERNIE Evolve fell short of the top result.

Sources give different figures for competitors’ results. In Baidu’s measurement (Evaluation Stack v1.1), Recursive scored 0.676 on SOL-ExecBench and Tencent’s Hyra-1.0 scored 0.736, but MarkTechPost said the systems’ self-reported results are 0.754 and 0.771. The publication also said their Speedrun times of 77.5 and 76.4 seconds were compared with an older record of about 79.7 seconds, so they are not directly comparable.

ERNIE-5.5-Flash inference got faster

Baidu also used the system to optimise inference, or running the model, for ERNIE-5.5-Flash, which is still in development. Engineers had already optimised most of the baselines, and the system’s changes had to pass the same load tests and correctness checks as theirs. According to Baidu:

• On the Kunlunxin P800, the model was not yet making full use of existing optimised execution paths. The first round enabled them and raised generation speed to 282.9 per cent of the baseline; the fourth round raised it to 623.6 per cent.
• Reinforcement learning training requires deterministic inference: outputs had to stay bit-for-bit identical regardless of which other requests were processed in the same batch. Its throughput rose by 33.7 per cent in about three days, with determinism preserved.
• Single-request decoding speed on GPUs rose by 14.9 per cent with accuracy unchanged. Some useful changes sped things up by only 0.3–0.5 per cent on their own, which is below measurement noise.

MarkTechPost reported that Baidu has not released the system itself, only the optimised code and measurement logs for the public benchmark tasks on GitHub (ernie-research/ERNIE-Evolve-Results). Baidu next plans to use the method to optimise training systems. The company said RSI must stay under human control and remain interruptible.