Daniel Lenton of unify published a preprint on arXiv on 10 November describing the Continual-ARC benchmark. The benchmark measures whether an AI agent saves effort on recurring tasks. The study has not been peer reviewed.

The benchmark uses visual puzzles from ARC-AGI-1. The agent has to infer, from a few examples, a rule that turns one grid into another. A task sequence contains 200 task instances, and 25 rules return after gaps, each time with new input. An extra example and a wrong answer each cost one unit; an unsolved task costs the full budget of 16 units.

At the end, the frozen agent solves new versions of familiar tasks and 100 unseen tasks. The former measure memory, the latter generalisation. The score is the fraction saved compared with the same agent whose memory is wiped before every task.

Harnesses remembered only when instructed

A harness is a program that controls an agent: it gives the model tools and runs its steps. Lenton ran three harnesses on GPT-6 Luna with their own memory features switched on: Prime Agent, Hermes Agent and OpenClaw. Memory produced no measurable savings at any reasoning effort.

Prime Agent kept an entry for later sessions only if the agent marked it as global. The agent never did. Hermes Agent’s skills contained general advice that did not reduce spending. OpenClaw carried nothing over to persistent memory.

When prompted to save liberally, only Prime Agent saved measurably: 0.42 on familiar tasks. The harnesses saved 0.62–0.72 on familiar tasks when their instructions told them to write a Python program for each task, check it against the examples and reuse it. Lenton’s reading is that the harnesses were capable of a good result, but neither the model’s judgement nor the harnesses’ default prompts steered the agent to save the right things.

Training the weights helped on new tasks

No harness-based method measurably improved results on unseen tasks. During the task sequence, Lenton trained the weights of the open-weight model Qwen3.5-9B to imitate GLM-5.3 Flash. With this method, DAgger, the model saved 0.29 on new tasks.

The table shows Qwen’s results with each method.

Fraction saved compared with the empty-memory version; higher is better. The program library stores a program checked against the examples for each task. Source: Continual-ARC preprint, ARC-AGI-1.

Apart from training, the figures for new tasks do not differ statistically from zero, and nor do the Prime Agent rows, even on familiar tasks.

Part of the gain from training came from learning to use the harness. Untrained, Qwen solved 18 of 100 new tasks under Prime Agent and 31 in a simple loop without tools. The trained model solved 47–52, so it also learned from the tasks themselves, Lenton said. DAgger needs a stronger teacher model whose reasoning tokens are available.

Lenton wrote on X that the harnesses remembered nothing of value. The preprint limits this finding to harnesses run with default settings. Most runs used relatively small models and the easiest of the three ARC tracks. How harnesses learn in other domains remains open, Lenton said. Lenton’s employer, unify, is itself developing continual learning based on weight training.

The code is on GitHub under the Apache-2.0 licence, and the run logs are under CC BY 4.0. Anyone can have their own solution checked against unify’s secret task set by paying for the model calls in advance.