Preprint: GPT-5.6 Sol solved 55.5 per cent of Terminal-Bench 4.0 tasks with the HEELO harness
Written by AI
AI agents often fail when they need to apply a tool or information they have already found, according to a preprint by researchers. The researchers’ HEELO system raised GPT-5.6 Sol’s Terminal-Bench 4.0 score from 37.3 per cent with Codex to 55.5 per cent.
Researchers from the University of Illinois (UIUC), Georgia Tech and heelo.dev published a preprint on arXiv on 5 November. It has not been peer reviewed. The researchers say agents fail at the stage where they should apply a resource that is already available. They call this stage the last mile.
Applying a resource can mean, for example, deriving the arguments of a tool call from the result of an earlier call. An error has to be traced to the work that caused it. For comparison, the authors report results on the Terminal-Bench 4.0 benchmark at maximum reasoning effort: GPT-5.6 Sol with Codex resolved 37.3 per cent of the tasks and Claude Opus 5 with Claude Code 51.8 per cent.
Each resource gets its own language model
A harness is the program that controls an agent: it gives the model its tools and runs its work steps. Current solutions bring a resource within reach but leave applying it to the caller, the authors said. As a result, every harness rebuilds the same know-how.
As a solution, they propose agent-native primitives: reusable units in which a resource has its own large language model. The resource can be, for example, a tool or a code component. The model applies the resource to each task and communicates with other primitives.
The HEELO system assembles the primitives a task needs from a library into a task-specific network. It checks the result against evidence from the environment, repairs the primitives linked to an error, and admits new primitives to the library only after validating them.
The authors said the benefit comes from applying resources. In a repository-building experiment, reading or calling retrieved code components unchanged barely changed the delivery score (scale 0–1). The scores were 0.3544 and 0.3430, against 0.3469 without a library. When the components were adapted to the target, the score rose to 0.5134.
Results from the authors’ own tests
The table summarises the preprint’s main results. Some of them come from the authors’ earlier sub-studies.
| Test | Metric | Baseline | HEELO |
|---|---|---|---|
| Terminal-Bench 4.0 | resolved tasks | 37.3% (Codex, GPT-5.6 Sol) | 55.5% |
| SWE-bench Verified, 100 tasks | Pass@1 | 73.0% (mini-SWE-agent) | 80.5% |
| ToolBench | pass rate | 55.8% (DFSDT, Qwen3-8B) | 75.1% |
| GAIA-Text-103 | accuracy | 45.6% (iterative sparse retrieval) | 58.3% |
Source: the authors’ preprint. Metrics and baselines vary by row.
The figures do not show that HEELO improves results equally for all models. On Terminal-Bench 4.0, Claude Opus 5 gained the least at two reasoning effort levels, 1.5 and 1.2 percentage points, and its Claude Code configurations were the strongest baselines.
The authors’ own heelo.dev platform has 46 practical tasks, such as publishing a page and handling an invoice email. Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex each completed nine tasks on their own. Connected to HEELO through the authors’ MCP server, they completed 39 and 35.
The difference arose in tasks where the agent has to act on an account or a service, the authors said. Without HEELO, the agents scored zero in 12 of 26 capabilities. They could not read email, calendars or Notion at all, for example.
Connected runs cost more on most tasks because the agent read more tokens. However, the connected agents completed more tasks. Per completed task, they therefore took 1.9 and 2.3 times less time, and the cost was 1.55 times lower.
The project’s website is heelo.dev. The ArXivSignals service marks the study’s code as released.
Read more: How an agent works · Models