The changelog of Joel Niklaus’s Harness Leaderboard describes fixes recorded on 6 and 7 November: not every harness received the models’ own limits. A harness is the program that drives an agent, giving the model its tools and running its steps.

Niklaus published the leaderboard on Hugging Face on 5 November 2026. It covers nine harnesses: Claude Code, Codex, Pi, OpenCode, Hermes Agent, mini-swe-agent, Kimi Code, Qwen Code and OpenClaw. The models are GLM-5.3-Flash and DeepSeek-V4.1-Flash at low and high reasoning effort, and Flint at its default effort. The benchmarks are Terminal-Bench 2.1 and DeepSWE v1.1.

How the leaderboard works

Each run keeps the provider, harness version and reasoning effort the same. A proxy running in the sandbox sets the reasoning effort on every model call and measures token use. Each task gets three trials. The headline score is the average of the resolve rates on the two benchmarks, with 95 per cent confidence intervals.

The leaderboard counts fetching published solutions or calling another model, for example, as a failure. The trajectories are checked with the atif-scan tool. Some can be checked only partially, so the number of detected cheating cases is a lower bound. Since 23 October, DeepSWE has run without internet access, so its results include trials both with and without network access.

What the changelog records

According to the changelog, Pi, Claude Code, Qwen Code, Codex and OpenCode did not recognise the leaderboard’s models and used their own default limits. Pi assumed a context window of 128,000 tokens, compacted its history at about 112,000 tokens and capped replies at 16,384 tokens. The models’ context windows, however, range from 262,144 to 1,048,576 tokens. Pi’s current cost per model call is lower mainly because of early compaction, the changelog says.

OpenClaw requested replies of at most 8,192 tokens. With Flint and DeepSeek-V4.1-Flash, the limit cut replies off and often ended the whole run. Kimi Code, meanwhile, was told that every model has a context window of 1 million tokens. With Flint, whose window is 262,144 tokens, 58 trials exceeded the limit and ended there. Claude Code did not see the reasoning of the GLM and DeepSeek models, because the proxy did not pass it on in the format the Hugging Face router uses.

Pi, Claude Code, Qwen Code and Codex will get the models’ own limits only for models added in future, and their current results will not be rerun. As a result, a harness’s cost per model call on a new model is not comparable with its current costs, the changelog says. OpenCode will be rerun on all models with the new version 2.2.6 and the models’ own context windows; until then the table shows results from version 1.19.0. OpenClaw will be rerun with version 2026.11.1, and the table shows its current results until the rerun is complete. The current OpenClaw trajectories are summaries without tool calls.

Also in preparation are Claude Code’s minimal mode, oh-my-pi, DeepSeek Harness, Grok Build and fx. They will join the table once they have run on all models. The trajectories and result files are publicly available in the leaderboard’s Hugging Face repository.