Makers say Qwen3.8-27B adapted for pi solved 62.7 per cent of a SWE-bench Pro subset
In its makers’ measurements, a LoRA adapter trained with reinforcement learning on top of Qwen3.8-27B raised the score in the pi coding agent from 29.4 per cent to 62.7 per cent on the 51 hardest SWE-bench Pro tasks. The gain has been measured only with pi.
Harness-hub published the adapter on Hugging Face on 16 November. The next day, Hugging Face chief executive Clément Delangue wrote on X that the adapter’s makers had adapted Qwen’s model specifically for the pi coding agent. Pi is a harness: a program that steers the agent, gives the model its tools and runs its work steps.
A LoRA adapter is a small set of extra weights loaded on top of the base model’s weights. The base model itself does not change. The model card says the adapter was trained in pi on 1,050 programming and command-line tasks. The reward came from the task verifier’s assessment, plus a small bonus for making few tool calls.
The model card’s table compares the adapter with the base model on the same tasks. Both ran in pi, with one attempt per task.
| Benchmark | Tasks | Base model | Adapter | Difference (percentage points) |
|---|---|---|---|---|
| Terminal-Bench 2.1 | 86 | 39.5% | 55.8% | +16.3 ± 9.1 |
| Terminal-Bench 2.1, not used for selection | 68 | 39.7% | 55.9% | +16.2 ± 10.6 |
| SWE-bench Pro, hardest 51 | 51 | 29.4% | 62.7% | +33.3 ± 14.2 |
Source: harness-hub model card. ± is the 95 per cent interval. Temperature was 0.8 and Qwen’s thinking mode was off. The card’s text gives 89 Terminal-Bench tasks; the table gives 86.
The figures apply specifically to the pi adaptation. Without the setting "api": "openai-completions", pi uses the Responses API, on which the adapter was neither trained nor tested, the card says. The setting goes in pi’s model definitions (~/.pi/agent/models.json).
The makers picked training step 160 based on results on SWE-bench Pro and 18 Terminal-Bench tasks. The SWE-bench Pro result is therefore not independent of that selection. On the 68 Terminal-Bench tasks not used for selection, the difference was 16.2 percentage points.
Averaged over all tasks, the adapter made 62 per cent fewer tool calls on Terminal-Bench and 55 per cent fewer on SWE-bench Pro. On tasks both models solved, the reductions were 48 per cent and 59 per cent respectively. In the card’s example, the adapter fixed an Ansible function in eight tool calls, while the base model searched for and ran code snippets until it hit the 100-step limit.
The adapter works only on top of Qwen3.8-27B. The card recommends running it with vLLM using the test settings, or with llama.cpp as a GGUF version. The GGUF version has not been retested. The card says its behaviour departs further from the trained behaviour the more heavily the base model is quantised.
Read more: How an agent works · Agent map