AI Aulis

AI NEWS IN ENGLISH · 21 November 2026

The most important AI news every morning. AI writes and checks the stories against the original sources – check important facts in the sources.

← Issue 21 November 2026Paper

Hugging Face: pi adapter raised Qwen from 39 to 65 per cent on hard SWE-bench Pro tasks

AI-generated contentWritten by AI

huggingface.co, thechenabtimes.com · 2 sources · Sources published: 20 November 2026

Hugging Face used reinforcement learning to train a separate adapter for Qwen3.8-27B for each of three harnesses, and published the recipe. In the company’s own tests, each adapter improved results in its own harness. On tasks the base model also solved, the adapters cut tool calls by more than half.

On 20 November, Hugging Face wrote in a blog post that it had trained a separate adapter for Qwen3.8-27B for each of three harnesses: pi, OpenCode and Hermes Agent. A harness is the program that controls an agent, giving the model its tools and running its steps. An adapter is a small set of extra weights made with LoRA. It is trained on top of the model weights without changing them.

Models are tuned for general performance, not for a particular harness or set of tasks, the company said. So Hugging Face trained the model with reinforcement learning inside the harness itself rather than in an imitation of it. A verifier that tests the solution gave the reward, and sparing tool use earned a small bonus.

When screening tasks, the company favoured those the base model solved on some attempts but not all. The GRPO method it used compares attempts at the same task against each other, so a task that always or never succeeds teaches the model nothing. Of the 442 scored tasks, 171 met this condition.

In the company’s own measurements, each adapter improved results in its own harness on both benchmarks. The table shows the share of tasks solved with the model’s thinking mode switched off.

Harness Benchmark Base model Own adapter Super-adapter
pi SWE-bench Pro 40.0% 64.0% 46.0%
Hermes Agent SWE-bench Pro 29.4% 43.1% 47.1%
OpenCode SWE-bench Pro 35.3% 51.0% 45.1%
pi Terminal-Bench 2.1 40.0% 56.5% 48.2%
Hermes Agent Terminal-Bench 2.1 44.7% 58.8% 55.3%
OpenCode Terminal-Bench 2.1 47.6% 52.4% 52.4%

Source: Hugging Face. The table covers tasks evaluated in every run (50–51 of the hardest SWE-bench Pro tasks, 82–85 from Terminal-Bench 2.1). One attempt per task.

The table cannot be used to compare the harnesses with one another, because they were run with default settings. For example, pi does not keep earlier reasoning steps. Across all 51 of the hardest SWE-bench Pro tasks, the pi adapter raised the score from 39.2 per cent to 64.7 per cent.

The super-adapter merges the three adapters into one that works in all three harnesses. With thinking mode off, the harness’s own adapter led in four of the six comparisons, and the company recommends it in that case. With thinking mode on, the super-adapter was the strongest of the models the company evaluated. Its merge weight was chosen using Terminal-Bench 2.1, so the Terminal-Bench figures are slightly optimistic, the company said.

On tasks that both the base model and the adapter solved, the adapters made 56–72 per cent fewer tool calls on SWE-bench Pro and 53–70 per cent fewer on Terminal-Bench 2.1.

It is still unclear what the model actually learned, the company said. With thinking mode off, the adapted model writes a short passage before a tool call that may resemble reasoning. Hugging Face calls this reasoning leakage but still considers the results valid. At the same token budget, the adapters clearly outperformed the base model, and they matched the base model’s thinking mode with fewer tokens, the company said.

On AutomationBench, which measures business automation, the adapters ran on the benchmark’s own agent loop and stayed close to the base model. The differences were within noise, the company said, except perhaps for the Hermes adapter, which scored 17.7 per cent against the base model’s 23.0 per cent.

The adapters and training tasks are in a Hugging Face collection. The recipe builds on the AsyncGRPOTrainer class in the TRL library, together with OpenEnv and Harbor. Training one adapter took two Nvidia B200 GPUs.

Previous coverage

Report an error in this story