Nous Research’s own test: fine-tuned Qwen repeats China’s official line less often
Written by AI
Nous Research fine-tuned Alibaba’s Qwen3.8-27B into Arete-27B. In the company’s own evaluation, the share of answers that adopted the Chinese state line fell from 22 per cent to about 1 per cent.
Nous Research said on 6 November, on its blog and its X account, that it had post-trained a Chinese open-weight model to reinforce Western values. Chinese open-weight models are the strongest available, the company said, but their weights carry state-aligned behaviour.
Nous said it had found that Qwen3.8-27B refused, evaded or answered from the Chinese Communist Party’s perspective when asked about politically sensitive topics. The company does not claim to have fixed all such behaviour.
A language model judged the answers
Nous’s evaluation contains 803 prompts across seven categories, including freedom of expression, democracy and privacy. Each prompt draws on a dated source article, which supplies the party’s official vocabulary on the topic and two to four facts that the answer must mention.
The judge was the language model Grok 4.7. It did not know which model had written an answer. It marked each fact as mentioned, missing or contradicted, and quoted the passages where an answer adopted the official line. Quoting the official line in order to identify it did not count as propaganda. Nous refined the judging instructions in a 25-prompt trial and also scored the trial by hand.
For the training data, Nous collected responses from the teacher model Inkling alongside the base model’s own. The fine-tuning used about 52.8 million tokens, and the model went through the data twice. Nous used no reinforcement learning or preference optimisation.
Propaganda fell, but Arete trailed its teacher on facts
The table shows the mean fact score on a scale of 0 to 5 and the share of answers in which the model adopted the state line. The figures come from Nous’s own measurement, with a single language model as judge.
| Model | Fact score (0–5) | Propaganda |
|---|---|---|
| Inkling (teacher) | 3.96 | 0.3% |
| Mistral Large 4 | 3.65 | 0.3% |
| Nemotron 3 Ultra | 3.22 | 0.6% |
| Gemma 4 31B | 2.58 | 0.5% |
| Arete-27B, reasoning on† | 3.07 | 0.6% |
| Arete-27B, reasoning off* | 2.98 | 0.9% |
| GLM 5.3 | 3.10 | 15% |
| DeepSeek V4.1 Flash | 2.39 | 16% |
| Qwen3.8-27B* | 2.32 | 22% |
Source: Nous Research. Average of five runs. Temperature, the setting that controls how random the answers are, was 0 on rows marked * and 0.6 on the others. Reasoning was off on rows marked * and for Nemotron 3 Ultra, GLM 5.3 and DeepSeek V4.1 Flash. † Preliminary result, two of five runs.
At the same temperature of 0.6, Arete’s fact score was 2.88 and Qwen’s 2.24. Arete gave 216 answers that scored zero, against 359 for the base model. The biggest change was in freedom of expression, where the share of answers containing propaganda fell from 60 per cent to 1 per cent.
On Nous’s internal Hermes Bench benchmark, Arete scored 65.60 per cent and the base Qwen 65.77. On Terminal-Bench-2.1, Arete scored 70 per cent and Qwen 67. Nous ran the tests with its own settings and does not claim credit for the improvement.
Nous reads the Chinese model’s bias as shallow and added late. ‘The values in any open model are only ever one fine-tune away,’ the company writes.
Nous’s blog post links to downloads of the model weights, the training data and the evaluation. The evaluation includes the prompts with their source articles, the judge prompt and the scoring scripts, so it can be run on other models too.
Read more: Models