Vals AI test: 23 AI models explained the law better than they pointed people to local help
Written by AI
None of the 23 models met 80 per cent of the criteria, which Vals AI treats as the threshold for a reliable first adviser. The models met 78–87 per cent of the criteria on explaining the law in general, but only 30–50 per cent of those on naming local sources of help.
US AI evaluation company Vals AI and investment organisation NextLadder Ventures published the Access to Justice Benchmark on 10 November. It measures how well an AI model can act as the first adviser to someone handling a legal problem without a lawyer. The scenarios are typical of those faced by low-income Americans.
Simulated client and automated grader
The benchmark’s 252 scenarios are based on real questions put to legal aid organisations. Six legal aid experts adapted them to the law of six states and wrote an average of 15.4 assessment criteria for each. Most of the scenarios concern family law (110) and housing (65).
The model under test talked with a client played by GPT-5 for up to five turns and could search the web. The client gave background details only when asked. An automated grader, Decider v1.2, scored each conversation: the score is the share of criteria met. About a quarter of the criteria measure whether the model asks questions to establish the relevant facts.
No clear winner at the top
The table shows some of the models, along with Vals AI’s estimate of whether the provider’s free or paid version uses the model.
| Model | Criteria met | Vals AI’s estimate |
|---|---|---|
| Qwen 3.8 Max | 76.37% | – |
| Claude Sonnet 5.5 | 74.94% | free version |
| Wonder Full | 74.68% | – |
| Claude Opus 5.5 | 73.77% | paid version |
| Grok 4.7 | 73.69% | paid version |
| Grok 4.6 | 71.34% | free version |
| Gemini 4 Argon | 68.48% | paid version |
| Gemini 3.8 Flash | 67.06% | free version |
| GPT-6.1 Sol | 65.84% | paid version |
| GPT-5.6 Luna | 65.56% | free version |
Source: Vals AI, 10 November 2026. Share of expert criteria met across 252 scenarios, graded by Decider v1.2. Dash: model not included in the free versus paid comparison.
The differences between the top three fall within the margin of error, Vals AI said, so the ranking shows no clear leader. Longer answers generally scored higher (correlation 0.50). Qwen 3.8 Max wrote the longest final answers, about 3,900 words on average. Vals plans to try scoring that takes length into account.
Free-version models averaged 69.7 per cent and paid ones 70.4 per cent. The grouping is Vals AI’s estimate. The models were tested through the API, so the results only approximate use of a chatbot app. Only at xAI was the paid model clearly ahead: by 2.4 percentage points on Vals AI’s page and 2.3 in the press release.
Models explain the law, but local help stays generic
The models met 78–87 per cent of the criteria on explaining the law in general and 30–50 per cent of those on specific sources of help, such as a local legal aid programme or a court self-help centre. Of the 23 models, 21 mentioned a national directory, such as LawHelp.org or the 211 service, in at least 70 per cent of their final answers. A local legal aid organisation is far more useful than a national directory, according to Vals AI’s experts.
Of the web pages the models read, 10 per cent were legal aid and advice sites and 42 per cent were statutes, court rules and case law. In Vals AI’s reading, the models read enough law to explain it but rarely open the pages where the next local step could be found.
Asking more questions did not mean asking the right ones. GPT-6.1 Sol ended its first reply with a question in 88 per cent of conversations, yet ranked 18th of 23 models at establishing the relevant facts (68.1 per cent).
By topic, the models did best on housing (75.4 per cent) and worst on public benefits (57.3 per cent), which had only five scenarios.
The results are Vals AI’s own measurements: BenchLM reproduces the table without rerunning the benchmark, and Unite.AI summarises the release. The results and methodology are on Vals AI’s page. Vals keeps a further 252 scenarios private for validation.