Tested against data, 77 of 156 major objections overturned AI agents’ headline conclusion
A preprint followed AI agents from 35 laboratories for 12 weeks as they published 936 studies. When objections were tested against the data the agents had uploaded with their studies, nearly half of the major objections overturned the headline conclusion.
A Society of AI Scientists is a preprint on alphaXiv that its authors describe as exploratory.
AI agents – programs that run on language models – worked on a shared preprint server and in a chat from 19 August to 12 November 2026. Humans chose the hardware, API budgets and broad topics. The agents chose the hypotheses and experimental designs. Nothing technically prevented humans from helping.
Every study had to include runnable code and raw data, plus a list showing the source file and field behind each number. Cypress, an agent from Lab 23, founded the Journal of Agent Research (JAR). For each version it reviewed, the journal commissioned four blind reviews from different language model families. Between 4 and 9 November it collected 1,408 reviews covering 352 versions of 263 studies.
Reviewers reading only the text could barely tell a flawed version from a corrected one. They flagged 11 of 16 flawed first versions as flawed, and 10 of their corrected versions. When scripts ran 276 objections against the data the agents had uploaded with their studies, 77 of 156 major objections (49.4 per cent) overturned the headline conclusion. Of 120 minor objections, 24 (20 per cent) did the same. The AI summary on the alphaXiv page says this does not show that every objection was correct. Nor did the reproducibility and number checks the agents ran across the whole archive prove the experiments valid, the authors wrote. The checks confirmed only that the reported figures matched the stored output files.
The researchers compared 202 agent studies with arXiv preprints on matching topics published over the same period. The agents scored higher on rigour and the humans on novelty:
| Metric (1–10) | Agents | Humans |
|---|---|---|
| Experimental rigour | 9.02 | 7.00 |
| Claim calibration | 9.32 | 7.23 |
| Novelty | 7.05 | 7.45 |
| Significance | 7.04 | 7.34 |
Source: summary on the study’s alphaXiv page; 808 language model reviews.
The table does not reflect expert opinion, because the language models assessed only the text. The agents’ advantage in the overall assessment came entirely from Lab 1. Labs 2–35 did not differ significantly from humans.
Ethan Mollick pointed out on X that the agents published null results more readily than humans do. In all, 22.6 per cent of the studies supported their primary hypothesis, and 34.7 per cent reported a null result or refuted their hypothesis. In human science, 84 per cent of hypothesis-testing papers report support for their hypothesis, the paper says, citing Fanelli (2010). Under the rules, however, a careful null result counted as an accepted study just as a positive one did. The AI summary on alphaXiv says some of these practices were therefore built into the design. Nor was the design randomised.
Locally run experiments mostly used open-weight models with up to 14 billion parameters. According to alphaXiv, the agents often gravitated towards familiar topics and made small variations on existing benchmarks. The archive can be downloaded from Hugging Face, and the agents’ server is now open to everyone.
Read more: Models