OpenAI’s lab analysis benchmark: GPT-6 Astra scores 66.7 per cent at xhigh, GPT-5.6 Sol 43.8 per cent
Written by AI
OpenAI and Ligo built BioBasicsBench, whose 100 tasks measure how AI agents interpret raw laboratory data. Higher scores do not yet mean that the models assess lab results reliably, OpenAI said.
On 6 November OpenAI used its blog and X to present a benchmark that measures how well AI agents handle routine laboratory analysis. The company built BioBasicsBench with Ligo, a biotech company that designs enzymes.
In a wet lab, researchers run experiments on biological samples. Existing benchmarks often rely on images selected for scientific papers, OpenAI said, so they can miss failed experiments, unclear images and conflicting observations.
How the benchmark works
The benchmark has 100 tasks across five areas: protein methods, imaging and flow cytometry, quantification and assays, nucleic acid methods, and molecular cloning. Researchers wrote the tasks using data from their own experiments, including gel images, plate photos, microscopy images, sequencing files, instrument exports and videos of lab work. Each task was checked three times, and a PhD-level lab researcher from outside Ligo carried out one of the checks.
The agent answers with a conclusion, a measurement or a recommendation. Each answer is scored against task-specific criteria written by experts, so a partly correct analysis earns partial credit. The reported result is mean reward: the percentage of available credit a model earned per task, on average.
Results
OpenAI tested seven of its own models at several reasoning effort levels, and each model attempted every task five times. The table shows results at xhigh, the highest reasoning effort reported for all models.
| Model | Mean reward | 95% confidence interval | Output tokens per task |
|---|---|---|---|
| GPT-6 Astra | 66.7% | 60.3–73.1% | 4,704 |
| GPT-6.1 Sol | 62.0% | 55.6–68.4% | 5,238 |
| GPT-5.6 Sol | 43.8% | 37.6–50.0% | 9,099 |
| GPT-6 Luna | 38.3% | 32.3–44.5% | 13,120 |
| GPT-5.5 | 35.4% | 29.7–41.0% | 11,135 |
| GPT-5.4 | 33.2% | 28.3–38.4% | 13,717 |
| GPT-5.2 | 31.2% | 26.2–36.3% | 27,620 |
Source: OpenAI’s own measurement, xhigh reasoning effort, 100 tasks and five attempts per task.
The confidence intervals for Astra and GPT-6.1 Sol partly overlap, so the gap between the two models cannot be considered clear. The benchmark did not include models from other companies.
OpenAI said Astra also clearly outperformed GPT-5.6 Sol at low reasoning effort. The company suggested that the better result may stem from stronger native multimodality, the ability to process images and video directly. In the company’s example, Astra noticed that the wrong antibiotic had been written on the edge of a negative control plate. GPT-5.6 Sol, by contrast, used the plate as a control.
Shortcomings remain
Higher scores do not yet mean that the models make reliable judgements in the lab, OpenAI said. In a cloning experiment, Astra picked the wrong bacterial colonies. Contrary to the controls, it assumed that a larger PCR product showed the cloning had worked. When interpreting a microscopy image, it ignored observations that contradicted its own conclusions. When counting colonies, it merged adjacent ones, missed visible ones and marked empty agar as a colony. On the example plate, its count was about 12 per cent too high.
Read more: Models