LlamaIndex’s ParseBench-2 scores PDF parsing errors by severity
LlamaIndex’s new benchmark measures how well tools convert PDFs for AI agents, and a single serious error can cap the score for a whole page. The company published the leaderboard itself, and its own LlamaParse leads it.
LlamaIndex released the ParseBench-2 benchmark on Hugging Face on 11 November. Jerry Liu of LlamaIndex presented it on X two days later. The benchmark contains 4,317 page-level test cases from 2,645 documents. The previous version had about 2,000 pages.
The benchmark measures document parsing: how well a tool converts a PDF page into structured text that an AI agent can work from. It covers six dimensions: text, formatting, tables, charts, layout (locating content on the page) and, new in this version, forms. There are 600 form pages.
A serious error caps the page score
A page’s score is the lower of two numbers. One measures how much of the content the tool recovered; the other is a cap set by the severity of the error. A serious error therefore lowers the score even if the rest of the page is almost correct. Such errors include a changed number, a flipped negation, swapped table headers and a misread form value. Presenting the same information in another supported format costs no points.
LlamaIndex said it tuned the metrics against 330 controlled scenarios, each with a target score drop ranging from critical to harmless. The metrics detect more than 140 error types. By default the scorer uses rules rather than a language model as judge. Liu said users can adjust error severities to suit their own needs.
The overall score is the unweighted mean of the six dimensions. Liu advises treating it with caution, saying the meaningful differences show up in individual dimensions.
LlamaParse tops the company’s own leaderboard
The table shows the top six entries on LlamaIndex’s GitHub leaderboard. The company that built the benchmark measured the results itself, and its own LlamaParse is in first place.
| Tool | Overall | Tables | Charts | Forms | Price/page |
|---|---|---|---|---|---|
| LlamaParse (Agentic Plus) | 90.07 | 94.61 | 92.58 | 81.76 | 2.81¢ |
| LlamaParse (Agentic) | 86.05 | 89.86 | 77.39 | 80.94 | 1.25¢ |
| Claude Opus 5.5 | 85.73 | 94.11 | 74.12 | 77.84 | 6.49¢ |
| Reducto (r-1) | 85.23 | 90.93 | 78.45 | 78.31 | 1.00¢ |
| LlamaParse (Cost Effective) | 84.59 | 87.19 | 73.07 | 81.21 | 0.38¢ |
| Gemini 4 argon (low) | 84.20 | 92.78 | 68.20 | 77.19 | 2.68¢ |
Source: ParseBench 2 leaderboard in the ParseBench repository. Scores are on a 0–100 scale; price is in cents per page. The leaderboard lists only systems scored on all six dimensions.
The scores cannot be compared with those from the first version of ParseBench. Liu noted that LlamaParse is not the best value for money in every dimension: Claude Opus 5.5, for example, is good at tables but weak at charts, and Extend does well on forms.
A corrected version of the dataset, v2.01, followed on 13 November and changed the answer keys for eight pages. The maintainers advise comparing results calculated with the same versions of the dataset and scorer. Liu said feedback had come in on the answer keys and metrics, and promised new versions soon. A preprint of the technical report is available as a PDF, and the report is due to appear on arXiv.
The dataset is on Hugging Face under the Apache 2.0 licence, and the scorers are part of the open parse-bench package. Evaluating a commercial service requires that service’s API key. ParseBench 2 runs with the --dataset parsebench-2 flag; without it, the command runs the first version.