BENCHMARKS
Every processing mode runs each night on two public benchmarks. Here’s how Nutrient scores next to other parsers, including where it scores lower.
Scores as of 24 September 2026. Nutrient’s scores come from our own nightly runs on the public datasets, and they haven’t been independently verified.
OPENDATALOADER-BENCH
opendataloader-bench is a public set of 200 PDFs with reference answers, scored on reading order, tables, and headings. Understand mode scores highest here, and it’s the default. Higher modes cost more credits per page, so start with the lowest mode that meets your bar.
Nightly run, 24 September 2026.
PUBLIC LEADERBOARD
The opendataloader.org leaderboard runs the same 200 PDFs through 12 parsers. Nutrient’s entry there is the free Nutrient CLI, not the Data Extraction API. On the same PDFs, the API’s understand mode scores 0.933 in our nightly run; the leaderboard’s top entry, opendataloader [hybrid], scores 0.907.
Source: opendataloader.org leaderboard, checked 23 September 2026. The Nutrient CLI’s Standard mode is free.
METHODOLOGY
opendataloader-bench compares the Markdown each parser produces with reference answers, using three scores from 0 to 1.
Checks that text comes out in the right order. It matters most for multicolumn pages and tables that span pages.
Checks rows, columns, cells, and headers, as well as the text in them. A table with the right text in the wrong cells still breaks what comes after.
Checks headings (H1, H2, H3) and the content under each one. Heading structure decides how well a document splits into chunks for RAG.
PARSEBENCH
ParseBench is a parsing benchmark from LlamaIndex, the maker of LlamaParse: 2,078 pages from 1,211 documents, scored from 0 to 100 on tables, charts, content faithfulness, semantic formatting, and visual grounding. Our nightly run of agentic mode scores 76.2. Nutrient’s row on ParseBench’s public leaderboard, which Nutrient measured with ParseBench’s own harness against the production API, scores 76.4.
On charts, agentic mode scores 80.3, and 4 of 111 entries score higher. That’s ahead of every general-purpose AI model on the leaderboard: The best of them, Qwen 3.8 27B (Thinking), scores 69.2. On visual grounding (placing each piece of content where it sits on the page), agentic mode scores 76.5, and 9 of 111 entries score higher.
Overall, agentic mode scores 76.2, and 11 of 111 entries score higher, led by LlamaParse Agentic Plus at 90.2. The biggest gap is tables: 72.4, with 73 entries higher. Content faithfulness scores 84.5 (68 higher) and semantic formatting scores 67.2 (21 higher). Understand mode, the default, costs half the credits and scores 57.9 overall. It reads tables as well as agentic mode (72.2), but not charts (1.3), so documents with charts need agentic mode.
Nutrient mode scores: our run on the public ParseBench dataset, 24 September 2026, using the latest nightly build. All other scores: parsebench.ai, checked 24 September 2026.
GROUNDING BENCHMARK
Nutrient also publishes an open benchmark for one question: Is an extracted number, date, or fact actually supported by the document? We built it, so treat it as a vendor benchmark. It compares Nutrient’s grounding models with open verification models and one commercial API, scored from 0 to 1 (ROC-AUC) on the English set (leaderboard, checked 23 September 2026).
On numbers, Nutrient’s models score 0.92–0.97, ahead of the one commercial API, Jev (TypeSafe), at 0.85. The best open verification model scores only 0.66. Across all claims, grounding-multilingual scores 0.925, Jev (TypeSafe) scores 0.884, and grounding-en scores 0.882.
grounding-en reads 8,192 tokens at once, 16 times the 512 tokens most open verification models accept, so long passages don’t need splitting.
The benchmark data (CC-BY-SA-4.0) and grounding-en (Apache-2.0) are on Hugging Face, so anyone can rerun the evaluation. grounding-multilingual isn’t open-weight.
CONTINUOUS BENCHMARKING
Every night, the latest build runs both benchmarks: opendataloader-bench in all four modes, and ParseBench in structure, understand, and agentic modes. Both datasets are public (opendataloader-bench, ParseBench), so anyone with an API key can run the same documents and compare.
