BENCHMARKS
Results run on the public opendataloader-bench corpus — 200 PDFs with hand-annotated ground truth, and three metrics covering reading order, table structure, and heading hierarchy. Nutrient’s open source tool is officially listed on the opendataloader.org leaderboard. Vision API modes (understand, agentic) are run internally against the same public corpus.
Benchmarks are rerun on every release.
PARSING BENCHMARK
Each API mode is benchmarked against the public opendataloader-bench corpus. The open source tool (text, structure modes) is officially listed on the opendataloader.org leaderboard. Vision API modes (understand, agentic) require a license and are run internally against the same public corpus.
200 PDF documents · July 2026 · scores vary by mode
OPEN SOURCE BENCHMARK
The opendataloader-bench corpus and metrics are applied across tools. Nutrient’s open source tool is officially listed on the opendataloader.org leaderboard. Benchmarks run on Apple M3 Ultra, with all libraries at their latest versions as of 6 July 2026.
opendataloader-bench · Apple M3 Ultra · 6 July 2026 · all libraries at latest versions · full leaderboard at opendataloader.org
PARSEBENCH
Nutrient’s agentic extraction mode achieved a score of 73.5 in an internal run against the ParseBench dataset, which would place it #7 among the 84 publicly listed systems — ahead of Reducto, Azure Document Intelligence, and many leading document parsing and general-purpose AI systems. Nutrient performed particularly strongly in content faithfulness and visual grounding.
ParseBench is published and maintained by LlamaParse, a competitor. The score reflects an internal run against the ParseBench dataset as of August 2026. Nutrient isn’t currently listed on the public leaderboard.
THROUGHPUT BENCHMARK
The default engine processes at 0.004 s/page. Docling 2.110.0 runs at 0.549 s/page on the same hardware. For high-throughput pipelines, the difference compounds quickly.
pymupdf4llm is a commonly used extraction library that runs at 0.218 s/page. Nutrient’s default engine runs at 0.004 s/page on the same corpus.
Even the AI-augmented vision pipeline (understand/agentic modes) runs at 0.354 s/page on local hardware (Apple M3 Ultra) — faster than Docling’s default 0.549 s/page, with higher overall accuracy. This is local processing time, not API roundtrip latency.
METHODOLOGY
Three complementary metrics capture the dimensions of extraction quality that matter most for downstream AI and automation workflows. All evaluations use the same 200-document corpus with hand-verified Markdown ground truth.
In understand mode, 96 of 100 pages return content in the correct reading sequence. It’s measured by Normalized Inverse Damerau-Levenshtein distance against hand-verified ground truth — critical for multicolumn layouts and tables that span pages.
In understand mode, 94 of 100 tables preserve correct row, column, cell, and header relationships. It’s measured by Tree Edit Distance Similarity against hand-verified Markdown table structure. A table with correct text but wrong structure produces incorrect downstream output.
In understand mode, 87 of 100 headings land at the correct level — H1, H2, H3 — in the Markdown output. It’s measured by Mean Heading Similarity against hand-verified heading annotations. Heading structure drives RAG chunking quality and document navigation.
GROUNDING BENCHMARK
Alongside extraction accuracy, Nutrient publishes an open grounding benchmark on Hugging Face for the grounding-en model — an Apache-2.0 licensed model that verifies extracted numbers and facts are traceable back to the source document. The model ranks first and second on the open leaderboard, scoring 0.92–0.97 on number grounding versus 0.45–0.66 for general NLI models at 16× the context window.
The grounding-en model ranks first and second on the public grounding leaderboard — independently evaluated alongside general NLI models on the same benchmark corpus.
The model scores 0.92–0.97 on number grounding tasks where general NLI models score 0.45–0.66. The gap reflects a model trained specifically on document-grounding tasks rather than general language inference.
The model operates at 16× the context window of comparable NLI models, allowing grounding verification across longer extracted passages without chunking artifacts.
CONTINUOUS BENCHMARKING
Every release runs the full opendataloader-bench corpus through the same deterministic pipeline before shipping. The corpus is publicly available at github.com/opendataloader-project/opendataloader-bench — anyone can reproduce the results independently.
Traditional document extraction confidence scores can be misleading. Source-grounded confidence makes automated review more reliable by tying scores to document evidence rather than model decisiveness.
Side-by-side Docling and Nutrient output on three real-world PDFs from opendataloader-bench — heading demotion, TOC misclassification, and table header absorption, with the downstream cost of each failure.
The corpus, three scoring metrics, and full accuracy and speed results — including where Nutrient doesn’t lead — produced from a public, reproducible benchmark.
