BENCHMARKS
Results run on the public opendataloader-bench corpus — 200 PDFs with hand-annotated ground truth, three metrics covering reading order, table structure, and heading hierarchy. Nutrient's open-source tool is officially listed on the opendataloader.org leaderboard. Vision API modes (understand, agentic) are run internally against the same public corpus.
Benchmarks are re-run on every release.
PARSING BENCHMARK
Each API mode benchmarked against the public opendataloader-bench corpus. The open-source tool (text, structure modes) is officially listed on the opendataloader.org leaderboard. Vision API modes (understand, agentic) require a license and are run internally against the same public corpus.
200 PDF documents · July 2026 · scores vary by mode
OPEN-SOURCE BENCHMARK
The opendataloader-bench corpus and metrics applied across tools. Nutrient's open-source tool is officially listed on the opendataloader.org leaderboard. Benchmarks run on Apple M3 Ultra, all libraries at latest versions as of July 6, 2026.
opendataloader-bench · Apple M3 Ultra · July 6, 2026 · all libraries at latest versions · full leaderboard at opendataloader.org
THROUGHPUT BENCHMARK
The default engine processes at 0.004 s/page. Docling 2.110.0 runs at 0.549 s/page on the same hardware. For high-throughput pipelines, the difference compounds quickly.
pymupdf4llm is a commonly used extraction library that runs at 0.218 s/page. Nutrient's default engine runs at 0.004 s/page on the same corpus.
Even the AI-augmented vision pipeline (understand / agentic modes) runs at 0.354 s/page — faster than docling's default 0.549 s/page, with higher overall accuracy.
METHODOLOGY
Three complementary metrics capture the dimensions of extraction quality that matter most for downstream AI and automation workflows. All evaluations use the same 200-document corpus with hand-verified Markdown ground truth.
In understand mode, 96 of 100 pages return content in the correct reading sequence. Measured by Normalized Inverse Damerau-Levenshtein distance against hand-verified ground truth. Critical for multi-column layouts and tables that span pages.
In understand mode, 94 of 100 tables preserve correct row, column, cell, and header relationships. Measured by Tree Edit Distance Similarity against hand-verified Markdown table structure. A table with correct text but wrong structure produces incorrect downstream output.
In understand mode, 87 of 100 headings land at the correct level — H1, H2, H3 — in the Markdown output. Measured by Mean Heading Similarity against hand-verified heading annotations. Heading structure drives RAG chunking quality and document navigation.
GROUNDING BENCHMARK
Alongside extraction accuracy, Nutrient publishes an open grounding benchmark on Hugging Face for the grounding-en model — an Apache-2.0 licensed model that verifies extracted numbers and facts are traceable back to the source document. The model ranks first and second on the open leaderboard, scoring 0.92–0.97 on number grounding versus 0.45–0.66 for general NLI models at 16× the context window.
The grounding-en model ranks first and second on the public grounding leaderboard — independently evaluated alongside general NLI models on the same benchmark corpus.
Scores 0.92–0.97 on number grounding tasks where general NLI models score 0.45–0.66. The gap reflects a model trained specifically on document-grounding tasks rather than general language inference.
Operates at 16× the context window of comparable NLI models, allowing grounding verification across longer extracted passages without chunking artifacts.
CONTINUOUS BENCHMARKING
Every release runs the full opendataloader-bench corpus through the same deterministic pipeline before shipping. The corpus is publicly available at github.com/opendataloader-project/opendataloader-bench — anyone can reproduce the results independently.
