How we test PDF extraction: the opendataloader-bench benchmark
Table of contents
Benchmark numbers in marketing copy are easy to distrust. A number published without a reproducible methodology, a named corpus, or an honest disclosure of where competitors win is a marketing claim, not a result.
This article documents exactly how Nutrient’s PDF extraction results are produced: the corpus, the three scoring metrics, the hardware, and what the numbers actually show — including where Nutrient does not lead.
Why most PDF benchmarks mislead
Most published extraction benchmarks share three problems: the corpus is proprietary, the document types are unrepresentative, and the evaluation is not reproducible.
A team evaluating parsers for a production workflow cannot verify whether a proprietary benchmark’s documents resemble their own, who annotated the ground truth and how, or whether results were produced from full runs or curated subsets. Closed benchmarks make strong claims without offering the means to check them.
Corpus composition matters more than corpus size. Many academic benchmarks draw primarily from LaTeX-rendered papers — documents where structure was encoded before being converted to PDF. Parsers that perform well on homogeneous academic corpora often degrade on two-column regulatory reports, scanned handouts with sidebar layouts, or tables embedded within narrative paragraphs. A parser that handles well-formed academic PDFs is not necessarily a parser for production document workflows.
The opendataloader-bench corpus was selected because it avoids these problems. All 200 documents are publicly accessible, the ground truth annotations are community-maintained, and the evaluation harness is open source. Any team can download the corpus, run the same parsers, and verify the numbers independently. When Nutrient publishes a benchmark score, anyone can check it.
The corpus: opendataloader-bench
All results in this article use the opendataloader-bench(opens in a new tab) corpus — a public dataset of 200 real-world PDFs with hand-annotated Markdown ground truth, maintained by the opendataloader project under the Apache 2.0 license.
The documents span a wide range of formats and layouts encountered in production: multi-column policy reports from international organizations, scientific papers with image grids and footnotes, legal and regulatory reports with styled tables, university course handouts with sidebar layouts, and dense academic papers with multiple comparison tables side by side.
This breadth matters. Benchmarks built on clean, single-column academic PDFs routinely overestimate accuracy on the documents that enterprise teams actually process. A parser that handles well-formed LaTeX output often fails on a two-column ADB report or a biology course handout with a sidebar.
The corpus is publicly available. The PDFs, ground truth annotations, and all tool predictions are committed to the repository — any team can download the files and reproduce the numbers independently.
The document types in the corpus map to categories that production workflows regularly encounter. Policy reports from international development organizations appear in financial due diligence, procurement, and ESG reporting pipelines. Scientific papers with image grids and footnotes are common source documents in life sciences RAG systems. Legal and regulatory reports with embedded tables represent insurance and compliance document processing. University course materials with sidebar layouts appear in educational technology and content digitization workflows. No single category dominates the corpus, which means a parser’s overall score reflects generalization across real document variety rather than performance on one document type.
Three metrics
Extraction quality is scored across three dimensions:
NID — Normalized Intersection Distance (reading order) Measures whether the extracted content appears in the correct sequence. In a two-column document, paragraphs from column one should not be interleaved with paragraphs from column two. A score of 0.96 means that 96 out of 100 content blocks are in the correct reading position.
TEDS — Tree Edit Distance Score (table structure) Measures whether tables are extracted with the correct number of rows, columns, and cell relationships. A score of 1.000 means the extracted table is structurally identical to the ground truth annotation.
MHS — Markdown Heading Score (heading hierarchy) Measures whether document sections are correctly identified and nested. A report with H1 chapter titles and H2 subsections should produce that structure in the output — not a flat list of paragraphs, and not a single heading level for all titles.
The overall score is a weighted mean across the three metrics. TEDS is computed only for documents that contain tables (42 of 200 in this corpus); MHS only for documents with headings (107 of 200). Documents without tables score only on NID and MHS; documents without headings score only on NID and TEDS.
What failure looks like in practice
The three metrics exist because the most damaging extraction failures are invisible at the character level. A parser can achieve near-perfect text recall — capturing 99% of the words in a document — while producing output that is functionally useless for downstream processing.
Reading order failures. A two-column policy report, read naively left-to-right across the full page width, produces interleaved text where sentences from column one alternate with sentences from column two. Word recall is high; the document is unreadable. A RAG system chunking this output creates fragments that span unrelated topics, making retrieval unreliable regardless of the embedding model used. NID measures this directly: it scores whether blocks of extracted content appear in the correct reading sequence.
Table structure failures. A parser can identify that a table exists and extract all of its cell text, but if it merges spanning cells incorrectly or loses the relationship between header rows and data rows, downstream queries will silently return wrong answers. A financial extraction workflow asking which row corresponds to a given quarter, or a compliance check mapping column headers to extracted values, will fail without any explicit error signal. TEDS measures whether the extracted table structure matches the ground truth annotation.
Heading hierarchy failures. A 50-page report that should produce H1 chapters, H2 sections, and H3 subsections will — if all headings are flattened to H1 — produce a list of chunks with similar apparent weight. A retrieval system navigating to “Section 3.2, eligibility criteria” searches the full document instead of routing to the correct subtree. MHS measures whether the extracted output preserves the heading structure of the source document.
These failures share a property: they pass basic quality checks. Extracted text looks plausible. Character counts are reasonable. Only downstream processing — or direct comparison against ground truth — exposes them.
Results
Accuracy
All parsers were run on the full 200-document corpus on an Apple M3 Ultra with no discrete GPU.
| Tool | Configuration | Version | Overall | NID | TEDS | MHS |
|---|---|---|---|---|---|---|
Nutrient --vision † | vision | 1.3.1 | 0.93 | 0.96 | 0.94 | 0.87 |
| opendataloader ‡ | hybrid (docling-fast) | — | 0.907 | 0.934 | 0.928 | 0.821 |
| Nutrient | standard | 1.3.0 | 0.89 | 0.93 | 0.74 | 0.82 |
| docling | default | 2.110.0 | 0.89 | 0.91 | 0.93 | 0.83 |
| pymupdf4llm | default | 1.28.0 | 0.86 | 0.90 | 0.73 | 0.78 |
| opendataloader | standard | 2.4.7 | 0.83 | 0.90 | 0.48 | 0.74 |
| markitdown | default | 0.1.6 | 0.59 | 0.84 | 0.27 | 0.00 |
| pypdf | default | 6.14.2 | 0.58 | 0.87 | 0.00 | 0.00 |
| liteparse | default | 2.4.1 | 0.57 | 0.86 | 0.00 | 0.00 |
† Requires a license key. Vision results are run internally against the same corpus and are not listed on the public leaderboard because the tier requires a license key.
‡ OpenDataLoader hybrid combines the OpenDataLoader engine with a docling-fast backend. Scores are drawn from the upstream public leaderboard rather than the frozen run used for all other rows; no pinned-version run was produced for this configuration.
Among the open-source tools, the OpenDataLoader hybrid configuration leads at 0.907 overall. Among tools available without a merged pipeline, docling 2.110 and the standard Nutrient CLI are separated by less than one point at the third decimal (0.892 vs 0.889). Nutrient has the best reading order score (NID 0.93 standard, 0.96 vision), and docling has the best table structure score among non-vision tools. The vision tier tops every metric outright, including table structure, while running faster than docling. Results from the standard CLI are independently run and listed by the opendataloader project on the public opendataloader.org(opens in a new tab) leaderboard; vision results are run internally against the same corpus and are not listed there because they require a license.
Speed
The standard Nutrient CLI processes at 0.004 seconds per page — 134× faster than docling at comparable overall accuracy. The vision tier runs at 0.354 seconds per page, still faster than docling, with higher accuracy across all three metrics.
The vision tier
The --vision flag activates a licensed machine-vision ICR pipeline that handles layout analysis, table reconstruction, and handwritten content locally. Vision results were independently verified by running the full 200-document harness on an Apple M3 Ultra using version 1.3.1, with all parsers at latest versions.
Verification confirmed: vision tops every accuracy metric including table structure, and at 0.354 seconds per page it remains faster than docling. There is no speed-for-accuracy tradeoff against the open-source field.
Vision is a licensed capability. The default engine (0.004 s/page, free for up to 1,000 documents per month) is listed on the opendataloader.org public leaderboard. Vision results are run internally against the same corpus because the tier requires a license key and cannot be submitted to the public leaderboard.
How benchmarks are run
Nutrient runs the full opendataloader-bench corpus through the same deterministic pipeline on every release. The corpus, methodology, and all tool predictions are committed to the public repository at github.com/opendataloader-project/opendataloader-bench(opens in a new tab). All competitor libraries are pinned to their latest published versions at the time of the run. No documents are cherry-picked, and no post-processing is applied to tool outputs before scoring.
The standard CLI predictions are independently run and listed by the opendataloader project on the public leaderboard at opendataloader.org. Vision-tier predictions are stored internally.
Resources
- opendataloader-bench corpus — github.com/opendataloader-project/opendataloader-bench(opens in a new tab)
- Nutrient pdf-to-markdown (open source) — github.com/PSPDFKit/pdf-to-markdown(opens in a new tab)
- Live benchmark results — nutrient.io/api/data-extraction-api/benchmarks/
