How we test PDF extraction: The opendataloader-bench benchmark

Benchmark numbers in marketing copy are easy to distrust. A number published without a reproducible methodology, a named corpus, or an honest disclosure of where competitors win is a marketing claim, not a result.

This article documents exactly how Nutrient’s PDF extraction results are produced: the corpus, the three scoring metrics, the hardware, and what the numbers actually show — including where Nutrient doesn’t lead.

Why most PDF benchmarks mislead

Most published extraction benchmarks share three problems: The corpus is proprietary, the document types are unrepresentative, and the evaluation isn’t reproducible.

A team evaluating parsers for a production workflow can’t verify whether a proprietary benchmark’s documents resemble their own, who annotated the ground truth and how, or whether results were produced from full runs or curated subsets. Closed benchmarks make strong claims without offering the means to check them.

Corpus composition matters more than corpus size. Many academic benchmarks draw primarily from LaTeX-rendered papers — documents where structure was encoded before being converted to PDF. Parsers that perform well on homogeneous academic corpora often degrade on two-column regulatory reports, scanned handouts with sidebar layouts, or tables embedded within narrative paragraphs. A parser that handles well-formed academic PDFs isn’t necessarily a parser for production document workflows.

The opendataloader-bench corpus was selected because it avoids these problems. All 200 documents are publicly accessible, the ground truth annotations are community-maintained, and the evaluation harness is open source. Any team can download the corpus, run the same parsers, and verify the numbers independently. When Nutrient publishes a benchmark score, anyone can check it.

The corpus: opendataloader-bench

All results in this article use the opendataloader-bench(opens in a new tab) corpus — a public dataset of 200 real-world PDFs with hand-annotated Markdown ground truth, maintained by the OpenDataLoader project under the Apache 2.0 license.

The documents span a wide range of formats and layouts encountered in production: multicolumn policy reports from international organizations, scientific papers with image grids and footnotes, legal and regulatory reports with styled tables, university course handouts with sidebar layouts, and dense academic papers with multiple comparison tables side by side.

This breadth matters. Benchmarks built on clean, single-column academic PDFs routinely overestimate accuracy on the documents that enterprise teams actually process. A parser that handles well-formed LaTeX output often fails on a two-column ADB report or a biology course handout with a sidebar.

The corpus is publicly available. The PDFs, ground truth annotations, and all tool predictions are committed to the repository, and any team can download the files and reproduce the numbers independently.

The document types in the corpus map to categories that production workflows regularly encounter. Policy reports from international development organizations appear in financial due diligence, procurement, and environmental, social, and governance (ESG) reporting pipelines. Scientific papers with image grids and footnotes are common source documents in life sciences retrieval-augmented generation (RAG) systems. Legal and regulatory reports with embedded tables represent insurance and compliance document processing. University course materials with sidebar layouts appear in educational technology and content digitization workflows. No single category dominates the corpus, which means a parser’s overall score reflects generalization across real document variety rather than performance on one document type.

Three metrics

Extraction quality is scored across three dimensions:

NID — Normalized Indel Distance (reading order) It compares the full extracted text with the ground truth, character by character, so content in the wrong sequence scores lower. In a two-column document, paragraphs from column one shouldn’t be interleaved with paragraphs from column two. A score of 1.0 means the extracted text matches the ground truth exactly.

TEDS — Tree Edit Distance Similarity (table structure) It measures whether tables are extracted with the correct rows, columns, and cell relationships, and scores the cell text too. A score of 1.000 means the extracted table is identical to the ground truth annotation.

MHS — Heading-Level Similarity (heading hierarchy) It measures whether document sections are correctly identified and nested. A report with H1 chapter titles and H2 subsections should produce that structure in the output — not a flat list of paragraphs, and not a single heading level for all titles.

The overall score is a weighted mean across the three metrics. TEDS is computed only for documents that contain tables (42 of 200 in this corpus); MHS only for documents with headings (107 of 200). Documents without tables score only on NID and MHS; documents without headings score only on NID and TEDS.

What failure looks like in practice

The three metrics exist because the most damaging extraction failures are invisible at the character level. A parser can achieve near-perfect text recall — capturing 99 percent of the words in a document — while producing output that’s functionally useless for downstream processing.

Reading order failures. A two-column policy report, read naively left-to-right across the full page width, produces interleaved text where sentences from column one alternate with sentences from column two. Word recall is high; the document is unreadable. A RAG system chunking this output creates fragments that span unrelated topics, making retrieval unreliable regardless of the embedding model used. NID measures this directly: It scores whether blocks of extracted content appear in the correct reading sequence.

Table structure failures. A parser can identify that a table exists and extract all of its cell text, but if it merges spanning cells incorrectly or loses the relationship between header rows and data rows, downstream queries will silently return wrong answers. A financial extraction workflow asking which row corresponds to a given quarter, or a compliance check mapping column headers to extracted values, will fail without any explicit error signal. TEDS measures whether the extracted table structure matches the ground truth annotation.

Heading hierarchy failures. A 50-page report that should produce H1 chapters, H2 sections, and H3 subsections will — if all headings are flattened to H1 — produce a list of chunks with similar apparent weight. A retrieval system navigating to “Section 3.2, eligibility criteria” searches the full document instead of routing to the correct subtree. MHS measures whether the extracted output preserves the heading structure of the source document.

These failures share a property: They pass basic quality checks. Extracted text looks plausible. Character counts are reasonable. Only downstream processing — or direct comparison against ground truth — exposes them.

Results

The following sections break down both dimensions of the benchmark: extraction accuracy across the three metrics, and processing speed per page.

Accuracy

All parsers were run on the full 200-document corpus on an Apple M3 Ultra with no discrete GPU, in our own run with the versions listed below. Nutrient’s PDF-to-Markdown tool runs in two modes referenced throughout: Standard, the free default, and Vision, an AI-enhanced mode priced per page for scans and complex layouts. The public leaderboard at opendataloader.org comes from the project’s own run with other versions, so its figures differ slightly: it lists Nutrient Standard at 0.885 overall and Docling at 0.882.

† Vision requires a Nutrient account, API key, or Nutrient CLI license key. We ran it against the same corpus internally, so its results don’t appear on the public leaderboard.

‡ OpenDataLoader hybrid combines the OpenDataLoader engine with a docling-fast backend. Scores are drawn from the upstream public leaderboard rather than the frozen run used for all other rows; no pinned-version run was produced for this configuration.

Among the open source tools, the OpenDataLoader hybrid configuration leads at 0.907 overall. Among tools available without a combined parser, Docling 2.110 and Nutrient Standard are 0.003 apart (0.892 vs. 0.889). On the public leaderboard, where the hybrid configuration’s scores come from, Standard scores 0.925 for reading order against the hybrid configuration’s 0.934. Vision has the highest score in every measure, including table structure, while running faster than Docling. Standard results are independently run and listed by the OpenDataLoader project on the public opendataloader.org(opens in a new tab) leaderboard. We ran Vision against the same corpus internally.

Speed

Nutrient Standard processes at 0.004 seconds per page — 134× faster than Docling at comparable overall accuracy. The two do different work: Standard reads the PDF text layer and runs no OCR, while Docling runs layout and OCR models. The opendataloader [hybrid] time comes from the upstream public run, on different hardware from the other rows. Vision runs at 0.354 seconds per page, still faster than Docling, and has the highest score in every accuracy measure shown above.

Vision

The --vision flag improves extraction for scans, handwriting, formulas, and complex layouts. It runs on your computer. We ran the full 200-document benchmark for Vision internally, on an Apple M3 Ultra, with version 1.3.1 and the latest available version of each parser. These results haven’t been independently verified.

In that run, Vision scored highest on every accuracy metric, including table structure, at 0.354 seconds per page. That’s faster than Docling in the same run, and slower than the text-layer tools.

Standard is free; Vision is priced per input page. Standard is listed on the opendataloader.org public leaderboard. We ran Vision against the same corpus internally because it requires a Nutrient account, API key, or Nutrient CLI license key.

How benchmarks are run

Nutrient runs the full opendataloader-bench corpus through the same deterministic pipeline on every release. The corpus, methodology, and all tool predictions are committed to the public repository at github.com/opendataloader-project/opendataloader-bench(opens in a new tab). All competitor libraries are pinned to their latest published versions at the time of the run. No documents are cherry-picked, and no post-processing is applied to tool outputs before scoring.

Standard results are independently run and listed by the OpenDataLoader project on the public leaderboard at opendataloader.org. Vision results are stored internally.

Resources

• opendataloader-bench corpus — github.com/opendataloader-project/opendataloader-bench(opens in a new tab)
• Nutrient pdf-to-markdown (free CLI, proprietary license) — github.com/PSPDFKit/pdf-to-markdown(opens in a new tab)
• Live benchmark results — nutrient.io/api/data-extraction-api/benchmarks/
• Interactive extraction demos — github.com/PSPDFKit/nutrient-extraction-samples(opens in a new tab) — eight demos showing grounded extraction on insurance claims, a birth record, mortgage assistance forms, a small claims court form, a prior authorization form, and a property appraisal report