Text
Correct: Every word is there, spelled as printed, with every accent.
Typical error: “Müller” comes out as “Muller.”
Search
Season 1 · Updated 5 November 2026
A parsing benchmark for whole documents
Invoices, statements, and reports often run for many pages. WholeDoc Bench scores every document from the first page to the last, so a table or a paragraph that continues on the next page must come out in one piece. It also tests scans, charts, barcodes, and the position of every item on the page.
Accuracy and cost
Top accuracy doesn’t have to cost the most
Each point is one way to run a parser, placed by its score and its price per page. The line marks the best score you can get for each budget. For every point below the line, another option scores higher for the same price or less.
- Nutrient
- Frontier labs
- Document APIs
- Best score at or below each price
Leaderboard
Which parser gets the whole document right?
13 parsers, run 19 ways, scored on whole documents from the first page to the last. The test is built around what breaks parsers in real work: tables that run across pages, scans, charts, and barcodes. The main test set stays private.
#
1
Nutrient Agentic
86.2 Best in this column
94.6 Best in this column
93.7
74.8 Best in this column
87.1
100 Best in this column
64.5 Best in this column
88.6 Best in this column
1.8
2
Nutrient Understand
80.8
92.7
91.9
73.3
86.1
100 Best in this column
35.1
86.4
0.9
3
Claude Fable 5.1 (high effort)
67.1
90.9
93.5
72.3
84.7
3.6
58.9
66.0
17.8
4
Claude Fable 5.1 (medium effort)
67.1
90.9
93.3
72.1
84.9
2.4
58.4
67.8
16.9
5
Azure Document Intelligence
65.9
83.5
91.7
50.0
69.4
94.0
0
72.5
≈2.8
6
GPT-6.1 Sol (medium effort)
62.9
89.2
94.9
74.3
82.8
2.4
47.6
49.3
≈3.1
7
GPT-6.1 Sol (high effort)
62.5
88.0
94.0
72.2
82.3
2.4
49.2
49.5
≈4.3
8
Extend (performance)
60.5
82.0
93.8
48.4
80.8
2.4 We’re still checking this score.
40.6
75.2
2.0
9
Databricks
60.4
85.7
95.0
47.2
77.4
2.4
51.0
64.2
≈0.45
10
Extend (performance, agentic text)
60.2
81.9
94.0
48.4
80.2
2.4 We’re still checking this score.
39.6
75.1
2.1
11
Reducto (R-1)
59.9
81.0
95.8 Best in this column
51.2
76.4
0
38.5
76.4
≈1.7
12
LlamaParse Agentic Plus
56.7
89.3
92.6
60.0
89.5 Best in this column
0
0 We’re still checking this score.
65.4
≈5.6
13
LlamaParse Agentic
56.5
91.8
94.8
57.3
82.2
0
0 We’re still checking this score.
69.5
≈1.3
14
AWS Textract
49.9
77.4
81.9
48.8
71.4
0
0
70.0
1.5
15
Docling
40.0
75.5
76.7
43.9
63.9
0
20.2
0
Local
16
Nutrient Structure (no AI) Reads text and basic layout without AI, for scans and simple documents. It doesn't read charts or barcodes.
39.4
71.9
82.0
34.8
44.2
0
1.9
41.2
0.15
17
PyMuPDF
37.5
80.8
84.1
49.4
47.3
0
0.8
0
Local
18
MarkItDown
21.4
54.8
64.9
15.9
12.7
0
1.3
0
Local
19
pdftotext
19.8
56.6
69.0
13.4
0
0
0
0
Local
- Documents scored as one piece
- Same rules for every parser
- Code under Apache 2.0
- Sample set: 10 documents
- Main test set kept private
- Failures count as zero
Where parsers fail
Most errors happen where documents get hard: at page breaks, in long tables, and in scans
Case
Correct output
Nutrient
Extend
Reducto
LlamaParse
Frontier model
Report
01
One table, five pages
Bank statement
1 table
1 table correct
In pieces wrong
1 table correct
1 table correct
In pieces wrong
02
One header, not four
Export invoice
1 table, 1 header
1 table correct
In pieces wrong
1 table correct
1 table correct
In pieces wrong
03
Three traps, kept as printed
Delivery note
As printed
As printed correct
As printed correct
Changed wrong
Changed wrong
Changed wrong
04
Two page breaks
Scanned report
Both right
Both right correct
Not both wrong
Not both wrong
Not both wrong
Not both wrong
- Correct on the point the case shows
- Wrong
What we test
What counts as correct
A typical error
Reading order
Correct: The text comes out in the order a person reads it.
Typical error: Two columns are mixed line by line.
Structure
Correct: Headings, paragraphs, captions, footnotes, and bold text stay as they are.
Typical error: A paragraph that runs onto the next page is cut in two.
Tables
Correct: Each cell is in the right row and column, even when the table continues on the next page.
Typical error: A table that runs across five pages comes back in five pieces.
Barcodes
Correct: The parser reads the content of each barcode.
Typical error: A code is skipped, or one character is wrong.
Charts
Correct: The chart becomes the right data points, series, chart type, and title.
Typical error: The parser invents values that aren’t in the chart.
Position on the page
Correct: Each item, and each table cell, has the right box on the page.
Typical error: A parser draws one box around the whole page.
How we built it
Documents
The set has 480 documents with 633 pages: bank statements, invoices, delivery notes, reports, and more. They include born-digital and scanned documents in more than 30 languages. A sample of 10 documents is public. The main test set stays private to reduce the risk of benchmark contamination or overfitting.
Answer keys
For real documents, three AI models from OpenAI, Anthropic, and Google read each page, and a value goes in only when at least two of them agree. A person resolves every disagreement, corrects the draft, and approves it. For generated documents, the key comes from the generator.
Scoring
Written rules decide every score, and code applies them the same way to every parser.
Settings
Nutrient runs through its public API — the same one our customers use. AI models run at medium and high effort, one page per request, with our layout prompt. Every setting for every parser is in the repository.
A snapshot
Season 1 is a snapshot. Some areas are still hard.
Tables that continue on the next page: Most settings rarely keep them whole.
Structure: No setting scores above 75.
Charts: The best score is 64 out of 100.
Add your parser to the leaderboard
The code and a sample of the test documents are public. Run the benchmark on the sample, or score your own parser on it. The leaderboard scores come from the full set.
Try Nutrient
Parse your own documents with the same API we tested and get back the text, the tables, the charts, and the position of every item. Agentic mode, the top score here, costs 1.8¢ per page.
Run the benchmark
Run it on the public sample. The code supports 13 parsers out of the box.
Score your own parser
Bring your parser’s output files for the sample documents. The code checks them, scores them, and adds your row to your own report, marked as imported. The official leaderboard ranks only runs the benchmark made itself.
For parser vendors
Does your parser deserve a better setting? Open a pull request with the setting, and we’ll run it again.
Benchmarks we don’t run
ParseBench (LlamaIndex) Our public score: 76.4 (#15 of 155)
Retrieved: 3 November 2026
View sourceopendataloader-bench Our public score: 0.885 (#2 of 12)
Retrieved: 3 November 2026
View sourceFrequently asked questions
Who built WholeDoc Bench?
The Nutrient Data Extraction team built WholeDoc Bench. Parse, which tests how well parsers read documents, is its first track. Nutrient’s own parser is one of the parsers we test.
Why trust a benchmark from a parser vendor?
You don’t have to. The code, the settings, and a sample of the test documents are public, so you can see how every score is made and run the benchmark yourself on the sample. The main test set stays private to reduce the risk of benchmark contamination or overfitting.
Which parsers are in it?
The benchmark covers 13 parsers in 19 settings: document APIs, frontier labs’ models, and open source tools. The leaderboard lists all of them.
How often do you update it?
We update it with each new season. Each season adds documents and tests. When a vendor sends a better setting, we run that parser again and date the new result.
Can I add my parser?
Yes. Run the benchmark on your parser’s output for the sample documents, and you get your own report, with your row marked as imported. For the official leaderboard, the benchmark has to run the parser itself, so open a pull request that adds an adapter and a configuration.