How evaluation scoring works
Evaluation sets use deterministic rules to compare each run with its saved answer key. Value accuracy and citation accuracy measure different aspects of the result. Grounding describes the engine’s evidence for a value but doesn’t determine whether the value matches the answer key.
Extract value accuracy
Each eligible expected leaf value contributes a match or a miss. A document’s answer accuracy equals its matched values divided by its total scored values.
The evaluator uses these comparison rules:
| Value | Comparison rule |
|---|---|
| Strings | The evaluator compares strings exactly after Unicode normalization, trimming, and collapsing whitespace. Case and punctuation still matter. |
| Numbers | The evaluator compares numbers by numeric equality. A number and a string containing exactly that number can match. |
| Booleans | The evaluator compares Boolean values by equality. A Boolean can also match its string form, "true" or "false". |
| Dates and datetimes | Equivalent parsed dates or instants match when both values have the same kind. A date and a datetime aren’t automatically equivalent. |
JSON null | Only a returned JSON null matches. A missing key, empty string, or literal string "null" is different. |
| Nested objects | The evaluator compares each expected leaf separately. |
| Scalar arrays | The evaluator compares values without requiring the same order. Duplicates count separately, and unmatched elements count as misses. |
| Arrays of objects | The evaluator pairs rows by value similarity before comparing their fields. Unmatched rows count as misses. Large comparisons use bounded fallback matching. |
The answer key defines which object fields the evaluator scores. The evaluator doesn’t score extra returned object fields because they have no expected answer. Extra array elements or table rows can count as misses because they change an expected array’s contents.
A missing expected field counts as a miss, even when its expected answer is null. It also counts as a miss when the executed preset no longer requests the field. Removing a field from the preset doesn’t remove its expected answer from the accuracy calculation.
The evaluator excludes an expected value incompatible with a field’s type in the executed schema from value accuracy. Studio flags the type mismatch so you can update the answer key or preset. If no eligible values remain, the document has no value accuracy score.
What similarity means
A string miss can show a similarity percentage to distinguish small text differences from unrelated answers. “Miss · 88% similar” still means a miss. Similarity doesn’t award partial credit for extract values.
Extract uses fixed matching rules without per-field fuzzy thresholds, large language model (LLM) judges, or vector matching settings. Row similarity helps align table rows but doesn’t make differing values correct.
Citation accuracy
Extract citation accuracy checks whether the returned source region agrees with the answer key’s labeled region. The evaluator scores citations separately from values, so a correct value can still have a citation miss.
Regions must share a page and overlap sufficiently to match. The default intersection-over-union threshold is 0.5. At this threshold, the overlapping area must cover at least half the combined area. A wider region that includes a label can therefore miss a smaller region around just the value.
For a scalar field with several labeled regions, citing any one is enough. Indexed table citations follow the row pairing from value scoring. The evaluator checks a reordered row against its paired answer.
Only eligible labeled citations contribute to the score. The evaluator excludes these cases:
- Fields without expected citations.
- Empty expected values.
- Regions without usable page dimensions.
A missing returned citation counts as a miss when its expected region qualifies for scoring, including when the executed preset no longer requests the field.
“0/1 citations” means the single scored citation expectation didn’t match. The engine might have returned a source region elsewhere rather than no region at all.
Grounding and source warnings
Grounding measures the engine’s support for its output. The grounded percentage and parse confidence score describe engine metadata, not agreement with the answer key. Studio displays these scores when available, but the evaluator doesn’t include them in value or citation accuracy.
If the engine returns "completed" with not_found metadata, the actual value remains "completed". This value misses against expected null and can match against expected "completed". Studio still shows a source-location warning.
Suggested answer keys also preserve returned values. Review unsupported suggestions before relying on evaluation scores. Agreement between two runs doesn’t prove that the document supports the answer.
Parse and classify
Parse compares expected and returned Markdown using line-level similarity. Its document score is continuous, so partially matching Markdown can receive partial credit. The markdown field verdict uses a default similarity threshold of 0.8. This threshold doesn’t enable fuzzy matching for extract fields.
Classify compares the returned top label with the expected label exactly. A document receives either a match or a miss.
Neither operation receives an extract citation score.
Run and field metrics
The evaluator calculates accuracy for runs and individual fields as follows:
- Run answer accuracy — The mean accuracy of scored documents. Each document has equal weight.
- Field value accuracy — Total matched values divided by total scored values for that field across documents. Table columns count values across all rows.
- Run citation accuracy — The mean citation accuracy of documents with scored citations.
- Field citation accuracy — Total matched citation expectations divided by total scored expectations for the field.
Errored and skipped items don’t contribute an accuracy score. A high percentage on an incomplete run doesn’t mean every document succeeded. Check item statuses and counts before comparing runs.
The history shows these metrics for each run instead of combining different preset versions into one score. Answer-key edits and changes to the document set can also affect comparisons.
For the Studio workflow, refer to the overview guide. For API workflows, refer to the API guide.