An evaluation set contains documents and their expected answers for a saved preset. Run the same documents against different preset versions to check whether a change improves results or introduces regressions.
Evaluation sets support extract, parse, and classify presets. They don’t support split or forms presets. Each set holds up to 25 documents.
Use a small set of representative documents, including difficult layouts and missing information. Review the answer keys before using the scores to judge a preset.
Create a set in Studio
Each evaluation set stays associated with the preset you choose when creating it. To create a set, follow these steps:
- Open Evaluations in Data Extraction Studio.
- Create an evaluation set.
- Enter a name for the set.
- Choose a saved extract, parse, or classify preset.
- Select Add files.
- Upload your documents.
Studio runs the preset to propose an answer key for each document. Open an item to check its answer key against the document. Alternatively, select Review next.
To reuse a completed Studio run, select Add to evaluation set. Its retained document and output become an unreviewed answer key without processing the document again. Reimporting that run into the same set preserves your existing edits.
Your account requires access to evaluation sets. If Evaluations isn’t available, refer to the support guide to contact Support.
Review the answer key
The answer key defines what counts as correct for each preset type:
- Extract — Expected field values, including nested objects and table rows. You can also label the document regions that support each value.
- Parse — Expected Markdown for the document.
- Classify — The expected classification label.
To review an answer key, follow these steps:
- Correct the proposed values.
- Check the source regions.
- Select Save corrections.
- Select Mark as reviewed when you’ve finished checking the answer key.
The preset version shown with an answer key identifies the version that generated its initial suggestion.
Generated answers are suggestions. When the engine reports a value without a supporting source location, Studio shows “Source location not found.” Check the value against the document. Don’t accept it solely because a later run returns the same value.
Review status records your check without changing scoring or excluding unreviewed items. You can run a set with unreviewed answers, but every item requires an answer key. Resolve any pending or failed answer-key preparation before starting a run.
Run an evaluation
Studio selects the newest saved preset version by default, including drafts. This enables you to evaluate changes before publishing them.
To start an evaluation, follow these steps:
- Select Run.
- Choose a Preset version.
- Start the evaluation.
Each run records the preset configuration and saves snapshots of the current answer keys and expected citations. Later edits to an answer key or deletion of an item don’t change that run’s scores.
Evaluation runs process whole documents and use credits, just like other Data Extraction runs. Preparing suggested answer keys also processes documents. Reusing a completed run avoids that initial processing step.
A queued evaluation continues if you leave the page. Select Cancel run to stop the remaining work. Completed results stay available, and unfinished items change to Skipped. Cancellation doesn’t undo processing that already happened.
Inspect results and compare versions
Each run shows answer accuracy, citation accuracy, processing time, and individual document results. Open a run to inspect these results. Check failures and completion counts alongside percentages. Average accuracy excludes errored items.
Open a document result to compare the actual and expected values. The actual value shows what the engine returned. For extract results, nested objects and tables show the comparison for each value.
Use these actions to inspect a cited value’s source:
- Hold the pointer over the value to preview its source.
- Focus or click the value to keep its source highlighted.
- Press Escape or move focus away to clear the highlight.
The set’s history shows measurements for each run. To investigate changes in field accuracy, follow these steps:
- Select a field.
- Switch between Values and Citations.
Compare runs with consistent answer keys and documents. You can download the comparison and available stored output from a result. For the exact matching rules, refer to the scoring guide.
Maintain a useful regression set
Changes to labels or preset fields can require updates to your answer keys.
When you discover labeling mistakes, update the answer keys. Then run the evaluation again. If a preset adds, removes, or changes fields, inspect Studio’s schema-change notices. Decide whether the answer key also requires changes.
Removing a requested field from the preset doesn’t remove its expected answer from evaluation. If the answer key still expects the field and the run doesn’t return it, the field counts as a miss. Studio shows newly returned fields without expected answers but doesn’t score them.
Historical results retain their recorded scores. Start a new run to evaluate changes to your preset or answer keys.
Next steps
Refer to these guides for scoring details, API workflows, and stored results:
- For value accuracy, citation accuracy, and grounding, refer to the scoring guide.
- To manage evaluation sets through the API, refer to the API guide.
- For stored processing runs and retention, refer to the runs guide.