Evaluation sets API

Use the API to create evaluation sets, manage answer keys, and compare preset versions. Run this workflow from your tools or continuous integration (CI) pipeline.

Send requests to https://api.nutrient.io. Authenticate with a Data Extraction API key in the Authorization: Bearer header. The key’s account requires access to evaluation sets, and endpoints can return 404 when the account hasn’t enabled the feature.

The API calls a saved preset a processor, and its ID starts with proc_. Create or select the preset in Studio before creating a set.

Create a set

Each set uses one parse, extract, or classify preset. You can’t change the preset associated with the set later.

Send POST /extraction/evaluations with a JSON body containing name and processorId, and optionally include a description. The response returns the set’s id, which starts with eval_.

Use these endpoints to manage the set:

Add documents and expected answers

The API supports three ways to add an item. Send POST /extraction/evaluations/{evaluationSetId}/items using one of these request formats:

Use the answer format that matches the preset:

• Extract — Use the object containing the expected field values.
• Parse — Use an object with a markdown string.
• Classify — Use an object with a label string.

For extract items, include optional expectedCitations keyed by field path. Encode expectedCitations as a JSON string in multipart requests. Each region contains a zero-based pageIndex and a rect object. The rect contains x, y, width, and height as normalized page fractions.

A set holds at most 25 items. The API limits serialized expected answers to 2 MiB for parse and 1 MiB for extract and classify. These limits include JSON property names and escaping.

Import requires a completed stored run for the same preset, with retained input and output. You can import ordinary processing runs and answer-key preparation runs. The API rejects runs created to score an evaluation with run_not_importable.

The importOutcome field returns added or existing. Repeating an import preserves the existing answer key and review status. If that item’s answer-key preparation is pending or failed, the API returns answer_key_not_ready. Resolve preparation before importing again.

Generating answers with prefill=true requires queued processing for your account. Without it, the API returns 422 prefill_unavailable before creating an item. Supply expectedOutput or import an eligible completed run instead.

After the API creates an item for generated answers, poll the item or set until prefill.status settles. A failed prefill returns an error, so correct the reported problem before retrying. For answer_key_too_large, use a shorter document or an answer within the size limit instead of retrying.

Use these endpoints to manage items:

Use expectedOutput, expectedCitations, and reviewed to update an answer key. Review status doesn’t exclude an item from scoring.

Run a preset version

Each run uses a specific preset version. Send POST /extraction/evaluations/{evaluationSetId}/runs. To select a version, include the optional processorVersion field in the JSON body. Without this field, the API uses the newest version, including drafts.

Every item requires an answer key. The API rejects runs with items_not_ready while any item has pending or failed answer-key preparation.

The API returns an evaluation run ID starting with evr_. Follow the workflow for the returned execution value:

• Queued execution (queued) — The server processes and scores the documents. Poll GET /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId} until the run reaches a terminal status.
• Client execution (client) — Execute each item through POST /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId}/items/{itemId}/execute, and then read the run. Creating the run doesn’t process its items.

Both modes perform billed processing. Terminal run statuses are processed, error, and canceled. A run with status processed can contain failed items. Check item statuses and failedCount alongside the run status.

To cancel a run, send POST /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId}/cancel. The server preserves completed outcomes. The remaining behavior depends on execution mode:

• Queued execution — Unfinished items become skipped. Late results don’t change the canceled evaluation.
• Client execution — Pending items become skipped. An active execute request can still finish and record its result; cancellation doesn’t abort that request.

Cancellation doesn’t undo processing that already happened.

Read and compare results

Run summaries provide accuracy metrics for comparing results. List runs with GET /extraction/evaluations/{evaluationSetId}/runs. Summaries include accuracy, fieldAccuracy, citationAccuracy, and fieldCitationAccuracy.

The API expresses scores as fractions from zero to one. A null accuracy means the API scored nothing. For calculation details, refer to the scoring guide.

Read a run’s details for the following information:

• Item outcomes.
• Snapshots of expected answers.
• Field scores and row pairings.
• Citation tallies.

Each item’s runId identifies its underlying processing run, which provides retained raw output and assets. For details, refer to the runs guide.

An evaluation run saves snapshots of its preset configuration and expected answers. Later edits apply to future runs and don’t change historical scores. Use per-field summaries to detect regressions without fetching every document’s full output.

Define a regression policy for your automated workflow. Check these factors alongside your accuracy target:

• Failed and skipped items.
• Reviewed answer keys.
• Consistent answer keys and documents.