--- title: "Evaluation sets API" canonical_url: "https://www.nutrient.io/guides/dws-data-extraction/evaluations/api/" md_url: "https://www.nutrient.io/guides/dws-data-extraction/evaluations/api.md" last_updated: "2026-10-27T00:00:00.000Z" description: "Create evaluation sets, maintain answer keys, run preset versions, and read value and citation metrics through the Data Extraction evaluation endpoints." --- # Evaluation sets API Use the API to create evaluation sets, manage answer keys, and compare preset versions. Run this workflow from your tools or continuous integration (CI) pipeline. Send requests to `https://api.nutrient.io`. Authenticate with a Data Extraction API key in the `Authorization: Bearer` header. The key’s account requires access to evaluation sets, and endpoints can return `404` when the account hasn’t enabled the feature. The API calls a saved preset a processor, and its ID starts with `proc_`. Create or select the preset in Studio before creating a set. ## Create a set Each set uses one parse, extract, or classify preset. You can’t change the preset associated with the set later. Send `POST /extraction/evaluations` with a JSON body containing `name` and `processorId`, and optionally include a `description`. The response returns the set’s `id`, which starts with `eval_`. Use these endpoints to manage the set: | Action | Method and path | | -------------------------------- | -------------------------------------------------- | | List sets | `GET /extraction/evaluations` | | Read a set and its current items | `GET /extraction/evaluations/{evaluationSetId}` | | Edit its name or description | `PATCH /extraction/evaluations/{evaluationSetId}` | | Delete a set | `DELETE /extraction/evaluations/{evaluationSetId}` | ## Add documents and expected answers The API supports three ways to add an item. Send `POST /extraction/evaluations/{evaluationSetId}/items` using one of these request formats: | Input | Request format | | -------------------------------------- | -------------------------------------------------------------------------------------------------------- | | A completed stored run | Send JSON containing only `fromRunId`. The retained document and output become an unreviewed answer key. | | A document with your own answers | Send multipart `document` and `expectedOutput`. Encode `expectedOutput` as a JSON object string. | | A document requiring suggested answers | Send multipart `document` and `prefill=true`. Don’t include `expectedOutput`. | Use the answer format that matches the preset: - **Extract** — Use the object containing the expected field values. - **Parse** — Use an object with a `markdown` string. - **Classify** — Use an object with a `label` string. For extract items, include optional `expectedCitations` keyed by field path. Encode `expectedCitations` as a JSON string in multipart requests. Each region contains a zero-based `pageIndex` and a `rect` object. The `rect` contains `x`, `y`, `width`, and `height` as normalized page fractions. A set holds at most 25 items. The API limits serialized expected answers to 2 MiB for parse and 1 MiB for extract and classify. These limits include JSON property names and escaping. Import requires a completed stored run for the same preset, with retained input and output. You can import ordinary processing runs and answer-key preparation runs. The API rejects runs created to score an evaluation with `run_not_importable`. The `importOutcome` field returns `added` or `existing`. Repeating an import preserves the existing answer key and review status. If that item’s answer-key preparation is pending or failed, the API returns `answer_key_not_ready`. Resolve preparation before importing again. Generating answers with `prefill=true` requires queued processing for your account. Without it, the API returns `422 prefill_unavailable` before creating an item. Supply `expectedOutput` or import an eligible completed run instead. After the API creates an item for generated answers, poll the item or set until `prefill.status` settles. A failed prefill returns an error, so correct the reported problem before retrying. For `answer_key_too_large`, use a shorter document or an answer within the size limit instead of retrying. Use these endpoints to manage items: | Action | Method and path | | ----------------------------------------- | ----------------------------------------------------------------------- | | Read an item | `GET /extraction/evaluations/{evaluationSetId}/items/{itemId}` | | Edit answers, citations, or review status | `PATCH /extraction/evaluations/{evaluationSetId}/items/{itemId}` | | Retry answer-key preparation | `POST /extraction/evaluations/{evaluationSetId}/items/{itemId}/prefill` | | Download its document | `GET /extraction/evaluations/{evaluationSetId}/items/{itemId}/document` | | Delete an item | `DELETE /extraction/evaluations/{evaluationSetId}/items/{itemId}` | Use `expectedOutput`, `expectedCitations`, and `reviewed` to update an answer key. Review status doesn’t exclude an item from scoring. ## Run a preset version Each run uses a specific preset version. Send `POST /extraction/evaluations/{evaluationSetId}/runs`. To select a version, include the optional `processorVersion` field in the JSON body. Without this field, the API uses the newest version, including drafts. Every item requires an answer key. The API rejects runs with `items_not_ready` while any item has pending or failed answer-key preparation. The API returns an evaluation run ID starting with `evr_`. Follow the workflow for the returned `execution` value: - **Queued execution (`queued`)** — The server processes and scores the documents. Poll `GET /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId}` until the run reaches a terminal status. - **Client execution (`client`)** — Execute each item through `POST /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId}/items/{itemId}/execute`, and then read the run. Creating the run doesn’t process its items. Both modes perform billed processing. Terminal run statuses are `processed`, `error`, and `canceled`. A run with status `processed` can contain failed items. Check item statuses and `failedCount` alongside the run status. To cancel a run, send `POST /extraction/evaluations/{evaluationSetId}/runs/{evaluationRunId}/cancel`. The server preserves completed outcomes. The remaining behavior depends on execution mode: - **Queued execution** — Unfinished items become skipped. Late results don’t change the canceled evaluation. - **Client execution** — Pending items become skipped. An active execute request can still finish and record its result; cancellation doesn’t abort that request. Cancellation doesn’t undo processing that already happened. ## Read and compare results Run summaries provide accuracy metrics for comparing results. List runs with `GET /extraction/evaluations/{evaluationSetId}/runs`. Summaries include `accuracy`, `fieldAccuracy`, `citationAccuracy`, and `fieldCitationAccuracy`. The API expresses scores as fractions from zero to one. A `null` accuracy means the API scored nothing. For calculation details, refer to the [scoring](https://www.nutrient.io/guides/dws-data-extraction/evaluations/scoring.md) guide. Read a run’s details for the following information: - Item outcomes. - Snapshots of expected answers. - Field scores and row pairings. - Citation tallies. Each item’s `runId` identifies its underlying processing run, which provides retained raw output and assets. For details, refer to the [runs](https://www.nutrient.io/guides/dws-data-extraction/runs.md) guide. An evaluation run saves snapshots of its preset configuration and expected answers. Later edits apply to future runs and don’t change historical scores. Use per-field summaries to detect regressions without fetching every document’s full output. Define a regression policy for your automated workflow. Check these factors alongside your accuracy target: - Failed and skipped items. - Reviewed answer keys. - Consistent answer keys and documents. --- ## Related pages - [Evaluation sets](/guides/dws-data-extraction/evaluations.md) - [How evaluation scoring works](/guides/dws-data-extraction/evaluations/scoring.md)