---
title: "Label form fields with a local VLM | Nutrient Python SDK"
canonical_url: "https://www.nutrient.io/guides/python/extraction/label-form-fields-with-local-vlm/"
md_url: "https://www.nutrient.io/guides/python/extraction/label-form-fields-with-local-vlm.md"
last_updated: "2026-10-28T00:00:00.000Z"
description: "Detect form fields and assign semantic labels with Nutrient Python SDK using an in-process vision model, without configuring an external model server."
---

# Label form fields with a local vision language model

Form-field detection locates fillable regions on a page, but a bounding box alone doesn't tell you what each field means. AI labeling adds a human-readable semantic label to each detected field, such as "First name" or "Date of birth", by sending the page to a vision language model (VLM).

Nutrient Python SDK 1.0.12 and later include a zero-configuration Local provider. The Local provider runs a vision model in process, downloads required model files automatically on first use, and uses GPU acceleration when available.

[Download sample](https://www.nutrient.io/downloads/samples/python/label-form-fields-with-local-vlm.zip)

## How Nutrient helps

Nutrient Python SDK runs detection and labeling behind a single method call. With labeling enabled, it also:

- Draws numbered marks over each detected field on a rendered copy of the page

- Sends the annotated page to the local VLM and reads each field's semantic label back

- Optionally drops detections the model judges to be false positives

- Records each field's type, bounding box, confidence, fill state, and assigned label in JSON

The result is structured data you can index, validate, or feed into a downstream workflow.

## Prerequisites

- Use Nutrient Python SDK 1.0.12 or later.

- Make sure the application can write to the SDK resource cache so model files can be stored after the first download.

- Allocate enough CPU/GPU memory for the local model.

If no AI provider is available, labeling fails at runtime. Leave `enable_ai_labeling` at its default of `False` to run detection only and keep the workflow offline.

## Load the document

Open the document in a [context manager](https://docs.python.org/3/reference/datamodel.html#context-managers) so resources are cleaned up after processing.

## Configure AI labeling

The Local provider is the default in version 1.0.12 and later, so no endpoint or model-server configuration is required. Turn labeling on with `enable_ai_labeling`.

## Detect and label form fields

Create a vision instance from the document with `Vision.set(document)`, then call `detect_forms()`. The same call covers both modes; it includes labels because `enable_ai_labeling` is set.

Write the JSON result to a file for downstream processing.

## Match labels to a vocabulary

Free-form labels can vary between runs ("First name" vs. "Given name"), which makes them hard to map to a database or template. Supply a vocabulary of preferred labels with `candidate_labels`, as shown above, and the model maps each field to one when it fits. If no label fits, it invents a concise new label.

Pass the labels as newline- or comma-separated text. A matched label uses the casing you supplied, and each field's `labelSource` records whether the label was `matched` or `invented`. Leave `candidate_labels` empty, which is the default, for free-form labeling.

## Complete implementation

This example detects and labels form fields with the default Local provider:

```python

from nutrient_sdk import Document, Vision, NutrientException

def main():
    try:
        with Document.open("input_forms_detection.pdf") as document:
            # Turn on labeling and drop detections the model judges to be false positives.

            form_labeling = document.settings.form_labeling_settings
            form_labeling.enable_ai_labeling = True
            form_labeling.enable_ai_remove_false_positives = True

            # Optional: constrain labels to a known vocabulary.

            form_labeling.candidate_labels = "First name, Last name, Date of birth, Signature"

            vision = Vision.set(document)
            forms_json = vision.detect_forms()

            with open("output.json", "w") as f:
                f.write(forms_json)
    except NutrientException as e:
        print(f"Error: {e}")

if __name__ == "__main__":
    main()

```

## Understand the output

`detect_forms()` returns structured JSON. The `elements` array holds one form element per page. Each form element includes its `pageNumber` and a `fields` list, so fields from a multi-page document stay grouped by the page they came from. Each field includes:

- **`fieldType`** — The detected type: `Text`, `Checkbox`, or `Signature`.

- **`bounds`** — The bounding box of the field on the page.

- **`confidence`** — The detection confidence for the field.

- **`fillState`** — Whether the field is filled or empty. Version 1.0.12 reports fill state for checked or unchecked checkboxes, signed or blank signatures, and filled or empty text fields.

- **`label`** — The AI-assigned semantic label (for example, "First name"). Present only when AI labeling is enabled.

- **`labelSource`** — `matched` or `invented`, present only when a candidate vocabulary was supplied.

- **`id`** — A unique identifier for the field.

## Handle errors

Vision API raises `VisionException`, which derives from `NutrientException`, when detection or labeling fails.

Common failure scenarios include:

- The document can't be read due to path or permission issues

- The page produces no renderable image

- The form detection model is missing or inaccessible, or the feature isn't licensed

- Required local model files can't be downloaded or read from the SDK resource cache

- Available CPU/GPU memory is insufficient for the local model

In production code:

- Catch `NutrientException`.

- Return a clear error message.

- Log failure details for debugging.

- Consider running detection only, with labeling disabled, as a fallback when the local model is unavailable.

## Conclusion

The workflow for labeling form fields with the Local provider is:

1. Open the source document using a [context manager](https://docs.python.org/3/reference/datamodel.html#context-managers) for automatic resource cleanup.

2. Enable AI labeling with `form_labeling_settings.enable_ai_labeling`.

3. Create a vision instance with `Vision.set()`.

4. Call `detect_forms()` to detect every field, assign a semantic label, and export the result as JSON.

5. Write the JSON to a file for indexing, validation, or downstream processing.

6. Handle `NutrientException` for robust error recovery.

Labeling adds semantic meaning when a vision model is available. For offline detection with no model contacted, refer to the [extract form fields from an image](https://www.nutrient.io/guides/python/extraction/extract-form-fields-from-image.md) guide. To use an external OpenAI-compatible VLM endpoint, refer to the [label form fields with a VLM](https://www.nutrient.io/guides/python/extraction/label-form-fields-with-custom-local-vlm.md) guide.

For related image extraction workflows, refer to the [Python SDK](https://www.nutrient.io/guides/python.md) guides.

Download the [sample package](https://www.nutrient.io/downloads/samples/python/label-form-fields-with-local-vlm.zip) to explore form-field labeling with the Local provider.
---

## Related pages

- [Nutrient Python SDK extraction guides](/guides/python/extraction.md)
- [Applying OCR to a PDF page](/guides/python/extraction/apply-ocr-to-pdf-page.md)
- [Applying OCR to a PDF document](/guides/python/extraction/apply-ocr-to-pdf.md)
- [Classifying documents](/guides/python/extraction/classify-document.md)
- [Making decisions from JSON data](/guides/python/extraction/decide-document.md)
- [Generating image descriptions using Claude](/guides/python/extraction/describe-image-with-claude.md)
- [Describe images with a custom local AI endpoint](/guides/python/extraction/describe-image-with-custom-local-ai.md)
- [Generating image descriptions using local AI](/guides/python/extraction/describe-image-with-local-ai.md)
- [Generating image descriptions using OpenAI](/guides/python/extraction/describe-image-with-openai.md)
- [Detecting document language](/guides/python/extraction/detect-document-language.md)
- [Extracting data from images using ICR](/guides/python/extraction/extract-data-from-image-icr.md)
- [Extracting data from images using OCR](/guides/python/extraction/extract-data-from-image-ocr.md)
- [Extracting data from images using vision language models](/guides/python/extraction/extract-data-from-image-vlm.md)
- [Extracting data from specific PDF pages](/guides/python/extraction/extract-data-from-specific-pages.md)
- [Extracting form fields from images](/guides/python/extraction/extract-form-fields-from-image.md)
- [Extracting structured data from documents](/guides/python/extraction/extract-structured-data.md)
- [Generating extraction schemas](/guides/python/extraction/generate-extraction-schema.md)
- [Extracting structured JSON data from PDF documents](/guides/python/extraction/json-data-extraction.md)
- [Label form fields with a custom local vision endpoint](/guides/python/extraction/label-form-fields-with-custom-local-vlm.md)
- [Labeling form fields with a vision language model](/guides/python/extraction/label-form-fields-with-vlm.md)
- [Opening password-protected PDFs](/guides/python/extraction/open-password-protected-pdf.md)
- [Extracting text from PDF documents](/guides/python/extraction/pdf-to-text.md)
- [Reading barcodes with vision extraction](/guides/python/extraction/read-barcodes-with-vision.md)
- [Extracting text from multilingual images](/guides/python/extraction/read-text-from-image-multi-language.md)
- [Extracting text from images](/guides/python/extraction/read-text-from-image.md)
- [Later, in another process — no document needed:](/guides/python/extraction/search-document-text.md)
- [Speeding up first ICR operation by predownloading models](/guides/python/extraction/speed-up-first-icr-by-downloading-requirements.md)
- [Split documents](/guides/python/extraction/split-document.md)