Label form fields with a local vision language model

Form-field detection locates fillable regions on a page, but a bounding box alone doesn’t tell you what each field means. AI labeling adds a human-readable semantic label to each detected field, such as “First name” or “Date of birth”, by sending the page to a vision language model (VLM).

Nutrient Python SDK 1.0.12 and later include a zero-configuration Local provider. The Local provider runs a vision model in process, downloads required model files automatically on first use, and uses GPU acceleration when available.

How Nutrient helps

Nutrient Python SDK runs detection and labeling behind a single method call. With labeling enabled, it also:

• Draws numbered marks over each detected field on a rendered copy of the page
• Sends the annotated page to the local VLM and reads each field’s semantic label back
• Optionally drops detections the model judges to be false positives
• Records each field’s type, bounding box, confidence, fill state, and assigned label in JSON

The result is structured data you can index, validate, or feed into a downstream workflow.

Prerequisites

• Use Nutrient Python SDK 1.0.12 or later.
• Make sure the application can write to the SDK resource cache so model files can be stored after the first download.
• Allocate enough CPU/GPU memory for the local model.

If no AI provider is available, labeling fails at runtime. Leave enable_ai_labeling at its default of False to run detection only and keep the workflow offline.

Load the document

Open the document in a context manager(opens in a new tab) so resources are cleaned up after processing.

Configure AI labeling

The Local provider is the default in version 1.0.12 and later, so no endpoint or model-server configuration is required. Turn labeling on with enable_ai_labeling.

Detect and label form fields

Create a vision instance from the document with Vision.set(document), then call detect_forms(). The same call covers both modes; it includes labels because enable_ai_labeling is set.

Write the JSON result to a file for downstream processing.

Match labels to a vocabulary

Free-form labels can vary between runs (“First name” vs. “Given name”), which makes them hard to map to a database or template. Supply a vocabulary of preferred labels with candidate_labels, as shown above, and the model maps each field to one when it fits. If no label fits, it invents a concise new label.

Pass the labels as newline- or comma-separated text. A matched label uses the casing you supplied, and each field’s labelSource records whether the label was matched or invented. Leave candidate_labels empty, which is the default, for free-form labeling.

Complete implementation

This example detects and labels form fields with the default Local provider:

from nutrient_sdk import Document, Vision, NutrientException


def main():
    try:
        with Document.open("input_forms_detection.pdf") as document:
            # Turn on labeling and drop detections the model judges to be false positives.
            form_labeling = document.settings.form_labeling_settings
            form_labeling.enable_ai_labeling = True
            form_labeling.enable_ai_remove_false_positives = True

            # Optional: constrain labels to a known vocabulary.
            form_labeling.candidate_labels = "First name, Last name, Date of birth, Signature"

            vision = Vision.set(document)
            forms_json = vision.detect_forms()

            with open("output.json", "w") as f:
                f.write(forms_json)
    except NutrientException as e:
        print(f"Error: {e}")


if __name__ == "__main__":
    main()

Understand the output

detect_forms() returns structured JSON. The elements array holds one form element per page. Each form element includes its pageNumber and a fields list, so fields from a multi-page document stay grouped by the page they came from. Each field includes:

• fieldType — The detected type: Text, Checkbox, or Signature.
• bounds — The bounding box of the field on the page.
• confidence — The detection confidence for the field.
• fillState — Whether the field is filled or empty. Version 1.0.12 reports fill state for checked or unchecked checkboxes, signed or blank signatures, and filled or empty text fields.
• label — The AI-assigned semantic label (for example, “First name”). Present only when AI labeling is enabled.
• labelSource — matched or invented, present only when a candidate vocabulary was supplied.
• id — A unique identifier for the field.

Handle errors

Vision API raises VisionException, which derives from NutrientException, when detection or labeling fails.

Common failure scenarios include:

• The document can’t be read due to path or permission issues
• The page produces no renderable image
• The form detection model is missing or inaccessible, or the feature isn’t licensed
• Required local model files can’t be downloaded or read from the SDK resource cache
• Available CPU/GPU memory is insufficient for the local model

In production code:

• Catch NutrientException.
• Return a clear error message.
• Log failure details for debugging.
• Consider running detection only, with labeling disabled, as a fallback when the local model is unavailable.

Conclusion

The workflow for labeling form fields with the Local provider is:

1. Open the source document using a context manager(opens in a new tab) for automatic resource cleanup.
2. Enable AI labeling with form_labeling_settings.enable_ai_labeling.
3. Create a vision instance with Vision.set().
4. Call detect_forms() to detect every field, assign a semantic label, and export the result as JSON.
5. Write the JSON to a file for indexing, validation, or downstream processing.
6. Handle NutrientException for robust error recovery.

Labeling adds semantic meaning when a vision model is available. For offline detection with no model contacted, refer to the extract form fields from an image guide. To use an external OpenAI-compatible VLM endpoint, refer to the label form fields with a VLM guide.

For related image extraction workflows, refer to the Python SDK guides.

Download the sample package to explore form-field labeling with the Local provider.