Label form fields with a custom local vision endpoint in C#

Form-field detection locates the fillable regions on a page, but a bounding box alone doesn’t tell you what each field means. AI labeling adds a human-readable semantic label to each detected field, such as “First name” or “Date of birth”, by sending the page to a vision language model (VLM).

This sample builds on offline form-field detection. For the detection basics — and a fully offline workflow with no model contacted — refer to the extract form fields from an image guide. Here you connect a custom local VLM endpoint and turn labeling on with Nutrient .NET SDK.

How Nutrient helps

Nutrient .NET SDK runs detection and labeling behind a single method call. With labeling enabled, it also:

• Draws numbered marks over each detected field on a rendered copy of the page
• Sends the annotated page to the VLM you select and reads each field’s semantic label back
• Optionally drops detections the model judges to be false positives
• Records each field’s type, bounding box, confidence, and assigned label in JSON

The result is structured data you can index, validate, or feed into a downstream workflow.

Prerequisites

This sample explicitly selects the Custom provider, which connects to an external OpenAI-compatible vision model server. The SDK doesn’t provision or start that server.

• Start a server such as LM Studio, Ollama, or vLLM and load a vision-capable model.
• Set the API endpoint and model identifier to match your server. Replace your-vision-model in the code with the loaded model’s identifier.
• Set an API key in the custom VLM API settings when your server requires authentication.
• Ensure the endpoint is reachable before processing documents.

Local server setup examples:

• LM Studio — Start server mode with a vision model loaded; the default endpoint is http://localhost:1234/v1.
• Ollama — Run ollama serve with a vision model; the default endpoint is http://localhost:11434/v1.
• vLLM — Launch a vision-capable model with an OpenAI-compatible server; configure its port and API key to match your deployment.

Use custom VLM API settings to configure the endpoint, model, authentication, and generation parameters.

Connect a vision model

Labeling uses the same provider configuration as the rest of the Vision API, so you don’t configure a separate endpoint for form labeling. Set the provider in vision settings and fill in the matching provider settings class:

• Custom — An OpenAI-compatible server such as LM Studio(opens in a new tab), Ollama, or vLLM. Configure custom VLM API settings.
• OpenAI — Configure OpenAI API endpoint settings.

Complete implementation

Start by importing the Nutrient namespace:

using Nutrient;

Load the document

Open the document with a using statement(opens in a new tab) so the handle is released when the block ends:

try
{
    using Document document = Document.Open("input_forms_detection.pdf");

Configure AI labeling

Select the provider, point it at your vision model, then opt in by setting EnableAiLabeling to true:

    DocumentSettings settings = document.Settings;

    // Select the vision model provider (the same setting Vision.Describe() uses)
    settings.VisionSettings.Provider = VlmProvider.Custom;

    // Configure the matching provider settings class
    CustomVlmApiSettings vlm = settings.CustomVlmApiSettings;
    vlm.ApiEndpoint = "http://localhost:1234/v1";
    vlm.Model = "your-vision-model";

    // Turn on labeling and drop detections the model judges to be false positives
    FormLabelingSettings formLabeling = settings.FormLabelingSettings;
    formLabeling.EnableAiLabeling = true;
    formLabeling.EnableAiRemoveFalsePositives = true;

    // Optional: constrain labels to a known vocabulary
    formLabeling.CandidateLabels = "First name, Last name, Date of birth, Signature";

Detect and label form fields

Create a vision instance from the document with Vision.Set(document), then call DetectForms(). The same call covers both modes; it includes labels because AI labeling is enabled:

    Vision vision = Vision.Set(document);
    string formsJson = vision.DetectForms();

Write the JSON result to a file for downstream processing:

    File.WriteAllText("output.json", formsJson);
}
catch (NutrientException e)
{
    Console.Error.WriteLine($"Error: {e.Message}");
    Environment.Exit(1);
}

Match labels to a vocabulary

Free-form labels can vary between runs (“First name” vs. “Given name”), which makes them hard to map to a database or template. Supply a vocabulary of preferred labels with the CandidateLabels property, as shown above, and the model maps each field to one when it fits. If no label fits, it invents a concise new label.

Pass the labels as newline- or comma-separated text. A matched label uses the casing you supplied, and each field’s labelSource records whether the label was matched or invented. Leave the candidate labels empty, which is the default, for free-form labeling.

Understand the output

DetectForms() returns structured JSON. The elements array holds one form element per page. Each form element includes its pageNumber and a fields list, so fields from a multi-page document stay grouped by the page they came from. Each field includes:

• fieldType — The detected type: Text, Checkbox, or Signature.
• bounds — The bounding box of the field on the page.
• confidence — The detection confidence for the field.
• label — The AI-assigned semantic label (for example, “First name”). Present only when AI labeling is enabled.
• labelSource — matched or invented, present only when a candidate vocabulary was supplied.
• id — A unique identifier for the field.

Handle errors

Vision API throws a NutrientException when detection or labeling fails.

Common failure scenarios include:

• The document can’t be read due to path or permission issues
• The page produces no renderable image
• The form detection model is missing or inaccessible, or the feature isn’t licensed
• AI labeling is enabled but the selected provider’s endpoint is unreachable

In production code:

• Catch NutrientException.
• Return a clear error message.
• Log failure details for debugging.
• Consider running detection only, with labeling disabled, as a fallback when the vision endpoint is unavailable.

Conclusion

The workflow for labeling form fields with a vision model is:

1. Open the source document with a using statement for automatic resource cleanup.
2. Select a provider with VisionSettings.Provider, configure the matching provider settings class, then set EnableAiLabeling to true on the form labeling settings.
3. Create a vision instance with Vision.Set().
4. Call DetectForms() to detect every field, assign a semantic label, and export the result as JSON.
5. Write the JSON to a file for indexing, validation, or downstream processing.
6. Handle NutrientException failures for robust error recovery.

Labeling adds semantic meaning when a vision model is available. For offline detection with no model contacted, refer to the extract form fields from an image guide. To produce a fillable PDF instead of data, refer to the detect and add form fields guide.

For related image extraction workflows, refer to the .NET SDK guides.

Download the sample package to explore form-field labeling.

To use Nutrient’s built-in model without a server, refer to the Local provider version of this sample.