---
title: "Extract image data with a custom local VLM in C# | Nutrient .NET SDK"
canonical_url: "https://www.nutrient.io/guides/dotnet/csharp/extraction/extract-data-from-image-with-custom-local-vlm/"
md_url: "https://www.nutrient.io/guides/dotnet/csharp/extraction/extract-data-from-image-with-custom-local-vlm.md"
last_updated: "2026-10-28T00:00:00.000Z"
description: "Extract structured image data with the Nutrient.SDK package using VLM-enhanced ICR and a custom local OpenAI-compatible vision endpoint configured in C#."
---

# Extract image data with a custom local vision endpoint in C#

Use VLM-enhanced ICR with a custom local vision model endpoint when you need higher extraction accuracy on complex documents.

Common use cases include:

- Financial documents with complex tables

- Invoices with varied layouts

- Medical records with specialized terminology

- Legal documents with strict structure requirements

- Multi-language document analysis

VLM-enhanced mode combines ICR layout analysis with language-model reasoning to improve classification and structure detection.

[Download sample](https://www.nutrient.io/downloads/samples/csharp/extract-data-from-image-with-custom-local-vlm.zip)

## How Nutrient helps

Nutrient.NET SDK handles VLM-enhanced configuration, model orchestration, and JSON output generation.

The SDK handles:

- Hybrid mode configuration for ICR + VLM processing

- Model loading and capability coordination

- Semantic classification and confidence scoring internals

- Complex layout analysis implementation details

## Prerequisites

This sample explicitly selects the Custom provider, which connects to an external OpenAI-compatible vision model server. The SDK doesn't provision or start that server.

- Start a server such as LM Studio, Ollama, or vLLM and load a vision-capable model.

- Set the API endpoint and model identifier to match your server. Replace `your-vision-model` in the code with the loaded model's identifier.

- Set an API key in the custom VLM API settings when your server requires authentication.

- Ensure the endpoint is reachable before processing documents.

Local server setup examples:

- **LM Studio** — Start server mode with a vision model loaded; the default endpoint is `http://localhost:1234/v1`.

- **Ollama** — Run `ollama serve` with a vision model; the default endpoint is `http://localhost:11434/v1`.

- **vLLM** — Launch a vision-capable model with an OpenAI-compatible server; configure its port and API key to match your deployment.

Use [custom VLM API settings](https://www.nutrient.io/api/csharp/settings/vision/advanced/custom-vlm-api-settings.md) to configure the endpoint, model, authentication, and generation parameters.

## Complete implementation

This example extracts structured JSON using `VisionEngine.VlmEnhancedIcr`:

```csharp

using Nutrient;

```

## Loading and processing the image

Open the image with a [using statement](https://learn.microsoft.com/dotnet/csharp/language-reference/statements/using) so the handle is released when the block ends:

```csharp

try
{
    using Document document = Document.Open("input.png");

```

## Configuring VLM-enhanced mode

Set the vision engine to `VisionEngine.VlmEnhancedIcr`, select the Custom provider, and configure your server endpoint and model.

This mode improves:

- Table boundary detection

- Semantic element classification

- Reading order in complex layouts

- Understanding across document variations

```csharp

    document.Settings.VisionSettings.Engine = VisionEngine.VlmEnhancedIcr;
    document.Settings.VisionSettings.Provider = VlmProvider.Custom;

    CustomVlmApiSettings vlmSettings = document.Settings.CustomVlmApiSettings;
    vlmSettings.ApiEndpoint = "http://localhost:1234/v1";
    vlmSettings.Model = "your-vision-model";

```

## Creating a vision instance

Create a vision instance bound to the opened document with `Vision.Set(document)`:

```csharp

    Vision vision = Vision.Set(document);

```

## Extracting structured content

Call `ExtractContent()` to run the VLM-enhanced pipeline.

In this mode, the pipeline performs:

- Initial ICR layout detection

- VLM-based semantic refinement

- Confidence scoring

- JSON generation with structure and coordinates

```csharp

    string contentJson = vision.ExtractContent();

```

Write the JSON result to a file for downstream processing.

Use this output for indexing, validation, storage, or custom analysis:

```csharp

    File.WriteAllText("output.json", contentJson);
}
catch (NutrientException e)
{
    Console.Error.WriteLine($"Error: {e.Message}");
    Environment.Exit(1);
}

```

## Understanding the output

`ExtractContent()` returns structured JSON with layout and semantic metadata.

VLM-enhanced output includes:

- **Document elements** — Paragraphs, headings, tables, figures, equations, detected barcodes, and form-related regions

- **Barcode data** — Decoded barcode values with symbology information for supported 1D and 2D barcode types

- **Bounding boxes** — Pixel coordinates with improved boundary accuracy

- **Hierarchical relationships** — Parent-child structure across sections and blocks

- **Element classification** — Semantic types with confidence scores

- **Reading order** — Sequence for complex layouts and multicolumn content

- **Semantic metadata** — Additional attributes used in downstream processing

### Key output fields

The following are the most commonly included fields in VLM JSON output:

- **`text`** — Extracted text for the element.

- **`words`** — Per-word OCR/extraction results.

- **`bounds`** — Bounding box coordinates for the element or word.

- **`confidence`** — Confidence score for the element or word.

- **`readingOrder`** — Sequence in which elements should be read.

- **`id`** — Unique identifier for the extracted element.

- **`pageNumber`** — Source page number.

- **`type` / `role`** — Semantic type of the extracted block, such as text, heading, table, image, or barcode.

When an element contains only one word, element-level and word-level `bounds`/`confidence` can appear identical.

### Confidence fields in VLM output

VLM output can contain two distinct confidence signals:

1. **`confidence` (or `classificationConfidence`)** — Zone classification confidence
   - **Definition**: How confident the model is in semantic classification (for example, text, heading, table, image), heading level detection, and language detection.
   - **Scale**: `0.0` to `1.0` (float).
   - **Interpretation**:
     - `0.0` = no confidence (often treated as unknown classification)
     - `1.0` = maximum confidence
   - **Use**: Decide whether to trust semantic zone labels in downstream logic.

2. **`textConfidence`** — Text extraction confidence
   - **Definition**: How confident the model is in the extracted text quality for a zone.
   - **Scale**: Categorical values: `high`, `medium`, `low` (not numeric).
   - **Interpretation**:
     - `high` = strong confidence in extracted text
     - `medium` = moderate confidence
     - `low` = uncertain text quality
   - **Use**: Prioritize review, fallback, or fusion strategies for lower-confidence text.

Use this JSON for form extraction, contract analysis, invoice parsing, and other high-accuracy workflows.

## Error handling

Vision API throws a `NutrientException` when extraction fails.

Common failure scenarios include:

- The image file can't be read due to path or permission issues

- Image data is corrupted or unsupported

- Required models are missing or inaccessible

- Available memory is insufficient for VLM-enhanced processing

- VLM enhancement fails due to connectivity or service issues when applicable

- Image format, resolution, or dimensions are unsupported

In production code:

- Catch `NutrientException`.

- Return a clear error message.

- Log failure details for debugging.

- Add fallback logic (for example, retry in ICR mode).

## Conclusion

Use this workflow for VLM-enhanced extraction:

1. Open the image document with a `using` statement for automatic resource cleanup.

2. Set `Engine` to `VisionEngine.VlmEnhancedIcr`, select the Custom provider, and configure the endpoint and model.

3. VLM-enhanced mode combines local ICR AI models with vision language model capabilities for superior document analysis.

4. Create a vision instance with `Vision.Set()` to bind content extraction operations to the document.

5. Call `ExtractContent()` to invoke the VLM-enhanced processing pipeline.

6. The pipeline performs initial ICR layout analysis, applies VLM enhancement for semantic understanding, calculates confidence scores, and generates JSON output.

7. VLM enhancement improves table cell boundary detection, element classification accuracy, and reading order determination for complex layouts.

8. The method returns a JSON-formatted string containing document structure with elements, bounding boxes, hierarchical relationships, reading order, and confidence scores.

9. Write the JSON content to a file for intelligent form extraction, contract analysis, invoice processing, and legal document parsing.

10. Handle `NutrientException` failures for robust error recovery with fallback strategies like pure ICR mode.

11. VLM-enhanced mode is ideal for complex documents where extraction accuracy is the priority.

For related image extraction workflows, refer to the [.NET SDK guides](https://www.nutrient.io/guides/dotnet/csharp.md).

Download [this ready-to-use sample package](https://www.nutrient.io/downloads/samples/csharp/extract-data-from-image-with-custom-local-vlm.zip) to explore VLM-enhanced extraction.

To use Nutrient's built-in model without a server, refer to the [Local provider version of this sample](https://www.nutrient.io/guides/dotnet/csharp/extraction/extract-data-from-image-vlm.md).
---

## Related pages

- [Nutrient .NET SDK extraction guides](/guides/dotnet/csharp/extraction.md)
- [Applying OCR to a PDF page](/guides/dotnet/csharp/extraction/apply-ocr-to-pdf-page.md)
- [Applying OCR to a PDF document](/guides/dotnet/csharp/extraction/apply-ocr-to-pdf.md)
- [Classifying documents](/guides/dotnet/csharp/extraction/classify-document.md)
- [Generating image descriptions using Claude](/guides/dotnet/csharp/extraction/describe-image-with-claude.md)
- [Describe images with a custom local AI endpoint in C#](/guides/dotnet/csharp/extraction/describe-image-with-custom-local-ai.md)
- [Generating image descriptions using local AI](/guides/dotnet/csharp/extraction/describe-image-with-local-ai.md)
- [Generating image descriptions using OpenAI](/guides/dotnet/csharp/extraction/describe-image-with-openai.md)
- [Detecting document language](/guides/dotnet/csharp/extraction/detect-document-language.md)
- [Extracting data from images using ICR](/guides/dotnet/csharp/extraction/extract-data-from-image-icr.md)
- [Extracting data from images using OCR](/guides/dotnet/csharp/extraction/extract-data-from-image-ocr.md)
- [Extracting data from images using vision language models](/guides/dotnet/csharp/extraction/extract-data-from-image-vlm.md)
- [Extracting data from specific pages](/guides/dotnet/csharp/extraction/extract-data-from-specific-pages.md)
- [Extracting form fields from images](/guides/dotnet/csharp/extraction/extract-form-fields-from-image.md)
- [Extracting structured data from documents](/guides/dotnet/csharp/extraction/extract-structured-data.md)
- [Generate extraction schemas with a custom local vision endpoint in C#](/guides/dotnet/csharp/extraction/generate-extraction-schema-with-custom-local-vlm.md)
- [Generating extraction schemas](/guides/dotnet/csharp/extraction/generate-extraction-schema.md)
- [Extracting JSON data from a PDF document](/guides/dotnet/csharp/extraction/json-data-extraction.md)
- [Label form fields with a custom local vision endpoint in C#](/guides/dotnet/csharp/extraction/label-form-fields-with-custom-local-vlm.md)
- [Labeling form fields with a vision language model](/guides/dotnet/csharp/extraction/label-form-fields-with-vlm.md)
- [Parsing a document into structured content](/guides/dotnet/csharp/extraction/parse-document.md)
- [Extracting text from PDF documents](/guides/dotnet/csharp/extraction/pdf-to-text.md)
- [Reading barcodes with vision extraction](/guides/dotnet/csharp/extraction/read-barcodes-with-vision.md)
- [Extracting text from multilingual images](/guides/dotnet/csharp/extraction/read-text-from-image-multi-language.md)
- [Extracting text from images](/guides/dotnet/csharp/extraction/read-text-from-image.md)
- [Searching document text](/guides/dotnet/csharp/extraction/search-document-text.md)
- [Speeding up first ICR operation by predownloading models](/guides/dotnet/csharp/extraction/speed-up-first-icr-by-downloading-requirements.md)