---
title: "Describe images with a custom local AI endpoint | Nutrient Java SDK"
canonical_url: "https://www.nutrient.io/guides/java/extraction/describe-image-with-custom-local-ai/"
md_url: "https://www.nutrient.io/guides/java/extraction/describe-image-with-custom-local-ai.md"
last_updated: "2026-10-27T00:00:00.000Z"
description: "Generate image descriptions with Nutrient Java SDK using a custom local OpenAI-compatible vision endpoint. Configure the provider, endpoint, and model."
---

# Describe images with a custom local AI endpoint

Use a custom local AI endpoint for image description when you need privacy, cost control, or an existing OpenAI-compatible vision model server.

Common use cases include:

- On-premises accessibility workflows

- Medical or regulated environments with local-only data handling

- Secure networks where external APIs are restricted

- High-volume processing without per-image API fees

- Offline-capable applications

This guide explicitly selects the Custom provider to use a local OpenAI-compatible VLM endpoint with Nutrient Vision API. The SDK's default Local provider instead runs a model in process.

[Download sample](https://www.nutrient.io/downloads/samples/java/describe-image-with-custom-local-ai.zip)

## How Nutrient helps

Nutrient Java SDK handles OpenAI-compatible VLM endpoint configuration and request/response processing.

The SDK handles:

- OpenAI-compatible endpoint formatting and communication

- Image encoding and multimodal payload construction

- Model parameters such as temperature and max tokens

- Local server and model runtime failure handling

## Opening the image file and configuring the local server

Open the image in try-with-resources and configure local endpoint settings.

In this sample:

- The SDK supports PNG, JPEG, GIF, BMP, and TIFF.

- The Custom provider endpoint defaults to `http://localhost:1234/v1`.

- Configure the Custom provider model to the vision-capable model loaded by your endpoint.

- The sample explicitly selects the Custom provider before configuring it.

## Creating a vision instance

Create a vision instance with `Vision.set(document)`.

Before calling vision methods, ensure:

- The local server is running.

- A vision-capable model is loaded.

- The configured endpoint is reachable.

## Generating the description

Call `vision.describe()` to generate a natural language description.

The SDK handles image encoding, request construction, and response parsing.

## Saving the description

Write the description to a text file, or store it in your application.

Common destinations include:

- Database fields

- JSON output files

- HTML `alt` attributes

## Configuring the VLM API endpoint

The Custom provider uses OpenAI-compatible endpoints. Configure `CustomVlmApiSettings` if your setup differs from the Custom defaults:

- `setApiEndpoint()` — Base URL for the OpenAI-compatible API (default: `http://localhost:1234/v1`)

- `setApiKey()` — API key for secured deployments (optional on many local servers)

- `setModel()` — Model identifier loaded by your endpoint

- `setTemperature()` — Creativity control (`0.0` deterministic, `1.0` varied phrasing)

- `setMaxTokens()` — Response token limit (`-1` unlimited)

### Choosing a model size

Larger models read charts, plots, and dense figures more accurately, but they need more memory. Pick the largest compatible model that fits your GPU. Memory requirements vary by model, quantization, and runtime.

**Local server setup examples**:

- **LM Studio** — Start server with vision model loaded, default endpoint `http://localhost:1234/v1`

- **Ollama** — Run `ollama serve` with vision model, default endpoint `http://localhost:11434/v1`

- **vLLM** — Launch with `--api-key` flag for authentication, custom port configuration

The SDK handles request formatting and response parsing for local endpoints.

## Complete implementation

This example generates image descriptions using a local OpenAI-compatible endpoint:

```java

package io.nutrient.Sample;

import io.nutrient.sdk.Document;
import io.nutrient.sdk.Vision;
import io.nutrient.sdk.enums.VlmProvider;
import io.nutrient.sdk.exceptions.NutrientException;

import java.io.FileWriter;
import java.io.IOException;

public class DescribeImageWithCustomLocalAi {
    public static void main(String[] args) throws NutrientException, IOException {
        try (Document document = Document.open("input_photo.png")) {
            document.getSettings().getVisionSettings().setProvider(VlmProvider.Custom);

            // Configure the selected Custom provider's endpoint settings.
            var vlmSettings = document.getSettings().getCustomVlmApiSettings();
            vlmSettings.setApiEndpoint("http://localhost:1234/v1");
            vlmSettings.setModel("your-vision-model");

            Vision vision = Vision.set(document);
            String description = vision.describe();

            try (FileWriter writer = new FileWriter("output.txt")) {
                writer.write(description);
            }
        }
    }
}

```

## Understanding the output

`describe()` returns natural language text generated by the local model.

Descriptions are typically:

- **Concise** — Focused on key subjects and details, often one to three sentences

- **Accessible** — Suitable for users who rely on screen readers

- **Accurate** — Based on visible content only

- **Model-dependent quality** — Quality varies by model architecture and size

Use this output for accessibility metadata, image search, and document workflows, all with local processing.

## Error handling

The sample can throw:

- `NutrientException` for vision and endpoint issues

- `IOException` for file I/O operations

Common failure scenarios include:

- The input image can’t be read due to path, permission, or format issues

- The local VLM server isn’t running or reachable

- No compatible vision model is loaded

- Responses time out on large images or heavy models

- Available CPU/GPU memory is insufficient

- Endpoint or authentication settings are invalid

- File writing fails due to path, disk, or permission issues

In production code:

- Catch `NutrientException` and `IOException`.

- Return clear error messages.

- Log failure details for debugging.

- Add retry logic for transient local server issues.

## Conclusion

Use this workflow to generate image descriptions with a custom local AI endpoint:

1. Open the image file using try-with-resources for automatic resource cleanup.

2. The SDK supports multiple image formats, including PNG, JPEG, GIF, BMP, and TIFF.

3. Select the Custom provider for an external OpenAI-compatible local VLM server; the SDK defaults to the in-process Local provider.

4. Configure the Custom provider endpoint and model to match your local server.

5. Create a vision instance with `Vision.set()` bound to the document.

6. Generate the description with `vision.describe()`, which sends the image to the local server endpoint and returns natural language text.

7. Handle `NutrientException` and `IOException` for endpoint, vision processing, and file operation failures.

For zero-configuration local image description, refer to the [local AI image description guide](https://www.nutrient.io/guides/java/extraction/describe-image-with-local-ai.md). For related image workflows, refer to the [Java SDK guides](https://www.nutrient.io/guides/java.md).

Download [this ready-to-use sample package](https://www.nutrient.io/downloads/samples/java/describe-image-with-custom-local-ai.zip) to explore image description with a custom local AI endpoint.
---

## Related pages

- [Nutrient Java SDK extraction guides](/guides/java/extraction.md)
- [Applying OCR to a PDF page](/guides/java/extraction/apply-ocr-to-pdf-page.md)
- [Applying OCR to a PDF document](/guides/java/extraction/apply-ocr-to-pdf.md)
- [Classifying documents](/guides/java/extraction/classify-document.md)
- [Making decisions from JSON data](/guides/java/extraction/decide-document.md)
- [Generating image descriptions using Claude](/guides/java/extraction/describe-image-with-claude.md)
- [Generating image descriptions using local AI](/guides/java/extraction/describe-image-with-local-ai.md)
- [Generating image descriptions using OpenAI](/guides/java/extraction/describe-image-with-openai.md)
- [Detecting document language](/guides/java/extraction/detect-document-language.md)
- [Extracting data from images using ICR](/guides/java/extraction/extract-data-from-image-icr.md)
- [Extracting data from images using OCR](/guides/java/extraction/extract-data-from-image-ocr.md)
- [Extracting data from images using vision language models](/guides/java/extraction/extract-data-from-image-vlm.md)
- [Extracting data from specific pages](/guides/java/extraction/extract-data-from-specific-pages.md)
- [Extracting form fields from images](/guides/java/extraction/extract-form-fields-from-image.md)
- [Extracting structured data from documents](/guides/java/extraction/extract-structured-data.md)
- [Generating extraction schemas](/guides/java/extraction/generate-extraction-schema.md)
- [Extracting JSON data from a PDF document](/guides/java/extraction/json-data-extraction.md)
- [Labeling form fields with a vision language model](/guides/java/extraction/label-form-fields-with-vlm.md)
- [Opening password-protected PDFs](/guides/java/extraction/open-password-protected-pdf.md)
- [Extracting text from PDF documents](/guides/java/extraction/pdf-to-text.md)
- [Reading barcodes with vision extraction](/guides/java/extraction/read-barcodes-with-vision.md)
- [Extracting text from multilingual images](/guides/java/extraction/read-text-from-image-multi-language.md)
- [Extracting text from images](/guides/java/extraction/read-text-from-image.md)
- [Searching document text](/guides/java/extraction/search-document-text.md)
- [Speeding up first ICR operation by predownloading models](/guides/java/extraction/speed-up-first-icr-by-downloading-requirements.md)
- [Split documents](/guides/java/extraction/split-document.md)