Generating image descriptions using local AI

Use local AI image description when you need privacy, cost control, or offline operation.

Common use cases include:

• On-premises accessibility workflows
• Medical or regulated environments with local-only data handling
• Secure networks where external APIs are restricted
• High-volume processing without per-image API fees
• Offline-capable applications

Nutrient Java SDK 2.1.12 and later include a Local provider for zero-configuration local AI. The Local provider runs a model in the SDK process, downloads required model files automatically on first use, and uses GPU acceleration when available.

How Nutrient helps

Nutrient Java SDK handles in-process local model loading, image preparation, and response parsing.

The SDK handles:

• Zero-configuration Local provider setup
• Automatic model downloads into the persistent SDK resource cache
• GPU acceleration when supported by the current system
• Image encoding and multimodal payload construction
• Model loading and processing errors

Complete implementation

This example generates image descriptions using the default in-process Local provider:

package io.nutrient.Sample;

Import required classes and define the sample class:

import io.nutrient.sdk.Document;
import io.nutrient.sdk.Vision;
import io.nutrient.sdk.enums.VlmProvider;
import io.nutrient.sdk.exceptions.NutrientException;

import java.io.FileWriter;
import java.io.IOException;

public class DescribeImageWithLocalAi {

Opening the image file

Create the main method and open the image in try-with-resources. The Local provider is the default in version 2.1.12 and later, so no endpoint or model-server configuration is required.

In this sample:

• Input can be PNG, JPEG, GIF, BMP, or TIFF.
• Required model files download automatically on first use.
• GPU acceleration is used when available.

    public static void main(String[] args) throws NutrientException, IOException {
        try (Document document = Document.open("input_photo.png")) {
            // Local is the default provider in 2.1.12 and later. This line is optional.
            document.getSettings().getVisionSettings().setProvider(VlmProvider.Local);

Creating a vision instance

Create a vision instance with Vision.set(document).

Before calling vision methods, ensure the process has access to the SDK resource cache location and enough memory for the local model.

            Vision vision = Vision.set(document);

Generating the description

Call vision.describe() to generate a natural language description.

The SDK handles image encoding, request construction, and response parsing:

            String description = vision.describe();

Saving the description

Write the description to a text file.

This sample uses try-with-resources for both document and file-writer cleanup:

            try (FileWriter writer = new FileWriter("output.txt")) {
                writer.write(description);
            }
        }
    }
}

Understanding the output

describe() returns natural language text generated by the local model.

Descriptions are typically:

• Concise — Focused on key subjects and details, often one to three sentences
• Accessible — Suitable for users who rely on screen readers
• Accurate — Based on visible content only
• Model-dependent quality — Quality varies by model architecture and size

Use this output for accessibility metadata, image search, and document workflows, all with local processing.

Error handling

The sample can throw:

• NutrientException for vision and local model issues
• IOException for file I/O operations

Common failure scenarios include:

• The input image can’t be read due to path, permission, or format issues
• Required local model files can’t be downloaded or read from the SDK resource cache
• Available CPU/GPU memory is insufficient for the local model
• Responses time out on large images or heavy models
• File writing fails due to path, disk, or permission issues

In production code:

• Catch NutrientException and IOException.
• Return clear error messages.
• Log failure details for debugging.
• Add retry logic for transient model download or processing issues.

Conclusion

Use this workflow to generate image descriptions with local AI:

1. Open the image file using try-with-resources for automatic resource cleanup.
2. The SDK supports multiple image formats, including PNG, JPEG, GIF, BMP, and TIFF.
3. The SDK uses the in-process Local provider by default, downloads required model files automatically, and uses GPU acceleration when available.
4. Create a vision instance with Vision.set() bound to the document.
5. Generate the description with vision.describe(), which processes the image with the Local provider and returns natural language text.
6. Write the description to a file using try-with-resources with FileWriter for automatic resource cleanup.
7. Handle NutrientException and IOException for vision processing and file operation failures.

To use an external OpenAI-compatible local VLM endpoint, refer to the custom local AI endpoint guide. For related image workflows, refer to the Java SDK guides.

Download this ready-to-use sample package to explore local AI image description.