Getting started with AI Assistant and Nutrient Android SDK

AI Assistant provides Nutrient Android SDK with AI functionality. Using intelligent document processing (IDP) technology, AI Assistant enables users to query, summarize, translate, and compare document text on the fly.

View example

Prefer to jump straight into code? View the example repo on GitHub.

To set up a fully functional AI system, you’ll need a Docker container service and a library working in unison:

Nutrient Android SDK — A document viewer for Android that also exposes a user interface (UI) for the AI features.
AI Assistant — A service to process the AI requests and process documents.

Prerequisites

AI Assistant is distributed as a Docker container. To run it on your computer, you need to install a Docker runtime distribution for your operating system.

Install and start Docker Desktop for Mac. Refer to the Docker website(opens in a new tab) for instructions.

Obtaining an OpenAI API key

AI Assistant requires an API key from either of these LLM providers:

OpenAI
Azure OpenAI

This example will use OpenAI, but if you want to use Azure OpenAI, refer to the Azure OpenAI guide.

If you don’t have an OpenAI key, create one by following the steps in the next section. Otherwise, skip to the setting up AI Assistant section.

Creating an OpenAI account

To create an OpenAI account, sign up(opens in a new tab) to obtain an API key(opens in a new tab).

The OpenAI API has attained SOC 2 Type II compliance (see the official announcement(opens in a new tab)).

Save your API key somewhere safe, as you’ll need it in the setting up AI Assistant step.

Setting up AI Assistant

AI Assistant requires a PostgreSQL database with the pgvector(opens in a new tab) extension to operate.

Copy the code snippet below and save it anywhere on your computer in a file called docker-compose.yml. Replace the <your-openai-api-key> placeholder with your OpenAI API key:

version: "3.8"

services:
  ai-assistant:
    image: pspdfkit/ai-assistant:nightly
    environment:
      OPENAI_API_KEY: <your-openai-api-key>
      PGUSER: db-user
      PGPASSWORD: password
      PGDATABASE: ai_assistant
      PGHOST: db
      PGPORT: 5432
      API_AUTH_TOKEN: secret
      JWT_PUBLIC_KEY: |
        -----BEGIN PUBLIC KEY-----
        MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2gzhmJ9TDanEzWdP1WG+
        0Ecwbe7f3bv6e5UUpvcT5q68IQJKP47AQdBAnSlFVi4X9SaurbWoXdS6jpmPpk24
        QvitzLNFphHdwjFBelTAOa6taZrSusoFvrtK9x5xsW4zzt/bkpUraNx82Z8MwLwr
        t6HlY7dgO9+xBAabj4t1d2t+0HS8O/ed3CB6T2lj6S8AbLDSEFc9ScO6Uc1XJlSo
        rgyJJSPCpNhSq3AubEZ1wMS1iEtgAzTPRDsQv50qWIbn634HLWxTP/UH6YNJBwzt
        3O6q29kTtjXlMGXCvin37PyX4Jy1IiPFwJm45aWJGKSfVGMDojTJbuUtM+8P9Rrn
        AwIDAQAB
        -----END PUBLIC KEY-----
      JWT_ALGORITHM: RS256
      DASHBOARD_USERNAME: dashboard
      DASHBOARD_PASSWORD: secret
      SECRET_KEY_BASE: secret-key-base
    ports:
      - 4000:4000
    depends_on:
      db:
        condition: service_healthy
  db:
    image: pgvector/pgvector:pg16
    healthcheck:
      test: [ "CMD-SHELL", "pg_isready -U db-user -d ai_assistant" ]
      interval: 3s
      timeout: 3s
      retries: 10
    environment:
      POSTGRES_USER: db-user
      POSTGRES_PASSWORD: password
      POSTGRES_DB: ai_assistant
      POSTGRES_INITDB_ARGS: --data-checksums
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

Starting AI Assistant

Now, open a terminal emulator.

Use the terminal emulator integrated with your code editor or IDE. Alternatively, you can use Terminal.app or iTerm2(opens in a new tab).

Go to the directory where you saved the docker-compose.yml file:

cd <path-to-directory-with-docker-compose-yml>

Run the following:

docker-compose up

This command might take a while to run, depending on your internet connection speed. Wait until you see the following message in the terminal:

ai_document_assistant  | info: AI Assistant started

AI Assistant is now up and running!

Setting up AI Assistant on Android

To attach AI Assistant directly to a document, follow the steps below.

Add the following dependencies to your Gradle file, as AI Assistant depends on them:
```
implementation("io.noties.markwon:core:4.6.2")
implementation("io.noties.markwon:html:4.6.2")
implementation("io.noties.markwon:linkify:4.6.2")
implementation("io.noties.markwon:ext-tables:4.6.2")
implementation("io.noties.markwon:ext-strikethrough:4.6.2")
implementation("io.socket:socket.io-client:2.1.1")
```
Create an instance of AI Assistant using the createAiAssistant method. This method requires:
- context — The Android context
- documentsDescriptors — List of document descriptors created from data providers (supports multiple documents)
- ipAddress — The AI Assistant server IP address
- sessionId — A unique session identifier
- jwtToken — A Lambda function that generates JSON Web Tokens (JWTs) with document IDs
```
val assetFiles = listOf("document1.pdf", "document2.pdf", "document3.pdf")
val documentDescriptors = assetFiles.map {
    DocumentDescriptor.fromDataProviders(listOf(AssetDataProvider(it)), listOf(), listOf())
}

val assistant = createAiAssistant(
    context = this,
    documentsDescriptors = documentDescriptors,
    ipAddress = "your-server-ip",
    sessionId = "your-session-id",
    jwtToken = { documentIds ->
        generateJwtToken(
            context = this,
            claims = mapOf(
                "document_ids" to documentIds,
                "session_ids" to listOf("your-session-id"),
                "request_limit" to mapOf(
                    "requests" to 160,
                    "time_period_s" to 600000 // 10 minutes
                )
            )
        )
    }
)
```
- It’s best practice for the JWT to be generated by your own server. For more details, refer to the guide on generating a JWT.
Implement the AiAssistantProvider interface in your Activity to provide the AI Assistant instance:
```
class YourActivity : AppCompatActivity(), AiAssistantProvider {
    override fun getAiAssistant(): AiAssistant {
        return assistant
    }

    // Optional: Implement navigation callback for multi-document support.
    override fun navigateTo(
        documentRect: List<RectF>,
        pageIndex: Int,
        documentIndex: Int
    ) {
        // Handle navigation to specific document, page, and highlight rectangles.
        // Example: Switch to the document at `documentIndex` and highlight the specified area.
    }
}
```
To display the AI Assistant icon in your toolbar, enable it with setAiAssistantEnabled in PdfActivityConfiguration.

Now you can access AI Assistant in your Activity by clicking the AI Assistant icon in the Nutrient MainToolbar.

Multi-document support

AI Assistant supports analyzing multiple documents simultaneously. When you provide multiple DocumentDescriptor objects, users can ask questions that span across all documents, compare content between documents, and navigate between them.

When AI Assistant references content from specific documents, it can automatically navigate to the relevant location using the navigateTo callback in the AiAssistantProvider interface. This callback provides:

documentRect — List of rectangles to highlight on the page
pageIndex — The specific page number within the document
documentIndex — The index of the document in the list you provided

This enables AI Assistant to guide users directly to relevant content across your document collection.

Alternative AI Assistant display methods

In addition to the built-in toolbar icon, you can show the AI Assistant dialog programmatically:

import com.pspdfkit.ai.showAiAssistant

// Show AI Assistant dialog.
showAiAssistant(context)

For interactive examples of AI Assistant with multiple documents, check out the AiAssistant examples(opens in a new tab) in the Catalog app, including the ViewPager and Compose multi-document examples.

Advanced usage

For advanced control over AI Assistant, you can use the lower-level API methods. The responseState flow enables observing responses, and various methods allow fine-grained control over initialization and messaging.

AI Assistant initialization

The initialize method provides a convenient way to set up AI Assistant:

// Initialize with session history (default).
aiAssistant.initialize(withSessionHistory = true)

// Initialize without session history for faster startup.
aiAssistant.initialize(withSessionHistory = false)

The initialize method automatically:

Checks if documents are already ingested
Ingests documents if necessary
Establishes a socket connection
Retrieves the session history (if enabled)

Manual initialization steps

For more granular control, you can perform initialization steps manually.

Check if documents are already ingested:
```
val result = aiAssistant.checkIfDocumentIsAlreadyIngested(documentId, fileHash)
```
Ingest documents:
```
val ingestionResponse = aiAssistant.ingestDocument(dataProvider, jwtToken)
```
Initialize socket connection:
```
// Include session history in connection.
aiAssistant.initializeSocketConnection(includeSessionHistory = true)

// Connect without session history for faster startup.
aiAssistant.initializeSocketConnection(includeSessionHistory = false)
```
Retrieve session history separately:
```
val history = aiAssistant.getSessionHistory()
```

Advanced messaging capabilities

AI Assistant provides several methods for different types of interactions.

Standard messaging:

aiAssistant.emitMessage("Your query here")

Context-specific messaging with selected text:

aiAssistant.emitContextSpecificMessage(
    message = "Explain this section",
    contextText = "Selected text from the document"
)

Emit selected text for processing:

aiAssistant.emitSelectedText("Text selected by the user")

Observing responses

After initialization, observe responses using the responseState flow and update your UI accordingly:

val responseState: Flow<CompletionResponse?> = aiAssistant.responseState

Terminating AI Assistant

To stop AI Assistant, call the terminate method:

aiAssistant.terminate()