Vercel AI SDK

This page shows how to register tools from the tool reference with Vercel AI SDK and execute them against a Nutrient Web SDK instance.

Registering tools

Tools defined without an execute function are forwarded to the client. Define each tool with a Zod schema matching the parameters from the tool reference.

The example below registers two tools: one for reading annotations, and one for creating them. Add tools for any SDK method by following the same pattern:

import { streamText, tool } from "ai";
import { openai } from "@ai-sdk/openai";
import { z } from "zod";


const result = streamText({
  model: openai("gpt-4o"),
  messages,
  tools: {
    get_annotations: tool({
      description:
        "Get all annotations on a given page. Returns " +
        "type, content, bounding box, and ID.",
      parameters: z.object({
        pageIndex: z
          .number()
          .describe("Zero-based page index"),
      }),
    }),
    create_annotation: tool({
      description:
        "Add a text annotation to a page. Call " +
        "get_page_info first to understand dimensions.",
      parameters: z.object({
        pageIndex: z.number(),
        text: z.string(),
        left: z.number(),
        top: z.number(),
        width: z.number(),
        height: z.number(),
      }),
    }),
  },
});

Executing tools

When the model calls a tool, the onToolCall callback receives the tool name and arguments. Each case runs the SDK method and returns the result to the model:

import NutrientViewer from "@nutrient-sdk/viewer";
import { useChat } from "@ai-sdk/react";


const { messages } = useChat({
  maxSteps: 10,
  async onToolCall({ toolCall }) {
    const instance = nutrientInstance;


    switch (toolCall.toolName) {
      case "get_annotations":
        return (
          await instance.getAnnotations(
            toolCall.args.pageIndex,
          )
        ).toJS();


      case "create_annotation": {
        const a = toolCall.args;
        const annotation =
          new NutrientViewer.Annotations.TextAnnotation({
            pageIndex: a.pageIndex,
            text: { format: "plain", value: a.text },
            boundingBox:
              new NutrientViewer.Geometry.Rect({
                left: a.left,
                top: a.top,
                width: a.width,
                height: a.height,
              }),
          });
        const [created] =
          await instance.create(annotation);
        return { id: created.id };
      }
    }
  },
});

Add a tool() definition on the server and a matching case on the client for each tool from the tool reference that your agent needs.