How to chat with your PDF using RAG

Chat-with-PDF tools let you upload a document and ask it questions like “what’s the penalty clause?”, “summarize section 4”, or “what was Q2 revenue?”, instead of scrolling through pages. Almost all of them work the same way: retrieval-augmented generation over the document’s text.

This article explains how that pipeline works, walks through building one, and covers the part that decides answer quality: getting clean content out of the PDF in the first place.

How chat with PDF works

A chat-with-PDF app is a RAG pipeline scoped to one document (or a small set). The flow is outlined below.

1. Extract the text (and tables, and any text from scanned pages) from the PDF.
2. Chunk the content into passages small enough to embed and retrieve.
3. Embed each chunk into a vector and store it.
4. Retrieve the chunks most relevant to the user’s question.
5. Generate an answer with an LLM, passing the retrieved chunks as context.

The model never “reads” the whole PDF on each question. It answers from the handful of chunks retrieval pulled, which is why both chunking and extraction matter as much as the model.

Building a chat-with-PDF app

Here’s a minimal version with LangChain(opens in a new tab): Load a PDF, index it, and answer questions with an agent that retrieves on demand.

You’ll need:

• Python 3.10+
• An OpenAI API key set as OPENAI_API_KEY
• The required packages — pip install langchain langchain-community langchain-openai langchain-text-splitters pypdf

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_core.vectorstores import InMemoryVectorStore
from langchain.chat_models import init_chat_model
from langchain.agents import create_agent
from langchain.tools import tool

# 1. Extract and chunk the PDF.
pages = PyPDFLoader("contract.pdf").load()
splits = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=200
).split_documents(pages)

# 2. Embed and store the chunks.
vector_store = InMemoryVectorStore(OpenAIEmbeddings(model="text-embedding-3-small"))
vector_store.add_documents(splits)

# 3. Give an agent a retrieval tool.
@tool(response_format="content_and_artifact")
def search_pdf(query: str):
    """Retrieve passages from the PDF relevant to the query."""
    docs = vector_store.similarity_search(query, k=4)
    return "\n\n".join(d.page_content for d in docs), docs

model = init_chat_model("gpt-4o-mini", model_provider="openai")
agent = create_agent(
    model,
    tools=[search_pdf],
    system_prompt=(
        "Answer questions about the PDF using the search tool. "
        "If the retrieved passages don't contain the answer, say you don't know."
    ),
)

# 4. Ask.
result = agent.invoke(
    {"messages": [{"role": "user", "content": "What is the penalty clause?"}]}
)
print(result["messages"][-1].content)

That’s a working chat-with-PDF backend for a clean, born-digital PDF. Wrap it in a chat UI and you have the app.

The part that decides accuracy: Extraction

Step 1, PyPDFLoader, is where this breaks on real documents. A loader that pulls the raw text layer works for a born-digital report. It fails on the documents people actually want to chat with:

• Scanned PDFs have no text layer, so the loader returns empty pages. Without optical character recognition (OCR), there’s nothing to index.
• Tables get flattened into a run of numbers, so “what was Q2 revenue?” retrieves a chunk where the figure has lost its label.
• Multicolumn and complex layouts come out in the wrong reading order, so chunks mix unrelated text.

When that happens, the model isn’t wrong; it’s answering faithfully from broken input. The fix is to replace naive text loading with real extraction: OCR for scans, table detection that preserves structure, and layout-aware reading order.

Swap the loader for an extraction step that handles those cases. Nutrient’s Data Extraction API returns clean Markdown — with OCR, tables, and reading order already resolved — which you chunk and embed exactly as before:

import requests
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Replace PyPDFLoader with extraction that runs OCR, detects tables,
# and preserves reading order, returning Markdown.
response = requests.post(
    "https://api.nutrient.io/extraction/parse",
    headers={"Authorization": "Bearer your_api_key_goes_here"},
    files={"file": open("contract.pdf", "rb")},
    data={"instructions": '{"mode":"understand","output":{"format":"markdown"}}'},
)
markdown = response.json()["output"]["markdown"]

# Chunk the extracted Markdown, then embed and index as before.
splits = RecursiveCharacterTextSplitter(
    chunk_size=1000, chunk_overlap=200
).create_documents([markdown])

Everything downstream stays identical. The chunks are just built from content that survived extraction, so the table figure and the scanned clause are actually in the index when the agent goes looking.

Skipping the plumbing with a managed AI Assistant

Build the pipeline yourself when you need control over chunking, the model, and the vector store. If you’d rather not run that stack, a managed document AI handles the retrieval, model orchestration, and UI for you.

Nutrient AI Assistant does this. It runs as a backend service your document viewer connects to, works with major LLM providers (OpenAI, Azure OpenAI, AWS Bedrock, and Anthropic), and is available for web, iOS, Android, and Flutter. Through it, users can:

• Ask questions and get context-aware answers from the document.
• Summarize long documents or specific pages.
• Translate content into other languages.
• Redact sensitive information through natural-language commands.
• Compare versions and categorize the differences.

Pair it with the Web SDK so the chat sits next to a real document viewer. The getting started guides and an open source demo(opens in a new tab) cover setup.

Other no-build options

• Nutrient PDF Editor for Claude Cowork lets you chat with and edit PDFs inside Claude, with no signup.
• Document Engine MCP Server lets AI agents like Claude process documents through natural-language commands.

FAQ

Conclusion

Chatting with a PDF is RAG pointed at a single document: extract, chunk, embed, retrieve, answer. The pipeline is a few lines of code, and the model does the visible work, but the answers are only as good as what extraction pulled out of the PDF. A flattened table or a skipped scan produces confident, wrong answers, no matter which model you use.

Get extraction right with the AI document and data extraction SDK, or skip the stack entirely with the managed AI Assistant. Contact our team to talk through your use case.