---
title: "Multimodal RAG: How to retrieve over text, tables, and images in documents"
canonical_url: "https://www.nutrient.io/blog/multimodal-rag/"
md_url: "https://www.nutrient.io/blog/multimodal-rag.md"
last_updated: "2026-07-08T14:42:17.197Z"
description: "Multimodal RAG retrieves over text, tables, and images, not plain text alone. How it works, why parsing makes or breaks it, and how to extract each from PDFs."
---

**TL;DR**

- Multimodal RAG retrieves over text, tables, and images together, instead of treating a document as one stream of plain text.

- Most documents are already multimodal: A single PDF page can hold prose, a financial table, a chart, and a scanned signature.

- Parsing is the hard part. A multimodal RAG system is only as good as the text, tables, and figures you extract from the source documents.

- This guide covers the architecture and how to extract each modality from PDFs so your pipeline retrieves clean, structured input.

Retrieval-augmented generation (RAG) grounds a language model’s answers in your own documents: You retrieve the passages most relevant to a question and pass them to the model as context.

Standard RAG treats every source as plain text. That works until the answer lives in a table, a chart, or a scanned page. In real documents, it usually does.

Multimodal RAG handles those cases. It retrieves across multiple modalities (text, tables, and images) rather than flattening everything into a single text stream. This article explains how it works, why document parsing is the part that determines quality, and how to extract each modality from PDFs to feed the pipeline.

## What is multimodal RAG?

Multimodal RAG is a RAG architecture that indexes and retrieves more than one content type. Where standard RAG embeds and retrieves text chunks, a multimodal system also represents tables and images so they can be retrieved and reasoned over.

|                                  | Standard RAG              | Multimodal RAG                   |
| -------------------------------- | ------------------------- | -------------------------------- |
| Indexes                          | Text chunks only          | Text, tables, and images         |
| Tables                           | Flattened into plain text | Serialized with structure intact |
| Figures and charts               | Ignored                   | Summarized or embedded           |
| Scanned pages                    | Returned empty            | OCR/ICR before indexing          |
| Answer quality on real documents | Degrades fast             | Holds up                         |

Much of the information in real documents isn’t prose. A quarterly report’s numbers live in tables. A specification’s meaning lives in a diagram. A signed contract’s proof lives in a scanned image. Flatten those into plain text, or drop them entirely, and the model never sees the answer.

## Why documents are inherently multimodal

A single page of a real-world PDF often contains several modalities at once:

- **Body text** — Paragraphs, headings, lists

- **Tables** — Financial figures, specifications, comparison grids

- **Figures** — Charts, diagrams, screenshots, photographs

- **Scanned content** — Pages that are images of text, with no extractable text layer

Standard text-only RAG mishandles all but the first. Tables get linearized into unreadable runs of numbers, figures are ignored, and scanned pages return nothing at all.

The retrieval step can only surface what the parsing step managed to extract, so the quality ceiling of the whole system is set before any embedding or retrieval happens.

This is the most common reason a RAG demo works on clean text and then fails on the customer’s actual documents is because the pipeline never extracted the table or the scanned page where the answer lived.

## How multimodal RAG works

A multimodal RAG pipeline has the same shape as standard RAG, with the ingestion and retrieval stages widened to handle each modality.![Multimodal RAG pipeline: A document is parsed and extracted into text, tables, and images. Text keeps its reading order, tables are serialized to Markdown, and figures are extracted as assets. Text and table text are embedded directly, while images are summarized by a VLM and then embedded. All three land in one vector index, which is retrieved across modalities to generate an answer.](@/assets/images/blog/2026/multimodal-rag/multimodal-rag-pipeline.svg)

1. **Parse** the document into separate elements: text blocks, tables, and images.

2. **Represent** each element. Text and tables are typically embedded as text (tables serialized to Markdown or HTML so structure survives). Images are either embedded with a multimodal embedding model or first summarized into text by a vision language model (VLM), and then embedded.

3. **Index** the representations in a vector store, keeping a reference back to the source element.

4. **Retrieve** the elements most relevant to the query across all modalities.

5. **Generate** an answer, passing the retrieved text, serialized tables, and image summaries (or the images themselves) to the model.

There are three common approaches to representing non-text content:

- **Joint multimodal embeddings** — A multimodal model embeds images and text into a shared vector space, so a figure or a scanned page is retrieved directly as an image, with no lossy text conversion in between. This is the closest representation to true multimodal RAG, and the models have matured fast: CLIP established the pattern, and newer models such as [Google’s Gemini embeddings](https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-embedding-2/) push multimodal quality much further. The cost is extra infrastructure and a model that can embed both modalities.

- **Summarize-then-embed** — A VLM writes a text description of each image or table; you embed the description. It works with any text embedding model and keeps everything in one text index, which makes it the simplest to run and a solid fallback when a joint embedding model isn’t practical, at the cost of whatever detail the summary drops.

- **Separate indexes with fusion** — Each modality is embedded and retrieved with its own model, and the results are merged at query time, which is the most flexible setup but the one with the most moving parts.

Embedding images directly is the stronger representation and increasingly the default; summarize-then-embed remains a valid fallback when you need to keep everything in a single text index.

## The part that breaks: Extracting modalities from PDFs

Everything above assumes step 1 produced clean, structured elements. With PDFs, that assumption is where pipelines fail.

PDFs encode layout, not meaning: Text can arrive out of reading order, a table is often just absolutely positioned text with no row or column structure, and a scanned page has no text layer at all.

So the real engineering work in multimodal RAG is extraction:

- **Text** must come out in correct reading order, with multicolumn layouts handled.

- **Tables** must be detected and serialized with their structure intact, not flattened into a stream of cells.

- **Images and figures** must be located and pulled out as discrete assets.

- **Scanned pages** must be run through [optical character recognition (OCR)](https://www.nutrient.io/blog/document-ai-vs-ocr.md), and handwriting through [intelligent character recognition (ICR) or a VLM](https://www.nutrient.io/blog/handwriting-recognition-ocr-vlm/), before any of the above is even possible.

That extraction layer is exactly what a document processing toolset handles.

Nutrient’s [AI document and data extraction SDK](https://www.nutrient.io/sdk/ai-document-processing/) is built for it: OCR, ICR, and VLM-enhanced extraction handle scanned and handwritten content, detect and extract tables with cell-level coordinates, and pull key-value pairs from forms and invoices without templates. Output is structured JSON with reading order and confidence scores, and every engine, including the VLM, can run on your own infrastructure. That turns a messy PDF into the structured text, tables, and figures a multimodal RAG pipeline can index.

If your documents include scans or handwriting, OCR/ICR quality sets the ceiling for the entire system. See [why your AI agent hallucinates PDF table data](https://www.nutrient.io/blog/why-your-ai-agent-hallucinates-pdf-table-data.md) for how parsing errors propagate into wrong answers downstream.

### Serializing a table so retrieval works

A table is only useful to the model if [its structure survives extraction](https://www.nutrient.io/sdk/python/pdf-data-extraction/). Linearized into plain text, the relationship between a row label and its value is lost. Serialized to Markdown, it stays intact:

```text

| Quarter | Revenue | Growth |
| ------- | ------- | ------ |
| Q1 2026 | $4.2M   | 12%    |
| Q2 2026 | $4.9M   | 17%    |

```

Embedding this representation, rather than a flat run of numbers, means a query like “Q2 revenue growth” retrieves a chunk where the structure makes the answer recoverable.

### Representing extracted elements in code

Once the document is parsed into elements, representing them is mechanical: Text and tables embed as-is, images get a short [VLM summary](https://www.nutrient.io/guides/python/extraction/describe-image-with-local-ai.md), and everything lands in one vector store. With LangChain:

```python

from langchain.chat_models import init_chat_model
from langchain_openai import OpenAIEmbeddings
from langchain_core.documents import Document
from langchain_core.vectorstores import InMemoryVectorStore

# `elements` come from your document parser — each has a type and content.

# Text and tables are already text; images are base64-encoded.

vlm = init_chat_model("gpt-4o-mini", model_provider="openai")
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = InMemoryVectorStore(embeddings)

docs = []
for element in elements:
    if element["type"] in ("text", "table"):
        docs.append(
            Document(page_content=element["content"], metadata={"type": element["type"]})
        )
    elif element["type"] == "image":
        summary = vlm.invoke(
            [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Describe this figure for retrieval."},
                        {"type": "image", "base64": element["content"], "mime_type": "image/png"},
                    ],
                }
            ]
        ).content
        docs.append(Document(page_content=summary, metadata={"type": "image"}))

vector_store.add_documents(docs)
results = vector_store.similarity_search("Q2 revenue growth", k=4)

```

Tables come in as Markdown so structure survives, images become searchable through their summaries, and one query retrieves across all three modalities. The quality of `elements`, whether the parser actually recovered that table and ran OCR on that scan, decides everything downstream.

## Evaluating a multimodal RAG system

Standard retrieval metrics still apply, but multimodal systems need extra checks tied to the modality split:

- **Per-modality retrieval** — Does the system surface tables and images for queries whose answers live there, or does it default to prose?

- **Extraction fidelity** — Sample parsed tables and OCR output against the source. Errors here are invisible at retrieval time but fatal at answer time.

- **Grounding** — Does the generated answer cite the right element, including non-text ones?

Measure extraction quality first. A retrieval bug is debuggable; a silent parsing error looks like a model hallucination and sends you chasing the wrong problem.

## Common pitfalls

- **Treating every PDF as text.** Scanned and image-heavy documents need OCR and figure extraction before indexing.

- **Flattening tables.** Serialize to Markdown or HTML so structure survives the embedding step.

- **Ignoring reading order.** Multicolumn and complex layouts produce scrambled text unless layout analysis runs first.

- **Skipping evaluation of the parse.** Most “the model is wrong” reports trace back to “the document was never extracted correctly.”

## FAQ

#### What is multimodal RAG?

Multimodal RAG is a retrieval-augmented generation architecture that indexes and retrieves more than one content type. Alongside text chunks, it represents tables and images so they can be retrieved and reasoned over, instead of flattening a document into a single text stream.

#### How is multimodal RAG different from standard RAG?

Standard RAG embeds and retrieves text only, so tables get flattened and figures and scanned pages are dropped. Multimodal RAG keeps tables as structured text, summarizes or embeds images, and runs OCR on scans, so the answer survives wherever it lives in the document.

#### Why does multimodal RAG give wrong answers on real documents?

It's almost always because extraction failed first. Retrieval can only surface what parsing pulled out, so a flattened table or a skipped scan means the answer was never indexed. Measure extraction quality before tuning embeddings or retrieval.

#### How do you represent images and tables for retrieval?

There are three common approaches: Use a joint multimodal embedding model (such as CLIP or Google’s Gemini embeddings) to embed images directly, summarize each image or table with a VLM and embed the text, or keep a separate index per modality and merge results at query time. Embedding images directly is the closest to true multimodal RAG; summarize-then-embed is a valid fallback when you need everything in a single text index.

#### Do I need a multimodal embedding model?

No. The summarize-then-embed approach turns images and tables into text descriptions, so a standard text embedding model and one vector index are enough. A multimodal model adds capability but also infrastructure.

## Conclusion

Multimodal RAG extends retrieval to text, tables, and images: what real documents contain. The retrieval and indexing patterns are well understood; the variable that decides quality is how cleanly you pull each modality out of the source PDFs. A pipeline fed scrambled tables and skipped scans retrieves noise, no matter how good the embeddings are.

If you’re building one, start by getting structured text, tables, and figures out of your documents reliably. Nutrient’s [AI document and data extraction SDK](https://www.nutrient.io/sdk/ai-document-processing/) is built for that ingestion layer. [Contact our team](https://www.nutrient.io/contact-sales/) to talk through your document workflow, or explore the [Data Extraction API guides](https://www.nutrient.io/guides/dws-data-extraction.md) to get started.

**Call to Action**

Turn messy PDFs into clean text, tables, and figures for your RAG pipeline with Nutrient

[Learn More](https://www.nutrient.io/sdk/ai-document-processing/)

## Related reading

- [Why your AI agent hallucinates PDF table data](https://www.nutrient.io/blog/why-your-ai-agent-hallucinates-pdf-table-data.md) — How parsing errors become wrong answers downstream

- [Document AI vs. traditional OCR](https://www.nutrient.io/blog/document-ai-vs-ocr.md) — Choosing between OCR, AI, and hybrid extraction pipelines

- [Handwriting OCR vs. VLMs](https://www.nutrient.io/blog/handwriting-recognition-ocr-vlm/) — Extracting handwritten content before indexing

- [Data Extraction API guides](https://www.nutrient.io/guides/dws-data-extraction.md) — Convert documents to RAG-ready Markdown
---

## Related pages

- [Accessibility Untangled Why It Matters Guide](/blog/accessibility-untangled-why-it-matters-guide.md)
- [The business case for accessibility: Five ways it drives enterprise value](/blog/5-ways-accessibility-drives-enterprise-value.md)
- [Advanced Techniques For React Native Ui Components](/blog/advanced-techniques-for-react-native-ui-components.md)
- [Auto Tagging And Document Accessibility In Dotnet Sdk](/blog/auto-tagging-and-document-accessibility-in-dotnet-sdk.md)
- [Ai Document Automation Extraction To Action](/blog/ai-document-automation-extraction-to-action.md)
- [Best Document Viewers](/blog/best-document-viewers.md)
- [The CEO’s AI playbook: Why decision architecture beats model selection](/blog/ceo-ai-playbook-decision-architecture.md)
- [Convert One Drive Files To Pdf In Sharepoint](/blog/convert-one-drive-files-to-pdf-in-sharepoint.md)
- [Complete Guide To Pdfjs](/blog/complete-guide-to-pdfjs.md)
- [Construction Document Data Extraction](/blog/construction-document-data-extraction.md)
- [Create And Edit Pdfs In Flutter](/blog/create-and-edit-pdfs-in-flutter.md)
- [Create Pdfs With React](/blog/create-pdfs-with-react.md)
- [Creating A Document Scanner With Ocr In Python](/blog/creating-a-document-scanner-with-ocr-in-python.md)
- [The CTO’s AI playbook: Why accountability architecture beats orchestration](/blog/cto-ai-playbook-accountability-architecture.md)
- [Digital Workflow Automation](/blog/digital-workflow-automation.md)
- [Digital Signatures](/blog/digital-signatures.md)
- [Document Ai Vs Ocr](/blog/document-ai-vs-ocr.md)
- [How To Build A Dotnet Maui Pdf Viewer](/blog/how-to-build-a-dotnet-maui-pdf-viewer.md)
- [Document Viewer](/blog/document-viewer.md)
- [How To Build A Flutter Pdf Viewer](/blog/how-to-build-a-flutter-pdf-viewer.md)
- [Emerging threats: Your logging system may be an agentic threat vector](/blog/emerging-threats-your-logging-system.md)
- [or](/blog/how-to-build-a-javascript-pdf-viewer-with-pdfjs.md)
- [How To Build A Powerpoint Viewer Using Javascript](/blog/how-to-build-a-powerpoint-viewer-using-javascript.md)
- [or](/blog/how-to-build-a-nextjs-pdf-viewer.md)
- [How To Build A React Powerpoint Viewer](/blog/how-to-build-a-react-powerpoint-viewer.md)
- [How To Build A React Native Pdf Viewer](/blog/how-to-build-a-react-native-pdf-viewer.md)
- [or](/blog/how-to-build-a-reactjs-pdf-viewer.md)
- [How To Build A Reactjs File Viewer](/blog/how-to-build-a-reactjs-file-viewer.md)
- [or](/blog/how-to-build-a-reactjs-pdf-viewer-with-react-pdf.md)
- [How To Build A Reactjs Viewer With Pdfjs](/blog/how-to-build-a-reactjs-viewer-with-pdfjs.md)
- [How To Build An Android Pdf Viewer](/blog/how-to-build-an-android-pdf-viewer.md)
- [or](/blog/how-to-convert-html-to-pdf-using-wkhtmltopdf-and-python.md)
- [How To Convert Word To Pdf In Nodejs](/blog/how-to-convert-word-to-pdf-in-nodejs.md)
- [How To Create Pdfs With React To Pdf](/blog/how-to-create-pdfs-with-react-to-pdf.md)
- [or](/blog/how-to-convert-html-to-pdf-using-react.md)
- [How To Embed A Pdf Viewer In Your Website](/blog/how-to-embed-a-pdf-viewer-in-your-website.md)
- [How To Convert Html To Pdf Using Html2pdf](/blog/how-to-convert-html-to-pdf-using-html2pdf.md)
- [How To Edit Pdfs Using Ios Pdf Library](/blog/how-to-edit-pdfs-using-ios-pdf-library.md)
- [How To Generate Pdf From Html With Nodejs](/blog/how-to-generate-pdf-from-html-with-nodejs.md)
- [base_url tells WeasyPrint where to resolve relative asset paths](/blog/how-to-generate-pdf-reports-from-html-in-python.md)
- [From an HTML string.](/blog/html-in-pdf-format.md)
- [Swift Package Manager](/blog/mobile-pdf-sdk.md)
- [Pdf Page Labels](/blog/pdf-page-labels.md)
- [Linearized Pdf](/blog/linearized-pdf.md)
- [Pdf Ua Compliance Guide](/blog/pdf-ua-compliance-guide.md)
- [Open Pdf In Your Web App](/blog/open-pdf-in-your-web-app.md)
- [Online Document Viewer](/blog/online-document-viewer.md)
- [Nutrient Vs Conga Composer](/blog/nutrient-vs-conga-composer.md)
- [Pdfjs Area Annotations Canvas Capture](/blog/pdfjs-area-annotations-canvas-capture.md)
- [Pdf Sdk Performance Benchmark](/blog/pdf-sdk-performance-benchmark.md)
- [Pdfjs Eventbus Guide](/blog/pdfjs-eventbus-guide.md)
- [Pdfjs Coordinate Systems Pdf To Screen](/blog/pdfjs-coordinate-systems-pdf-to-screen.md)
- [Pdf Sdk Compliance Security Checklist](/blog/pdf-sdk-compliance-security-checklist.md)
- [Pdfjs Navigation Zoom Rotation](/blog/pdfjs-navigation-zoom-rotation.md)
- [Pdfjs Annotation Editor Layer](/blog/pdfjs-annotation-editor-layer.md)
- [Pdfjs Limitations Commercial Upgrade](/blog/pdfjs-limitations-commercial-upgrade.md)
- [Pdfjs Native Annotation Layer Forms](/blog/pdfjs-native-annotation-layer-forms.md)
- [Pdfjs Rendering Overlays React Portals](/blog/pdfjs-rendering-overlays-react-portals.md)
- [Pdfjs React Viewer Setup](/blog/pdfjs-react-viewer-setup.md)
- [Pdfjs Server Side Text Extraction](/blog/pdfjs-server-side-text-extraction.md)
- [Pdfjs Pdf Page Manipulation Pdf Lib](/blog/pdfjs-pdf-page-manipulation-pdf-lib.md)
- [Pdfjs Sticky Note Annotations](/blog/pdfjs-sticky-note-annotations.md)
- [Pdfjs Thumbnail Sidebar](/blog/pdfjs-thumbnail-sidebar.md)
- [Pdfjs Text Search Pdffindcontroller](/blog/pdfjs-text-search-pdffindcontroller.md)
- [Pdfjs Text Highlight Annotations](/blog/pdfjs-text-highlight-annotations.md)
- [Process Flows](/blog/process-flows.md)
- [React Native Pdf Annotation](/blog/react-native-pdf-annotation.md)
- [Vector Pdf](/blog/vector-pdf.md)
- [or](/blog/sample-blog-updated.md)
- [Wcag2 Accessibility Requirements Documents](/blog/wcag2-accessibility-requirements-documents.md)
- [Web Sdk Is Now Headless](/blog/web-sdk-is-now-headless.md)
- [What Are Annotations](/blog/what-are-annotations.md)
- [Convert an HTML file to PDF.](/blog/top-ten-ways-to-convert-html-to-pdf.md)
- [What Is A Vpat](/blog/what-is-a-vpat.md)
- [Why Pdfium Is A Trusted Platform For Pdf Rendering](/blog/why-pdfium-is-a-trusted-platform-for-pdf-rendering.md)
- [Why Your Ai Agent Hallucinates Pdf Table Data](/blog/why-your-ai-agent-hallucinates-pdf-table-data.md)
- [What Is Pdf Ua](/blog/what-is-pdf-ua.md)