--- title: "How to extract data from patient documents on-premises with local AI" canonical_url: "https://www.nutrient.io/blog/extract-patient-data-on-premises/" md_url: "https://www.nutrient.io/blog/extract-patient-data-on-premises.md" last_updated: "2026-08-12T15:45:44.233Z" description: "A step-by-step tutorial for extracting structured data from patient documents on-premises — self-hosted Document Engine plus local ICR, no PHI leaving the network." --- **TL;DR** Patient documents contain protected health information (PHI) that often can’t leave the network of the covered entity. This tutorial builds a document AI pipeline that runs entirely on-premises: a self-hosted Document Engine for storage and processing, plus local intelligent content recognition (ICR) that extracts structured data without any external API call. Cloud enhancement stays opt-in. Healthcare teams process a constant stream of scanned intake forms, lab reports, and referral letters. Extracting structured data from those documents is a task AI handles well, but most document AI services require uploading the file to a vendor-hosted model. For protected health information governed by the Health Insurance Portability and Accountability Act ([HIPAA](https://www.hhs.gov/hipaa/index.html)), that upload is often not an option. This tutorial covers the alternative: an on-premises pipeline where the model runs next to the data, so a patient record is understood without ever leaving the covered entity’s network — the same principle behind [automated PII removal](https://www.nutrient.io/blog/automated-pii-removal/) that keeps sensitive fields in-house. ## What “on-premises” means for patient documents A private pipeline has two parts, and both must stay inside the network for the guarantee to hold: 1. **Self-hosted Document Engine** — Handles storage, rendering, and orchestration inside the owner’s infrastructure rather than a managed cloud. 2. **Local ICR extraction** — Analyzes layout, tables, and handwriting using AI models that run on local hardware, returning structured JSON with no external call. When both run on-premises, a scanned intake form is processed end to end without a network hop to a third party.![On-premises document AI pipeline: A patient document flows through a self-hosted Document Engine and local ICR to structured JSON entirely inside the network boundary, with an optional cloud VLM shown as an opt-in branch that leaves the network](@/assets/images/blog/2026/extract-patient-data-on-premises/on-prem-architecture.png) ## Prerequisites - A [Kubernetes](https://kubernetes.io/) cluster or [Docker](https://www.docker.com/) host inside the network where PHI is permitted - Python 3.8 or higher - Nutrient Python SDK (`pip install nutrient-sdk`) - A sample patient document (PNG, JPEG, or TIFF) for testing ## Step 1 — Deploy a self-hosted Document Engine [Document Engine](https://www.nutrient.io/sdk/document-engine/) is self-hosted, so storage and processing stay on local infrastructure. Deploy it with the method that matches the environment: - [Deploy with Kubernetes](https://www.nutrient.io/guides/document-engine/deployment/kubernetes.md) for a clustered, production setup - [Deploy with Helm](https://www.nutrient.io/guides/document-engine/deployment/helm.md) for a chart-based install Follow the [deployment overview](https://www.nutrient.io/guides/document-engine/deployment.md) for configuration, and confirm the instance is reachable only from inside the network before processing live documents. ## Step 2 — Configure local ICR extraction With the engine in place, extraction runs through the Nutrient Vision API. ICR is the default engine and runs locally, so the following configuration keeps every document on local hardware (see the [local ICR extraction guide](https://www.nutrient.io/guides/python/extraction/extract-data-from-image-icr.md) for the full reference): ```python from nutrient_sdk import Document, Vision, VisionEngine with Document.open("patient-intake-form.png") as document: # ICR runs locally and is the default engine — no external calls. document.settings.vision_settings.engine = VisionEngine.ICR vision = Vision.set(document) content_json = vision.extract_content() # structured JSON, all processing stays local. ``` ## Step 3 — Extract structured data from an intake form `extract_content()` returns JSON with layout and semantic structure — paragraphs, tables, and key-value regions — plus a bounding box for every element. Write it to disk and add error handling for production use: ```python from nutrient_sdk import Document, Vision, VisionEngine, VisionException try: with Document.open("patient-intake-form.png") as document: document.settings.vision_settings.engine = VisionEngine.ICR vision = Vision.set(document) content_json = vision.extract_content() with open("intake-form.json", "w") as f: f.write(content_json) except VisionException as error: # Log and handle extraction failures (unreadable file, missing models, etc.) print(f"Extraction failed: {error}") ``` The bounding box on each extracted value makes the output auditable: Any field on the parsed form traces back to the exact region of the source page, which supports review interfaces and record-keeping. For a deeper walkthrough of the extraction output across engines, see [how to build a document extraction pipeline](https://www.nutrient.io/blog/build-document-extraction-pipeline-nutrient-vision-api/). ## Keeping the pipeline compliant On-premises processing addresses a key requirement — PHI stays inside the covered entity’s environment — but a compliant deployment depends on the surrounding controls: - **Network isolation** — Restrict the Document Engine instance to internal traffic so no document is reachable externally. - **Traceability** — Retain the element-level coordinates from extraction for audit and review. - **Access control** — Apply the organization’s existing authentication and authorization to the pipeline. Running extraction on-premises keeps PHI within the covered entity’s control, which is what HIPAA-regulated workloads typically require. The overall compliance posture still depends on how the full system is configured and operated. ## When cloud enhancement is acceptable Local ICR handles most real-world layouts. For unusually complex documents, an opt-in tier enhanced by a vision language model (VLM) can send *layout data* — not the raw document — to a cloud provider for higher accuracy, and the pipeline owner controls whether it activates. The tradeoffs are compared in [OCR vs. intelligent document processing](https://www.nutrient.io/blog/ocr-vs-intelligent-document-processing-choosing-extraction-engine/). For workloads with strict data-residency rules, the local tiers keep the entire pipeline in-house; teams that can accept a hosted endpoint can use the [Data Extraction API](https://www.nutrient.io/api/data-extraction-api/) instead. ## The Nutrient products in this pipeline The pipeline is built from three Nutrient components: - **Vision API** — Runs the extraction engines. Local ICR (the default) performs layout analysis, table extraction, handwriting recognition, and reading-order detection on local hardware, turning a scanned form into structured JSON without a document leaving the machine. An opt-in VLM tier is available for the hardest layouts. - **Self-hosted Document Engine** — Stores, renders, and orchestrates documents inside the network. Running it in-house is what lets the pipeline operate on-premises or air-gapped, keeping PHI under the organization’s control. - **Data Extraction API** — A fully hosted option for teams that don’t require on-premises processing, offering the same structured extraction through a managed endpoint. For a healthcare team, the practical benefit is control over where processing happens: Sensitive documents can be handled entirely in-house, while the same tools remain available as a managed service where that is acceptable. [Start a free trial](https://www.nutrient.io/try/) [Talk to our team](https://www.nutrient.io/contact-sales/) **Call to Action** *See how a medical center streamlined its document workflows with Nutrient* Read the Holyoke Medical Center story [Learn More](https://www.nutrient.io/blog/customers-holyoke-medical-center/) ## FAQ #### Can patient documents be processed without sending PHI to the cloud? Yes. With a self-hosted Document Engine and local ICR extraction, scanned patient documents are stored and analyzed inside the covered entity’s network, and the default extraction path makes no external API call. #### Does on-premises extraction work offline or air-gapped? Local ICR runs on local hardware, so extraction can operate without internet access once the engine and models are deployed. [Air-gapped](https://en.wikipedia.org/wiki/Air_gap_(networking)) operation depends on the surrounding deployment being fully self-contained. #### What structured data does ICR return from a patient form? ICR returns JSON with layout and semantic elements — paragraphs, tables with cell coordinates, key-value regions, and reading order — plus a bounding box for every element, so each value maps back to its location on the page. #### Is cloud AI ever required for healthcare documents? No. Local ICR covers most layouts on its own. A cloud VLM tier is available for the hardest cases and is strictly opt-in, sending layout data rather than the raw document. For the broader argument behind this approach, read [the case for private document AI][private-ai]; for more on document automation in the sector, see [the future of healthcare document automation](https://www.nutrient.io/blog/future-healthcare-document-automation/). --- ## Related pages - [The business case for accessibility: Five ways it drives enterprise value](/blog/5-ways-accessibility-drives-enterprise-value.md) - [Accessibility Untangled Why It Matters Guide](/blog/accessibility-untangled-why-it-matters-guide.md) - [Advanced Techniques For React Native Ui Components](/blog/advanced-techniques-for-react-native-ui-components.md) - [`vector_store` holds your indexed documents (see the multimodal RAG post](/blog/agentic-rag.md) - [Ai Document Automation Extraction To Action](/blog/ai-document-automation-extraction-to-action.md) - [Ai Legal Assistant Document Authoring](/blog/ai-legal-assistant-document-authoring.md) - [Angular File Viewer Pdf Image Office Files](/blog/angular-file-viewer-pdf-image-office-files.md) - [Auto Tagging And Document Accessibility In Dotnet Sdk](/blog/auto-tagging-and-document-accessibility-in-dotnet-sdk.md) - [Best Document Ai Platforms](/blog/best-document-ai-platforms.md) - [Best Document Viewers](/blog/best-document-viewers.md) - [The CEO’s AI playbook: Why decision architecture beats model selection](/blog/ceo-ai-playbook-decision-architecture.md) - [1. Extract and chunk the PDF.](/blog/chat-with-pdf.md) - [Complete Guide To Pdfjs](/blog/complete-guide-to-pdfjs.md) - [Construction Document Data Extraction](/blog/construction-document-data-extraction.md) - [Convert One Drive Files To Pdf In Sharepoint](/blog/convert-one-drive-files-to-pdf-in-sharepoint.md) - [Create And Edit Pdfs In Flutter](/blog/create-and-edit-pdfs-in-flutter.md) - [Create Pdfs With React](/blog/create-pdfs-with-react.md) - [Creating A Document Scanner With Ocr In Python](/blog/creating-a-document-scanner-with-ocr-in-python.md) - [Creating And Filling Pdf Forms Programmatically In Javascript](/blog/creating-and-filling-pdf-forms-programmatically-in-javascript.md) - [The CTO’s AI playbook: Why accountability architecture beats orchestration](/blog/cto-ai-playbook-accountability-architecture.md) - [Digital Signatures](/blog/digital-signatures.md) - [Digital Workflow Automation](/blog/digital-workflow-automation.md) - [Document Ai Vs Ocr](/blog/document-ai-vs-ocr.md) - [Document Extraction Confidence Scores](/blog/document-extraction-confidence-scores.md) - [Document Viewer](/blog/document-viewer.md) - [Document Watermarking](/blog/document-watermarking.md) - [Emerging threats: Your logging system may be an agentic threat vector](/blog/emerging-threats-your-logging-system.md) - [app.py](/blog/extract-text-from-pdf-using-python.md) - [Fillable Pdf](/blog/fillable-pdf.md) - [How To Add Digital Signature To Pdf Using React](/blog/how-to-add-digital-signature-to-pdf-using-react.md) - [How To Build A Dotnet Maui Pdf Viewer](/blog/how-to-build-a-dotnet-maui-pdf-viewer.md) - [How To Build A Flutter Pdf Viewer](/blog/how-to-build-a-flutter-pdf-viewer.md) - [or](/blog/how-to-build-a-javascript-pdf-viewer-with-pdfjs.md) - [How To Build A Javascript Pdf Viewer](/blog/how-to-build-a-javascript-pdf-viewer.md) - [or](/blog/how-to-build-a-nextjs-pdf-viewer.md) - [How To Build A Powerpoint Viewer Using Javascript](/blog/how-to-build-a-powerpoint-viewer-using-javascript.md) - [Using Yarn](/blog/how-to-build-a-react-excel-viewer.md) - [How To Build A React Native Pdf Viewer](/blog/how-to-build-a-react-native-pdf-viewer.md) - [How To Build A React Powerpoint Viewer](/blog/how-to-build-a-react-powerpoint-viewer.md) - [How To Build A Reactjs File Viewer](/blog/how-to-build-a-reactjs-file-viewer.md) - [or](/blog/how-to-build-a-reactjs-pdf-viewer-with-react-pdf.md) - [or](/blog/how-to-build-a-reactjs-pdf-viewer.md) - [How To Build A Reactjs Viewer With Pdfjs](/blog/how-to-build-a-reactjs-viewer-with-pdfjs.md) - [How To Build A Vuejs Pdf Viewer With Pdfjs](/blog/how-to-build-a-vuejs-pdf-viewer-with-pdfjs.md) - [How To Build A Vuejs Pdf Viewer](/blog/how-to-build-a-vuejs-pdf-viewer.md) - [How To Build An Android Pdf Viewer](/blog/how-to-build-an-android-pdf-viewer.md) - [How To Build An Angular Pdf Viewer With Ng2 Pdf Viewer](/blog/how-to-build-an-angular-pdf-viewer-with-ng2-pdf-viewer.md) - [How To Build An Angular Pdf Viewer With Pdfjs](/blog/how-to-build-an-angular-pdf-viewer-with-pdfjs.md) - [How To Convert Docx To Pdf Using Javascript](/blog/how-to-convert-docx-to-pdf-using-javascript.md) - [How To Convert Docx To Pdf Using Python](/blog/how-to-convert-docx-to-pdf-using-python.md) - [How To Convert Html To Pdf Using Html2pdf](/blog/how-to-convert-html-to-pdf-using-html2pdf.md) - [or](/blog/how-to-convert-html-to-pdf-using-react.md) - [How To Convert Html To Pdf Using Wkhtmltopdf And Csharp](/blog/how-to-convert-html-to-pdf-using-wkhtmltopdf-and-csharp.md) - [or](/blog/how-to-convert-html-to-pdf-using-wkhtmltopdf-and-python.md) - [How To Convert Word To Pdf In Nodejs](/blog/how-to-convert-word-to-pdf-in-nodejs.md) - [or](/blog/how-to-create-a-react-js-signature-pad.md) - [How To Create Pdfs With React To Pdf](/blog/how-to-create-pdfs-with-react-to-pdf.md) - [How To Edit Pdfs Using Ios Pdf Library](/blog/how-to-edit-pdfs-using-ios-pdf-library.md) - [How To Embed A Pdf Viewer In Your Website](/blog/how-to-embed-a-pdf-viewer-in-your-website.md) - [How To Extract Tables From Pdf And Images](/blog/how-to-extract-tables-from-pdf-and-images.md) - [How To Generate Pdf From Html With Nodejs](/blog/how-to-generate-pdf-from-html-with-nodejs.md) - [base_url tells WeasyPrint where to resolve relative asset paths](/blog/how-to-generate-pdf-reports-from-html-in-python.md) - [How To Merge Pdfs Using Javascript](/blog/how-to-merge-pdfs-using-javascript.md) - [How To Ocr Pdfs In Linux](/blog/how-to-ocr-pdfs-in-linux.md) - [How To Print Pdf In Csharp](/blog/how-to-print-pdf-in-csharp.md) - [Open an image.](/blog/how-to-use-tesseract-ocr-in-python.md) - [From an HTML string.](/blog/html-in-pdf-format.md) - [Javascript Pdf Editors](/blog/javascript-pdf-editors.md) - [Javascript Pdf Libraries](/blog/javascript-pdf-libraries.md) - [Linearized Pdf](/blog/linearized-pdf.md) - [or](/blog/merge-pdfs.md) - [Swift Package Manager](/blog/mobile-pdf-sdk.md) - [`elements` come from your document parser — each has a type and content.](/blog/multimodal-rag.md) - [Nutrient Vs Conga Composer](/blog/nutrient-vs-conga-composer.md) - [Online Document Viewer](/blog/online-document-viewer.md) - [Open Pdf In Your Web App](/blog/open-pdf-in-your-web-app.md) - [Building WCAG 2.2, Section 508, and PDF/UA-compliant PDFs with an SDK](/blog/pdf-accessibility.md) - [Pdf Data Extraction Developer Guide](/blog/pdf-data-extraction-developer-guide.md) - [Pdf Extraction Benchmark Opendataloader Bench](/blog/pdf-extraction-benchmark-opendataloader-bench.md) - [Pdf Extraction Document Case Studies](/blog/pdf-extraction-document-case-studies.md) - [Pdf Page Labels](/blog/pdf-page-labels.md) - [Pdf Sdk Compliance Security Checklist](/blog/pdf-sdk-compliance-security-checklist.md) - [Pdf Sdk Performance Benchmark](/blog/pdf-sdk-performance-benchmark.md) - [Pdf Ua Compliance Guide](/blog/pdf-ua-compliance-guide.md) - [Pdfjs Accessibility Structtree Printing](/blog/pdfjs-accessibility-structtree-printing.md) - [Pdfjs Advanced Loading Streaming Workers](/blog/pdfjs-advanced-loading-streaming-workers.md) - [Pdfjs Annotation Editor Layer](/blog/pdfjs-annotation-editor-layer.md) - [Pdfjs Area Annotations Canvas Capture](/blog/pdfjs-area-annotations-canvas-capture.md) - [Pdfjs Coordinate Systems Pdf To Screen](/blog/pdfjs-coordinate-systems-pdf-to-screen.md) - [Pdfjs Document Outline Bookmarks Metadata](/blog/pdfjs-document-outline-bookmarks-metadata.md) - [Pdfjs Eventbus Guide](/blog/pdfjs-eventbus-guide.md) - [macOS](/blog/pdfjs-file-format-conversion-to-pdf.md) - [macOS](/blog/pdfjs-generating-pdf-thumbnails-pdf2pic.md) - [Pdfjs Limitations Commercial Upgrade](/blog/pdfjs-limitations-commercial-upgrade.md) - [Pdfjs Native Annotation Layer Forms](/blog/pdfjs-native-annotation-layer-forms.md) - [Pdfjs Navigation Zoom Rotation](/blog/pdfjs-navigation-zoom-rotation.md) - [Pdfjs Pdf Page Manipulation Pdf Lib](/blog/pdfjs-pdf-page-manipulation-pdf-lib.md) - [Pdfjs React Viewer Setup](/blog/pdfjs-react-viewer-setup.md) - [Pdfjs Rendering Overlays React Portals](/blog/pdfjs-rendering-overlays-react-portals.md) - [Pdfjs Server Side Text Extraction](/blog/pdfjs-server-side-text-extraction.md) - [Pdfjs Sticky Note Annotations](/blog/pdfjs-sticky-note-annotations.md) - [Pdfjs Text Highlight Annotations](/blog/pdfjs-text-highlight-annotations.md) - [Pdfjs Text Search Pdffindcontroller](/blog/pdfjs-text-search-pdffindcontroller.md) - [Pdfjs Thumbnail Sidebar](/blog/pdfjs-thumbnail-sidebar.md) - [Process Flows](/blog/process-flows.md) - [React Native Pdf Annotation](/blog/react-native-pdf-annotation.md) - [Using Yarn](/blog/react-pdf-editor.md) - [or](/blog/sample-blog-updated.md) - [Sdk Product Updates Q2 2026](/blog/sdk-product-updates-q2-2026.md) - [Add DWS MCP Server to your Claude Code project.](/blog/teaching-llms-to-read-pdfs.md) - [Open an image file.](/blog/tesseract-python-guide.md) - [Define the HTML part of the document.](/blog/top-10-ways-to-generate-pdfs-in-python.md) - [Top 5 Javascript Pdf Viewers](/blog/top-5-javascript-pdf-viewers.md) - [or](/blog/top-js-pdf-libraries.md) - [Convert an HTML file to PDF.](/blog/top-ten-ways-to-convert-html-to-pdf.md) - [Vector Pdf](/blog/vector-pdf.md) - [Wcag2 Accessibility Requirements Documents](/blog/wcag2-accessibility-requirements-documents.md) - [Web Sdk Is Now Headless](/blog/web-sdk-is-now-headless.md) - [What Are Annotations](/blog/what-are-annotations.md) - [What Is A Vpat](/blog/what-is-a-vpat.md) - [What Is Document Processing](/blog/what-is-document-processing.md) - [What Is Intelligent Document Processing](/blog/what-is-intelligent-document-processing.md) - [What Is Pdf Ua](/blog/what-is-pdf-ua.md) - [Why Pdfium Is A Trusted Platform For Pdf Rendering](/blog/why-pdfium-is-a-trusted-platform-for-pdf-rendering.md) - [Why Your Ai Agent Hallucinates Pdf Table Data](/blog/why-your-ai-agent-hallucinates-pdf-table-data.md)