Speed up search with a full-text index
Document Engine can use a cached full-text search (FTS) index to make plain text searches on large documents faster. Document Engine uses the index to find the pages that can contain a match. It then searches only those pages instead of every page of the source PDF.
The feature is opt-in and disabled by default. Deploying a release that includes it doesn’t change search behavior until you enable it, and the /search response shape is unchanged — SDK clients need no changes. FTS has been available since Document Engine 1.17.
In Document Engine 1.17.0 to 1.19.0, FTS is a preview. Index builds need an extra queue configuration, and search results can differ from the regular search. Read the known issues in Document Engine 1.17 to 1.19 before you enable FTS on these versions.
For a cross-product overview of Web SDK, Document Engine, and DWS Viewer API search and document navigation workflows, refer to the choose the right search, bookmarks, and document navigation setup guide.
When to use it
Without the index, every /search request opens the source PDF and searches every page in the requested range. That cost grows with document size, so search on large blueprint- or manual-style PDFs can take several seconds per query.
With the index, a search is fastest when the search term occurs on few pages. A search term that occurs on most pages takes about as long as a search without the index, because Document Engine still searches most pages. For small documents, where a scan is already fast, the two paths perform similarly.
Enable it
FTS is controlled by two environment variables on the document-engine container. A typical rollout enables them in this order:
- Set
SEARCH_FTS_BUILD_ON_WRITE=truefirst, so indexes are built in the background as documents are created or their source PDFs are replaced. This lets indexes accumulate before any query relies on them. - Set
SEARCH_FTS_ENABLED=trueto route eligible plain text searches through the index. For a document without a built index, Document Engine enqueues a build and uses the regular search for that request.
Both default to false. A third option, SEARCH_FTS_ENRICHMENT, controls how Document Engine turns matches into preview text and highlight rectangles. The supported value is daemon (the default). Document Engine then computes previews and rectangles with the regular search on the pages that the index selects.
Refer to the search options in the configuration options guide for the full list.
Choose the nodes that build indexes
Index builds run as background jobs on the internal search_index queue. From Document Engine 1.20.0, every node that uses persistent database storage processes this queue, and ASYNC_WORKER_QUEUES doesn’t affect it.
To keep index builds off a node, set SEARCH_FTS_BUILD_WORKER=false on that node. At least one node must keep the default value, true. When clustering is enabled, a build runs on the document owner node, which can be a node with this option set to false. To keep builds off a node completely, process them only on worker nodes with CLUSTERING_ENABLED=false. The async_jobs_queue_state_jobs metric with queue="search_index" and state="available" shows build jobs that no node processes.
Search results
From Document Engine 1.20.0, a search that uses the index returns the same results as one without it. The pages, preview text, highlight ranges, and highlight rectangles match. This includes phrases, matches inside a word, and matches across a line break.
What uses the FTS path
To keep behavior safe and predictable, FTS only handles plain text search. The following requests continue to use the previous per-query scan:
- Searches that include annotations (
include_annotations=true) - Regex and preset searches
- Case-sensitive searches (
case_sensitive=true) - Any document without a built or reachable index — a build is enqueued and the request falls back for that query
How indexing works
- Document Engine stores the index per source PDF, so documents that share a source PDF share one index. Replacing a document’s source PDF invalidates its index.
- Indexes are built by background jobs and persisted to your configured asset storage backend (S3, Azure Blob Storage, or the built-in PostgreSQL storage). A node restart, or a request routed to a different node, reuses the persisted index rather than rebuilding it.
- From Document Engine 1.20.0, each release builds its own indexes. After an upgrade, a document gets a new index on its first search or write. Until then, its searches use the regular search. When Document Engine stores a new index, it deletes the indexes of earlier releases for the same PDF if they’re older than one day.
- Index builds never block document upload, source-PDF replacement, or deletion.
Known issues in Document Engine 1.17 to 1.19
Two issues affect FTS in Document Engine 1.17.0 to 1.19.0. Document Engine 1.20.0 fixes both issues.
Index builds need the search_index queue
ASYNC_WORKER_QUEUES sets the complete list of background job queues that a node processes. The default value, default=5, doesn’t include the search_index queue for FTS index builds. Without it, index build jobs stay queued, no index is built, and searches use the previous per-query scan.
Add search_index to ASYNC_WORKER_QUEUES on at least one node. A typical single-node configuration is:
ASYNC_WORKER_QUEUES: "default=5,search_index=2"
At least one node must also process default for regular asynchronous operations and for asynchronous result and metadata cleanup. In a split deployment, different nodes can process default and search_index. After a restart, the node processes the index build jobs that are already queued, so you don’t need to recreate documents.
Document Engine 1.20.0 ignores search_index in ASYNC_WORKER_QUEUES, so you can remove it after you upgrade.
Search results differ from the regular search
With SEARCH_FTS_ENABLED=true, a plain text search that uses the index can return different hits than the same search without the index:
- A phrase search returns each word of the phrase as a separate hit, including on pages that don’t contain the phrase.
- A word also matches longer words with the same stem.
- Highlights can end before the end of the matched word.
- Matches across a line break and matches inside a word are missing.
If your application needs the same results as the regular search, set SEARCH_FTS_ENABLED=false on these versions.