1.18 release notes

Before attempting to upgrade to Document Engine 1.18, make sure your application runs as expected on your current version. If you’re on version 1.6.1 or later, you can upgrade directly to 1.18. If you’re on an earlier version, follow the step-by-step upgrade path outlined in our upgrade guide.

Highlights

Instant synchronization performance and observability

Since Document Engine 1.18.1.

Document Engine now spends less time inside the serialized Instant synchronization write path for hot layers by avoiding unnecessary full-layer reads, fetching full changed-record details after the write transaction, and pre-storing valid attachment data before taking the layer write lock. Document Engine also skips database lookups that cannot match a record, no longer rereads the layer row after a write, and removes an unused index that prevented PostgreSQL from optimizing the layer revision update.

Document Engine also reports more detailed Instant synchronization telemetry, including layer lock wait time, write transaction duration, attachment work, changed-record counts, and accepted or rejected write counts.

GdPicture broker observability

Document Engine now reports dedicated GdPicture broker metrics for queue wait time, dropped requests, execution duration, and worker saturation. These metrics make it easier to distinguish slow GdPicture operations from broker saturation and are shown in the local metrics dashboard.

PowerPoint and Excel Office templates

The Office templating API now populates PowerPoint and Excel templates in addition to Word templates. The processed response uses the same Office Open XML format as the input template.

Breaking changes

This release doesn’t include any breaking changes.

Deprecations

This release doesn’t include any deprecations.

Instant synchronization performance and observability

Since Document Engine 1.18.1.

This release improves Instant synchronization behavior for workloads where many clients write to the same layer at the same time.

Document Engine now avoids a full-layer record read when validating anonymity updates unless a request explicitly changes isAnonymous. When a write request is based on an older record revision, Document Engine now uses lightweight record ID checks inside the write transaction and fetches the full changed-record response after the transaction commits.

For attachment-backed records, Document Engine can also store valid request attachment data before entering the layer write transaction. The transaction still validates and links accepted attachments, but the expensive blob storage work no longer has to run while the layer write lock is held.

Document Engine also removes further redundant work from the layer write transaction. Permission and record lookups that receive an empty list of record IDs no longer query the database, so a request that only creates annotations no longer runs queries that cannot match a record. After writing records, Document Engine no longer rereads the layer row to learn the revision it has just assigned.

Document Engine removes an unused index on the layers table. The index covered the record revision column that every Instant write updates, which prevented PostgreSQL from applying the heap-only tuple (HOT) optimization to that update. Removing it lets those updates stay on the same page and reduces table bloat on layers that are written to frequently. See the database migrations section for more information.

Document Engine now emits additional Instant synchronization telemetry on the existing [:layer, :sync] event. These measurements include layer lock wait time, layer revision update time, transaction duration, database write time, attachment pre-storage time, stale attachment cleanup time, changed-record count, attachment count, input change counts, accepted change counts, rejected change counts, and response change counts. Deployments using the StatsD reporter also receive corresponding layer.sync.* timing and count metrics.

Remote URL fetch loopback allowlists

Since Document Engine 1.18.1.

Document Engine now allows controlled self-hosted deployments to add an exception to fetch loopback URLs through the remote URL fetch allowlist configuration. This isn’t recommended unless absolutely necessary because it could allow undesired queries to reach otherwise firewalled destinations on the local host.

Use REMOTE_URL_FETCH_ALLOWED_HOSTS=localhost for localhost URLs. Use REMOTE_URL_FETCH_ALLOWED_CIDRS=127.0.0.1/32 for literal IPv4 loopback URLs such as http://127.0.0.1/..., or ::1/128 for IPv6 loopback URLs. Other critical SSRF ranges, including link-local and cloud metadata addresses, remain blocked unless REMOTE_URL_FETCH_POLICY=allow_all is used.

GdPicture broker metrics

Document Engine now exposes dedicated metrics for the GdPicture broker used by OCR, conversion, PDF/A, optimization, analysis, Office template, and PDF-to-Office operations.

The new metrics include:

• gdpicture_queue_duration_milliseconds — Time spent waiting for an available GdPicture worker.
• gdpicture_queue_drop_duration_milliseconds — Time a request waited before the broker dropped it.
• gdpicture_queue_drop_duration_milliseconds_count — Count of dropped GdPicture broker requests.
• gdpicture_exec_duration_milliseconds — Time spent executing a GdPicture task after a worker accepts it.
• gdpicture_worker_pool_size — Configured GdPicture worker pool size.
• gdpicture_queue_depth and gdpicture_queue_capacity — Current queue pressure.
• gdpicture_queue_fill_ratio — Current queue fill as a ratio of configured capacity.
• gdpicture_tasks_in_flight — Current GdPicture tasks running on workers.

Duration and count metrics use bounded labels such as ocr, convert, convert_pdfa, validate_pdfa, optimize, analyze, process_office_template, and convert_to_office. They don’t include file paths, tenant IDs, request IDs, or document IDs.

The local metrics dashboard now includes panels for GdPicture queue wait, drop rate, saturation, and execution latency. Use these metrics to distinguish slow GdPicture processing from broker saturation: High execution latency with low queue pressure points to slow GdPicture work, while rising queue wait, fill ratio, and drop counts point to broker saturation.

PowerPoint and Excel Office templates

POST /api/process_office_template now accepts PowerPoint (.pptx) and Excel (.xlsx) templates in addition to Word (.docx) templates. The endpoint applies the supplied JSON model and returns the populated document in the same format as the input template.

PowerPoint templates support placeholders, loops, conditions, tables, and images on slides. Slide masters, slide layouts, and speaker notes aren’t processed.

Excel templates process every worksheet and support placeholders, conditions, row loops, images, and typed number, date, percentage, and Boolean cells. Column loops and list expansion within a single cell aren’t supported. Formula references, structured table ranges, and defined names aren’t adjusted when row loops insert rows. Merged-cell ranges below inserted rows shift with their contents, but a merged range on a repeated row is only retained for the first generated row.

The optional config.delimiter.objectDelimiter setting changes the separator used to navigate nested model properties. This allows dots in template markers to be treated as literal model key characters.

For example, to populate an Excel template:

curl -X POST http://localhost:5000/api/process_office_template \
  -H "Authorization: Token token=<API_AUTH_TOKEN>" \
  -F "document=@invoice-template.xlsx" \
  -F "model=<invoice-model.json;type=application/json" \
  --output result.xlsx

PDF/UA validation

Document Engine can now validate PDF/UA-1 conformance through the POST /api/validate_pdfua endpoint. The endpoint accepts uploaded PDFs, remote URLs, and existing Document Engine documents, and it returns a machine-readable validation report.

Performance

Faster orphaned-asset cleanup on large databases

Document Engine periodically removes PDF assets that are no longer referenced by any document or layer — after bulk document deletion and when POST /api/cleanup is called. This cleanup now scans the asset table in short, index-backed batches instead of one unbounded query.

On large databases, the previous query could run for hours at default PostgreSQL settings and delay queued background jobs, including subsequent bulk deletion jobs. In our benchmarks on a database with 1 million documents, a full cleanup pass now completes in a few seconds.

The cleanup is also no longer skipped on databases containing documents without a stored source-file checksum (for example, remote documents that were never fetched). If your database accumulated orphaned assets because of this, the first cleanup after upgrading will remove them, and you may notice a corresponding drop in storage usage. Assets referenced by any document or layer are never removed.

Indexed asset-reference checks

Before an asset is removed, cleanup checks whether any document or layer still references it. These checks look up assets by their identifier in the source_pdf and original_file columns, which weren’t indexed. On large databases, each check fell back to a full table scan and could run for minutes, adding load to the database and holding connections open long enough to strain the connection pool.

This release adds indexes for those identifiers, so the checks now use an index lookup instead of a table scan. See the database migrations section for what to expect when upgrading.

Rendering fidelity configuration

Document Engine now accepts per-request options for simulating overprint and rendering exact 100 percent K-only CMYK black. Full-page REST requests use the overprint_preview and black_rendering query parameters, tile requests use the overprint and pureblack path segments, and batch print requests use the corresponding JSON fields.

Document Engine doesn’t define instance defaults or enable overprint simulation itself. When omitted, overprint simulation is disabled and 100 percent K-only black renders through the regular CMYK conversion as a dark charcoal, matching Acrobat. Nutrient Web SDK’s next release and later sends these options automatically when used with Document Engine 1.18 and later.

Database migrations

This release adds three indexes used by asset cleanup:

• documents_source_pdf_uuid_idx on documents
• documents_original_file_uuid_idx on documents
• layers_source_pdf_uuid_idx on layers

The indexes are created with CREATE INDEX CONCURRENTLY. This doesn’t block reads or writes to the documents or layers tables — they stay fully writable throughout — so it doesn’t require database downtime. It isn’t lock-free, though: The build holds a lighter lock that blocks other schema changes and vacuum on the same table while it runs, and it waits for transactions that started before it to finish before it can complete. If a concurrent build is interrupted, the migration is safe to rerun — it drops and rebuilds any index left in an invalid state.

On a large database, plan for the build to take a while. Each index is built over the source_pdf/original_file document data, which Document Engine stores out of line and compressed once it grows past a few kilobytes. Building the index reads and decompresses that data for every row, so on a large instance, a build can take anywhere from minutes to hours, depending on the number of rows and on disk I/O throughput. The documents_source_pdf_uuid_idx build is the long pole: Nearly every document has a source-file identifier, so it processes the whole table. The documents_original_file_uuid_idx and layers_source_pdf_uuid_idx builds are usually much cheaper because they’re partial (see below). Make sure there’s enough free disk and WAL/replication headroom, and prefer a lower-traffic window. The tables stay writable throughout.

The indexes are partial: They only include rows whose identifier is set. Rows without one — documents uploaded as a PDF (which have no separate original file) and remote documents (which have no stored asset) — aren’t indexed at all, so the cost below scales with the number of referenced rows, not the total row count.

Rough figures per one million indexed (non-null) rows, per index (measured on realistic data, so treat as an order-of-magnitude guide, not a guarantee):

• Permanent extra storage: about 56 MB per million indexed rows, for each index.
• Write-ahead log (WAL) generated during the build: about 50 MB per million indexed rows, per index. This is a one-time cost, but it’s streamed to any replicas and archived, so a replicated setup needs that headroom.
• Temporary disk used while building: about 50 MB per million indexed rows, per index. This is freed as soon as the index build finishes.
• Peak build memory doesn’t grow with row count. It’s bounded by the PostgreSQL maintenance_work_mem setting (64 MB by default); above that limit, the build sorts on disk (the temporary disk above) rather than using more memory.

Running this on a large cluster

Document Engine runs migrations before the server starts, so on a large instance, a node stays unready until the index builds finish. In a rolling deployment, a startup probe or deployment timeout shorter than the build can repeatedly kill the node mid-build; each interrupted CREATE INDEX CONCURRENTLY leaves an invalid index that the next run rebuilds from scratch, which can turn into a restart loop that never completes. For a large cluster:

• Run migrations as a single dedicated one-shot job — set EXIT_AFTER_DATABASE_MIGRATIONS=true on that job — and set ENABLE_DATABASE_MIGRATIONS=false on the application nodes, so only the job builds the indexes and app nodes don’t each attempt it.
• Give that job a generous timeout — hours, not minutes — and don’t let a health-check or deployment timeout kill it mid-build.
• Before starting, check for long-running transactions. A concurrent index build cannot finish until every transaction that started before it has ended, so a single idle-in-transaction session or long query can stall the build regardless of table size.
• While it runs, you can monitor progress with pg_stat_progress_create_index, and watch disk, WAL/archive capacity, replica lag, and I/O latency.
• If the job is interrupted, it’s safe to rerun: It drops and rebuilds any index left in an invalid state.

This release also contains a migration that removes the unused layers_greatest_idx index from the layers table.

Since Document Engine 1.18.1.

The index covered GREATEST(mrev, rrev). No query used it. Because it covered the record revision column that every Instant write updates, it prevented PostgreSQL from applying the heap-only tuple (HOT) optimization to that update, which made each write add an index entry and leave a dead row version that HOT pruning cannot reclaim.

Operational note: The index is dropped with DROP INDEX CONCURRENTLY, so the migration doesn’t take a lock that blocks reads or writes on layers. Document Engine continues to serve Instant synchronization while the migration runs. The migration doesn’t rewrite the table.