1.18 release notes

Before attempting to upgrade to Document Engine 1.18, make sure your application runs as expected on your current version. If you’re on version 1.6.1 or later, you can upgrade directly to 1.18. If you’re on an earlier version, follow the step-by-step upgrade path outlined in our upgrade guide.

Highlights

GdPicture broker observability

Document Engine now reports dedicated GdPicture broker metrics for queue wait time, dropped requests, execution duration, and worker saturation. These metrics make it easier to distinguish slow GdPicture operations from broker saturation and are shown in the local metrics dashboard.

PowerPoint and Excel Office templates

The Office templating API now populates PowerPoint and Excel templates in addition to Word templates. The processed response uses the same Office Open XML format as the input template.

Breaking changes

This release doesn’t include any breaking changes.

Deprecations

This release doesn’t include any deprecations.

GdPicture broker metrics

Document Engine now exposes dedicated metrics for the GdPicture broker used by OCR, conversion, PDF/A, optimization, analysis, Office template, and PDF-to-Office operations.

The new metrics include:

• gdpicture_queue_duration_milliseconds — Time spent waiting for an available GdPicture worker.
• gdpicture_queue_drop_duration_milliseconds — Time a request waited before the broker dropped it.
• gdpicture_queue_drop_duration_milliseconds_count — Count of dropped GdPicture broker requests.
• gdpicture_exec_duration_milliseconds — Time spent executing a GdPicture task after a worker accepts it.
• gdpicture_worker_pool_size — Configured GdPicture worker pool size.
• gdpicture_queue_depth and gdpicture_queue_capacity — Current queue pressure.
• gdpicture_queue_fill_ratio — Current queue fill as a ratio of configured capacity.
• gdpicture_tasks_in_flight — Current GdPicture tasks running on workers.

Duration and count metrics use bounded labels such as ocr, convert, convert_pdfa, validate_pdfa, optimize, analyze, process_office_template, and convert_to_office. They don’t include file paths, tenant IDs, request IDs, or document IDs.

The local metrics dashboard now includes panels for GdPicture queue wait, drop rate, saturation, and execution latency. Use these metrics to distinguish slow GdPicture processing from broker saturation: High execution latency with low queue pressure points to slow GdPicture work, while rising queue wait, fill ratio, and drop counts point to broker saturation.

PowerPoint and Excel Office templates

POST /api/process_office_template now accepts PowerPoint (.pptx) and Excel (.xlsx) templates in addition to Word (.docx) templates. The endpoint applies the supplied JSON model and returns the populated document in the same format as the input template.

PowerPoint templates support placeholders, loops, conditions, tables, and images on slides. Slide masters, slide layouts, and speaker notes aren’t processed.

Excel templates process every worksheet and support placeholders, conditions, row loops, images, and typed number, date, percentage, and Boolean cells. Column loops and list expansion within a single cell aren’t supported. Formula references, structured table ranges, and defined names aren’t adjusted when row loops insert rows. Merged-cell ranges below inserted rows shift with their contents, but a merged range on a repeated row is only retained for the first generated row.

The optional config.delimiter.objectDelimiter setting changes the separator used to navigate nested model properties. This allows dots in template markers to be treated as literal model key characters.

For example, to populate an Excel template:

curl -X POST http://localhost:5000/api/process_office_template \
  -H "Authorization: Token token=<API_AUTH_TOKEN>" \
  -F "document=@invoice-template.xlsx" \
  -F "model=<invoice-model.json;type=application/json" \
  --output result.xlsx

PDF/UA validation

Document Engine can now validate PDF/UA-1 conformance through the POST /api/validate_pdfua endpoint. The endpoint accepts uploaded PDFs, remote URLs, and existing Document Engine documents, and it returns a machine-readable validation report.

Performance

Faster orphaned-asset cleanup on large databases

Document Engine periodically removes PDF assets that are no longer referenced by any document or layer — after bulk document deletion and when POST /api/cleanup is called. This cleanup now scans the asset table in short, index-backed batches instead of one unbounded query.

On large databases, the previous query could run for hours at default PostgreSQL settings and delay queued background jobs, including subsequent bulk deletion jobs. In our benchmarks on a database with 1 million documents, a full cleanup pass now completes in a few seconds.

The cleanup is also no longer skipped on databases containing documents without a stored source-file checksum (for example, remote documents that were never fetched). If your database accumulated orphaned assets because of this, the first cleanup after upgrading will remove them, and you may notice a corresponding drop in storage usage. Assets referenced by any document or layer are never removed.

Indexed asset-reference checks

Before an asset is removed, cleanup checks whether any document or layer still references it. These checks look up assets by their identifier in the source_pdf and original_file columns, which weren’t indexed. On large databases, each check fell back to a full table scan and could run for minutes, adding load to the database and holding connections open long enough to strain the connection pool.

This release adds indexes for those identifiers, so the checks now use an index lookup instead of a table scan. See the database migrations section for what to expect when upgrading.

Rendering fidelity configuration

Document Engine now accepts per-request options for simulating overprint and rendering exact 100 percent K-only CMYK black. Full-page REST requests use the overprint_preview and black_rendering query parameters, tile requests use the overprint and pureblack path segments, and batch print requests use the corresponding JSON fields.

Document Engine doesn’t define instance defaults or enable overprint simulation itself. When omitted, overprint simulation is disabled and 100 percent K-only black renders through the regular CMYK conversion as a dark charcoal, matching Acrobat. Nutrient Web SDK’s next release and later sends these options automatically when used with Document Engine 1.18 and later.

Database migrations

This release adds three indexes used by asset cleanup:

• documents_source_pdf_uuid_idx on documents
• documents_original_file_uuid_idx on documents
• layers_source_pdf_uuid_idx on layers

The indexes are created with CREATE INDEX CONCURRENTLY. This doesn’t block reads or writes to the documents or layers tables — they stay fully writable throughout — so it doesn’t require database downtime. It isn’t lock-free, though: The build holds a lighter lock that blocks other schema changes and vacuum on the same table while it runs, and it waits for transactions that started before it to finish before it can complete. If a concurrent build is interrupted, the migration is safe to rerun — it drops and rebuilds any index left in an invalid state.

On a large database, plan for the build to take a while. Each index is built over the source_pdf/original_file document data, which Document Engine stores out of line and compressed once it grows past a few kilobytes. Building the index reads and decompresses that data for every row, so on a large instance, a build can take anywhere from minutes to hours, depending on the number of rows and on disk I/O throughput. The documents_source_pdf_uuid_idx build is the long pole: Nearly every document has a source-file identifier, so it processes the whole table. The documents_original_file_uuid_idx and layers_source_pdf_uuid_idx builds are usually much cheaper because they’re partial (see below). Make sure there’s enough free disk and WAL/replication headroom, and prefer a lower-traffic window. The tables stay writable throughout.

The indexes are partial: They only include rows whose identifier is set. Rows without one — documents uploaded as a PDF (which have no separate original file) and remote documents (which have no stored asset) — aren’t indexed at all, so the cost below scales with the number of referenced rows, not the total row count.

Rough figures per one million indexed (non-null) rows, per index (measured on realistic data, so treat as an order-of-magnitude guide, not a guarantee):

• Permanent extra storage: about 56 MB per million indexed rows, for each index.
• Write-ahead log (WAL) generated during the build: about 50 MB per million indexed rows, per index. This is a one-time cost, but it’s streamed to any replicas and archived, so a replicated setup needs that headroom.
• Temporary disk used while building: about 50 MB per million indexed rows, per index. This is freed as soon as the index build finishes.
• Peak build memory doesn’t grow with row count. It’s bounded by the PostgreSQL maintenance_work_mem setting (64 MB by default); above that limit, the build sorts on disk (the temporary disk above) rather than using more memory.

Running this on a large cluster

Document Engine runs migrations before the server starts, so on a large instance, a node stays unready until the index builds finish. In a rolling deployment, a startup probe or deployment timeout shorter than the build can repeatedly kill the node mid-build; each interrupted CREATE INDEX CONCURRENTLY leaves an invalid index that the next run rebuilds from scratch, which can turn into a restart loop that never completes. For a large cluster:

• Run migrations as a single dedicated one-shot job — set EXIT_AFTER_DATABASE_MIGRATIONS=true on that job — and set ENABLE_DATABASE_MIGRATIONS=false on the application nodes, so only the job builds the indexes and app nodes don’t each attempt it.
• Give that job a generous timeout — hours, not minutes — and don’t let a health-check or deployment timeout kill it mid-build.
• Before starting, check for long-running transactions. A concurrent index build cannot finish until every transaction that started before it has ended, so a single idle-in-transaction session or long query can stall the build regardless of table size.
• While it runs, you can monitor progress with pg_stat_progress_create_index, and watch disk, WAL/archive capacity, replica lag, and I/O latency.
• If the job is interrupted, it’s safe to rerun: It drops and rebuilds any index left in an invalid state.