This HTML page is not optimized for LLM or AI agent consumption. Fetch the Markdown version instead: /guides/document-engine/troubleshooting/errors-and-warnings/overload.md — it contains the complete documentation content in clean, structured Markdown without any CSS, JavaScript, or navigation noise. Document Engine: 503 service unavailable (overload)

Document Engine includes a back-pressure system that prevents a node from accepting more work than it can safely process. When the internal processing queue becomes full, the node responds with an HTTP 503 service unavailable error. This overload condition is intentional: It protects node stability and signals to clients that they must reduce request concurrency and retry failed operations with backoff.

Since Document Engine 1.20, Build API requests also return 503 when a node exhausts its remote URL fetch pool. The response has reason set to overload and identifies the failing input in failingPaths. Earlier releases returned 400 for this condition. Handle it as a temporary capacity issue and retry with backoff, not as a malformed request. Asynchronous Build jobs retry rejected fetches.

Why overload can occur

Although overload reflects expected back-pressure rather than an unexpected failure, several conditions can increase the likelihood of queue saturation.

Local caching and initial workload

Document Engine maintains a local file system cache on each node. When a document is accessed for the first time on a node, the system must fetch, parse, and prepare it before serving additional operations. This parsing work is CPU-intensive and occupies workers. New nodes, or nodes that restart, begin with empty caches and therefore perform this initial work more often.

Repeated work across multiple nodes

In distributed or autoscaling deployments, multiple nodes may be required to process the same documents before their caches have warmed. Since the local file system cache isn’t shared between nodes, each node repeats the same initial processing steps. During peaks, this duplicated work can increase processing load across the cluster.

Scaling behavior in dynamic environments

Deployments that rapidly create new nodes may result in multiple instances starting with empty caches. If high traffic arrives before these caches have warmed, the additional initial work performed on each node can increase overall processing load and make queue saturation more likely.

Remote URL fetch capacity

URL inputs to the Build API and documents added by URL share a fixed-size downloader pool on each node. Document Engine rejects a fetch when the waiting queue is full or the fetch waits too long for a worker. The remote URL fetch tuning section describes the tuning options.

Strategies that improve stability

The following configuration patterns help reduce repeated work and improve stability under load. These approaches are grounded in the system’s documented behavior and have been effective in real deployments.

Effective use of local caching

Ensuring that frequently accessed documents remain in the local asset cache reduces the amount of repeated fetch-and-parse work. Document Engine’s caching behavior is documented in the cache configuration guide. You can adjust the size of the on-disk document cache using the ASSET_STORAGE_CACHE_SIZE environment variable; see the configuration options limits and timeouts for details and defaults.

Shared rendering cache

When supported in the deployment environment, using a shared cache for rendered pages allows multiple nodes to reuse completed rendering work. This reduces the per-node CPU load, especially during periods of high activity.

Node stability and cache locality

Some workloads benefit from maintaining a smaller number of stable, appropriately resourced nodes rather than rapidly creating many short-lived instances. A node that remains active with a sufficiently large cache can avoid repeated initialization work and handle load more consistently.

Remote URL fetch tuning

Use these configuration options to tune remote URL fetch capacity for the workload:

  • DOWNLOADER_WORKER_POOL_SIZE — Concurrent remote fetches per node. The default is 16.
  • DOWNLOADER_BROKER_CLIENT_QUEUE_TIMEOUT — Maximum wait for a free worker, in milliseconds. The default is 5000.
  • DOWNLOADER_BROKER_CLIENT_QUEUE_MAX_SIZE — Maximum number of waiting fetches. The default is 128.

Reduce client concurrency before increasing these limits, and size the pool for available network and processing capacity.

Observability and diagnostics

Understanding overload events requires insight into cache behavior, queue status, and request patterns. Deployments should ensure that metrics remain available even when nodes scale up rapidly, as these metrics provide essential visibility into performance.

Useful metrics include:

  • cache.fs_hit/cache.fs_miss — Show the effectiveness of the local file-system cache
  • cache.fs_size/cache.fs_free — Indicate cache usage and cleanup frequency
  • http_server.req_end (status tags) — Reveal the distribution of successful and error responses
  • Rendering cache metrics — When using a shared cache backend
  • remote_doc.response_start/remote_doc.response_end — Show how long remote URL fetches take

Source PDF fetch errors on password-protected documents

Under heavy load, the logs can show the following errors with the same request_id:

[error] Error accessing cache: :overload.
[error] Could not find PDF for layer <layer> in document <document_id>

For the default layer, <layer> is empty.

In this pair, the first line means Document Engine needed the document’s source PDF, but it wasn’t in the local asset cache. Document Engine queues fetches from S3-compatible storage and remote URLs. The queue dropped this fetch because it was full or the wait in the queue timed out.

The second line comes from a password check that Document Engine runs for requests to password-protected documents. It means the check couldn’t get the source PDF. On its own, the second line can have other causes, such as a missing file.

When both lines appear together, they don’t mean the file is missing or corrupt, or that the password is wrong. The request failed because Document Engine dropped the fetch. Document Engine doesn’t cache a failed fetch, so a retry can succeed once the queue has capacity. Keeping frequently accessed documents in the local cache, as described in effective use of local caching, avoids the fetch.

Summary

HTTP 503 service unavailable signals overload from internal processing back-pressure or exhausted remote URL fetch capacity. Situations involving new or frequently restarted nodes, repeated work across multiple nodes, or limited cache warmth can increase the likelihood of overload. Ensuring adequate caching, maintaining observability, and using appropriate client-side throttling reduces load pressure and improves resiliency during peak periods.

If overload errors persist, contact Support(opens in a new tab) with structured logs (LOG_STRUCTURED=true) from a peak period and your deployment configuration.