Metrics Reference

This is a reference page for all internal metrics exported by Document Engine.

Metrics follow the DogStatsD protocol(opens in a new tab) format, a variant of the popular StatsD protocol. Document Engine sends a metric update to the compatible monitoring agent either when some event happens (e.g. an HTTP response is sent), or when a measurement is collected periodically (e.g. the memory used is sampled). The agent aggregates metrics in fixed time windows and forwards them to the monitoring system, where they are persisted for further analysis. How the metric is aggregated depends on its type and the agent implementation (e.g. Telegraf(opens in a new tab) might perform different aggregations than CloudWatch agent(opens in a new tab) does). Refer to our Integration guide to learn how to export Document Engine metrics in different environments and deployment settings.

Metric Types

Document Engine exports three types of metrics:

  • Counters — Each metric update carries a value that increases a counter by that value. An example of this is when a file system cache hits metric, where a counter increment is sent every time an item is found in the cache.
  • Gauges — A metric update carries the most recent value of a particular measurement. This is a very common type for metrics gathered periodically, e.g. memory usage.
  • Timings — Each metric update carries the duration of a particular event. An example of this is an HTTP request handling duration. Usually agents aggregate timings by calculating statistics based on measurements falling into the time window, e.g. count, minimum, maximum, mean, percentiles, etc. Timings are often used when we need to both count the events and measure their duration.

Tags

Apart from the metric name and value, each metric update includes a set of tags that allow you to group and filter measurements belonging to the same metric when analyzing them.

This is a list of common tags attached to every metric exported by Document Engine:

TagDescription
hostThe hostname of the Document Engine container
nodeThe unique ID of the Document Engine node in the cluster
familyThis is always set to pspdfkit-document-engine

Metrics Reference

HTTP Performance

NameTypeUnit
http_server.req_endtimingmillisecond

This is the duration it takes Document Engine to process the HTTP request and respond.

TagDescription
statusHTTP response status
methodHTTP request method
groupEither standard for regular HTTP requests, or long_poll for long polling requests

When analyzing HTTP performance metrics, make sure to separate metrics based on the group. By definition, long polling requests take a long time to complete because the client keeps the connection open to allow Document Engine to send a response only when it has data ready. In most situations, you’re most likely interested in metrics with the group set to standard.

PostgreSQL Performance

NameTypeUnit
pg_client.querytimingmillisecond
pg_client.queuetimingmillisecond
pg_client.decodetimingmillisecond
pg_client.result_sizegauge-

These measurements concern the performance of SQL queries made by Document Engine against PostgreSQL:

  • pg_client.query tells you how long it took to actually execute the query.
  • pg_client.queue tells you how long the query waited for the connection to be available from the pool.
  • pg_client.decode tells you how long it took to decode the query results.
  • pg_client.result_size is a measurement that tells you how many rows were returned per query.

To get insight into the total time it takes to complete the database query, you need to take the sum of the query, queue, and decode measurements.

TagDescription
resultEither success or error, indicating if the query succeeded.
commandThe SQL command that was executed. One of select, update, delete, insert, begin, commit, or rollback.
error_codePostgreSQL error code, only set when result is error.
severityError severity, only set when result is error

Asset Storage

NameTypeUnit
assets.fetch_assettimingmillisecond
assets.store_assettimingmillisecond

These measurements track the time it takes to retrieve or store the asset in the asset storage. Note that the asset is only fetched from the storage if it’s not found in the cache.

TagDescription
resultEither success or error, indicating if the operation was successful.

File System Cache

NameTypeUnit
cache.fs_hitcounter-
cache.fs_misscounter-
cache.fs_sizegaugebyte
cache.fs_freetimingmillisecond

These measurements are related to the Document Engine file system cache used for document source files:

  • The cache.fs_hit and cache.fs_miss measurements count cache hits and misses.
  • The cache.fs_size measurement reports the current size of the file system cache. The cache size is limited by the ASSET_STORAGE_CACHE_SIZE configuration option.
  • The cache.fs_free measurement tells you how long it took to clear a full cache.

In-Memory Cache

NameTypeUnit
cache.memory_hitcounter-
cache.memory_misscounter-

These measurements are related to the in-memory cache for PDF metadata. cache.fs_hit and cache.fs_miss measurements count cache hits and misses.

Redis Cache

NameTypeUnit
cache.redis_hittimingmillisecond
cache.redis_misstimingmillisecond
cache.redis_settimingmillisecond
cache.redis_errortimingmillisecond

These measurements are related to the optional Redis cache used for caching rendering results between multiple Document Engine instances.

  • cache.redis_hit indicates how long it took to fetch an item from Redis when there was a cache hit.
  • cache.redis_miss indicates how long the request to Redis took when there was a cache miss.
  • cache.redis_set indicates how long it took to store an item in Redis.
  • cache_redis_error indicates how long a Redis operation that errored out took.
TagDescription
opThe Redis operation that was performed. Only set for the cache.redis_error metric.

Remote Documents

NameTypeUnit
remote_doc.response_starttimingmillisecond
remote_doc.response_endtimingmillisecond

These measurements concern the time it takes the Document Engine to fetch documents from remote URLs.

  • remote_doc.response_start tells you the time between when the Document Engine sent the request and when the first byte of data has been received.
  • remote_doc.response_end tells you how long the actual data transfer took after the remote server started responding.

To get the total remote document response time, sum up both metrics.

TagDescription
resultsuccess, error, or timeout, indicating if fetching the remote document succeeded.

Document Conversion

NameTypeUnit
document_conversion.converttimingmillisecond

The duration of the Office documents conversion.

TagDescription
resultEither success or error, indicating the conversion result.

PDF Processing

NameTypeUnit
pspdfkitd.queuetimingmillisecond
pspdfkitd.exectimingmillisecond

These measurements concern all Document Engine operations that involve working with PDFs, including rendering, extracting content, and preparing PDFs for being downloaded.

  • pspdfkit.queue tells you how long an operation had to wait until there was a worker available to execute it.
  • pspdfkit.exec tells you how long the operation actually took.
TagDescription
requestThe PDF operation performed

Signing Service

NameTypeUnit
signing_service.signtimingmillisecond

How long it took the signing service to respond to the signing request.

TagDescription
resultEither success or error, indicating if the call to the signing service succeeded.

Instant

Metric NameMetric TypeUnitDescription
layer.sync.hookstimingmillisecondsDuration of the time spent across the various hooks that are registered to perform various operations on sync
layer.sync.totaltimingmillisecondsTotal duration of the sync operation

These metrics track the duration of Instant sync. For operations that fail, Document Engine emits layer.sync.total only.

TagDescription
resultOnly set for layer.sync.total. Either success or error, indicating if Instant sync succeeded.

Memory Total

NameTypeUnit
vm_memory.totalgaugebyte

The total amount of memory allocated by the Document Engine process. Note that the amount of memory taken by the Document Engine container is usually larger than this number, since there are also other processes running inside the container.

Compute Resources Utilization

NameTypeUnit
vm_scheduler_wall_time.activetimingmillisecond
vm_scheduler_wall_time.totaltimingmillisecond
  • vm_scheduler_wall_time.active tells you how much time the Erlang VM spent being active in the last interval.
  • vm_scheduler_wall_time.total tells you the total uptime of the Erlang VM.

If you divide the active time by the total time, the resulting number indicates the utilization of compute resources assigned to Document Engine. In other words, it says which percentage of the time the Document Engine was busy doing work.

Note that, as with memory, this only concerns the Document Engine process — the CPU utilization of the container may be different, as there are other processes running inside it as well.

TagDescription
scheduler_numberThe internal Document Engine scheduler number

Document Engine starts as many schedulers as there are logical CPU cores available. In most cases, you’ll want to take the average of metrics described here across schedulers.