Asset storage configuration

Document Engine supports several backends for storing PDFs and other assets. This guide explains the available options, how to choose and configure a backend, and how to work with multiple backends or existing file stores.

Storage options overview

Document Engine supports four storage modes: no asset storage, built-in database storage, S3-compatible object storage, and Azure Blob Storage.

Without storage

Document Engine can run without asset storage. In this mode, you can use only stateless operations, primarily the Build API.

Built-in asset storage

By default, Document Engine stores assets as binary large objects (BLOBs) in the database.

Built-in storage suits small, stable document collections. For production deployments with larger PDFs, use object storage(opens in a new tab). For guidance on large PDFs, refer to the large documents guide.

S3 object storage

Document Engine can store assets in Amazon S3(opens in a new tab) and S3-compatible object storage services.

Use S3-compatible storage when you want Document Engine to store assets in AWS S3 or a compatible service, such as Garage, Ceph, SeaweedFS, or Google Cloud Storage interoperability mode.

Azure Blob Storage

Document Engine can also store assets in Azure Blob Storage(opens in a new tab).

Choose a storage backend

Choose a backend based on document size, how often documents change, and the infrastructure you operate.

Use built-in storage for a small, stable collection of PDFs that changes infrequently. It offers these benefits:

• You don’t need separate object storage infrastructure.
• Backing up the Document Engine PostgreSQL instance also backs up your assets.

However, storing assets in the database affects database operations. Large or frequently changed assets increase the size of the primary database, write-ahead log, replication stream, and backups. PostgreSQL uses TOAST(opens in a new tab) to compress and store large bytea values outside the main table rows. These values still affect database writes, replication, vacuuming, backups, and restores.

For larger or frequently changed files, use object storage, typically S3 object storage. Object storage supports concurrent uploads and downloads.

S3-compatible storage requires a separate backup routine:

• Document Engine stores files by their secure hash algorithm (SHA) checksums. In most cases, a daily incremental backup is sufficient.
• If your backup solution doesn’t coordinate point-in-time backups across storage types, schedule the asset storage backup immediately after the PostgreSQL database backup. This helps keep the backups consistent.

Configuring storage backends

Set the ASSET_STORAGE_BACKEND environment variable to choose a backend. The following sections explain each supported value.

Built-in asset storage

Set ASSET_STORAGE_BACKEND to built-in to use the built-in asset storage.

When deploying with Helm, use the pspdfkit.storage.assetStorageBackend value.

S3 object storage

Set ASSET_STORAGE_BACKEND to s3. Other configuration options depend on whether you’re using AWS S3 or an S3-compatible storage provider.

Here are the available parameters as Helm values(opens in a new tab):

assetStorage:
  # `ASSET_STORAGE_BACKEND`: `built-in`, `s3` or `azure`
  assetStorageBackend: s3
  # S3 backend storage settings, in case `pspdfkit.storage.assetStorageBackend` is set to `s3`
  s3:
    # `ASSET_STORAGE_S3_ACCESS_KEY_ID`
    accessKeyId: "<...>"
    # `ASSET_STORAGE_S3_SECRET_ACCESS_KEY`
    secretAccessKey: "<...>"
    # `ASSET_STORAGE_S3_BUCKET`
    bucket: "<...>"
    # `ASSET_STORAGE_S3_REGION`
    region: "<...>"
    # `ASSET_STORAGE_S3_HOST`
    #host: "os.local"
    # `ASSET_STORAGE_S3_PORT`
    port: 443
    # `ASSET_STORAGE_S3_SCHEME`
    #scheme: "https://"
    # External secret name
    #externalSecretName: ""

AWS S3

For AWS S3, set ASSET_STORAGE_S3_BUCKET and ASSET_STORAGE_S3_REGION.

Document Engine checks for AWS credentials in this order:

• ASSET_STORAGE_S3_ACCESS_KEY_ID and ASSET_STORAGE_S3_SECRET_ACCESS_KEY
• IAM Role for Service Accounts(opens in a new tab)
• ECS Task Role(opens in a new tab)
• EC2 Instance Role(opens in a new tab)

Use role-based permissions instead of static credentials when your platform supports them.

AWS S3 bucket and key policy

If you’re using AWS S3, the AWS Identity and Access Management (IAM) identity used by Document Engine needs the following permissions:

• s3:ListBucket on the configured bucket
• s3:PutObject on all objects in the bucket (<bucket-arn>/*)
• s3:GetObjectAcl on all objects in the bucket (<bucket-arn>/*)
• s3:GetObject on all objects in the bucket (<bucket-arn>/*)
• s3:DeleteObject on all objects in the bucket (<bucket-arn>/*)

If you use server-side encryption with Key Management Service, enable these actions on the encryption key:

• kms:Decrypt
• kms:Encrypt
• kms:GenerateDataKey

Other S3-compatible storage providers

For an object storage provider other than Amazon S3(opens in a new tab), set ASSET_STORAGE_S3_ACCESS_KEY_ID and ASSET_STORAGE_S3_SECRET_ACCESS_KEY. You can also configure these options:

• ASSET_STORAGE_S3_HOST — Host name of the storage service
• ASSET_STORAGE_S3_PORT — Port used to access the storage service. The default port is 443
• ASSET_STORAGE_S3_SCHEME — URL scheme used when accessing the service, either http:// or https://. The default is https://

For more details about using Google Cloud Storage(opens in a new tab) as the storage backend, take a look at the Google Cloud Storage interoperability(opens in a new tab) guide.

Required APIs for S3-compatible providers

For providers other than Amazon S3, Document Engine’s S3 backend uses the APIs listed below for runtime storage operations. These APIs cover object reads, writes, copies, metadata checks, deletions, and multipart uploads. Runtime storage doesn’t require bucket-listing or access control list (ACL) APIs. The broader Amazon S3 IAM policy above includes permissions such as s3:ListBucket and s3:GetObjectAcl. Those permissions apply to AWS-specific or optional bucket checks and ACL inspection, not the core runtime path.

Example S3-compatible providers

Common S3-compatible providers include Garage, Ceph, and SeaweedFS.

Use the standard S3-compatible configuration shown above and set the provider-specific values for:

• ASSET_STORAGE_S3_HOST
• ASSET_STORAGE_S3_PORT
• ASSET_STORAGE_S3_SCHEME
• ASSET_STORAGE_S3_ACCESS_KEY_ID
• ASSET_STORAGE_S3_SECRET_ACCESS_KEY
• ASSET_STORAGE_S3_BUCKET
• ASSET_STORAGE_S3_REGION, if your provider requires it

For example, in docker-compose.yml:

environment:
  ASSET_STORAGE_BACKEND: s3
  ASSET_STORAGE_S3_BUCKET: <bucket name>
  ASSET_STORAGE_S3_ACCESS_KEY_ID: <access key>
  ASSET_STORAGE_S3_SECRET_ACCESS_KEY: <secret access key>
  ASSET_STORAGE_S3_SCHEME: http://
  ASSET_STORAGE_S3_HOST: <s3-compatible host>
  ASSET_STORAGE_S3_PORT: 9000
  ASSET_STORAGE_S3_REGION: us-east-1

Check your storage provider’s documentation for its endpoint, port, credential, and region requirements.

Azure Blob Storage

To configure Azure Blob Storage as the default asset store, set ASSET_STORAGE_BACKEND to azure in your Document Engine configuration.

Set these configuration options:

• AZURE_STORAGE_ACCOUNT_NAME
• AZURE_STORAGE_ACCOUNT_KEY
• AZURE_STORAGE_DEFAULT_CONTAINER

You can provide a connection string with AZURE_STORAGE_ACCOUNT_CONNECTION_STRING instead of setting AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY.

These are the corresponding Helm values(opens in a new tab):

assetStorage:
  # `ASSET_STORAGE_BACKEND`: `built-in`, `s3` or `azure`
  assetStorageBackend: azure
  # Azure backend storage settings, in case `pspdfkit.storage.assetStorageBackend` is set to `azure`
  azure:
    # `AZURE_STORAGE_ACCOUNT_NAME`
    accountName: "<...>"
    # `AZURE_STORAGE_ACCOUNT_KEY`
    accountKey: "<...>"
    # `AZURE_STORAGE_DEFAULT_CONTAINER`
    container: "<...>"
    # `AZURE_STORAGE_ACCOUNT_CONNECTION_STRING`, takes priority over `accountName` and `accountKey`
    #connectionString: ""
    # `AZURE_STORAGE_API_URL` for custom endpoints
    #apiUrl: ""
    # External secret name
    #externalSecretName: ""

Azurite

Use Azurite(opens in a new tab) with Document Engine in development and test environments when ASSET_STORAGE_BACKEND is set to azure.

Set AZURE_STORAGE_API_URL to the address of your Azurite deployment.

Azurite is Microsoft’s open source emulator for Azure Blob Storage. Use it in development and test environments to align them with production.

To run Azurite in Docker, run the following commands:

docker pull mcr.microsoft.com/azure-storage/azurite
docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite

Configure Document Engine to use the Azurite default storage account. For details, refer to Microsoft’s Azurite default storage account documentation(opens in a new tab).

Add these settings to your Docker Compose file:

environment:
  AZURE_STORAGE_ACCOUNT_NAME: devstoreaccount1
  AZURE_STORAGE_ACCOUNT_KEY: Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
  AZURE_STORAGE_DEFAULT_CONTAINER: pspdfkit-dev
  AZURE_STORAGE_API_URL: http://localhost:10000/devstoreaccount1

Upload timeouts for object storage

All upload operations to S3-compatible object storage and Azure Blob Storage have a timeout of 30 seconds.

Working with multiple storage backends

This section covers per-document storage, fallbacks, per-document uploads, and document-level asset migration.

Set ASSET_STORAGE_BACKEND to choose the default backend. To upload documents to other backends, enable those backends as fallbacks in your Document Engine configuration.

Enabling fallbacks for asset storage

To use multiple asset stores in your Document Engine instance, configure the main asset store by setting ASSET_STORAGE_BACKEND to built-in, azure, or s3.

Document Engine uses the backend set in ASSET_STORAGE_BACKEND for all documents by default. To use another backend for a document, enable it as a fallback.

For example, if ASSET_STORAGE_BACKEND is set to azure, Document Engine stores documents and assets in Azure Blob Storage by default. You can store a specific document in AWS S3 when uploading the document. To do this, enable S3 as a fallback asset store.

To enable fallback asset storage, set ENABLE_ASSET_STORAGE_FALLBACK to true. After that, enable the specific fallbacks you want by setting any of the following to true:

• ENABLE_ASSET_STORAGE_FALLBACK_POSTGRES
• ENABLE_ASSET_STORAGE_FALLBACK_S3
• ENABLE_ASSET_STORAGE_FALLBACK_AZURE

In addition to enabling the specific fallback, you also need to set the relevant configuration options for every backend you enable. For example, if you enable S3 as an asset fallback, you need to provide the relevant configuration for S3, including the default S3 bucket.

Uploading documents to different storage

Set the storage option when uploading a document to choose a backend. The backend must be the default or an enabled fallback. For available options, refer to the API reference.

This request uploads a document to a specified S3 bucket:

// With Document Engine running on `http://localhost:5000`.

curl -X POST http://localhost:5000/api/documents \
    -H "Authorization: Token token=secret" \
    -H "Content-Type: multipart/form-data" \
    -F 'file=@blank.pdf' \
    -F 'storage={
      "backend": "s3",
      "bucketName": "a-different-bucket-from-default-s3-bucket",
      "bucketRegion": "us-west-2"
      }'

Migrating a document’s assets to different storage

To migrate a document’s assets and layers to another backend, send a request to /api/documents/{documentId}/migrate_assets.

This curl request migrates a document’s assets to built-in storage:

// With Document Engine running on `http://localhost:5000`.

curl -X POST http://localhost:5000/api/documents/{documentID}/migrate_assets \
    -H "Authorization: Token token=secret" \
    -H "Content-type: application/json" \
    -d '{
      "storage": {
        "backend": "built-in"
      }
      }'

Learn more about migrating assets from our API reference.

Multiple S3 buckets

You can upload or migrate documents to multiple S3 buckets. The instance role for your Document Engine nodes or the configured AWS credentials must have access to each bucket.

Migrate between default storage backends

Use the migration commands below to move assets between default storage backends. Migrations don’t delete files from the original backend.

Migrations run incrementally. You can pause and resume a migration, and you can run it while Document Engine is online. For large collections, consider running the migration during periods of low load.

Before migrating, set ENABLE_ASSET_STORAGE_FALLBACK to true and enable the required fallback backends. Document Engine can then serve assets that haven’t migrated yet.

After migrating all documents, set ENABLE_ASSET_STORAGE_FALLBACK to false. Fallback storage slightly reduces asset-fetch performance.

Check the Storage tab in the Document Engine dashboard to see how many documents use each backend. You can also set these options in the Helm chart values(opens in a new tab).

Migrating to S3 from built-in storage

Migrate from built-in asset storage to S3 with these steps:

1. Set ENABLE_ASSET_STORAGE_FALLBACK to true.
2. Enable the built-in database storage as a fallback by setting ENABLE_ASSET_STORAGE_FALLBACK_POSTGRES to true.
3. Set ASSET_STORAGE_BACKEND to s3 and configure the rest of the S3 options.
4. Run the migration script by executing pspdfkit assets:migrate:from-built-in-to-s3 in the Document Engine container.
   • If you use docker-compose, run the following command in the directory where you have your docker-compose.yml file: docker-compose run pspdfkit pspdfkit assets:migrate:from-built-in-to-s3.
   • If you don’t use docker-compose, first find the name of the Document Engine container using docker ps -a. This will list all running containers and their names. Then run the following command, replacing <container name> with the actual Document Engine container name: docker exec <container name> pspdfkit assets:migrate:from-built-in-to-s3.
5. When all your documents have been migrated, set ENABLE_ASSET_STORAGE_FALLBACK back to false.

Migrating to built-in storage from S3

Migrate from S3 asset storage to built-in storage with these steps:

1. Set ENABLE_ASSET_STORAGE_FALLBACK to true.
2. Enable S3 asset storage as a fallback by setting ENABLE_ASSET_STORAGE_FALLBACK_S3 to true.
3. Set ASSET_STORAGE_BACKEND to built-in. Don’t remove any of the S3 configuration options.
4. Run the migration script by executing pspdfkit assets:migrate:from-s3-to-built-in in the Document Engine container.
   • If you use docker-compose, run the following command in the directory where you have your docker-compose.yml file: docker-compose run pspdfkit pspdfkit assets:migrate:from-s3-to-built-in.
   • If you don’t use docker-compose, first find the name of the Document Engine container using docker ps -a. This will list all running containers and their names. Then run the following command, replacing <container name> with the actual Document Engine container name: docker exec <container name> pspdfkit assets:migrate:from-s3-to-built-in.
5. When all your documents have been migrated, set ENABLE_ASSET_STORAGE_FALLBACK back to false and remove all S3 configuration options.

Migrating to and from Azure Blob Storage

Document Engine doesn’t support batch migrations to or from Azure Blob Storage. You can migrate one document’s assets at a time. Refer to migrating a document’s assets to different storage.

Working with existing file storage in your infrastructure

Document Engine can use an existing file store if it exposes PDF files through HTTP endpoints. To connect the file store, add documents from a URL.

Treat each PDF URL as a permalink. Document Engine fetches the file when needed and keeps only a local cached copy, which can expire at any time.

For lower latency, run Document Engine instances and the file store on the same physical or virtual network.

Since version 2019.4, you can edit a document with a remote URL. Document Engine stores the resulting PDF in one of the supported storage backends. To copy the transformed file back to your file store, fetch the transformed file first.

If your file store requires authentication, use an internal proxy. Point the document URL to the proxy endpoint. The proxy can handle authentication and redirect requests to the PDF file. For details and sample code, refer to the guide to accessing external storage with credentials or tokens.