v2.0 IBM/docling-pipelines

Docling Pipelines Advanced Configuration Guide

This guide covers advanced configuration topics for production deployments, distributed execution, and performance optimization.

Job Stats Storage Configuration

Docling Pipelines supports pluggable job stats storage for job runs, node execution state, and micro-batch progress tracking.

Available Backends

  • In-memory - test and development scenarios
  • Filesystem storage - local file-backed persistence
  • DuckDB - embedded database for local development (no server required)
  • PostgreSQL - durable storage for concurrent and distributed execution

Backend Selection

Job-management components are wired through JobManagementFactory. Backend selection is controlled by environment variables and defaults.

The main user-facing configuration is controlled via environment variables:

job_management:
  framework:
    type: default
    config: {}
  store:
    type: filesystem
    config:
      base_dir: ./data/job_stats_store_data

This allows users to configure:

  • the job framework type via DOCPIPE_FRAMEWORK_TYPE environment variable
  • the job stats store backend via DOCPIPE_STORAGE_BACKEND environment variable
  • the job stats store runtime config via backend-specific environment variables
  • the flow repository via LOCAL_FLOWS_DIR environment variable

Common overrides include:

  • DOCPIPE_CONFIG_PATH
  • DOCPIPE_STORAGE_BACKEND
  • DOCPIPE_FRAMEWORK_TYPE
  • DOCPIPE_JOB_STATS_BASE_DIR
  • DOCPIPE_POSTGRES_HOST
  • DOCPIPE_POSTGRES_PORT
  • DOCPIPE_POSTGRES_DB
  • DOCPIPE_POSTGRES_USER
  • DOCPIPE_POSTGRES_PASSWORD

Effective precedence for job-management runtime selection is:

  1. explicit environment variables
  2. built-in defaults in JobManagementFactory

Filesystem Storage Guidance

Filesystem storage is useful for single-host execution and simple local testing.

Important requirements:

  • the job stats base directory must be writable
  • for distributed Prefect workers, the configured path must resolve to the same shared filesystem location for the submitter and workers
  • local-only paths on the submitter machine are not sufficient for distributed workers
  • when JSON storage is selected from config, worker propagation resolves base_dir to an absolute path before injecting it into the worker environment

DuckDB Guidance

DuckDB is an embedded analytical database that provides persistent storage without requiring a server. It's ideal for local development and testing.

Use DuckDB when:

  • you want persistent storage without running a database server
  • you need SQL query capabilities for analyzing job statistics
  • you're working in a single-host environment
  • you want faster queries than JSON storage

Configuration example:

job_management:
  store:
    type: duckdb
    config:
      database_path: ./data/duckdb/job_stats.duckdb

Important notes:

  • DuckDB stores all data in a single file
  • Not recommended for distributed execution (use PostgreSQL instead)
  • Provides SQL interface for querying job statistics
  • See examples/duckdb_job_stats/ for usage examples

PostgreSQL Guidance

PostgreSQL is the recommended backend for multi-process and distributed execution because it provides durable shared storage and stronger concurrency behavior than file-backed JSON storage.

Use PostgreSQL when:

  • multiple workers need to update job stats concurrently
  • workers do not share a reliable filesystem path
  • you need a single durable backend for job status APIs
  • you want workers on different containers, pods, or machines to share a single backend without filesystem coupling

Metadata Aggregation Maintenance

Node stats are aggregated on the read path, not in the storage adapter. When operators add new metadata fields, maintainers must review DEFAULT_STRATEGIES and update it if the field should not use the default LAST aggregation behavior.

See docs/internals/NODE_METADATA_AGGREGATION_STRATEGY.md for the maintainer workflow.

Distributed Execution and Work Pool Environment Inheritance

For distributed Prefect execution, work pool runtime configuration is modeled in work_pool_config.py and applied by WorkPoolAdapter.

Important behavior:

  • worker env values configured directly in the work pool take highest precedence
  • if job-management env values are omitted from the work pool config, workers inherit the submitter's effective job-management configuration
  • the inherited effective configuration is resolved from:
  • submitter environment variables
  • code defaults

This makes it possible to configure job management via environment variables per environment or per deployment.

For full distributed execution examples and work-pool-specific configuration, see DISTRIBUTED_EXECUTION_GUIDE.md.

Incremental Metadata Configuration

Docling Pipelines supports incremental processing to avoid reprocessing unchanged input data. Incremental metadata stores processing state such as file identity and modification information so ingest operators can determine whether an item is new, changed, or already processed.

Configuration

Use the incremental_metadata section in docling-pipelines-config.yaml:

incremental_metadata:
  storage:
    type: "filesystem"  # Options: filesystem, postgresql
    config:
      base_dir: "./data"
      lock_timeout: 30.0

This configuration is the single source of truth for incremental metadata backend selection and runtime settings.

Storage Backends

JSON

JSON storage is the default backend. It is file-based, easy to inspect, and suitable for development or smaller single-host deployments.

incremental_metadata:
  storage:
    type: "json"
    config:
      base_dir: "./data/incremental_metadata"
      lock_timeout: 30.0

Use JSON when:

  • you want a simple file-based backend
  • you are running locally or on a single host
  • you want metadata files that are easy to inspect during debugging

Parquet

Parquet storage uses a columnar file format and is better suited to larger datasets or analytics-oriented workflows.

incremental_metadata:
  storage:
    type: "parquet"
    config:
      base_dir: "./data/incremental_metadata"
      lock_timeout: 30.0

Use Parquet when:

  • you need more efficient columnar storage than JSON
  • you want better compression for larger metadata volumes
  • you are operating on a shared filesystem but do not need a database backend

PostgreSQL

PostgreSQL is the recommended backend for production deployments that require stronger concurrency behavior and durable centralized storage.

incremental_metadata:
  storage:
    type: "postgresql"
    config:
      base_dir: "./data"
      lock_timeout: 30.0
  postgres:
    host: "${POSTGRES_HOST:-localhost}"
    port: 5432
    database: "${POSTGRES_DB:-docpipe}"
    user: "${POSTGRES_USER:-docpipe_user}"
    password: "${POSTGRES_PASSWORD}"
    schema: "incremental_metadata"

Use PostgreSQL when:

  • multiple workers need to access incremental metadata concurrently
  • submitters and workers do not share a reliable local filesystem path
  • you need durable centralized state for distributed or production execution

Environment Variables for Sensitive Data

Use environment variable substitution in docling-pipelines-config.yaml for credentials and deployment-specific values.

incremental_metadata:
  storage:
    type: "postgresql"
    config:
      base_dir: "${DATA_DIR:-./data}"
      lock_timeout: "${LOCK_TIMEOUT:-30.0}"
  postgres:
    host: "${INCR_META_DB_HOST:-localhost}"
    port: "${INCR_META_DB_PORT:-5432}"
    database: "${INCR_META_DB_NAME:-docpipe}"
    user: "${INCR_META_DB_USER:-docpipe_user}"
    password: "${INCR_META_DB_PASSWORD}"
    schema: "${INCR_META_DB_SCHEMA:-incremental_metadata}"

Guidance:

  • use ${VAR_NAME} for required secrets
  • use ${VAR_NAME:-default} for optional values with safe defaults
  • do not commit real credentials into version control

See docling-pipelines-config.yaml.example for complete backend examples and environment variable patterns.

Execution Models

Docling Pipelines uses Prefect as its orchestration engine and supports two Prefect execution modes:

Ephemeral Mode (Default)

By default, Docling Pipelines runs Prefect in ephemeral mode with a temporary in-memory server. This mode is ideal for:

  • Development and testing
  • Small to medium workloads (< 1000 documents)
  • Single-machine processing
  • Quick prototyping and learning

How it works:

  • Prefect server runs temporarily in-memory
  • No external Prefect server setup required
  • Automatic cleanup after execution
  • Zero configuration needed

Usage:

# Simply run your flow - Prefect ephemeral mode is automatic
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

Distributed Execution with Prefect Work Pools (Optional)

For production workloads and large-scale processing, Docling Pipelines supports Prefect's distributed execution using work pools and workers.

When to use:

  • Processing large document collections (1000+ documents)
  • Horizontal scaling across multiple machines
  • Production deployments with high availability
  • Resource-intensive operations requiring distributed processing

Deployment options:

  1. Local POC: Test distributed patterns with Prefect server and workers on a single machine
  2. Docker Compose: Multi-worker setup with containerized Prefect infrastructure

For complete setup instructions, work pool configuration, and deployment guides, see:

Prefect Distributed Execution Guide

This guide covers:

  • Prefect server and work pool setup
  • Worker deployment for different environments
  • Batch storage strategies (inline and local filesystem)
  • Docker deployment configurations
  • Troubleshooting and performance tuning