HashiCorp Vault Integration

HashiCorp Vault enables secure credential resolution for Docling Pipelines without storing sensitive credentials in flow JSON files, source control, or database storage.

Overview

In standard flow execution, operators often require sensitive configuration parameters such as database passwords, API tokens, and cloud storage access keys. Hardcoding these credentials inside flow definitions presents security risks.

The Vault integration replaces cleartext credentials in flow definitions with immutable vault:// URIs. When a pipeline runs:
1. Flow JSON stores only reference pointers (e.g., vault://hashicorp/docpipe/embeddings#api_key).
2. Flow validation sanitizes and verifies reference formats without exposing or leaking credentials.
3. The orchestrator resolves references dynamically from Vault just prior to operator initialization in memory.
4. Resolved values remain strictly transient and are never written back to disk, database, or API response payloads.

Architecture

The following diagram illustrates the lifecycle of a vault:// reference from authoring to operator execution:

graph TD
    UI[Web UI Canvas / Flow JSON] -->|vault:// URI| API[REST API / Storage]
    API -->|Authoring Flow intact| Val[Flow Validator]
    Val -->|Dry-run sanitization| Orch[Python Orchestrator]
    Orch -->|Resolve URIs| SecProv[SecretProvider / HashiCorpVault]
    SecProv -->|AppRole or Token Auth| Vault[(HashiCorp Vault)]
    Vault -->|Decrypted Secret| SecProv
    SecProv -->|In-memory Config| Op[Operator Instance]
    Op -->|Execute transform| Svc[External Service / DB]

Key Components

• SecretProvider / HashiCorpVault (src/docpipe/integrations/secrets/): Interacts with Vault's KV v2 engine, handles authentication (Token or AppRole), auto-refreshes tokens, and auto-deserializes JSON strings into dictionaries.
• VaultInitializer (src/docpipe/integrations/secrets/vault_initializer.py): Initializes configured secret providers on server and CLI startup.
• FlowValidator (src/docpipe/core/orchestration/flow_validator.py): Sanitizes vault:// placeholders during schema validation so schema type-check rules pass without network dependency on Vault.
• PythonOperatorExecutor (src/docpipe/core/orchestration/python/python_operator_executor.py): Recursively walks operator configuration trees and replaces all vault:// paths with resolved values before instantiating operator classes.
• FlowMapper (src/docpipe/api/dto/mappers/flow_mapper.py): Preserves vault:// strings unchanged in API responses so authoring workflows round-trip cleanly without masking them as ********.

URI Format and Patterns

URI syntax

Every Vault secret reference follows the standard format:

vault://<provider>/<secret_path>#<field_key>

Pattern A — individual field references

Use this pattern to store secrets as distinct key-value pairs in Vault. Recommended for granular access control and standard credential fields:

Vault Secret Data (docpipe/s3):

{
  "auth_token": "placeholder-token-value"
}

Flow Configuration:

{
  "type": "embeddings",
  "name": "embed",
  "config": {
    "provider": "openai",
    "provider_config": {
      "api_key": "vault://hashicorp/docpipe/openai#api_key"  # pragma: allowlist secret
    }
  }
}

Pattern B — whole credential JSON blobs

For connectors requiring complex or multiline credentials (such as Google Cloud Service Account keys, complete cloud credential sets, or SharePoint configuration), store the whole credential JSON as a single field value in Vault.

Docpipe automatically parses the resolved string into a dictionary if the target field expects an object or dictionary:

Vault Secret Data (docpipe/s3_full):

{
  "connection_creds": "{\"key_id\": \"sample-id\", \"region\": \"us-east-1\"}"
}

Flow Configuration:

{
  "type": "storage_output",
  "name": "export_s3",
  "config": {
    "destination_type": "s3",
    "destination_config": {
      "bucket_name": "processed-output",
      "credentials": "vault://hashicorp/docpipe/s3_full#connection_creds"
    }
  }
}

Supported Operators and Fields

All sensitive credential fields across docpipe operators support vault:// URIs:

Configuration

Configuration file (docling-pipelines-config.yaml)

Enable Vault in docling-pipelines-config.yaml under the secrets block:

secrets:
  vault:
    enabled: true
    provider: hashicorp
    url: "http://127.0.0.1:8200"
    mount_point: "secret"
    auth_method: "approle"      # "approle" (production) or "token" (dev/testing)
    role_id: ""                 # set via VAULT_ROLE_ID environment variable
    secret_id: ""               # set via VAULT_SECRET_ID environment variable
    token: ""                   # set via VAULT_TOKEN environment variable (if auth_method is token)
    timeout: 30

Authentication methods

1. AppRole Authentication (Production Recommended)

AppRole authentication is intended for automated services. Docpipe uses a role_id and secret_id to authenticate against Vault, obtain a transient client token, and automatically manage token renewal.

1. Enable AppRole in Vault:
   bash vault auth enable approle
2. Create a policy (docpipe-read-policy.hcl):
   hcl path "secret/data/docpipe/*" { capabilities = ["read"] }
bash vault policy write docpipe-read docpipe-read-policy.hcl
3. Create AppRole binding:
   bash vault write auth/approle/role/docpipe-role \ secret_id_ttl=0 \ token_num_uses=0 \ token_ttl=1h \ token_max_ttl=24h \ token_policies="docpipe-read"
4. Fetch Role ID and generate Secret ID:
   bash vault read auth/approle/role/docpipe-role/role-id vault write -f auth/approle/role/docpipe-role/secret-id

2. Token Authentication (Development / Local Testing)

For local testing or development, pass a static developer or root token directly:

export VAULT_AUTH_METHOD=token
export VAULT_TOKEN="root-dev-token"

Environment variables

Environment variables override values configured in docling-pipelines-config.yaml:

Web UI and Properties Panel Usage

The Docling Pipelines web canvas provides integrated support for Vault credential references via the shared VaultInput component.

Interactive properties panel controls

When editing any sensitive parameter in an operator's properties panel:

1. Direct Entry vs Vault Reference Toggle:
2. Click the Vault icon or mode toggle next to the input field to switch from direct text entry to Vault reference mode.
3. Simplified Reference Input:
4. In Vault mode, the prefix vault://hashicorp/ is managed automatically. You only type the path and key: docpipe/opensearch#password.
5. Round-Trip Safety:
6. Existing vault:// URIs in loaded flows are automatically detected and displayed in Vault mode.
7. Saving the flow preserves the exact vault://... URI in the flow definition without converting it into a plaintext masked password.

Validation and Masking Lifecycle

To prevent credential leakage while maintaining robust flow authoring and execution:

1. Schema Validation (FlowValidator):
2. When validating flow JSON, fields marked as sensitive that contain vault:// strings are replaced in-memory with dummy mock values matching the required schema type (e.g. valid dummy strings).
3. If Vault is not configured in the active environment, the validator issues a clear warning (Config key '...' references unregistered vault provider '...') instead of failing execution unconditionally.
4. REST API Masking (FlowMapper):
5. Direct cleartext credentials stored in flows are masked as ******** when retrieved through REST endpoints (GET /api/v1/flows).
6. vault:// URI references are recognized as safe pointer strings and are returned unmasked, enabling UI canvas editors to load and edit the flow without corrupting the reference.
7. Logging Hygiene:
8. Resolved secret values are never printed in application logs. Operator executor logs record only the configuration path names being resolved (e.g. ['provider_config.username', 'provider_config.password']), never the resolved values.

Integration Guide

For end-to-end setup instructions, AppRole configuration walkthroughs, and security details, see the Vault Integration Guide.

Troubleshooting