Docling Pipelines Pipeline User Guide: Complete Setup and Execution

This comprehensive guide walks you through setting up and executing a complete Docling Pipelines pipeline from document ingestion to vector storage in OpenSearch.

Quick Reference

⚠️ CRITICAL: Working Directory Requirements

All docling-pipelines commands MUST be run from the project root directory (docling-pipelines/).

Correct:

```bash

From project root (docling-pipelines/)

docling-pipelines --flow-file sample_flows/use_cases/invoice_processing.json
```

Incorrect:

```bash

From docpipe directory - WILL FAIL with ModuleNotFoundError

cd src/docpipe
docling-pipelines --flow-file ... # ERROR: No module named 'docpipe'
```

Why: The PYTHONPATH must point to src as the source root. Running from subdirectories breaks Python imports.

1. Introduction

Quick Start

For fast setup (5 minutes), see QUICKSTART.md which provides automated installation and your first pipeline execution.

This guide provides comprehensive details for:
- Manual installation and configuration
- Understanding each component
- Troubleshooting and debugging

Prerequisites

Before proceeding, ensure you have:
- Python 3.12 installed
- 5GB+ available disk space
- Internet connection for downloading dependencies

Quick setup: Use the automated script from QUICKSTART.md:

./scripts/setup_docling_pipelines_environment.sh

# Combine options
./scripts/setup_docling_pipelines_environment.sh --interactive --models granite4

Available Options

Option	Description
`--interactive`	Enable interactive mode with prompts for each step
`--models MODEL1,MODEL2`	Specify Ollama models (comma-separated). Default: granite4,llama3.2,nomic-embed-text
`--skip-ollama`	Skip Ollama installation and setup
`--skip-opensearch`	Skip OpenSearch installation and setup
`--skip-python`	Skip Python environment setup
`--help`	Show help message with all options

What the Script Installs

Python Environment:

Verifies Python 3.12 is installed
Installs uv package manager
Creates virtual environment in .venv
Installs all project dependencies

Ollama (for LLM operations):

Installs Ollama server
Starts Ollama service on http://localhost:11434
Downloads specified models (default: granite4, llama3.2, nomic-embed-text)

OpenSearch (for vector storage):

Installs Podman or uses existing Docker
Installs podman-compose
Starts OpenSearch on http://localhost:9200
Starts OpenSearch Dashboards on http://localhost:5601
Default credentials: admin / <YOUR_OPENSEARCH_PASSWORD> (configured in docker/docker-compose.opensearch.yml)

After Setup Completes

The script creates two files:

.docpipe_setup_config - Configuration settings
docpipe_setup.log - Detailed setup log

Next steps:

Set PYTHONPATH from project root:

bash export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

Warning: This must be run from the project root directory (docling-pipelines), not from subdirectories. The PYTHONPATH must point to src as the source root for Python imports to work correctly.

Activate the virtual environment:

bash # From project root source .venv/bin/activate

Verify installation:

bash docling-pipelines --help

Run your first flow:
bash docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

Troubleshooting the Setup Script

Script fails with "Python 3.12 not found":

Install Python 3.12 manually (see Prerequisites)
Run the script again

Ollama fails to start:

Check if port 11434 is already in use: lsof -i :11434
Start manually: ollama serve

OpenSearch fails to start:

Check if ports 9200 or 5601 are in use
View logs: podman-compose -f docker/docker-compose.opensearch.yml logs
Ensure you're in the project root directory

Permission denied errors:

Make script executable: chmod +x scripts/setup_docling_pipelines_environment.sh
Some operations may require sudo (script will prompt)

Want to start fresh?

# Stop services
podman-compose -f docker/docker-compose.opensearch.yml down
pkill -f "ollama serve"

# Remove configuration
rm .docpipe_setup_config docpipe_setup.log

# Run setup again
./scripts/setup_docling_pipelines_environment.sh

Manual Setup

If you prefer manual control or the automated script doesn't work for your environment, follow the detailed manual setup instructions below.

What is Docling Pipelines?

Docling Pipelines is a modular, operator-based data processing framework designed for building flexible data pipelines. It enables you to:

Ingest documents from various sources
Extract structured content using AI-powered tools
Chunk documents for optimal processing
Generate embeddings for semantic search
Store vectors in OpenSearch for similarity search

What This Guide Covers

This guide demonstrates the complete Ingest → Extract → Chunk → Embeddings → OpenSearch pipeline, which is the foundation for building Retrieval-Augmented Generation (RAG) systems and semantic search applications.

Pipeline Overview

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Ingest    │───▶│   Extract   │───▶│    Chunk    │───▶│ Embeddings  │───▶│ OpenSearch  │
│   Local     │    │   Docling   │    │   Hybrid    │    │   Ollama    │    │   Vector    │
│   Folder    │    │             │    │             │    │             │    │   Storage   │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

Prerequisites Overview

Before starting, you'll need:

Python 3.12 - Required for Docling Pipelines
uv package manager - Fast Python package management
Ollama - Local LLM server for embeddings
Podman or Docker - For running OpenSearch
Basic command-line knowledge - For running commands
ffmpeg - Required for audio/video processing (M4A, AAC, OGG, FLAC, MP4, AVI, MOV formats)

2. Prerequisites and Installation

Python 3.12 Requirement

Docling Pipelines requires Python 3.12. Check your version:

python --version
# Should output: Python 3.12.x

If you need to install Python 3.12:

macOS (using Homebrew):

brew install python@3.12

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install python3.12 python3.12-venv

Installing uv Package Manager

uv is a fast Python package manager that Docling Pipelines uses for dependency management.

Install uv:

curl -LsSf https://astral.sh/uv/install.sh | sh

Verify installation:

uv --version

Installing ffmpeg (Required for Audio/Video Processing)

ffmpeg is required for processing audio formats (M4A, AAC, OGG, FLAC) and all video formats (MP4, AVI, MOV) with Docling ASR.

macOS (using Homebrew):

brew install ffmpeg

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install ffmpeg

Linux (RHEL/CentOS/Fedora):

sudo dnf install ffmpeg

Verify installation:

ffmpeg -version

Note: If you only process WAV or MP3 audio files, ffmpeg is not required. However, for M4A, AAC, OGG, FLAC audio formats and all video formats, ffmpeg must be installed and available on your PATH.

Cloning and Setting Up the Project

1. Clone the repository:

git clone https://github.com/IBM/docling-pipelines.git
cd docling-pipelines

Note: If you already have the repository cloned, simply navigate to it:

bash cd docling-pipelines

2. Create virtual environment and install dependencies:

# From project root
uv sync --extra dev

This command:

Installs CPython 3.12.13 in a virtual environment (.venv/ at project root)
Installs all project dependencies
Installs development dependencies

Note: VLM, ASR, and Docling entity extraction require an additional install step.

If you intend to use VLM-based text extraction (text_extraction.provider_config.vlm_pipeline),
ASR transcription (text_extraction.provider_config.asr_pipeline), or Docling template-based
entity extraction (entity_extraction.provider: "docling"), install the vlm_asr optional
dependency group which provides docling[vlm,asr]:

bash uv sync --extra vlm_asr

3. Activate the virtual environment:

macOS/Linux:

# From project root
source .venv/bin/activate
pre-commit install

Windows:

.venv\Scripts\activate
pre-commit install

4. Configure environment variables:

# Copy environment template
cp .env.example .env

# The default values work for local development
# Edit .env if you need to customize settings

HashiCorp Vault (optional): If your organisation uses Vault for secret management, set secrets.vault.enabled: true in docling-pipelines-config.yaml and supply VAULT_ROLE_ID and VAULT_SECRET_ID as environment variables. See the Architecture Guide — HashiCorp Vault Integration for full details.

Verify Installation

# Check Docling Pipelines CLI is available
docling-pipelines --help

# List available operators
docling-pipelines --list-operators

3. Ollama Setup

Installing Ollama

Ollama provides local LLM inference for embeddings and entity extraction.

macOS:

brew install ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh

Verify installation:

ollama --version

Starting the Ollama Server

# Start Ollama server (runs in background)
ollama serve

The server will start on http://localhost:11434.

Downloading Models

Download the models you'll use for embeddings and extraction:

# Embedding model (required for embeddings operator)
ollama pull nomic-embed-text

# Alternative embedding models
ollama pull granite4
ollama pull mxbai-embed-large

# LLM models for entity extraction (optional)
ollama pull llama3.2
ollama pull llama3.1

Verify models are downloaded:

ollama list

Verifying Ollama is Running

# Check server status
curl http://localhost:11434/api/tags

# Should return JSON with list of available models

Expected response:

{
  "models": [
    {
      "name": "nomic-embed-text:latest",
      ...
    }
  ]
}

Troubleshooting Common Ollama Issues

Issue: Port 11434 already in use

# Find process using port 11434
lsof -i :11434

# Kill the process if needed
kill -9 <PID>

# Restart Ollama
ollama serve

Issue: Model download fails

# Check internet connection
# Try downloading again
ollama pull nomic-embed-text

# If still fails, check Ollama logs
ollama logs

4. OpenSearch Setup with Podman

Installing Podman

Podman is a container engine similar to Docker, used to run OpenSearch.

macOS:

brew install podman

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install podman

Linux (Fedora/RHEL):

sudo dnf install podman

Verify installation:

podman --version

Initialize Podman machine (macOS only):

podman machine init
podman machine start

Starting OpenSearch Using podman-compose

1. Install podman-compose:

pip install podman-compose

2. Start OpenSearch:

# From project root
podman-compose -f docker/docker-compose.opensearch.yml up -d

This starts:
- OpenSearch on http://localhost:9200
- OpenSearch Dashboards on http://localhost:5601

3. Wait for OpenSearch to start (30-60 seconds):

# Check if OpenSearch is ready
curl -u admin:<YOUR_OPENSEARCH_PASSWORD> http://localhost:9200

# Should return cluster information

Note: Replace <YOUR_OPENSEARCH_PASSWORD> with the password from docker/docker-compose.opensearch.yml.

Verifying OpenSearch is Running

# Check cluster health
curl -u admin:<YOUR_OPENSEARCH_PASSWORD> http://localhost:9200/_cluster/health?pretty

# Expected: "status": "green" or "yellow"

Check container status:

podman-compose -f docker/docker-compose.opensearch.yml ps

Accessing OpenSearch Dashboards

Open your browser to: http://localhost:5601

Login credentials:
- Username: admin
- Password: <YOUR_OPENSEARCH_PASSWORD>

Note: Use the password configured in docker/docker-compose.opensearch.yml.

Navigate to Dev Tools to run queries against your indexed documents.

Default Credentials and Security

Default credentials (for development only):
- Username: admin
- Password: <YOUR_OPENSEARCH_PASSWORD>

⚠️ Security Warning: The default password in the docker-compose file is for development only. For production:
- Change the OPENSEARCH_INITIAL_ADMIN_PASSWORD in docker-compose or use environment variables
- Enable SSL/TLS
- Configure proper authentication
- See Advanced Configuration Guide for production setup

Stopping and Cleaning Up OpenSearch

Stop OpenSearch:

podman-compose -f docker/docker-compose.opensearch.yml down

Remove volumes (deletes all data):

podman-compose -f docker/docker-compose.opensearch.yml down -v

View logs:

podman-compose -f docker/docker-compose.opensearch.yml logs

5. Creating Your First Flow

Testing with the Sample Flow

Before creating your own flow, test your setup using the provided sample flow.

The repository includes a complete, ready-to-run pipeline in sample_flows/quickstart/complete_pipeline_ollama.json that demonstrates the full document processing workflow:

Ingest → Reads documents from ./sample_documents
Extract → Extracts content using Docling
Chunk → Splits documents using hybrid chunking (512 tokens, 128 overlap)
Embed → Generates embeddings using Ollama
Store → Saves vectors to OpenSearch index sample-documents-index

To test your setup:

⚠️ Important: Commands must be run from the project root directory (docling-pipelines/), not from subdirectories.

Ensure you have sample documents in ./sample_documents/ directory (create it if needed)
Run the sample flow:
bash # From project root docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

Understanding Pipeline Output

Docling Pipelines provides clean, formatted console output showing pipeline progress in real-time.

Example output:

================================================================================
 FLOW: complete-document-pipeline
 Operators: 5
 Started: 2024-01-15 10:30:00
================================================================================

[ingest] Starting ingest_source...

================================================================================
 ingest (COMPLETED)
================================================================================
 Duration: 0.50s | Documents: 1 processed, 0 failed, 0 skipped

 Data Columns: 8 total (8 added by this operator)
   Added (8): id, name, path, size, created_time, modified_time, content, extension
================================================================================

[extract] Starting extract_operator...
[chunk] Starting chunker...
[embeddings] Starting embeddings...
[vectordb] Starting vectordb...

================================================================================
 FLOW EXECUTION SUMMARY
================================================================================
 Status: COMPLETED
 Total Duration: 15.30s
 Documents: 1 completed, 0 failed, 0 skipped (of 1 total)

 Operator Summary:
 Operator                       Status               Duration     Docs
 ------------------------------------------------------------------------------
 ingest                         COMPLETED            0.50s        1/1
 extract                        COMPLETED            3.20s        1/1
 chunk                          COMPLETED            1.10s        1/1
 embeddings                     COMPLETED            8.50s        1/1
 vectordb                       COMPLETED            2.00s        1/1
================================================================================

Output Features:
- Real-time operator progress with document counts
- Schema changes showing new columns added by each operator
- Operator-specific metrics (chunk counts, embedding dimensions, etc.)
- Final summary table with per-operator statistics

Controlling Log Verbosity

Control log output using the DS_LOG_LEVEL environment variable:

# Debug: Detailed information
export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

# Info: Standard output (default)
export DS_LOG_LEVEL=INFO
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

# Warning: Only warnings and errors
export DS_LOG_LEVEL=WARNING
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

# Error: Only errors
export DS_LOG_LEVEL=ERROR
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

Creating a Custom Flow

For detailed information on creating and configuring flows, see:

Flow Configuration Guide

This comprehensive guide covers:
- Flow structure and required fields
- Global configuration options
- Complete operator configuration examples
- Connecting operators
- Schema templates and best practices

6. Running the Pipeline

Setting PYTHONPATH

Critical: Set PYTHONPATH before running any Docling Pipelines commands:

# From project root
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

⚠️ Important: This must be run from the project root directory, not from subdirectories.

Activating the Virtual Environment

# From project root
source .venv/bin/activate

Executing the Flow

# From project root
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

Common CLI options:

# List available operators
docling-pipelines --list-operators

# List operators with detailed information
docling-pipelines --list-operators --verbose

# Validate flow without executing
docling-pipelines --flow-file my_flow.json --validate

# Run with debug logging
DS_LOG_LEVEL=DEBUG docling-pipelines --flow-file my_flow.json

Understanding the Output

The pipeline will:
1. Load and validate the flow configuration
2. Execute operators in dependency order
3. Display real-time progress for each operator
4. Show final execution summary with statistics

Success indicators:
- All operators show COMPLETED status
- No errors in the output
- Final summary shows all documents processed

7. Verification and Testing

Verify Ollama Setup

# Check Ollama is running
curl http://localhost:11434/api/tags

# List downloaded models
ollama list

# Test embedding generation
curl http://localhost:11434/api/embeddings -d '{
  "model": "nomic-embed-text",
  "prompt": "test"
}'

Verify OpenSearch Setup

# Check cluster health
curl -u admin:<YOUR_OPENSEARCH_PASSWORD> http://localhost:9200/_cluster/health?pretty

# List indices
curl -u admin:<YOUR_OPENSEARCH_PASSWORD> http://localhost:9200/_cat/indices?v

# Query sample index
curl -u admin:<YOUR_OPENSEARCH_PASSWORD> \
  "http://localhost:9200/sample-documents-index/_search?pretty" \
  -H 'Content-Type: application/json' \
  -d '{"query": {"match_all": {}}, "size": 1}'

Using OpenSearch Dashboards

Open browser to: http://localhost:5601
Login with credentials: admin / <YOUR_OPENSEARCH_PASSWORD>
Navigate to Dev Tools
Run queries:

GET sample-documents-index/_search
{
  "query": {
    "match_all": {}
  }
}

8. Troubleshooting

Common Issues

Issue: ModuleNotFoundError

# Solution: Set PYTHONPATH from project root
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

# Verify you're in project root
pwd  # Should end with /docling-pipelines

Issue: Ollama connection refused

# Check if Ollama is running
curl http://localhost:11434/api/tags

# If not running, start it
ollama serve

# Verify models are downloaded
ollama list

Issue: OpenSearch connection refused

# Check OpenSearch status
podman-compose -f docker/docker-compose.opensearch.yml ps

# View logs
podman-compose -f docker/docker-compose.opensearch.yml logs

# Restart if needed
podman-compose -f docker/docker-compose.opensearch.yml restart

Issue: Flow validation failed

# Validate flow to see specific errors
docling-pipelines --flow-file my_flow.json --validate

# Check flow syntax and operator configurations
# See Flow Configuration Guide for correct format

Debug Logging

Enable debug logging for detailed troubleshooting:

export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file sample_flows/quickstart/complete_pipeline_ollama.json

For comprehensive troubleshooting, see TROUBLESHOOTING.md.

9. Next Steps

Explore More Operators

# List all available operators
docling-pipelines --list-operators

# Get detailed operator information
docling-pipelines --list-operators --verbose

Available operator categories:
- Ingest: IngestSourceOperator (filesystem, S3, IBM COS, SharePoint, OneDrive, Google Drive, Box, CSV, web)
- Extract: ExtractOperator (with multiple extraction providers)
- Quality: DocumentClassifier, LanguageDetect, Redaction, Dedup
- Functional: Chunker, Embeddings, Branching, Merge, NOOP
- VectorDB: VectorDBOperator (OpenSearch, Milvus)
- Storage: DocumentSetOperator

See Operator Reference for complete specifications.

Customize Your Pipeline

Copy the sample flow: cp sample_flows/quickstart/complete_pipeline_ollama.json my_flow.json
Edit the configuration: Change paths, chunk sizes, models, etc.
Run your custom flow: docling-pipelines --flow-file my_flow.json

For detailed flow configuration, see Flow Configuration Guide.

Performance Tuning

For production deployments and performance optimization, see Advanced Configuration Guide which covers:
- Job stats storage backends (DuckDB, PostgreSQL)
- Incremental metadata for avoiding reprocessing
- Distributed execution with Prefect work pools
- Production deployment patterns

Programmatic Usage

Use Docling Pipelines as a Python library in your applications:

from docpipe.lib.docpipe_flow_manager import DocpipeFlowManager

manager = DocpipeFlowManager(flow_file="my_flow.json")
result = manager.execute()

For complete Python API documentation, see Python API Guide.

Additional Resources

Documentation

QUICKSTART.md - Fast 5-minute setup
Flow Configuration Guide - Creating and configuring flows
Python API Guide - Programmatic usage
Advanced Configuration - Production deployment
Operator Reference - Complete operator specifications
TROUBLESHOOTING.md - Comprehensive troubleshooting
ARCHITECTURE.md - System design and architecture
CONTRIBUTING.md - Contributing guidelines

Examples

Sample Flows - Example flow configurations
Examples Directory - Code examples and patterns
Operator Examples - Detailed operator guides

Community

Issues: Report bugs and request features
Discussions: Ask questions and share ideas
Contributing: See CONTRIBUTING.md for guidelines