Docling Pipelines Troubleshooting Guide

This comprehensive guide helps you diagnose and resolve common issues when working with Docling Pipelines pipelines.

Quick Diagnostics

Run these commands to quickly check your Docling Pipelines environment:

# 1. Check Python version (must be 3.12)
python3.12 --version

# 2. Check if virtual environment is activated
which python  # Should point to .venv/bin/python

# 3. Verify PYTHONPATH is set correctly
echo $PYTHONPATH  # Should include src

# 4. Check Ollama service
curl http://localhost:11434/api/tags

# 5. Check OpenSearch service
curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

# 6. Check Milvus service
python3 -c "from pymilvus import connections; connections.connect(host='localhost', port=19530); print('Milvus: Connected')"

# 7. Verify you're in the project root
pwd  # Should end with /docling-pipelines

# 8. List available Ollama models
ollama list

# 9. Check OpenSearch cluster health
curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/health?pretty"

Health Check Script:

#!/bin/bash
echo "=== Docling Pipelines Environment Health Check ==="
echo ""
echo "1. Python Version:"
python3.12 --version || echo "❌ Python 3.12 not found"
echo ""
echo "2. Virtual Environment:"
which python | grep -q ".venv" && echo "✅ Virtual environment active" || echo "❌ Virtual environment not active"
echo ""
echo "3. PYTHONPATH:"
[[ -n "$PYTHONPATH" ]] && echo "✅ PYTHONPATH set: $PYTHONPATH" || echo "❌ PYTHONPATH not set"
echo ""
echo "4. Ollama Service:"
curl -s http://localhost:11434/api/tags > /dev/null && echo "✅ Ollama running" || echo "❌ Ollama not responding"
echo ""
echo "5. OpenSearch Service:"
curl -s -u admin:MyStrongPass123! http://localhost:9200 > /dev/null && echo "✅ OpenSearch running" || echo "❌ OpenSearch not responding"
echo ""
echo "6. Milvus Service:"
python3 -c "from pymilvus import connections; connections.connect(host='localhost', port=19530)" 2>/dev/null && echo "✅ Milvus running" || echo "❌ Milvus not responding"

Common Issues by Category

Installation & Setup

Issue: Python 3.12 Not Found

Symptoms:

bash: python3.12: command not found

Solutions:

macOS:

brew install python@3.12

Ubuntu/Debian:

sudo apt update
sudo apt install python3.12 python3.12-venv python3.12-dev

Fedora/RHEL:

sudo dnf install python3.12 python3.12-devel

Verify installation:

python3.12 --version

Issue: uv Package Manager Not Found

Symptoms:

bash: uv: command not found

Solution:

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Add to PATH (add to ~/.bashrc or ~/.zshrc)
export PATH="$HOME/.cargo/bin:$PATH"

# Reload shell
source ~/.bashrc  # or source ~/.zshrc

# Verify installation
uv --version

Issue: Virtual Environment Creation Failed

Symptoms:

Error: Failed to create virtual environment

Solutions:

1. Ensure Python 3.12 is installed:

python3.12 --version

1. Remove existing .venv and recreate:

# From project root
rm -rf .venv
uv venv --python python3.12

1. Install dependencies:

# From project root
uv sync --extra dev

Issue: fasttext-wheel Build Failure on macOS 15+ (Sequoia / Tahoe)

Symptoms:

× Failed to build `fasttext-wheel==0.9.2`
...
fatal error: 'istream' file not found
error: command '/usr/bin/c++' failed with exit code 1

Cause:

fasttext-wheel 0.9.2 compiles a C++ extension from source. On macOS 15+ with clang 17, the
compiler no longer resolves the C++ standard library headers at the deployment target hardcoded
by fasttext-wheel (-mmacosx-version-min=11.0). No pre-built binary is available for arm64.

Solution:

Set these environment variables before running uv sync:

CPLUS_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include \
CXXFLAGS="-isysroot $(xcrun --show-sdk-path) -mmacosx-version-min=14.0" \
uv sync --extra dev

To avoid setting these every time, add them to a .envrc in the project root (if you use
direnv) or to your ~/.zshrc:

export CPLUS_INCLUDE_PATH=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1:/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
export CXXFLAGS="-isysroot $(xcrun --show-sdk-path) -mmacosx-version-min=14.0"

Verify Xcode CLT is installed:

xcode-select --print-path
# Expected: /Library/Developer/CommandLineTools

xcrun --show-sdk-path
# Expected: /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk

If xcode-select prints nothing, install the tools first:

xcode-select --install

Issue: Permission Denied During Setup

Symptoms:

Permission denied: './scripts/setup_docling_pipelines_environment.sh'

Solution:

chmod +x scripts/setup_docling_pipelines_environment.sh
./scripts/setup_docling_pipelines_environment.sh

Issue: Distributed workers show inconsistent or missing job status

Symptoms:

Job run exists on the submitter but worker updates do not appear in job status APIs

Likely cause:
- Filesystem job stats storage is configured with a local path that workers cannot access
- submitter and workers resolve different filesystem paths
- worker environment does not receive the same job stats backend configuration

Solutions:
1. Check job management configuration via environment variables and confirm the selected backend matches your deployment model.
2. For distributed execution, prefer PostgreSQL job stats storage.
3. If using filesystem storage, configure a shared filesystem path visible to both submitter and workers.
4. Ensure worker environments inherit the same effective backend configuration and connection settings.
5. If needed, override config explicitly with DOCPIPE_STORAGE_BACKEND, DOCPIPE_FRAMEWORK_TYPE, DOCPIPE_JOB_STATS_BASE_DIR, and PostgreSQL env variables.
6. Review docs/integrations/prefect/DISTRIBUTED_EXECUTION_GUIDE.md for distributed storage guidance.

Issue: Job status is stored, but aggregated node metadata looks wrong

Symptoms:

Counters are too low or too high
Lists are overwritten instead of combined
Batch progress looks incorrect in aggregated results

Likely cause:
- a new operator metadata field was added without reviewing aggregation behavior
- the field was left on the default LAST strategy when it should use SUM, UNION, WEIGHTED_AVERAGE, or another explicit strategy

Solutions:
1. Review DEFAULT_STRATEGIES in strategies.py.
2. Add explicit mappings for newly introduced metadata fields when needed.
3. Add or update tests covering multi-batch aggregation behavior.
4. See docs/internals/NODE_METADATA_AGGREGATION_STRATEGY.md for maintainer guidance.

Issue: Job run ends in crashed or canceled state during batch execution

Symptoms:

Batch work starts, but the final job state becomes CRASHED or CANCELED unexpectedly

Likely cause:
- the outer Prefect flow finished before submitted batch futures completed
- task-runner shutdown canceled in-flight batch work before final job state was recorded cleanly

Solutions:
1. Confirm the execution path waits for submitted batch work before the outer flow exits.
2. Check PrefectEngine behavior when debugging batch failures.
3. Validate that job-management terminal-state updates are still reached on failure paths.
4. Prefer PostgreSQL storage in concurrent/distributed environments to reduce ambiguity in final state updates.

Ollama Issues

Issue: Ollama Connection Refused

Error Code: OLLAMA_CONNECTION_FAILED

Symptoms:

ConnectionError: Failed to connect to Ollama at http://localhost:11434
Error code: ollama_connection_failed

Diagnosis:

# Check if Ollama is running
curl http://localhost:11434/api/tags

Solutions:

1. Start Ollama server:

ollama serve

1. Run in background:

ollama serve > /dev/null 2>&1 &

1. Check if port 11434 is in use:

lsof -i :11434
# If another process is using it, kill it or change Ollama port

1. Verify Ollama installation:

which ollama
ollama --version

Issue: Ollama Model Not Found

Error Code: OLLAMA_MODEL_NOT_FOUND

Symptoms:

Error: Model 'granite4' not found
Error code: ollama_model_not_found

Diagnosis:

# List installed models
ollama list

Solutions:

1. Pull the required model:

# For LLM operations
ollama pull granite4

# For embeddings
ollama pull nomic-embed-text

# For entity extraction
ollama pull llama3.2

1. Verify model is available:

ollama list | grep granite4

1. Check model size before pulling:

# granite4: ~2.5GB
# llama3.2: ~2GB
# nomic-embed-text: ~274MB

Issue: Ollama Using Too Much Memory

Symptoms:

• System slowdown
• Out of memory errors
• Ollama process consuming >8GB RAM

Solutions:

1. Use smaller models:

# Instead of granite4 (2.5GB), use smaller variants
ollama pull granite4:3b  # Smaller version

# For embeddings, nomic-embed-text is already small (274MB)
ollama pull nomic-embed-text

1. Limit Ollama memory usage:

# Set environment variable before starting Ollama
export OLLAMA_MAX_LOADED_MODELS=1
ollama serve

1. Stop unused models:

# Ollama automatically unloads models after 5 minutes of inactivity
# To force unload, restart Ollama
pkill -f "ollama serve"
ollama serve

Issue: Slow Ollama Model Downloads

Symptoms:

• Model download taking >30 minutes
• Download speed <1MB/s

Solutions:

1. Download smaller models first:

# Start with the smallest model
ollama pull nomic-embed-text  # Only 274MB, ~30 seconds

# Then larger models
ollama pull granite4  # 2.5GB, ~5 minutes

1. Check internet connection:

# Test download speed
curl -o /dev/null http://speedtest.wdc01.softlayer.com/downloads/test10.zip

1. Use a different mirror (if available):

# Check Ollama documentation for mirror options

WatsonX Issues

Issue: WatsonX Authentication Failed

Error Code: WATSONX_AUTH_FAILED

Symptoms:

AuthenticationError: Failed to authenticate with WatsonX
Error: Invalid API key or credentials

Diagnosis:

# Check if environment variables are set
echo $WATSONX_API_KEY
echo $WATSONX_CONTAINER_ID

Solutions:

1. Set required environment variables:

export WATSONX_API_KEY="your-api-key"   #pragma: allowlist secret
export WATSONX_CONTAINER_ID="your-project-or-space-id"

1. Verify API key is valid:

# Test authentication with curl
curl -X POST "https://us-south.ml.cloud.ibm.com/ml/v1/text/generation?version=2023-05-29" \
  -H "Authorization: Bearer ${WATSONX_API_KEY}" \
  -H "Content-Type: application/json"

1. Check container ID format:

# Project ID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
# Space ID format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

1. Verify container kind:

# Set container kind if using space instead of project
export WATSONX_CONTAINER_KIND="space"  # or "project" (default)

Issue: WatsonX Model Not Found

Error Code: WATSONX_MODEL_NOT_FOUND

Symptoms:

Error: Model 'ibm/granite-13b-chat-v2' not found or not accessible

Solutions:

1. Verify model is available in your WatsonX instance:

Check WatsonX.ai console for available models in your project/space.

1. Use correct model identifier:

{
  "entity_extraction": {
    "provider": "watsonx",
    "provider_config": {
      "model_id": "ibm/granite-13b-chat-v2",
      "api_key": "${WATSONX_API_KEY}",
      "container_id": "${WATSONX_CONTAINER_ID}"
    }
  }
}

1. Check model access permissions:

Ensure your API key has access to the specified model in the project/space.

Issue: WatsonX Connection Timeout

Symptoms:

TimeoutError: Request to WatsonX timed out after 120 seconds

Solutions:

1. Increase timeout in provider config:

{
  "entity_extraction": {
    "provider": "watsonx",
    "provider_config": {
      "api_key": "${WATSONX_API_KEY}",
      "container_id": "${WATSONX_CONTAINER_ID}",
      "timeout": 300
    }
  }
}

1. Check network connectivity:

# Test connection to WatsonX endpoint
curl -I https://us-south.ml.cloud.ibm.com

1. Verify API base URL:

# Set correct region endpoint
export WATSONX_API_BASE_URL="https://us-south.ml.cloud.ibm.com"
# Or use eu-de, eu-gb, jp-tok, etc.

Issue: WatsonX Rate Limiting

Symptoms:

Error: Rate limit exceeded (429 Too Many Requests)

Solutions:

1. Reduce parallel workers:

{
  "max_workers": 1,
  "entity_extraction": {"provider": "watsonx"}
}

1. Add retry logic in provider config:

{
  "entity_extraction": {
    "provider": "watsonx",
    "provider_config": {
      "max_retries": 3,
      "retry_delay": 2
    }
  }
}

1. Check WatsonX quota limits:

Review your WatsonX plan limits and usage in the IBM Cloud console.

OpenSearch Issues

Issue: OpenSearch Connection Refused

Error Code: OPENSEARCH_CONNECTION_FAILED

Symptoms:

ConnectionError: Failed to connect to OpenSearch at http://localhost:9200
Error code: opensearch_connection_failed

Diagnosis:

# Check if OpenSearch is running
curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

Solutions:

1. Start OpenSearch using podman-compose:

podman-compose -f docker/docker-compose.opensearch.yml up -d

1. Or using docker-compose:

docker-compose -f docker/docker-compose.opensearch.yml up -d

1. Wait for startup (30-60 seconds):

# Check logs
podman-compose -f docker/docker-compose.opensearch.yml logs -f opensearch-node

# Wait for "Node started" message

1. Verify cluster health:

curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/health?pretty"

Issue: OpenSearch Index Creation Failed

Error Code: OPENSEARCH_INDEX_ERROR

Symptoms:

Error: Failed to create index 'my-index'
Error code: opensearch_index_error

Diagnosis:

# Check if index already exists
curl -u admin:MyStrongPass123! "http://localhost:9200/_cat/indices?v"

Solutions:

1. Delete existing index and recreate:

# Delete index
curl -X DELETE -u admin:MyStrongPass123! "http://localhost:9200/my-index"

# Verify deletion
curl -u admin:MyStrongPass123! "http://localhost:9200/_cat/indices?v"

1. Check index settings in flow configuration:

{
  "type": "vectordb",
  "name": "opensearch_store",
  "config": {
    "provider": "opensearch",
    "index_name": "my-index",  // Must be lowercase, no spaces
    "vector_dimension": 768,  // Must match embedding model dimension
    "provider_config": {
        "host": "localhost",
        "port": 9200,
        "engine": "nmslib",  // Valid: nmslib, faiss, lucene
        "algorithm": "hnsw",
        "space_type": "l2"
    }
  }
}

1. Verify dimension matches embedding model:

# nomic-embed-text: 768 dimensions
# Check your model's documentation for correct dimension

Issue: OpenSearch Cluster Status Red

Symptoms:

{
  "status": "red",
  "unassigned_shards": 5
}

Diagnosis:

# Check cluster health
curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/health?pretty"

# Check shard allocation
curl -u admin:MyStrongPass123! "http://localhost:9200/_cat/shards?v"

Solutions:

1. Restart OpenSearch:

podman-compose -f docker/docker-compose.opensearch.yml restart

1. Check disk space:

df -h
# OpenSearch requires at least 10% free disk space

1. Reset cluster (WARNING: deletes all data):

podman-compose -f docker/docker-compose.opensearch.yml down -v
podman-compose -f docker/docker-compose.opensearch.yml up -d

Issue: OpenSearch Authentication Failed

Symptoms:

401 Unauthorized

Solutions:

1. Verify credentials:

# Default credentials from docker/docker-compose.opensearch.yml
# Username: admin
# Password: <your-opensearch-password>

curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

1. Check if credentials were changed:

# View docker/docker-compose.opensearch.yml
cat docker/docker-compose.opensearch.yml | grep -A 5 "OPENSEARCH_INITIAL_ADMIN_PASSWORD"

1. Reset admin password:

# Stop OpenSearch
podman-compose -f docker/docker-compose.opensearch.yml down

# Edit docker/docker-compose.opensearch.yml and change password
# Restart
podman-compose -f docker/docker-compose.opensearch.yml up -d

Milvus Issues

Issue: Milvus Connection Refused

Symptoms:
- Error: Connection refused or Failed to connect to Milvus
- Pipeline fails at VectorDB operator with Milvus adapter

Causes:
1. Milvus server not running
2. Wrong host/port configuration
3. Firewall blocking connection

Solutions:

1. Check if Milvus is running:
   ```bash
   # Docker
   docker ps | grep milvus

# Podman
podman ps | grep milvus
```

1. Start Milvus if not running:
   bash docker-compose -f docker-compose.milvus.yml up -d

2. Verify connection:
   bash python3 -c "from pymilvus import connections; connections.connect(host='localhost', port=19530); print('Connected successfully')"

3. Check configuration in flow:
   json { "type": "vectordb", "name": "milvus_store", "config": { "provider": "milvus", "index_name": "my_collection", "provider_config": { "host": "localhost", "port": 19530 } } }

Issue: Milvus Collection Creation Failed

Symptoms:
- Error: Collection already exists or Invalid collection name
- VectorDB operator fails during initialization

Causes:
1. Collection name conflicts
2. Invalid collection name format
3. Dimension mismatch with existing collection

Solutions:

1. Drop existing collection:
   bash python3 -c "from pymilvus import connections, utility; connections.connect(host='localhost', port=19530); utility.drop_collection('your_collection_name')"

2. Use unique collection names:

3. Avoid special characters
4. Use alphanumeric and underscores only

5. Keep names under 255 characters

6. Verify dimension matches embeddings:
   json { "type": "vectordb", "name": "milvus_collection", "config": { "provider": "milvus", "index_name": "my_collection", "vector_dimension": 768, "provider_config": { "index_type": "HNSW" } } }

Issue: Milvus Dimension Mismatch

Symptoms:
- Error: Dimension mismatch or Invalid vector dimension
- Insertion fails with dimension error

Causes:
1. Configured dimension doesn't match embedding dimension
2. Chunked embeddings not properly detected
3. Wrong embedding model used

Solutions:

1. Verify embedding dimension:
   python # Check your embedding model's output dimension # For Ollama nomic-embed-text: 768 # For sentence-transformers: varies by model

2. Let auto-detection work:

3. Remove vector_dimension from config to enable auto-detection

4. System will detect dimension from first batch

5. Match embedding model:
   json { "type": "embeddings", "name": "generate_embeddings", "config": { "provider": "litellm", "provider_config": { "model_id": "openai/nomic-embed-text", "api_base": "http://localhost:11434/v1" }, "embeddings_column": "embeddings" } }, { "type": "vectordb", "name": "store_in_milvus", "config": { "provider": "milvus", "index_name": "my_collection", "vector_dimension": 768, "provider_config": { "index_type": "HNSW" } } }

Issue: Milvus Index Build Failed

Symptoms:
- Error: Index build failed or Invalid index parameters
- Collection created but index not built

Causes:
1. Invalid index type for metric type
2. Insufficient memory for index building
3. Invalid index parameters

Solutions:

1. Use compatible index and metric combinations:
   json { "type": "vectordb", "name": "milvus_hnsw", "config": { "provider": "milvus", "index_name": "my_collection", "provider_config": { "index_type": "HNSW", "metric_type": "L2" } } }

2. Adjust index parameters:
   json { "type": "vectordb", "name": "milvus_tuned", "config": { "provider": "milvus", "index_name": "my_collection", "provider_config": { "index_type": "HNSW", "index_parameters": { "M": 16, "efConstruction": 256 } } } }

3. Check Milvus memory:
   bash docker stats milvus-standalone

Issue: Unable to Access Watsonx.data Milvus from macOS

Symptoms:
- Error: MilvusException: (code=2, message=Fail connecting to server on <host>:<port>)
- Connection fails when accessing watsonx.data Milvus from non-VM macOS system
- Works fine from Linux or VM environments

Cause:
- gRPC DNS resolution issue on macOS when connecting to watsonx.data Milvus
- The default DNS resolver doesn't work properly with GRPC certificates on macOS

Solution:

Set the GRPC_DNS_RESOLVER environment variable to "native" before running your Python script:

import os
os.environ["GRPC_DNS_RESOLVER"] = "native"

from pymilvus import connections
connections.connect(
    alias="<db_name>",
    host="<host>",
    port=443,
    secure=True,
    server_name="<host>",
    user="<username>",
    password="<password>"
)

Alternative: Set environment variable before execution:

export GRPC_DNS_RESOLVER="native"
docling-pipelines --flow-file sample_flows/vectordb/milvus_integration.json

Verification Steps:

1. Check if Milvus instance was deleted and recreated (GRPC server/host URL might have changed)
2. Verify the certificate is a GRPC certificate (not an https certificate)
3. Add the environment variable setting to your script or shell

Reference:
- IBM watsonx.data Documentation

Pipeline Execution

Issue: Flow Validation Failed

Error Code: FLOW_VALIDATION_FAILED

Symptoms:

FlowValidationException: Invalid Flow definition
Error code: flow_validation_failed

Common Validation Errors:

1. Missing Required Operator:

Error: The first operator in the flow must be an "Ingest data" operator
Message code: INGEST_OPERATOR_MISPLACED

Solution: Ensure your flow starts with an ingest operator:

{
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_1",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["sample_documents"]}
      }
    }
  ]
}

1. Disconnected Operators:

Error: Flow contains disconnected operators
Message code: DISJOINT_OPERATORS_DETECTED

Solution: Ensure all operators are connected via edges:

{
  "edges": [
    { "source": "ingest_1", "target": "extract_1" },
    { "source": "extract_1", "target": "chunk_1" }
  ]
}

1. Duplicate Operator Names:

Error: Same operator name(s) are used for multiple operators
Message code: OPERATOR_NAME_REPEATED

Solution: Use unique IDs for each operator node.

Issue: Flow Execution Failed

Error Code: FLOW_EXECUTION_FAILED

Symptoms:

FlowExecutionFailedException: Pipeline execution failed
Error code: flow_execution_failed

Diagnosis:

# Enable debug logging
export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file my_flow.json

Common Causes:

1. Input files not found:

# Verify input folder exists
ls -la sample_documents/

# Check file paths in flow configuration

1. Operator configuration invalid:

# Check operator parameters match requirements
# See operator documentation in README.md

1. Service unavailable (Ollama/OpenSearch):

# Verify services are running
curl http://localhost:11434/api/tags
curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

Issue: Prefect Flow Task Failed

Error Code: PREFECT_FLOW_TASK_FAILED

Symptoms:

PrefectFlowFailed: Task execution failed
Error code: prefect_flow_failed

Solutions:

1. Check Prefect logs:

# Logs are in the terminal output
# Look for the specific task that failed

1. Run with verbose logging:

export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file my_flow.json

1. Check operator-specific errors:

# Each operator logs its own errors
# Look for the operator name in the error message

Issue: Documents Being Skipped (Already Ingested) - CLI Execution

Applies To: docling-pipelines CLI execution only

Symptoms:

Documents are being skipped during ingestion with "already ingested" messages, even though they should be processed

Cause:

When using docling-pipelines, multiple flows with the same flow_name will generate the same job_id, causing incremental metadata conflicts. Documents processed in one pipeline may be incorrectly marked as processed in another pipeline with the same flow_name.

Diagnosis:

# Check if multiple flows use the same flow_name
grep -r "flow_name" sample_flows/*.json

# Example: Both flows below would generate the same job_id (UUID v5)
# Flow 1: flow_name "my-pipeline" → job_id "5f543fcf-ccf4-5537-a2a5-e6bdca4f6060"
# Flow 2: flow_name "my-pipeline" → job_id "5f543fcf-ccf4-5537-a2a5-e6bdca4f6060" (conflict!)

Solution:

1. Use unique flow_name for each logically different pipeline:

Before (Problematic):

// flow1.json
{
  "flow_name": "document-pipeline",
  "description": "PDF processing pipeline",
  "flow": [...]
}

// flow2.json  
{
  "flow_name": "document-pipeline",
  "description": "Word document processing pipeline",
  "flow": [...]
}

After (Fixed):

// flow1.json
{
  "flow_name": "pdf-document-pipeline",
  "description": "PDF processing pipeline",
  "flow": [...]
}

// flow2.json
{
  "flow_name": "word-document-pipeline",
  "description": "Word document processing pipeline",
  "flow": [...]
}

1. Force re-ingestion if you need to reprocess documents:

{
  "flow_name": "my-pipeline",
  "global_config": {
    "force_ingest": true
  },
  "flow": [...]
}

Prevention:

• Use descriptive, unique flow_name values for each logically different pipeline
• Document your flow_name conventions in your project
• The job_id is automatically generated from flow_name as a deterministic UUID v5 (36-character format)

Note: This issue does not apply when using DocpipeFlowManager programmatically with custom job_id parameters.

Related Documentation:
- See docs/guides/FLOW_CONFIGURATION_GUIDE.md section "Flow Identification: flow_name and job_id"

Import/Path Errors

Issue: ModuleNotFoundError

Symptoms:

ModuleNotFoundError: No module named 'docpipe_app'
ModuleNotFoundError: No module named 'common'
ModuleNotFoundError: No module named 'core'

Root Cause: PYTHONPATH not set correctly or running from wrong directory.

Solutions:

1. Set PYTHONPATH correctly (from project root):

# Navigate to project root
cd /path/to/docling-pipelines

# Set PYTHONPATH
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

# Verify
echo $PYTHONPATH

1. Ensure you're in the project root:

pwd  # Should end with /docling-pipelines

# If not, navigate to project root
cd /path/to/docling-pipelines

1. Activate virtual environment:

# From project root
source .venv/bin/activate

1. Add to shell profile for persistence:

# Add to ~/.bashrc or ~/.zshrc
echo 'export PYTHONPATH="/path/to/docling-pipelines/src:${PYTHONPATH}"' >> ~/.bashrc
source ~/.bashrc

Issue: Running from Wrong Directory

Symptoms:

FileNotFoundError: [Errno 2] No such file or directory: 'sample_flows/...'

Solution:

Always run docling-pipelines from the project root:

# ✅ CORRECT: From project root
cd /path/to/docling-pipelines
docling-pipelines --flow-file sample_flows/use_cases/invoice_processing.json

# ❌ INCORRECT: From subdirectory - will cause path resolution issues
cd some/subdirectory
docling-pipelines --flow-file ...  # WILL FAIL

Issue: File Paths in Flow Configuration

Symptoms:

FileNotFoundError: Input folder not found: ~/documents

Solutions:

1. Use absolute paths:

{
  "config": {
    "paths": "/Users/username/docling-pipelines/sample_documents"
  }
}

1. Use relative paths from project root:

{
  "config": {
    "paths": "sample_documents"
  }
}

1. Avoid using ~ or $HOME:

# ❌ Don't use
"paths": "~/documents"

# ✅ Use instead
"paths": "/Users/username/documents"

Operator-Specific Issues

Issue: Chunker Invalid Chunk Type

Error Message:

Invalid chunk_type: invalid_type
Message code: CHUNKER_INVALID_CHUNK_TYPE

Valid chunk types:

• simple: Fixed-size chunking
• semantic: Semantic similarity-based chunking
• hybrid: Docling-based chunking (requires ExtractOperator)

Solution:

{
  "type": "chunker",
  "name": "semantic_chunker",
  "config": {
    "chunk_type": "semantic", // Must be: simple, semantic, or hybrid
    "chunk_size": 512,
    "chunk_overlap": 50
  }
}

Issue: Semantic Chunking Requires Embeddings Support

Error Message:

This server does not support embeddings

or chunking operation fails when using chunk_type: "semantic".

Root Cause:

Semantic chunking requires Ollama embeddings support to calculate semantic similarity between text segments. This error occurs when:
- Ollama doesn't have an embeddings model installed
- Ollama version is outdated and lacks embeddings support
- The embeddings endpoint is not accessible

Solutions:

1. Install an embeddings model:

ollama pull granite4

1. Update Ollama to the latest version:

# macOS
brew upgrade ollama

# Linux
curl -fsSL https://ollama.com/install.sh | sh

1. Switch to simple chunking in your flow JSON:

{
  "type": "chunker",
  "name": "chunker_node",
  "config": {
    "chunk_type": "simple",  // Changed from "semantic"
    "chunk_size": 512,
    "chunk_overlap": 50
  }
}

Verification:

Test embeddings support:

curl http://localhost:11434/api/embeddings -d '{
  "model": "granite4",
  "prompt": "test"
}'

See Also:
- QUICKSTART.md for Ollama setup instructions
- Chunker documentation for chunking strategies

Issue: Audio/Video Processing Fails with ffmpeg Error

Error Message:

ffmpeg not found or codec error during audio/video processing

Symptoms:

• Audio files (M4A, AAC, OGG, FLAC) fail to process
• Video files (MP4, AVI, MOV) fail to process
• Error mentions "ffmpeg" or "codec"

Diagnosis:

# Check if ffmpeg is installed
ffmpeg -version

# Check if ffmpeg is in PATH
which ffmpeg  # macOS/Linux
where ffmpeg  # Windows

Solutions:

1. Install ffmpeg:

macOS:

brew install ffmpeg

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install ffmpeg

Linux (RHEL/CentOS/Fedora):

sudo dnf install ffmpeg

1. Verify installation:

ffmpeg -version

1. Add ffmpeg to PATH (if installed but not found):

macOS:

export PATH="/opt/homebrew/bin:$PATH"
# Add to ~/.zshrc or ~/.bash_profile for persistence

Linux:

export PATH="/usr/local/bin:$PATH"
# Add to ~/.bashrc for persistence

1. Restart terminal after installation

Note: WAV and MP3 audio files do not require ffmpeg and should work without it.

Supported Formats:
- Requires ffmpeg: M4A, AAC, OGG, FLAC (audio), MP4, AVI, MOV (video)
- No ffmpeg needed: WAV, MP3 (audio)

Issue: Embeddings Invalid Type

Error Message:

Invalid embeddings type: invalid_type
Message code: EMBEDDINGS_INVALID_TYPE

Valid embeddings types:

• huggingface: Native HuggingFace local or API embeddings
• litellm: LiteLLM-managed embedding providers (100+ including Ollama, HuggingFace API, OpenAI, etc.)
• watsonx: Native IBM watsonx.ai adapter

Solution:

{
  "operator_type": "EmbeddingsOperator",
  "operator_params": {
    "provider": "watsonx",
    "model_id": "ibm/slate-125m-english-rtrvr",
    "provider_config": {
      "api_key": "${WATSONX_API_KEY}",
      "api_base": "${WATSONX_API_BASE}",
      "container_id": "${WATSONX_CONTAINER_ID}",
      "container_kind": "project"
    }
  }
}

Important distinction:

• Native watsonx uses api_base and container_id
• LiteLLM watsonx uses LiteLLM-specific parameter names such as api_base and provider-specific IDs

Issue: SQL Filter Dropping Mandatory Columns

Error Message:

Mandatory features drop attempted: ['id', 'content']
Message code: DROPPING_MANDATORY_FEATURES

Mandatory columns that cannot be dropped:

• id: Document identifier
• content: Document content
• pages_processed: Processing metadata

Solution:

{
  "type": "sql_filter",
  "name": "filter_documents",
  "config": {
    "filter_criteria": "SELECT * FROM table WHERE length > 100"
    // Don't use: SELECT column1, column2 (missing id, content)
  }
}

Issue: Extract Operator Missing

Error Message:

Extract operator is either missing or not connected in the flow
Message code: EXTRACT_OPERATOR_MISSING

Solution: Add an extract operator before chunking:

{
  "flow": [
    {
      "type": "ingest_source",
      "name": "ingest_1",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["./sample_documents"]}
      }
    },
    {
      "type": "extract_operator",
      "name": "extract_1",
      "config": {
        "text_extraction": {
          "provider": "docling_library"
        }
      },
      "depends_on": ["ingest_1"]
    },
    {
      "type": "chunker",
      "name": "chunk_1",
      "config": {
        "chunk_type": "simple",
        "chunk_size": 512
      },
      "depends_on": ["extract_1"]
    }
  ]
}

### Document Set Storage Issues

**Problem:** Database file not created or cannot be found

**Symptoms:**
```text
FileNotFoundError: Database file not found at data/duckdb/document_sets.db

Solutions:
1. Verify workspace directory is correct:
bash pwd # Should be at project root

1. Check database path configuration:

python # Default path is relative to workspace data/duckdb/document_sets.db

1. Ensure directory exists:

bash mkdir -p data/duckdb

1. Check file permissions:
   bash ls -la data/duckdb/ chmod 755 data/duckdb

Problem: Document set name validation errors

Symptoms:

ValueError: Invalid document_set_name: must be alphanumeric with underscores/hyphens

Solutions:

1. Use valid characters only (alphanumeric, underscore, hyphen):

json "document_set_name": "my_documents_2024" // Valid "document_set_name": "my documents!" // Invalid

1. Avoid special characters and spaces
2. Keep names under 255 characters

Problem: Schema evolution errors when adding new columns

Symptoms:

DuckDBError: Column 'new_field' does not exist in table

Solutions:

1. Schema evolution is automatic - verify the operator is receiving the new column
2. Check PyArrow table schema before storage:

python print(table.schema)

1. Ensure column names are valid SQL identifiers
2. Restart pipeline if schema changes are not detected

Problem: Row count mismatch after storage

Symptoms:

AssertionError: Row count mismatch: expected 100, got 95

Solutions:

1. Check for data corruption in upstream operators
2. Verify no filtering is happening before storage
3. Enable debug logging to see detailed row counts:

bash DS_LOG_LEVEL=DEBUG docling-pipelines --flow-file your_flow.json

1. Inspect the PyArrow table before storage:
   python print(f"Table has {len(table)} rows")

Problem: Database locked or in use

Symptoms:

sqlite3.OperationalError: database is locked

Solutions:

1. Close other connections to the database
2. Check for running processes:

bash lsof data/duckdb/document_sets.db

1. Wait for concurrent operations to complete
2. Use connection pooling for concurrent access

Problem: Metadata not being stored correctly

Symptoms:

• Custom metadata fields missing from document set
• Metadata appears as empty dictionary

Solutions:

1. Verify metadata is valid JSON:

json "metadata": { "key": "value", "number": 123, "boolean": true }

1. Check metadata is not null or undefined
2. Ensure metadata values are JSON-serializable
3. Avoid nested objects deeper than 2-3 levels

Problem: Cannot query stored documents

Symptoms:

DuckDBError: Table 'document_set_name' does not exist

Solutions:

1. Verify document set was created successfully:

python from storage.duckdb_storage import DuckDBStorage storage = DuckDBStorage(db_path="data/duckdb/document_sets.db") tables = storage.list_tables() print(tables)

1. Check table name matches exactly (case-sensitive)
2. Ensure database file path is correct
3. Verify flow completed without errors

Problem: Performance issues with large document sets

Symptoms:

• Slow write operations
• High memory usage
• Database file growing very large

Solutions:

1. Use batch processing for large datasets
2. Consider partitioning large document sets
3. Monitor database file size:

bash du -h data/duckdb/document_sets.db

1. Optimize queries with proper indexing
2. Use retain_deleted_docs: false to avoid storing deleted documents

Problem: REST API document set operations failing

Symptoms:

HTTP 500: Internal server error when creating document set

Solutions:

1. Check API server logs for detailed errors
2. Verify database path is accessible by API server
3. Ensure proper permissions on database directory
4. Check API authentication if enabled
5. Verify request payload matches expected schema

Error Code Reference

Flow Validation and Execution Errors

Flow CRUD Operation Errors

Operator Errors

Integration Errors

Configuration and Service Errors

Debugging Guide

Enable Debug Logging

Set log level to DEBUG:

DS_LOG_LEVEL=DEBUG docling-pipelines --flow-file my_flow.json

Or set as environment variable:

export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file my_flow.json

Log levels:

• DEBUG: Detailed diagnostic information
• INFO: General informational messages (default)
• WARNING: Warning messages
• ERROR: Error messages
• CRITICAL: Critical errors

Log File Locations

Prefect logs:

# Logs are output to terminal by default
# To save to file:
docling-pipelines --flow-file my_flow.json > pipeline.log 2>&1

Operator logs:

# Each operator logs to the same output stream
# Look for operator name in log messages:
# [INFO] Operator: ingest_source_filesystem - Processing...

Service logs:

Ollama:

# If running in background
tail -f /tmp/ollama.log  # Or wherever you redirected output

OpenSearch:

podman-compose -f docker/docker-compose.opensearch.yml logs -f opensearch-node

Reading Log Output

Typical log format:

[2024-01-15 10:30:45] [INFO] [ingest_source_filesystem] Processing folder: sample_documents
[2024-01-15 10:30:46] [INFO] [ingest_source_filesystem] Found 5 files
[2024-01-15 10:30:47] [INFO] [extract_operator] Extracting content from file1.pdf
[2024-01-15 10:30:50] [ERROR] [extract_operator] Failed to extract: Connection refused

Key information:

• Timestamp: When the event occurred
• Level: Severity (INFO, WARNING, ERROR)
• Operator: Which operator generated the log
• Message: What happened

Common Log Patterns

Pattern 1: Connection Refused

[ERROR] Failed to connect to http://localhost:11434

Meaning: Service (Ollama/OpenSearch) not running
Action: Start the service

Pattern 2: File Not Found

[ERROR] FileNotFoundError: [Errno 2] No such file or directory: 'sample_documents'

Meaning: Input path doesn't exist
Action: Verify path, create directory if needed

Pattern 3: Model Not Found

[ERROR] Model 'granite4' not found in Ollama

Meaning: Ollama model not pulled
Action: Run ollama pull granite4

Pattern 4: Import Error

[ERROR] ModuleNotFoundError: No module named 'docpipe_app'

Meaning: PYTHONPATH not set correctly
Action: Set PYTHONPATH from project root

Using Python Debugger

Add breakpoints in code:

import pdb; pdb.set_trace()

Run with debugger:

python -m pdb -m docpipe.cli.docpipe_cli --flow-file my_flow.json

Verbose Operator Output

Enable verbose output using environment variable:

# Set debug level for detailed operator output
DS_LOG_LEVEL=DEBUG docling-pipelines --flow-file my_flow.json

Note: Individual operator verbosity settings (if supported) are separate from the global log level.

Service-Specific Troubleshooting

Ollama Troubleshooting

Check Ollama Status

# Check if running
curl http://localhost:11434/api/tags

# Check process
ps aux | grep ollama

# Check port
lsof -i :11434

Ollama Performance Issues

Slow inference:

# Use smaller models
ollama pull granite4:3b  # Instead of granite4:7b

# Check system resources
top  # Look for ollama process

High memory usage:

# Limit loaded models
export OLLAMA_MAX_LOADED_MODELS=1
ollama serve

Ollama Model Issues

List all models:

ollama list

Remove unused models:

ollama rm model-name

Update model:

ollama pull model-name  # Re-pulls latest version

OpenSearch Troubleshooting

Check OpenSearch Status

# Cluster health
curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/health?pretty"

# Node info
curl -u admin:MyStrongPass123! "http://localhost:9200/_nodes?pretty"

# Indices
curl -u admin:MyStrongPass123! "http://localhost:9200/_cat/indices?v"

OpenSearch Performance Issues

Slow indexing:

# Check cluster stats
curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/stats?pretty"

# Increase refresh interval
curl -X PUT -u admin:MyStrongPass123! "http://localhost:9200/my-index/_settings" \
  -H 'Content-Type: application/json' \
  -d '{"index": {"refresh_interval": "30s"}}'

High memory usage:

# Check heap usage
curl -u admin:MyStrongPass123! "http://localhost:9200/_cat/nodes?v&h=heap.percent,heap.current,heap.max"

# Adjust heap size in docker/docker-compose.opensearch.yml
# OPENSEARCH_JAVA_OPTS: "-Xms512m -Xmx512m"

OpenSearch Index Issues

Delete index:

curl -X DELETE -u admin:MyStrongPass123! "http://localhost:9200/my-index"

Reindex data:

curl -X POST -u admin:MyStrongPass123! "http://localhost:9200/_reindex" \
  -H 'Content-Type: application/json' \
  -d '{
    "source": {"index": "old-index"},
    "dest": {"index": "new-index"}
  }'

Check index mapping:

curl -u admin:MyStrongPass123! "http://localhost:9200/my-index/_mapping?pretty"

Milvus Troubleshooting

Check Milvus Status

# Check if running
docker ps | grep milvus

# Test connection
python3 -c "from pymilvus import connections; connections.connect(host='localhost', port=19530); print('Connected')"

# Check collections
python3 -c "from pymilvus import connections, utility; connections.connect(host='localhost', port=19530); print(utility.list_collections())"

Milvus Performance Issues

Slow search:

# Check collection stats
python3 -c "
from pymilvus import connections, Collection
connections.connect(host='localhost', port=19530)
collection = Collection('your_collection')
print(f'Entities: {collection.num_entities}')
print(f'Index: {collection.index().params}')
"

# Use faster index type
# HNSW is generally faster than IVF_FLAT for most use cases

High memory usage:

# Check Milvus container stats
docker stats milvus-standalone

# Reduce index parameters in flow config:
# "M": 8,  # Lower value = less memory
# "efConstruction": 128  # Lower value = less memory

Milvus Collection Issues

List collections:

python3 -c "from pymilvus import connections, utility; connections.connect(host='localhost', port=19530); print(utility.list_collections())"

Drop collection:

python3 -c "from pymilvus import connections, utility; connections.connect(host='localhost', port=19530); utility.drop_collection('collection_name')"

Check collection schema:

python3 -c "
from pymilvus import connections, Collection
connections.connect(host='localhost', port=19530)
collection = Collection('your_collection')
print(collection.schema)
"

Query collection:

python3 -c "
from pymilvus import connections, Collection
connections.connect(host='localhost', port=19530)
collection = Collection('your_collection')
collection.load()
results = collection.query(expr='pk >= \"\"', output_fields=['pk'], limit=5)
print(results)
"

Docling Troubleshooting

Docling Extraction Failures

Symptoms:

[ERROR] Failed to extract content from document.pdf

Solutions:

1. Check document format:

# Supported formats: PDF, DOCX, PPTX, HTML
file document.pdf  # Verify file type

1. Check document size:

# Large documents may timeout
ls -lh document.pdf

# Split large documents if needed

1. Verify Docling dependencies:

# From project root
uv sync --extra dev

Docling Timeout Issues

Increase timeout in operator configuration:

{
  "type": "extract_operator",
  "name": "extract_with_timeout",
  "config": {
    "text_extraction": {
      "provider": "docling_library"
    },
    "entity_extraction": {
      "provider": "none"
    },
    "timeout": 300  // Increase from default 60 seconds
  }
}

Watsonx Troubleshooting

Authentication Issues

Symptoms:
- 401 Unauthorized errors
- "Failed to fetch IAM access token"
- Token refresh failures

Solutions:

1. Verify API key:
   bash echo $WATSONX_API_KEY

2. Check environment variables:
   bash echo $WATSONX_API_BASE echo $WATSONX_CONTAINER_ID

3. Test connectivity:
   bash curl -X GET "${WATSONX_API_BASE}/ml/v1/foundation_model_specs?version=2023-05-29"

4. Enable debug logging:
   python import logging logging.basicConfig(level=logging.DEBUG)

Rate Limit Errors

Symptoms:
- 429 Too Many Requests errors
- "Exceeded limit of calls to endpoint"

Solutions:
- Reduce batch size in operator configuration
- Add delays between requests
- Request higher rate limits from IBM Cloud

Configuration Issues

Common Mistakes:
- Using api_base instead of url
- Using project_id instead of container_id
- Missing required parameters

Correct Configuration:

{
  "provider": "watsonx",
  "model_name": "ibm/slate-125m-english-rtrvr",
  "provider_config": {
    "api_key": "${WATSONX_API_KEY}",
    "api_base": "${WATSONX_API_BASE}",
    "container_id": "${WATSONX_CONTAINER_ID}"
  }
}

Environment Issues

Python Version Problems

Issue: Wrong Python version

# Check version
python --version  # Should be 3.12.x

# If wrong version, use python3.12 explicitly
python3.12 --version

Solution: Use correct Python version

# Recreate virtual environment with Python 3.12 (from project root)
rm -rf .venv
uv venv --python python3.12
source .venv/bin/activate
uv sync --extra dev

Virtual Environment Issues

Issue: Virtual environment not activated

# Check if activated
which python  # Should point to .venv/bin/python

Solution: Activate virtual environment

# From project root
source .venv/bin/activate

Issue: Virtual environment corrupted

# Remove and recreate (from project root)
rm -rf .venv
uv venv --python python3.12
source .venv/bin/activate
uv sync --extra dev

Dependency Conflicts

Issue: Package version conflicts

ERROR: Cannot install package-a and package-b because these package versions have conflicting dependencies

Solution: Update dependencies

# From project root
uv sync --extra dev --upgrade

Issue: Missing dependencies

ModuleNotFoundError: No module named 'package_name'

Solution: Install dependencies

# From project root
uv sync --extra dev

PYTHONPATH Issues

Issue: PYTHONPATH not set

ModuleNotFoundError: No module named 'docpipe_app'

Solution: Set PYTHONPATH

# From project root
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"

Verify PYTHONPATH is set correctly:

echo $PYTHONPATH

Expected output:

/Users/username/codebase/docling-pipelines/src:...

The output should show your project's src directory as the first entry. The actual path will match your project location.

Make permanent:

# Add to ~/.bashrc or ~/.zshrc
echo 'export PYTHONPATH="/path/to/docling-pipelines/src:${PYTHONPATH}"' >> ~/.bashrc
source ~/.bashrc

FAQ

General Questions

Q: What Python version is required?
A: Python 3.12 is required. Earlier or later versions are not supported.

Q: Can I use Docker instead of Podman?
A: Yes, Docker and Podman are interchangeable. Use docker-compose instead of podman-compose.

Q: Where are the log files stored?
A: Logs are output to the terminal by default. Redirect to a file: docling-pipelines --flow-file my_flow.json > pipeline.log 2>&1

Q: How do I stop all services?
A:

# Stop OpenSearch
podman-compose -f docker/docker-compose.opensearch.yml down

# Stop Ollama
pkill -f "ollama serve"

Q: Can I run multiple pipelines simultaneously?
A: Yes, but be aware of resource constraints (CPU, memory, GPU if using Ollama).

Installation Questions

Q: Do I need GPU for Ollama?
A: No, Ollama works on CPU. GPU acceleration is optional and improves performance.

Q: How much disk space is needed?
A: Minimum 5GB for models and data. Recommended 10GB+ for production use.

Q: Can I install on Windows?
A: Yes, but use WSL2 (Windows Subsystem for Linux) for best compatibility.

Q: What if I don't have admin/sudo access?
A: You can install in user space using uv and run services in containers without root.

Configuration Questions

Q: How do I change the OpenSearch password?
A: Edit docker/docker-compose.opensearch.yml and change OPENSEARCH_INITIAL_ADMIN_PASSWORD, then restart.

Q: Can I use a different port for Ollama?
A: Yes, set OLLAMA_HOST environment variable: export OLLAMA_HOST=0.0.0.0:11435

Q: How do I use a remote OpenSearch cluster?
A: Update the opensearch_url in your flow configuration to point to the remote cluster.

Q: Can I use different embedding models?
A: Yes, any Ollama model that supports embeddings. Update model_name in EmbeddingsOperator configuration.

Pipeline Questions

Q: How do I process multiple folders?
A: Use IngestSourceOperator with multiple paths in connection_params.paths (e.g. ["./folder1", "./folder2"]).

Q: Can I skip the chunking step?
A: Yes, but embeddings will be generated for entire documents, which may not be optimal for large documents.

Q: How do I handle different document types?
A: Use BranchingOperator to route different document types to different processing paths.

Q: What's the maximum document size?
A: Depends on available memory. Large documents (>100MB) should be split or processed with increased timeouts.

Performance Questions

Q: Why is my pipeline slow?
A: Common causes:

• Large documents
• Slow embedding generation (use smaller models)
• Network latency to OpenSearch
• Insufficient CPU/memory

Q: How can I speed up processing?
A:

• Use smaller embedding models
• Increase batch sizes
• Use GPU for Ollama (if available)
• Optimize chunk sizes

Q: How many documents can I process?
A: No hard limit, but performance depends on:

• Document size
• Available memory
• OpenSearch cluster capacity

Error Questions

Q: What does "Connection refused" mean?
A: The service (Ollama or OpenSearch) is not running. Start the service and try again.

Q: What does "Model not found" mean?
A: The Ollama model hasn't been pulled. Run ollama pull model-name.

Q: What does "Flow validation failed" mean?
A: Your flow configuration has errors. Check the error message for specific issues.

Q: What does "PYTHONPATH not set" mean?
A: The Python import path is not configured. Set PYTHONPATH from the project root.

Getting Help

Before Reporting Issues

1. Check this troubleshooting guide
2. Review the documentation:
3. README.md - Overview and operator reference
4. USER_GUIDE_PIPELINE_SETUP.md - Complete setup guide
5. QUICKSTART.md - Quick start guide

6. ARCHITECTURE.md - System architecture

7. Run diagnostics:

# Health check
curl http://localhost:11434/api/tags
curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

# Check logs
docling-pipelines --flow-file my_flow.json 2>&1 | tee debug.log

1. Try with debug logging:

export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file my_flow.json

Reporting Bugs

When reporting issues, include:

1. Environment information:

# Python version
python3.12 --version

# OS information
uname -a

# Package versions (from project root)
uv pip list

1. Error message:

# Full error output
docling-pipelines --flow-file my_flow.json 2>&1 | tee error.log

1. Flow configuration:

# Sanitized flow.json (remove sensitive data)
cat my_flow.json

1. Steps to reproduce:

2. What you were trying to do

3. Commands you ran

4. Expected vs actual behavior

5. Service status:

# Ollama
curl http://localhost:11434/api/tags

# OpenSearch
curl -u admin:MyStrongPass123! "http://localhost:9200/_cluster/health?pretty"

Community Resources

GitHub Issues:

• Search existing issues: https://github.com/your-org/docling-pipelines/issues
• Create new issue: https://github.com/your-org/docling-pipelines/issues/new

Documentation:

• Main README: README.md
• User Guide: USER_GUIDE_PIPELINE_SETUP.md
• Quick Start: QUICKSTART.md
• Architecture: ARCHITECTURE.md

Examples:

• Sample flows: sample_flows/
• Test flows: sample_flows/

Support Channels

For bugs and feature requests:

• GitHub Issues: https://github.com/your-org/docling-pipelines/issues

For questions and discussions:

• GitHub Discussions: https://github.com/your-org/docling-pipelines/discussions

For security issues:

• Email: security@your-org.com
• Do not post security issues publicly

Additional Resources

Useful Commands Reference

# Environment setup (from project root)
export PYTHONPATH="$(pwd)/src:${PYTHONPATH}"
source .venv/bin/activate

# Service management
ollama serve &
podman-compose -f docker/docker-compose.opensearch.yml up -d

# Pipeline execution
docling-pipelines --flow-file my_flow.json

# Debugging
export DS_LOG_LEVEL=DEBUG
docling-pipelines --flow-file my_flow.json 2>&1 | tee debug.log

# Service health checks
curl http://localhost:11434/api/tags
curl -u admin:${OPENSEARCH_PASSWORD} http://localhost:9200

# Cleanup
podman-compose -f docker/docker-compose.opensearch.yml down -v
pkill -f "ollama serve"

Quick Links

• Installation Guide: USER_GUIDE_PIPELINE_SETUP.md#2-prerequisites-and-installation
• Ollama Setup: USER_GUIDE_PIPELINE_SETUP.md#3-ollama-setup
• OpenSearch Setup: USER_GUIDE_PIPELINE_SETUP.md#4-opensearch-setup-with-podman
• Flow Configuration: USER_GUIDE_PIPELINE_SETUP.md#5-understanding-flow-configuration
• Operator Reference: README.md#operators

Last Updated: 2024-01-15
Version: 1.0.0