ChunkerOperator

Splits document text into overlapping chunks for embedding generation and vector storage.

• Short Name: chunker
• Category: Functional

Overview

ChunkerOperator breaks extracted document text into smaller, contextually coherent pieces
(chunks) that fit within embedding model token limits. It supports three strategies — simple
fixed-size, semantic content-aware, and hybrid Docling-based — and can optionally summarise
each chunk using an LLM.

Chunking can run locally (default) or be offloaded to a remote docling-serve instance for
distributed workloads.

Key Features

• Three chunking strategies: simple, semantic, hybrid
• Optional LLM-based chunk summarisation (LiteLLM or WatsonX)
• Remote chunking via docling-serve API
• Configurable chunk size and overlap
• Optionally retains the original content column alongside chunks
• Outputs chunk_sequence_number and start_index for precise retrieval

Operator Configuration

{
  "type": "chunker",
  "name": "chunk_documents",
  "config": {
    "chunk_type": "hybrid",
    "chunk_size": 512,
    "chunk_overlap": 128,
    "provider": "docling_library",
    "doc_column": "content",
    "retain_original_content": false
  },
  "depends_on": ["extract_documents"]
}

Parameters

Common parameters

Semantic chunking parameters

Remote (docling-serve) chunking parameters

Summarisation parameters (summarization object)

Output Columns

All original columns (except content when retain_original_content: false) are preserved.

Each item in chunked_content:
- chunk — text content of the chunk
- start_index — position in the original text
- summary — present only when summarisation is enabled

Examples

Example 1 — Simple chunking

{
  "type": "chunker",
  "name": "simple_chunker",
  "config": {
    "chunk_type": "simple",
    "chunk_size": 1000,
    "chunk_overlap": 200,
    "doc_column": "content"
  },
  "depends_on": ["extract"]
}

Example 2 — Semantic chunking

{
  "type": "chunker",
  "name": "semantic_chunker",
  "config": {
    "chunk_type": "semantic",
    "semantic_embeddings_model": "nomic-embed-text",
    "breakpoint_threshold_type": "percentile",
    "breakpoint_threshold_amount": 95.0,
    "doc_column": "content"
  },
  "depends_on": ["extract"]
}

Example 3 — Hybrid chunking with summarisation

{
  "type": "chunker",
  "name": "hybrid_chunker",
  "config": {
    "chunk_type": "hybrid",
    "provider": "docling_library",
    "chunk_size": 512,
    "chunk_overlap": 50,
    "docling_tokenizer": "sentence-transformers/all-MiniLM-L6-v2",
    "summarization": {
      "provider": "litellm",
      "provider_config": {
        "model_id": "llama3.2",
        "api_base": "http://localhost:11434/v1",
        "api_key": "<ollama>"
      },
      "summary_sentences": 2,
      "summary_max_words": 30
    }
  },
  "depends_on": ["extract"]
}

Example 4 — Remote chunking via docling-serve

{
  "type": "chunker",
  "name": "remote_chunker",
  "config": {
    "chunk_type": "hybrid",
    "chunk_size": 512,
    "chunk_overlap": 128,
    "provider": "docling_serve",
    "provider_config": {
      "api_base": "https://docling-serve.example.com",
      "timeout": 120,
      "max_retries": 3
    }
  },
  "depends_on": ["extract"]
}

Troubleshooting

Architecture

Strategy selection guide

Chunk size guidance

• 512–1024 tokens: Better for precise retrieval
• 1024–2048 tokens: Better for preserving context
• Overlap: Recommend 10–25% of chunk size (e.g. 128 for 512)

Typical pipeline position

Ingest → Extract → Chunker → Embeddings → VectorDB

Sample flow

See sample_flows/use_cases/invoice_processing.json.