LanguageDetectionOperator

Detects the language of document text and appends ISO 639-1 language code and confidence score columns.

• Short Name: lang_detect
• Category: Quality

Overview

LanguageDetectionOperator runs each document's text through a pluggable language detection
adapter and adds two columns — lang_name (ISO 639-1 code) and lang_score (confidence
0.0–1.0) — to the output table. Documents that fail detection are either removed (when
filter_unknown_language: true) or marked as UNKNOWN with a score of 0.0.

Two adapters are bundled: fasttext (default, 176+ languages) and langdetect (55 languages,
no setup required). The adapter system is extensible — add a custom adapter by implementing
LanguageServicePort.

Key Features

• Two built-in providers: fasttext (176+ languages) and langdetect (55 languages)
• FastText model loaded as a singleton (thread-safe, memory-efficient)
• Configurable handling of unknown/failed detections
• Extensible via LanguageServicePort adapter interface

Operator Configuration

{
  "name": "detect_language",
  "type": "lang_detect",
  "config": {
    "doc_column": "content",
    "filter_unknown_language": false
  },
  "depends_on": ["extract"]
}

With langdetect provider:

{
  "name": "detect_language",
  "type": "lang_detect",
  "config": {
    "doc_column": "content",
    "language_provider": "langdetect",
    "filter_unknown_language": false
  },
  "depends_on": ["extract"]
}

Complete Pipeline Example

{
  "flow_name": "Language Detection Pipeline",
  "description": "Example flow with language detection",
  "global_config": {
    "doc_column": "content",
    "storage": "in-memory",
    "execute_type": "local"
  },
  "flow": [
    {
      "name": "ingest",
      "type": "ingest_source",
      "config": {
        "provider": "filesystem",
        "connection_params": {"paths": ["data/documents"]}
      }
    },
    {
      "name": "extract",
      "type": "extract_operator",
      "depends_on": ["ingest"],
      "config": {
        "text_extraction": {
          "provider": "docling_library"
        },
        "entity_extraction": {
          "provider": "none"
        }
      }
    },
    {
      "name": "detect_language",
      "type": "lang_detect",
      "depends_on": ["extract"],
      "config": {
        "language_provider": "fasttext",
        "filter_unknown_language": false
      }
    },
    {
      "name": "chunk",
      "type": "chunker",
      "depends_on": ["detect_language"],
      "config": {
        "chunk_size": 512
      }
    }
  ]
}

Parameters

Output Columns

All input columns are preserved. The operator appends:

Examples

Example 1 — fasttext (default)

{
  "name": "detect_language",
  "type": "lang_detect",
  "config": {
    "language_provider": "fasttext",
    "filter_unknown_language": false
  },
  "depends_on": ["extract"]
}

Example 2 — langdetect, filter unknowns

{
  "name": "detect_language",
  "type": "lang_detect",
  "config": {
    "language_provider": "langdetect",
    "filter_unknown_language": true
  },
  "depends_on": ["extract"]
}

Example 3 — Complete pipeline

{
  "flow_name": "Language Detection Pipeline",
  "flow": [
    { "name": "ingest", "type": "ingest_source", "config": { "provider": "filesystem", "connection_params": {"paths": ["data/documents"]} } },
    { "name": "extract", "type": "extract_operator", "depends_on": ["ingest"],
      "config": { "text_extraction": { "provider": "docling_library" }, "entity_extraction": { "provider": "none" } } },
    { "name": "detect_language", "type": "lang_detect", "depends_on": ["extract"],
      "config": { "language_provider": "fasttext", "filter_unknown_language": false } },
    { "name": "chunk", "type": "chunker", "depends_on": ["detect_language"],
      "config": { "chunk_size": 512 } }
  ]
}

Troubleshooting

Architecture

Supported providers

Supported Providers

fasttext (Default)

Supported Languages: 176+ languages including all langdetect languages plus:
- Uzbek (uz), Kazakh (kk), Azerbaijani (az)
- Bengali (bn), Tamil (ta), Telugu (te)
- Swahili (sw), Amharic (am), Yoruba (yo)
- And 120+ more languages

Pros:
- ✅ 176+ language support (3x more than langdetect)
- ✅ High accuracy for both long and short texts
- ✅ Better handling of mixed-language content
- ✅ Memory-efficient singleton model with reference counting
- ✅ Thread-safe model management

Cons:
- ❌ Requires model download (~131MB) on first use
- ❌ Slightly slower than langdetect for very short texts
- ❌ Requires fasttext library installation

langdetect

Supported Languages: 55+ languages including:
- English (en), Spanish (es), French (fr), German (de)
- Chinese (zh-cn, zh-tw), Japanese (ja), Korean (ko)
- Arabic (ar), Russian (ru), Portuguese (pt)
- Italian (it), Dutch (nl), Polish (pl)

Pros:
- ✅ No setup required
- ✅ Fast detection
- ✅ Good accuracy for common languages
- ✅ No external dependencies
- ✅ Probabilistic approach

Cons:
- ❌ Limited to 55 languages
- ❌ Less accurate for very short texts (<20 characters)
- ❌ May struggle with mixed-language content

Architecture

Hexagonal Architecture

The language detection operator follows hexagonal architecture (ports and adapters pattern):

┌─────────────────────────────────────────┐
│    LanguageDetect (Core Logic)          │
│                                         │
│  - Document processing                  │
│  - Error handling                       │
│  - PyArrow table management             │
│  - Filtering logic                      │
└──────────────┬──────────────────────────┘
               │
               │ Uses
               ▼
┌─────────────────────────────────────────┐
│   LanguageServicePort (Interface)       │
│   (ports/outbound/language_service.py)  │
│                                         │
│  - detect_language()                    │
│  - Returns: LanguageDetectionResult     │
└──────────────┬──────────────────────────┘
               │
               │ Implemented by
               ▼
┌─────────────────────────────────────────┐
│           Adapters                      │
│   (adapters/outbound/)                  │
│                                         │
│  - LangdetectAdapter                    │
│  - Custom adapters (extensible)         │
└──────────────┬──────────────────────────┘
               │
               │ Uses
               ▼
┌─────────────────────────────────────────┐
│      Language Detection Libraries       │
│                                         │
│  - langdetect                           │
│  - Future: spaCy, polyglot, etc.        │
└─────────────────────────────────────────┘
               ▲
               │
               │ Uses
┌──────────────┴──────────────────────────┐
│      Domain Models                      │
│      (domain/models.py)                 │
│                                         │
│  - LanguageDetectionResult (dataclass)  │
│    * language_code: str                 │
│    * confidence: float                  │
└─────────────────────────────────────────┘

Directory Structure

language_detection/
├── domain/
│   ├── __init__.py
│   └── models.py                    # Domain models (LanguageDetectionResult)
├── ports/
│   └── outbound/
│       ├── __init__.py
│       └── language_service.py      # Port interface (LanguageServicePort)
├── adapters/
│   └── outbound/
│       ├── __init__.py
│       ├── langdetect_adapter.py    # Langdetect implementation
│       └── factories/
│           ├── __init__.py
│           └── language_adapter_factory.py  # Factory for creating adapters
└── README.md

Current Adapters

LangdetectAdapter

• Location: adapters/outbound/langdetect_adapter.py
• Provider: langdetect library
• Languages: 55+
• Use Case: Fast detection for common languages, no setup required

FastTextAdapter

• Location: adapters/outbound/fasttext_adapter.py
• Provider: Facebook's FastText model
• Languages: 176+
• Use Case: High accuracy, support for rare languages, production workloads
• Infrastructure: Uses FastTextModelManager for singleton pattern and reference counting

Adding a New Provider

The architecture allows easy addition of new language detection providers:

1. Create Adapter (in adapters/outbound/):

from core.operators.quality.language_detection.domain.models import LanguageDetectionResult
from core.operators.quality.language_detection.ports.outbound.language_service import LanguageServicePort
from core.operators.quality.language_detection.adapters.outbound.factories.language_adapter_factory import (
    register_language_adapter,
)

@register_language_adapter
class MyLanguageAdapter(LanguageServicePort):
    ADAPTER_NAME = "mylang"
    ADAPTER_DISPLAY_NAME = "My Language Detector"

    def detect_language(self, text: str) -> LanguageDetectionResult:
        # Your implementation here
        language_code = "en"  # ISO 639-1 code
        confidence = 0.95     # 0.0 to 1.0
        return LanguageDetectionResult(language_code=language_code, confidence=confidence)

Note: LanguageDetectionResult is a dataclass defined in domain/models.py with two fields:
- language_code: ISO 639-1 language code (str)
- confidence: Confidence score between 0.0 and 1.0 (float)

1. Register Adapter (add to adapters/outbound/__init__.py):

from .mylang_adapter import MyLanguageAdapter

__all__ = [
    "FastTextAdapter",
    "LangdetectAdapter",
    "MyLanguageAdapter",  # Add your adapter
]

1. Use in Configuration:

The adapter will be automatically available through the factory pattern:

{
  "name": "detect_language",
  "type": "lang_detect",
  "config": {
    "language_provider": "mylang"
  }
}

Adding a new provider

Implement LanguageServicePort and register the adapter:

@register_language_adapter
class MyLanguageAdapter(LanguageServicePort):
    ADAPTER_NAME = "mylang"

    def detect_language(self, text: str) -> LanguageDetectionResult:
        return LanguageDetectionResult(language_code="en", confidence=0.95)

Then use "language_provider": "mylang" in the flow config.

Typical pipeline position

Ingest → Extract → LanguageDetectionOperator → [SQLFilter / Chunker → Embeddings]

Sample Flow

See sample_flows/advanced/quality_branching_merge_pipeline.json for a complete example using language detection for quality-based routing.