v2.0 IBM/docling-pipelines

Docling Pipelines OpenShift Deployment Guide

Complete guide for deploying the Docling Pipelines project on OpenShift using the automated deployment script.

Overview

This guide covers deploying Docling Pipelines on OpenShift by:
1. Creating a project namespace
2. Building the application from Git repository
3. Creating an ImageStream for the container image
4. Deploying the application with proper configuration
5. Exposing the service via OpenShift Route

Prerequisites

Required Tools

  • OpenShift CLI (oc): Version 4.10 or higher
  • Git: For repository access
  • OpenShift Cluster: Access to an OpenShift cluster with appropriate permissions

Required Permissions

  • Create projects/namespaces
  • Create builds, deployments, services, and routes
  • Create imagestreams

Verify Prerequisites

# Check oc CLI installation
oc version

# Login to OpenShift cluster
oc login https://api.your-cluster.com:6443

# Verify login
oc whoami

Quick Start

1. Clone the Repository

git clone https://github.com/IBM/docling-pipelines.git
cd docling-pipelines

2. Configure Git Credentials for Private Repositories

If the repository is private, export the Git username and personal access token before starting the deployment script. The script uses these values to create the OpenShift secret docpipe-git-auth and attach it to the build source.

export GIT_USERNAME="your-git-username"
export GIT_TOKEN="your-personal-access-token"

# Optional: override the secret name created by the script
export GIT_SECRET_NAME="docpipe-git-auth"  # pragma: allowlist secret

Notes:
- Use an HTTPS repository URL, not SSH
- The personal access token must have permission to read the repository
- Avoid putting the token directly on the command line or committing it to shell history files

3. Run Deployment Script

# Deploy with default settings (project name: docpipe)
./scripts/deploy-openshift.sh

# Or specify custom project name and Git repository
./scripts/deploy-openshift.sh my-docpipe-project https://github.com/IBM/docling-pipelines.git main

4. Monitor Deployment

# Watch build progress
oc logs -f bc/docpipe-app

# Watch pod status
oc get pods -w

# Check deployment status
oc get deployment docpipe-backend

5. Access the Application

# Get the route URL
oc get route docpipe-route -o jsonpath='{.spec.host}'

# Test the application
DOCPIPE_URL=$(oc get route docpipe-route -o jsonpath='{.spec.host}')
curl -k https://$DOCPIPE_URL/health

Deployment Script Details

Script Parameters

./scripts/deploy-openshift.sh [project-name] [git-repo-url] [git-branch]

Parameters:
- project-name (optional): OpenShift project name (default: docpipe)
- git-repo-url (optional): Git repository URL (default: https://github.com/IBM/docling-pipelines.git)
- git-branch (optional): Git branch to deploy (default: main)

What the Script Does

  1. Prerequisites Check
  2. Verifies oc CLI is installed
  3. Confirms user is logged into OpenShift

  4. Displays current user and cluster information

  5. Project Creation

  6. Creates new OpenShift project with specified name
  7. Checks for existing project and prompts for confirmation

  8. Sets project as current context

  9. Git Credentials Secret Creation

  10. Reads GIT_USERNAME and GIT_TOKEN from the environment
  11. Creates or updates an OpenShift basic-auth secret for Git access

  12. Attaches that secret to the build source for private repository checkout

  13. Application BuildConfig Creation

  14. Creates a Docker build from the Git repository
  15. Uses docker/Dockerfile for the image build

  16. Starts the build explicitly

  17. ImageStream Creation

  18. Creates an internal ImageStream for storing built container images

  19. Configures local lookup policy

  20. Deployment Creation

  21. Creates an OpenShift DeploymentConfig with 2 replicas
  22. Triggers rollout from the built ImageStream tag

  23. Configures resource limits and health checks

  24. Service Creation

  25. Creates ClusterIP Service for backend pods

  26. Route Creation

  27. Creates edge-terminated TLS route
  28. Enables automatic HTTP to HTTPS redirect

  29. Exposes service externally

  30. Verification

  31. Monitors build completion
  32. Verifies pod, service, and route creation
  33. Displays deployment summary

Manual Deployment Steps

If you prefer manual deployment or need to customize the process:

1. Create Project

oc new-project docpipe --display-name="Docling Pipelines" \
  --description="Docling Pipelines - Modular document processing framework"

2. Create Application from Git

oc new-app python:3.12~https://github.com/IBM/docling-pipelines.git \
  --name=docpipe-app \
  --strategy=source \
  --context-dir=. \
  --env PYTHONPATH=/opt/app-root/src/src \
  --env PYTHONUNBUFFERED=1

3. Create ImageStream

cat <<EOF | oc apply -f -
apiVersion: image.openshift.io/v1
kind: ImageStream
metadata:
  name: docpipe-image
spec:
  lookupPolicy:
    local: true
EOF

4. Create BuildConfig

cat <<EOF | oc apply -f -
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: docpipe-build
spec:
  output:
    to:
      kind: ImageStreamTag
      name: docpipe-image:latest
  source:
    type: Git
    git:
      uri: https://github.com/IBM/docling-pipelines.git
      ref: main
  strategy:
    type: Source
    sourceStrategy:
      from:
        kind: ImageStreamTag
        namespace: openshift
        name: python:3.12
EOF

5. Create Deployment

cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: docpipe-backend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: docpipe-app
  template:
    metadata:
      labels:
        app: docpipe-app
    spec:
      containers:
      - name: docpipe
        image: docpipe-image:latest
        ports:
        - containerPort: 8000
        resources:
          requests:
            memory: "2Gi"
            cpu: "1000m"
          limits:
            memory: "4Gi"
            cpu: "2000m"
EOF

6. Create Service

oc expose deployment docpipe-backend --port=8000 --name=docpipe-service

7. Create Route

oc create route edge docpipe-route --service=docpipe-service --insecure-policy=Redirect

Configuration

Environment Variables

The deployment configures the following environment variables:

  • PYTHONPATH=/opt/app-root/src/src: Ensures Python can find docpipe modules
  • PYTHONUNBUFFERED=1: Enables real-time log output

Resource Limits

Default resource configuration per pod:

Resource Request Limit
Memory 2Gi 4Gi
CPU 1 core 2 cores

Health Checks

Liveness Probe:
- Endpoint: /health
- Initial Delay: 30 seconds
- Period: 10 seconds

Readiness Probe:
- Endpoint: /health
- Initial Delay: 10 seconds
- Period: 5 seconds

Operations

Scaling

# Scale to 3 replicas
oc scale deployment/docpipe-backend --replicas=3

# Check scaling status
oc get deployment docpipe-backend

Viewing Logs

# View application logs
oc logs -f deployment/docpipe-backend

# View specific pod logs
oc logs -f <pod-name>

# View build logs
oc logs -f bc/docpipe-app

Rebuilding

# Trigger new build from Git
oc start-build docpipe-app

# Follow build logs
oc logs -f bc/docpipe-app

# Check build status
oc get builds

Updating Configuration

# Update environment variable
oc set env deployment/docpipe-backend NEW_VAR=value

# Update resource limits
oc set resources deployment/docpipe-backend \
  --requests=cpu=1500m,memory=3Gi \
  --limits=cpu=2500m,memory=5Gi

# Rollout restart
oc rollout restart deployment/docpipe-backend

Rolling Updates

# Update image
oc set image deployment/docpipe-backend docpipe=docpipe-image:v2.0

# Check rollout status
oc rollout status deployment/docpipe-backend

# View rollout history
oc rollout history deployment/docpipe-backend

# Rollback to previous version
oc rollout undo deployment/docpipe-backend

Monitoring

Pod Status

# List all pods
oc get pods

# Watch pod status
oc get pods -w

# Describe pod for events
oc describe pod <pod-name>

Resource Usage

# View resource usage
oc adm top pods

# View node resource usage
oc adm top nodes

Events

# View recent events
oc get events --sort-by='.lastTimestamp'

# Watch events
oc get events -w

Troubleshooting

Build Failures

# Check build logs
oc logs -f bc/docpipe-app

# Describe build for errors
oc describe build <build-name>

# Check build config
oc describe bc/docpipe-app

Pod Not Starting

# Check pod status
oc get pods

# Describe pod for events
oc describe pod <pod-name>

# Check pod logs
oc logs <pod-name>

# Check previous container logs (if crashed)
oc logs <pod-name> --previous

Image Pull Errors

# Check imagestream
oc get imagestream docpipe-image

# Describe imagestream
oc describe imagestream docpipe-image

# Check image tags
oc get imagestreamtag

Route Not Accessible

# Check route configuration
oc get route docpipe-route

# Describe route
oc describe route docpipe-route

# Check service endpoints
oc get endpoints docpipe-service

# Test from inside cluster
oc run test-pod --image=curlimages/curl -it --rm -- \
  curl http://docpipe-service:8000/health

Network Issues

# Check service
oc get svc docpipe-service

# Check endpoints
oc get endpoints docpipe-service

# Test connectivity from debug pod
oc debug deployment/docpipe-backend

Cleanup

Remove All Resources

# Delete all resources in project
oc delete all -l app=docpipe-app

# Or delete entire project
oc delete project docpipe

Selective Cleanup

# Delete deployment only
oc delete deployment docpipe-backend

# Delete service
oc delete svc docpipe-service

# Delete route
oc delete route docpipe-route

# Delete buildconfig
oc delete bc docpipe-build

# Delete imagestream
oc delete imagestream docpipe-image

Advanced Configuration

Custom Dockerfile Build

If you need to use a custom Dockerfile instead of S2I:

# Create BuildConfig with Docker strategy
cat <<EOF | oc apply -f -
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: docpipe-docker-build
spec:
  output:
    to:
      kind: ImageStreamTag
      name: docpipe-image:latest
  source:
    type: Git
    git:
      uri: https://github.com/IBM/docling-pipelines.git
  strategy:
    type: Docker
    dockerStrategy:
      dockerfilePath: Dockerfile
EOF

Persistent Storage

Add persistent volume for data storage:

# Create PVC
cat <<EOF | oc apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: docpipe-data
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
EOF

# Mount in deployment
oc set volume deployment/docpipe-backend \
  --add --name=data-volume \
  --type=persistentVolumeClaim \
  --claim-name=docpipe-data \
  --mount-path=/data

ConfigMaps and Secrets

# Create ConfigMap
oc create configmap docpipe-config \
  --from-file=config.yaml

# Create Secret
oc create secret generic docpipe-secrets \
  --from-literal=api-key=your-secret-key

# Mount in deployment
oc set volume deployment/docpipe-backend \
  --add --name=config \
  --type=configmap \
  --configmap-name=docpipe-config \
  --mount-path=/config

oc set env deployment/docpipe-backend \
  --from=secret/docpipe-secrets

Integration with CI/CD

Webhook Triggers

The BuildConfig includes webhook triggers for automated builds:

# Get webhook URL
oc describe bc/docpipe-build | grep -A 1 "Webhook GitHub"

# Configure in GitHub repository settings:
# Settings > Webhooks > Add webhook
# Payload URL: <webhook-url>
# Content type: application/json

Jenkins Integration

# Create Jenkins pipeline
cat <<EOF | oc apply -f -
apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  name: docpipe-pipeline
spec:
  strategy:
    type: JenkinsPipeline
    jenkinsPipelineStrategy:
      jenkinsfile: |
        pipeline {
          agent any
          stages {
            stage('Build') {
              steps {
                script {
                  openshift.withCluster() {
                    openshift.withProject('docpipe') {
                      openshift.startBuild('docpipe-build').logs('-f')
                    }
                  }
                }
              }
            }
            stage('Deploy') {
              steps {
                script {
                  openshift.withCluster() {
                    openshift.withProject('docpipe') {
                      openshift.selector('deployment', 'docpipe-backend').rollout().latest()
                    }
                  }
                }
              }
            }
          }
        }
EOF

Best Practices

  1. Use Specific Image Tags: Avoid using latest tag in production
  2. Set Resource Limits: Always define resource requests and limits
  3. Enable Health Checks: Configure liveness and readiness probes
  4. Use Secrets: Store sensitive data in OpenShift Secrets
  5. Enable Monitoring: Configure Prometheus metrics and alerts
  6. Implement Logging: Use centralized logging (EFK stack)
  7. Regular Backups: Backup persistent data and configurations
  8. Security Scanning: Scan images for vulnerabilities
  9. Network Policies: Implement network policies for security
  10. Documentation: Keep deployment documentation up to date

Support

For issues or questions:
- Check pod logs: oc logs <pod-name>
- Review events: oc get events --sort-by='.lastTimestamp'
- Consult main documentation: README.md
- Review architecture: ARCHITECTURE.md