gh-christianlouis-docuelevate/docs/ProductionReadiness.md

# Production Readiness Guide

This guide bridges the gap between a working local installation and a hardened, production-ready DocuElevate deployment.  Work through the checklist below — every item should be addressed before exposing DocuElevate to real users or sensitive documents.

## Table of Contents

- [Quick Checklist](#quick-checklist)
- [1. Database](#1-database)
- [2. Persistent Storage](#2-persistent-storage)
- [3. Security Hardening](#3-security-hardening)
- [4. TLS / HTTPS](#4-tls--https)
- [5. Authentication](#5-authentication)
- [6. Scaling Workers](#6-scaling-workers)
- [7. Monitoring & Alerting](#7-monitoring--alerting)
- [8. Backup Strategy](#8-backup-strategy)
- [9. Updates & Maintenance](#9-updates--maintenance)
- [10. Helm / Kubernetes Specifics](#10-helm--kubernetes-specifics)

---

## Quick Checklist

Use this checklist to track readiness before going live.

- [ ] **Database** — PostgreSQL configured; SQLite is not used in production
- [ ] **Migrations** — `alembic upgrade head` runs cleanly on every deploy
- [ ] **Persistent storage** — workdir volume is mounted on durable, backed-up storage
- [ ] **TLS** — All traffic served over HTTPS; HTTP redirects to HTTPS
- [ ] **Session secret** — `SESSION_SECRET` is a random 32-byte (64-hex-char) value, not a placeholder
- [ ] **Admin password** — Strong password set; default placeholder removed
- [ ] **Auth enabled** — `AUTH_ENABLED=true`
- [ ] **Security headers** — Configured at the reverse proxy or via `SECURITY_HEADERS_ENABLED=true`
- [ ] **Rate limiting** — `RATE_LIMITING_ENABLED=true` (default)
- [ ] **Redis** — Running and accessible only from internal network
- [ ] **Meilisearch** — Running and accessible only from internal network
- [ ] **Worker replicas** — At least 2 workers configured for redundancy
- [ ] **Monitoring** — `/api/diagnostic/health` polled by uptime checker
- [ ] **Backups** — Automated backup of database, workdir, and Meilisearch data
- [ ] **Log retention** — Logs shipped to a persistent store or aggregator
- [ ] **Secrets management** — API keys not committed to source control

---

## 1. Database

### SQLite → PostgreSQL Migration

SQLite is the default database and is suitable only for **development or single-node, low-traffic setups**.  For any multi-replica deployment or meaningful production load, use PostgreSQL.

```bash
DATABASE_URL=postgresql://docuelevate:strongpassword@postgres-host:5432/docuelevate
```

See the [Database Configuration Guide](DatabaseConfiguration.md) for detailed setup instructions, migration steps, and optimization tips.

### Alembic Migrations

Always run database migrations on every deploy **before** new application code starts serving traffic:

```bash
alembic upgrade head
```

- **Docker Compose**: Add a one-shot `migrate` service that runs before `api` and `worker`:

  ```yaml
  migrate:
    image: ghcr.io/christianlouis/docuelevate:latest
    command: alembic upgrade head
    env_file: .env
    depends_on:
      - redis
  ```

- **Helm / Kubernetes**: The Helm chart includes a pre-install/pre-upgrade Job hook that runs `alembic upgrade head` automatically before pods are updated.

---

## 2. Persistent Storage

The `WORKDIR` directory (`/workdir` by default) is where documents are staged during processing.  **This must be backed by persistent, durable storage.**

### Docker Compose

Map a named volume or a host path:

```yaml
services:
  api:
    volumes:
      - docuelevate_workdir:/workdir
  worker:
    volumes:
      - docuelevate_workdir:/workdir   # Same volume — both services share it

volumes:
  docuelevate_workdir:
    driver: local
```

For production, replace `driver: local` with an NFS or other network-backed volume driver so data survives host failures.

### Kubernetes

Use a `ReadWriteMany` (RWX) PersistentVolumeClaim when running multiple replicas:

```yaml
workdir:
  persistence:
    enabled: true
    accessMode: ReadWriteMany   # Required for multi-replica
    size: 50Gi
    storageClass: "nfs-client"  # Or your cluster's RWX storage class
```

Single-replica clusters can use `ReadWriteOnce`.

---

## 3. Security Hardening

### Session Secret

Generate a strong, unique secret and set it as `SESSION_SECRET`:

```bash
python -c "import secrets; print(secrets.token_hex(32))"
# or
openssl rand -hex 32
```

This value **must be at least 32 characters** and must not be a known placeholder.  Rotating it invalidates all active sessions.

### HTTP Security Headers

DocuElevate's built-in security headers are **disabled by default** because most production deployments sit behind a reverse proxy that already adds them.

**Option A** — Let your reverse proxy add headers (recommended):

- See the Nginx and Traefik examples in the [Deployment Guide](DeploymentGuide.md#security-headers).

**Option B** — Enable built-in headers (only if no reverse proxy):

```bash
SECURITY_HEADERS_ENABLED=true
```

Recommended headers to configure at the proxy level:

| Header | Recommended Value |
|--------|-------------------|
| `Strict-Transport-Security` | `max-age=31536000; includeSubDomains` |
| `X-Frame-Options` | `DENY` |
| `X-Content-Type-Options` | `nosniff` |
| `Referrer-Policy` | `strict-origin-when-cross-origin` |
| `Content-Security-Policy` | See CSP notes below |

#### Content-Security-Policy Notes

DocuElevate's frontend uses Tailwind CSS v3 compiled at Docker build time. No external CDN requests are needed for CSS. In production, your CSP does not need to allow any external style sources beyond your own static file origin.  A starting point:

```
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:;
```

Audit and tighten this policy for your specific deployment.

### Rate Limiting

Rate limiting is enabled by default and requires Redis.  Verify it is active:

```bash
RATE_LIMITING_ENABLED=true   # default — ensure not overridden to false
REDIS_URL=redis://redis:6379/0
```

See the [Configuration Guide — Rate Limiting](ConfigurationGuide.md#rate-limiting) for per-endpoint tuning.

### Secrets Management

- **Never commit `.env` files** containing real secrets to source control.
- For Kubernetes, use an external secret manager (HashiCorp Vault, External Secrets Operator, Sealed Secrets) and reference secrets by name in Helm values rather than embedding them.
- Rotate API keys, the session secret, and database credentials on a regular schedule. See the [Credential Rotation Guide](CredentialRotationGuide.md).

### File Upload Limits

Set appropriate upload size limits to prevent resource exhaustion:

```bash
MAX_UPLOAD_SIZE=104857600       # 100 MB — adjust for your use case
MAX_REQUEST_BODY_SIZE=1048576   # 1 MB for non-file requests (default)
```

---

## 4. TLS / HTTPS

**All production traffic must be served over HTTPS.**

### Docker Compose with Traefik

```yaml
services:
  api:
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.docuelevate.rule=Host(`docuelevate.example.com`)"
      - "traefik.http.routers.docuelevate.entrypoints=websecure"
      - "traefik.http.routers.docuelevate.tls.certresolver=letsencrypt"
```

### Nginx Reverse Proxy

```nginx
server {
    listen 80;
    server_name docuelevate.example.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    server_name docuelevate.example.com;

    ssl_certificate /etc/nginx/ssl/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        client_max_body_size 1g;
    }
}
```

### Kubernetes (Helm)

```yaml
ingress:
  enabled: true
  className: nginx
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: "1g"
  hosts:
    - host: docuelevate.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: docuelevate-tls
      hosts:
        - docuelevate.example.com
```

---

## 5. Authentication

Enable authentication and choose an auth method appropriate for your organization.

```bash
AUTH_ENABLED=true
ADMIN_USERNAME=admin
ADMIN_PASSWORD=<strong-password>
SESSION_SECRET=<min-32-char-random-string>
```

For SSO/OIDC (Authentik, Keycloak, Auth0, etc.) see the [Authentication Setup Guide](AuthenticationSetup.md).

**Best practices:**

- Use OIDC/SSO for team deployments to centralize access control.
- Enforce strong password policies or delegate password management to your identity provider.
- Set an appropriate session timeout (handled by the identity provider for OIDC, or by session middleware for basic auth).

---

## 6. Scaling Workers

### Docker Compose

Scale workers independently:

```bash
docker compose up -d --scale worker=3
```

Each worker processes tasks from the Celery queue independently.  Ensure the shared `workdir` volume is accessible from all worker containers.

> **Important:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance.  It is defined as a dedicated service in `docker-compose.yaml` with a fixed `container_name`.  Do not scale it.

### Scaling the API

API pods are fully stateless (sessions use encrypted cookies, not server-side state) and can be scaled behind a load balancer:

```bash
docker compose up -d --scale api=3
```

### Kubernetes (Helm)

```yaml
worker:
  replicaCount: 3
  autoscaling:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 75
```

### Worker Queue Tuning

Celery workers process three queues with different priorities:

| Queue | Purpose |
|-------|---------|
| `document_processor` | Main document processing tasks (OCR, conversion) |
| `default` | Metadata extraction, storage uploads |
| `celery` | Built-in Celery management tasks |

To dedicate workers to specific queues in high-volume deployments:

```bash
# High-priority worker — document processing only
celery -A app.celery_worker worker -Q document_processor --concurrency=4

# General worker — everything else
celery -A app.celery_worker worker -Q default,celery --concurrency=2
```

---

## 7. Monitoring & Alerting

### Health Check Endpoint

DocuElevate exposes three health-related endpoints:

| Endpoint | Auth | Purpose |
|----------|------|---------|
| `GET /api/diagnostic/healthz/live` | None | Lightweight liveness probe — returns 200 if the process is running |
| `GET /api/diagnostic/healthz/ready` | None | Readiness probe — checks database and Redis (503 when DB is down) |
| `GET /api/diagnostic/health` | Required | Full status for monitoring dashboards (Grafana, Uptime Kuma) |

For **Kubernetes probes**, use the unauthenticated endpoints:

```yaml
livenessProbe:
  httpGet:
    path: /api/diagnostic/healthz/live
    port: 8000
readinessProbe:
  httpGet:
    path: /api/diagnostic/healthz/ready
    port: 8000
```

For **uptime monitors** (Uptime Kuma, Grafana, etc.), use the authenticated endpoint:

```bash
curl http://docuelevate.example.com/api/diagnostic/health
# Expected: {"status": "healthy", ...}
```

Set `UPTIME_KUMA_URL` to your Uptime Kuma push URL for heartbeat monitoring:

```bash
UPTIME_KUMA_URL=https://uptime.example.com/api/push/abc123
```

### Prometheus / Grafana

Scrape the `/api/health` endpoint or add a custom Prometheus exporter.  Useful metrics to track:

- Number of documents processed per hour
- Queue length (via Redis `LLEN` on Celery queues)
- Worker concurrency and CPU utilization
- API request latency (p50, p95, p99)

### Log Aggregation

- **Docker Compose**: Use the `logging` driver to ship to Loki, Fluentd, or CloudWatch:

  ```yaml
  services:
    api:
      logging:
        driver: "json-file"
        options:
          max-size: "50m"
          max-file: "5"
  ```

- **Kubernetes**: Logs are written to stdout/stderr and can be captured by your cluster's log aggregator (Fluentd, Vector, Promtail).

---

## 8. Backup Strategy

Back up all three data stores regularly:

### Database

**PostgreSQL:**

```bash
pg_dump -h postgres-host -U docuelevate docuelevate > backup_$(date +%Y%m%d).sql
```

Automate with a cron job or your cloud provider's managed backup feature.

**SQLite** (development only):

```bash
cp app/database.db backup_$(date +%Y%m%d).db
```

### Workdir Volume

The `WORKDIR` directory contains original uploads and processed documents.  Use your volume provider's snapshot feature or rsync to a secondary location:

```bash
rsync -av /workdir/ /backup/workdir/
```

### Meilisearch Data

Meilisearch stores its index in the directory specified by `MEILI_DB_PATH` (default `/meili_data`).  Snapshot it regularly or use [Meilisearch's dump feature](https://www.meilisearch.com/docs/reference/api/dumps):

```bash
curl -X POST http://localhost:7700/dumps \
  -H "Authorization: Bearer $MEILISEARCH_API_KEY"
```

### Configuration / Secrets

Back up your `.env` file or Helm values file to a **secure, encrypted** store (e.g., a password manager or secrets vault).  Do not commit it to source control.

---

## 9. Updates & Maintenance

### Docker Compose

```bash
git pull
docker-compose pull
docker-compose down && docker-compose up -d
```

The `alembic upgrade head` command is run automatically if you include the `migrate` service (see [Database](#1-database)).

### Helm / Kubernetes

```bash
helm upgrade docuelevate ./helm/docuelevate \
  --namespace docuelevate \
  -f my-values.yaml
```

The Helm chart's pre-upgrade hook runs `alembic upgrade head` before new pods start.

### Keep Dependencies Updated

```bash
pip install --upgrade -r requirements.txt
safety check   # Scan for known CVEs
```

Enable [GitHub Dependabot](https://docs.github.com/en/code-security/dependabot) or similar automated dependency update tooling.

---

## 10. Helm / Kubernetes Specifics

For a dedicated Kubernetes deployment guide, including architecture diagrams, PVC configuration, HPA, and ingress examples, see:

- [Deployment Guide — Kubernetes / Helm](DeploymentGuide.md#kubernetes--helm-deployment)

**Additional production recommendations for Kubernetes:**

- **Pod Disruption Budgets (PDB)**: Ensure at least one API pod is always available during node maintenance.

  ```yaml
  apiVersion: policy/v1
  kind: PodDisruptionBudget
  metadata:
    name: docuelevate-api-pdb
  spec:
    minAvailable: 1
    selector:
      matchLabels:
        app.kubernetes.io/component: api
  ```

- **Resource Requests & Limits**: Set CPU/memory requests and limits on all containers to ensure the scheduler can place pods correctly.

  ```yaml
  api:
    resources:
      requests:
        cpu: "250m"
        memory: "512Mi"
      limits:
        cpu: "1000m"
        memory: "2Gi"
  worker:
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2000m"
        memory: "4Gi"
  ```

- **Network Policies**: Restrict traffic so that Redis and Meilisearch are reachable only from DocuElevate pods, not from the internet or other namespaces.

- **Image Pull Policy**: Use `IfNotPresent` in production with pinned image tags (not `latest`) for reproducible deployments.

- **Liveness & Readiness Probes**: Already configured in the Helm chart via unauthenticated endpoints (`/api/diagnostic/healthz/live` and `/api/diagnostic/healthz/ready`).  Verify they are tuned to your startup time.