From 4d019d53d9ad161a68639d6ba2109ba3ac77df34 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Tue, 17 Mar 2026 00:08:17 +0000 Subject: [PATCH] docs: update scaling, health probe, and beat scheduler documentation - Update DeploymentGuide.md with scaling instructions and beat service info - Update KubernetesDeployment.md with unauthenticated probe paths and beat note - Update ProductionReadiness.md with new health endpoints table and beat guidance - Update API.md with new healthz/live and healthz/ready endpoint docs Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> --- docs/API.md | 41 ++++++++++++++++++++++++++++ docs/DeploymentGuide.md | 16 +++++++---- docs/KubernetesDeployment.md | 18 ++++++++---- docs/ProductionReadiness.md | 53 ++++++++++++++++++++++++++---------- 4 files changed, 102 insertions(+), 26 deletions(-) diff --git a/docs/API.md b/docs/API.md index 29a3dc7a..5e42751d 100644 --- a/docs/API.md +++ b/docs/API.md @@ -1571,6 +1571,47 @@ Lightweight endpoint returning the total number of queued + in-progress items. D ## Diagnostic +### GET /api/diagnostic/healthz/live + +Lightweight liveness probe for Kubernetes. Returns **200 OK** as long as the process is running. This endpoint does **not** check external dependencies and is intentionally cheap. + +**Authentication:** None (designed for kubelet probes) + +**Response (200 OK):** +```json +{ + "status": "ok" +} +``` + +### GET /api/diagnostic/healthz/ready + +Readiness probe for Kubernetes. Verifies that the application can serve traffic by checking database and Redis connectivity. + +**Authentication:** None (designed for kubelet probes) + +**Response (200 OK) – ready to serve traffic:** +```json +{ + "status": "ready", + "checks": { + "database": {"status": "ok"}, + "redis": {"status": "ok"} + } +} +``` + +**Response (503 Service Unavailable) – database unreachable:** +```json +{ + "status": "not_ready", + "checks": { + "database": {"status": "error", "detail": "..."}, + "redis": {"status": "ok"} + } +} +``` + ### GET /api/diagnostic/health System health endpoint designed for monitoring tools such as Grafana, Uptime Kuma, Prometheus blackbox exporter, or any HTTP-based health checker. diff --git a/docs/DeploymentGuide.md b/docs/DeploymentGuide.md index cb66bfa9..72f9c30a 100644 --- a/docs/DeploymentGuide.md +++ b/docs/DeploymentGuide.md @@ -349,16 +349,18 @@ workdir: ## Scaling +DocuElevate is designed for horizontal scaling. Both API and worker pods are stateless and can be scaled independently. + ### Docker Compose -Add more worker containers: +Scale workers (task processing) and API pods (request handling) independently: -```yaml -worker: - deploy: - replicas: 3 +```bash +docker compose up -d --scale worker=3 --scale api=2 ``` +> **Note:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. Do not scale it. It publishes periodic tasks to the Redis broker; workers pick them up. + ### Kubernetes / Helm Enable HPA: @@ -377,13 +379,15 @@ worker: maxReplicas: 10 ``` +The Helm chart deploys a separate **beat** pod (always 1 replica, `Recreate` strategy) so that scheduled tasks are never duplicated when workers scale. + --- ## Monitoring - **Docker Compose**: `docker-compose logs -f`, `docker stats` - **Kubernetes**: `kubectl logs -l app.kubernetes.io/component=api -f` -- **Prometheus / Grafana**: Scrape the `/api/health` endpoint for readiness; add custom metrics as needed. +- **Prometheus / Grafana**: Scrape the `/api/diagnostic/healthz/ready` endpoint for readiness; add custom metrics as needed. - **Uptime Kuma**: Set `UPTIME_KUMA_URL` to your push URL for heartbeat monitoring. --- diff --git a/docs/KubernetesDeployment.md b/docs/KubernetesDeployment.md index 94247873..c23ddc81 100644 --- a/docs/KubernetesDeployment.md +++ b/docs/KubernetesDeployment.md @@ -373,6 +373,8 @@ worker: replicaCount: 4 ``` +> **Beat scheduler:** The Helm chart deploys a dedicated `beat` pod (always exactly 1 replica with `Recreate` strategy) that publishes periodic tasks to the Redis broker. Workers consume these tasks — scaling workers does **not** duplicate scheduled jobs. + ### Horizontal Pod Autoscaler ```yaml @@ -433,24 +435,30 @@ externalRedis: ### Kubernetes Probes -The Helm chart configures liveness and readiness probes on the API pods via `/api/health`. Default settings: +The Helm chart configures **unauthenticated** liveness and readiness probes on the API pods so kubelet can reach them without credentials. Default settings: ```yaml api: livenessProbe: httpGet: - path: /api/health + path: /api/diagnostic/healthz/live port: 8000 initialDelaySeconds: 30 - periodSeconds: 30 + periodSeconds: 20 readinessProbe: httpGet: - path: /api/health + path: /api/diagnostic/healthz/ready port: 8000 - initialDelaySeconds: 10 + initialDelaySeconds: 15 periodSeconds: 10 ``` +| Endpoint | Auth | Purpose | +|----------|------|---------| +| `/api/diagnostic/healthz/live` | None | Lightweight liveness check — returns 200 if the process is running | +| `/api/diagnostic/healthz/ready` | None | Readiness check — verifies database and Redis connectivity (503 when DB is down) | +| `/api/diagnostic/health` | Required | Full health status for monitoring dashboards (Grafana, Uptime Kuma) | + ### Prometheus Scraping Add annotations to expose metrics (if using a Prometheus-compatible exporter): diff --git a/docs/ProductionReadiness.md b/docs/ProductionReadiness.md index 2f93c7c5..a938e39e 100644 --- a/docs/ProductionReadiness.md +++ b/docs/ProductionReadiness.md @@ -34,7 +34,7 @@ Use this checklist to track readiness before going live. - [ ] **Redis** — Running and accessible only from internal network - [ ] **Meilisearch** — Running and accessible only from internal network - [ ] **Worker replicas** — At least 2 workers configured for redundancy -- [ ] **Monitoring** — `/api/health` polled by uptime checker +- [ ] **Monitoring** — `/api/diagnostic/health` polled by uptime checker - [ ] **Backups** — Automated backup of database, workdir, and Meilisearch data - [ ] **Log retention** — Logs shipped to a persistent store or aggregator - [ ] **Secrets management** — API keys not committed to source control @@ -285,22 +285,24 @@ For SSO/OIDC (Authentik, Keycloak, Auth0, etc.) see the [Authentication Setup Gu ### Docker Compose -Use the `deploy.replicas` setting (requires Docker Swarm mode) or simply run multiple workers: - -```yaml -worker: - deploy: - replicas: 3 -``` - -Or scale after deployment: +Scale workers independently: ```bash -docker-compose up -d --scale worker=3 +docker compose up -d --scale worker=3 ``` Each worker processes tasks from the Celery queue independently. Ensure the shared `workdir` volume is accessible from all worker containers. +> **Important:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. It is defined as a dedicated service in `docker-compose.yaml` with a fixed `container_name`. Do not scale it. + +### Scaling the API + +API pods are fully stateless (sessions use encrypted cookies, not server-side state) and can be scaled behind a load balancer: + +```bash +docker compose up -d --scale api=3 +``` + ### Kubernetes (Helm) ```yaml @@ -339,11 +341,32 @@ celery -A app.celery_worker worker -Q default,celery --concurrency=2 ### Health Check Endpoint -DocuElevate exposes `/api/health` for readiness probing. Configure your uptime monitor to poll this endpoint: +DocuElevate exposes three health-related endpoints: + +| Endpoint | Auth | Purpose | +|----------|------|---------| +| `GET /api/diagnostic/healthz/live` | None | Lightweight liveness probe — returns 200 if the process is running | +| `GET /api/diagnostic/healthz/ready` | None | Readiness probe — checks database and Redis (503 when DB is down) | +| `GET /api/diagnostic/health` | Required | Full status for monitoring dashboards (Grafana, Uptime Kuma) | + +For **Kubernetes probes**, use the unauthenticated endpoints: + +```yaml +livenessProbe: + httpGet: + path: /api/diagnostic/healthz/live + port: 8000 +readinessProbe: + httpGet: + path: /api/diagnostic/healthz/ready + port: 8000 +``` + +For **uptime monitors** (Uptime Kuma, Grafana, etc.), use the authenticated endpoint: ```bash -curl http://docuelevate.example.com/api/health -# Expected: {"status": "ok", ...} +curl http://docuelevate.example.com/api/diagnostic/health +# Expected: {"status": "healthy", ...} ``` Set `UPTIME_KUMA_URL` to your Uptime Kuma push URL for heartbeat monitoring: @@ -502,4 +525,4 @@ For a dedicated Kubernetes deployment guide, including architecture diagrams, PV - **Image Pull Policy**: Use `IfNotPresent` in production with pinned image tags (not `latest`) for reproducible deployments. -- **Liveness & Readiness Probes**: Already configured in the Helm chart via `/api/health`. Verify they are tuned to your startup time. +- **Liveness & Readiness Probes**: Already configured in the Helm chart via unauthenticated endpoints (`/api/diagnostic/healthz/live` and `/api/diagnostic/healthz/ready`). Verify they are tuned to your startup time.