docs: update scaling, health probe, and beat scheduler documentation
- Update DeploymentGuide.md with scaling instructions and beat service info - Update KubernetesDeployment.md with unauthenticated probe paths and beat note - Update ProductionReadiness.md with new health endpoints table and beat guidance - Update API.md with new healthz/live and healthz/ready endpoint docs Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
+41
@@ -1571,6 +1571,47 @@ Lightweight endpoint returning the total number of queued + in-progress items. D
|
||||
|
||||
## Diagnostic
|
||||
|
||||
### GET /api/diagnostic/healthz/live
|
||||
|
||||
Lightweight liveness probe for Kubernetes. Returns **200 OK** as long as the process is running. This endpoint does **not** check external dependencies and is intentionally cheap.
|
||||
|
||||
**Authentication:** None (designed for kubelet probes)
|
||||
|
||||
**Response (200 OK):**
|
||||
```json
|
||||
{
|
||||
"status": "ok"
|
||||
}
|
||||
```
|
||||
|
||||
### GET /api/diagnostic/healthz/ready
|
||||
|
||||
Readiness probe for Kubernetes. Verifies that the application can serve traffic by checking database and Redis connectivity.
|
||||
|
||||
**Authentication:** None (designed for kubelet probes)
|
||||
|
||||
**Response (200 OK) – ready to serve traffic:**
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"checks": {
|
||||
"database": {"status": "ok"},
|
||||
"redis": {"status": "ok"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response (503 Service Unavailable) – database unreachable:**
|
||||
```json
|
||||
{
|
||||
"status": "not_ready",
|
||||
"checks": {
|
||||
"database": {"status": "error", "detail": "..."},
|
||||
"redis": {"status": "ok"}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GET /api/diagnostic/health
|
||||
|
||||
System health endpoint designed for monitoring tools such as Grafana, Uptime Kuma, Prometheus blackbox exporter, or any HTTP-based health checker.
|
||||
|
||||
+10
-6
@@ -349,16 +349,18 @@ workdir:
|
||||
|
||||
## Scaling
|
||||
|
||||
DocuElevate is designed for horizontal scaling. Both API and worker pods are stateless and can be scaled independently.
|
||||
|
||||
### Docker Compose
|
||||
|
||||
Add more worker containers:
|
||||
Scale workers (task processing) and API pods (request handling) independently:
|
||||
|
||||
```yaml
|
||||
worker:
|
||||
deploy:
|
||||
replicas: 3
|
||||
```bash
|
||||
docker compose up -d --scale worker=3 --scale api=2
|
||||
```
|
||||
|
||||
> **Note:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. Do not scale it. It publishes periodic tasks to the Redis broker; workers pick them up.
|
||||
|
||||
### Kubernetes / Helm
|
||||
|
||||
Enable HPA:
|
||||
@@ -377,13 +379,15 @@ worker:
|
||||
maxReplicas: 10
|
||||
```
|
||||
|
||||
The Helm chart deploys a separate **beat** pod (always 1 replica, `Recreate` strategy) so that scheduled tasks are never duplicated when workers scale.
|
||||
|
||||
---
|
||||
|
||||
## Monitoring
|
||||
|
||||
- **Docker Compose**: `docker-compose logs -f`, `docker stats`
|
||||
- **Kubernetes**: `kubectl logs -l app.kubernetes.io/component=api -f`
|
||||
- **Prometheus / Grafana**: Scrape the `/api/health` endpoint for readiness; add custom metrics as needed.
|
||||
- **Prometheus / Grafana**: Scrape the `/api/diagnostic/healthz/ready` endpoint for readiness; add custom metrics as needed.
|
||||
- **Uptime Kuma**: Set `UPTIME_KUMA_URL` to your push URL for heartbeat monitoring.
|
||||
|
||||
---
|
||||
|
||||
@@ -373,6 +373,8 @@ worker:
|
||||
replicaCount: 4
|
||||
```
|
||||
|
||||
> **Beat scheduler:** The Helm chart deploys a dedicated `beat` pod (always exactly 1 replica with `Recreate` strategy) that publishes periodic tasks to the Redis broker. Workers consume these tasks — scaling workers does **not** duplicate scheduled jobs.
|
||||
|
||||
### Horizontal Pod Autoscaler
|
||||
|
||||
```yaml
|
||||
@@ -433,24 +435,30 @@ externalRedis:
|
||||
|
||||
### Kubernetes Probes
|
||||
|
||||
The Helm chart configures liveness and readiness probes on the API pods via `/api/health`. Default settings:
|
||||
The Helm chart configures **unauthenticated** liveness and readiness probes on the API pods so kubelet can reach them without credentials. Default settings:
|
||||
|
||||
```yaml
|
||||
api:
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
path: /api/diagnostic/healthz/live
|
||||
port: 8000
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 30
|
||||
periodSeconds: 20
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/health
|
||||
path: /api/diagnostic/healthz/ready
|
||||
port: 8000
|
||||
initialDelaySeconds: 10
|
||||
initialDelaySeconds: 15
|
||||
periodSeconds: 10
|
||||
```
|
||||
|
||||
| Endpoint | Auth | Purpose |
|
||||
|----------|------|---------|
|
||||
| `/api/diagnostic/healthz/live` | None | Lightweight liveness check — returns 200 if the process is running |
|
||||
| `/api/diagnostic/healthz/ready` | None | Readiness check — verifies database and Redis connectivity (503 when DB is down) |
|
||||
| `/api/diagnostic/health` | Required | Full health status for monitoring dashboards (Grafana, Uptime Kuma) |
|
||||
|
||||
### Prometheus Scraping
|
||||
|
||||
Add annotations to expose metrics (if using a Prometheus-compatible exporter):
|
||||
|
||||
+38
-15
@@ -34,7 +34,7 @@ Use this checklist to track readiness before going live.
|
||||
- [ ] **Redis** — Running and accessible only from internal network
|
||||
- [ ] **Meilisearch** — Running and accessible only from internal network
|
||||
- [ ] **Worker replicas** — At least 2 workers configured for redundancy
|
||||
- [ ] **Monitoring** — `/api/health` polled by uptime checker
|
||||
- [ ] **Monitoring** — `/api/diagnostic/health` polled by uptime checker
|
||||
- [ ] **Backups** — Automated backup of database, workdir, and Meilisearch data
|
||||
- [ ] **Log retention** — Logs shipped to a persistent store or aggregator
|
||||
- [ ] **Secrets management** — API keys not committed to source control
|
||||
@@ -285,22 +285,24 @@ For SSO/OIDC (Authentik, Keycloak, Auth0, etc.) see the [Authentication Setup Gu
|
||||
|
||||
### Docker Compose
|
||||
|
||||
Use the `deploy.replicas` setting (requires Docker Swarm mode) or simply run multiple workers:
|
||||
|
||||
```yaml
|
||||
worker:
|
||||
deploy:
|
||||
replicas: 3
|
||||
```
|
||||
|
||||
Or scale after deployment:
|
||||
Scale workers independently:
|
||||
|
||||
```bash
|
||||
docker-compose up -d --scale worker=3
|
||||
docker compose up -d --scale worker=3
|
||||
```
|
||||
|
||||
Each worker processes tasks from the Celery queue independently. Ensure the shared `workdir` volume is accessible from all worker containers.
|
||||
|
||||
> **Important:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. It is defined as a dedicated service in `docker-compose.yaml` with a fixed `container_name`. Do not scale it.
|
||||
|
||||
### Scaling the API
|
||||
|
||||
API pods are fully stateless (sessions use encrypted cookies, not server-side state) and can be scaled behind a load balancer:
|
||||
|
||||
```bash
|
||||
docker compose up -d --scale api=3
|
||||
```
|
||||
|
||||
### Kubernetes (Helm)
|
||||
|
||||
```yaml
|
||||
@@ -339,11 +341,32 @@ celery -A app.celery_worker worker -Q default,celery --concurrency=2
|
||||
|
||||
### Health Check Endpoint
|
||||
|
||||
DocuElevate exposes `/api/health` for readiness probing. Configure your uptime monitor to poll this endpoint:
|
||||
DocuElevate exposes three health-related endpoints:
|
||||
|
||||
| Endpoint | Auth | Purpose |
|
||||
|----------|------|---------|
|
||||
| `GET /api/diagnostic/healthz/live` | None | Lightweight liveness probe — returns 200 if the process is running |
|
||||
| `GET /api/diagnostic/healthz/ready` | None | Readiness probe — checks database and Redis (503 when DB is down) |
|
||||
| `GET /api/diagnostic/health` | Required | Full status for monitoring dashboards (Grafana, Uptime Kuma) |
|
||||
|
||||
For **Kubernetes probes**, use the unauthenticated endpoints:
|
||||
|
||||
```yaml
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /api/diagnostic/healthz/live
|
||||
port: 8000
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /api/diagnostic/healthz/ready
|
||||
port: 8000
|
||||
```
|
||||
|
||||
For **uptime monitors** (Uptime Kuma, Grafana, etc.), use the authenticated endpoint:
|
||||
|
||||
```bash
|
||||
curl http://docuelevate.example.com/api/health
|
||||
# Expected: {"status": "ok", ...}
|
||||
curl http://docuelevate.example.com/api/diagnostic/health
|
||||
# Expected: {"status": "healthy", ...}
|
||||
```
|
||||
|
||||
Set `UPTIME_KUMA_URL` to your Uptime Kuma push URL for heartbeat monitoring:
|
||||
@@ -502,4 +525,4 @@ For a dedicated Kubernetes deployment guide, including architecture diagrams, PV
|
||||
|
||||
- **Image Pull Policy**: Use `IfNotPresent` in production with pinned image tags (not `latest`) for reproducible deployments.
|
||||
|
||||
- **Liveness & Readiness Probes**: Already configured in the Helm chart via `/api/health`. Verify they are tuned to your startup time.
|
||||
- **Liveness & Readiness Probes**: Already configured in the Helm chart via unauthenticated endpoints (`/api/diagnostic/healthz/live` and `/api/diagnostic/healthz/ready`). Verify they are tuned to your startup time.
|
||||
|
||||
Reference in New Issue
Block a user