Merge pull request #448 from christianlouis/copilot/add-retry-logic-backoff

docs/API.md

@@ -766,6 +766,77 @@ Lightweight endpoint returning the total number of queued + in-progress items. D
 }
 ```
 
+## Diagnostic
+
+### GET /api/diagnostic/health
+
+System health endpoint designed for monitoring tools such as Grafana, Uptime Kuma, Prometheus blackbox exporter, or any HTTP-based health checker.
+
+Checks the database and Redis connectivity and returns a machine-readable JSON summary.
+
+**Authentication:** Required (bypassed when `AUTH_ENABLED=False`)
+
+**Response (200 OK) – all subsystems healthy:**
+```json
+{
+  "status": "healthy",
+  "version": "1.2.3",
+  "timestamp": "2024-01-15T10:30:00+00:00",
+  "checks": {
+    "database": {"status": "ok"},
+    "redis":    {"status": "ok"}
+  }
+}
+```
+
+**Response (200 OK) – one or more non-critical checks failed:**
+```json
+{
+  "status": "degraded",
+  "version": "1.2.3",
+  "timestamp": "2024-01-15T10:30:00+00:00",
+  "checks": {
+    "database": {"status": "ok"},
+    "redis":    {"status": "error", "detail": "Connection refused"}
+  }
+}
+```
+
+**Response (503 Service Unavailable) – critical check (database) failed:**
+```json
+{
+  "status": "unhealthy",
+  "version": "1.2.3",
+  "timestamp": "2024-01-15T10:30:00+00:00",
+  "checks": {
+    "database": {"status": "error", "detail": "..."},
+    "redis":    {"status": "ok"}
+  }
+}
+```
+
+The `status` field is always one of:
+- `"healthy"` – all checks passed
+- `"degraded"` – at least one non-critical check failed (Redis unavailable)
+- `"unhealthy"` – a critical check failed (database unavailable); HTTP 503 is returned
+
+**Grafana / Uptime Kuma integration:** point your health check at `GET /api/diagnostic/health` and check for HTTP 200 or the JSON `status` field.
+
+### POST /api/diagnostic/test-notification
+
+Send a test notification through all configured notification channels.
+
+**Authentication:** Required
+
+**Response (200 OK):**
+```json
+{
+  "status": "success",
+  "message": "Test notification sent successfully to 2 service(s)",
+  "services_count": 2
+}
+```
+
 ## Rate Limiting
 
 The API implements rate limiting to ensure system stability. If you exceed the limits, you'll receive a `429 Too Many Requests` response.

docs/ConfigurationGuide.md

@@ -31,6 +31,40 @@ Control how the `/processall` endpoint handles large batches of files to prevent
 - Total queue time: (25-1) × 3 = 72 seconds
 - Prevents API rate limit issues and ensures smooth processing
 
+### Task Retry Settings
+
+Failed Celery tasks are automatically retried with exponential backoff and optional jitter. Different task types use different default delays (OCR tasks wait longer than upload tasks to account for API rate limits).
+
+| **Variable**             | **Description**                                                                                                                                   | **Default**     |
+|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
+| `TASK_RETRY_MAX_RETRIES` | Maximum number of retry attempts for any failed task.                                                                                             | `3`             |
+| `TASK_RETRY_DELAYS`      | Comma-separated list of countdown values in seconds for each retry attempt. Values beyond the list double the last entry for subsequent retries. | `60,300,900`    |
+| `TASK_RETRY_JITTER`      | Apply ±20 % random jitter to countdowns to prevent thundering-herd problems when many tasks fail at the same time.                               | `true`          |
+
+**Per-task-type policies** (not overridable via environment variables; set in code):
+
+| Task type               | Default delays (s)     | Notes                                               |
+|-------------------------|------------------------|-----------------------------------------------------|
+| General tasks           | 60, 300, 900           | Controlled by `TASK_RETRY_DELAYS`                   |
+| OCR / AI tasks          | 120, 600, 1800         | Longer waits for API rate-limit windows to clear    |
+| Cloud-storage uploads   | 60, 300, 900           | Controlled by `TASK_RETRY_DELAYS`                   |
+
+**Example – aggressive retries for a high-availability setup:**
+
+```dotenv
+TASK_RETRY_MAX_RETRIES=5
+TASK_RETRY_DELAYS=30,120,600,1800,3600
+TASK_RETRY_JITTER=true
+```
+
+**Example – conservative retries with longer back-off:**
+
+```dotenv
+TASK_RETRY_MAX_RETRIES=3
+TASK_RETRY_DELAYS=300,900,3600
+TASK_RETRY_JITTER=true
+```
+
 ### Client-Side Upload Throttling
 
 Control how the web UI queues and paces file uploads to avoid overwhelming the backend, especially when dragging large directories (potentially thousands of files) onto the upload area.