gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(tasks): add retry logic with exponential backoff and jitter

Browse Source 
- Rewrite app/tasks/retry_config.py with compute_countdown() function
  implementing per-retry delays with ±20% jitter (default: 60s, 300s, 900s)
- Add BaseTaskWithRetry.retry() override to inject proper countdown
- Add OcrTaskWithRetry (120s, 600s, 1800s) for OCR/AI tasks
- Add UploadTaskWithRetry for cloud-storage upload tasks
- Add config settings: TASK_RETRY_MAX_RETRIES, TASK_RETRY_DELAYS, TASK_RETRY_JITTER
- Update process_with_ocr and process_with_azure tasks to use OcrTaskWithRetry
- Update all 11 upload tasks to use UploadTaskWithRetry
- Add 38 unit tests in tests/test_retry_config.py
- Update docs/ConfigurationGuide.md and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-01 17:37:41 +00:00
parent af3ff0b581
commit 5ff7b72a80
18 changed files with 642 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ConfigurationGuide.md
+34
View File
@@ -31,6 +31,40 @@ Control how the `/processall` endpoint handles large batches of files to prevent
- Total queue time: (25-1) × 3 = 72 seconds
- Prevents API rate limit issues and ensures smooth processing
### Task Retry Settings
Failed Celery tasks are automatically retried with exponential backoff and optional jitter. Different task types use different default delays (OCR tasks wait longer than upload tasks to account for API rate limits).
| **Variable** | **Description** | **Default** |
|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
| `TASK_RETRY_MAX_RETRIES` | Maximum number of retry attempts for any failed task. | `3` |
| `TASK_RETRY_DELAYS` | Comma-separated list of countdown values in seconds for each retry attempt. Values beyond the list double the last entry for subsequent retries. | `60,300,900` |
| `TASK_RETRY_JITTER` | Apply ±20 % random jitter to countdowns to prevent thundering-herd problems when many tasks fail at the same time. | `true` |
**Per-task-type policies** (not overridable via environment variables; set in code):
| Task type | Default delays (s) | Notes |
|-------------------------|------------------------|-----------------------------------------------------|
| General tasks | 60, 300, 900 | Controlled by `TASK_RETRY_DELAYS` |
| OCR / AI tasks | 120, 600, 1800 | Longer waits for API rate-limit windows to clear |
| Cloud-storage uploads | 60, 300, 900 | Controlled by `TASK_RETRY_DELAYS` |
**Example – aggressive retries for a high-availability setup:**
```dotenv
TASK_RETRY_MAX_RETRIES=5
TASK_RETRY_DELAYS=30,120,600,1800,3600
TASK_RETRY_JITTER=true
```
**Example – conservative retries with longer back-off:**
```dotenv
TASK_RETRY_MAX_RETRIES=3
TASK_RETRY_DELAYS=300,900,3600
TASK_RETRY_JITTER=true
```
### Client-Side Upload Throttling
Control how the web UI queues and paces file uploads to avoid overwhelming the backend, especially when dragging large directories (potentially thousands of files) onto the upload area.