feat(tasks): add retry logic with exponential backoff and jitter

- Rewrite app/tasks/retry_config.py with compute_countdown() function
  implementing per-retry delays with ±20% jitter (default: 60s, 300s, 900s)
- Add BaseTaskWithRetry.retry() override to inject proper countdown
- Add OcrTaskWithRetry (120s, 600s, 1800s) for OCR/AI tasks
- Add UploadTaskWithRetry for cloud-storage upload tasks
- Add config settings: TASK_RETRY_MAX_RETRIES, TASK_RETRY_DELAYS, TASK_RETRY_JITTER
- Update process_with_ocr and process_with_azure tasks to use OcrTaskWithRetry
- Update all 11 upload tasks to use UploadTaskWithRetry
- Add 38 unit tests in tests/test_retry_config.py
- Update docs/ConfigurationGuide.md and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-01 17:37:41 +00:00
parent af3ff0b581
commit 5ff7b72a80
18 changed files with 642 additions and 29 deletions
+6
View File
@@ -16,6 +16,12 @@ ALLOW_FILE_DELETE=true # Allow deletion of file records
PROCESSALL_THROTTLE_THRESHOLD=20 # Number of files above which throttling is applied (default: 20)
PROCESSALL_THROTTLE_DELAY=3 # Delay in seconds between each task submission when throttling (default: 3)
# **Task Retry Settings**
# Failed tasks are automatically retried with exponential backoff and jitter.
# TASK_RETRY_MAX_RETRIES=3 # Max retry attempts per task (default: 3)
# TASK_RETRY_DELAYS=60,300,900 # Countdown (seconds) before each retry; 1 min, 5 min, 15 min
# TASK_RETRY_JITTER=true # Add ±20% random jitter to prevent thundering-herd (default: true)
# **Client-Side Upload Throttling**
# Controls pacing when the browser uploads files (especially large directory drops).
# The browser auto-detects rate-limit (HTTP 429) responses and backs off accordingly.