feat(tasks): add retry logic with exponential backoff and jitter
- Rewrite app/tasks/retry_config.py with compute_countdown() function implementing per-retry delays with ±20% jitter (default: 60s, 300s, 900s) - Add BaseTaskWithRetry.retry() override to inject proper countdown - Add OcrTaskWithRetry (120s, 600s, 1800s) for OCR/AI tasks - Add UploadTaskWithRetry for cloud-storage upload tasks - Add config settings: TASK_RETRY_MAX_RETRIES, TASK_RETRY_DELAYS, TASK_RETRY_JITTER - Update process_with_ocr and process_with_azure tasks to use OcrTaskWithRetry - Update all 11 upload tasks to use UploadTaskWithRetry - Add 38 unit tests in tests/test_retry_config.py - Update docs/ConfigurationGuide.md and .env.demo Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -15,7 +15,7 @@ from googleapiclient.http import MediaFileUpload
|
||||
|
||||
from app.celery_app import celery
|
||||
from app.config import settings
|
||||
from app.tasks.retry_config import BaseTaskWithRetry
|
||||
from app.tasks.retry_config import UploadTaskWithRetry
|
||||
from app.utils import log_task_progress
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -152,7 +152,7 @@ def truncate_property_value(key, value, max_bytes=100):
|
||||
return str_value
|
||||
|
||||
|
||||
@celery.task(base=BaseTaskWithRetry, bind=True)
|
||||
@celery.task(base=UploadTaskWithRetry, bind=True)
|
||||
def upload_to_google_drive(self, file_path: str, include_metadata=True, file_id: int = None):
|
||||
"""
|
||||
Uploads a file to Google Drive in the configured folder with optional metadata.
|
||||
|
||||
Reference in New Issue
Block a user