feat(step-timeout): add automatic recovery for stalled processing steps
- Implement step timeout detection to prevent files from getting stuck in 'pending' state - Add monitor_stalled_steps periodic task running every minute (Celery Beat) - Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes) - Add step_timeout configuration setting (default: 600 seconds) - Recover stalled steps with error message indicating when timeout was triggered - Fix duplicate check to exclude self-comparison (file not duplicate of itself) When processing crashes or hangs: 1. Worker detects stalled steps (in_progress for >10 minutes) 2. Marks them as failed with timeout error message 3. Updates UI to show failure status 4. Allows file to be retried or handled by user This prevents files from being indefinitely stuck in processing state and provides visibility into what went wrong.
This commit is contained in:
@@ -33,6 +33,7 @@ from app.tasks.upload_to_s3 import upload_to_s3 # noqa: F401
|
||||
from app.tasks.upload_to_sftp import upload_to_sftp # noqa: F401
|
||||
from app.tasks.upload_to_webdav import upload_to_webdav # noqa: F401
|
||||
from app.tasks.uptime_kuma_tasks import ping_uptime_kuma # noqa: F401
|
||||
from app.tasks.monitor_stalled_steps import monitor_stalled_steps # noqa: F401
|
||||
|
||||
celery.conf.task_routes = {
|
||||
"app.tasks.*": {"queue": "default"},
|
||||
@@ -79,6 +80,12 @@ celery.conf.beat_schedule = {
|
||||
"schedule": crontab(hour="0", minute="0"), # Midnight
|
||||
"options": {"expires": 3600}, # 1 hour expiry
|
||||
},
|
||||
# Monitor for stalled processing steps every minute
|
||||
"monitor-stalled-steps": {
|
||||
"task": "app.tasks.monitor_stalled_steps.monitor_stalled_steps",
|
||||
"schedule": crontab(minute="*/1"), # Every minute
|
||||
"options": {"expires": 55}, # Must complete within 55 seconds
|
||||
},
|
||||
}
|
||||
|
||||
# Remove None entries from beat_schedule
|
||||
|
||||
Reference in New Issue
Block a user