feat(step-timeout): add automatic recovery for stalled processing steps
- Implement step timeout detection to prevent files from getting stuck in 'pending' state - Add monitor_stalled_steps periodic task running every minute (Celery Beat) - Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes) - Add step_timeout configuration setting (default: 600 seconds) - Recover stalled steps with error message indicating when timeout was triggered - Fix duplicate check to exclude self-comparison (file not duplicate of itself) When processing crashes or hangs: 1. Worker detects stalled steps (in_progress for >10 minutes) 2. Marks them as failed with timeout error message 3. Updates UI to show failure status 4. Allows file to be retried or handled by user This prevents files from being indefinitely stuck in processing state and provides visibility into what went wrong.
This commit is contained in:
@@ -189,6 +189,12 @@ class Settings(BaseSettings):
|
||||
description="Show the 'Check for Duplicates' step in processing history. If False, the check is still performed but not displayed. Default: True.",
|
||||
)
|
||||
|
||||
# Processing step timeout - prevents files from getting stuck in "in_progress" state
|
||||
step_timeout: int = Field(
|
||||
default=600,
|
||||
description="Timeout in seconds for processing steps. If a step is 'in_progress' for longer than this, it will be marked as failed. Default: 600 seconds (10 minutes).",
|
||||
)
|
||||
|
||||
# Security Headers Configuration (see SECURITY_AUDIT.md and docs/DeploymentGuide.md)
|
||||
# Disabled by default since most deployments use a reverse proxy (Traefik, Nginx, etc.)
|
||||
# that already adds these headers. Enable if deploying directly without a reverse proxy.
|
||||
|
||||
Reference in New Issue
Block a user