- Implement step timeout detection to prevent files from getting stuck in 'pending' state
- Add monitor_stalled_steps periodic task running every minute (Celery Beat)
- Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes)
- Add step_timeout configuration setting (default: 600 seconds)
- Recover stalled steps with error message indicating when timeout was triggered
- Fix duplicate check to exclude self-comparison (file not duplicate of itself)
When processing crashes or hangs:
1. Worker detects stalled steps (in_progress for >10 minutes)
2. Marks them as failed with timeout error message
3. Updates UI to show failure status
4. Allows file to be retried or handled by user
This prevents files from being indefinitely stuck in processing state and provides
visibility into what went wrong.
- Remove duplicate record creation to avoid UNIQUE constraint on filehash
- When duplicate detected, return original file_id instead of creating new record
- Avoids sqlite3.IntegrityError: UNIQUE constraint failed
- Simpler approach: duplicates not tracked as separate records, just rejected
- Revert filehash column back to NOT NULL (required for original files)
- Fixes error: (sqlite3.IntegrityError) UNIQUE constraint failed: files.filehash
- Add enable_deduplication and show_deduplication_step config options
- Rename hash_file step to check_for_duplicates
- Make deduplication step conditional based on configuration
- Add is_duplicate and duplicate_of_id fields to FileRecord model
- Create database migration for new deduplication fields
- Update process_document task to log deduplication results
- Update step visualization to show/hide step based on config
- Update status calculations to include deduplication step conditionally
- Default: deduplication enabled, step displayed
- Can be configured to hide from UI while still processing
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add file_id parameter to process_document to skip duplicate hash check on reprocess
- Pass file_id from reprocess_single_file and bulk_reprocess_files endpoints
- Extend retry-subtask endpoint to support pipeline steps (process_document,
process_with_azure_document_intelligence, extract_metadata_with_gpt,
embed_metadata_into_pdf) in addition to upload tasks
- Add retry button for failed main pipeline steps in file detail UI
- Add comprehensive tests for reprocessing and pipeline step retry
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Added database logging to all major processing tasks
- Created API endpoints for retrieving processing logs
- Updated frontend to display processing logs per file
- Logging includes: process_document, convert_to_pdf, extract_metadata_with_gpt, embed_metadata_into_pdf, finalize_document_storage, send_to_all_destinations
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Rename upload_to_s3.py to process_document.py to better reflect its purpose
- Update all import statements across the codebase to use new module name
- Remove S3-specific code and references
- Keep the core document processing logic intact
- Update docstrings and comments to reflect new functionality
This change is part of removing AWS S3 dependencies and simplifying the
document processing pipeline.
copilot-swe-agent[bot]