- Fix _get_pipeline_ocr_language: remove redundant `or None` in step_config.get()
- Add Session type hint to _get_pipeline_ocr_language db parameter via TYPE_CHECKING
- Update process_with_ocr to use modern str | None syntax instead of Optional[str]
- Fix test_get_pipeline_ocr_language_explicit_pipeline_takes_priority: properly add
sys_step to db_session so the system pipeline step is persisted in the test DB
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add OCR_LANGUAGES constant (28 languages, EN/DE/FR/ES/IT/PT/RU/ZH/JA/KO/AR/etc.)
- Add TESSERACT_TO_EASYOCR mapping for automatic code translation
- Add optional language constructor arg to TesseractOCRProvider/EasyOCRProvider
- Update get_ocr_providers() to accept and pass per-call language override
- Add language parameter to process_with_ocr Celery task
- Add _get_pipeline_ocr_language() helper to resolve OCR language from pipeline step config
- Update process_document to look up and pass pipeline OCR language to process_with_ocr
- Add ocr_language select config field (28 options) to pipeline OCR step schema
- Add language dropdown to pipeline UI (pipelines.html)
- Update docs/UserGuide.md and docs/API.md with language override documentation
- Add 27 new tests covering language constants, provider overrides, and pipeline lookup
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.
- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
completed filter
- process_document: call initialize_file_steps after creating a new
file record so all mandatory steps are pre-created as pending
Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.
Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Raise quality acceptance threshold from 65→85 (configurable via TEXT_QUALITY_THRESHOLD)
- Reject text with significant issues (excessive_typos, garbage_characters,
incoherent_text, fragmented_sentences) even when score is above threshold
(configurable via TEXT_QUALITY_SIGNIFICANT_ISSUES)
- Add compare_text_quality() for AI-powered head-to-head comparison of
original embedded text vs fresh OCR output
- Update process_document to pass original text to OCR task for comparison
- Update process_with_ocr to run comparison and keep the higher-quality text
- Add new settings to settings_service.py metadata
- Update docs/ConfigurationGuide.md with new settings
- Add comprehensive tests for new threshold and comparison logic
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _compute_processing_flow to recognize process_with_ocr as the OCR
stage and remap legacy process_with_azure_document_intelligence log entries
for backward compatibility
- Normalize legacy OCR step name in _compute_step_summary log fallback
- Add process_with_ocr to REAL_MAIN_STEPS/REAL_STEPS in step_manager,
file_status, and file_queries (keeping legacy name for old DB entries)
- Update retry logic in api/files.py to retry failed OCR via process_with_ocr
(handles both step names as aliases)
- Fix process_document.py to log process_with_ocr as skipped (not azure step)
for the local text extraction path
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Implement step timeout detection to prevent files from getting stuck in 'pending' state
- Add monitor_stalled_steps periodic task running every minute (Celery Beat)
- Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes)
- Add step_timeout configuration setting (default: 600 seconds)
- Recover stalled steps with error message indicating when timeout was triggered
- Fix duplicate check to exclude self-comparison (file not duplicate of itself)
When processing crashes or hangs:
1. Worker detects stalled steps (in_progress for >10 minutes)
2. Marks them as failed with timeout error message
3. Updates UI to show failure status
4. Allows file to be retried or handled by user
This prevents files from being indefinitely stuck in processing state and provides
visibility into what went wrong.
- Remove duplicate record creation to avoid UNIQUE constraint on filehash
- When duplicate detected, return original file_id instead of creating new record
- Avoids sqlite3.IntegrityError: UNIQUE constraint failed
- Simpler approach: duplicates not tracked as separate records, just rejected
- Revert filehash column back to NOT NULL (required for original files)
- Fixes error: (sqlite3.IntegrityError) UNIQUE constraint failed: files.filehash
- Add enable_deduplication and show_deduplication_step config options
- Rename hash_file step to check_for_duplicates
- Make deduplication step conditional based on configuration
- Add is_duplicate and duplicate_of_id fields to FileRecord model
- Create database migration for new deduplication fields
- Update process_document task to log deduplication results
- Update step visualization to show/hide step based on config
- Update status calculations to include deduplication step conditionally
- Default: deduplication enabled, step displayed
- Can be configured to hide from UI while still processing
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add file_id parameter to process_document to skip duplicate hash check on reprocess
- Pass file_id from reprocess_single_file and bulk_reprocess_files endpoints
- Extend retry-subtask endpoint to support pipeline steps (process_document,
process_with_azure_document_intelligence, extract_metadata_with_gpt,
embed_metadata_into_pdf) in addition to upload tasks
- Add retry button for failed main pipeline steps in file detail UI
- Add comprehensive tests for reprocessing and pipeline step retry
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Added database logging to all major processing tasks
- Created API endpoints for retrieving processing logs
- Updated frontend to display processing logs per file
- Logging includes: process_document, convert_to_pdf, extract_metadata_with_gpt, embed_metadata_into_pdf, finalize_document_storage, send_to_all_destinations
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Rename upload_to_s3.py to process_document.py to better reflect its purpose
- Update all import statements across the codebase to use new module name
- Remove S3-specific code and references
- Keep the core document processing logic intact
- Update docstrings and comments to reflect new functionality
This change is part of removing AWS S3 dependencies and simplifying the
document processing pipeline.
copilot-swe-agent[bot]