The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Both get_file_preview and download_file were ignoring the DB-stored
original_file_path and processed_file_path fields, instead relying on
local_filename (a temp path that may be gone) and guessing patterns for
the processed file. This caused "Processed file not found" and
potentially "Original file not found" in the /files/{id} view even when
the files existed at their stored paths.
- version=original: check original_file_path first, fall back to local_filename
- version=processed: check processed_file_path first, fall back to
hash/filename guessing patterns
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _compute_processing_flow to recognize process_with_ocr as the OCR
stage and remap legacy process_with_azure_document_intelligence log entries
for backward compatibility
- Normalize legacy OCR step name in _compute_step_summary log fallback
- Add process_with_ocr to REAL_MAIN_STEPS/REAL_STEPS in step_manager,
file_status, and file_queries (keeping legacy name for old DB entries)
- Update retry logic in api/files.py to retry failed OCR via process_with_ocr
(handles both step names as aliases)
- Fix process_document.py to log process_with_ocr as skipped (not azure step)
for the local text extraction path
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
A) Per-option Save Button
- Add per-setting Save button in settings.html (visible only when value changed)
- Button calls POST /api/settings/{key} directly; existing bulk Save retained
- Add Audit Log link in settings page header
B) Immediate Worker Sync
- New app/utils/settings_sync.py with notify_settings_updated() (Redis version key)
and register_settings_reload_signal() (Celery task_prerun handler)
- Register signal in celery_worker.py at startup
- All API write paths call notify_settings_updated() after successful saves
C) Audit Log
- Add SettingsAuditLog model (key, old_value, new_value, changed_by, changed_at, action)
- save_setting_to_db / delete_setting_from_db accept changed_by and write audit entries
- New get_audit_log() service function (masks sensitive values)
- New GET /api/settings/audit-log endpoint (admin-only)
- New GET /admin/settings/audit-log view + audit_log.html template
- Visible to all admins (per clarified requirement)
D) Config Rollback / History
- New get_setting_history() and rollback_setting() service functions
- New GET /api/settings/{key}/history endpoint
- New POST /api/settings/{key}/rollback/{history_id} endpoint
- Rollback buttons in audit_log.html with confirmation dialog
- Tests: 25 new tests covering audit log, rollback, worker sync helpers, and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
`GET /api/settings/{key}` was calling `validate_setting_key()` which raises
HTTP 404 for keys not in SETTING_METADATA. The test expects 200 with value=None
for unknown keys.
Added `validate_setting_key_format()` to `input_validation.py` that validates
only the key format without the SETTING_METADATA existence check. Updated
`get_setting` to use the format-only validator; POST/DELETE endpoints continue
using the full `validate_setting_key()` for write-side security.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add RequestSizeLimitMiddleware that checks Content-Length header
before request body is read: non-multipart requests capped at
MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
in 64 KB chunks (bounded memory usage), removing the post-write
os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
the now-unused os.path.getsize mock
Closes#173
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive logging to all pipeline retry branches (process_document, process_with_azure_document_intelligence, extract_metadata_with_gpt, embed_metadata_into_pdf)
- Add original_file_path as 3rd fallback for embed_metadata_into_pdf (checks: local_filename, processed_file_path, original_file_path, workdir/tmp fallback)
- Include all checked paths with existence status in 400 error responses for easier debugging
- Add enhanced logging to upload task retry path showing which processed file paths were checked
- Log successful file path when found
- Add comprehensive test suite covering new logging and fallback behavior
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _retry_pipeline_step to check for file in tmp, processed, and fallback locations
- Pass full path to extract_metadata_with_gpt instead of just basename
- Update extract_metadata_with_gpt to handle both basename and full path parameters
- Add test case for retrying when file is in processed directory
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix PLW2901: Use different variable name for stripped lines in loop
- Fix E721: Use 'is' instead of '==' for type comparisons
- Add noqa comments for intentional security warnings (S321, S507, S110, S603)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add @staticmethod to 9 test methods in test_external_integrations.py
that don't use self (PYL-R0201)
- Extract hard-coded password literals to constants in 6 test files
to resolve S2068 warnings (fixtures_integration, test_imap_tasks,
test_upload_tasks, test_upload_webdav_comprehensive,
test_upload_webdav_integration, test_views_coverage)
- Migrate Form() dependency injection to Annotated type hints in
dropbox.py, google_drive.py, onedrive.py (Sonar fastapi convention)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The @require_login decorator expects a Starlette Request as the first
parameter to access request.session, but process_url only had a
URLUploadRequest Pydantic model parameter. This caused an
AttributeError: 'URLUploadRequest' object has no attribute 'session'
when POST /api/process-url was called.
Fix: Add `request: Request` as the first parameter and rename the
Pydantic model parameter from `request` to `url_request`.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Extract duplicated PDF text extraction into _extract_text_from_pdf helper
- Clarify empty metadata dict comment for embed_metadata_into_pdf retry
- Make test assertion for file_id passing more explicit
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add file_id parameter to process_document to skip duplicate hash check on reprocess
- Pass file_id from reprocess_single_file and bulk_reprocess_files endpoints
- Extend retry-subtask endpoint to support pipeline steps (process_document,
process_with_azure_document_intelligence, extract_metadata_with_gpt,
embed_metadata_into_pdf) in addition to upload tasks
- Add retry button for failed main pipeline steps in file detail UI
- Add comprehensive tests for reprocessing and pipeline step retry
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Created app/utils/file_queries.py with apply_status_filter function
- Updated app/api/files.py to use shared function
- Updated app/views/files.py to use shared function
- Removed unused 'or_' import from app/api/files.py
- Added comprehensive tests in tests/test_file_queries.py
- All tests pass (10 new tests, 14 existing tests verified)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add MAX_UPLOAD_SIZE config (default 1GB) to prevent resource exhaustion
- Add MAX_SINGLE_FILE_SIZE config for optional PDF file splitting
- Implement automatic PDF splitting when files exceed single file limit
- Update upload endpoint to use configured limits instead of hardcoded 500MB
- Add comprehensive tests for upload limits and file splitting
- Document configuration in ConfigurationGuide.md and SECURITY_AUDIT.md
- Reference SECURITY_AUDIT.md in error messages for user guidance
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Import and use sanitize_filename utility in ui_upload endpoint
- Enhance sanitize_filename to handle Windows-style paths (backslashes)
- Add protection against path traversal patterns (..)
- Replace all path separators with underscores
- Add comprehensive security tests for Windows-style paths and mixed separators
- All existing tests pass with improved security
This addresses the "Uncontrolled data used in path expression" code scanning alert
by ensuring all user-provided filenames are properly sanitized before being used
in any file operations or stored in the database.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]