- Add RequestSizeLimitMiddleware that checks Content-Length header
before request body is read: non-multipart requests capped at
MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
in 64 KB chunks (bounded memory usage), removing the post-write
os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
the now-unused os.path.getsize mock
Closes#173
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix ruff format: add blank line before nested function in app/views/filemanager.py
- Fix ruff format: use double-quote escaping in tests/test_config.py
- Remove deploy job (Portainer webhook) from ci.yml
- Add update-k8s-manifest job: after main-branch build, updates
apps/docuelevate/preprod/docuelevate-stack.yaml in christianlouis/k8s-cluster-state
with the new GHCR image tag (ghcr.io/christianlouis/docuelevate:main-<short-sha>)
using mikefarah/yq@v4.44.6 and GH_PAT secret for cross-repo write access
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add [[tool.mypy.overrides]] section for app/utils/** with disallow_untyped_defs=true
- Add type annotations to all functions in app/utils/ (12 files)
- Fix type annotations in app/config.py and app/database.py (imported by utils)
- All 85 source files now pass mypy type checking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive tests for imap_tasks.py covering:
- Lock acquire/release mechanisms
- pull_all_inboxes task with various scenarios
- pull_inbox for Gmail and non-Gmail with edge cases
- find_all_mail_xlist functionality
- Extended fetch_attachments tests for all MIME types
- Edge cases: invalid JSON, missing Message-ID, already processed, etc.
- Achieve 98.26% coverage for imap_tasks.py (up from 48.78%)
- Fix config_validator.py to include validate_auth_config export
- Update tests to verify all exports including validate_auth_config
- Note: config_validator.py file is shadowed by config_validator/ directory
in Python's module resolution, so it cannot be directly imported or tested.
The package's __init__.py (which has 100% coverage) is what's actually used.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive logging to all pipeline retry branches (process_document, process_with_azure_document_intelligence, extract_metadata_with_gpt, embed_metadata_into_pdf)
- Add original_file_path as 3rd fallback for embed_metadata_into_pdf (checks: local_filename, processed_file_path, original_file_path, workdir/tmp fallback)
- Include all checked paths with existence status in 400 error responses for easier debugging
- Add enhanced logging to upload task retry path showing which processed file paths were checked
- Log successful file path when found
- Add comprehensive test suite covering new logging and fallback behavior
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _retry_pipeline_step to check for file in tmp, processed, and fallback locations
- Pass full path to extract_metadata_with_gpt instead of just basename
- Update extract_metadata_with_gpt to handle both basename and full path parameters
- Add test case for retrying when file is in processed directory
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Move get_all_settings_from_db import to module level in app/views/settings.py
- Move get_provider_status and get_settings_for_display imports to module level in app/views/status.py
- Fix CI workflow: replace deprecated 'file' parameter with 'files' in codecov-action
- Fix CI workflow: update test results upload to use codecov-action@v5 with report_type instead of deprecated test-results-action@v1
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Replace isinstance(ftp, ftplib.FTP_TLS) with a boolean flag to avoid issues when FTP_TLS is mocked in tests. Also fix Google Drive test parameter passing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix PLW2901: Use different variable name for stripped lines in loop
- Fix E721: Use 'is' instead of '==' for type comparisons
- Add noqa comments for intentional security warnings (S321, S507, S110, S603)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Implement step timeout detection to prevent files from getting stuck in 'pending' state
- Add monitor_stalled_steps periodic task running every minute (Celery Beat)
- Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes)
- Add step_timeout configuration setting (default: 600 seconds)
- Recover stalled steps with error message indicating when timeout was triggered
- Fix duplicate check to exclude self-comparison (file not duplicate of itself)
When processing crashes or hangs:
1. Worker detects stalled steps (in_progress for >10 minutes)
2. Marks them as failed with timeout error message
3. Updates UI to show failure status
4. Allows file to be retried or handled by user
This prevents files from being indefinitely stuck in processing state and provides
visibility into what went wrong.
- Remove duplicate record creation to avoid UNIQUE constraint on filehash
- When duplicate detected, return original file_id instead of creating new record
- Avoids sqlite3.IntegrityError: UNIQUE constraint failed
- Simpler approach: duplicates not tracked as separate records, just rejected
- Revert filehash column back to NOT NULL (required for original files)
- Fixes error: (sqlite3.IntegrityError) UNIQUE constraint failed: files.filehash
- Implement lightweight migration system for is_duplicate and duplicate_of_id columns
- Migrations run automatically on application startup
- Idempotent migrations safe to run multiple times
- Fixes SQLite OperationalError for missing columns
- Resolves issue where database schema didn't match model definitions
- Add enable_deduplication and show_deduplication_step config options
- Rename hash_file step to check_for_duplicates
- Make deduplication step conditional based on configuration
- Add is_duplicate and duplicate_of_id fields to FileRecord model
- Create database migration for new deduplication fields
- Update process_document task to log deduplication results
- Update step visualization to show/hide step based on config
- Update status calculations to include deduplication step conditionally
- Default: deduplication enabled, step displayed
- Can be configured to hide from UI while still processing
- Added endpoints for previewing original and processed PDF files.
- Implemented on-demand text extraction from original and processed PDFs.
- Updated file detail page to show original and processed file paths with existence status.
- Introduced GPT metadata display with a collapsible JSON view.
- Enhanced front-end with PDF.js for in-browser PDF rendering and improved user experience.
- Added integration tests for new features including metadata display and file previews.
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix _compute_status_from_logs to track latest status per unique step
- Fix _compute_step_summary to count only latest status per step
- Add comprehensive tests for both fixes
- Resolves issue where completed files showed as "Processing"
- Resolves issue where metrics showed incorrect counts
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
github-actions[bot]