- Remove duplicate record creation to avoid UNIQUE constraint on filehash
- When duplicate detected, return original file_id instead of creating new record
- Avoids sqlite3.IntegrityError: UNIQUE constraint failed
- Simpler approach: duplicates not tracked as separate records, just rejected
- Revert filehash column back to NOT NULL (required for original files)
- Fixes error: (sqlite3.IntegrityError) UNIQUE constraint failed: files.filehash
- Implement lightweight migration system for is_duplicate and duplicate_of_id columns
- Migrations run automatically on application startup
- Idempotent migrations safe to run multiple times
- Fixes SQLite OperationalError for missing columns
- Resolves issue where database schema didn't match model definitions
- Add enable_deduplication and show_deduplication_step config options
- Rename hash_file step to check_for_duplicates
- Make deduplication step conditional based on configuration
- Add is_duplicate and duplicate_of_id fields to FileRecord model
- Create database migration for new deduplication fields
- Update process_document task to log deduplication results
- Update step visualization to show/hide step based on config
- Update status calculations to include deduplication step conditionally
- Default: deduplication enabled, step displayed
- Can be configured to hide from UI while still processing
- Added endpoints for previewing original and processed PDF files.
- Implemented on-demand text extraction from original and processed PDFs.
- Updated file detail page to show original and processed file paths with existence status.
- Introduced GPT metadata display with a collapsible JSON view.
- Enhanced front-end with PDF.js for in-browser PDF rendering and improved user experience.
- Added integration tests for new features including metadata display and file previews.
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix _compute_status_from_logs to track latest status per unique step
- Fix _compute_step_summary to count only latest status per step
- Add comprehensive tests for both fixes
- Resolves issue where completed files showed as "Processing"
- Resolves issue where metrics showed incorrect counts
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add @staticmethod to 9 test methods in test_external_integrations.py
that don't use self (PYL-R0201)
- Extract hard-coded password literals to constants in 6 test files
to resolve S2068 warnings (fixtures_integration, test_imap_tasks,
test_upload_tasks, test_upload_webdav_comprehensive,
test_upload_webdav_integration, test_views_coverage)
- Migrate Form() dependency injection to Annotated type hints in
dropbox.py, google_drive.py, onedrive.py (Sonar fastapi convention)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update generate_build_metadata.sh to sync VERSION from latest git tag
- Change BUILD_DATE format from date-only to ISO 8601 with time (YYYY-MM-DDTHH:MM:SSZ)
- Fix VERSION file from 0.5.0 to 0.9.1 (matching latest git tag v0.9.1)
- Change version fallback from hardcoded '0.5.0-dev' to 'unknown' in config.py
- Update release.yml to commit all build metadata files (not just VERSION)
- Update BuildMetadata.md documentation to reflect automated versioning
- Add tests for build_date with time format and version unknown fallback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The @require_login decorator expects a Starlette Request as the first
parameter to access request.session, but process_url only had a
URLUploadRequest Pydantic model parameter. This caused an
AttributeError: 'URLUploadRequest' object has no attribute 'session'
when POST /api/process-url was called.
Fix: Add `request: Request` as the first parameter and rename the
Pydantic model parameter from `request` to `url_request`.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Extract duplicated PDF text extraction into _extract_text_from_pdf helper
- Clarify empty metadata dict comment for embed_metadata_into_pdf retry
- Make test assertion for file_id passing more explicit
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add file_id parameter to process_document to skip duplicate hash check on reprocess
- Pass file_id from reprocess_single_file and bulk_reprocess_files endpoints
- Extend retry-subtask endpoint to support pipeline steps (process_document,
process_with_azure_document_intelligence, extract_metadata_with_gpt,
embed_metadata_into_pdf) in addition to upload tasks
- Add retry button for failed main pipeline steps in file detail UI
- Add comprehensive tests for reprocessing and pipeline step retry
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Created app/utils/file_queries.py with apply_status_filter function
- Updated app/api/files.py to use shared function
- Updated app/views/files.py to use shared function
- Removed unused 'or_' import from app/api/files.py
- Added comprehensive tests in tests/test_file_queries.py
- All tests pass (10 new tests, 14 existing tests verified)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Security headers are now disabled by default since most deployments use a reverse proxy (Traefik, Nginx) that already adds these headers. Enable with SECURITY_HEADERS_ENABLED=true for direct deployments.
Changes:
- Set security_headers_enabled default to False in app/config.py
- Update all documentation to reflect new default
- Comment out examples in .env.demo (now showing disabled state)
- Update SECURITY_AUDIT.md to reflect reverse proxy as default deployment
- Tests still pass (3 passed, 8 skipped as expected with headers disabled)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add prominent documentation that splitting uses page boundaries
- Update docstring with IMPORTANT note about page-level splitting
- Add test to validate split PDFs are valid and readable
- Update ConfigurationGuide.md to emphasize page-based approach
- Update SECURITY_AUDIT.md with implementation details
- Ensures users understand no risk of corrupted PDFs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Use BytesIO for size checking instead of temporary disk writes (major performance improvement)
- Add constant and comment for PDF overhead multiplier in tests
- Address code review feedback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add MAX_UPLOAD_SIZE config (default 1GB) to prevent resource exhaustion
- Add MAX_SINGLE_FILE_SIZE config for optional PDF file splitting
- Implement automatic PDF splitting when files exceed single file limit
- Update upload endpoint to use configured limits instead of hardcoded 500MB
- Add comprehensive tests for upload limits and file splitting
- Document configuration in ConfigurationGuide.md and SECURITY_AUDIT.md
- Reference SECURITY_AUDIT.md in error messages for user guidance
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Christian Krakau-Louis