- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add health check endpoint at GET /api/diagnostic/health
- Auth-protected via @require_login (no-op when AUTH_ENABLED=False)
- Checks database (SELECT 1) and Redis (ping) with 2s timeouts
- Returns healthy/degraded/unhealthy with per-check detail
- Returns HTTP 503 when database is down, 200 otherwise
- 7 new unit tests covering all status scenarios
- Update docs/API.md with Grafana/monitoring integration notes
- Fixes test_cors_headers_absent_when_disabled CI timeout
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default `version` param from 'original' to 'processed' so
GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Both get_file_preview and download_file were ignoring the DB-stored
original_file_path and processed_file_path fields, instead relying on
local_filename (a temp path that may be gone) and guessing patterns for
the processed file. This caused "Processed file not found" and
potentially "Original file not found" in the /files/{id} view even when
the files existed at their stored paths.
- version=original: check original_file_path first, fall back to local_filename
- version=processed: check processed_file_path first, fall back to
hash/filename guessing patterns
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _compute_processing_flow to recognize process_with_ocr as the OCR
stage and remap legacy process_with_azure_document_intelligence log entries
for backward compatibility
- Normalize legacy OCR step name in _compute_step_summary log fallback
- Add process_with_ocr to REAL_MAIN_STEPS/REAL_STEPS in step_manager,
file_status, and file_queries (keeping legacy name for old DB entries)
- Update retry logic in api/files.py to retry failed OCR via process_with_ocr
(handles both step names as aliases)
- Fix process_document.py to log process_with_ocr as skipped (not azure step)
for the local text extraction path
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
A) Per-option Save Button
- Add per-setting Save button in settings.html (visible only when value changed)
- Button calls POST /api/settings/{key} directly; existing bulk Save retained
- Add Audit Log link in settings page header
B) Immediate Worker Sync
- New app/utils/settings_sync.py with notify_settings_updated() (Redis version key)
and register_settings_reload_signal() (Celery task_prerun handler)
- Register signal in celery_worker.py at startup
- All API write paths call notify_settings_updated() after successful saves
C) Audit Log
- Add SettingsAuditLog model (key, old_value, new_value, changed_by, changed_at, action)
- save_setting_to_db / delete_setting_from_db accept changed_by and write audit entries
- New get_audit_log() service function (masks sensitive values)
- New GET /api/settings/audit-log endpoint (admin-only)
- New GET /admin/settings/audit-log view + audit_log.html template
- Visible to all admins (per clarified requirement)
D) Config Rollback / History
- New get_setting_history() and rollback_setting() service functions
- New GET /api/settings/{key}/history endpoint
- New POST /api/settings/{key}/rollback/{history_id} endpoint
- Rollback buttons in audit_log.html with confirmation dialog
- Tests: 25 new tests covering audit log, rollback, worker sync helpers, and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
`GET /api/settings/{key}` was calling `validate_setting_key()` which raises
HTTP 404 for keys not in SETTING_METADATA. The test expects 200 with value=None
for unknown keys.
Added `validate_setting_key_format()` to `input_validation.py` that validates
only the key format without the SETTING_METADATA existence check. Updated
`get_setting` to use the format-only validator; POST/DELETE endpoints continue
using the full `validate_setting_key()` for write-side security.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add RequestSizeLimitMiddleware that checks Content-Length header
before request body is read: non-multipart requests capped at
MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
in 64 KB chunks (bounded memory usage), removing the post-write
os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
the now-unused os.path.getsize mock
Closes#173
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive logging to all pipeline retry branches (process_document, process_with_azure_document_intelligence, extract_metadata_with_gpt, embed_metadata_into_pdf)
- Add original_file_path as 3rd fallback for embed_metadata_into_pdf (checks: local_filename, processed_file_path, original_file_path, workdir/tmp fallback)
- Include all checked paths with existence status in 400 error responses for easier debugging
- Add enhanced logging to upload task retry path showing which processed file paths were checked
- Log successful file path when found
- Add comprehensive test suite covering new logging and fallback behavior
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _retry_pipeline_step to check for file in tmp, processed, and fallback locations
- Pass full path to extract_metadata_with_gpt instead of just basename
- Update extract_metadata_with_gpt to handle both basename and full path parameters
- Add test case for retrying when file is in processed directory
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix PLW2901: Use different variable name for stripped lines in loop
- Fix E721: Use 'is' instead of '==' for type comparisons
- Add noqa comments for intentional security warnings (S321, S507, S110, S603)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]