When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default `version` param from 'original' to 'processed' so
GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Also updates UserGuide.md with documentation for pdf.js viewer,
image zoom/pan, text preview, and the file-list preview modal.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The terminal-step guard requires send_to_all_destinations: success before
a file is considered "completed". Update test_status_filter_completed to
include this step so it matches the new semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The previous fix (requiring send_to_all_destinations to be present
before marking a file as completed) broke 3 tests that used only
partial step sets and expected "completed":
- test_coverage_polish.py::TestFileQueriesDeduplicationEnabled::
test_deduplication_enabled_adds_check_for_duplicates
- test_file_listing.py::TestFileListingPagination::
test_processing_status_included
- test_file_listing.py::TestFileDetailEndpoint::
test_file_detail_status_determination
Add send_to_all_destinations: success to each test's dataset so
"completed" status is reached correctly under the new semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.
- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
completed filter
- process_document: call initialize_file_steps after creating a new
file record so all mandatory steps are pre-created as pending
Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.
Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The mock_oauth_server session fixture tried to pull ghcr.io/navikt/mock-oauth2-server:2.1.1
from Docker, which times out in sandboxed CI, causing all 14 OAuth integration tests to ERROR.
Changes to tests/conftest_oauth.py:
- mock_oauth_server: catch container startup exceptions, attempt cleanup, yield None
instead of propagating (static fallback config is used instead)
- oauth_config: add elif mock_oauth_server is None branch returning a static hardcoded
config (mode="static") using module-level URL constants
- oauth_enabled_app: use authorize_url/access_token_url directly (no HTTP metadata
discovery), clear/restore authlib _clients/_registry cache per test, add cleanup in teardown
- Extract _STATIC_OAUTH_* constants to avoid URL duplication
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The test was using patch.object(type(settings), "onedrive_client_id", property(...))
to make settings.onedrive_client_id raise. Pydantic v2 Settings fields are not plain
Python descriptors so this approach raises AttributeError.
Fix: patch app.api.onedrive.settings with a MagicMock whose onedrive_client_id
is a PropertyMock(side_effect=Exception), which correctly triggers the except
branch in get_onedrive_full_config and returns {"status": "error"}.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The test was patching os.path.join, which is called inside an inner
try/except block in save_onedrive_settings. This meant the exception
was silently caught and logged, never reaching the outer exception
handler that returns HTTP 500.
Fix by patching notify_settings_updated instead, which is called in
the outer try block, so exceptions correctly propagate to the outer
handler.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The debounceSearch() function in files.html called clearFullTextSearch()
when the query was shorter than 2 characters. Since clearFullTextSearch()
sets input.value = '', every single keystroke was immediately erased —
users could paste text but not type.
Fix: debounceSearch now only hides the results panel for short queries
without touching the input value.
Also adds a dedicated /search page with Google-style results showing
content previews (document title, filename, type badges, tag badges,
sender, and OCR text snippets with highlighted matches).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Replace module-level getattr patch with a MagicMock that has a property
raising on google_drive_use_oauth access, as suggested by code review.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change @patch("app.api.google_drive.get_google_drive_service") to
@patch("app.tasks.upload_to_google_drive.get_google_drive_service")
because the function is imported locally inside the endpoint function
body, not at module level
- Replace patch.object(type(settings), "google_drive_use_oauth", ...)
with patch("app.api.google_drive.getattr", ...) because Pydantic v2
models don't expose fields as regular class attributes
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Rewrite tests/test_celery_worker.py to mock check_credentials.apply_async
at import time, enabling all 35 statements to be covered without Redis
- Add tests for conditional beat schedule entries (IMAP, Uptime Kuma)
- Delete app/utils/config_validator.py — dead code shadowed by the
config_validator/ package directory (Python gives packages precedence)
- Remove both files from coverage omit in pyproject.toml
- celery_worker.py now at 100% coverage (was 0%)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add dismissable cookie notice banner to base template (essential
cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
covering cookie strategy, data transfer mechanisms, data subject
rights handling matrix with response timelines, and market-specific
notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
key compliance content areas
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Raise quality acceptance threshold from 65→85 (configurable via TEXT_QUALITY_THRESHOLD)
- Reject text with significant issues (excessive_typos, garbage_characters,
incoherent_text, fragmented_sentences) even when score is above threshold
(configurable via TEXT_QUALITY_SIGNIFICANT_ISSUES)
- Add compare_text_quality() for AI-powered head-to-head comparison of
original embedded text vs fresh OCR output
- Update process_document to pass original text to OCR task for comparison
- Update process_with_ocr to run comparison and keep the higher-quality text
- Add new settings to settings_service.py metadata
- Update docs/ConfigurationGuide.md with new settings
- Add comprehensive tests for new threshold and comparison logic
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Root cause: ensure_ocr_languages_from_settings() only downloaded tessdata
when the 'tesseract' provider was active, but embed_text_layer() uses
ocrmypdf (which needs tessdata) as a fallback for ALL OCR providers.
- embed_text_layer(): call ensure_tesseract_languages(language) after
confirming ocrmypdf is on PATH, so language data is present before
ocrmypdf is invoked (prevents exit code 3 for fra/deu/etc.)
- ensure_ocr_languages_from_settings(): extend the condition from
'tesseract' in active_providers to also trigger when ocrmypdf is
on PATH, enabling proactive pre-download at startup for any config
- Tests: mock shutil.which and ensure_tesseract_languages in affected
test cases; rename azure-only test and add new test for ocrmypdf case
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]