- Replace bare Exception catches with sa.exc.OperationalError in migration 007
- Use ScriptDirectory.get_heads() for dynamic revision ID assertions in tests
- Make migration count assertion flexible (>= 8 instead of == 8)
- Rewrite pending migrations test to actually test upgrade from revision 006
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add pytest-timeout>=2.3.0 to requirements-dev.txt with 120s global default
- Fix test_fallback_file_id_lookup: add missing _should_upload_* mocks that
caused .delay() calls to hang on Redis broker connection
- Split CI test job into two stages:
- Quick Tests (timeout: 10min, ~2min run): unit + basic integration
- Integration Tests (timeout: 20min): Docker containers, external services
- Quick tests gate integration tests for fast-fail feedback
- Build job now depends on both test stages
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default `version` param from 'original' to 'processed' so
GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Also updates UserGuide.md with documentation for pdf.js viewer,
image zoom/pan, text preview, and the file-list preview modal.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The terminal-step guard requires send_to_all_destinations: success before
a file is considered "completed". Update test_status_filter_completed to
include this step so it matches the new semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The previous fix (requiring send_to_all_destinations to be present
before marking a file as completed) broke 3 tests that used only
partial step sets and expected "completed":
- test_coverage_polish.py::TestFileQueriesDeduplicationEnabled::
test_deduplication_enabled_adds_check_for_duplicates
- test_file_listing.py::TestFileListingPagination::
test_processing_status_included
- test_file_listing.py::TestFileDetailEndpoint::
test_file_detail_status_determination
Add send_to_all_destinations: success to each test's dataset so
"completed" status is reached correctly under the new semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.
- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
completed filter
- process_document: call initialize_file_steps after creating a new
file record so all mandatory steps are pre-created as pending
Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.
Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The mock_oauth_server session fixture tried to pull ghcr.io/navikt/mock-oauth2-server:2.1.1
from Docker, which times out in sandboxed CI, causing all 14 OAuth integration tests to ERROR.
Changes to tests/conftest_oauth.py:
- mock_oauth_server: catch container startup exceptions, attempt cleanup, yield None
instead of propagating (static fallback config is used instead)
- oauth_config: add elif mock_oauth_server is None branch returning a static hardcoded
config (mode="static") using module-level URL constants
- oauth_enabled_app: use authorize_url/access_token_url directly (no HTTP metadata
discovery), clear/restore authlib _clients/_registry cache per test, add cleanup in teardown
- Extract _STATIC_OAUTH_* constants to avoid URL duplication
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The test was using patch.object(type(settings), "onedrive_client_id", property(...))
to make settings.onedrive_client_id raise. Pydantic v2 Settings fields are not plain
Python descriptors so this approach raises AttributeError.
Fix: patch app.api.onedrive.settings with a MagicMock whose onedrive_client_id
is a PropertyMock(side_effect=Exception), which correctly triggers the except
branch in get_onedrive_full_config and returns {"status": "error"}.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The test was patching os.path.join, which is called inside an inner
try/except block in save_onedrive_settings. This meant the exception
was silently caught and logged, never reaching the outer exception
handler that returns HTTP 500.
Fix by patching notify_settings_updated instead, which is called in
the outer try block, so exceptions correctly propagate to the outer
handler.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The debounceSearch() function in files.html called clearFullTextSearch()
when the query was shorter than 2 characters. Since clearFullTextSearch()
sets input.value = '', every single keystroke was immediately erased —
users could paste text but not type.
Fix: debounceSearch now only hides the results panel for short queries
without touching the input value.
Also adds a dedicated /search page with Google-style results showing
content previews (document title, filename, type badges, tag badges,
sender, and OCR text snippets with highlighted matches).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Replace module-level getattr patch with a MagicMock that has a property
raising on google_drive_use_oauth access, as suggested by code review.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change @patch("app.api.google_drive.get_google_drive_service") to
@patch("app.tasks.upload_to_google_drive.get_google_drive_service")
because the function is imported locally inside the endpoint function
body, not at module level
- Replace patch.object(type(settings), "google_drive_use_oauth", ...)
with patch("app.api.google_drive.getattr", ...) because Pydantic v2
models don't expose fields as regular class attributes
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Rewrite tests/test_celery_worker.py to mock check_credentials.apply_async
at import time, enabling all 35 statements to be covered without Redis
- Add tests for conditional beat schedule entries (IMAP, Uptime Kuma)
- Delete app/utils/config_validator.py — dead code shadowed by the
config_validator/ package directory (Python gives packages precedence)
- Remove both files from coverage omit in pyproject.toml
- celery_worker.py now at 100% coverage (was 0%)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add dismissable cookie notice banner to base template (essential
cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
covering cookie strategy, data transfer mechanisms, data subject
rights handling matrix with response timelines, and market-specific
notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
key compliance content areas
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Christian Krakau-Louis