- Add dismissable cookie notice banner to base template (essential
cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
covering cookie strategy, data transfer mechanisms, data subject
rights handling matrix with response timelines, and market-specific
notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
key compliance content areas
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Raise quality acceptance threshold from 65→85 (configurable via TEXT_QUALITY_THRESHOLD)
- Reject text with significant issues (excessive_typos, garbage_characters,
incoherent_text, fragmented_sentences) even when score is above threshold
(configurable via TEXT_QUALITY_SIGNIFICANT_ISSUES)
- Add compare_text_quality() for AI-powered head-to-head comparison of
original embedded text vs fresh OCR output
- Update process_document to pass original text to OCR task for comparison
- Update process_with_ocr to run comparison and keep the higher-quality text
- Add new settings to settings_service.py metadata
- Update docs/ConfigurationGuide.md with new settings
- Add comprehensive tests for new threshold and comparison logic
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Root cause: ensure_ocr_languages_from_settings() only downloaded tessdata
when the 'tesseract' provider was active, but embed_text_layer() uses
ocrmypdf (which needs tessdata) as a fallback for ALL OCR providers.
- embed_text_layer(): call ensure_tesseract_languages(language) after
confirming ocrmypdf is on PATH, so language data is present before
ocrmypdf is invoked (prevents exit code 3 for fra/deu/etc.)
- ensure_ocr_languages_from_settings(): extend the condition from
'tesseract' in active_providers to also trigger when ocrmypdf is
on PATH, enabling proactive pre-download at startup for any config
- Tests: mock shutil.which and ensure_tesseract_languages in affected
test cases; rename azure-only test and add new test for ocrmypdf case
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- PDFs are now uploaded to Mistral Files API (POST /v1/files) and
processed via a signed document_url, resolving the 422 error caused
by passing data:application/pdf;base64,... to an image endpoint
- Images (JPEG/PNG/GIF/WEBP/BMP/TIFF) use base64 image_url directly
- Unsupported MIME types raise a clear ValueError
- Magic-byte fallback detects PDFs with no file extension
- Switches from openai chat completions to requests HTTP calls
- Adds helper method _upload_pdf_and_get_document()
- Adds TestMistralOCRProvider with 9 unit tests covering all paths
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
E) Wizard DB persistence + worker sync
- app/api/dropbox.py: save-settings persists to DB (primary); .env write
is now best-effort (no 500 on missing file); notify_settings_updated()
called; update-settings already done in previous commit
- app/api/google_drive.py: update-settings + save-settings both persist
to DB and call notify_settings_updated(); .env write remains best-effort
- app/api/onedrive.py: save-settings + update-settings persist to DB +
notify; test-token auto-refresh path persists rotated token via
SessionLocal + notifies; .env write is best-effort throughout
- app/views/wizard.py: setup-wizard POST calls notify_settings_updated()
when settings are saved; GET pre-fills fields from DB > ENV > default
with a source badge; new GET /setup/undo-skip route removes skip marker
F) ENV Exporter
- app/utils/settings_service.py: get_settings_for_export(db, source)
supports source=db (DB-only) and source=effective (full runtime config)
- app/api/settings.py: GET /api/settings/export-env admin-only endpoint
returns downloadable .env file; source= query param selects scope
- frontend/templates/settings.html: Export .env dropdown (DB / effective)
+ Setup Wizard button added alongside existing Audit Log button
G) Setup Wizard improvements
- frontend/templates/setup_wizard.html: inputs pre-filled with
current_value; DB/ENV/DEFAULT source badges; undo-skip messaging
- app/views/wizard.py: passes setup_skipped flag to template
Tests
- tests/test_wizard_db_persist.py: 28 tests across 7 classes covering
wizard DB persistence, undo-skip, ENV exporter service + endpoint
- tests/test_api_dropbox.py: updated two tests to match new best-effort
.env behavior (was: assert 500; now: assert 200)
- tests/test_api_onedrive_comprehensive.py: same for two OneDrive tests;
fixed settings singleton pollution by adding @patch("app.api.*.settings")
to all new wizard tests that call save/update endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- TestSetupWizardDbPersist: verify save_setting_to_db and notify_settings_updated called
- TestSetupWizardUndoSkip: verify skip marker removed and redirect
- TestDropboxSaveSettingsDbPersist: verify DB written even without .env
- TestGoogleDriveUpdateSettingsDbPersist: verify per-field DB persistence
- TestOneDriveSaveSettingsDbPersist: verify DB written without .env file
- TestGetSettingsForExport: unit tests for source=db and source=effective
- TestExportEnvEndpoint: admin-only, text/plain, content-disposition, 400 on invalid source
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
A) Per-option Save Button
- Add per-setting Save button in settings.html (visible only when value changed)
- Button calls POST /api/settings/{key} directly; existing bulk Save retained
- Add Audit Log link in settings page header
B) Immediate Worker Sync
- New app/utils/settings_sync.py with notify_settings_updated() (Redis version key)
and register_settings_reload_signal() (Celery task_prerun handler)
- Register signal in celery_worker.py at startup
- All API write paths call notify_settings_updated() after successful saves
C) Audit Log
- Add SettingsAuditLog model (key, old_value, new_value, changed_by, changed_at, action)
- save_setting_to_db / delete_setting_from_db accept changed_by and write audit entries
- New get_audit_log() service function (masks sensitive values)
- New GET /api/settings/audit-log endpoint (admin-only)
- New GET /admin/settings/audit-log view + audit_log.html template
- Visible to all admins (per clarified requirement)
D) Config Rollback / History
- New get_setting_history() and rollback_setting() service functions
- New GET /api/settings/{key}/history endpoint
- New POST /api/settings/{key}/rollback/{history_id} endpoint
- Rollback buttons in audit_log.html with confirmation dialog
- Tests: 25 new tests covering audit log, rollback, worker sync helpers, and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Two root causes identified and fixed:
1. tests/test_api_settings.py (TestListCredentials):
asyncio.get_event_loop().run_until_complete() raised RuntimeError in
Python 3.12 because test_api_auth_enabled.py's asyncio.run() sets the
current event loop to None on completion. Replace all 7 occurrences
with asyncio.run() which creates its own event loop each time.
2. tests/test_cors.py:
reload(app.config) replaced the app.config.settings singleton with a
new instance, so app modules holding the original reference no longer
saw patches applied to app.config.settings.X. This caused the
notification, OpenAI, and file-upload tests to behave as if unpatched.
Remove the redundant reload() calls — the tests only need a fresh
Settings(...) instance constructed with the env var already set.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add CORSMiddleware (disabled by default, enabled via CORS_ENABLED=true)
- Add cors_enabled, cors_allowed_origins, cors_allow_credentials,
cors_allowed_methods, cors_allowed_headers settings to config.py
- Add parse_comma_separated_list validator for CORS list env vars
- Insert CORS middleware between SessionMiddleware and ProxyHeaders
so preflight runs before CSRF/auth but after proxy-header processing
- Document CORS env vars in .env.demo with rationale for proxy-first approach
- Mark CORS TODO as completed in SECURITY_AUDIT.md
- Add tests/test_cors.py with 12 unit and integration tests
Closes#175
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Christian Krakau-Louis