Commit Graph

178 Commits

Author SHA1 Message Date
Christian Krakau-Louis d5fb0da335 Merge pull request #639 from christianlouis/refactor-audit-logs-annotated-b008-6104828608171783200
🧹 Use Annotated pattern in audit_logs.py to resolve Ruff B008
2026-03-14 12:44:26 +01:00
copilot-swe-agent[bot] 1416d901d0 fix(i18n): persist language preference server-side, fix dropdown rendering and avatar 404
- Fix language dropdown in base.html by moving tojson data out of HTML
  attribute into a script tag (prevents quote conflicts breaking Alpine.js)
- Fix avatar placeholder 404 by correcting filename reference from
  avatar-placeholder.svg to default-avatar.svg
- Add session hydration from DB in _inject_global_context() so
  detect_language() uses the stored preference on every request
- Sync session and cookie in PATCH /api/profile when language changes
- Reload page after language change in profile to reflect new locale
- Add tests for session/cookie sync and DB hydration

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-14 10:30:50 +00:00
google-labs-jules[bot] f96fcbe6c9 style: use Annotated pattern in audit_logs.py to resolve Ruff B008
Refactor `app/api/audit_logs.py` to use the `Annotated` type hint pattern for FastAPI dependencies (`Depends`) and query parameters (`Query`).

- Resolves B008: Function-call in default argument.
- Improves code maintainability and readability by following modern FastAPI best practices.
- Maintains consistency with other modules in the codebase (e.g., `files.py`, `integrations.py`).
- No changes to API runtime behavior.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-14 09:42:23 +00:00
copilot-swe-agent[bot] f70fa65969 Merge branch 'main' into copilot/add-conditional-routing
Resolve conflicts in app/api/__init__.py and app/models.py.
Renumber migration 027_add_routing_rules → 035_add_routing_rules.
Fix migration chain: down_revision → 034_add_user_profile_settings.
Add PipelineRoutingRule to migrations/env.py.
2026-03-12 22:07:54 +00:00
Christian Krakau-Louis 4345d5128f Merge pull request #607 from christianlouis/copilot/add-user-settings-management
fix(tests): update whoami_handler call sites to pass required db argument
2026-03-12 22:44:13 +01:00
copilot-swe-agent[bot] 9583d6d96f fix(profile): address code review feedback - early size check, CSRF helper, test constants
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:10:46 +00:00
copilot-swe-agent[bot] b0d6f1ab60 feat(profile): add user self-service profile settings page and API
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:00:19 +00:00
copilot-swe-agent[bot] b483ed16fc fix(i18n): register _ in all Jinja2 template instances, sync translation files, fix pipelines.html
- Register _() translation function in app/auth.py, app/api/local_auth.py, and
  app/api/billing.py template env globals to fix jinja2.exceptions.UndefinedError
- Update test_settings_template_has_db_wizard_link to assert i18n key instead of
  literal "DB Wizard" (template now uses {{ _("settings.db_wizard_btn") }})
- Add 418 missing en.json keys to all 30 non-English translation files as English
  fallbacks to fix test_all_languages_have_same_keys
- Fix orphan </template> tag in pipelines.html by adding missing
  <template x-if="pipelineModal.saving"> opening tag (fixes djlint H025)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 12:46:11 +00:00
copilot-swe-agent[bot] 2e087731b9 fix(imap): address code review feedback on ingestion profiles
- Improve warning log in _resolve_categories_for_profile() to include
  exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
  category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
  full profiles documentation including category table and API reference

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:19:32 +00:00
copilot-swe-agent[bot] c9f554465d feat(imap): add ImapIngestionProfile model, API, migration and UI
Replaces the simple binary attachment_filter string with a full
ingestion profiles system:

- Add FILE_TYPE_CATEGORIES dict to allowed_types.py (6 categories:
  pdf, office, opendocument, text, web, images) + DEFAULT_CATEGORIES
  / ALL_CATEGORIES + get_allowed_types_for_categories() helper
- Add ImapIngestionProfile model (id, name, description, owner_id,
  allowed_categories JSON, is_builtin)
- Update UserImapAccount: replace attachment_filter string with
  profile_id FK to imap_ingestion_profiles
- Migration 033: creates profiles table, seeds 2 built-in profiles
  (Documents Only, All Files), migrates attachment_filter → profile_id
- New /api/imap-profiles/ CRUD endpoints (list, create, get, update,
  delete) with category validation
- Register imap_profiles router in app/api/__init__.py
- Update imap_tasks.py: replace attachment_filter string param with
  profile-based allowed_categories; add _resolve_categories_for_profile()
- Update imap_accounts.py API to use profile_id instead of attachment_filter
- Update imap_accounts view to pass profiles + categories to template
- Full UI overhaul: profiles panel + profile create/edit modal with
  category checkboxes; profile selector in account modal
- 17 new tests (141 total), all passing

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:14:19 +00:00
copilot-swe-agent[bot] 554bb21d32 feat(imap): add attachment type filter for IMAP ingestion
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.

- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
  `_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
  display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 00:59:54 +00:00
copilot-swe-agent[bot] 054af636e3 fix(migrations): merge main and rechain compliance migration as 031 after 030_add_mobile_devices
Merge main branch into compliance templates feature branch.

Main had advanced with migrations 027-030 (ensure_shared_links,
audit_logs, user_language_preference, mobile_devices) since this
branch forked. Our compliance migration was 027 with down_revision
026, which conflicted with main's 027_ensure_shared_links_table.

Changes:
- Merge main (including i18n, audit logs, mobile, GraphQL features)
- Resolve conflicts in app/api/__init__.py, app/models.py, tests/conftest.py
- Rename 027_add_compliance_templates → 031_add_compliance_templates
- Rechain: down_revision 026_add_scheduled_jobs → 030_add_mobile_devices
- Add ComplianceTemplate to migrations/env.py imports
- Alembic now has single head: 031_add_compliance_templates
2026-03-12 00:00:29 +00:00
copilot-swe-agent[bot] a41ded535f feat(api): add GraphQL endpoint at /graphql with Strawberry
- Add strawberry-graphql[fastapi] dependency
- Implement GraphQL schema covering documents, pipelines, settings, users
- Enable GraphiQL playground at /graphql
- Mount GraphQL router in main.py
- Add 24 tests for all query types and auth enforcement
- Update docs/API.md with GraphQL documentation section

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 22:17:25 +00:00
copilot-swe-agent[bot] 1d5eee4281 fix(migrations): re-chain mobile devices migration against main and restore merge conflicts
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 20:43:29 +00:00
copilot-swe-agent[bot] d538c0879d feat(mobile): add iOS/Android mobile app with SSO login, camera upload, and push notifications
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 09:52:02 +00:00
copilot-swe-agent[bot] 542fb46ee7 fix(compliance): address code review feedback - add aria-busy, input validation, use IntegrityError
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 00:09:08 +00:00
copilot-swe-agent[bot] 666f739f4e feat(compliance): add GDPR, HIPAA, SOC2 compliance templates with one-click apply and dashboard
- Add ComplianceTemplate model in app/models.py
- Create database migration 027_add_compliance_templates
- Add compliance_enabled feature flag to config and settings metadata
- Create compliance_service.py with pre-built template definitions and evaluation
- Create compliance API endpoints (list, get, apply, status, summary)
- Create compliance admin view and dashboard template
- Add compliance link to admin navigation (desktop and mobile)
- Seed compliance templates at application startup
- Add comprehensive tests (29 passing)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:55:29 +00:00
copilot-swe-agent[bot] 1e5e35a26a refactor(routing): address code review feedback - simplify list filter, fix docs example
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:38:25 +00:00
copilot-swe-agent[bot] 40d56f0396 feat(routing): add conditional routing rules for document-to-pipeline assignment
- Add PipelineRoutingRule model to app/models.py
- Add Alembic migration 027_add_routing_rules
- Add routing engine (app/utils/routing_engine.py) with rule evaluation
- Add CRUD API endpoints (app/api/routing_rules.py)
- Register router in app/api/__init__.py
- Add comprehensive tests (73 tests, all passing)

Supported fields: file_type, document_type, category, filename, size, metadata.*
Supported operators: equals, not_equals, contains, not_contains, regex, gt, lt, gte, lte
First-match-wins evaluation with default pipeline fallback

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:34:37 +00:00
copilot-swe-agent[bot] ec77c51cb5 fix: code standards audit — 100% coverage, Pydantic v2 model_config, asyncio.run(), targeted edge-case tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 19:00:13 +00:00
copilot-swe-agent[bot] 5bc1fac79c feat(tasks): add 5 additional scheduled jobs (expire_shared_links, prune_processing_logs, prune_old_notifications, backfill_missing_metadata, sync_search_index)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:25:26 +00:00
copilot-swe-agent[bot] c2d56d7475 feat(tasks): add scheduled batch processing infrastructure (model, migration, tasks, API, view, template)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:01:34 +00:00
Christian Krakau-Louis 201d7c8fdc Merge branch 'main' into copilot/add-document-sharing-feature 2026-03-08 23:12:27 +01:00
Christian Krakau-Louis 6376b6d73b Merge pull request #568 from christianlouis/copilot/add-multi-language-ocr-support
feat(ocr): per-pipeline language override for multi-language OCR
2026-03-08 23:11:41 +01:00
copilot-swe-agent[bot] 06091546b0 fix(sharing): address security review findings
- Use per-password random salt with PBKDF2-HMAC-SHA256 (stored as salt:hash)
- Increase PBKDF2 iterations to 600,000 (OWASP 2023 recommendation)
- Password for downloads now accepted via POST body (never URL query param)
- Fail download request if view count cannot be incremented (prevents bypass)
- Update tests to match new hashing format and POST password download

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 21:55:55 +00:00
copilot-swe-agent[bot] 0f91b8bb7c feat(sharing): add document sharing with expiring links
- Add SharedLink model with token, expiry, view limit, password hash
- Add migration 025_add_shared_links
- Add API endpoints: create, list, revoke (auth) + public info/download
- Add management UI at /shared-links with revoke controls
- Add public share landing page at /share/{token}
- Add Share button on file_view.html
- Add Shared Links to user dropdown in common.js
- Write 35 unit tests covering all scenarios
- Update UserGuide.md with sharing documentation

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 21:51:30 +00:00
copilot-swe-agent[bot] a2a4c6fc9a feat(ocr): add multi-language OCR support with per-pipeline language override
- Add OCR_LANGUAGES constant (28 languages, EN/DE/FR/ES/IT/PT/RU/ZH/JA/KO/AR/etc.)
- Add TESSERACT_TO_EASYOCR mapping for automatic code translation
- Add optional language constructor arg to TesseractOCRProvider/EasyOCRProvider
- Update get_ocr_providers() to accept and pass per-call language override
- Add language parameter to process_with_ocr Celery task
- Add _get_pipeline_ocr_language() helper to resolve OCR language from pipeline step config
- Update process_document to look up and pass pipeline OCR language to process_with_ocr
- Add ocr_language select config field (28 options) to pipeline OCR step schema
- Add language dropdown to pipeline UI (pipelines.html)
- Update docs/UserGuide.md and docs/API.md with language override documentation
- Add 27 new tests covering language constants, provider overrides, and pipeline lookup

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 21:47:12 +00:00
copilot-swe-agent[bot] fcefd0978f feat(notifications): add per-user notification system with inbox, email, and webhook targets
- Add UserNotificationTarget, UserNotificationPreference, InAppNotification models
- Add migration 025_add_user_notifications (tables + indexes)
- Add app/utils/user_notification.py dispatch service
- Add app/api/notifications.py REST endpoints (inbox, targets, preferences)
- Add app/views/notifications.py view route
- Add frontend/templates/notifications_dashboard.html Alpine.js dashboard
- Add bell icon with unread badge in base.html nav (desktop + mobile)
- Register routers in app/api/__init__.py and app/views/__init__.py
- Add 32 unit tests in tests/test_notifications_api.py

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-08 21:34:25 +00:00
copilot-swe-agent[bot] a7eff2c033 fix(security): address CodeQL clear-text logging and weak hashing alerts
- Convert f-string log interpolation to %s-style formatting in
  app/api/pipelines.py and app/api/saved_searches.py to prevent
  clear-text logging of request-derived data (CodeQL: clear-text
  logging of sensitive information)
- Replace plain hashlib.sha256() with PBKDF2-HMAC-SHA256 via
  hash_token() in app/auth.py for Bearer token verification,
  consistent with how tokens are stored in api_tokens.py (CodeQL:
  use of weak cryptographic hashing on sensitive data)
- Remove redundant {exc} from logger.exception() calls (the
  traceback is already captured by logger.exception())
- Update test to verify PBKDF2 hash instead of plain SHA-256

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 19:33:55 +00:00
Christian Krakau-Louis 52ebbad335 Potential fix for code scanning alert no. 344: Use of a broken or weak cryptographic hashing algorithm on sensitive data
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-03-08 20:22:39 +01:00
copilot-swe-agent[bot] 2f95febf71 fix: correct token_prefix comment to match implementation (12 chars)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 18:54:55 +00:00
copilot-swe-agent[bot] c3bb93c197 feat(api): add personal API tokens and enhance webhook integration UI
- Add ApiToken model with SHA-256 hashed storage and usage tracking
- Create API token CRUD endpoints (POST/GET/DELETE /api/api-tokens/)
- Add Bearer token authentication to require_login decorator
- Exempt Bearer-authenticated requests from CSRF validation
- Add API tokens management page with create/revoke/copy UI
- Enhance webhook integration type with detailed explanation and code snippets
- Add navigation links (desktop + mobile) to API tokens page
- Include 19 tests covering CRUD, auth resolution, and utility functions
- Create migration 024_add_api_tokens

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 18:42:55 +00:00
Christian Krakau-Louis a7a807bb11 Merge pull request #551 from christianlouis/copilot/verify-stripe-billing-implementation
fix(billing): cast SQLAlchemy Column[str] to str to resolve mypy typeddict-item error
2026-03-08 18:11:58 +01:00
Christian Krakau-Louis 6cdca24333 Potential fix for code scanning alert no. 332: Information exposure through an exception
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-03-08 18:11:38 +01:00
copilot-swe-agent[bot] c4b4643f6e fix(integrations): sanitize error messages and add SSRF protection for WebDAV test
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:57:27 +00:00
copilot-swe-agent[bot] 35abdb0e69 feat(integrations): add quota enforcement, connection test, and quota endpoint
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:45:41 +00:00
copilot-swe-agent[bot] 9627667d82 fix(billing): cast plan.name to str to fix mypy typeddict-item error
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:33:57 +00:00
Christian Krakau-Louis 48e3552e1a Potential fix for code scanning alert no. 333: Information exposure through an exception
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-03-08 16:27:53 +01:00
copilot-swe-agent[bot] 002cdf312b fix(billing): address code review - remove duplicate mode logic, accessibility improvements, robust test assertions
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:18:53 +00:00
copilot-swe-agent[bot] 223539bebb feat(billing): expose Stripe price IDs in plan API, add Stripe Setup Wizard and sync endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:14:42 +00:00
copilot-swe-agent[bot] 743627ecc2 feat(models): add UserIntegration model, encrypt IMAP passwords, add integrations API
- Add IntegrationDirection/IntegrationType constants and UserIntegration SQLAlchemy model
  with owner_id, direction, integration_type, name, config (JSON), credentials (encrypted),
  is_active, last_used_at, last_error, created_at, updated_at fields
- Fix UserImapAccount password encryption: encrypt_value() on create/update,
  decrypt_value() in IMAP tasks and test-connection endpoint (backward compatible)
- Add Alembic migration 023_add_user_integrations
- Add app/api/integrations.py: CRUD + credentials endpoint with owner-scoped access
- Register integrations router in app/api/__init__.py
- Add 39 tests in tests/test_api_integrations.py covering model, CRUD, encryption

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 15:02:38 +00:00
copilot-swe-agent[bot] 43bcfe5653 feat(imap): add per-user IMAP ingestion accounts with quota enforcement
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 13:01:14 +00:00
copilot-swe-agent[bot] 86f9f5f9b1 fix(auth): activate account on password reset and fix is_active check order
- reset_password sets is_active=True so users with unverified accounts
  can log in after using the forgot-password flow
- admin set_password also sets is_active=True for the same reason
- auth() now checks is_active before verifying the password, ensuring
  inactive users always see the email-verification prompt regardless of
  password correctness (avoids leaking password validity)"

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 12:07:25 +00:00
Christian Krakau-Louis 4109bf65d2 Merge pull request #509 from christianlouis/copilot/add-password-reset-functionality
fix(admin-users): restore missing @router.get decorator on get_user endpoint
2026-03-08 12:14:04 +01:00
copilot-swe-agent[bot] 9b8eb911c3 fix(admin-users): restore missing @router.get decorator on get_user endpoint
The @router.get("/{user_id:path}") decorator was accidentally dropped from the
get_user function when the /local/... routes were inserted above it in the
previous PR. Without the decorator the function was never registered as a GET
handler, so GET /api/admin/users/<id> matched the PUT/DELETE catch-all routes
and Starlette correctly returned 405 Method Not Allowed instead of 200/403.

Adding the decorator back restores the GET endpoint and fixes the 5 tests that
were failing with 405.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 11:07:37 +00:00
copilot-swe-agent[bot] d36ba88de7 feat(auth): password reset, forgot username, and admin user management for local accounts
- Add /forgot-password and /forgot-username page routes and templates
- Update login page label to "Username or Email" (both already accepted by backend)
- Add "Forgot password?" and "Forgot username?" links to login page
- Add POST /api/auth/forgot-username endpoint + send_forgot_username_email() utility
- Add admin endpoints: PATCH /local/{id}, POST /local/{id}/send-password-reset, POST /local/{id}/set-password
- Update admin_users.html with Edit, Password, and Reset action buttons + modals
- Add 23 tests; fix code review issues (import style, display_name clearing behaviour)
- Update docs/API.md and docs/UserGuide.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 10:06:08 +00:00
copilot-swe-agent[bot] a0f5ba1799 feat(backup): extend backup and restore to PostgreSQL and MySQL/MariaDB
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 09:56:58 +00:00
copilot-swe-agent[bot] 44ea43f9cf chore: update plan to include forgot-username and login label clarification
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 09:44:59 +00:00
copilot-swe-agent[bot] 2dd1ca0197 feat(backup): add database backup/restore with scheduled retention and admin dashboard
- Add BackupRecord model for tracking backup archives
- Add migration 021_add_backup_records
- Add backup configuration settings (backup_enabled, backup_dir, backup_remote_destination, backup_remote_folder, backup_retain_hourly/daily/weekly)
- Add backup_tasks.py with create_backup, cleanup_old_backups, and helpers
- Register hourly/daily/weekly Celery beat schedules
- Add /api/admin/backup/* REST endpoints (list, create, download, restore, delete, cleanup)
- Add /admin/backup dashboard view and template
- Add backup link to admin dropdown navigation in base.html
- Add backup settings to SETTING_METADATA in settings_service.py
- Add comprehensive test suite (39 tests passing)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-07 22:03:27 +00:00
Christian Krakau-Louis e929930bae Merge pull request #499 from christianlouis/copilot/add-subscription-management-features
Merge main → subscription-management-features; fix migration chain collision
2026-03-07 22:03:02 +01:00