Commit Graph

543 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] f70fa65969 Merge branch 'main' into copilot/add-conditional-routing
Resolve conflicts in app/api/__init__.py and app/models.py.
Renumber migration 027_add_routing_rules → 035_add_routing_rules.
Fix migration chain: down_revision → 034_add_user_profile_settings.
Add PipelineRoutingRule to migrations/env.py.
2026-03-12 22:07:54 +00:00
Christian Krakau-Louis 4345d5128f Merge pull request #607 from christianlouis/copilot/add-user-settings-management
fix(tests): update whoami_handler call sites to pass required db argument
2026-03-12 22:44:13 +01:00
copilot-swe-agent[bot] 270e1a56b5 fix(tests): pass mock db to whoami_handler in test_api_auth_enabled.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 17:03:30 +00:00
copilot-swe-agent[bot] 9583d6d96f fix(profile): address code review feedback - early size check, CSRF helper, test constants
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:10:46 +00:00
copilot-swe-agent[bot] b0d6f1ab60 feat(profile): add user self-service profile settings page and API
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:00:19 +00:00
copilot-swe-agent[bot] b483ed16fc fix(i18n): register _ in all Jinja2 template instances, sync translation files, fix pipelines.html
- Register _() translation function in app/auth.py, app/api/local_auth.py, and
  app/api/billing.py template env globals to fix jinja2.exceptions.UndefinedError
- Update test_settings_template_has_db_wizard_link to assert i18n key instead of
  literal "DB Wizard" (template now uses {{ _("settings.db_wizard_btn") }})
- Add 418 missing en.json keys to all 30 non-English translation files as English
  fallbacks to fix test_all_languages_have_same_keys
- Fix orphan </template> tag in pipelines.html by adding missing
  <template x-if="pipelineModal.saving"> opening tag (fixes djlint H025)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 12:46:11 +00:00
copilot-swe-agent[bot] be6b49c872 fix(tests): fix two failing tests - missing DB table and MagicMock IP address
- tests/test_database.py: add user_imap_accounts table to the regression
  test's initial DB setup (migration 022 creates it before rev 026, so it
  must exist for migration 032's ALTER TABLE to succeed)
- tests/test_local_auth.py: configure mock_request.headers.get to return
  None and client=None so get_client_ip() returns "unknown" instead of an
  un-serialisable MagicMock that broke the audit_logs INSERT in
  test_local_login_success and test_local_login_by_email

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 10:49:49 +00:00
copilot-swe-agent[bot] 8060a79b9c style: fix ruff formatting in tests/test_imap_profiles.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 10:25:32 +00:00
Christian Krakau-Louis 7f644a2388 Merge pull request #604 from christianlouis/copilot/configure-attachment-ingestion
feat(imap): fine-grained attachment ingestion profiles with per-category selection
2026-03-12 08:53:45 +01:00
Christian Krakau-Louis 87fca21c7f Merge pull request #605 from christianlouis/copilot/enable-disable-storage-locations
feat(storage): explicit enable/disable flag per global storage destination
2026-03-12 08:51:14 +01:00
copilot-swe-agent[bot] 2e087731b9 fix(imap): address code review feedback on ingestion profiles
- Improve warning log in _resolve_categories_for_profile() to include
  exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
  category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
  full profiles documentation including category table and API reference

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:19:32 +00:00
copilot-swe-agent[bot] c9f554465d feat(imap): add ImapIngestionProfile model, API, migration and UI
Replaces the simple binary attachment_filter string with a full
ingestion profiles system:

- Add FILE_TYPE_CATEGORIES dict to allowed_types.py (6 categories:
  pdf, office, opendocument, text, web, images) + DEFAULT_CATEGORIES
  / ALL_CATEGORIES + get_allowed_types_for_categories() helper
- Add ImapIngestionProfile model (id, name, description, owner_id,
  allowed_categories JSON, is_builtin)
- Update UserImapAccount: replace attachment_filter string with
  profile_id FK to imap_ingestion_profiles
- Migration 033: creates profiles table, seeds 2 built-in profiles
  (Documents Only, All Files), migrates attachment_filter → profile_id
- New /api/imap-profiles/ CRUD endpoints (list, create, get, update,
  delete) with category validation
- Register imap_profiles router in app/api/__init__.py
- Update imap_tasks.py: replace attachment_filter string param with
  profile-based allowed_categories; add _resolve_categories_for_profile()
- Update imap_accounts.py API to use profile_id instead of attachment_filter
- Update imap_accounts view to pass profiles + categories to template
- Full UI overhaul: profiles panel + profile create/edit modal with
  category checkboxes; profile selector in account modal
- 17 new tests (141 total), all passing

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:14:19 +00:00
copilot-swe-agent[bot] b72ab3b318 feat(storage): add explicit enable/disable flag for each global storage destination
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:04:28 +00:00
copilot-swe-agent[bot] 554bb21d32 feat(imap): add attachment type filter for IMAP ingestion
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.

- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
  `_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
  display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 00:59:54 +00:00
Christian Krakau-Louis 14b892b6d4 Merge pull request #601 from christianlouis/copilot/improve-code-test-coverage
test: improve coverage for app/utils/user_notification.py from ~50% to 100%
2026-03-12 01:55:49 +01:00
copilot-swe-agent[bot] 78204c2490 test(notifications): improve coverage for user_notification.py to 100%
Add 22 new unit tests in tests/test_user_notification_service.py covering
all previously-uncovered branches in app/utils/user_notification.py:

- create_in_app_notification: exception/rollback path
- _send_email_notification: SMTP success (TLS+creds, no-TLS, no-creds),
  SMTP exception, sender_email fallbacks
- _send_webhook_notification: success with/without secret, HTTP errors
- dispatch_user_notification: email/webhook pref dispatch, no target_id
  skip, inactive target skip, invalid JSON config, null config, outer
  exception handling, push notification sent/exception, unknown channel_type

Combined coverage: 49.52% → 100%

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 00:44:59 +00:00
Christian Krakau-Louis 0aa1df3920 Merge pull request #588 from christianlouis/copilot/add-social-logins-configuration
feat(auth): add social login support for Google, Microsoft, Apple, and Dropbox
2026-03-12 01:40:04 +01:00
copilot-swe-agent[bot] 7438551080 test: improve coverage for app/utils/audit_service.py from 59% to 100%
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 00:28:42 +00:00
Christian Krakau-Louis 80dd117ebf Merge pull request #587 from christianlouis/copilot/add-apple-drive-icloud-support
feat(storage): add Apple iCloud Drive storage provider
2026-03-12 01:14:37 +01:00
copilot-swe-agent[bot] 054af636e3 fix(migrations): merge main and rechain compliance migration as 031 after 030_add_mobile_devices
Merge main branch into compliance templates feature branch.

Main had advanced with migrations 027-030 (ensure_shared_links,
audit_logs, user_language_preference, mobile_devices) since this
branch forked. Our compliance migration was 027 with down_revision
026, which conflicted with main's 027_ensure_shared_links_table.

Changes:
- Merge main (including i18n, audit logs, mobile, GraphQL features)
- Resolve conflicts in app/api/__init__.py, app/models.py, tests/conftest.py
- Rename 027_add_compliance_templates → 031_add_compliance_templates
- Rechain: down_revision 026_add_scheduled_jobs → 030_add_mobile_devices
- Add ComplianceTemplate to migrations/env.py imports
- Alembic now has single head: 031_add_compliance_templates
2026-03-12 00:00:29 +00:00
copilot-swe-agent[bot] 50af0ea679 fix(tests): add iCloud mocks to all test files and address code review feedback
- Add _should_upload_to_icloud mock to test_coverage_uploads_notification.py
  (_all_should_upload_false helper + 3 inline patch blocks)
- Add cfg.icloud_username/password = None to all onboarding test mocks
- Add iCloud creds to fully-configured onboarding test
- Fix noqa comment accuracy (unofficial third-party, not first-party)
- Replace generic Exception with RuntimeError in upload error handler

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 23:59:19 +00:00
Christian Krakau-Louis 39db5fc564 Merge pull request #600 from christianlouis/copilot/add-graphql-api
feat: add GraphQL API alongside REST at /graphql
2026-03-12 00:43:10 +01:00
copilot-swe-agent[bot] 528f0a624d fix: remove accidental pip artifact file and update docs for iCloud Drive
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 23:38:35 +00:00
copilot-swe-agent[bot] 30f06e0b32 feat(storage): add Apple iCloud Drive storage provider
Add iCloud Drive as a new storage destination using the pyicloud library.
Includes upload task, configuration, user integration handler, provider
status, onboarding support, and comprehensive tests.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 23:38:35 +00:00
copilot-swe-agent[bot] 9bc23aa40b test(tasks): add _should_upload_to_icloud mock to send_to_all tests
Add icloud upload check mock alongside existing _should_upload_to_*
function mocks in all TestSendToAllDestinations test methods.

Changes:
- Import _should_upload_to_icloud from app.tasks.send_to_all
- Add @patch decorator for _should_upload_to_icloud in 9 test methods
- Add mock_icloud parameter to each test method signature
- Set mock_icloud.return_value = False where other mocks are set to False

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-11 23:38:34 +00:00
Christian Krakau-Louis 1c4a261f01 Merge pull request #599 from christianlouis/copilot/fix-main-test-failures
fix: resolve 47 failing CI tests across i18n, templates, and migrations
2026-03-12 00:34:18 +01:00
copilot-swe-agent[bot] df4c91a586 fix: resolve all 47 failing tests in main
- Fix detect_language() to safely handle unhashable session values and
  requests missing cookies/headers attributes (TypeError + AttributeError)
- Add default English `_()` translation function to Jinja2 template
  environment globals so error pages always have it available
- Fix app/main.py exception handlers to use a dedicated error templates
  instance with `_` registered, keeping it separate from view templates
  to avoid test patches breaking error rendering
- Fix app/views/plans.py to import shared templates from app.views.base
  instead of creating its own Jinja2Templates instance
- Make migration 029_add_user_language_preference idempotent: skip
  ALTER TABLE if user_profiles table does not exist
- Update test_i18n.py expectations to reflect 31 supported languages
- Create 21 missing translation files (nb, da, sv, fi, is, ga, lb, ca,
  cs, sk, hu, sl, hr, ro, bg, el, et, lv, lt, tr, uk) with English
  placeholder translations
- Update de.json with 117 missing translation keys including proper
  German translations
- Update es, fr, it, nl, pl, pt, ru, zh translation files with missing
  keys using English fallbacks

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 22:28:01 +00:00
copilot-swe-agent[bot] a41ded535f feat(api): add GraphQL endpoint at /graphql with Strawberry
- Add strawberry-graphql[fastapi] dependency
- Implement GraphQL schema covering documents, pipelines, settings, users
- Enable GraphiQL playground at /graphql
- Mount GraphQL router in main.py
- Add 24 tests for all query types and auth enforcement
- Update docs/API.md with GraphQL documentation section

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-11 22:17:25 +00:00
copilot-swe-agent[bot] 6bcabfef77 fix(merge): resolve conflicts in app/api/__init__.py and migrations/env.py against main
Merge main into branch to pull in translation files, audit log templates,
CI workflow updates, and other changes from PRs #595, #597, #598.
Keep mobile_router and MobileDevice additions from this branch.
2026-03-11 21:53:00 +00:00
copilot-swe-agent[bot] a66eb4b2a2 fix: merge main into i18n branch and relink migration to 029
- Resolve conflict in app/api/__init__.py (keep both audit_logs_router and i18n_router)
- Incorporate AuditLog model, audit_service, audit_logs API/views from main
- Relink migration from 026→027 to 028→029 (chain after 028_add_audit_logs)
- Update migrations/env.py with full model import list from main

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 21:59:58 +00:00
copilot-swe-agent[bot] 4eb04bd7f6 fix(mobile): address code review findings in mobile app config and tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 09:57:58 +00:00
copilot-swe-agent[bot] d538c0879d feat(mobile): add iOS/Android mobile app with SSO login, camera upload, and push notifications
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 09:52:02 +00:00
copilot-swe-agent[bot] 5d716ad78f fix(auth): address code review feedback - sanitize error messages, remove unused import
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 00:27:07 +00:00
copilot-swe-agent[bot] 9c26d412d7 test(auth): add tests for social login and fix existing config validator tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 00:24:53 +00:00
copilot-swe-agent[bot] 542fb46ee7 fix(compliance): address code review feedback - add aria-busy, input validation, use IntegrityError
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 00:09:08 +00:00
copilot-swe-agent[bot] 666f739f4e feat(compliance): add GDPR, HIPAA, SOC2 compliance templates with one-click apply and dashboard
- Add ComplianceTemplate model in app/models.py
- Create database migration 027_add_compliance_templates
- Add compliance_enabled feature flag to config and settings metadata
- Create compliance_service.py with pre-built template definitions and evaluation
- Create compliance API endpoints (list, get, apply, status, summary)
- Create compliance admin view and dashboard template
- Add compliance link to admin navigation (desktop and mobile)
- Seed compliance templates at application startup
- Add comprehensive tests (29 passing)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:55:29 +00:00
copilot-swe-agent[bot] c286e1b394 test(i18n): add comprehensive tests and documentation for i18n system
- Add 45 tests covering translation files, translate(), Accept-Language
  parsing, language detection, l10n formatters, and API endpoints
- Create InternationalizationGuide.md documentation
- Fix linting issues (E741, PLW2901)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:35:32 +00:00
copilot-swe-agent[bot] 40d56f0396 feat(routing): add conditional routing rules for document-to-pipeline assignment
- Add PipelineRoutingRule model to app/models.py
- Add Alembic migration 027_add_routing_rules
- Add routing engine (app/utils/routing_engine.py) with rule evaluation
- Add CRUD API endpoints (app/api/routing_rules.py)
- Register router in app/api/__init__.py
- Add comprehensive tests (73 tests, all passing)

Supported fields: file_type, document_type, category, filename, size, metadata.*
Supported operators: equals, not_equals, contains, not_contains, regex, gt, lt, gte, lte
First-match-wins evaluation with default pipeline fallback

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:34:37 +00:00
Christian Krakau-Louis 5fd3f0661b Merge pull request #577 from christianlouis/copilot/create-user-auth-workflow
fix(templates): remove orphan `</div>` in google_drive.html breaking HTML accessibility lint
2026-03-10 00:00:39 +01:00
copilot-swe-agent[bot] 0529ae53ff fix: 100% coverage, WCAG accessibility fixes, and documentation updates for per-user OAuth wizards
Test coverage:
- Add config=None branch test for dropbox, onedrive, google_drive views (100% coverage)
- Add WATCH_FOLDER source-type test (folder_path vs folder key)
- Add integration-not-found fallback-to-admin-mode test

WCAG 2.1 AA fixes:
- Add aria-labelledby="modalTitle" to role="dialog" modals in setup templates
- Add aria-hidden="true" to decorative SVGs in callback templates
- Add role="status" aria-label="Loading" to spinner divs
- Add aria-live="polite" to processing-message and success/folder-selection regions
- Add role="alert" aria-live="assertive" to error containers
- Update "Return to Setup" link to preserve integration_id in user mode

Docs: update DropboxSetup.md, GoogleDriveSetup.md, OneDriveSetup.md with per-user OAuth flow section

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 20:19:19 +00:00
copilot-swe-agent[bot] ec77c51cb5 fix: code standards audit — 100% coverage, Pydantic v2 model_config, asyncio.run(), targeted edge-case tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 19:00:13 +00:00
copilot-swe-agent[bot] 2d01e71afd test(views): add user-mode coverage tests and fix dashboard helper duplication
- Add tests for user-mode with invalid JSON config (covers exception handling path)
- Add tests for user-mode with valid config (verifies folder path pre-population)
- Extract _watchFolderOAuthSource() helper to reduce isOAuthType/oauthLink duplication

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:32:04 +00:00
copilot-swe-agent[bot] 5bc1fac79c feat(tasks): add 5 additional scheduled jobs (expire_shared_links, prune_processing_logs, prune_old_notifications, backfill_missing_metadata, sync_search_index)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:25:26 +00:00
copilot-swe-agent[bot] 1741ab1011 test(tasks): add comprehensive tests for per-user cloud scan functions and gmail_apply_labels
Add 68 new unit tests covering:
- _scan_user_s3_folder: 10 tests (bucket config, credentials, client creation,
  pagination, download, cache, file types, delete, error handling)
- _scan_user_dropbox_folder: 9 tests (credentials, folder path, auth, listing,
  download, cache, non-file entries, delete, download failure)
- _scan_user_google_drive_folder: 8 tests (credentials JSON, folder ID, auth,
  download, cache, delete, download/listing failures)
- _scan_user_onedrive_folder: 10 tests (credentials, folder path, token exchange,
  download, folders, cache, download URL, delete, download/listing failures)
- _scan_user_nextcloud_folder: 9 tests (settings, PROPFIND, XML parse, download,
  self-entry, cache, delete, download failure, absolute href)
- _scan_user_webdav_folder: 9 tests (URL config, PROPFIND, XML parse, download,
  directories, cache, delete, download failure, absolute href)
- Dispatch tests for google_drive, onedrive, nextcloud, webdav source types
- gmail_apply_labels forwarding and default-to-true in IMAP integration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 09:29:43 +00:00
copilot-swe-agent[bot] 4acad71c8f fix: address code review feedback - fix type hint and rename test
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 02:38:50 +00:00
copilot-swe-agent[bot] 79fc11b452 test: add tests for gmail_apply_labels=False and cloud source type dispatching
Add test_gmail_labels_disabled_when_gmail_apply_labels_false to verify
that Gmail star/label operations are skipped when gmail_apply_labels=False.

Add four tests to TestPullUserIntegrationWatchFolders for cloud source
type dispatching: S3, Dropbox, unknown provider, and explicit local type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 02:26:10 +00:00
Christian Krakau-Louis 0b9bba7158 Merge pull request #562 from christianlouis/copilot/refactor-dynamic-routing-user-destinations
fix(tasks): register upload_to_user_integration in Celery and add handler coverage
2026-03-09 01:05:36 +01:00
Christian Krakau-Louis e916001e08 Merge pull request #573 from christianlouis/copilot/fix-sentry-settings-startup
fix(sentry): initialize Sentry after DB settings load in lifespan
2026-03-09 00:46:36 +01:00
github-actions[bot] b22f9a6af6 style: apply ruff auto-fix
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-08 22:40:15 +00:00
copilot-swe-agent[bot] 5fb11b978c fix(tasks): register upload_to_user_integration in celery_worker and add handler tests
- Add `from app.tasks.upload_to_user_integration import upload_to_user_integration`
  to app/celery_worker.py — fixes TestCeleryWorkerConfig test that enforces
  every tasks/ module is registered with Celery.

- Add tests/test_upload_handlers.py with 43 unit tests covering all 11
  per-type upload handler functions (_upload_dropbox, _upload_s3,
  _upload_google_drive, _upload_onedrive, _upload_webdav, _upload_nextcloud,
  _upload_ftp, _upload_sftp, _upload_paperless, _upload_email, _upload_rclone)
  plus 2 additional finalize_document_storage branch tests. All external
  libraries (dropbox, boto3, msal, paramiko, smtplib, subprocess, requests)
  are mocked so tests are hermetic and fast. Coverage on changed files:
  upload_to_user_integration.py 94.71%, finalize_document_storage.py 95.51%
  (both well above the 70% Codecov diff threshold).

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 22:21:15 +00:00