Commit Graph

14 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] c9f554465d feat(imap): add ImapIngestionProfile model, API, migration and UI
Replaces the simple binary attachment_filter string with a full
ingestion profiles system:

- Add FILE_TYPE_CATEGORIES dict to allowed_types.py (6 categories:
  pdf, office, opendocument, text, web, images) + DEFAULT_CATEGORIES
  / ALL_CATEGORIES + get_allowed_types_for_categories() helper
- Add ImapIngestionProfile model (id, name, description, owner_id,
  allowed_categories JSON, is_builtin)
- Update UserImapAccount: replace attachment_filter string with
  profile_id FK to imap_ingestion_profiles
- Migration 033: creates profiles table, seeds 2 built-in profiles
  (Documents Only, All Files), migrates attachment_filter → profile_id
- New /api/imap-profiles/ CRUD endpoints (list, create, get, update,
  delete) with category validation
- Register imap_profiles router in app/api/__init__.py
- Update imap_tasks.py: replace attachment_filter string param with
  profile-based allowed_categories; add _resolve_categories_for_profile()
- Update imap_accounts.py API to use profile_id instead of attachment_filter
- Update imap_accounts view to pass profiles + categories to template
- Full UI overhaul: profiles panel + profile create/edit modal with
  category checkboxes; profile selector in account modal
- 17 new tests (141 total), all passing

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 01:14:19 +00:00
copilot-swe-agent[bot] 554bb21d32 feat(imap): add attachment type filter for IMAP ingestion
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.

- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
  `_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
  display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 00:59:54 +00:00
copilot-swe-agent[bot] 1741ab1011 test(tasks): add comprehensive tests for per-user cloud scan functions and gmail_apply_labels
Add 68 new unit tests covering:
- _scan_user_s3_folder: 10 tests (bucket config, credentials, client creation,
  pagination, download, cache, file types, delete, error handling)
- _scan_user_dropbox_folder: 9 tests (credentials, folder path, auth, listing,
  download, cache, non-file entries, delete, download failure)
- _scan_user_google_drive_folder: 8 tests (credentials JSON, folder ID, auth,
  download, cache, delete, download/listing failures)
- _scan_user_onedrive_folder: 10 tests (credentials, folder path, token exchange,
  download, folders, cache, download URL, delete, download/listing failures)
- _scan_user_nextcloud_folder: 9 tests (settings, PROPFIND, XML parse, download,
  self-entry, cache, delete, download failure, absolute href)
- _scan_user_webdav_folder: 9 tests (URL config, PROPFIND, XML parse, download,
  directories, cache, delete, download failure, absolute href)
- Dispatch tests for google_drive, onedrive, nextcloud, webdav source types
- gmail_apply_labels forwarding and default-to-true in IMAP integration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 09:29:43 +00:00
copilot-swe-agent[bot] 79fc11b452 test: add tests for gmail_apply_labels=False and cloud source type dispatching
Add test_gmail_labels_disabled_when_gmail_apply_labels_false to verify
that Gmail star/label operations are skipped when gmail_apply_labels=False.

Add four tests to TestPullUserIntegrationWatchFolders for cloud source
type dispatching: S3, Dropbox, unknown provider, and explicit local type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 02:26:10 +00:00
copilot-swe-agent[bot] 4f9f458151 test(tasks): add comprehensive tests for multi-tenant polling functions
- Add tests for owner_id passthrough in fetch_attachments_and_enqueue
- Add tests for _pull_user_integration_imap (success, failure, DB error, incomplete config)
- Add tests for _is_safe_watch_path path traversal security
- Add tests for _enqueue_file owner_id forwarding
- Add tests for _scan_user_watch_folder (scan, skip processed, nonexistent dir)
- Add tests for _pull_user_integration_watch_folders (success, unsafe path, failure, DB error)
- Add tests verifying scan_all_watch_folders calls user integration polling
- Add tests verifying pull_all_inboxes calls both legacy and integration polling

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-08 18:43:18 +00:00
copilot-swe-agent[bot] f8c5dd539d feat(imap): add IMAP_READONLY_MODE feature flag to safeguard shared mailboxes
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:28:51 +00:00
copilot-swe-agent[bot] 8460249ee1 test: add comprehensive tests for IMAP, OpenAI, send_to_all, config_loader, and diagnostic APIs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-14 00:34:34 +00:00
github-actions[bot] d0e85d0690 style: apply ruff auto-fix
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-02-14 00:12:30 +00:00
copilot-swe-agent[bot] fc4f5631ee test: increase coverage for imap_tasks.py and fix config_validator.py
- Add comprehensive tests for imap_tasks.py covering:
  - Lock acquire/release mechanisms
  - pull_all_inboxes task with various scenarios
  - pull_inbox for Gmail and non-Gmail with edge cases
  - find_all_mail_xlist functionality
  - Extended fetch_attachments tests for all MIME types
  - Edge cases: invalid JSON, missing Message-ID, already processed, etc.
- Achieve 98.26% coverage for imap_tasks.py (up from 48.78%)

- Fix config_validator.py to include validate_auth_config export
- Update tests to verify all exports including validate_auth_config
- Note: config_validator.py file is shadowed by config_validator/ directory
  in Python's module resolution, so it cannot be directly imported or tested.
  The package's __init__.py (which has 100% coverage) is what's actually used.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 23:42:49 +00:00
copilot-swe-agent[bot] 43bc58770d refactor: consolidate linting tools into Ruff
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 09:10:52 +00:00
copilot-swe-agent[bot] ff9a3ff49f style: fix code formatting with black, isort, and flake8
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 03:21:41 +00:00
copilot-swe-agent[bot] 5e82f7c03a refactor: fix static method warnings, hard-coded credentials, and Annotated type hints
- Add @staticmethod to 9 test methods in test_external_integrations.py
  that don't use self (PYL-R0201)
- Extract hard-coded password literals to constants in 6 test files
  to resolve S2068 warnings (fixtures_integration, test_imap_tasks,
  test_upload_tasks, test_upload_webdav_comprehensive,
  test_upload_webdav_integration, test_views_coverage)
- Migrate Form() dependency injection to Annotated type hints in
  dropbox.py, google_drive.py, onedrive.py (Sonar fastapi convention)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 17:03:54 +00:00
copilot-swe-agent[bot] 7c4d6fd025 fix: extract hard-coded test credentials to module-level constants (S2068)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 16:50:37 +00:00
copilot-swe-agent[bot] c3bfb26c73 test: add comprehensive tests across modules to increase coverage above 60%
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 11:57:09 +00:00