Commit Graph

2900 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] 289dcc375c fix(db): add migration to create shared_links table for databases that skipped 025
Migration 025_add_shared_links was inserted into the Alembic chain
(between 024_add_api_tokens and 025_add_user_notifications) after some
databases had already been migrated past that point. Those databases
never had the shared_links table created, causing OperationalError when
the expire-shared-links scheduled task runs or when users try to create
shared links.

This commit:
- Adds migration 027_ensure_shared_links_table that idempotently creates
  the table if it doesn't exist
- Updates migrations/env.py to import all models for autogenerate support
- Adds shared_links to db_migrate.py _TABLE_ORDER for proper migration
  ordering
- Adds a regression test verifying the fix

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-10 00:01:33 +00:00
copilot-swe-agent[bot] ce2a76fb77 fix(automation): register automation task in celery worker and add docs
- Import deliver_automation_hook_task in celery_worker.py
- Add Automation (Zapier / Make.com) section to docs/API.md
- Add AUTOMATION_HOOKS_ENABLED to docs/ConfigurationGuide.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:59:15 +00:00
copilot-swe-agent[bot] 666f739f4e feat(compliance): add GDPR, HIPAA, SOC2 compliance templates with one-click apply and dashboard
- Add ComplianceTemplate model in app/models.py
- Create database migration 027_add_compliance_templates
- Add compliance_enabled feature flag to config and settings metadata
- Create compliance_service.py with pre-built template definitions and evaluation
- Create compliance API endpoints (list, get, apply, status, summary)
- Create compliance admin view and dashboard template
- Add compliance link to admin navigation (desktop and mobile)
- Seed compliance templates at application startup
- Add comprehensive tests (29 passing)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:55:29 +00:00
copilot-swe-agent[bot] d167be8274 feat(automation): add Zapier and Make.com integration
Add REST hooks subscription endpoints, incoming action endpoints, and
Zapier-compatible flat payload format for automation platform integration.

- AutomationHook model for webhook subscriptions
- POST /api/automation/hooks/subscribe and DELETE /hooks/{id}
- GET /api/automation/triggers/sample/{event} for Zapier field mapping
- POST /api/automation/actions/upload for incoming document uploads
- Celery task with retry for async hook delivery
- Integration with existing webhook dispatch flow
- 30 passing tests covering all new functionality

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:50:12 +00:00
copilot-swe-agent[bot] 23498e0a98 fix: address code review feedback on spelling
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:40:09 +00:00
copilot-swe-agent[bot] bd898605de docs: add classification rules API and user guide documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:39:09 +00:00
copilot-swe-agent[bot] 1e5e35a26a refactor(routing): address code review feedback - simplify list filter, fix docs example
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:38:25 +00:00
copilot-swe-agent[bot] df051e8b81 feat(classify): add rule-based document classification engine, task, and API
Implements the classify pipeline step with:
- Classification rules engine (app/utils/classification_rules.py) with
  pre-built categories (invoice, contract, receipt, letter, report,
  bank_statement, tax_document, insurance, payslip) and support for
  filename patterns, content keywords, and metadata matching rules
- Celery task (app/tasks/classify_document.py) that runs as a pipeline step
- CRUD API (app/api/classification_rules.py) for managing custom rules
- ClassificationRuleModel in app/models.py with migration 027
- Updated pipeline step config_schema and stage mapping
- Comprehensive tests for engine, API, and task

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:38:05 +00:00
copilot-swe-agent[bot] e95693d684 docs(routing): add routing rules documentation to API.md and UserGuide.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:36:13 +00:00
copilot-swe-agent[bot] c286e1b394 test(i18n): add comprehensive tests and documentation for i18n system
- Add 45 tests covering translation files, translate(), Accept-Language
  parsing, language detection, l10n formatters, and API endpoints
- Create InternationalizationGuide.md documentation
- Fix linting issues (E741, PLW2901)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:35:32 +00:00
copilot-swe-agent[bot] 40d56f0396 feat(routing): add conditional routing rules for document-to-pipeline assignment
- Add PipelineRoutingRule model to app/models.py
- Add Alembic migration 027_add_routing_rules
- Add routing engine (app/utils/routing_engine.py) with rule evaluation
- Add CRUD API endpoints (app/api/routing_rules.py)
- Register router in app/api/__init__.py
- Add comprehensive tests (73 tests, all passing)

Supported fields: file_type, document_type, category, filename, size, metadata.*
Supported operators: equals, not_equals, contains, not_contains, regex, gt, lt, gte, lte
First-match-wins evaluation with default pipeline fallback

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:34:37 +00:00
copilot-swe-agent[bot] 653c137222 fix(audit): address code review - header validation, touch targets, env.demo cleanup
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:33:43 +00:00
copilot-swe-agent[bot] 3b491ea84c docs(audit): add SIEM configuration to docs, .env.demo, and settings metadata
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:29:12 +00:00
copilot-swe-agent[bot] ff76855f29 feat(i18n): add internationalization framework with 10 languages
- Create i18n utility module (app/utils/i18n.py) with translation loading,
  browser language detection, AI fallback, and l10n helpers
- Add JSON translation files for EN, DE, FR, ES, IT, PT, NL, PL, ZH, RU
- Add preferred_language column to UserProfile model with migration
- Register _() translation function as Jinja2 global
- Update base.html with translated navigation, footer, cookie notice
- Add language selector dropdown in nav bar (desktop + mobile)
- Create API endpoints for language preference (POST/GET /api/i18n/)
- Support language detection: user profile > cookie > Accept-Language > default

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:28:02 +00:00
copilot-swe-agent[bot] 35db9f88de feat(audit): add comprehensive audit logging with SIEM integration
- Add AuditLog model with append-only design (timestamp, user, action, resource, IP, details, severity)
- Add audit_service.py with record/query helpers and SIEM forwarding (Syslog RFC 5424, HTTP/webhook)
- Add /api/audit-logs REST endpoints with filtering and pagination
- Add /admin/audit-logs viewer UI with real-time filters
- Add SIEM config settings (syslog, HTTP for Splunk HEC/Logstash/Grafana Loki)
- Add Alembic migration 027_add_audit_logs
- Add navigation link in admin menu
- Add comprehensive tests (20 tests covering model, service, SIEM, API, view)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 23:18:18 +00:00
copilot-swe-agent[bot] 73295cad33 Initial plan 2026-03-09 23:15:51 +00:00
copilot-swe-agent[bot] b4131e0d19 Initial plan 2026-03-09 23:14:47 +00:00
copilot-swe-agent[bot] 3afd406c59 Initial plan 2026-03-09 23:13:25 +00:00
copilot-swe-agent[bot] 954e67640c Initial plan 2026-03-09 23:13:07 +00:00
copilot-swe-agent[bot] 91e50a4441 Initial plan 2026-03-09 23:07:46 +00:00
copilot-swe-agent[bot] 76353349e7 Initial plan 2026-03-09 23:07:18 +00:00
copilot-swe-agent[bot] 341dad643f Initial plan 2026-03-09 23:06:40 +00:00
copilot-swe-agent[bot] 9ee3249146 Initial plan 2026-03-09 23:06:22 +00:00
copilot-swe-agent[bot] e6dfa079cc Initial plan 2026-03-09 23:05:35 +00:00
copilot-swe-agent[bot] 1875986ce8 Initial plan 2026-03-09 23:05:10 +00:00
copilot-swe-agent[bot] 7685b787f2 Initial plan 2026-03-09 23:04:32 +00:00
copilot-swe-agent[bot] 95f1798908 Initial plan 2026-03-09 23:04:14 +00:00
github-actions[bot] e709f2646c chore(release): update build metadata files [skip ci] 2026-03-09 23:01:00 +00:00
semantic-release d5ed512612 0.114.0
Automatically generated by python-semantic-release
2026-03-09 23:00:57 +00:00
Christian Krakau-Louis 5fd3f0661b Merge pull request #577 from christianlouis/copilot/create-user-auth-workflow
fix(templates): remove orphan `</div>` in google_drive.html breaking HTML accessibility lint
2026-03-10 00:00:39 +01:00
copilot-swe-agent[bot] 256d9cb674 fix(templates): remove orphan </div> in google_drive.html causing HTML accessibility lint failure
The extra </div> at line 260 (before the {% endif %} closing the admin
auth-method block) was introduced when the per-user OAuth wizard PR
restructured the admin section. Removing it balances the div tree so
that the outer container div is properly closed by line 427.

djlint frontend/templates/ --lint now reports 0 errors across 58 files.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 21:46:38 +00:00
github-actions[bot] 5d5784f069 chore(release): update build metadata files [skip ci] 2026-03-09 21:30:44 +00:00
semantic-release 91549c8f5e 0.113.0
Automatically generated by python-semantic-release
2026-03-09 21:30:41 +00:00
Christian Krakau-Louis 4555cdb7df Merge pull request #576 from christianlouis/copilot/add-scheduled-batch-processing
feat: scheduled batch processing — 8 admin-managed jobs, DB-driven beat schedule, 100% coverage
2026-03-09 22:30:20 +01:00
copilot-swe-agent[bot] 0529ae53ff fix: 100% coverage, WCAG accessibility fixes, and documentation updates for per-user OAuth wizards
Test coverage:
- Add config=None branch test for dropbox, onedrive, google_drive views (100% coverage)
- Add WATCH_FOLDER source-type test (folder_path vs folder key)
- Add integration-not-found fallback-to-admin-mode test

WCAG 2.1 AA fixes:
- Add aria-labelledby="modalTitle" to role="dialog" modals in setup templates
- Add aria-hidden="true" to decorative SVGs in callback templates
- Add role="status" aria-label="Loading" to spinner divs
- Add aria-live="polite" to processing-message and success/folder-selection regions
- Add role="alert" aria-live="assertive" to error containers
- Update "Return to Setup" link to preserve integration_id in user mode

Docs: update DropboxSetup.md, GoogleDriveSetup.md, OneDriveSetup.md with per-user OAuth flow section

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 20:19:19 +00:00
copilot-swe-agent[bot] ec77c51cb5 fix: code standards audit — 100% coverage, Pydantic v2 model_config, asyncio.run(), targeted edge-case tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 19:00:13 +00:00
copilot-swe-agent[bot] 2d01e71afd test(views): add user-mode coverage tests and fix dashboard helper duplication
- Add tests for user-mode with invalid JSON config (covers exception handling path)
- Add tests for user-mode with valid config (verifies folder path pre-population)
- Extract _watchFolderOAuthSource() helper to reduce isOAuthType/oauthLink duplication

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:32:04 +00:00
copilot-swe-agent[bot] 5bc1fac79c feat(tasks): add 5 additional scheduled jobs (expire_shared_links, prune_processing_logs, prune_old_notifications, backfill_missing_metadata, sync_search_index)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:25:26 +00:00
copilot-swe-agent[bot] c71f33a214 feat(integrations): add per-user OAuth wizard with user-mode for Dropbox, OneDrive, Google Drive
- Add user_mode to dropbox/onedrive/google_drive setup views that loads integration config
- Show user-friendly auth wizard when integration_id is provided (user mode)
- In user mode: show integration name, current folder, back-to-integrations link
- In callback templates: only save credentials (not config) for user integrations
- In integrations dashboard: show Authorize/Re-Authorize button for all OAuth types
- Add WATCH_FOLDER OAuth support: detect source_type in config for auth button
- isOAuthType() and oauthLink() now accept full integration object

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:24:07 +00:00
copilot-swe-agent[bot] c2d56d7475 feat(tasks): add scheduled batch processing infrastructure (model, migration, tasks, API, view, template)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 18:01:34 +00:00
copilot-swe-agent[bot] 80b1b01beb Initial plan 2026-03-09 17:58:17 +00:00
copilot-swe-agent[bot] 07d90758b8 Initial plan 2026-03-09 17:52:43 +00:00
github-actions[bot] 04d51bab22 chore(release): update build metadata files [skip ci] 2026-03-09 12:01:14 +00:00
semantic-release 8fa3e9bc4a 0.112.0
Automatically generated by python-semantic-release
2026-03-09 12:01:11 +00:00
Christian Krakau-Louis 2132e7e8d1 Merge pull request #575 from christianlouis/copilot/fix-imap-integration-ui-issues
feat(integrations): IMAP Gmail labels/delete controls, Watch Folder cloud sources, and comprehensive test coverage
2026-03-09 13:00:50 +01:00
copilot-swe-agent[bot] 9a342af609 fix(imap): normalize CRLF line endings to LF in imap_tasks.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 09:42:20 +00:00
copilot-swe-agent[bot] 1741ab1011 test(tasks): add comprehensive tests for per-user cloud scan functions and gmail_apply_labels
Add 68 new unit tests covering:
- _scan_user_s3_folder: 10 tests (bucket config, credentials, client creation,
  pagination, download, cache, file types, delete, error handling)
- _scan_user_dropbox_folder: 9 tests (credentials, folder path, auth, listing,
  download, cache, non-file entries, delete, download failure)
- _scan_user_google_drive_folder: 8 tests (credentials JSON, folder ID, auth,
  download, cache, delete, download/listing failures)
- _scan_user_onedrive_folder: 10 tests (credentials, folder path, token exchange,
  download, folders, cache, download URL, delete, download/listing failures)
- _scan_user_nextcloud_folder: 9 tests (settings, PROPFIND, XML parse, download,
  self-entry, cache, delete, download failure, absolute href)
- _scan_user_webdav_folder: 9 tests (URL config, PROPFIND, XML parse, download,
  directories, cache, delete, download failure, absolute href)
- Dispatch tests for google_drive, onedrive, nextcloud, webdav source types
- gmail_apply_labels forwarding and default-to-true in IMAP integration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 09:29:43 +00:00
copilot-swe-agent[bot] 4acad71c8f fix: address code review feedback - fix type hint and rename test
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 02:38:50 +00:00
copilot-swe-agent[bot] 8c62a6b714 test(integrations): add tests for gmail_apply_labels and cloud watch folder dispatch
docs(integrations): update UserGuide and ConfigurationGuide for new IMAP and Watch Folder features

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-09 02:35:30 +00:00
copilot-swe-agent[bot] 79fc11b452 test: add tests for gmail_apply_labels=False and cloud source type dispatching
Add test_gmail_labels_disabled_when_gmail_apply_labels_false to verify
that Gmail star/label operations are skipped when gmail_apply_labels=False.

Add four tests to TestPullUserIntegrationWatchFolders for cloud source
type dispatching: S3, Dropbox, unknown provider, and explicit local type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-09 02:26:10 +00:00