The test_successful_conversion test asserts that '--' precedes the input/output
file paths in the ocrmypdf command as a security measure against argument
injection (file paths starting with '-' being interpreted as options).
The implementation was missing this separator, causing the test to fail and
triggering a downstream pytest INTERNALERROR (OSError: Bad file descriptor)
when pytest's terminal writer tried to report the failure.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Agent-Logs-Url: https://github.com/christianlouis/DocuElevate/sessions/bd7ef195-4b30-456e-8118-5c200fb4bf45
Move the valid filename regex pattern to a shared constant in `app/utils/filename_utils.py` and update both the task logic and security tests to use it. This eliminates duplication and ensures consistency across the codebase.
Changes:
- Defined `VALID_FILENAME_PATTERN` and `VALID_FILENAME_RE` in `app/utils/filename_utils.py`.
- Updated `app/tasks/extract_metadata_with_gpt.py` to use `VALID_FILENAME_RE`.
- Updated `tests/test_path_traversal_security.py` to use `VALID_FILENAME_PATTERN`.
This refactoring addresses the duplication mentioned in the TODO in `tests/test_path_traversal_security.py`.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
🚨 Severity: HIGH
💡 Vulnerability: User-provided IMAP `host` in `_test_imap_connection` and `pull_inbox` was not validated against private IPs, creating an SSRF risk.
🎯 Impact: Attackers could abuse the endpoints to port-scan or interact with internal/private network services.
🔧 Fix: Integrated `is_private_ip` from `app.utils.network` to block connections resolving to private, loopback, link-local, or reserved IPs.
✅ Verification: Ran `test_imap_tasks.py` and `test_api_imap_accounts.py` successfully. Checked `ruff` output and diffs. Removed all scratch files from the commit.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
🚨 Severity: HIGH
💡 Vulnerability: User-provided IMAP `host` in `_test_imap_connection` and `pull_inbox` was not validated against private IPs, creating an SSRF risk.
🎯 Impact: Attackers could abuse the endpoints to port-scan or interact with internal/private network services.
🔧 Fix: Integrated `is_private_ip` from `app.utils.network` to block connections resolving to private, loopback, link-local, or reserved IPs.
✅ Verification: Ran `test_imap_tasks.py` and `test_api_imap_accounts.py` successfully. Checked `ruff` output and diffs.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Resolve all merge conflicts between our automation feature branch and
current main (v0.163.0, 920 commits ahead).
Conflicts resolved:
- app/api/__init__.py: add automation_router alongside main's new routers
(classification_rules, qr_auth, sessions, system_reset)
- app/config.py: add main's new settings (dropbox_use_global_credentials,
factory_reset_on_startup, enable_factory_reset)
- app/models.py: add main's new models (ClassificationRuleModel, UserSession,
QRLoginChallenge, SharePoint integration type)
- app/utils/settings_service.py: merge automation_hooks_enabled with main's
new metadata entries
- docs/API.md: merge automation API docs with main's classification rules docs
- docs/ConfigurationGuide.md: add factory reset settings
- tests/conftest.py: import both AutomationHook and new main models
Migration renumbered:
- 037_add_automation_hooks → 040_add_automation_hooks
- down_revision: 039_add_classification_rules (was 036_add_document_translation_fields)
- Chain: 036 → 037 → 038 → 039 → 040 (automation hooks)
For all non-automation files with conflicts, main's version was taken since
our branch did not modify those files (conflicts were from a stale prior merge).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Agent-Logs-Url: https://github.com/christianlouis/DocuElevate/sessions/cb62f012-3b69-4415-835e-3857ce3e9f45
- Add Privacy Policy, Terms of Service, and Imprint links to WelcomeScreen
and LoginScreen for GDPR/Apple compliance (pre-login access)
- Enable multiple image selection in photo library picker
- Add HEIC/HEIF image support to backend (allowed_types, convert_to_pdf, upload handler)
- Create FileDetailScreen with processing status and logs
- Add search bar to FilesScreen with debounced search
- Set up i18n with expo-localization (EN, DE, ES, FR, IT)
- Add language selector to ProfileScreen settings
- Add Imprint link to ProfileScreen legal section
- Update docs and tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Merge origin/main into feature branch, resolving 3 conflicts:
- app/api/__init__.py: add classification_rules_router alongside new
routers from main (audit_logs, i18n, mobile, compliance, translation)
- app/models.py: keep ClassificationRuleModel alongside new models from
main (MobileDevice, ComplianceTemplate, PipelineRoutingRule)
- tests/conftest.py: import both ClassificationRuleModel and new models
from main (AuditLog, ComplianceTemplate)
Also renumber migration from 027 to 037 to chain from the latest
migration on main (036_add_document_translation_fields).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Merge origin/main into copilot/add-sharepoint-integration.
All four conflicts were resolved by keeping both the SharePoint
additions (from this branch) and the iCloud additions (from main):
- app/models.py: added both SHAREPOINT and ICLOUD to IntegrationType
- app/tasks/send_to_all.py: added both to service_map and services list
- app/tasks/upload_to_user_integration.py: kept both upload handlers
- frontend/templates/files.html: added both filter options
No database migration conflicts — SharePoint does not require schema changes.
The per-user notification functions (notify_user_document_processed /
notify_user_document_failed) were defined but never called from the
document processing pipeline.
- Call notify_user_document_processed in finalize_document_storage
when owner_id is available (creates in-app + email/webhook notifications)
- Add _dispatch_user_failure_notification helper to celery_app.py that
extracts file_id from failed task args and dispatches
notify_user_document_failed for document pipeline tasks
- Add comprehensive tests for both success and failure notification paths
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Added the `--` argument before positional arguments in rclone subprocess calls (link, mkdir, copy) in `app/tasks/upload_with_rclone.py`. This ensures that filenames or destinations starting with a hyphen are treated as paths rather than unintended command-line flags.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Move the valid filename regex pattern to a shared constant in app/utils/filename_utils.py and update both the task logic and security tests to use it. This eliminates duplication and ensures consistency across the codebase. Also normalized line endings in app/tasks/extract_metadata_with_gpt.py.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Improve warning log in _resolve_categories_for_profile() to include
exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
full profiles documentation including category table and API reference
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.
- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
`_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add iCloud Drive as a new storage destination using the pyicloud library.
Includes upload task, configuration, user integration handler, provider
status, onboarding support, and comprehensive tests.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add Microsoft SharePoint Online as a storage provider using the
Microsoft Graph API. Includes OAuth2 authentication via MSAL,
site/drive resolution, chunked upload sessions, and metadata sync.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add REST hooks subscription endpoints, incoming action endpoints, and
Zapier-compatible flat payload format for automation platform integration.
- AutomationHook model for webhook subscriptions
- POST /api/automation/hooks/subscribe and DELETE /hooks/{id}
- GET /api/automation/triggers/sample/{event} for Zapier field mapping
- POST /api/automation/actions/upload for incoming document uploads
- Celery task with retry for async hook delivery
- Integration with existing webhook dispatch flow
- 30 passing tests covering all new functionality
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Implements the classify pipeline step with:
- Classification rules engine (app/utils/classification_rules.py) with
pre-built categories (invoice, contract, receipt, letter, report,
bank_statement, tax_document, insurance, payslip) and support for
filename patterns, content keywords, and metadata matching rules
- Celery task (app/tasks/classify_document.py) that runs as a pipeline step
- CRUD API (app/api/classification_rules.py) for managing custom rules
- ClassificationRuleModel in app/models.py with migration 027
- Updated pipeline step config_schema and stage mapping
- Comprehensive tests for engine, API, and task
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Christian Krakau-Louis