Move the valid filename regex pattern to a shared constant in `app/utils/filename_utils.py` and update both the task logic and security tests to use it. This eliminates duplication and ensures consistency across the codebase.
Also normalized line endings to LF in affected files to ensure CI compatibility.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Improve warning log in _resolve_categories_for_profile() to include
exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
full profiles documentation including category table and API reference
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.
- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
`_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add iCloud Drive as a new storage destination using the pyicloud library.
Includes upload task, configuration, user integration handler, provider
status, onboarding support, and comprehensive tests.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix _get_pipeline_ocr_language: remove redundant `or None` in step_config.get()
- Add Session type hint to _get_pipeline_ocr_language db parameter via TYPE_CHECKING
- Update process_with_ocr to use modern str | None syntax instead of Optional[str]
- Fix test_get_pipeline_ocr_language_explicit_pipeline_takes_priority: properly add
sys_step to db_session so the system pipeline step is persisted in the test DB
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add OCR_LANGUAGES constant (28 languages, EN/DE/FR/ES/IT/PT/RU/ZH/JA/KO/AR/etc.)
- Add TESSERACT_TO_EASYOCR mapping for automatic code translation
- Add optional language constructor arg to TesseractOCRProvider/EasyOCRProvider
- Update get_ocr_providers() to accept and pass per-call language override
- Add language parameter to process_with_ocr Celery task
- Add _get_pipeline_ocr_language() helper to resolve OCR language from pipeline step config
- Update process_document to look up and pass pipeline OCR language to process_with_ocr
- Add ocr_language select config field (28 options) to pipeline OCR step schema
- Add language dropdown to pipeline UI (pipelines.html)
- Update docs/UserGuide.md and docs/API.md with language override documentation
- Add 27 new tests covering language constants, provider overrides, and pipeline lookup
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/tasks/upload_to_user_integration.py: new Celery task that
uploads a processed document to a specific UserIntegration using its
own per-user config and Fernet-decrypted credentials. Supports all
DESTINATION types: Dropbox, S3, Google Drive, OneDrive, WebDAV,
Nextcloud, FTP, SFTP, Paperless-ngx, Email (SMTP), and Rclone.
- Extend app/tasks/send_to_all.py: add send_to_user_destinations task
(queries active DESTINATION UserIntegrations for an owner and
dispatches one upload_to_user_integration task per integration) and
get_user_destination_count helper used by finalize_document_storage.
- Refactor app/tasks/finalize_document_storage.py: after processing,
look up the document owner; if the owner has active DESTINATION
integrations route exclusively to those (user-specific routing),
otherwise fall back to the global send_to_all_destinations.
- Update tests/test_finalize_storage.py: add autouse fixture to prevent
Redis hangs, update all existing tests with new mock parameters, add
TestFinalizeDocumentStorageUserRouting class with four new tests that
validate user-specific vs global routing decisions.
- Add tests/test_user_integration_upload.py: 14 new unit tests covering
upload_to_user_integration (handler dispatch, error persistence,
last_used_at update, credential decryption, skip for unknown types)
and send_to_user_destinations / get_user_destination_count.
- Update docs/StorageArchitecture.md: document the user-specific
destination routing feature, supported types, multiple-destination
behaviour, and global fallback semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add owner_id parameter to pull_inbox() and fetch_attachments_and_enqueue()
to attribute ingested documents to the correct user
- Add _pull_user_integration_imap() to poll IMAP sources from UserIntegration model
- Add _pull_user_integration_watch_folders() to scan watch folders from UserIntegration model
- Add _is_safe_watch_path() for path traversal security on user-configured paths
- Add _scan_user_watch_folder() that passes owner_id to _enqueue_file()
- Update _enqueue_file() to forward owner_id to process_document/convert_to_pdf
- Update celery beat schedule to always enable IMAP and watch folder polling
(user integrations can exist without system-level config)
- Ensure individual connection failures don't crash the polling loop
- Update existing tests for new function signatures
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Create app/templates/email/default.html (fixes 'default.html not found' error)
- Add DEST_EMAIL_* settings to app/config.py (decoupled from shared EMAIL_* settings)
- Update upload_to_email task to use dest_email_* settings exclusively
- Update _should_upload_to_email() to check dest_email_* settings
- Update config validator, providers, and settings_service for dest_email_*
- Update .env.demo and docs/ConfigurationGuide.md
- Update all tests to use dest_email_* settings where appropriate"
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix platform-specific %%-d format → use .day and .year directly in templates and messages
- Fix Tailwind JIT dynamic class interpolation → use static class variables in showFlash()
- Fix Jinja pending_date rendering → use .strftime('%B') + .day + .year
- Add aria-atomic=true to flash container for full screen-reader announcements
- Move SessionLocal() creation inside try block in Celery task for proper session management
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add subscription_change_pending_tier and subscription_change_pending_date fields to UserProfile
- Create migration 019_add_subscription_change_pending
- Add apply_pending_subscription_changes(), request_subscription_change(), cancel_pending_subscription_change() utilities
- Add POST /api/subscriptions/change and DELETE /api/subscriptions/change endpoints
- Update GET /api/subscriptions/my to apply pending changes and return pending change info
- Update subscription view to apply pending changes and pass period_start + pending info
- Update subscription.html: per-tier action buttons (upgrade/downgrade/cancel), pending-change banner, period start date
- Add Celery daily task apply_pending_subscription_changes_all at 00:05 UTC
- Add 19 new tests covering all new utility functions and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
google-labs-jules[bot]