- Improve warning log in _resolve_categories_for_profile() to include
exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
full profiles documentation including category table and API reference
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a configurable switch to control which attachment types are ingested
via IMAP. Images are excluded by default; office files and PDFs are ingested.
- Add global `IMAP_ATTACHMENT_FILTER` config setting (default: `documents_only`)
- Add `attachment_filter` column to `UserImapAccount` model for per-user override
- Migration 032 adds the column to `user_imap_accounts` table
- Update `fetch_attachments_and_enqueue()` to respect filter (documents_only/all)
- Update `pull_inbox()`, `_pull_user_imap_accounts()`, and
`_pull_user_integration_imap()` to pass the resolved filter
- Update IMAP accounts API (schemas, create/update handlers, response serializer)
- Update IMAP accounts UI to show attachment filter dropdown in modal and
display filter badges on account cards
- Add 6 new tests covering attachment filter behaviour
- Update ConfigurationGuide.md, EmailIngestion.md, and .env.demo
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add iCloud Drive as a new storage destination using the pyicloud library.
Includes upload task, configuration, user integration handler, provider
status, onboarding support, and comprehensive tests.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix _get_pipeline_ocr_language: remove redundant `or None` in step_config.get()
- Add Session type hint to _get_pipeline_ocr_language db parameter via TYPE_CHECKING
- Update process_with_ocr to use modern str | None syntax instead of Optional[str]
- Fix test_get_pipeline_ocr_language_explicit_pipeline_takes_priority: properly add
sys_step to db_session so the system pipeline step is persisted in the test DB
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add OCR_LANGUAGES constant (28 languages, EN/DE/FR/ES/IT/PT/RU/ZH/JA/KO/AR/etc.)
- Add TESSERACT_TO_EASYOCR mapping for automatic code translation
- Add optional language constructor arg to TesseractOCRProvider/EasyOCRProvider
- Update get_ocr_providers() to accept and pass per-call language override
- Add language parameter to process_with_ocr Celery task
- Add _get_pipeline_ocr_language() helper to resolve OCR language from pipeline step config
- Update process_document to look up and pass pipeline OCR language to process_with_ocr
- Add ocr_language select config field (28 options) to pipeline OCR step schema
- Add language dropdown to pipeline UI (pipelines.html)
- Update docs/UserGuide.md and docs/API.md with language override documentation
- Add 27 new tests covering language constants, provider overrides, and pipeline lookup
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/tasks/upload_to_user_integration.py: new Celery task that
uploads a processed document to a specific UserIntegration using its
own per-user config and Fernet-decrypted credentials. Supports all
DESTINATION types: Dropbox, S3, Google Drive, OneDrive, WebDAV,
Nextcloud, FTP, SFTP, Paperless-ngx, Email (SMTP), and Rclone.
- Extend app/tasks/send_to_all.py: add send_to_user_destinations task
(queries active DESTINATION UserIntegrations for an owner and
dispatches one upload_to_user_integration task per integration) and
get_user_destination_count helper used by finalize_document_storage.
- Refactor app/tasks/finalize_document_storage.py: after processing,
look up the document owner; if the owner has active DESTINATION
integrations route exclusively to those (user-specific routing),
otherwise fall back to the global send_to_all_destinations.
- Update tests/test_finalize_storage.py: add autouse fixture to prevent
Redis hangs, update all existing tests with new mock parameters, add
TestFinalizeDocumentStorageUserRouting class with four new tests that
validate user-specific vs global routing decisions.
- Add tests/test_user_integration_upload.py: 14 new unit tests covering
upload_to_user_integration (handler dispatch, error persistence,
last_used_at update, credential decryption, skip for unknown types)
and send_to_user_destinations / get_user_destination_count.
- Update docs/StorageArchitecture.md: document the user-specific
destination routing feature, supported types, multiple-destination
behaviour, and global fallback semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add owner_id parameter to pull_inbox() and fetch_attachments_and_enqueue()
to attribute ingested documents to the correct user
- Add _pull_user_integration_imap() to poll IMAP sources from UserIntegration model
- Add _pull_user_integration_watch_folders() to scan watch folders from UserIntegration model
- Add _is_safe_watch_path() for path traversal security on user-configured paths
- Add _scan_user_watch_folder() that passes owner_id to _enqueue_file()
- Update _enqueue_file() to forward owner_id to process_document/convert_to_pdf
- Update celery beat schedule to always enable IMAP and watch folder polling
(user integrations can exist without system-level config)
- Ensure individual connection failures don't crash the polling loop
- Update existing tests for new function signatures
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Create app/templates/email/default.html (fixes 'default.html not found' error)
- Add DEST_EMAIL_* settings to app/config.py (decoupled from shared EMAIL_* settings)
- Update upload_to_email task to use dest_email_* settings exclusively
- Update _should_upload_to_email() to check dest_email_* settings
- Update config validator, providers, and settings_service for dest_email_*
- Update .env.demo and docs/ConfigurationGuide.md
- Update all tests to use dest_email_* settings where appropriate"
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix platform-specific %%-d format → use .day and .year directly in templates and messages
- Fix Tailwind JIT dynamic class interpolation → use static class variables in showFlash()
- Fix Jinja pending_date rendering → use .strftime('%B') + .day + .year
- Add aria-atomic=true to flash container for full screen-reader announcements
- Move SessionLocal() creation inside try block in Celery task for proper session management
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add subscription_change_pending_tier and subscription_change_pending_date fields to UserProfile
- Create migration 019_add_subscription_change_pending
- Add apply_pending_subscription_changes(), request_subscription_change(), cancel_pending_subscription_change() utilities
- Add POST /api/subscriptions/change and DELETE /api/subscriptions/change endpoints
- Update GET /api/subscriptions/my to apply pending changes and return pending change info
- Update subscription view to apply pending changes and pass period_start + pending info
- Update subscription.html: per-tier action buttons (upgrade/downgrade/cancel), pending-change banner, period start date
- Add Celery daily task apply_pending_subscription_changes_all at 00:05 UTC
- Add 19 new tests covering all new utility functions and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.
- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
completed filter
- process_document: call initialize_file_steps after creating a new
file record so all mandatory steps are pre-created as pending
Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.
Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Christian Krakau-Louis