Merges origin/main (v0.155.0) into the classification feature branch,
resolving all 18 conflicted files by accepting main's version and
re-applying only classification-specific additions:
- Renumber migration from 037 to 038 (chains from 037_user_sessions)
- Re-add ClassificationRuleModel to models.py, env.py, conftest.py
- Re-add classification_rules_router to api/__init__.py
- All session management, QR auth, and devices code preserved from main
Migration chain validated. 62 classification tests pass.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Merge origin/main into feature branch, resolving 3 conflicts:
- app/api/__init__.py: add classification_rules_router alongside new
routers from main (audit_logs, i18n, mobile, compliance, translation)
- app/models.py: keep ClassificationRuleModel alongside new models from
main (MobileDevice, ComplianceTemplate, PipelineRoutingRule)
- tests/conftest.py: import both ClassificationRuleModel and new models
from main (AuditLog, ComplianceTemplate)
Also renumber migration from 027 to 037 to chain from the latest
migration on main (036_add_document_translation_fields).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Implements the classify pipeline step with:
- Classification rules engine (app/utils/classification_rules.py) with
pre-built categories (invoice, contract, receipt, letter, report,
bank_statement, tax_document, insurance, payslip) and support for
filename patterns, content keywords, and metadata matching rules
- Celery task (app/tasks/classify_document.py) that runs as a pipeline step
- CRUD API (app/api/classification_rules.py) for managing custom rules
- ClassificationRuleModel in app/models.py with migration 027
- Updated pipeline step config_schema and stage mapping
- Comprehensive tests for engine, API, and task
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add user_mode to dropbox/onedrive/google_drive setup views that loads integration config
- Show user-friendly auth wizard when integration_id is provided (user mode)
- In user mode: show integration name, current folder, back-to-integrations link
- In callback templates: only save credentials (not config) for user integrations
- In integrations dashboard: show Authorize/Re-Authorize button for all OAuth types
- Add WATCH_FOLDER OAuth support: detect source_type in config for auth button
- isOAuthType() and oauthLink() now accept full integration object
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add `from app.tasks.upload_to_user_integration import upload_to_user_integration`
to app/celery_worker.py — fixes TestCeleryWorkerConfig test that enforces
every tasks/ module is registered with Celery.
- Add tests/test_upload_handlers.py with 43 unit tests covering all 11
per-type upload handler functions (_upload_dropbox, _upload_s3,
_upload_google_drive, _upload_onedrive, _upload_webdav, _upload_nextcloud,
_upload_ftp, _upload_sftp, _upload_paperless, _upload_email, _upload_rclone)
plus 2 additional finalize_document_storage branch tests. All external
libraries (dropbox, boto3, msal, paramiko, smtplib, subprocess, requests)
are mocked so tests are hermetic and fast. Coverage on changed files:
upload_to_user_integration.py 94.71%, finalize_document_storage.py 95.51%
(both well above the 70% Codecov diff threshold).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Move init_sentry() from module level into the FastAPI lifespan context
manager, immediately after load_settings_from_db() completes. This
ensures that SENTRY_DSN and other Sentry settings configured via the
database admin UI are picked up on every restart.
Also update tests and docs accordingly.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Use per-password random salt with PBKDF2-HMAC-SHA256 (stored as salt:hash)
- Increase PBKDF2 iterations to 600,000 (OWASP 2023 recommendation)
- Password for downloads now accepted via POST body (never URL query param)
- Fail download request if view count cannot be incremented (prevents bypass)
- Update tests to match new hashing format and POST password download
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add SharedLink model with token, expiry, view limit, password hash
- Add migration 025_add_shared_links
- Add API endpoints: create, list, revoke (auth) + public info/download
- Add management UI at /shared-links with revoke controls
- Add public share landing page at /share/{token}
- Add Share button on file_view.html
- Add Shared Links to user dropdown in common.js
- Write 35 unit tests covering all scenarios
- Update UserGuide.md with sharing documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix _get_pipeline_ocr_language: remove redundant `or None` in step_config.get()
- Add Session type hint to _get_pipeline_ocr_language db parameter via TYPE_CHECKING
- Update process_with_ocr to use modern str | None syntax instead of Optional[str]
- Fix test_get_pipeline_ocr_language_explicit_pipeline_takes_priority: properly add
sys_step to db_session so the system pipeline step is persisted in the test DB
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add OCR_LANGUAGES constant (28 languages, EN/DE/FR/ES/IT/PT/RU/ZH/JA/KO/AR/etc.)
- Add TESSERACT_TO_EASYOCR mapping for automatic code translation
- Add optional language constructor arg to TesseractOCRProvider/EasyOCRProvider
- Update get_ocr_providers() to accept and pass per-call language override
- Add language parameter to process_with_ocr Celery task
- Add _get_pipeline_ocr_language() helper to resolve OCR language from pipeline step config
- Update process_document to look up and pass pipeline OCR language to process_with_ocr
- Add ocr_language select config field (28 options) to pipeline OCR step schema
- Add language dropdown to pipeline UI (pipelines.html)
- Update docs/UserGuide.md and docs/API.md with language override documentation
- Add 27 new tests covering language constants, provider overrides, and pipeline lookup
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/tasks/upload_to_user_integration.py: new Celery task that
uploads a processed document to a specific UserIntegration using its
own per-user config and Fernet-decrypted credentials. Supports all
DESTINATION types: Dropbox, S3, Google Drive, OneDrive, WebDAV,
Nextcloud, FTP, SFTP, Paperless-ngx, Email (SMTP), and Rclone.
- Extend app/tasks/send_to_all.py: add send_to_user_destinations task
(queries active DESTINATION UserIntegrations for an owner and
dispatches one upload_to_user_integration task per integration) and
get_user_destination_count helper used by finalize_document_storage.
- Refactor app/tasks/finalize_document_storage.py: after processing,
look up the document owner; if the owner has active DESTINATION
integrations route exclusively to those (user-specific routing),
otherwise fall back to the global send_to_all_destinations.
- Update tests/test_finalize_storage.py: add autouse fixture to prevent
Redis hangs, update all existing tests with new mock parameters, add
TestFinalizeDocumentStorageUserRouting class with four new tests that
validate user-specific vs global routing decisions.
- Add tests/test_user_integration_upload.py: 14 new unit tests covering
upload_to_user_integration (handler dispatch, error persistence,
last_used_at update, credential decryption, skip for unknown types)
and send_to_user_destinations / get_user_destination_count.
- Update docs/StorageArchitecture.md: document the user-specific
destination routing feature, supported types, multiple-destination
behaviour, and global fallback semantics.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Convert f-string log interpolation to %s-style formatting in
app/api/pipelines.py and app/api/saved_searches.py to prevent
clear-text logging of request-derived data (CodeQL: clear-text
logging of sensitive information)
- Replace plain hashlib.sha256() with PBKDF2-HMAC-SHA256 via
hash_token() in app/auth.py for Bearer token verification,
consistent with how tokens are stored in api_tokens.py (CodeQL:
use of weak cryptographic hashing on sensitive data)
- Remove redundant {exc} from logger.exception() calls (the
traceback is already captured by logger.exception())
- Update test to verify PBKDF2 hash instead of plain SHA-256
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Update API.md with API token management endpoints, usage examples,
and authentication guide. Update UserGuide.md with webhook ingestion
and API tokens sections.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add 'Authorize' button to integration cards for OAuth types without credentials
- Add isOAuthType() helper and update info box in create/edit modal
- Accept integration_id query param in Dropbox, Google Drive, OneDrive setup views
- Store integration_id in sessionStorage on setup pages
- Add per-user flow in OAuth callbacks: PUT credentials to /api/integrations/{id}
- Preserve existing global flow as fallback when no integration_id is present
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add owner_id parameter to pull_inbox() and fetch_attachments_and_enqueue()
to attribute ingested documents to the correct user
- Add _pull_user_integration_imap() to poll IMAP sources from UserIntegration model
- Add _pull_user_integration_watch_folders() to scan watch folders from UserIntegration model
- Add _is_safe_watch_path() for path traversal security on user-configured paths
- Add _scan_user_watch_folder() that passes owner_id to _enqueue_file()
- Update _enqueue_file() to forward owner_id to process_document/convert_to_pdf
- Update celery beat schedule to always enable IMAP and watch folder polling
(user integrations can exist without system-level config)
- Ensure individual connection failures don't crash the polling loop
- Update existing tests for new function signatures
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Pass authenticated user's name, email, and username to Zammad form
and chat widgets. For the ticket form: pre-fill name/email fields
and append a User Context metadata block to the ticket body via
$.ajaxPrefilter. For the chat widget: pass name/email to the
ZammadChat constructor.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]