feat(tasks): dynamic routing to user-specific destination integrations
- Add app/tasks/upload_to_user_integration.py: new Celery task that uploads a processed document to a specific UserIntegration using its own per-user config and Fernet-decrypted credentials. Supports all DESTINATION types: Dropbox, S3, Google Drive, OneDrive, WebDAV, Nextcloud, FTP, SFTP, Paperless-ngx, Email (SMTP), and Rclone. - Extend app/tasks/send_to_all.py: add send_to_user_destinations task (queries active DESTINATION UserIntegrations for an owner and dispatches one upload_to_user_integration task per integration) and get_user_destination_count helper used by finalize_document_storage. - Refactor app/tasks/finalize_document_storage.py: after processing, look up the document owner; if the owner has active DESTINATION integrations route exclusively to those (user-specific routing), otherwise fall back to the global send_to_all_destinations. - Update tests/test_finalize_storage.py: add autouse fixture to prevent Redis hangs, update all existing tests with new mock parameters, add TestFinalizeDocumentStorageUserRouting class with four new tests that validate user-specific vs global routing decisions. - Add tests/test_user_integration_upload.py: 14 new unit tests covering upload_to_user_integration (handler dispatch, error persistence, last_used_at update, credential decryption, skip for unknown types) and send_to_user_destinations / get_user_destination_count. - Update docs/StorageArchitecture.md: document the user-specific destination routing feature, supported types, multiple-destination behaviour, and global fallback semantics. Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -295,6 +295,78 @@ with SessionLocal() as db:
|
||||
pass
|
||||
```
|
||||
|
||||
## User-Specific Destination Routing
|
||||
|
||||
### Overview
|
||||
|
||||
When a document has an identified owner (non-anonymous user), DocuElevate
|
||||
routes the processed file to **that user's own configured destinations** instead
|
||||
of the system-wide global destinations. This enables true multi-tenant
|
||||
operation: each user's documents are stored where *they* configured, using
|
||||
*their* OAuth tokens or API credentials.
|
||||
|
||||
### Routing Decision
|
||||
|
||||
The routing decision is made in `finalize_document_storage` after all
|
||||
processing steps are complete:
|
||||
|
||||
```
|
||||
Document owner has active DESTINATION integrations?
|
||||
├── YES → send_to_user_destinations (user-specific routing)
|
||||
└── NO → send_to_all_destinations (global fallback)
|
||||
```
|
||||
|
||||
"Active DESTINATION integrations" means rows in the `user_integrations` table
|
||||
where `owner_id` matches, `direction = "DESTINATION"`, and `is_active = True`.
|
||||
|
||||
### User Integrations as Destinations
|
||||
|
||||
Users configure their own upload targets via the **Integrations** dashboard
|
||||
(`/integrations`). A DESTINATION integration stores:
|
||||
|
||||
- **Config** (`config` column, JSON): non-sensitive settings such as bucket
|
||||
name, remote folder, SMTP host, etc.
|
||||
- **Credentials** (`credentials` column, Fernet-encrypted JSON): sensitive
|
||||
values such as OAuth refresh tokens, API keys, and passwords.
|
||||
|
||||
When uploading, credentials are decrypted at task execution time and passed
|
||||
directly to the appropriate upload handler — they never appear in plain text
|
||||
in task messages or logs.
|
||||
|
||||
### Supported Destination Types
|
||||
|
||||
| Integration Type | Upload Method |
|
||||
|-----------------|--------------|
|
||||
| `DROPBOX` | Dropbox SDK, OAuth refresh-token flow |
|
||||
| `S3` | boto3 `upload_file`, per-user access key |
|
||||
| `GOOGLE_DRIVE` | Google Drive API v3, OAuth or service account |
|
||||
| `ONEDRIVE` | Microsoft Graph API, MSAL confidential-client |
|
||||
| `WEBDAV` | HTTP PUT request, Basic Auth |
|
||||
| `NEXTCLOUD` | WebDAV (same as WEBDAV, Nextcloud-compatible path) |
|
||||
| `FTP` | ftplib FTPS (TLS preferred, plaintext configurable) |
|
||||
| `SFTP` | Paramiko, password or private-key auth |
|
||||
| `PAPERLESS` | Paperless-ngx REST API, API token |
|
||||
| `EMAIL` | SMTP/STARTTLS, file as attachment |
|
||||
| `RCLONE` | `rclone copyto` subprocess, per-user rclone config |
|
||||
|
||||
### Multiple Destinations
|
||||
|
||||
If a user configures multiple active DESTINATION integrations, the file is
|
||||
uploaded to **each one asynchronously and independently**. Success or failure
|
||||
per destination is logged separately so a single failing destination does not
|
||||
block the others.
|
||||
|
||||
### Fallback to Global Destinations
|
||||
|
||||
Global destinations (configured via environment variables / admin settings)
|
||||
are used whenever:
|
||||
|
||||
- The document has no owner (`owner_id` is `None`), e.g., uploaded in
|
||||
single-user / anonymous mode.
|
||||
- The owner exists but has **zero** active DESTINATION integrations.
|
||||
|
||||
This ensures backward compatibility with existing single-user deployments.
|
||||
|
||||
## See Also
|
||||
|
||||
- [API Documentation](API.md) - API endpoints for file operations
|
||||
|
||||
Reference in New Issue
Block a user