feat(watch-folders): add local/FTP/SFTP watch folder ingest with settings, tasks, tests, docs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
@@ -121,9 +121,97 @@ MAX_SINGLE_FILE_SIZE=524288000
|
||||
- **With splitting**: Recommended for servers with limited memory or when processing very large scanned documents
|
||||
- **Higher limits**: For environments specifically designed to handle large architectural plans, books, or scanned archives
|
||||
|
||||
### IMAP Configuration
|
||||
### Watch Folder Ingestion
|
||||
|
||||
DocuElevate can monitor multiple IMAP mailboxes for document attachments. Each mailbox uses a numbered prefix (e.g., `IMAP1_`, `IMAP2_`).
|
||||
DocuElevate can automatically monitor directories for new files and ingest them without any manual action.
|
||||
This works for:
|
||||
- **Local filesystem paths** — including SMB/CIFS shares, NFS mounts, or any path accessible to the Docker container
|
||||
- **FTP server directories** — using the configured FTP connection credentials
|
||||
- **SFTP server directories** — using the configured SFTP connection credentials
|
||||
|
||||
#### Local Watch Folders
|
||||
|
||||
Mount the share or directory into the Docker container and configure one or more paths to watch.
|
||||
|
||||
| **Variable** | **Description** | **Default** |
|
||||
|-------------------------------------|----------------------------------------------------------------------------------------------|-------------|
|
||||
| `WATCH_FOLDERS` | Comma-separated list of **absolute** local filesystem paths to poll for new files. | *(empty)* |
|
||||
| `WATCH_FOLDER_POLL_INTERVAL` | How often to scan the folders, in minutes. | `1` |
|
||||
| `WATCH_FOLDER_DELETE_AFTER_PROCESS` | Delete source files from the watch folder after they are successfully enqueued. When `false`, processed files are tracked in a cache file to prevent re-ingestion. | `false` |
|
||||
|
||||
**Example (docker-compose.yaml):**
|
||||
|
||||
```yaml
|
||||
services:
|
||||
worker:
|
||||
volumes:
|
||||
- /mnt/smb/scanner:/watchfolders/scanner # SMB/CIFS share mounted on the host
|
||||
- /mnt/nfs/inbox:/watchfolders/inbox # NFS mount
|
||||
environment:
|
||||
WATCH_FOLDERS: /watchfolders/scanner,/watchfolders/inbox
|
||||
WATCH_FOLDER_POLL_INTERVAL: 1
|
||||
WATCH_FOLDER_DELETE_AFTER_PROCESS: false
|
||||
```
|
||||
|
||||
> **Tip for HP Scanners and MFPs**: Configure your scanner's "Scan to Network Folder" to point at an SMB share that is also mounted into the DocuElevate worker container. DocuElevate will pick up the scan files automatically every minute. No email forwarding is required.
|
||||
|
||||
#### FTP Ingest (Watch Folder)
|
||||
|
||||
DocuElevate can poll an FTP server directory for new files. It reuses the FTP connection settings already configured for uploads.
|
||||
|
||||
| **Variable** | **Description** | **Default** |
|
||||
|-----------------------------------|------------------------------------------------------------------------------------------------|-------------|
|
||||
| `FTP_INGEST_ENABLED` | Enable FTP folder watching (`true`/`false`). | `false` |
|
||||
| `FTP_INGEST_FOLDER` | Path on the FTP server to poll (e.g. `/incoming`). Uses the existing FTP connection settings. | *(empty)* |
|
||||
| `FTP_INGEST_DELETE_AFTER_PROCESS` | Delete files from the FTP server after they are downloaded and enqueued. | `false` |
|
||||
|
||||
**Example:**
|
||||
|
||||
```dotenv
|
||||
# Existing FTP upload settings (also used for ingest)
|
||||
FTP_HOST=ftp.example.com
|
||||
FTP_USERNAME=docuelevate
|
||||
FTP_PASSWORD=secret
|
||||
|
||||
# FTP ingest configuration
|
||||
FTP_INGEST_ENABLED=true
|
||||
FTP_INGEST_FOLDER=/incoming
|
||||
FTP_INGEST_DELETE_AFTER_PROCESS=false
|
||||
```
|
||||
|
||||
#### SFTP Ingest (Watch Folder)
|
||||
|
||||
DocuElevate can poll an SFTP server directory for new files. It reuses the SFTP connection settings already configured for uploads.
|
||||
|
||||
| **Variable** | **Description** | **Default** |
|
||||
|------------------------------------|-------------------------------------------------------------------------------------------------|-------------|
|
||||
| `SFTP_INGEST_ENABLED` | Enable SFTP folder watching (`true`/`false`). | `false` |
|
||||
| `SFTP_INGEST_FOLDER` | Path on the SFTP server to poll (e.g. `/uploads/inbox`). Uses the existing SFTP connection settings. | *(empty)* |
|
||||
| `SFTP_INGEST_DELETE_AFTER_PROCESS` | Delete files from the SFTP server after they are downloaded and enqueued. | `false` |
|
||||
|
||||
**Example:**
|
||||
|
||||
```dotenv
|
||||
# Existing SFTP upload settings (also used for ingest)
|
||||
SFTP_HOST=sftp.example.com
|
||||
SFTP_USERNAME=docuelevate
|
||||
SFTP_PRIVATE_KEY=/run/secrets/sftp_key
|
||||
|
||||
# SFTP ingest configuration
|
||||
SFTP_INGEST_ENABLED=true
|
||||
SFTP_INGEST_FOLDER=/uploads/inbox
|
||||
SFTP_INGEST_DELETE_AFTER_PROCESS=false
|
||||
```
|
||||
|
||||
#### Supported File Types for Watch Folders
|
||||
|
||||
Watch folder ingestion accepts the same file types as the web upload interface: PDF, Word, Excel, PowerPoint, images (JPEG, PNG, TIFF, BMP, GIF), plain text, CSV, RTF, and more. Unsupported files (executables, archives, etc.) are silently skipped.
|
||||
|
||||
### IMAP Email Ingestion
|
||||
|
||||
DocuElevate can automatically pull document attachments from IMAP mailboxes — no need to forward emails manually. Configure one or two mailboxes and DocuElevate polls them on the schedule you set.
|
||||
|
||||
> **For HP Scanners (Scan to Email)**: If your scanner is set up to email scanned documents to a dedicated mailbox, configure that mailbox in DocuElevate using the settings below. DocuElevate will automatically retrieve the scanned PDFs from the inbox and process them. You do **not** need to configure DocuElevate as an email server — it acts as an email *client* that reads from your existing mailbox.
|
||||
|
||||
| **Variable** | **Description** | **Example** |
|
||||
|-------------------------------|--------------------------------------------------------------|-------------------|
|
||||
|
||||
+29
-6
@@ -74,14 +74,37 @@ For even more convenience, you can upload files directly from the **Files** page
|
||||
|
||||
This feature allows you to quickly add new files without navigating away from your document management view.
|
||||
|
||||
### Email Attachments
|
||||
### Email Attachments (IMAP Ingestion)
|
||||
|
||||
If configured, DocuElevate can automatically fetch documents from email attachments:
|
||||
DocuElevate acts as an email *client* that automatically retrieves document attachments from one or more IMAP mailboxes. You do not need to set up DocuElevate as an email server — it simply polls an existing mailbox that you designate for document delivery.
|
||||
|
||||
1. Send an email with attachments to the configured email account
|
||||
2. DocuElevate will poll the mailbox at the configured interval
|
||||
3. Attachments will be automatically downloaded and processed
|
||||
4. No further action is required
|
||||
**How it works:**
|
||||
1. A document is sent as an email attachment to the configured mailbox (e.g. from a scanner, a colleague, or any email client)
|
||||
2. DocuElevate polls the mailbox at the configured interval (typically every 1–5 minutes)
|
||||
3. Email attachments in supported formats are automatically downloaded and enqueued for processing
|
||||
4. Processed emails are marked with a label or star (Gmail) or tracked locally, so they are not re-processed
|
||||
|
||||
> **HP Scanners and MFPs (Scan to Email)**: Configure your scanner's "Scan to Email" feature to send scanned documents to a dedicated email account. Point DocuElevate at that mailbox using the IMAP settings. DocuElevate will retrieve the scanned PDFs automatically — no manual forwarding required.
|
||||
|
||||
### Watch Folders (Automatic Folder Ingestion)
|
||||
|
||||
Watch folders allow DocuElevate to automatically monitor directories for new files and ingest them without any manual action.
|
||||
|
||||
#### Local Watch Folders (including SMB/CIFS and NFS)
|
||||
|
||||
Mount a network share or local directory into the DocuElevate worker container and configure the path in `WATCH_FOLDERS`. DocuElevate scans the folder every minute (configurable via `WATCH_FOLDER_POLL_INTERVAL`) and enqueues any new documents it finds.
|
||||
|
||||
This is the recommended approach for:
|
||||
- **HP Scanners / MFPs** using "Scan to Network Folder" — point the scanner at a shared folder that DocuElevate also has access to
|
||||
- **SMB/CIFS shares** — mount the Windows/Samba share and add the path to `WATCH_FOLDERS`
|
||||
- **NFS mounts** — works identically, just configure the mount path
|
||||
- **Any local directory** on the server running DocuElevate
|
||||
|
||||
#### FTP / SFTP Watch Folders
|
||||
|
||||
DocuElevate can poll an FTP or SFTP directory for new files. Enable this with `FTP_INGEST_ENABLED` or `SFTP_INGEST_ENABLED` and set the corresponding ingest folder. DocuElevate downloads new files, enqueues them for processing, and optionally deletes them from the remote server.
|
||||
|
||||
See [Configuration Guide — Watch Folder Ingestion](ConfigurationGuide.md#watch-folder-ingestion) for full setup instructions.
|
||||
|
||||
## Managing Documents
|
||||
|
||||
|
||||
Reference in New Issue
Block a user