fix(imap): address code review feedback on ingestion profiles
- Improve warning log in _resolve_categories_for_profile() to include
exception type name for better troubleshooting
- Add SQLAlchemy IS NULL comment to imap_profiles.py filter
- Pass default_categories from server to template to avoid hardcoded
category list in JS (now uses {{ default_categories | tojson }})
- Simplify view profiles query (remove redundant unauthenticated path)
- Update docs: ConfigurationGuide.md and EmailIngestion.md with
full profiles documentation including category table and API reference
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
@@ -304,7 +304,42 @@ DocuElevate can automatically pull document attachments from IMAP mailboxes —
|
||||
| `IMAP1_SSL` | Use SSL (`true`/`false`). | `true` |
|
||||
| `IMAP1_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll for new mail. | `5` |
|
||||
| `IMAP_READONLY_MODE` | When `true`, fetches and processes attachments but does **not** modify the mailbox (no starring, labeling, deleting, or flag changes). Use for pre-production instances sharing a mailbox with production. Default: `false`. | `false` |
|
||||
| `IMAP_ATTACHMENT_FILTER` | Controls which attachment types are ingested from emails. `documents_only` (default) ingests PDFs and office files only — images are skipped. `all` ingests every supported file type including images. Individual per-user IMAP accounts can override this global default. | `documents_only` |
|
||||
| `IMAP_ATTACHMENT_FILTER` | System-wide fallback for which attachment types are ingested when no ingestion profile is assigned to a mailbox. `documents_only` (default) ingests PDFs and office files only — images are skipped. `all` ingests every supported file type including images. Individual IMAP accounts can override this using ingestion profiles. | `documents_only` |
|
||||
|
||||
#### IMAP Ingestion Profiles
|
||||
|
||||
For fine-grained control, DocuElevate supports **Ingestion Profiles** — named configurations that let you choose exactly which file-type categories to accept from each mailbox.
|
||||
|
||||
Each profile contains a list of enabled **categories**:
|
||||
|
||||
| Category | Description |
|
||||
|----------|-------------|
|
||||
| `pdf` | PDF documents (`.pdf`) |
|
||||
| `office` | Microsoft Office files (Word, Excel, PowerPoint — `.docx`, `.xlsx`, `.pptx`, …) |
|
||||
| `opendocument` | LibreOffice/OpenOffice files (`.odt`, `.ods`, `.odp`, …) |
|
||||
| `text` | Plain text, CSV and RTF files (`.txt`, `.csv`, `.rtf`) |
|
||||
| `web` | HTML and Markdown files (`.html`, `.htm`, `.md`, `.markdown`) |
|
||||
| `images` | Image files (`.jpg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg`) |
|
||||
|
||||
Two built-in system profiles are seeded automatically:
|
||||
|
||||
| Profile | Categories |
|
||||
|---------|------------|
|
||||
| **Documents Only** | pdf, office, opendocument, text, web (no images) |
|
||||
| **All Files** | All categories, including images |
|
||||
|
||||
Users can create their own custom profiles via the **Email Ingestion** dashboard (`/imap-accounts`) by clicking the **Manage profiles** link or the **+** button next to the profile dropdown. Custom profiles are private to the creating user and can be freely edited or deleted.
|
||||
|
||||
**API endpoints for ingestion profiles:**
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|--------|----------|-------------|
|
||||
| `GET` | `/api/imap-profiles/` | List all visible profiles (system + user's own) |
|
||||
| `POST` | `/api/imap-profiles/` | Create a new profile |
|
||||
| `GET` | `/api/imap-profiles/categories` | List available file-type categories |
|
||||
| `GET` | `/api/imap-profiles/{id}` | Get a single profile |
|
||||
| `PUT` | `/api/imap-profiles/{id}` | Update a profile (not built-in) |
|
||||
| `DELETE` | `/api/imap-profiles/{id}` | Delete a profile (not built-in) |
|
||||
|
||||
#### Per-User IMAP Integrations
|
||||
|
||||
|
||||
@@ -69,30 +69,47 @@ By default, DocuElevate only ingests **document** attachments (PDFs, Word, Excel
|
||||
|
||||
#### Global Default (Admin Setting)
|
||||
|
||||
Set the `IMAP_ATTACHMENT_FILTER` environment variable to control the system-wide default:
|
||||
Set the `IMAP_ATTACHMENT_FILTER` environment variable to control the system-wide fallback when no ingestion profile is assigned to a mailbox:
|
||||
|
||||
| Value | Behaviour |
|
||||
|-------|-----------|
|
||||
| `documents_only` | **(Default)** Only PDFs and office/document files. Images (JPEG, PNG, GIF, BMP, TIFF, WebP, SVG) are skipped. |
|
||||
| `documents_only` | **(Default)** Only PDFs and office/document files. Images are skipped. |
|
||||
| `all` | All supported file types, including images. |
|
||||
|
||||
```env
|
||||
# Only ingest document-type attachments (default behaviour)
|
||||
IMAP_ATTACHMENT_FILTER=documents_only
|
||||
|
||||
# Ingest all supported file types, including images
|
||||
IMAP_ATTACHMENT_FILTER=all
|
||||
```
|
||||
|
||||
#### Per-User Override
|
||||
#### Ingestion Profiles (Fine-Grained Per-Mailbox Control)
|
||||
|
||||
Each user can override the global default for their personal IMAP accounts via the **Email Ingestion** dashboard (`/imap-accounts`). When creating or editing an account, select the desired setting from the **Attachment Types to Ingest** dropdown:
|
||||
For precise control, you can create **Ingestion Profiles** that let you pick exactly which file-type categories to accept from each mailbox. This is more powerful than the binary global toggle and works independently per mailbox.
|
||||
|
||||
- **Use global default** — inherits the `IMAP_ATTACHMENT_FILTER` setting above.
|
||||
- **Documents only** — PDFs and office files, no images.
|
||||
- **All supported types (including images)** — overrides the global setting to allow images for this specific account.
|
||||
**Available categories:**
|
||||
|
||||
This allows administrators to restrict image ingestion system-wide while individual users can opt-in to image ingestion on a per-mailbox basis.
|
||||
| Category | File types included |
|
||||
|----------|---------------------|
|
||||
| PDF | `.pdf` |
|
||||
| Microsoft Office | `.doc`, `.docx`, `.xls`, `.xlsx`, `.ppt`, `.pptx`, and macro-enabled variants |
|
||||
| OpenDocument | `.odt`, `.ods`, `.odp`, `.odg`, `.odf` (LibreOffice / OpenOffice) |
|
||||
| Text & Data | `.txt`, `.csv`, `.rtf` |
|
||||
| Web & Markup | `.html`, `.htm`, `.md`, `.markdown` |
|
||||
| Images | `.jpg`, `.png`, `.gif`, `.bmp`, `.tiff`, `.webp`, `.svg` |
|
||||
|
||||
**Managing profiles:**
|
||||
|
||||
1. Go to **Email Ingestion** (`/imap-accounts`)
|
||||
2. Click **Manage profiles** (or the **+** icon next to the profile dropdown)
|
||||
3. Create a new profile, give it a name, and tick the categories you want
|
||||
4. When adding or editing a mailbox, select your profile from the dropdown
|
||||
|
||||
Two built-in profiles are always available and cannot be deleted:
|
||||
|
||||
- **Documents Only** — PDF, Office, OpenDocument, Text, Web (no images)
|
||||
- **All Files** — all categories including images
|
||||
|
||||
Users can also create unlimited **custom profiles** to mix and match exactly the categories they need per mailbox (e.g. a scanner mailbox that only accepts PDFs, or a finance mailbox that accepts Office and CSV but not images).
|
||||
|
||||
Custom profiles are created via the UI or the `/api/imap-profiles/` API.
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user