Update documentation for hybrid config and Gmail API vs SMTP delivery

- README.md: document hybrid config model, Gmail API vs SMTP comparison, update architecture diagram, replace Postmarkapp with Apprise - ARCHITECTURE.md: add Gmail API vs SMTP comparison table, document ConfigService and AppSetting, add settings/gmail API endpoints - CHANGELOG.md: add entries for database-backed config and documentation - TODO.md: update progress (59% coverage, production readiness 30%) - .env.example: document bootstrap vs database-managed settings Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/ce88d4a8-d8c2-49b3-95a1-30592105a769
2026-03-23 13:39:52 +00:00
parent 702650376e
commit 71a97379a1
5 changed files with 209 additions and 41 deletions
@@ -12,13 +12,14 @@ A Docker-based solution that automatically fetches emails from POP3 mailboxes an

 ## Features

- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables
- **Automatic Forwarding** — sends emails to your Gmail account via SMTP
+- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes
+- **Dual Delivery** — inject emails via **Gmail API** (preferred) or forward via **SMTP**
+- **Hybrid Configuration** — configure via environment variables, `.env` files, **or** the database
 - **Smart Throttling** — configurable rate limiting to stay within Gmail quotas
- **Error Reporting** — notifications via Postmarkapp when issues occur
+- **Error Reporting** — multi-channel notifications (Apprise: email, Telegram, Slack, Discord, webhooks)
 - **Scheduled Polling** — configurable check intervals (default: every 5 minutes)
 - **Docker Ready** — fully containerized with Docker Compose support
- **Secure** — runs as non-root user, SSL/TLS connections
+- **Secure** — runs as non-root user, SSL/TLS connections, encrypted credential storage

 ### SaaS Platform (in development)

@@ -53,6 +54,20 @@ See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions.

 ## Configuration

+### Hybrid Configuration (Environment + Database)
+
+The application supports a **hybrid configuration model**:
+
+| Source | Priority | Use For |
+|--------|----------|---------|
+| **Database** (`app_settings` table) | Highest | SMTP, processing, Gmail API, notifications |
+| **Environment variables / `.env`** | Fallback | All settings; required for bootstrap settings |
+| **Built-in defaults** | Lowest | Sensible defaults for all non-bootstrap settings |
+
+**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) always come from environment variables because the database connection depends on them.
+
+All other settings (SMTP, processing intervals, Gmail API, etc.) can be managed via the admin API at `/api/v1/settings` and are stored in the PostgreSQL database. When a database setting exists, it takes priority over the corresponding environment variable.
+
 ### POP3 Accounts

 Add multiple POP3 accounts by incrementing the account number in your `.env`:
@@ -67,15 +82,62 @@ POP3_ACCOUNT_2_USER=user2@provider2.com
 POP3_ACCOUNT_2_PASSWORD=password2
 ```

-### Gmail App Password
+### Email Delivery Methods
+
+The forwarder supports two delivery methods for getting emails into Gmail:
+
+#### Gmail API Injection (Preferred)
+
+Emails are injected directly into your Gmail account using Google's `users.messages.insert()` API. This is the **recommended method** because it:
+
+- Preserves original email headers and metadata exactly as-is
+- Does not modify `From`, `Reply-To`, or `Message-ID` headers
+- Applies Gmail labels (e.g., `INBOX`) on injection
+- Does not count against Gmail's SMTP sending quotas
+- Does not require an SMTP App Password
+
+**Setup:**
+
+1. Configure Google OAuth2 credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`)
+2. Authenticate via the SaaS web UI or API (`POST /api/v1/providers/gmail-credential`)
+3. Set `delivery_method` to `gmail_api` when creating mail accounts
+
+**Required OAuth2 Scopes:**
+- `https://www.googleapis.com/auth/gmail.insert`
+- `https://www.googleapis.com/auth/gmail.labels`
+
+#### SMTP Forwarding (Fallback)
+
+Emails are forwarded to Gmail via SMTP. This is the legacy method and is used as a fallback when Gmail API credentials are not available.
+
+**Limitations vs Gmail API:**
+- Modifies email headers (adds `Received`, may rewrite `From`)
+- Counts against Gmail's SMTP sending quota (500/day for free accounts)
+- Requires a Gmail App Password (see below)
+- May trigger spam filters for forwarded mail
+
+**Setup:**

 1. Go to your [Google Account Security](https://myaccount.google.com/security)
 2. Under "Signing in to Google," select **App Passwords**
 3. Generate a new app password for "Mail"
-4. Use this password as `SMTP_PASSWORD`
+4. Set `SMTP_PASSWORD` in your environment or database settings

 ### Environment Variables

+> **Note:** All settings marked ★ can also be managed via the database
+> through the admin API (`/api/v1/settings`). Database values take precedence.
+
+#### Bootstrap Settings (env only)
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `DATABASE_URL` | Yes | `postgresql+asyncpg://...` | PostgreSQL connection string |
+| `SECRET_KEY` | Yes | — | JWT signing key (min 32 chars) |
+| `ENCRYPTION_KEY` | Yes | — | Credential encryption key (min 32 chars) |
+
+#### POP3/IMAP Accounts (env only — or via API)
+
 | Variable | Required | Default | Description |
 |----------|----------|---------|-------------|
 | `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname |
@@ -83,18 +145,32 @@ POP3_ACCOUNT_2_PASSWORD=password2
 | `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username |
 | `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password |
 | `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS |
+
+#### SMTP Settings ★
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
 | `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server |
 | `SMTP_PORT` | No | `587` | SMTP port |
-| `SMTP_USER` | Yes | — | SMTP username |
-| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) |
+| `SMTP_USER` | For SMTP | — | SMTP username |
+| `SMTP_PASSWORD` | For SMTP | — | SMTP password (App Password) |
 | `SMTP_USE_TLS` | No | `true` | Use STARTTLS |
-| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address |
+
+#### Gmail API Settings ★
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
+| `GOOGLE_CLIENT_ID` | For Gmail API | — | Google OAuth2 client ID |
+| `GOOGLE_CLIENT_SECRET` | For Gmail API | — | Google OAuth2 client secret |
+| `GMAIL_API_ENABLED` | No | `true` | Enable Gmail API delivery |
+
+#### Processing Settings ★
+
+| Variable | Required | Default | Description |
+|----------|----------|---------|-------------|
 | `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval |
 | `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run |
 | `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit |
-| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token |
-| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender |
-| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient |
 | `LOG_LEVEL` | No | `INFO` | Logging level |

 ## How It Works
@@ -105,23 +181,29 @@ POP3_ACCOUNT_2_PASSWORD=password2
 └────────┬────────┘
         │ (Fetch emails)
         ▼
-┌─────────────────┐      ┌──────────────┐      ┌─────────────┐
-│  POP3 Server 2  │─────▶│  Forwarder   │─────▶│    Gmail    │
-└─────────────────┘      │  Container   │      │   (SMTP)    │
-         │               └──────┬───────┘      └─────────────┘
-┌────────▼────────┐             │ (Error notifications)
-│  POP3 Server N  │             ▼
-└─────────────────┘   ┌─────────────────┐
-                      │   Postmarkapp   │
+┌─────────────────┐      ┌──────────────────┐      ┌──────────────────┐
+│  POP3 Server 2  │─────▶│    Forwarder     │─────▶│  Gmail API       │
+└─────────────────┘      │    Container     │      │  (Preferred)     │
+         │               │                  │      └──────────────────┘
+┌────────▼────────┐      │  Config from:    │      ┌──────────────────┐
+│  POP3 Server N  │      │  • Database      │─────▶│  Gmail SMTP      │
+└─────────────────┘      │  • Environment   │      │  (Fallback)      │
+                         └──────┬───────────┘      └──────────────────┘
+                                │ (Notifications)
+                                ▼
+                      ┌─────────────────┐
+                      │  Apprise        │
+                      │  (Email, Slack, │
+                      │   Telegram ...) │
                      └─────────────────┘
 ```

-1. **Polling** — checks POP3 mailboxes at the configured interval
+1. **Polling** — checks POP3/IMAP mailboxes at the configured interval
 2. **Fetching** — retrieves new emails from each account
-3. **Forwarding** — delivers to Gmail with original metadata preserved
-4. **Cleanup** — deletes from POP3 after successful forwarding
+3. **Delivery** — injects into Gmail via API (preferred) or forwards via SMTP (fallback)
+4. **Cleanup** — deletes from source after successful delivery
 5. **Throttling** — respects rate limits to avoid quota issues
-6. **Error Handling** — sends notifications if something goes wrong
+6. **Notifications** — sends alerts via Apprise (email, Telegram, Slack, Discord, webhooks)

 ## Development