From 71a97379a185e570fbbf4483f8e098f7098995c1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 13:39:52 +0000 Subject: [PATCH] Update documentation for hybrid config and Gmail API vs SMTP delivery - README.md: document hybrid config model, Gmail API vs SMTP comparison, update architecture diagram, replace Postmarkapp with Apprise - ARCHITECTURE.md: add Gmail API vs SMTP comparison table, document ConfigService and AppSetting, add settings/gmail API endpoints - CHANGELOG.md: add entries for database-backed config and documentation - TODO.md: update progress (59% coverage, production readiness 30%) - .env.example: document bootstrap vs database-managed settings Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/ce88d4a8-d8c2-49b3-95a1-30592105a769 --- .env.example | 22 +++++--- CHANGELOG.md | 9 +++ README.md | 130 +++++++++++++++++++++++++++++++++++-------- docs/ARCHITECTURE.md | 77 +++++++++++++++++++++++-- docs/TODO.md | 12 ++-- 5 files changed, 209 insertions(+), 41 deletions(-) diff --git a/.env.example b/.env.example index caf26a5..af39a01 100644 --- a/.env.example +++ b/.env.example @@ -13,20 +13,26 @@ POP3_ACCOUNT_1_USE_SSL=true # POP3_ACCOUNT_2_PASSWORD=another_password # POP3_ACCOUNT_2_USE_SSL=true -# Gmail/SMTP Configuration +# ── Bootstrap Settings (always from env, never from database) ────── +# DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/pop3_forwarder +# SECRET_KEY= +# ENCRYPTION_KEY= + +# ── Settings below can also be managed via the database ──────────── +# Use the admin API (PUT /api/v1/settings/{key}) to store them in +# PostgreSQL. Database values take precedence over env vars. + +# Gmail/SMTP Configuration (fallback delivery method) SMTP_HOST=smtp.gmail.com SMTP_PORT=587 SMTP_USER=your-email@gmail.com SMTP_PASSWORD=your-app-password SMTP_USE_TLS=true -# Gmail destination -GMAIL_DESTINATION=your-email@gmail.com - -# Postmarkapp for Error Notifications -POSTMARK_API_TOKEN=your-postmark-api-token -POSTMARK_FROM_EMAIL=errors@yourdomain.com -POSTMARK_TO_EMAIL=admin@yourdomain.com +# Gmail API Configuration (preferred delivery method) +# GOOGLE_CLIENT_ID=your-google-client-id +# GOOGLE_CLIENT_SECRET=your-google-client-secret +# GMAIL_API_ENABLED=true # Scheduling CHECK_INTERVAL_MINUTES=5 diff --git a/CHANGELOG.md b/CHANGELOG.md index b2a3b14..485d2c0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added +- **Database-backed configuration**: `AppSetting` model and `ConfigService` for hybrid config (DB-first, env-var fallback) +- Admin API endpoints for managing settings (`GET/PUT/DELETE /api/v1/settings`) +- Default settings seeded into database on first startup (SMTP, processing, Gmail API, notifications) +- Unit tests for `ConfigService` (24 tests covering resolution order, CRUD, SMTP helper, defaults) +- Gmail API delivery documentation with comparison table (Gmail API vs SMTP forwarding) - GitHub issue templates (bug report, feature request, test needed) - Pull request template with comprehensive checklist - `docs/CODING_PATTERNS.md` with development best practices @@ -34,6 +39,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Reached 57% test coverage (up from 54%) ### Changed +- Configuration system now supports database-backed settings in addition to environment variables +- Celery tasks (`tasks.py`) use `ConfigService` for SMTP config instead of raw `os.getenv()` calls +- README updated with hybrid configuration docs, Gmail API vs SMTP comparison, and Apprise notifications +- Architecture docs updated to reflect Gmail API service, hybrid config, and new API endpoints - Reorganized documentation into `docs/` directory - Improved error handling with specific exception types - Updated datetime usage to timezone-aware diff --git a/README.md b/README.md index 55f21c8..b7e0385 100644 --- a/README.md +++ b/README.md @@ -12,13 +12,14 @@ A Docker-based solution that automatically fetches emails from POP3 mailboxes an ## Features -- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables -- **Automatic Forwarding** — sends emails to your Gmail account via SMTP +- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes +- **Dual Delivery** — inject emails via **Gmail API** (preferred) or forward via **SMTP** +- **Hybrid Configuration** — configure via environment variables, `.env` files, **or** the database - **Smart Throttling** — configurable rate limiting to stay within Gmail quotas -- **Error Reporting** — notifications via Postmarkapp when issues occur +- **Error Reporting** — multi-channel notifications (Apprise: email, Telegram, Slack, Discord, webhooks) - **Scheduled Polling** — configurable check intervals (default: every 5 minutes) - **Docker Ready** — fully containerized with Docker Compose support -- **Secure** — runs as non-root user, SSL/TLS connections +- **Secure** — runs as non-root user, SSL/TLS connections, encrypted credential storage ### SaaS Platform (in development) @@ -53,6 +54,20 @@ See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions. ## Configuration +### Hybrid Configuration (Environment + Database) + +The application supports a **hybrid configuration model**: + +| Source | Priority | Use For | +|--------|----------|---------| +| **Database** (`app_settings` table) | Highest | SMTP, processing, Gmail API, notifications | +| **Environment variables / `.env`** | Fallback | All settings; required for bootstrap settings | +| **Built-in defaults** | Lowest | Sensible defaults for all non-bootstrap settings | + +**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) always come from environment variables because the database connection depends on them. + +All other settings (SMTP, processing intervals, Gmail API, etc.) can be managed via the admin API at `/api/v1/settings` and are stored in the PostgreSQL database. When a database setting exists, it takes priority over the corresponding environment variable. + ### POP3 Accounts Add multiple POP3 accounts by incrementing the account number in your `.env`: @@ -67,15 +82,62 @@ POP3_ACCOUNT_2_USER=user2@provider2.com POP3_ACCOUNT_2_PASSWORD=password2 ``` -### Gmail App Password +### Email Delivery Methods + +The forwarder supports two delivery methods for getting emails into Gmail: + +#### Gmail API Injection (Preferred) + +Emails are injected directly into your Gmail account using Google's `users.messages.insert()` API. This is the **recommended method** because it: + +- Preserves original email headers and metadata exactly as-is +- Does not modify `From`, `Reply-To`, or `Message-ID` headers +- Applies Gmail labels (e.g., `INBOX`) on injection +- Does not count against Gmail's SMTP sending quotas +- Does not require an SMTP App Password + +**Setup:** + +1. Configure Google OAuth2 credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`) +2. Authenticate via the SaaS web UI or API (`POST /api/v1/providers/gmail-credential`) +3. Set `delivery_method` to `gmail_api` when creating mail accounts + +**Required OAuth2 Scopes:** +- `https://www.googleapis.com/auth/gmail.insert` +- `https://www.googleapis.com/auth/gmail.labels` + +#### SMTP Forwarding (Fallback) + +Emails are forwarded to Gmail via SMTP. This is the legacy method and is used as a fallback when Gmail API credentials are not available. + +**Limitations vs Gmail API:** +- Modifies email headers (adds `Received`, may rewrite `From`) +- Counts against Gmail's SMTP sending quota (500/day for free accounts) +- Requires a Gmail App Password (see below) +- May trigger spam filters for forwarded mail + +**Setup:** 1. Go to your [Google Account Security](https://myaccount.google.com/security) 2. Under "Signing in to Google," select **App Passwords** 3. Generate a new app password for "Mail" -4. Use this password as `SMTP_PASSWORD` +4. Set `SMTP_PASSWORD` in your environment or database settings ### Environment Variables +> **Note:** All settings marked ★ can also be managed via the database +> through the admin API (`/api/v1/settings`). Database values take precedence. + +#### Bootstrap Settings (env only) + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `DATABASE_URL` | Yes | `postgresql+asyncpg://...` | PostgreSQL connection string | +| `SECRET_KEY` | Yes | — | JWT signing key (min 32 chars) | +| `ENCRYPTION_KEY` | Yes | — | Credential encryption key (min 32 chars) | + +#### POP3/IMAP Accounts (env only — or via API) + | Variable | Required | Default | Description | |----------|----------|---------|-------------| | `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname | @@ -83,18 +145,32 @@ POP3_ACCOUNT_2_PASSWORD=password2 | `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username | | `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password | | `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS | + +#### SMTP Settings ★ + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| | `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server | | `SMTP_PORT` | No | `587` | SMTP port | -| `SMTP_USER` | Yes | — | SMTP username | -| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) | +| `SMTP_USER` | For SMTP | — | SMTP username | +| `SMTP_PASSWORD` | For SMTP | — | SMTP password (App Password) | | `SMTP_USE_TLS` | No | `true` | Use STARTTLS | -| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address | + +#### Gmail API Settings ★ + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| +| `GOOGLE_CLIENT_ID` | For Gmail API | — | Google OAuth2 client ID | +| `GOOGLE_CLIENT_SECRET` | For Gmail API | — | Google OAuth2 client secret | +| `GMAIL_API_ENABLED` | No | `true` | Enable Gmail API delivery | + +#### Processing Settings ★ + +| Variable | Required | Default | Description | +|----------|----------|---------|-------------| | `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval | | `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run | | `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit | -| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token | -| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender | -| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient | | `LOG_LEVEL` | No | `INFO` | Logging level | ## How It Works @@ -105,23 +181,29 @@ POP3_ACCOUNT_2_PASSWORD=password2 └────────┬────────┘ │ (Fetch emails) ▼ -┌─────────────────┐ ┌──────────────┐ ┌─────────────┐ -│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail │ -└─────────────────┘ │ Container │ │ (SMTP) │ - │ └──────┬───────┘ └─────────────┘ -┌────────▼────────┐ │ (Error notifications) -│ POP3 Server N │ ▼ -└─────────────────┘ ┌─────────────────┐ - │ Postmarkapp │ +┌─────────────────┐ ┌──────────────────┐ ┌──────────────────┐ +│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail API │ +└─────────────────┘ │ Container │ │ (Preferred) │ + │ │ │ └──────────────────┘ +┌────────▼────────┐ │ Config from: │ ┌──────────────────┐ +│ POP3 Server N │ │ • Database │─────▶│ Gmail SMTP │ +└─────────────────┘ │ • Environment │ │ (Fallback) │ + └──────┬───────────┘ └──────────────────┘ + │ (Notifications) + ▼ + ┌─────────────────┐ + │ Apprise │ + │ (Email, Slack, │ + │ Telegram ...) │ └─────────────────┘ ``` -1. **Polling** — checks POP3 mailboxes at the configured interval +1. **Polling** — checks POP3/IMAP mailboxes at the configured interval 2. **Fetching** — retrieves new emails from each account -3. **Forwarding** — delivers to Gmail with original metadata preserved -4. **Cleanup** — deletes from POP3 after successful forwarding +3. **Delivery** — injects into Gmail via API (preferred) or forwards via SMTP (fallback) +4. **Cleanup** — deletes from source after successful delivery 5. **Throttling** — respects rate limits to avoid quota issues -6. **Error Handling** — sends notifications if something goes wrong +6. **Notifications** — sends alerts via Apprise (email, Telegram, Slack, Discord, webhooks) ## Development diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index fe92608..4bf0f10 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -28,16 +28,18 @@ pop_puller_to_gmail/ │ │ │ ├── endpoints/ # Individual route modules │ │ │ └── api.py # Router aggregation │ │ ├── core/ # Core configuration -│ │ │ ├── config.py # Settings management +│ │ │ ├── config.py # Bootstrap settings (env / .env) │ │ │ ├── database.py # Database connection │ │ │ ├── security.py # Security utilities │ │ │ └── deps.py # FastAPI dependencies │ │ ├── models/ # Data models -│ │ │ ├── database_models.py # SQLAlchemy models +│ │ │ ├── database_models.py # SQLAlchemy models (incl. AppSetting) │ │ │ └── schemas.py # Pydantic schemas │ │ ├── services/ # Business logic │ │ │ ├── auth_service.py # OAuth authentication -│ │ │ └── mail_processor.py # Email processing +│ │ │ ├── config_service.py # Hybrid config (DB + env) +│ │ │ ├── gmail_service.py # Gmail API injection +│ │ │ └── mail_processor.py # POP3/IMAP email processing │ │ ├── workers/ # Celery background tasks │ │ ├── utils/ # Utility functions │ │ └── main.py # FastAPI application @@ -112,10 +114,31 @@ pop_puller_to_gmail/ - **Background Jobs**: Celery workers for async processing - **Scheduled Checks**: Configurable intervals per account -- **Smart Forwarding**: Preserves metadata, handles MIME types +- **Dual Delivery**: Gmail API injection (preferred) or SMTP forwarding (fallback) - **Error Handling**: Automatic retries with exponential backoff - **Statistics**: Track success/failure rates, last check times +### 3a. Gmail API vs SMTP Delivery + +The platform supports two methods for delivering fetched emails to Gmail: + +| Feature | Gmail API (`gmail_api`) | SMTP Forwarding (`smtp`) | +|---------|------------------------|--------------------------| +| **Header preservation** | ✅ All original headers intact | ⚠️ Adds `Received` headers, may rewrite `From` | +| **Gmail sending quota** | ✅ Does not count against quota | ❌ Counts against 500/day free limit | +| **Authentication** | OAuth2 tokens (per-user) | App Password (shared) | +| **Setup complexity** | Requires OAuth2 consent flow | Requires App Password only | +| **Spam risk** | ✅ Low (email appears native) | ⚠️ Higher (forwarded mail may be flagged) | +| **Fallback** | Falls back to SMTP if no credentials | Primary legacy method | + +**How it works:** + +1. Each mail account has a `delivery_method` field (`gmail_api` or `smtp`) +2. When `gmail_api` is selected, the worker looks up the user's `GmailCredential` +3. The `GmailService` calls `users.messages.insert()` to inject the raw RFC 2822 email +4. If no valid Gmail credential is found, the worker falls back to SMTP automatically +5. SMTP settings are loaded from the database (via `ConfigService`) with env-var fallback + ### 4. Subscription Management - **Stripe Integration**: Secure payment processing @@ -151,6 +174,8 @@ pop_puller_to_gmail/ - **subscription_plans**: Available subscription tiers - **mail_server_presets**: Known provider configurations - **audit_logs**: Security and compliance audit trail +- **gmail_credentials**: Per-user OAuth2 tokens for Gmail API injection +- **app_settings**: Database-backed application configuration (key-value store) ## 🔌 API Endpoints @@ -184,10 +209,54 @@ pop_puller_to_gmail/ ### Admin - `GET /api/v1/admin/stats` - System statistics (admin only) +### Settings (Admin) +- `GET /api/v1/settings` - List all database-backed settings +- `PUT /api/v1/settings/{key}` - Create or update a setting +- `DELETE /api/v1/settings/{key}` - Delete a setting +- `POST /api/v1/settings/seed-defaults` - Seed default settings + +### Providers & Gmail +- `GET /api/v1/providers/presets` - List mail provider presets +- `GET /api/v1/providers/presets/{id}` - Get a specific preset +- `POST /api/v1/providers/gmail-credential` - Save Gmail API credentials +- `GET /api/v1/providers/gmail-credential` - Get Gmail credential status +- `DELETE /api/v1/providers/gmail-credential` - Remove Gmail credentials + See full API documentation at `/api/docs` when running. ## 🔧 Configuration +### Hybrid Configuration Model + +The application uses a **hybrid configuration model** where settings can come +from either the database or environment variables: + +``` +┌─────────────────────────────────────────────────────┐ +│ Setting Lookup Priority │ +│ │ +│ 1. Database (app_settings table) ← highest │ +│ 2. Environment variable / .env file │ +│ 3. Built-in default ← lowest │ +└─────────────────────────────────────────────────────┘ +``` + +**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) +always come from environment variables because the database connection +depends on them. + +All other settings (SMTP config, processing intervals, Gmail API options, +etc.) can be managed via the **Admin Settings API** (`/api/v1/settings`) +and are stored in PostgreSQL. On first startup the application seeds +sensible defaults into the `app_settings` table. + +**Key components:** + +- `app.core.config.Settings` — Pydantic Settings for bootstrap config +- `app.models.database_models.AppSetting` — SQLAlchemy model for DB-backed settings +- `app.services.config_service.ConfigService` — Hybrid resolver (DB → env → default) +- `app.api.v1.endpoints.app_settings` — Admin CRUD endpoints + ### Environment Variables Key configuration options in `backend/.env`: diff --git a/docs/TODO.md b/docs/TODO.md index b5c8bba..5059d6a 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -69,7 +69,7 @@ Comprehensive task breakdown for repository improvements and production readines - [x] Write unit tests for middleware (98% coverage) - [x] Write unit tests for schemas and validation - [x] Write unit tests for application factory and core endpoints -- [x] Reach 50%+ test coverage (currently 57%) +- [x] Reach 50%+ test coverage (currently 59%) ### In Progress 🔨 - [ ] Write unit tests for authentication (target 80%+ coverage) @@ -135,10 +135,12 @@ Comprehensive task breakdown for repository improvements and production readines ### Completed ✅ - [x] Basic health check endpoint exists +- [x] Database-backed configuration (`AppSetting` model + `ConfigService`) +- [x] Admin API for managing settings (`/api/v1/settings`) +- [x] Default settings seeded on first startup ### In Progress 🔨 - [ ] Improve health checks (DB/Redis connectivity) -- [ ] Add environment variable validation ### Not Started 📋 - [ ] Create production docker-compose.yml @@ -292,12 +294,12 @@ because the API client layer is missing. |----------|----------|--------| | Security | 60% | 🟡 In Progress | | Agentic Infrastructure | 95% | 🟢 Near Complete | -| Testing | 57% | 🟡 In Progress | +| Testing | 59% | 🟡 In Progress | | CI/CD | 80% | 🟢 Near Complete | | Code Quality | 40% | 🔴 Needs Work | -| Production Ready | 20% | 🔴 Needs Work | +| Production Ready | 30% | 🔴 Needs Work | | Observability | 10% | 🔴 Needs Work | -| Backend Features | 80% | 🟢 Near Complete | +| Backend Features | 85% | 🟢 Near Complete | | Frontend | 50% | 🟡 In Progress | **Overall Repository Readiness**: 55% ⚠️