gitea-admin/gh-christianlouis-inboxconverge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update documentation for hybrid config and Gmail API vs SMTP delivery

Browse Source 
- README.md: document hybrid config model, Gmail API vs SMTP comparison,
  update architecture diagram, replace Postmarkapp with Apprise
- ARCHITECTURE.md: add Gmail API vs SMTP comparison table, document
  ConfigService and AppSetting, add settings/gmail API endpoints
- CHANGELOG.md: add entries for database-backed config and documentation
- TODO.md: update progress (59% coverage, production readiness 30%)
- .env.example: document bootstrap vs database-managed settings

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/ce88d4a8-d8c2-49b3-95a1-30592105a769
This commit is contained in:
copilot-swe-agent[bot]
2026-03-23 13:39:52 +00:00
parent 702650376e
commit 71a97379a1
5 changed files with 209 additions and 41 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+14 -8
View File
@@ -13,20 +13,26 @@ POP3_ACCOUNT_1_USE_SSL=true
# POP3_ACCOUNT_2_PASSWORD=another_password
# POP3_ACCOUNT_2_USE_SSL=true
# Gmail/SMTP Configuration
# ── Bootstrap Settings (always from env, never from database) ──────
# DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/pop3_forwarder
# SECRET_KEY=<generate with: python -c 'import secrets; print(secrets.token_urlsafe(32))'>
# ENCRYPTION_KEY=<generate with: python -c 'import secrets; print(secrets.token_urlsafe(32))'>
# ── Settings below can also be managed via the database ────────────
# Use the admin API (PUT /api/v1/settings/{key}) to store them in
# PostgreSQL. Database values take precedence over env vars.
# Gmail/SMTP Configuration (fallback delivery method)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_USE_TLS=true
# Gmail destination
GMAIL_DESTINATION=your-email@gmail.com
# Postmarkapp for Error Notifications
POSTMARK_API_TOKEN=your-postmark-api-token
POSTMARK_FROM_EMAIL=errors@yourdomain.com
POSTMARK_TO_EMAIL=admin@yourdomain.com
# Gmail API Configuration (preferred delivery method)
# GOOGLE_CLIENT_ID=your-google-client-id
# GOOGLE_CLIENT_SECRET=your-google-client-secret
# GMAIL_API_ENABLED=true
# Scheduling
CHECK_INTERVAL_MINUTES=5
CHANGELOG.md
+9
View File
@@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Database-backed configuration**: `AppSetting` model and `ConfigService` for hybrid config (DB-first, env-var fallback)
- Admin API endpoints for managing settings (`GET/PUT/DELETE /api/v1/settings`)
- Default settings seeded into database on first startup (SMTP, processing, Gmail API, notifications)
- Unit tests for `ConfigService` (24 tests covering resolution order, CRUD, SMTP helper, defaults)
- Gmail API delivery documentation with comparison table (Gmail API vs SMTP forwarding)
- GitHub issue templates (bug report, feature request, test needed)
- Pull request template with comprehensive checklist
- `docs/CODING_PATTERNS.md` with development best practices
@@ -34,6 +39,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Reached 57% test coverage (up from 54%)
### Changed
- Configuration system now supports database-backed settings in addition to environment variables
- Celery tasks (`tasks.py`) use `ConfigService` for SMTP config instead of raw `os.getenv()` calls
- README updated with hybrid configuration docs, Gmail API vs SMTP comparison, and Apprise notifications
- Architecture docs updated to reflect Gmail API service, hybrid config, and new API endpoints
- Reorganized documentation into `docs/` directory
- Improved error handling with specific exception types
- Updated datetime usage to timezone-aware
README.md
+106 -24
View File
@@ -12,13 +12,14 @@ A Docker-based solution that automatically fetches emails from POP3 mailboxes an
## Features
- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables
- **Automatic Forwarding** — sends emails to your Gmail account via SMTP
- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes
- **Dual Delivery** — inject emails via **Gmail API** (preferred) or forward via **SMTP**
- **Hybrid Configuration** — configure via environment variables, `.env` files, **or** the database
- **Smart Throttling** — configurable rate limiting to stay within Gmail quotas
- **Error Reporting** — notifications via Postmarkapp when issues occur
- **Error Reporting** — multi-channel notifications (Apprise: email, Telegram, Slack, Discord, webhooks)
- **Scheduled Polling** — configurable check intervals (default: every 5 minutes)
- **Docker Ready** — fully containerized with Docker Compose support
- **Secure** — runs as non-root user, SSL/TLS connections
- **Secure** — runs as non-root user, SSL/TLS connections, encrypted credential storage
### SaaS Platform (in development)
@@ -53,6 +54,20 @@ See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions.
## Configuration
### Hybrid Configuration (Environment + Database)
The application supports a **hybrid configuration model**:
| Source | Priority | Use For |
|--------|----------|---------|
| **Database** (`app_settings` table) | Highest | SMTP, processing, Gmail API, notifications |
| **Environment variables / `.env`** | Fallback | All settings; required for bootstrap settings |
| **Built-in defaults** | Lowest | Sensible defaults for all non-bootstrap settings |
**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) always come from environment variables because the database connection depends on them.
All other settings (SMTP, processing intervals, Gmail API, etc.) can be managed via the admin API at `/api/v1/settings` and are stored in the PostgreSQL database. When a database setting exists, it takes priority over the corresponding environment variable.
### POP3 Accounts
Add multiple POP3 accounts by incrementing the account number in your `.env`:
@@ -67,15 +82,62 @@ POP3_ACCOUNT_2_USER=user2@provider2.com
POP3_ACCOUNT_2_PASSWORD=password2
```
### Gmail App Password
### Email Delivery Methods
The forwarder supports two delivery methods for getting emails into Gmail:
#### Gmail API Injection (Preferred)
Emails are injected directly into your Gmail account using Google's `users.messages.insert()` API. This is the **recommended method** because it:
- Preserves original email headers and metadata exactly as-is
- Does not modify `From`, `Reply-To`, or `Message-ID` headers
- Applies Gmail labels (e.g., `INBOX`) on injection
- Does not count against Gmail's SMTP sending quotas
- Does not require an SMTP App Password
**Setup:**
1. Configure Google OAuth2 credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`)
2. Authenticate via the SaaS web UI or API (`POST /api/v1/providers/gmail-credential`)
3. Set `delivery_method` to `gmail_api` when creating mail accounts
**Required OAuth2 Scopes:**
- `https://www.googleapis.com/auth/gmail.insert`
- `https://www.googleapis.com/auth/gmail.labels`
#### SMTP Forwarding (Fallback)
Emails are forwarded to Gmail via SMTP. This is the legacy method and is used as a fallback when Gmail API credentials are not available.
**Limitations vs Gmail API:**
- Modifies email headers (adds `Received`, may rewrite `From`)
- Counts against Gmail's SMTP sending quota (500/day for free accounts)
- Requires a Gmail App Password (see below)
- May trigger spam filters for forwarded mail
**Setup:**
1. Go to your [Google Account Security](https://myaccount.google.com/security)
2. Under "Signing in to Google," select **App Passwords**
3. Generate a new app password for "Mail"
4. Use this password as `SMTP_PASSWORD`
4. Set `SMTP_PASSWORD` in your environment or database settings
### Environment Variables
> **Note:** All settings marked ★ can also be managed via the database
> through the admin API (`/api/v1/settings`). Database values take precedence.
#### Bootstrap Settings (env only)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DATABASE_URL` | Yes | `postgresql+asyncpg://...` | PostgreSQL connection string |
| `SECRET_KEY` | Yes | — | JWT signing key (min 32 chars) |
| `ENCRYPTION_KEY` | Yes | — | Credential encryption key (min 32 chars) |
#### POP3/IMAP Accounts (env only — or via API)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname |
@@ -83,18 +145,32 @@ POP3_ACCOUNT_2_PASSWORD=password2
| `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username |
| `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password |
| `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS |
#### SMTP Settings ★
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server |
| `SMTP_PORT` | No | `587` | SMTP port |
| `SMTP_USER` | Yes | — | SMTP username |
| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) |
| `SMTP_USER` | For SMTP | — | SMTP username |
| `SMTP_PASSWORD` | For SMTP | — | SMTP password (App Password) |
| `SMTP_USE_TLS` | No | `true` | Use STARTTLS |
| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address |
#### Gmail API Settings ★
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `GOOGLE_CLIENT_ID` | For Gmail API | — | Google OAuth2 client ID |
| `GOOGLE_CLIENT_SECRET` | For Gmail API | — | Google OAuth2 client secret |
| `GMAIL_API_ENABLED` | No | `true` | Enable Gmail API delivery |
#### Processing Settings ★
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval |
| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run |
| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit |
| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token |
| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender |
| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient |
| `LOG_LEVEL` | No | `INFO` | Logging level |
## How It Works
@@ -105,23 +181,29 @@ POP3_ACCOUNT_2_PASSWORD=password2
└────────┬────────┘
│ (Fetch emails)
▼
┌─────────────────┐ ┌──────────────┐ ┌─────────────┐
│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail │
└─────────────────┘ │ Container │ │ (SMTP) │
│ └──────┬───────┘ └─────────────┘
┌────────▼────────┐ │ (Error notifications)
│ POP3 Server N │ ▼
└─────────────────┘ ┌─────────────────┐
│ Postmarkapp │
┌─────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail API │
└─────────────────┘ │ Container │ │ (Preferred) │
│ │ │ └──────────────────┘
┌────────▼────────┐ │ Config from: │ ┌──────────────────┐
│ POP3 Server N │ │ • Database │─────▶│ Gmail SMTP │
└─────────────────┘ │ • Environment │ │ (Fallback) │
└──────┬───────────┘ └──────────────────┘
│ (Notifications)
▼
┌─────────────────┐
│ Apprise │
│ (Email, Slack, │
│ Telegram ...) │
└─────────────────┘
```
1. **Polling** — checks POP3 mailboxes at the configured interval
1. **Polling** — checks POP3/IMAP mailboxes at the configured interval
2. **Fetching** — retrieves new emails from each account
3. **Forwarding** — delivers to Gmail with original metadata preserved
4. **Cleanup** — deletes from POP3 after successful forwarding
3. **Delivery** — injects into Gmail via API (preferred) or forwards via SMTP (fallback)
4. **Cleanup** — deletes from source after successful delivery
5. **Throttling** — respects rate limits to avoid quota issues
6. **Error Handling** — sends notifications if something goes wrong
6. **Notifications** — sends alerts via Apprise (email, Telegram, Slack, Discord, webhooks)
## Development
docs/ARCHITECTURE.md
+73 -4
View File
@@ -28,16 +28,18 @@ pop_puller_to_gmail/
│ │ │ ├── endpoints/ # Individual route modules
│ │ │ └── api.py # Router aggregation
│ │ ├── core/ # Core configuration
│ │ │ ├── config.py # Settings management
│ │ │ ├── config.py # Bootstrap settings (env / .env)
│ │ │ ├── database.py # Database connection
│ │ │ ├── security.py # Security utilities
│ │ │ └── deps.py # FastAPI dependencies
│ │ ├── models/ # Data models
│ │ │ ├── database_models.py # SQLAlchemy models
│ │ │ ├── database_models.py # SQLAlchemy models (incl. AppSetting)
│ │ │ └── schemas.py # Pydantic schemas
│ │ ├── services/ # Business logic
│ │ │ ├── auth_service.py # OAuth authentication
│ │ │ └── mail_processor.py # Email processing
│ │ │ ├── config_service.py # Hybrid config (DB + env)
│ │ │ ├── gmail_service.py # Gmail API injection
│ │ │ └── mail_processor.py # POP3/IMAP email processing
│ │ ├── workers/ # Celery background tasks
│ │ ├── utils/ # Utility functions
│ │ └── main.py # FastAPI application
@@ -112,10 +114,31 @@ pop_puller_to_gmail/
- **Background Jobs**: Celery workers for async processing
- **Scheduled Checks**: Configurable intervals per account
- **Smart Forwarding**: Preserves metadata, handles MIME types
- **Dual Delivery**: Gmail API injection (preferred) or SMTP forwarding (fallback)
- **Error Handling**: Automatic retries with exponential backoff
- **Statistics**: Track success/failure rates, last check times
### 3a. Gmail API vs SMTP Delivery
The platform supports two methods for delivering fetched emails to Gmail:
| Feature | Gmail API (`gmail_api`) | SMTP Forwarding (`smtp`) |
|---------|------------------------|--------------------------|
| **Header preservation** | ✅ All original headers intact | ⚠️ Adds `Received` headers, may rewrite `From` |
| **Gmail sending quota** | ✅ Does not count against quota | ❌ Counts against 500/day free limit |
| **Authentication** | OAuth2 tokens (per-user) | App Password (shared) |
| **Setup complexity** | Requires OAuth2 consent flow | Requires App Password only |
| **Spam risk** | ✅ Low (email appears native) | ⚠️ Higher (forwarded mail may be flagged) |
| **Fallback** | Falls back to SMTP if no credentials | Primary legacy method |
**How it works:**
1. Each mail account has a `delivery_method` field (`gmail_api` or `smtp`)
2. When `gmail_api` is selected, the worker looks up the user's `GmailCredential`
3. The `GmailService` calls `users.messages.insert()` to inject the raw RFC 2822 email
4. If no valid Gmail credential is found, the worker falls back to SMTP automatically
5. SMTP settings are loaded from the database (via `ConfigService`) with env-var fallback
### 4. Subscription Management
- **Stripe Integration**: Secure payment processing
@@ -151,6 +174,8 @@ pop_puller_to_gmail/
- **subscription_plans**: Available subscription tiers
- **mail_server_presets**: Known provider configurations
- **audit_logs**: Security and compliance audit trail
- **gmail_credentials**: Per-user OAuth2 tokens for Gmail API injection
- **app_settings**: Database-backed application configuration (key-value store)
## 🔌 API Endpoints
@@ -184,10 +209,54 @@ pop_puller_to_gmail/
### Admin
- `GET /api/v1/admin/stats` - System statistics (admin only)
### Settings (Admin)
- `GET /api/v1/settings` - List all database-backed settings
- `PUT /api/v1/settings/{key}` - Create or update a setting
- `DELETE /api/v1/settings/{key}` - Delete a setting
- `POST /api/v1/settings/seed-defaults` - Seed default settings
### Providers & Gmail
- `GET /api/v1/providers/presets` - List mail provider presets
- `GET /api/v1/providers/presets/{id}` - Get a specific preset
- `POST /api/v1/providers/gmail-credential` - Save Gmail API credentials
- `GET /api/v1/providers/gmail-credential` - Get Gmail credential status
- `DELETE /api/v1/providers/gmail-credential` - Remove Gmail credentials
See full API documentation at `/api/docs` when running.
## 🔧 Configuration
### Hybrid Configuration Model
The application uses a **hybrid configuration model** where settings can come
from either the database or environment variables:
```
┌─────────────────────────────────────────────────────┐
│ Setting Lookup Priority │
│ │
│ 1. Database (app_settings table) ← highest │
│ 2. Environment variable / .env file │
│ 3. Built-in default ← lowest │
└─────────────────────────────────────────────────────┘
```
**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`)
always come from environment variables because the database connection
depends on them.
All other settings (SMTP config, processing intervals, Gmail API options,
etc.) can be managed via the **Admin Settings API** (`/api/v1/settings`)
and are stored in PostgreSQL. On first startup the application seeds
sensible defaults into the `app_settings` table.
**Key components:**
- `app.core.config.Settings` — Pydantic Settings for bootstrap config
- `app.models.database_models.AppSetting` — SQLAlchemy model for DB-backed settings
- `app.services.config_service.ConfigService` — Hybrid resolver (DB → env → default)
- `app.api.v1.endpoints.app_settings` — Admin CRUD endpoints
### Environment Variables
Key configuration options in `backend/.env`:
docs/TODO.md
+7 -5
View File
@@ -69,7 +69,7 @@ Comprehensive task breakdown for repository improvements and production readines
- [x] Write unit tests for middleware (98% coverage)
- [x] Write unit tests for schemas and validation
- [x] Write unit tests for application factory and core endpoints
- [x] Reach 50%+ test coverage (currently 57%)
- [x] Reach 50%+ test coverage (currently 59%)
### In Progress 🔨
- [ ] Write unit tests for authentication (target 80%+ coverage)
@@ -135,10 +135,12 @@ Comprehensive task breakdown for repository improvements and production readines
### Completed ✅
- [x] Basic health check endpoint exists
- [x] Database-backed configuration (`AppSetting` model + `ConfigService`)
- [x] Admin API for managing settings (`/api/v1/settings`)
- [x] Default settings seeded on first startup
### In Progress 🔨
- [ ] Improve health checks (DB/Redis connectivity)
- [ ] Add environment variable validation
### Not Started 📋
- [ ] Create production docker-compose.yml
@@ -292,12 +294,12 @@ because the API client layer is missing.
|----------|----------|--------|
| Security | 60% | 🟡 In Progress |
| Agentic Infrastructure | 95% | 🟢 Near Complete |
| Testing | 57% | 🟡 In Progress |
| Testing | 59% | 🟡 In Progress |
| CI/CD | 80% | 🟢 Near Complete |
| Code Quality | 40% | 🔴 Needs Work |
| Production Ready | 20% | 🔴 Needs Work |
| Production Ready | 30% | 🔴 Needs Work |
| Observability | 10% | 🔴 Needs Work |
| Backend Features | 80% | 🟢 Near Complete |
| Backend Features | 85% | 🟢 Near Complete |
| Frontend | 50% | 🟡 In Progress |
**Overall Repository Readiness**: 55% ⚠️