Merge branch 'main' into testing-auth-store-improvements-16756141904230151053

This commit is contained in:
Christian Krakau-Louis
2026-03-23 17:05:43 +01:00
committed by GitHub
14 changed files with 1055 additions and 53 deletions
+14 -8
View File
@@ -13,20 +13,26 @@ POP3_ACCOUNT_1_USE_SSL=true
# POP3_ACCOUNT_2_PASSWORD=another_password
# POP3_ACCOUNT_2_USE_SSL=true
# Gmail/SMTP Configuration
# ── Bootstrap Settings (always from env, never from database) ──────
# DATABASE_URL=postgresql+asyncpg://user:password@localhost:5432/pop3_forwarder
# SECRET_KEY=<generate with: python -c 'import secrets; print(secrets.token_urlsafe(32))'>
# ENCRYPTION_KEY=<generate with: python -c 'import secrets; print(secrets.token_urlsafe(32))'>
# ── Settings below can also be managed via the database ────────────
# Use the admin API (PUT /api/v1/settings/{key}) to store them in
# PostgreSQL. Database values take precedence over env vars.
# Gmail/SMTP Configuration (fallback delivery method)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USER=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_USE_TLS=true
# Gmail destination
GMAIL_DESTINATION=your-email@gmail.com
# Postmarkapp for Error Notifications
POSTMARK_API_TOKEN=your-postmark-api-token
POSTMARK_FROM_EMAIL=errors@yourdomain.com
POSTMARK_TO_EMAIL=admin@yourdomain.com
# Gmail API Configuration (preferred delivery method)
# GOOGLE_CLIENT_ID=your-google-client-id
# GOOGLE_CLIENT_SECRET=your-google-client-secret
# GMAIL_API_ENABLED=true
# Scheduling
CHECK_INTERVAL_MINUTES=5
+10
View File
@@ -8,6 +8,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Added
- **Database-backed configuration**: `AppSetting` model and `ConfigService` for hybrid config (DB-first, env-var fallback)
- Admin API endpoints for managing settings (`GET/PUT/DELETE /api/v1/settings`)
- Default settings seeded into database on first startup (SMTP, processing, Gmail API, notifications)
- Unit tests for `ConfigService` (24 tests covering resolution order, CRUD, SMTP helper, defaults)
- Gmail API delivery documentation with comparison table (Gmail API vs SMTP forwarding)
- GitHub issue templates (bug report, feature request, test needed)
- Pull request template with comprehensive checklist
- `docs/CODING_PATTERNS.md` with development best practices
@@ -34,6 +39,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Reached 57% test coverage (up from 54%)
### Changed
- Configuration system now supports database-backed settings in addition to environment variables
- Celery tasks (`tasks.py`) use `ConfigService` for SMTP config instead of raw `os.getenv()` calls
- README updated with hybrid configuration docs, Gmail API vs SMTP comparison, and Apprise notifications
- Architecture docs updated to reflect Gmail API service, hybrid config, and new API endpoints
- Reorganized documentation into `docs/` directory
- Improved error handling with specific exception types
- Updated datetime usage to timezone-aware
@@ -44,6 +53,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Bumped GitHub Actions: `actions/setup-python` v5 → v6, `actions/setup-node` v4 → v6, `docker/setup-buildx-action` v3 → v4, `codecov/codecov-action` v3 → v5
- Bumped backend dependencies: pydantic 2.5.3 → 2.12.5, pydantic-settings 2.1.0 → 2.13.1, psycopg2-binary 2.9.9 → 2.9.11, asyncpg 0.29.0 → 0.31.0, stripe 7.11.0 → 14.4.1, aioimaplib 1.0.1 → 2.0.1, google-auth-httplib2 0.2.0 → 0.3.0, celery 5.3.6 → 5.6.2, redis 5.0.1 → 7.3.0, tenacity 8.2.3 → 9.1.4
- Bumped frontend dependencies: react 19.2.3 → 19.2.4, @tanstack/react-query ^5.90.20 → ^5.95.0, axios ^1.13.5 → ^1.13.6, zustand ^5.0.11 → ^5.0.12, eslint ^9 → ^10, eslint-config-next 16.1.6 → 16.2.1
- Synced `frontend/package.json` `eslint-config-next` to `16.2.1` to match `package-lock.json` (resolves `npm ci` EUSAGE failure)
### Removed
- Removed CodeQL analysis from CI pipeline (was blocking builds)
+106 -24
View File
@@ -12,13 +12,14 @@ A Docker-based solution that automatically fetches emails from POP3 mailboxes an
## Features
- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables
- **Automatic Forwarding** — sends emails to your Gmail account via SMTP
- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes
- **Dual Delivery** — inject emails via **Gmail API** (preferred) or forward via **SMTP**
- **Hybrid Configuration** — configure via environment variables, `.env` files, **or** the database
- **Smart Throttling** — configurable rate limiting to stay within Gmail quotas
- **Error Reporting** — notifications via Postmarkapp when issues occur
- **Error Reporting** — multi-channel notifications (Apprise: email, Telegram, Slack, Discord, webhooks)
- **Scheduled Polling** — configurable check intervals (default: every 5 minutes)
- **Docker Ready** — fully containerized with Docker Compose support
- **Secure** — runs as non-root user, SSL/TLS connections
- **Secure** — runs as non-root user, SSL/TLS connections, encrypted credential storage
### SaaS Platform (in development)
@@ -53,6 +54,20 @@ See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions.
## Configuration
### Hybrid Configuration (Environment + Database)
The application supports a **hybrid configuration model**:
| Source | Priority | Use For |
|--------|----------|---------|
| **Database** (`app_settings` table) | Highest | SMTP, processing, Gmail API, notifications |
| **Environment variables / `.env`** | Fallback | All settings; required for bootstrap settings |
| **Built-in defaults** | Lowest | Sensible defaults for all non-bootstrap settings |
**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) always come from environment variables because the database connection depends on them.
All other settings (SMTP, processing intervals, Gmail API, etc.) can be managed via the admin API at `/api/v1/settings` and are stored in the PostgreSQL database. When a database setting exists, it takes priority over the corresponding environment variable.
### POP3 Accounts
Add multiple POP3 accounts by incrementing the account number in your `.env`:
@@ -67,15 +82,62 @@ POP3_ACCOUNT_2_USER=user2@provider2.com
POP3_ACCOUNT_2_PASSWORD=password2
```
### Gmail App Password
### Email Delivery Methods
The forwarder supports two delivery methods for getting emails into Gmail:
#### Gmail API Injection (Preferred)
Emails are injected directly into your Gmail account using Google's `users.messages.insert()` API. This is the **recommended method** because it:
- Preserves original email headers and metadata exactly as-is
- Does not modify `From`, `Reply-To`, or `Message-ID` headers
- Applies Gmail labels (e.g., `INBOX`) on injection
- Does not count against Gmail's SMTP sending quotas
- Does not require an SMTP App Password
**Setup:**
1. Configure Google OAuth2 credentials (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`)
2. Authenticate via the SaaS web UI or API (`POST /api/v1/providers/gmail-credential`)
3. Set `delivery_method` to `gmail_api` when creating mail accounts
**Required OAuth2 Scopes:**
- `https://www.googleapis.com/auth/gmail.insert`
- `https://www.googleapis.com/auth/gmail.labels`
#### SMTP Forwarding (Fallback)
Emails are forwarded to Gmail via SMTP. This is the legacy method and is used as a fallback when Gmail API credentials are not available.
**Limitations vs Gmail API:**
- Modifies email headers (adds `Received`, may rewrite `From`)
- Counts against Gmail's SMTP sending quota (500/day for free accounts)
- Requires a Gmail App Password (see below)
- May trigger spam filters for forwarded mail
**Setup:**
1. Go to your [Google Account Security](https://myaccount.google.com/security)
2. Under "Signing in to Google," select **App Passwords**
3. Generate a new app password for "Mail"
4. Use this password as `SMTP_PASSWORD`
4. Set `SMTP_PASSWORD` in your environment or database settings
### Environment Variables
> **Note:** All settings marked ★ can also be managed via the database
> through the admin API (`/api/v1/settings`). Database values take precedence.
#### Bootstrap Settings (env only)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `DATABASE_URL` | Yes | `postgresql+asyncpg://...` | PostgreSQL connection string |
| `SECRET_KEY` | Yes | — | JWT signing key (min 32 chars) |
| `ENCRYPTION_KEY` | Yes | — | Credential encryption key (min 32 chars) |
#### POP3/IMAP Accounts (env only — or via API)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname |
@@ -83,18 +145,32 @@ POP3_ACCOUNT_2_PASSWORD=password2
| `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username |
| `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password |
| `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS |
#### SMTP Settings (★ database-configurable)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server |
| `SMTP_PORT` | No | `587` | SMTP port |
| `SMTP_USER` | Yes | — | SMTP username |
| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) |
| `SMTP_USER` | For SMTP | — | SMTP username |
| `SMTP_PASSWORD` | For SMTP | — | SMTP password (App Password) |
| `SMTP_USE_TLS` | No | `true` | Use STARTTLS |
| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address |
#### Gmail API Settings (★ database-configurable)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `GOOGLE_CLIENT_ID` | For Gmail API | — | Google OAuth2 client ID |
| `GOOGLE_CLIENT_SECRET` | For Gmail API | — | Google OAuth2 client secret |
| `GMAIL_API_ENABLED` | No | `true` | Enable Gmail API delivery |
#### Processing Settings (★ database-configurable)
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval |
| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run |
| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit |
| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token |
| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender |
| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient |
| `LOG_LEVEL` | No | `INFO` | Logging level |
## How It Works
@@ -105,23 +181,29 @@ POP3_ACCOUNT_2_PASSWORD=password2
└────────┬────────┘
│ (Fetch emails)
┌─────────────────┐ ┌──────────────┐ ┌─────────────┐
│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail │
└─────────────────┘ │ Container │ │ (SMTP)
└──────┬───────┘ └─────────────┘
┌────────▼────────┐ │ (Error notifications)
│ POP3 Server N │
└─────────────────┘ ┌─────────────────┐
│ Postmarkapp │
┌─────────────────┐ ┌──────────────────┐ ┌──────────────────
│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail API
└─────────────────┘ │ Container │ │ (Preferred)
│ │ └──────────────────
┌────────▼────────┐ Config from: │ ┌──────────────────┐
│ POP3 Server N │ • Database │─────▶│ Gmail SMTP │
└─────────────────┘ │ • Environment │ │ (Fallback) │
└──────┬───────────┘ └──────────────────┘
│ (Notifications)
┌─────────────────┐
│ Apprise │
│ (Email, Slack, │
│ Telegram ...) │
└─────────────────┘
```
1. **Polling** — checks POP3 mailboxes at the configured interval
1. **Polling** — checks POP3/IMAP mailboxes at the configured interval
2. **Fetching** — retrieves new emails from each account
3. **Forwarding** — delivers to Gmail with original metadata preserved
4. **Cleanup** — deletes from POP3 after successful forwarding
3. **Delivery** — injects into Gmail via API (preferred) or forwards via SMTP (fallback)
4. **Cleanup** — deletes from source after successful delivery
5. **Throttling** — respects rate limits to avoid quota issues
6. **Error Handling** — sends notifications if something goes wrong
6. **Notifications** — sends alerts via Apprise (email, Telegram, Slack, Discord, webhooks)
## Development
+2
View File
@@ -12,6 +12,7 @@ from app.api.v1.endpoints import (
subscriptions,
admin,
providers,
app_settings,
)
api_router = APIRouter()
@@ -32,3 +33,4 @@ api_router.include_router(
subscriptions.router, prefix="/subscriptions", tags=["Subscriptions"]
)
api_router.include_router(admin.router, prefix="/admin", tags=["Admin"])
api_router.include_router(app_settings.router, prefix="/settings", tags=["Settings"])
@@ -0,0 +1,148 @@
"""Admin API endpoints for managing database-backed application settings."""
from typing import List, Optional
from fastapi import APIRouter, Depends, HTTPException, Query, status
from pydantic import BaseModel, ConfigDict
from sqlalchemy.ext.asyncio import AsyncSession
from app.core.database import get_db
from app.core.deps import get_current_superuser
from app.models.database_models import User
from app.services.config_service import BOOTSTRAP_KEYS, ConfigService
router = APIRouter()
# ── Schemas ────────────────────────────────────────────────────────
class AppSettingResponse(BaseModel):
"""Public representation of a stored setting."""
id: int
key: str
value: Optional[str] = None
value_type: str
description: Optional[str] = None
is_secret: bool
category: Optional[str] = None
model_config = ConfigDict(from_attributes=True)
class AppSettingCreate(BaseModel):
"""Payload for creating / updating a setting."""
key: str
value: str
value_type: str = "string"
description: Optional[str] = None
is_secret: bool = False
category: Optional[str] = None
# ── Endpoints ──────────────────────────────────────────────────────
@router.get("", response_model=List[AppSettingResponse])
async def list_settings(
category: Optional[str] = Query(None, description="Filter by category"),
current_user: User = Depends(get_current_superuser),
db: AsyncSession = Depends(get_db),
):
"""List all application settings (admin only).
Secrets are returned with their values masked.
"""
settings = await ConfigService.list_all(db, category=category)
results: List[AppSettingResponse] = []
for s in settings:
val = s.value if not s.is_secret else "********" # type: ignore[arg-type]
results.append(
AppSettingResponse(
id=s.id, # type: ignore[arg-type]
key=s.key, # type: ignore[arg-type]
value=val, # type: ignore[arg-type]
value_type=s.value_type or "string", # type: ignore[arg-type]
description=s.description, # type: ignore[arg-type]
is_secret=s.is_secret, # type: ignore[arg-type]
category=s.category, # type: ignore[arg-type]
)
)
return results
@router.put("/{key}", response_model=AppSettingResponse)
async def upsert_setting(
key: str,
payload: AppSettingCreate,
current_user: User = Depends(get_current_superuser),
db: AsyncSession = Depends(get_db),
):
"""Create or update a setting (admin only).
Bootstrap settings (DATABASE_URL, SECRET_KEY, ENCRYPTION_KEY) cannot
be stored in the database — they must be set via environment variables.
"""
if key in BOOTSTRAP_KEYS:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(
f"'{key}' is a bootstrap setting and cannot be stored in the "
"database. Set it via environment variables."
),
)
setting = await ConfigService.set(
key=key,
value=payload.value,
db=db,
value_type=payload.value_type,
description=payload.description,
is_secret=payload.is_secret,
category=payload.category,
)
return AppSettingResponse(
id=setting.id, # type: ignore[arg-type]
key=setting.key, # type: ignore[arg-type]
value="********" if setting.is_secret else setting.value, # type: ignore[arg-type]
value_type=setting.value_type or "string", # type: ignore[arg-type]
description=setting.description, # type: ignore[arg-type]
is_secret=setting.is_secret, # type: ignore[arg-type]
category=setting.category, # type: ignore[arg-type]
)
@router.delete("/{key}", status_code=status.HTTP_204_NO_CONTENT)
async def delete_setting(
key: str,
current_user: User = Depends(get_current_superuser),
db: AsyncSession = Depends(get_db),
):
"""Delete a setting from the database (admin only)."""
if key in BOOTSTRAP_KEYS:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail=(f"'{key}' is a bootstrap setting and cannot be deleted."),
)
deleted = await ConfigService.delete(key, db)
if not deleted:
raise HTTPException(
status_code=status.HTTP_404_NOT_FOUND,
detail=f"Setting '{key}' not found",
)
@router.post("/seed-defaults", status_code=status.HTTP_200_OK)
async def seed_default_settings(
current_user: User = Depends(get_current_superuser),
db: AsyncSession = Depends(get_db),
):
"""Seed default settings into the database (admin only).
Only creates settings that do not already exist.
"""
count = await ConfigService.seed_defaults(db)
return {"message": f"Seeded {count} default settings", "created": count}
+12 -1
View File
@@ -1,6 +1,17 @@
"""
Application configuration using Pydantic settings.
Supports environment variables and .env files.
Supports a hybrid configuration model:
• **Bootstrap settings** (DATABASE_URL, SECRET_KEY, ENCRYPTION_KEY)
are always loaded from environment variables or ``.env`` files.
• **Application settings** (SMTP, processing, Gmail API, etc.)
can be managed in the database via the ``AppSetting`` model and
the ``/api/v1/settings`` admin endpoints. When a setting exists
in the database it takes precedence over environment variables.
See ``app.services.config_service.ConfigService`` for the runtime
lookup logic and ``app.models.database_models.AppSetting`` for the
database model.
"""
from typing import Optional, List
+11
View File
@@ -28,6 +28,17 @@ async def lifespan(app: FastAPI) -> AsyncIterator[None]:
logger.info(f"Starting {settings.APP_NAME} v{settings.APP_VERSION}")
logger.info(f"Debug mode: {settings.DEBUG}")
logger.info("API documentation: /api/docs")
# Seed default database-backed settings (no-op if they already exist)
try:
from app.core.database import async_session_maker
from app.services.config_service import ConfigService
async with async_session_maker() as db:
await ConfigService.seed_defaults(db)
except Exception as exc:
logger.warning("Could not seed default settings: %s", exc, exc_info=True)
yield
# Shutdown
logger.info("Shutting down application")
+27
View File
@@ -430,3 +430,30 @@ class GmailCredential(Base):
# Relationships
user = relationship("User", backref="gmail_credential")
class AppSetting(Base):
"""
Application settings stored in the database.
Provides database-backed configuration that supplements or overrides
environment variable settings. Bootstrap settings (DATABASE_URL,
SECRET_KEY, ENCRYPTION_KEY) must still come from environment variables,
but all other settings can be managed via the database.
"""
__tablename__ = "app_settings"
id = Column(Integer, primary_key=True, index=True)
key = Column(String(255), unique=True, nullable=False, index=True)
value = Column(Text, nullable=True)
value_type = Column(String(50), default="string") # string, int, float, bool, json
description = Column(Text, nullable=True)
is_secret = Column(Boolean, default=False)
category = Column(String(100), nullable=True, index=True)
# Timestamps
created_at = Column(DateTime, default=datetime.utcnow, nullable=False)
updated_at = Column(
DateTime, default=datetime.utcnow, onupdate=datetime.utcnow, nullable=False
)
+341
View File
@@ -0,0 +1,341 @@
"""
Configuration service for hybrid env + database settings.
Provides a unified interface for reading application configuration.
Settings are resolved in this order:
1. Database (app_settings table) — highest priority
2. Environment variables / .env file — fallback
3. Hard-coded defaults — last resort
Bootstrap settings (DATABASE_URL, SECRET_KEY, ENCRYPTION_KEY) always
come from environment variables because the database connection itself
depends on them.
"""
import json
import logging
import os
from typing import Any, Dict, List, Optional
from sqlalchemy import select
from sqlalchemy.ext.asyncio import AsyncSession
from app.models.database_models import AppSetting
logger = logging.getLogger(__name__)
# Settings that MUST come from env vars (bootstrap / chicken-and-egg)
BOOTSTRAP_KEYS = frozenset(
{
"DATABASE_URL",
"DATABASE_POOL_SIZE",
"DATABASE_MAX_OVERFLOW",
"SECRET_KEY",
"ENCRYPTION_KEY",
}
)
# Default settings to seed into the database on first run
DEFAULT_SETTINGS: List[Dict[str, Any]] = [
# SMTP / Forwarding
{
"key": "SMTP_HOST",
"value": "smtp.gmail.com",
"value_type": "string",
"description": "SMTP server hostname for email forwarding",
"is_secret": False,
"category": "smtp",
},
{
"key": "SMTP_PORT",
"value": "587",
"value_type": "int",
"description": "SMTP server port",
"is_secret": False,
"category": "smtp",
},
{
"key": "SMTP_USER",
"value": "",
"value_type": "string",
"description": "SMTP username for authentication",
"is_secret": False,
"category": "smtp",
},
{
"key": "SMTP_PASSWORD",
"value": "",
"value_type": "string",
"description": "SMTP password for authentication",
"is_secret": True,
"category": "smtp",
},
{
"key": "SMTP_USE_TLS",
"value": "true",
"value_type": "bool",
"description": "Whether to use STARTTLS for SMTP connections",
"is_secret": False,
"category": "smtp",
},
# Email Processing
{
"key": "MAX_EMAILS_PER_RUN",
"value": "50",
"value_type": "int",
"description": "Maximum emails to process per account per run",
"is_secret": False,
"category": "processing",
},
{
"key": "CHECK_INTERVAL_MINUTES",
"value": "5",
"value_type": "int",
"description": "Default interval between mail checks (minutes)",
"is_secret": False,
"category": "processing",
},
{
"key": "THROTTLE_EMAILS_PER_MINUTE",
"value": "10",
"value_type": "int",
"description": "Rate limit for email forwarding",
"is_secret": False,
"category": "processing",
},
# Gmail API
{
"key": "GMAIL_API_ENABLED",
"value": "true",
"value_type": "bool",
"description": "Enable Gmail API for direct email injection (preferred over SMTP)",
"is_secret": False,
"category": "gmail",
},
{
"key": "GMAIL_INJECT_LABEL_IDS",
"value": '["INBOX"]',
"value_type": "json",
"description": "Gmail label IDs to apply when injecting emails via API",
"is_secret": False,
"category": "gmail",
},
# Notifications
{
"key": "APPRISE_ENABLED",
"value": "true",
"value_type": "bool",
"description": "Enable Apprise multi-channel notifications",
"is_secret": False,
"category": "notifications",
},
# Logging
{
"key": "LOG_LEVEL",
"value": "INFO",
"value_type": "string",
"description": "Application log level (DEBUG, INFO, WARNING, ERROR)",
"is_secret": False,
"category": "general",
},
]
def _cast_value(raw: str, value_type: str) -> Any:
"""Cast a raw string value to its declared type."""
if raw is None:
return None
if value_type == "int":
return int(raw)
if value_type == "float":
return float(raw)
if value_type == "bool":
return raw.lower() in ("true", "1", "yes")
if value_type == "json":
return json.loads(raw)
return raw # string
class ConfigService:
"""
Hybrid configuration service: database-first with env-var fallback.
Usage::
value = await ConfigService.get("SMTP_HOST", db=session)
smtp_config = await ConfigService.get_smtp_config(db=session)
"""
@staticmethod
async def get(
key: str,
db: Optional[AsyncSession] = None,
default: Any = None,
) -> Any:
"""
Retrieve a single setting value.
Resolution order: database → environment variable → *default*.
Bootstrap keys (DATABASE_URL, SECRET_KEY, ENCRYPTION_KEY) skip the
database lookup entirely.
"""
if key not in BOOTSTRAP_KEYS and db is not None:
try:
result = await db.execute(
select(AppSetting).where(AppSetting.key == key)
)
setting = result.scalar_one_or_none()
if setting is not None and setting.value is not None:
return _cast_value(
setting.value, # type: ignore[arg-type]
setting.value_type or "string", # type: ignore[arg-type]
)
except Exception as exc:
logger.warning(
"DB lookup failed for key=%s, falling back to env: %s",
key,
exc,
)
env_val = os.getenv(key)
if env_val is not None:
return env_val
return default
@staticmethod
async def get_many(
keys: List[str],
db: Optional[AsyncSession] = None,
) -> Dict[str, Any]:
"""Retrieve multiple settings at once."""
result: Dict[str, Any] = {}
if db is not None:
try:
db_result = await db.execute(
select(AppSetting).where(AppSetting.key.in_(keys))
)
for setting in db_result.scalars().all():
result[setting.key] = _cast_value( # type: ignore[index]
setting.value, # type: ignore[arg-type]
setting.value_type or "string", # type: ignore[arg-type]
)
except Exception as exc:
logger.warning("DB bulk lookup failed, falling back to env: %s", exc)
for key in keys:
if key not in result:
env_val = os.getenv(key)
if env_val is not None:
result[key] = env_val
return result
@staticmethod
async def set(
key: str,
value: str,
db: AsyncSession,
value_type: str = "string",
description: Optional[str] = None,
is_secret: bool = False,
category: Optional[str] = None,
) -> AppSetting:
"""Create or update a setting in the database."""
if key in BOOTSTRAP_KEYS:
raise ValueError(
f"'{key}' is a bootstrap setting and cannot be stored in the database. "
"Set it via environment variables instead."
)
result = await db.execute(select(AppSetting).where(AppSetting.key == key))
existing = result.scalar_one_or_none()
if existing:
existing.value = value # type: ignore[assignment]
if value_type:
existing.value_type = value_type # type: ignore[assignment]
if description is not None:
existing.description = description # type: ignore[assignment]
existing.is_secret = is_secret # type: ignore[assignment]
if category is not None:
existing.category = category # type: ignore[assignment]
await db.commit()
await db.refresh(existing)
return existing
setting = AppSetting(
key=key,
value=value,
value_type=value_type,
description=description,
is_secret=is_secret,
category=category,
)
db.add(setting)
await db.commit()
await db.refresh(setting)
return setting
@staticmethod
async def delete(key: str, db: AsyncSession) -> bool:
"""Delete a setting from the database."""
if key in BOOTSTRAP_KEYS:
raise ValueError(
f"'{key}' is a bootstrap setting and cannot be deleted from the database."
)
result = await db.execute(select(AppSetting).where(AppSetting.key == key))
existing = result.scalar_one_or_none()
if existing:
await db.delete(existing)
await db.commit()
return True
return False
@staticmethod
async def list_all(
db: AsyncSession,
category: Optional[str] = None,
) -> List[AppSetting]:
"""List all settings, optionally filtered by category."""
query = select(AppSetting).order_by(AppSetting.category, AppSetting.key)
if category:
query = query.where(AppSetting.category == category)
result = await db.execute(query)
return list(result.scalars().all())
# ── convenience helpers ─────────────────────────────────────────
@staticmethod
async def get_smtp_config(db: Optional[AsyncSession] = None) -> Dict[str, Any]:
"""Return a ready-to-use SMTP configuration dict."""
keys = ["SMTP_HOST", "SMTP_PORT", "SMTP_USER", "SMTP_PASSWORD", "SMTP_USE_TLS"]
values = await ConfigService.get_many(keys, db=db)
return {
"host": values.get("SMTP_HOST", "smtp.gmail.com"),
"port": int(values.get("SMTP_PORT", 587)),
"username": values.get("SMTP_USER", ""),
"password": values.get("SMTP_PASSWORD", ""),
"use_tls": str(values.get("SMTP_USE_TLS", "true")).lower()
in ("true", "1", "yes"),
}
@staticmethod
async def seed_defaults(db: AsyncSession) -> int:
"""
Populate the database with default settings (skip existing keys).
Returns the number of settings created.
"""
created = 0
for item in DEFAULT_SETTINGS:
result = await db.execute(
select(AppSetting).where(AppSetting.key == item["key"])
)
if result.scalar_one_or_none() is None:
db.add(AppSetting(**item))
created += 1
if created:
await db.commit()
logger.info("Seeded %d default settings into the database", created)
return created
+3 -9
View File
@@ -3,7 +3,6 @@ Celery tasks for background email processing.
"""
import asyncio
import os
from datetime import datetime, timedelta, timezone
from celery import Task
import logging
@@ -21,6 +20,7 @@ from app.models.database_models import (
)
from app.services.mail_processor import MailProcessor
from app.services.gmail_service import GmailService
from app.services.config_service import ConfigService
from app.core.config import settings
from sqlalchemy import select, and_
@@ -118,14 +118,8 @@ async def process_mail_account(account_id: int):
use_gmail_api = False # type: ignore[assignment]
if not use_gmail_api:
# Fall back to SMTP
smtp_config = {
"host": os.getenv("SMTP_HOST", "smtp.gmail.com"),
"port": int(os.getenv("SMTP_PORT", "587")),
"username": os.getenv("SMTP_USER", ""),
"password": os.getenv("SMTP_PASSWORD", ""),
"use_tls": os.getenv("SMTP_USE_TLS", "true").lower() == "true",
}
# Fall back to SMTP read config from DB with env fallback
smtp_config = await ConfigService.get_smtp_config(db=db)
if not smtp_config["username"] or not smtp_config["password"]:
logger.error(
+299
View File
@@ -0,0 +1,299 @@
"""
Unit tests for the database-backed configuration service.
"""
import os
import pytest
from unittest.mock import AsyncMock, MagicMock, patch
from app.services.config_service import (
ConfigService,
BOOTSTRAP_KEYS,
DEFAULT_SETTINGS,
_cast_value,
)
# ── _cast_value tests ─────────────────────────────────────────────
class TestCastValue:
"""Test low-level value casting."""
def test_cast_string(self):
assert _cast_value("hello", "string") == "hello"
def test_cast_int(self):
assert _cast_value("42", "int") == 42
def test_cast_float(self):
assert _cast_value("3.14", "float") == 3.14
def test_cast_bool_true(self):
for val in ("true", "True", "1", "yes"):
assert _cast_value(val, "bool") is True
def test_cast_bool_false(self):
for val in ("false", "False", "0", "no"):
assert _cast_value(val, "bool") is False
def test_cast_json(self):
assert _cast_value('["INBOX"]', "json") == ["INBOX"]
assert _cast_value('{"a": 1}', "json") == {"a": 1}
def test_cast_none(self):
assert _cast_value(None, "string") is None
assert _cast_value(None, "int") is None
# ── ConfigService.get tests ────────────────────────────────────────
class TestConfigServiceGet:
"""Test ConfigService.get() resolution order."""
@pytest.mark.asyncio
async def test_returns_default_when_no_source(self):
"""When neither DB nor env has the key, return the default."""
with patch.dict(os.environ, {}, clear=False):
# Make sure the key is not in env
os.environ.pop("MY_TEST_KEY_XYZ", None)
result = await ConfigService.get("MY_TEST_KEY_XYZ", default="fallback")
assert result == "fallback"
@pytest.mark.asyncio
async def test_env_overrides_default(self):
"""Env var should override the default."""
with patch.dict(os.environ, {"MY_TEST_KEY_XYZ": "from_env"}):
result = await ConfigService.get("MY_TEST_KEY_XYZ", default="fallback")
assert result == "from_env"
@pytest.mark.asyncio
async def test_db_overrides_env(self):
"""Database value should take precedence over env var."""
mock_setting = MagicMock()
mock_setting.value = "from_db"
mock_setting.value_type = "string"
mock_result = MagicMock()
mock_result.scalar_one_or_none.return_value = mock_setting
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
with patch.dict(os.environ, {"SMTP_HOST": "from_env"}):
result = await ConfigService.get("SMTP_HOST", db=mock_db, default="default")
assert result == "from_db"
@pytest.mark.asyncio
async def test_bootstrap_key_skips_db(self):
"""Bootstrap keys must never read from the database."""
mock_db = AsyncMock()
with patch.dict(os.environ, {"SECRET_KEY": "env_secret_value"}):
result = await ConfigService.get("SECRET_KEY", db=mock_db)
# DB should not have been called
mock_db.execute.assert_not_called()
assert result == "env_secret_value"
@pytest.mark.asyncio
async def test_db_exception_falls_back_to_env(self):
"""If DB lookup fails, fall back to env var gracefully."""
mock_db = AsyncMock()
mock_db.execute.side_effect = Exception("DB down")
with patch.dict(os.environ, {"SMTP_HOST": "env_value"}):
result = await ConfigService.get("SMTP_HOST", db=mock_db)
assert result == "env_value"
# ── ConfigService.get_many tests ───────────────────────────────────
class TestConfigServiceGetMany:
"""Test ConfigService.get_many() bulk retrieval."""
@pytest.mark.asyncio
async def test_get_many_from_env(self):
"""When no DB, all values come from env."""
with patch.dict(
os.environ, {"SMTP_HOST": "host", "SMTP_PORT": "587"}, clear=False
):
result = await ConfigService.get_many(["SMTP_HOST", "SMTP_PORT"])
assert result["SMTP_HOST"] == "host"
assert result["SMTP_PORT"] == "587"
@pytest.mark.asyncio
async def test_get_many_with_db(self):
"""DB values should be included in results."""
mock_setting = MagicMock()
mock_setting.key = "SMTP_HOST"
mock_setting.value = "db_host"
mock_setting.value_type = "string"
mock_scalars = MagicMock()
mock_scalars.all.return_value = [mock_setting]
mock_result = MagicMock()
mock_result.scalars.return_value = mock_scalars
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
with patch.dict(os.environ, {"SMTP_PORT": "465"}, clear=False):
os.environ.pop("SMTP_HOST", None)
result = await ConfigService.get_many(
["SMTP_HOST", "SMTP_PORT"], db=mock_db
)
assert result["SMTP_HOST"] == "db_host"
assert result["SMTP_PORT"] == "465"
# ── ConfigService.set tests ────────────────────────────────────────
class TestConfigServiceSet:
"""Test ConfigService.set() create and update."""
@pytest.mark.asyncio
async def test_set_rejects_bootstrap_key(self):
"""Setting a bootstrap key must raise ValueError."""
mock_db = AsyncMock()
with pytest.raises(ValueError, match="bootstrap setting"):
await ConfigService.set("SECRET_KEY", "value", db=mock_db)
@pytest.mark.asyncio
async def test_set_creates_new_setting(self):
"""set() should create a new record when key does not exist."""
mock_result = MagicMock()
mock_result.scalar_one_or_none.return_value = None
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
# The method will call db.add() and db.commit()
await ConfigService.set(
"SMTP_HOST", "new.host.com", db=mock_db, category="smtp"
)
mock_db.add.assert_called_once()
mock_db.commit.assert_called()
@pytest.mark.asyncio
async def test_set_updates_existing_setting(self):
"""set() should update an existing record."""
existing = MagicMock()
existing.value = "old_value"
mock_result = MagicMock()
mock_result.scalar_one_or_none.return_value = existing
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
await ConfigService.set("SMTP_HOST", "updated.host.com", db=mock_db)
assert existing.value == "updated.host.com"
mock_db.commit.assert_called()
# ── ConfigService.delete tests ─────────────────────────────────────
class TestConfigServiceDelete:
"""Test ConfigService.delete()."""
@pytest.mark.asyncio
async def test_delete_rejects_bootstrap_key(self):
mock_db = AsyncMock()
with pytest.raises(ValueError, match="bootstrap setting"):
await ConfigService.delete("DATABASE_URL", db=mock_db)
@pytest.mark.asyncio
async def test_delete_returns_false_when_missing(self):
mock_result = MagicMock()
mock_result.scalar_one_or_none.return_value = None
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
assert await ConfigService.delete("NONEXISTENT", db=mock_db) is False
@pytest.mark.asyncio
async def test_delete_removes_existing(self):
existing = MagicMock()
mock_result = MagicMock()
mock_result.scalar_one_or_none.return_value = existing
mock_db = AsyncMock()
mock_db.execute.return_value = mock_result
assert await ConfigService.delete("SMTP_HOST", db=mock_db) is True
mock_db.delete.assert_called_once_with(existing)
# ── ConfigService.get_smtp_config tests ────────────────────────────
class TestConfigServiceSmtpConfig:
"""Test the SMTP convenience helper."""
@pytest.mark.asyncio
async def test_smtp_config_defaults(self):
"""When nothing is configured, sensible defaults are returned."""
with patch.dict(os.environ, {}, clear=False):
for key in (
"SMTP_HOST",
"SMTP_PORT",
"SMTP_USER",
"SMTP_PASSWORD",
"SMTP_USE_TLS",
):
os.environ.pop(key, None)
config = await ConfigService.get_smtp_config()
assert config["host"] == "smtp.gmail.com"
assert config["port"] == 587
assert config["username"] == ""
assert config["password"] == ""
assert config["use_tls"] is True
@pytest.mark.asyncio
async def test_smtp_config_from_env(self):
"""Env vars should populate the SMTP config."""
env = {
"SMTP_HOST": "mail.example.com",
"SMTP_PORT": "465",
"SMTP_USER": "user",
"SMTP_PASSWORD": "pass",
"SMTP_USE_TLS": "false",
}
with patch.dict(os.environ, env, clear=False):
config = await ConfigService.get_smtp_config()
assert config["host"] == "mail.example.com"
assert config["port"] == 465
assert config["username"] == "user"
assert config["password"] == "pass"
assert config["use_tls"] is False
# ── DEFAULT_SETTINGS sanity check ──────────────────────────────────
class TestDefaultSettings:
"""Verify the built-in defaults are well-formed."""
def test_all_defaults_have_required_fields(self):
for item in DEFAULT_SETTINGS:
assert "key" in item
assert "value" in item
assert "value_type" in item
def test_no_default_is_a_bootstrap_key(self):
for item in DEFAULT_SETTINGS:
assert item["key"] not in BOOTSTRAP_KEYS
+73 -4
View File
@@ -28,16 +28,18 @@ pop_puller_to_gmail/
│ │ │ ├── endpoints/ # Individual route modules
│ │ │ └── api.py # Router aggregation
│ │ ├── core/ # Core configuration
│ │ │ ├── config.py # Settings management
│ │ │ ├── config.py # Bootstrap settings (env / .env)
│ │ │ ├── database.py # Database connection
│ │ │ ├── security.py # Security utilities
│ │ │ └── deps.py # FastAPI dependencies
│ │ ├── models/ # Data models
│ │ │ ├── database_models.py # SQLAlchemy models
│ │ │ ├── database_models.py # SQLAlchemy models (incl. AppSetting)
│ │ │ └── schemas.py # Pydantic schemas
│ │ ├── services/ # Business logic
│ │ │ ├── auth_service.py # OAuth authentication
│ │ │ ── mail_processor.py # Email processing
│ │ │ ── config_service.py # Hybrid config (DB + env)
│ │ │ ├── gmail_service.py # Gmail API injection
│ │ │ └── mail_processor.py # POP3/IMAP email processing
│ │ ├── workers/ # Celery background tasks
│ │ ├── utils/ # Utility functions
│ │ └── main.py # FastAPI application
@@ -112,10 +114,31 @@ pop_puller_to_gmail/
- **Background Jobs**: Celery workers for async processing
- **Scheduled Checks**: Configurable intervals per account
- **Smart Forwarding**: Preserves metadata, handles MIME types
- **Dual Delivery**: Gmail API injection (preferred) or SMTP forwarding (fallback)
- **Error Handling**: Automatic retries with exponential backoff
- **Statistics**: Track success/failure rates, last check times
### 3a. Gmail API vs SMTP Delivery
The platform supports two methods for delivering fetched emails to Gmail:
| Feature | Gmail API (`gmail_api`) | SMTP Forwarding (`smtp`) |
|---------|------------------------|--------------------------|
| **Header preservation** | ✅ All original headers intact | ⚠️ Adds `Received` headers, may rewrite `From` |
| **Gmail sending quota** | ✅ Does not count against quota | ❌ Counts against 500/day free limit |
| **Authentication** | OAuth2 tokens (per-user) | App Password (shared) |
| **Setup complexity** | Requires OAuth2 consent flow | Requires App Password only |
| **Spam risk** | ✅ Low (email appears native) | ⚠️ Higher (forwarded mail may be flagged) |
| **Fallback** | Falls back to SMTP if no credentials | Primary legacy method |
**How it works:**
1. Each mail account has a `delivery_method` field (`gmail_api` or `smtp`)
2. When `gmail_api` is selected, the worker looks up the user's `GmailCredential`
3. The `GmailService` calls `users.messages.insert()` to inject the raw RFC 2822 email
4. If no valid Gmail credential is found, the worker falls back to SMTP automatically
5. SMTP settings are loaded from the database (via `ConfigService`) with env-var fallback
### 4. Subscription Management
- **Stripe Integration**: Secure payment processing
@@ -151,6 +174,8 @@ pop_puller_to_gmail/
- **subscription_plans**: Available subscription tiers
- **mail_server_presets**: Known provider configurations
- **audit_logs**: Security and compliance audit trail
- **gmail_credentials**: Per-user OAuth2 tokens for Gmail API injection
- **app_settings**: Database-backed application configuration (key-value store)
## 🔌 API Endpoints
@@ -184,10 +209,54 @@ pop_puller_to_gmail/
### Admin
- `GET /api/v1/admin/stats` - System statistics (admin only)
### Settings (Admin)
- `GET /api/v1/settings` - List all database-backed settings
- `PUT /api/v1/settings/{key}` - Create or update a setting
- `DELETE /api/v1/settings/{key}` - Delete a setting
- `POST /api/v1/settings/seed-defaults` - Seed default settings
### Providers & Gmail
- `GET /api/v1/providers/presets` - List mail provider presets
- `GET /api/v1/providers/presets/{id}` - Get a specific preset
- `POST /api/v1/providers/gmail-credential` - Save Gmail API credentials
- `GET /api/v1/providers/gmail-credential` - Get Gmail credential status
- `DELETE /api/v1/providers/gmail-credential` - Remove Gmail credentials
See full API documentation at `/api/docs` when running.
## 🔧 Configuration
### Hybrid Configuration Model
The application uses a **hybrid configuration model** where settings can come
from either the database or environment variables:
```
┌─────────────────────────────────────────────────────┐
│ Setting Lookup Priority │
│ │
│ 1. Database (app_settings table) ← highest │
│ 2. Environment variable / .env file │
│ 3. Built-in default ← lowest │
└─────────────────────────────────────────────────────┘
```
**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`)
always come from environment variables because the database connection
depends on them.
All other settings (SMTP config, processing intervals, Gmail API options,
etc.) can be managed via the **Admin Settings API** (`/api/v1/settings`)
and are stored in PostgreSQL. On first startup the application seeds
sensible defaults into the `app_settings` table.
**Key components:**
- `app.core.config.Settings` — Pydantic Settings for bootstrap config
- `app.models.database_models.AppSetting` — SQLAlchemy model for DB-backed settings
- `app.services.config_service.ConfigService` — Hybrid resolver (DB → env → default)
- `app.api.v1.endpoints.app_settings` — Admin CRUD endpoints
### Environment Variables
Key configuration options in `backend/.env`:
+7 -5
View File
@@ -69,7 +69,7 @@ Comprehensive task breakdown for repository improvements and production readines
- [x] Write unit tests for middleware (98% coverage)
- [x] Write unit tests for schemas and validation
- [x] Write unit tests for application factory and core endpoints
- [x] Reach 50%+ test coverage (currently 57%)
- [x] Reach 50%+ test coverage (currently 59%)
### In Progress 🔨
- [ ] Write unit tests for authentication (target 80%+ coverage)
@@ -135,10 +135,12 @@ Comprehensive task breakdown for repository improvements and production readines
### Completed ✅
- [x] Basic health check endpoint exists
- [x] Database-backed configuration (`AppSetting` model + `ConfigService`)
- [x] Admin API for managing settings (`/api/v1/settings`)
- [x] Default settings seeded on first startup
### In Progress 🔨
- [ ] Improve health checks (DB/Redis connectivity)
- [ ] Add environment variable validation
### Not Started 📋
- [ ] Create production docker-compose.yml
@@ -292,12 +294,12 @@ because the API client layer is missing.
|----------|----------|--------|
| Security | 60% | 🟡 In Progress |
| Agentic Infrastructure | 95% | 🟢 Near Complete |
| Testing | 57% | 🟡 In Progress |
| Testing | 59% | 🟡 In Progress |
| CI/CD | 80% | 🟢 Near Complete |
| Code Quality | 40% | 🔴 Needs Work |
| Production Ready | 20% | 🔴 Needs Work |
| Production Ready | 30% | 🔴 Needs Work |
| Observability | 10% | 🔴 Needs Work |
| Backend Features | 80% | 🟢 Near Complete |
| Backend Features | 85% | 🟢 Near Complete |
| Frontend | 50% | 🟡 In Progress |
**Overall Repository Readiness**: 55% ⚠️
+2 -2
View File
@@ -22,8 +22,8 @@
"@types/node": "^20",
"@types/react": "^19",
"@types/react-dom": "^19",
"eslint": "^9",
"eslint-config-next": "16.1.7",
"eslint": "^10",
"eslint-config-next": "16.2.1",
"tailwindcss": "^4",
"typescript": "^5"
}