Add ADR-003 through ADR-010, update TODO.md and CHANGELOG.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/dd261f53-4891-437e-bd71-5561ece62d7d
This commit is contained in:
@@ -32,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- Upgraded `sqlalchemy` from `2.0.25` to `2.0.48` to fix `AssertionError: Class ... directly inherits TypingOnly but has additional attributes` on Python 3.14 (`__static_attributes__`, `__firstlineno__`)
|
||||
|
||||
### Added
|
||||
- **Architecture Decision Records ADR-003 through ADR-010**: Added eight new ADRs covering FastAPI web framework (ADR-003), PostgreSQL database (ADR-004), Celery task retry strategy (ADR-005), key management in production (ADR-006), JWT authentication (ADR-007), Next.js frontend (ADR-008), Gmail API email delivery (ADR-009), and hybrid configuration model (ADR-010)
|
||||
- `userApi.updateProfile()` method in `frontend/src/lib/api.ts` for updating user profile via `PUT /users/me`
|
||||
- **Backend URL logged at startup**: The Next.js server now logs the resolved `BACKEND_URL` (e.g. `[proxy] BACKEND_URL = http://backend:8000`) via `src/instrumentation.ts` when the server starts, making it easy to diagnose `ECONNREFUSED` proxy errors. The per-request error log now also includes the full target URL.
|
||||
- **Dual-registry Docker deployment**: CI now builds separate backend and frontend images and pushes to both GHCR (`ghcr.io`) and private registry (`registry.cklnet.com`) using a matrix strategy
|
||||
|
||||
+4
-4
@@ -44,7 +44,7 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
|
||||
### In Progress 🔨
|
||||
- [x] Reorganize documentation into `docs/` directory
|
||||
- [ ] Complete ADR documentation (add ADR-003 through ADR-010)
|
||||
- [x] Complete ADR documentation (add ADR-003 through ADR-010)
|
||||
- [ ] Create GitHub Projects board for task management
|
||||
|
||||
### Not Started 📋
|
||||
@@ -301,7 +301,7 @@ because the API client layer is missing.
|
||||
| Category | Progress | Status |
|
||||
|----------|----------|--------|
|
||||
| Security | 60% | 🟡 In Progress |
|
||||
| Agentic Infrastructure | 95% | 🟢 Near Complete |
|
||||
| Agentic Infrastructure | 98% | 🟢 Near Complete |
|
||||
| Testing | 59% | 🟡 In Progress |
|
||||
| CI/CD | 80% | 🟢 Near Complete |
|
||||
| Code Quality | 40% | 🔴 Needs Work |
|
||||
@@ -325,7 +325,7 @@ because the API client layer is missing.
|
||||
- [ ] Enable rate limiting
|
||||
- [ ] Add audit logging
|
||||
- [ ] Write more unit tests (target 70% coverage)
|
||||
- [ ] Complete ADR documentation
|
||||
- [x] Complete ADR documentation
|
||||
- [ ] End-to-end test frontend against backend
|
||||
|
||||
3. **Next Week**:
|
||||
@@ -368,6 +368,6 @@ Must complete before production:
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-03-23
|
||||
**Last Updated**: 2026-03-25
|
||||
**Maintained By**: Development Team
|
||||
**Review Frequency**: Weekly
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
# ADR 003: Use FastAPI as Web Framework
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-01-25
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The multi-tenant SaaS backend requires a Python web framework that can handle:
|
||||
|
||||
1. High-concurrency API requests (multiple users, simultaneous email polls)
|
||||
2. Asynchronous database and network I/O without blocking
|
||||
3. Automatic input validation and serialization
|
||||
4. Built-in interactive API documentation
|
||||
5. Easy integration with the async ecosystem (asyncpg, aiosmtplib, aioimaplib)
|
||||
|
||||
## Decision
|
||||
|
||||
We will use **FastAPI** as the primary web framework for the backend API.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Django REST Framework (DRF)
|
||||
- **Pros**: Mature ecosystem, batteries-included ORM, admin panel, well-known
|
||||
- **Cons**: Synchronous-first, heavier footprint, complex async support, more boilerplate
|
||||
|
||||
### 2. Flask
|
||||
- **Pros**: Lightweight, flexible, large community
|
||||
- **Cons**: No native async support, requires extensions for validation, no auto-docs, more manual wiring
|
||||
|
||||
### 3. Starlette (bare)
|
||||
- **Pros**: Minimal, pure async, very fast
|
||||
- **Cons**: No built-in validation, no auto-documentation, requires writing more boilerplate
|
||||
|
||||
### 4. Tornado
|
||||
- **Pros**: Battle-tested async framework, good WebSocket support
|
||||
- **Cons**: Older API design, less active development, no modern type annotation support
|
||||
|
||||
## Rationale
|
||||
|
||||
FastAPI was chosen because:
|
||||
|
||||
1. **Native Async Support**: First-class `async`/`await` support matches our async database (asyncpg) and mail protocol (aiosmtplib, aioimaplib) libraries
|
||||
2. **Automatic Validation**: Pydantic models provide request/response validation with no extra code
|
||||
3. **Auto-generated Docs**: OpenAPI spec and Swagger UI at `/api/docs` out of the box
|
||||
4. **Type Safety**: Python type annotations drive both validation and editor tooling
|
||||
5. **Performance**: Comparable to Node.js and Go for async I/O workloads (Starlette/uvicorn underneath)
|
||||
6. **Dependency Injection**: Built-in `Depends()` system for auth, database sessions, and config
|
||||
7. **Modern Python**: Designed for Python 3.8+ with full typing support
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
```python
|
||||
# Application factory pattern with lifespan context manager
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup: create tables, seed defaults
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
yield
|
||||
# Shutdown: cleanup
|
||||
|
||||
app = FastAPI(
|
||||
title="POP3 Forwarder API",
|
||||
lifespan=lifespan,
|
||||
openapi_url="/api/openapi.json",
|
||||
docs_url="/api/docs",
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
# Versioned API routing
|
||||
from fastapi import APIRouter
|
||||
|
||||
api_router = APIRouter(prefix="/api/v1")
|
||||
api_router.include_router(auth.router, prefix="/auth", tags=["auth"])
|
||||
api_router.include_router(mail_accounts.router, prefix="/mail-accounts", tags=["mail-accounts"])
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Clean, self-documenting API with zero extra work
|
||||
- Async I/O throughout eliminates thread-pool bottlenecks
|
||||
- Pydantic validation catches bad input before it reaches business logic
|
||||
- Dependency injection decouples auth, DB, and config from route handlers
|
||||
- Fast iteration: type errors and validation errors caught at startup
|
||||
|
||||
### Negative
|
||||
- Smaller ecosystem than Django (fewer ready-made plugins)
|
||||
- Pydantic v2 migration required breaking changes from v1
|
||||
- Lifespan/startup patterns require careful structuring to avoid import cycles
|
||||
|
||||
### Neutral
|
||||
- Uvicorn required as ASGI server for production
|
||||
- Gunicorn can be used to manage multiple uvicorn workers
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-004 for database choice (asyncpg/SQLAlchemy async)
|
||||
- See ADR-007 for JWT authentication via `Depends()`
|
||||
|
||||
## References
|
||||
|
||||
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
|
||||
- [Starlette Documentation](https://www.starlette.io/)
|
||||
- [Pydantic Documentation](https://docs.pydantic.dev/)
|
||||
- [TechEmpower Framework Benchmarks](https://www.techempower.com/benchmarks/)
|
||||
@@ -0,0 +1,129 @@
|
||||
# ADR 004: Use PostgreSQL as Primary Database
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-01-28
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The application requires a relational database to store:
|
||||
|
||||
1. User accounts and subscription data
|
||||
2. Mail account configurations (servers, protocols, check intervals)
|
||||
3. Processing run history and per-email logs
|
||||
4. Notification configurations
|
||||
5. Subscription plans and audit logs
|
||||
6. Database-backed application settings (`app_settings` key-value store)
|
||||
|
||||
Requirements:
|
||||
- ACID transactions for financial/subscription data
|
||||
- Foreign key constraints for referential integrity
|
||||
- JSON support for flexible metadata storage
|
||||
- Async driver support for FastAPI integration
|
||||
- Horizontal read-scaling capability
|
||||
|
||||
## Decision
|
||||
|
||||
We will use **PostgreSQL 15+** as the primary relational database, accessed via **SQLAlchemy 2.x** with the **asyncpg** driver.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. MySQL / MariaDB
|
||||
- **Pros**: Wide adoption, good tooling, familiar to many developers
|
||||
- **Cons**: Historically weaker JSON support, slightly different SQL dialect, asyncio driver (aiomysql) less mature than asyncpg
|
||||
|
||||
### 2. SQLite
|
||||
- **Pros**: Zero infrastructure, simple setup, file-based
|
||||
- **Cons**: No concurrent writes, no horizontal scaling, not suitable for multi-user production SaaS
|
||||
|
||||
### 3. MongoDB
|
||||
- **Pros**: Flexible schema, easy horizontal sharding, native JSON
|
||||
- **Cons**: No ACID transactions across collections (before 4.0), weaker relational integrity, harder to query with joins, async support less mature
|
||||
|
||||
### 4. CockroachDB
|
||||
- **Pros**: Distributed SQL, auto-sharding, highly available
|
||||
- **Cons**: More complex deployment, higher cost, unnecessary for initial scale
|
||||
|
||||
## Rationale
|
||||
|
||||
PostgreSQL was chosen because:
|
||||
|
||||
1. **ACID Compliance**: Full transaction support critical for subscription billing and user data integrity
|
||||
2. **JSON/JSONB Support**: Native JSON columns allow flexible metadata without schema migrations
|
||||
3. **asyncpg Driver**: The fastest PostgreSQL async driver for Python, purpose-built for asyncio
|
||||
4. **SQLAlchemy 2.x Async**: Mature async ORM integration via `create_async_engine` and `AsyncSession`
|
||||
5. **Extension Ecosystem**: uuid-ossp, pgcrypto, and other extensions available if needed
|
||||
6. **Alembic Migrations**: SQLAlchemy's Alembic integrates seamlessly for schema version control
|
||||
7. **Row-Level Security**: Available for multi-tenant data isolation if required in future
|
||||
8. **Industry Standard**: Well-understood operational characteristics, strong community, excellent documentation
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Async Engine Configuration
|
||||
```python
|
||||
from sqlalchemy.ext.asyncio import create_async_engine, AsyncSession
|
||||
from sqlalchemy.orm import sessionmaker
|
||||
|
||||
engine = create_async_engine(
|
||||
settings.DATABASE_URL,
|
||||
echo=False,
|
||||
connect_args={"prepared_statement_cache_size": 0}, # Avoids plan invalidation when create_all() runs CREATE TYPE DDL at startup
|
||||
)
|
||||
|
||||
AsyncSessionLocal = sessionmaker(
|
||||
engine, class_=AsyncSession, expire_on_commit=False
|
||||
)
|
||||
```
|
||||
|
||||
### Session Dependency
|
||||
```python
|
||||
async def get_db() -> AsyncSession:
|
||||
async with AsyncSessionLocal() as session:
|
||||
yield session
|
||||
```
|
||||
|
||||
### Schema Management
|
||||
```bash
|
||||
# Auto-generate migration from model changes
|
||||
alembic revision --autogenerate -m "add gmail_credentials table"
|
||||
|
||||
# Apply all pending migrations
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Full relational integrity with foreign keys and constraints
|
||||
- Async I/O with asyncpg eliminates blocking database calls
|
||||
- Alembic provides version-controlled, reviewable schema changes
|
||||
- Familiar SQL tooling (pgAdmin, psql, etc.) for debugging
|
||||
- Supports connection pooling (PgBouncer) for high-concurrency deployments
|
||||
|
||||
### Negative
|
||||
- Additional infrastructure to deploy and operate (unlike SQLite)
|
||||
- asyncpg prepared statement cache must be disabled when `create_all()` runs `CREATE TYPE … AS ENUM` DDL at startup — this DDL invalidates cached plans on the same connection, causing the next enum-type existence check to fail with `ProgrammingError: cached statement plan is invalid` (fix: `prepared_statement_cache_size=0`)
|
||||
- Async SQLAlchemy patterns are more complex than synchronous ORM patterns
|
||||
|
||||
### Neutral
|
||||
- Redis is still required as a separate service (for Celery broker/result backend)
|
||||
- Database backups must be configured separately (pg_dump or WAL archiving)
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
All schema changes are managed via Alembic:
|
||||
- Development: auto-generate from SQLAlchemy model changes
|
||||
- Production: migrations run explicitly via `alembic upgrade head`
|
||||
- Rollback: `alembic downgrade -1` for single-step rollback
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-001 for Celery (uses Redis, separate from PostgreSQL)
|
||||
- See ADR-010 for hybrid configuration model (uses `app_settings` table in PostgreSQL)
|
||||
|
||||
## References
|
||||
|
||||
- [PostgreSQL Documentation](https://www.postgresql.org/docs/)
|
||||
- [SQLAlchemy Async](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html)
|
||||
- [asyncpg Documentation](https://magicstack.github.io/asyncpg/)
|
||||
- [Alembic Documentation](https://alembic.sqlalchemy.org/)
|
||||
@@ -0,0 +1,128 @@
|
||||
# ADR 005: Celery Task Retry Strategy
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-01
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
Email processing tasks can fail for many transient reasons:
|
||||
|
||||
1. Mail server temporarily unavailable or overloaded
|
||||
2. Network timeouts during POP3/IMAP connections
|
||||
3. SMTP delivery failures (temporary, e.g. greylisting)
|
||||
4. Gmail API rate limits or transient 5xx errors
|
||||
5. Database connection errors
|
||||
|
||||
Without a retry strategy, transient failures result in permanently missed emails. However, aggressive retries can overload external services or cause duplicate deliveries.
|
||||
|
||||
## Decision
|
||||
|
||||
We will use **Celery's built-in autoretry mechanism** with **exponential backoff**, capped at **3 retries** per task invocation. Failed tasks after exhausting retries are recorded in the processing log and flagged on the mail account for user visibility.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. No Retries (fail-fast)
|
||||
- **Pros**: Simple, predictable
|
||||
- **Cons**: Transient failures cause permanent data loss, poor user experience
|
||||
|
||||
### 2. Infinite Retries
|
||||
- **Pros**: Guarantees eventual processing
|
||||
- **Cons**: Fills task queue, may mask persistent failures, delays detection of real errors
|
||||
|
||||
### 3. Fixed-Interval Retries
|
||||
- **Pros**: Simple to reason about
|
||||
- **Cons**: Hammers failing services, does not give them time to recover
|
||||
|
||||
### 4. External Retry Orchestrator (e.g., Temporal, AWS Step Functions)
|
||||
- **Pros**: Advanced workflow management, visual debugging
|
||||
- **Cons**: Significant infrastructure overhead, unnecessary complexity at current scale
|
||||
|
||||
### 5. Manual Retry Queue
|
||||
- **Pros**: Full control
|
||||
- **Cons**: Re-implements what Celery already provides
|
||||
|
||||
## Rationale
|
||||
|
||||
Celery's `autoretry_for` with `retry_backoff=True` was chosen because:
|
||||
|
||||
1. **Native Integration**: No additional libraries required, built into Celery
|
||||
2. **Exponential Backoff**: Gives external services time to recover between retries
|
||||
3. **Jitter**: Prevents thundering herd when many accounts fail simultaneously
|
||||
4. **Bounded Retries**: `max_retries=3` ensures tasks eventually fail rather than running forever
|
||||
5. **Configurable**: Per-task retry configuration allows tuning per failure type
|
||||
6. **Visibility**: Failed tasks appear in Flower dashboard and processing logs
|
||||
|
||||
## Implementation
|
||||
|
||||
```python
|
||||
from celery import Task
|
||||
from app.core.celery_app import celery_app
|
||||
|
||||
@celery_app.task(
|
||||
bind=True,
|
||||
autoretry_for=(ConnectionError, TimeoutError, OSError),
|
||||
retry_kwargs={"max_retries": 3, "countdown": 60},
|
||||
retry_backoff=True, # Exponential: 60s, 120s, 240s
|
||||
retry_backoff_max=600, # Cap at 10 minutes
|
||||
retry_jitter=True, # Add randomness to spread load
|
||||
)
|
||||
def process_mail_account(self: Task, account_id: int) -> dict:
|
||||
"""Fetch and forward emails for a single mail account."""
|
||||
try:
|
||||
# ... email processing logic ...
|
||||
pass
|
||||
except Exception as exc:
|
||||
# Log failure to processing_logs table before re-raising
|
||||
_record_failure(account_id, str(exc))
|
||||
raise
|
||||
```
|
||||
|
||||
### Retry Schedule (default config)
|
||||
|
||||
| Attempt | Delay (approx.) | Total elapsed |
|
||||
|---------|-----------------|---------------|
|
||||
| 1st | immediate | 0s |
|
||||
| 2nd | ~60s | 60s |
|
||||
| 3rd | ~120s | 3m |
|
||||
| 4th | ~240s | 7m |
|
||||
| Final failure | — recorded to DB | ~7m |
|
||||
|
||||
### Failure Handling After All Retries
|
||||
|
||||
When all retries are exhausted, the task:
|
||||
1. Marks the `ProcessingRun` as `failed` in the database
|
||||
2. Increments the `consecutive_failures` counter on the `MailAccount`
|
||||
3. If `consecutive_failures` exceeds threshold: marks account as `error` state
|
||||
4. Triggers a user notification via Apprise (if configured)
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Transient failures (network blips, greylisting) recover automatically
|
||||
- Exponential backoff respects external service rate limits
|
||||
- Bounded retry count prevents runaway task accumulation
|
||||
- Failure visibility in Flower and processing logs
|
||||
|
||||
### Negative
|
||||
- Up to ~7 minutes of additional delay for permanently failing accounts
|
||||
- Retry state stored in Redis; Redis failure loses retry context
|
||||
- Duplicate delivery possible if task succeeds after partial completion (must ensure idempotency)
|
||||
|
||||
### Idempotency Requirement
|
||||
|
||||
Tasks **must be idempotent**: re-running a task for the same mail account must not deliver duplicate emails. Implementations must:
|
||||
- Track message UIDs already processed in `processing_logs`
|
||||
- Use POP3 `UIDL` or IMAP `UID` to identify messages
|
||||
- Check for existing `ProcessingLog` entries before delivery
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-001 for Celery architecture overview
|
||||
- See ADR-009 for Gmail API delivery (idempotency considerations)
|
||||
|
||||
## References
|
||||
|
||||
- [Celery Retrying Tasks](https://docs.celeryq.dev/en/stable/userguide/tasks.html#retrying)
|
||||
- [Celery autoretry_for](https://docs.celeryq.dev/en/stable/userguide/tasks.html#automatic-retry-for-known-exceptions)
|
||||
- [Exponential Backoff and Jitter (AWS blog)](https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/)
|
||||
@@ -0,0 +1,183 @@
|
||||
# ADR 006: Key Management in Production
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-05
|
||||
**Deciders:** Security Team, Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The application requires several cryptographic secrets for secure operation:
|
||||
|
||||
1. `SECRET_KEY` — signs and verifies JWT tokens (HMAC-SHA256)
|
||||
2. `ENCRYPTION_KEY` — Fernet master key for encrypting POP3/IMAP passwords at rest (see ADR-002)
|
||||
3. `GOOGLE_CLIENT_SECRET` — OAuth2 client secret for Google Sign-In
|
||||
4. `STRIPE_API_KEY` — Stripe payment processing API key
|
||||
|
||||
These secrets must be:
|
||||
- **Available at startup** (bootstrap dependency before database is ready)
|
||||
- **Never committed to source control**
|
||||
- **Rotatable** without application downtime
|
||||
- **Auditable** (who accessed what, when)
|
||||
|
||||
## Decision
|
||||
|
||||
For the initial production deployment we will manage secrets via **environment variables injected at container runtime**, loaded from a **secrets management backend** (Docker secrets, Kubernetes Secrets, or HashiCorp Vault depending on deployment target). Local development uses `.env` files that are `.gitignore`d.
|
||||
|
||||
Startup validation rejects the application launch if any required secret is missing or too short.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Hardcoded / In-Code Defaults
|
||||
- **Pros**: Simple, no external dependency
|
||||
- **Cons**: Catastrophic security failure, impossible to rotate without code deployment
|
||||
|
||||
### 2. Plain `.env` Files in Production
|
||||
- **Pros**: Simple, portable
|
||||
- **Cons**: Files on disk are a security risk, not auditable, hard to rotate across multiple instances
|
||||
|
||||
### 3. HashiCorp Vault
|
||||
- **Pros**: Industry-standard secret management, dynamic secrets, full audit trail, automatic rotation
|
||||
- **Cons**: Significant operational overhead for initial deployment, requires dedicated Vault cluster
|
||||
|
||||
### 4. AWS Secrets Manager / Azure Key Vault
|
||||
- **Pros**: Managed service, automatic rotation, IAM integration
|
||||
- **Cons**: Cloud vendor lock-in, adds latency on secret retrieval, requires cloud SDK
|
||||
|
||||
### 5. Docker Swarm Secrets / Kubernetes Secrets
|
||||
- **Pros**: Native to container orchestration platform, mounted as files, not in env
|
||||
- **Cons**: Still requires base64 encoding, secrets accessible to anyone with cluster access unless using encrypted etcd
|
||||
|
||||
## Rationale
|
||||
|
||||
Environment variable injection was chosen as the **pragmatic starting point** because:
|
||||
|
||||
1. **Universally Supported**: Works identically in Docker Compose, Kubernetes, and bare-metal
|
||||
2. **No Additional Infrastructure**: No Vault cluster to operate initially
|
||||
3. **Startup Validation**: FastAPI lifespan validates all required secrets before accepting requests
|
||||
4. **Platform Agnostic**: Easy to migrate to Vault or cloud KMS later without code changes
|
||||
5. **12-Factor App Compliance**: Follows 12-factor app principle for configuration
|
||||
|
||||
HashiCorp Vault is documented as the **target architecture** for enterprise deployments (see Future Roadmap).
|
||||
|
||||
## Implementation
|
||||
|
||||
### Startup Validation
|
||||
```python
|
||||
# backend/app/core/config.py
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
class Settings(BaseSettings):
|
||||
SECRET_KEY: str
|
||||
ENCRYPTION_KEY: str
|
||||
|
||||
@validator("SECRET_KEY")
|
||||
def secret_key_min_length(cls, v: str) -> str:
|
||||
if len(v) < 32:
|
||||
raise ValueError("SECRET_KEY must be at least 32 characters")
|
||||
return v
|
||||
|
||||
@validator("ENCRYPTION_KEY")
|
||||
def encryption_key_must_be_valid_fernet(cls, v: str) -> str:
|
||||
try:
|
||||
Fernet(v.encode())
|
||||
except Exception:
|
||||
raise ValueError("ENCRYPTION_KEY must be a valid Fernet key")
|
||||
return v
|
||||
```
|
||||
|
||||
### Docker Compose (Development / Staging)
|
||||
```yaml
|
||||
# docker-compose.new.yml
|
||||
services:
|
||||
backend:
|
||||
env_file:
|
||||
- backend/.env # Never committed to git
|
||||
environment:
|
||||
- SECRET_KEY=${SECRET_KEY}
|
||||
- ENCRYPTION_KEY=${ENCRYPTION_KEY}
|
||||
```
|
||||
|
||||
### Kubernetes (Production)
|
||||
```yaml
|
||||
# Create secret from secure source (not echo/printf)
|
||||
kubectl create secret generic app-secrets \
|
||||
--from-literal=SECRET_KEY="$(vault kv get -field=SECRET_KEY secret/app)" \
|
||||
--from-literal=ENCRYPTION_KEY="$(vault kv get -field=ENCRYPTION_KEY secret/app)"
|
||||
|
||||
# Reference in Deployment
|
||||
envFrom:
|
||||
- secretRef:
|
||||
name: app-secrets
|
||||
```
|
||||
|
||||
### Generating Secure Keys
|
||||
```bash
|
||||
# Generate SECRET_KEY (min 32 chars, cryptographically random)
|
||||
python -c "import secrets; print(secrets.token_hex(32))"
|
||||
|
||||
# Generate ENCRYPTION_KEY (valid Fernet key)
|
||||
python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
|
||||
```
|
||||
|
||||
## Key Rotation Procedure
|
||||
|
||||
### JWT Secret Key Rotation
|
||||
1. Generate new `SECRET_KEY`
|
||||
2. Deploy with both old and new key (dual-key verification)
|
||||
3. All existing tokens expire within TTL (default: 30 minutes access, 7 days refresh)
|
||||
4. Remove old key from config after max TTL has elapsed
|
||||
|
||||
### Encryption Key Rotation (Fernet)
|
||||
1. Generate new `ENCRYPTION_KEY`
|
||||
2. Run migration script to re-encrypt all stored credentials with new key
|
||||
3. Deploy new key in production
|
||||
4. Verify decryption works on all accounts
|
||||
5. Delete old key from secrets store
|
||||
|
||||
```python
|
||||
# Key rotation migration (run as one-off script)
|
||||
async def rotate_encryption_key(old_key: str, new_key: str, db: AsyncSession):
|
||||
old_fernet = Fernet(old_key.encode())
|
||||
new_fernet = Fernet(new_key.encode())
|
||||
accounts = await db.execute(select(MailAccount).where(MailAccount.password.is_not(None)))
|
||||
for account in accounts.scalars():
|
||||
plaintext = old_fernet.decrypt(account.password.encode())
|
||||
account.password = new_fernet.encrypt(plaintext).decode()
|
||||
await db.commit()
|
||||
```
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Startup validation prevents misconfigured deployments
|
||||
- Secrets never touch the filesystem in container (env-var injection)
|
||||
- Rotation procedure documented and tested
|
||||
- Clear migration path to Vault for enterprise use
|
||||
|
||||
### Negative
|
||||
- Environment variables are visible to all processes in the container
|
||||
- Docker inspect can reveal env vars if host is compromised
|
||||
- No automatic rotation — manual procedure required
|
||||
|
||||
### Mitigation
|
||||
- Use Docker secrets or Kubernetes secrets (mounted as files) to avoid env-var exposure
|
||||
- Regularly audit secret access patterns
|
||||
- Rotate keys on any suspected compromise
|
||||
|
||||
## Future Roadmap
|
||||
|
||||
1. **Phase 2**: Migrate to HashiCorp Vault for dynamic secrets and automatic rotation
|
||||
2. **Phase 3**: Implement per-user encryption key derivation (separate Fernet keys per user)
|
||||
3. **Phase 4**: Add HSM support for root key protection
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-002 for Fernet encryption implementation
|
||||
- See ADR-007 for JWT token management
|
||||
|
||||
## References
|
||||
|
||||
- [12-Factor App: Config](https://12factor.net/config)
|
||||
- [HashiCorp Vault](https://www.vaultproject.io/)
|
||||
- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||||
- [OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html)
|
||||
@@ -0,0 +1,150 @@
|
||||
# ADR 007: Use JWT for API Authentication
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-08
|
||||
**Deciders:** Security Team, Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The multi-tenant SaaS API needs a stateless authentication mechanism that:
|
||||
|
||||
1. Works with both browser-based frontends and programmatic API clients
|
||||
2. Supports short-lived access tokens to limit exposure on compromise
|
||||
3. Allows session refresh without re-entering credentials
|
||||
4. Integrates with Google OAuth2 (users who sign in via Google)
|
||||
5. Requires no server-side session store
|
||||
|
||||
## Decision
|
||||
|
||||
We will use **JWT (JSON Web Tokens)** with **HMAC-SHA256 (HS256)** signing for both access tokens and refresh tokens, implemented via the `python-jose` library.
|
||||
|
||||
- **Access tokens**: 30-minute TTL, sent in `Authorization: Bearer <token>` header
|
||||
- **Refresh tokens**: 7-day TTL, exchanged for a new access token
|
||||
- **Token subject**: User ID (stored as `str` in `sub` claim, decoded to `int` in deps)
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Session Cookies (server-side sessions)
|
||||
- **Pros**: Easy revocation, browser-native, CSRF protectable with SameSite
|
||||
- **Cons**: Requires server-side session store (Redis), doesn't work well for API-first architecture, harder to use from mobile/CLI clients
|
||||
|
||||
### 2. OAuth2 Opaque Tokens
|
||||
- **Pros**: Easy revocation, no token content leakage
|
||||
- **Cons**: Every request requires a database lookup to validate the token (not stateless)
|
||||
|
||||
### 3. JWT with RS256 (RSA asymmetric signing)
|
||||
- **Pros**: Public key verification allows third-party validation without sharing secret
|
||||
- **Cons**: Additional key management complexity, slower to sign/verify, not required at current scale
|
||||
|
||||
### 4. API Keys (long-lived static tokens)
|
||||
- **Pros**: Simple for machine-to-machine, easy to understand
|
||||
- **Cons**: Long-lived tokens increase risk on compromise, no user-session semantics
|
||||
|
||||
### 5. Paseto (Platform-Agnostic Security Tokens)
|
||||
- **Pros**: Better defaults than JWT (no algorithm confusion attacks), cleaner spec
|
||||
- **Cons**: Less widespread adoption, fewer library options in Python, migration cost from JWT
|
||||
|
||||
## Rationale
|
||||
|
||||
JWT with HS256 was chosen because:
|
||||
|
||||
1. **Stateless**: No database lookup required to validate a token — the signature is self-authenticating
|
||||
2. **Short TTL**: 30-minute access tokens limit the window of opportunity if a token is stolen
|
||||
3. **Refresh Token Pattern**: 7-day refresh tokens allow long sessions without exposing long-lived access tokens
|
||||
4. **Standard**: JWT is the de-facto standard for REST API authentication
|
||||
5. **OAuth2 Compatibility**: Google OAuth2 tokens can be exchanged for our own JWTs, giving unified auth handling
|
||||
6. **`python-jose`**: Well-maintained library with HS256/RS256 support and JWKS endpoint capability
|
||||
|
||||
## Implementation
|
||||
|
||||
```python
|
||||
# backend/app/core/security.py
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from jose import jwt, JWTError
|
||||
|
||||
ACCESS_TOKEN_EXPIRE_MINUTES = 30
|
||||
REFRESH_TOKEN_EXPIRE_DAYS = 7
|
||||
|
||||
def create_access_token(subject: str) -> str:
|
||||
expire = datetime.now(timezone.utc) + timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
|
||||
payload = {"sub": subject, "exp": expire, "type": "access"}
|
||||
return jwt.encode(payload, settings.SECRET_KEY, algorithm="HS256")
|
||||
|
||||
def create_refresh_token(subject: str) -> str:
|
||||
expire = datetime.now(timezone.utc) + timedelta(days=REFRESH_TOKEN_EXPIRE_DAYS)
|
||||
payload = {"sub": subject, "exp": expire, "type": "refresh"}
|
||||
return jwt.encode(payload, settings.SECRET_KEY, algorithm="HS256")
|
||||
|
||||
def decode_token(token: str) -> dict:
|
||||
return jwt.decode(token, settings.SECRET_KEY, algorithms=["HS256"])
|
||||
```
|
||||
|
||||
```python
|
||||
# backend/app/core/deps.py
|
||||
from fastapi import Depends, HTTPException, status
|
||||
from fastapi.security import OAuth2PasswordBearer
|
||||
|
||||
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="/api/v1/auth/login")
|
||||
|
||||
async def get_current_user(
|
||||
token: str = Depends(oauth2_scheme),
|
||||
db: AsyncSession = Depends(get_db),
|
||||
) -> User:
|
||||
try:
|
||||
payload = decode_token(token)
|
||||
user_id = int(payload["sub"]) # sub is str (jose requirement), decode to int
|
||||
except (JWTError, KeyError, ValueError):
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)
|
||||
user = await db.get(User, user_id)
|
||||
if user is None:
|
||||
raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED)
|
||||
return user
|
||||
```
|
||||
|
||||
### Token Subject Encoding
|
||||
|
||||
The `sub` claim uses `str(user.id)` when creating tokens and `int(payload["sub"])` when decoding. This is required because `python-jose` mandates string subjects per the JWT spec.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- No database query per request for authentication (stateless validation)
|
||||
- Short access token TTL limits blast radius of token theft
|
||||
- Works identically for browser, mobile, and CLI clients
|
||||
- Google OAuth flow exchanges Google's token for our own JWT (unified handling)
|
||||
|
||||
### Negative
|
||||
- No instant token revocation (must wait for TTL to expire)
|
||||
- Refresh token theft allows extended session hijacking
|
||||
- `SECRET_KEY` compromise invalidates all tokens and requires rotation
|
||||
|
||||
### Token Revocation Strategy
|
||||
|
||||
For logout and forced revocation, a **token blocklist** stored in Redis can be used:
|
||||
```python
|
||||
# Add jti (JWT ID) claim to tokens
|
||||
# On logout: store jti in Redis with TTL = token TTL
|
||||
# On each request: check if jti is blocklisted
|
||||
```
|
||||
|
||||
This is planned but not yet implemented; current logout deletes the token client-side only.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **HTTPS Only**: JWTs must only be transmitted over TLS in production
|
||||
2. **No Sensitive Data in Payload**: JWT payload is base64-encoded, not encrypted — never put passwords or PII in claims
|
||||
3. **Algorithm Pinning**: Always specify `algorithms=["HS256"]` in `jwt.decode()` to prevent algorithm confusion attacks
|
||||
4. **Secret Key Length**: `SECRET_KEY` must be ≥ 32 characters (validated at startup)
|
||||
5. **Token Storage**: Frontend stores tokens in `localStorage` (XSS risk); consider `httpOnly` cookies for hardened deployments
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-006 for SECRET_KEY management
|
||||
- See ADR-003 for FastAPI dependency injection (`Depends(get_current_user)`)
|
||||
|
||||
## References
|
||||
|
||||
- [RFC 7519: JSON Web Token](https://www.rfc-editor.org/rfc/rfc7519)
|
||||
- [python-jose Documentation](https://python-jose.readthedocs.io/)
|
||||
- [JWT Best Practices (RFC 8725)](https://www.rfc-editor.org/rfc/rfc8725)
|
||||
- [OWASP Authentication Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html)
|
||||
@@ -0,0 +1,145 @@
|
||||
# ADR 008: Use Next.js for Frontend
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-10
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The SaaS platform requires a web frontend that provides:
|
||||
|
||||
1. User authentication (email/password + Google OAuth)
|
||||
2. Mail account management dashboard (CRUD operations)
|
||||
3. Processing run history and statistics
|
||||
4. Settings and subscription management
|
||||
5. Responsive design for mobile and desktop
|
||||
|
||||
Technical requirements:
|
||||
- Type-safe API integration with the FastAPI backend
|
||||
- Server-side rendering (SSR) for SEO and initial load performance
|
||||
- Static export capability for CDN hosting
|
||||
- Easy Docker containerization
|
||||
- Modern development tooling (TypeScript, ESLint, hot-reload)
|
||||
|
||||
## Decision
|
||||
|
||||
We will use **Next.js 14+** (App Router) with **TypeScript** and **Tailwind CSS** for the frontend.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. React (Create React App / Vite SPA)
|
||||
- **Pros**: Simpler build setup, no SSR complexity, large ecosystem
|
||||
- **Cons**: Client-side rendering only (SEO limitations), no built-in routing, additional setup for SSR
|
||||
|
||||
### 2. Vue.js / Nuxt.js
|
||||
- **Pros**: Excellent developer experience, reactive by default, SSR via Nuxt
|
||||
- **Cons**: Smaller ecosystem than React, team less familiar, different component model
|
||||
|
||||
### 3. Svelte / SvelteKit
|
||||
- **Pros**: Very small bundle size, simple reactive model, SSR via SvelteKit
|
||||
- **Cons**: Smaller community, fewer UI component libraries, less mature ecosystem
|
||||
|
||||
### 4. Angular
|
||||
- **Pros**: Full framework (routing, forms, HTTP), TypeScript-first
|
||||
- **Cons**: Heavy boilerplate, steep learning curve, over-engineered for a dashboard application
|
||||
|
||||
### 5. Server-Side Templates (Jinja2 / FastAPI with Jinja)
|
||||
- **Pros**: Simplest deployment (single backend), no separate frontend build
|
||||
- **Cons**: No reactive UI, poor UX for dynamic dashboards, hard to test independently
|
||||
|
||||
## Rationale
|
||||
|
||||
Next.js was chosen because:
|
||||
|
||||
1. **App Router**: File-system based routing with layout nesting simplifies page organization
|
||||
2. **TypeScript First**: Strong typing catches API integration errors at compile time
|
||||
3. **API Route Proxy**: Next.js Route Handlers allow proxying backend requests at runtime — eliminating CORS issues and enabling `BACKEND_URL` to be set at container runtime (not build time)
|
||||
4. **SSR + Static**: Supports both server-rendered pages (authenticated dashboard) and static pages (marketing landing page)
|
||||
5. **Tailwind CSS**: Utility-first CSS eliminates the need for a separate CSS framework and enables rapid UI development
|
||||
6. **Large Ecosystem**: Extensive component libraries, excellent documentation, widely used
|
||||
7. **Docker Friendly**: `next start` serves the production build, easily containerized
|
||||
|
||||
## Implementation Architecture
|
||||
|
||||
### Frontend Proxy Pattern
|
||||
|
||||
The frontend proxies all API calls through a Next.js Route Handler to avoid CORS and build-time URL issues:
|
||||
|
||||
```
|
||||
Browser → Next.js Server (/api/v1/*) → FastAPI Backend (BACKEND_URL/api/v1/*)
|
||||
```
|
||||
|
||||
```typescript
|
||||
// src/app/api/v1/[...path]/route.ts
|
||||
const BACKEND_URL = process.env.BACKEND_URL ?? "http://localhost:8000";
|
||||
|
||||
export async function GET(req: NextRequest, { params }: { params: { path: string[] } }) {
|
||||
const path = params.path.join("/");
|
||||
const url = `${BACKEND_URL}/api/v1/${path}${req.nextUrl.search}`;
|
||||
return fetch(url, { headers: req.headers });
|
||||
}
|
||||
```
|
||||
|
||||
This means `BACKEND_URL` is a **runtime** environment variable (set in Docker Compose / Kubernetes), not a build-time variable. The Axios client uses a relative base URL:
|
||||
|
||||
```typescript
|
||||
// src/lib/api.ts
|
||||
const api = axios.create({ baseURL: "/api/v1" });
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
frontend/src/
|
||||
├── app/ # Next.js App Router pages
|
||||
│ ├── page.tsx # Landing page
|
||||
│ ├── login/ # Authentication pages
|
||||
│ ├── dashboard/ # Main dashboard
|
||||
│ ├── accounts/ # Mail account management
|
||||
│ ├── settings/ # User settings
|
||||
│ └── api/v1/[...path]/ # Backend proxy Route Handler
|
||||
├── components/ # Reusable UI components
|
||||
│ ├── AddMailAccountModal.tsx
|
||||
│ ├── DashboardLayout.tsx
|
||||
│ └── AuthGuard.tsx
|
||||
├── lib/
|
||||
│ └── api.ts # Typed Axios API client
|
||||
├── store/
|
||||
│ └── authStore.ts # Zustand authentication state
|
||||
└── instrumentation.ts # Server startup hook (logs BACKEND_URL)
|
||||
```
|
||||
|
||||
### Authentication State
|
||||
|
||||
Authentication state is managed via **Zustand** store, with JWT tokens persisted in `localStorage`. The `AuthGuard` component protects all dashboard routes.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- TypeScript catches API contract mismatches at compile time
|
||||
- Proxy pattern eliminates CORS configuration and build-time URL embedding
|
||||
- App Router layouts reduce boilerplate for authenticated vs public pages
|
||||
- Tailwind CSS enables rapid UI iteration without writing custom CSS
|
||||
- Server instrumentation hook logs `BACKEND_URL` at startup for easy debugging
|
||||
|
||||
### Negative
|
||||
- Next.js adds complexity vs a plain React SPA (SSR concepts, server/client component boundary)
|
||||
- Node.js 20+ required at runtime (not just build time)
|
||||
- Separate Docker container needed (backend and frontend are distinct services)
|
||||
- `localStorage` token storage is vulnerable to XSS (future: migrate to `httpOnly` cookies)
|
||||
|
||||
### Neutral
|
||||
- ESLint with `eslint-config-next` enforces Next.js-specific rules
|
||||
- `npm ci` used in CI to ensure deterministic installs from `package-lock.json`
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-007 for JWT tokens consumed by the frontend
|
||||
- See ADR-003 for the FastAPI backend the frontend proxies to
|
||||
|
||||
## References
|
||||
|
||||
- [Next.js Documentation](https://nextjs.org/docs)
|
||||
- [Next.js App Router](https://nextjs.org/docs/app)
|
||||
- [Tailwind CSS](https://tailwindcss.com/)
|
||||
- [Zustand State Management](https://github.com/pmndrs/zustand)
|
||||
@@ -0,0 +1,146 @@
|
||||
# ADR 009: Use Gmail API for Email Delivery
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-15
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The core function of the application is forwarding emails fetched from POP3/IMAP accounts into a user's Gmail inbox. There are two primary mechanisms for delivering an email into Gmail:
|
||||
|
||||
1. **SMTP**: Send the email as a new message via Gmail's SMTP server (smtp.gmail.com)
|
||||
2. **Gmail API `messages.insert`**: Inject the raw RFC 2822 email directly into the inbox using the Gmail REST API
|
||||
|
||||
Both require user authentication, but they have significantly different characteristics in terms of header preservation, quota, and delivery behavior.
|
||||
|
||||
## Decision
|
||||
|
||||
We will support **both delivery methods**, with **Gmail API injection as the preferred (default) method** and **SMTP as the fallback**. Each mail account stores a `delivery_method` field (`gmail_api` or `smtp`) chosen at account creation. If `gmail_api` is selected but no valid `GmailCredential` exists, the worker falls back to SMTP automatically.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. SMTP Only
|
||||
- **Pros**: Simple, no OAuth2 setup required, works with any email provider
|
||||
- **Cons**: Counts against sending quota (500/day free), adds forwarding headers (`Received`, `X-Forwarded-To`), may be flagged as spam, rewrites `From` header
|
||||
|
||||
### 2. Gmail API Only
|
||||
- **Pros**: Perfect header preservation, no sending quota, lower spam risk
|
||||
- **Cons**: Requires OAuth2 setup per user, more complex credential management, tied to Gmail
|
||||
|
||||
### 3. IMAP APPEND
|
||||
- **Pros**: Direct inbox write via standard protocol
|
||||
- **Cons**: Requires IMAP access to the destination Gmail account, separate credential management, less reliable than API
|
||||
|
||||
### 4. PubSub Push (Gmail Push Notifications)
|
||||
- **Pros**: Real-time delivery notifications
|
||||
- **Cons**: Unrelated to delivery mechanism, solves a different problem
|
||||
|
||||
## Rationale
|
||||
|
||||
The dual-method approach was chosen because:
|
||||
|
||||
1. **New users benefit from Gmail API**: Perfect header preservation means the email appears exactly as it was sent; no spam risk from forwarding
|
||||
2. **Legacy users supported via SMTP**: Existing setups using App Passwords continue to work without migration
|
||||
3. **User Choice**: Different users have different technical comfort levels with OAuth2 setup
|
||||
4. **Quota Protection**: Gmail API `messages.insert` does not count against the 500/day sending quota
|
||||
5. **Graceful Degradation**: Automatic fallback to SMTP prevents complete failures when Gmail credentials expire
|
||||
|
||||
## Implementation
|
||||
|
||||
### GmailService
|
||||
|
||||
```python
|
||||
# backend/app/services/gmail_service.py
|
||||
from googleapiclient.discovery import build
|
||||
from google.oauth2.credentials import Credentials
|
||||
|
||||
class GmailService:
|
||||
def inject_email(self, raw_email: bytes, user_id: int) -> dict:
|
||||
"""Inject a raw RFC 2822 email directly into Gmail inbox."""
|
||||
creds = self._get_credentials(user_id)
|
||||
service = build("gmail", "v1", credentials=creds)
|
||||
message = {"raw": base64.urlsafe_b64encode(raw_email).decode()}
|
||||
return service.users().messages().insert(
|
||||
userId="me",
|
||||
body=message,
|
||||
).execute()
|
||||
```
|
||||
|
||||
### Delivery Decision in Worker
|
||||
|
||||
```python
|
||||
# backend/app/workers/tasks.py
|
||||
async def _deliver_email(account: MailAccount, raw_email: bytes, db: AsyncSession):
|
||||
if account.delivery_method == "gmail_api":
|
||||
credential = await db.get(GmailCredential, account.user_id)
|
||||
if credential and not credential.is_expired():
|
||||
gmail_service = GmailService(credential)
|
||||
return gmail_service.inject_email(raw_email, account.user_id)
|
||||
# Fall through to SMTP if no valid credential
|
||||
await _deliver_via_smtp(account, raw_email)
|
||||
```
|
||||
|
||||
### Per-Account Delivery Method
|
||||
|
||||
| Field | Type | Values | Default |
|
||||
|-------|------|--------|---------|
|
||||
| `delivery_method` | string | `gmail_api`, `smtp` | `gmail_api` |
|
||||
|
||||
### Gmail OAuth2 Credential Storage
|
||||
|
||||
Gmail credentials are stored in the `gmail_credentials` table:
|
||||
- `access_token`: Short-lived OAuth2 access token
|
||||
- `refresh_token`: Long-lived token for refreshing access
|
||||
- `token_expiry`: Expiration timestamp
|
||||
- Tokens are encrypted at rest using Fernet (see ADR-002)
|
||||
|
||||
## Comparison Table
|
||||
|
||||
| Feature | Gmail API (`gmail_api`) | SMTP Forwarding (`smtp`) |
|
||||
|---------|------------------------|--------------------------|
|
||||
| Header preservation | ✅ All original headers intact | ⚠️ Adds `Received`, may rewrite `From` |
|
||||
| Gmail sending quota | ✅ Does not consume quota | ❌ Counts against 500/day |
|
||||
| Authentication | OAuth2 tokens per user | App Password (shared per server) |
|
||||
| Setup complexity | OAuth2 consent flow required | App Password only |
|
||||
| Spam risk | ✅ Low (email appears native) | ⚠️ Higher (forwarded mail may be flagged) |
|
||||
| Fallback behavior | Falls back to SMTP automatically | Primary legacy method |
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Gmail API delivery produces the cleanest inbox experience for users
|
||||
- No sending quota concerns for high-volume users
|
||||
- SMTP fallback ensures continued operation when OAuth tokens expire
|
||||
- Per-account delivery method supports heterogeneous user setups
|
||||
|
||||
### Negative
|
||||
- OAuth2 setup adds friction for new Gmail API users
|
||||
- Credential refresh logic must be implemented and maintained
|
||||
- Gmail API credentials must be encrypted at rest (additional complexity)
|
||||
- Two code paths to test and maintain
|
||||
|
||||
### Token Refresh Strategy
|
||||
|
||||
Gmail access tokens expire after 1 hour. The worker automatically refreshes tokens using the stored `refresh_token` before delivery:
|
||||
|
||||
```python
|
||||
from google.auth.transport.requests import Request
|
||||
|
||||
if creds.expired and creds.refresh_token:
|
||||
creds.refresh(Request())
|
||||
# Persist refreshed tokens back to database
|
||||
await _update_gmail_credential(user_id, creds, db)
|
||||
```
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-001 for Celery worker that executes delivery
|
||||
- See ADR-002 for Fernet encryption of Gmail credentials
|
||||
- See ADR-005 for retry strategy when delivery fails
|
||||
|
||||
## References
|
||||
|
||||
- [Gmail API: messages.insert](https://developers.google.com/gmail/api/reference/rest/v1/users.messages/insert)
|
||||
- [Google OAuth2 for Web Apps](https://developers.google.com/identity/protocols/oauth2/web-server)
|
||||
- [Gmail Sending Limits](https://support.google.com/mail/answer/22839)
|
||||
- [RFC 2822: Internet Message Format](https://www.rfc-editor.org/rfc/rfc2822)
|
||||
@@ -0,0 +1,186 @@
|
||||
# ADR 010: Hybrid Configuration Model (Database + Environment Variables)
|
||||
|
||||
**Status:** Accepted
|
||||
**Date:** 2026-02-20
|
||||
**Deciders:** Development Team
|
||||
|
||||
## Context
|
||||
|
||||
The application requires configuration for many operational parameters:
|
||||
|
||||
1. **Bootstrap secrets** that must be available before the database is ready: `DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`
|
||||
2. **SMTP relay settings** that operators want to change without redeployment: host, port, username, password
|
||||
3. **Processing tuning parameters** that may need runtime adjustment: check interval, max emails per run, throttle rate
|
||||
4. **Feature flags** for Gmail API, notifications, subscriptions
|
||||
5. **Tier limits** (max accounts per subscription tier)
|
||||
|
||||
A pure environment-variable approach requires container redeployment for every config change. A pure database approach creates a chicken-and-egg problem for bootstrap settings.
|
||||
|
||||
## Decision
|
||||
|
||||
We will use a **hybrid configuration model** with the following priority chain:
|
||||
|
||||
```
|
||||
1. Database (app_settings table) ← highest priority
|
||||
2. Environment variable / .env file
|
||||
3. Built-in default ← lowest priority
|
||||
```
|
||||
|
||||
**Bootstrap settings** (`DATABASE_URL`, `SECRET_KEY`, `ENCRYPTION_KEY`) are **always** resolved from environment variables only, because the database connection depends on them.
|
||||
|
||||
**All other settings** are resolved by `ConfigService`, which checks the database first, then falls back to environment/defaults.
|
||||
|
||||
## Alternatives Considered
|
||||
|
||||
### 1. Environment Variables Only
|
||||
- **Pros**: Simple, 12-factor compliant, works everywhere
|
||||
- **Cons**: Config changes require redeployment, hard to manage across many instances, no UI for non-technical operators
|
||||
|
||||
### 2. Database-Only Configuration
|
||||
- **Pros**: Runtime changes, admin UI possible, audit trail
|
||||
- **Cons**: Chicken-and-egg for `DATABASE_URL` itself; database must exist before the app can read its own connection string
|
||||
|
||||
### 3. External Config Service (Consul, etcd)
|
||||
- **Pros**: Distributed config, live reload, service discovery
|
||||
- **Cons**: Additional infrastructure to deploy and operate, overkill for initial deployment
|
||||
|
||||
### 4. TOML / YAML Config Files
|
||||
- **Pros**: Version-controlled, human-readable
|
||||
- **Cons**: Requires file system mounts in Docker, config changes need file edits + possible restart
|
||||
|
||||
### 5. Pydantic Settings Only (env + .env files)
|
||||
- **Pros**: Type-safe, validated at startup, no database dependency
|
||||
- **Cons**: No runtime mutation, no admin UI, requires env changes for any tuning
|
||||
|
||||
## Rationale
|
||||
|
||||
The hybrid model was chosen because:
|
||||
|
||||
1. **Bootstrap Problem Solved**: Database connection string and secrets are always env-only — no circular dependency
|
||||
2. **Runtime Mutability**: SMTP settings, tier limits, and feature flags can be changed via admin API without redeployment
|
||||
3. **Operator Friendly**: Non-technical operators can use the admin UI or API to adjust settings
|
||||
4. **Developer Friendly**: Developers can use `.env` files for local overrides without touching the database
|
||||
5. **Gradual Migration**: Existing env-based deployments continue to work; database settings are additive
|
||||
|
||||
## Implementation
|
||||
|
||||
### Data Model
|
||||
|
||||
```python
|
||||
# backend/app/models/database_models.py
|
||||
class AppSetting(Base):
|
||||
__tablename__ = "app_settings"
|
||||
|
||||
key: Mapped[str] = mapped_column(String(255), primary_key=True)
|
||||
value: Mapped[str] = mapped_column(Text, nullable=False)
|
||||
description: Mapped[str | None] = mapped_column(Text)
|
||||
updated_at: Mapped[datetime] = mapped_column(
|
||||
DateTime(timezone=True),
|
||||
default=lambda: datetime.now(timezone.utc),
|
||||
onupdate=lambda: datetime.now(timezone.utc),
|
||||
)
|
||||
```
|
||||
|
||||
### ConfigService
|
||||
|
||||
```python
|
||||
# backend/app/services/config_service.py
|
||||
class ConfigService:
|
||||
def __init__(self, db: AsyncSession):
|
||||
self.db = db
|
||||
|
||||
async def get(self, key: str, default: str | None = None) -> str | None:
|
||||
"""Resolve setting: DB → env → default."""
|
||||
# 1. Check database
|
||||
result = await self.db.get(AppSetting, key)
|
||||
if result is not None:
|
||||
return result.value
|
||||
# 2. Check environment
|
||||
env_value = os.getenv(key)
|
||||
if env_value is not None:
|
||||
return env_value
|
||||
# 3. Return default
|
||||
return default
|
||||
|
||||
async def set(self, key: str, value: str, description: str | None = None) -> None:
|
||||
"""Persist setting to database (creates or updates)."""
|
||||
setting = AppSetting(key=key, value=value, description=description)
|
||||
await self.db.merge(setting)
|
||||
await self.db.commit()
|
||||
```
|
||||
|
||||
### Default Seeding at Startup
|
||||
|
||||
On first boot, sensible defaults are written to the `app_settings` table if they don't already exist:
|
||||
|
||||
```python
|
||||
# Called during lifespan startup
|
||||
async def seed_default_settings(db: AsyncSession):
|
||||
defaults = {
|
||||
"CHECK_INTERVAL_MINUTES": ("5", "How often to check mail accounts"),
|
||||
"MAX_EMAILS_PER_RUN": ("50", "Maximum emails to process per run"),
|
||||
"SMTP_PORT": ("587", "SMTP relay port"),
|
||||
"TIER_FREE_MAX_ACCOUNTS": ("1", "Free tier account limit"),
|
||||
# ... etc.
|
||||
}
|
||||
for key, (value, description) in defaults.items():
|
||||
existing = await db.get(AppSetting, key)
|
||||
if existing is None:
|
||||
await db.add(AppSetting(key=key, value=value, description=description))
|
||||
await db.commit()
|
||||
```
|
||||
|
||||
### Admin API Endpoints
|
||||
|
||||
```
|
||||
GET /api/v1/settings # List all database settings (admin only)
|
||||
PUT /api/v1/settings/{key} # Create or update a setting
|
||||
DELETE /api/v1/settings/{key} # Delete a setting (falls back to env/default)
|
||||
POST /api/v1/settings/seed-defaults # Re-seed all defaults
|
||||
```
|
||||
|
||||
### Bootstrap Settings (Env-Only)
|
||||
|
||||
These are read via `pydantic-settings` (`BaseSettings`) and never looked up in the database:
|
||||
|
||||
| Setting | Required | Purpose |
|
||||
|---------|----------|---------|
|
||||
| `DATABASE_URL` | ✅ | PostgreSQL connection string |
|
||||
| `SECRET_KEY` | ✅ | JWT signing key (≥32 chars) |
|
||||
| `ENCRYPTION_KEY` | ✅ | Fernet master encryption key |
|
||||
| `GOOGLE_CLIENT_ID` | Optional | Google OAuth2 |
|
||||
| `GOOGLE_CLIENT_SECRET` | Optional | Google OAuth2 |
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
- Bootstrap settings always resolved from env (no circular dependency)
|
||||
- Runtime-mutable settings require no redeployment
|
||||
- Default values seeded at startup so application works out-of-the-box
|
||||
- Clear priority chain: database overrides env, env overrides code default
|
||||
- Admin API provides complete CRUD on runtime settings
|
||||
|
||||
### Negative
|
||||
- Two sources of truth for settings requires careful documentation
|
||||
- Database settings take precedence over env — operators must know to check the database if env changes seem to have no effect
|
||||
- `ConfigService` requires a database session (async), adding overhead for high-frequency config reads
|
||||
- Settings cache is not implemented; every lookup hits the database (future improvement: TTL-based cache)
|
||||
|
||||
### Future Improvements
|
||||
|
||||
1. **In-Memory Cache**: Cache settings with a short TTL (e.g., 30 seconds) to reduce database load
|
||||
2. **Change Notifications**: Pub/sub via Redis to notify workers of setting changes
|
||||
3. **Typed Settings Schema**: JSON Schema validation for setting values
|
||||
4. **Audit Trail**: Record who changed each setting and when
|
||||
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-004 for PostgreSQL (stores `app_settings` table)
|
||||
- See ADR-006 for key management (bootstrap secrets are env-only)
|
||||
- See ADR-003 for FastAPI dependency injection of `ConfigService`
|
||||
|
||||
## References
|
||||
|
||||
- [12-Factor App: Config](https://12factor.net/config)
|
||||
- [Pydantic Settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/)
|
||||
- [SQLAlchemy ORM](https://docs.sqlalchemy.org/en/20/orm/)
|
||||
Reference in New Issue
Block a user