gitea-admin/gh-christianlouis-inboxconverge
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Add security hardening, agentic coding infrastructure, and test framework

Browse Source 
- Add SECRET_KEY and ENCRYPTION_KEY validation on startup
- Implement security headers middleware (X-Frame-Options, CSP, HSTS)
- Add CSRF protection middleware
- Create comprehensive GitHub issue templates and PR template
- Add Makefile with common development tasks
- Configure pre-commit hooks (black, ruff, mypy, bandit, detect-secrets)
- Create docs/CODING_PATTERNS.md with best practices
- Create docs/ERRORS.md documenting all error codes
- Add Architecture Decision Records (ADR) for Celery and Fernet encryption
- Create CHANGELOG.md for version tracking
- Set up pytest test infrastructure with fixtures and factories
- Add sample unit tests for security and config validation
- Create CI/CD workflows (test, lint, security)
- Add comprehensive TODO.md with milestones and progress tracking

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-02-06 22:01:11 +00:00
parent 24e7d161d7
commit e64f0f2705
24 changed files with 2910 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/CODING_PATTERNS.md
+583
View File
@@ -0,0 +1,583 @@
# Coding Patterns and Best Practices
This document outlines the coding patterns, conventions, and best practices for the POP3 to Gmail Forwarder project.
## Table of Contents
- [General Principles](#general-principles)
- [Python Style](#python-style)
- [API Development](#api-development)
- [Database Patterns](#database-patterns)
- [Error Handling](#error-handling)
- [Security Patterns](#security-patterns)
- [Testing Patterns](#testing-patterns)
- [Async/Await Patterns](#asyncawait-patterns)
---
## General Principles
### 1. Explicit is Better Than Implicit
```python
# Good ✅
async def get_user_by_id(db: AsyncSession, user_id: int) -> Optional[User]:
result = await db.execute(select(User).where(User.id == user_id))
return result.scalar_one_or_none()
# Bad ❌
async def get_user(db, id): # Missing type hints
return await db.execute(select(User).where(User.id == id)).scalar() # Chained calls
```
### 2. Dependency Injection
Use FastAPI's dependency injection for shared resources:
```python
# Good ✅
async def create_mail_account(
account_in: MailAccountCreate,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user)
) -> MailAccount:
# Function implementation
pass
# Bad ❌
# Accessing global db connection or parsing tokens manually
```
---
## Python Style
### Type Hints (Required)
```python
# Good ✅
from typing import Optional, List
from datetime import datetime
def process_emails(
account_id: int,
max_count: int = 50,
since: Optional[datetime] = None
) -> List[Email]:
pass
# Bad ❌
def process_emails(account_id, max_count=50, since=None): # No type hints
pass
```
### Docstrings (Required for Public APIs)
```python
# Good ✅
async def fetch_emails_from_pop3(account: MailAccount) -> List[Email]:
"""
Fetch emails from a POP3 account.
Args:
account: The mail account to fetch from
Returns:
List of Email objects retrieved from the server
Raises:
ConnectionError: If POP3 connection fails
AuthenticationError: If credentials are invalid
"""
pass
```
### Constants
```python
# Good ✅ - In backend/app/core/constants.py
MAX_EMAILS_PER_RUN = 50
DEFAULT_CHECK_INTERVAL_MINUTES = 5
PBKDF2_ITERATIONS = 100_000
# Bad ❌ - Magic numbers in code
if len(emails) > 50:
pass
```
---
## API Development
### Endpoint Structure
```python
# Good ✅
from fastapi import APIRouter, Depends, HTTPException, status
from app.models.schemas import MailAccountCreate, MailAccountResponse
from app.core.deps import get_current_user, get_db
router = APIRouter(prefix="/mail-accounts", tags=["Mail Accounts"])
@router.post(
"/",
response_model=MailAccountResponse,
status_code=status.HTTP_201_CREATED,
summary="Create a new mail account"
)
async def create_mail_account(
account_in: MailAccountCreate,
db: AsyncSession = Depends(get_db),
current_user: User = Depends(get_current_user)
) -> MailAccount:
"""Create a new POP3/IMAP mail account for the current user."""
# Validate subscription limits
# Create account with encrypted credentials
# Return response
pass
```
### Error Responses
```python
# Good ✅
from app.core.errors import ErrorCode, ErrorResponse
if not can_add_account:
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=ErrorResponse(
code=ErrorCode.SUBSCRIPTION_LIMIT_REACHED,
message="Your plan allows maximum 5 mail accounts",
details={"current": 5, "limit": 5, "upgrade_url": "/pricing"}
).dict()
)
# Bad ❌
if not can_add_account:
raise HTTPException(status_code=403, detail="Limit reached") # Not helpful
```
### Validation
```python
# Good ✅ - Use Pydantic validators
from pydantic import BaseModel, validator
class MailAccountCreate(BaseModel):
host: str
port: int
username: str
password: str
@validator('port')
def validate_port(cls, v):
if not 1 <= v <= 65535:
raise ValueError('Port must be between 1 and 65535')
return v
@validator('host')
def validate_host(cls, v):
if not v or v.strip() == "":
raise ValueError('Host cannot be empty')
return v.strip()
```
---
## Database Patterns
### Queries
```python
# Good ✅ - Use SQLAlchemy select statements
from sqlalchemy import select
async def get_user_mail_accounts(db: AsyncSession, user_id: int) -> List[MailAccount]:
result = await db.execute(
select(MailAccount)
.where(MailAccount.user_id == user_id)
.order_by(MailAccount.created_at.desc())
)
return result.scalars().all()
# Bad ❌ - Raw SQL or no await
def get_accounts(db, user_id):
return db.query(MailAccount).filter_by(user_id=user_id).all() # Sync, old style
```
### Transactions
```python
# Good ✅ - Explicit transaction management
async def create_user_with_account(
db: AsyncSession,
user_data: UserCreate,
account_data: MailAccountCreate
) -> User:
try:
user = User(**user_data.dict())
db.add(user)
await db.flush() # Get user.id
account = MailAccount(**account_data.dict(), user_id=user.id)
db.add(account)
await db.commit()
await db.refresh(user)
return user
except Exception as e:
await db.rollback()
raise
# Bad ❌ - No explicit error handling
async def create_user_with_account(db, user_data, account_data):
user = User(**user_data.dict())
db.add(user)
await db.commit() # What if this fails?
```
### Relationships
```python
# Good ✅ - Use eager loading when needed
from sqlalchemy.orm import selectinload
async def get_user_with_accounts(db: AsyncSession, user_id: int) -> Optional[User]:
result = await db.execute(
select(User)
.options(selectinload(User.mail_accounts))
.where(User.id == user_id)
)
return result.scalar_one_or_none()
# Bad ❌ - N+1 queries
user = await get_user(db, user_id)
for account in user.mail_accounts: # Lazy loads each account
print(account.email)
```
---
## Error Handling
### Specific Exceptions
```python
# Good ✅ - Catch specific exceptions
from smtplib import SMTPAuthenticationError, SMTPException
from poplib import error_proto
try:
await send_email(message)
except SMTPAuthenticationError as e:
logger.error(f"SMTP authentication failed: {e}")
raise HTTPException(status_code=401, detail="Invalid email credentials")
except SMTPException as e:
logger.error(f"SMTP error: {e}")
raise HTTPException(status_code=500, detail="Email delivery failed")
# Bad ❌ - Bare except
try:
await send_email(message)
except Exception as e: # Too broad
logger.error(f"Error: {e}")
```
### Logging
```python
# Good ✅ - Structured logging with context
logger.info(
"Email forwarded successfully",
extra={
"user_id": user.id,
"account_id": account.id,
"email_size": len(email_data),
"destination": destination_email
}
)
# Bad ❌ - String formatting in logs
logger.info(f"Email forwarded for user {user.id}") # No structure
```
### Resource Cleanup
```python
# Good ✅ - Use context managers
async with aiosmtplib.SMTP(hostname=smtp_host, port=smtp_port) as smtp:
await smtp.login(username, password)
await smtp.send_message(message)
# Connection automatically closed
# Bad ❌ - Manual cleanup
smtp = aiosmtplib.SMTP(hostname=smtp_host, port=smtp_port)
try:
await smtp.connect()
await smtp.send_message(message)
finally:
smtp.close() # Might be forgotten
```
---
## Security Patterns
### Credential Encryption
```python
# Good ✅ - Always encrypt credentials before storage
from app.core.security import encrypt_password
async def create_mail_account(
db: AsyncSession,
account_data: MailAccountCreate,
user_id: int
) -> MailAccount:
encrypted_password = encrypt_password(account_data.password, user_id)
account = MailAccount(
**account_data.dict(exclude={'password'}),
encrypted_password=encrypted_password,
user_id=user_id
)
db.add(account)
await db.commit()
return account
# Bad ❌ - Plain text storage
account = MailAccount(password=account_data.password) # NEVER DO THIS
```
### Input Validation
```python
# Good ✅ - Validate all external inputs
from urllib.parse import urlparse
@validator('redirect_uri')
def validate_redirect_uri(cls, v):
allowed_domains = ['localhost', 'app.yourdomain.com']
parsed = urlparse(v)
if parsed.netloc not in allowed_domains:
raise ValueError('Invalid redirect URI')
return v
# Bad ❌ - Trust user input
redirect_uri = request.args.get('redirect_uri')
return redirect(redirect_uri) # Open redirect vulnerability
```
### Never Log Secrets
```python
# Good ✅
logger.info(f"Connecting to POP3 server {host} as {username}")
# Bad ❌
logger.debug(f"Connecting with password: {password}") # NEVER LOG PASSWORDS
```
---
## Testing Patterns
### Test Structure
```python
# Good ✅ - Arrange, Act, Assert
import pytest
from httpx import AsyncClient
@pytest.mark.asyncio
async def test_create_mail_account(
client: AsyncClient,
auth_headers: dict,
db_session: AsyncSession
):
# Arrange
account_data = {
"host": "pop.example.com",
"port": 995,
"username": "test@example.com",
"password": "secure_password"
}
# Act
response = await client.post(
"/api/v1/mail-accounts/",
json=account_data,
headers=auth_headers
)
# Assert
assert response.status_code == 201
data = response.json()
assert data["username"] == account_data["username"]
assert "password" not in data # Never return passwords
```
### Fixtures
```python
# Good ✅ - In conftest.py
import pytest
from sqlalchemy.ext.asyncio import AsyncSession
@pytest.fixture
async def test_user(db_session: AsyncSession) -> User:
"""Create a test user."""
user = User(
email="test@example.com",
hashed_password=get_password_hash("testpass")
)
db_session.add(user)
await db_session.commit()
await db_session.refresh(user)
return user
```
### Mocking
```python
# Good ✅ - Mock external services
from unittest.mock import AsyncMock, patch
@pytest.mark.asyncio
async def test_send_email_success():
with patch('aiosmtplib.SMTP') as mock_smtp:
mock_instance = AsyncMock()
mock_smtp.return_value.__aenter__.return_value = mock_instance
await send_email("test@example.com", "Subject", "Body")
mock_instance.send_message.assert_called_once()
```
---
## Async/Await Patterns
### Always Await Async Functions
```python
# Good ✅
result = await db.execute(query)
await db.commit()
# Bad ❌
result = db.execute(query) # Returns coroutine, not result!
```
### Use AsyncSession
```python
# Good ✅ - Backend uses AsyncSession
from sqlalchemy.ext.asyncio import AsyncSession
async def get_user(db: AsyncSession, user_id: int) -> Optional[User]:
result = await db.execute(select(User).where(User.id == user_id))
return result.scalar_one_or_none()
# Bad ❌ - Mixing sync code with async
from sqlalchemy.orm import Session # Wrong import
def get_user(db: Session, user_id: int): # Sync function
return db.query(User).filter_by(id=user_id).first()
```
### Don't Block the Event Loop
```python
# Good ✅ - Use async libraries
import aiofiles
async def read_large_file(filepath: str) -> str:
async with aiofiles.open(filepath, 'r') as f:
return await f.read()
# Bad ❌ - Blocking I/O in async function
async def read_large_file(filepath: str) -> str:
with open(filepath, 'r') as f: # Blocks event loop!
return f.read()
```
---
## Celery Task Patterns
### Task Definition
```python
# Good ✅ - With retry and error handling
from celery import Task
from app.core.celery_app import celery_app
@celery_app.task(
bind=True,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 3, 'countdown': 60},
retry_backoff=True
)
def process_mail_account(self: Task, account_id: int) -> dict:
"""Process emails for a mail account."""
try:
# Task logic
return {"status": "success", "count": 10}
except Exception as exc:
logger.error(f"Task failed for account {account_id}: {exc}")
raise self.retry(exc=exc)
# Bad ❌ - No retry logic
@celery_app.task
def process_mail_account(account_id):
# If this fails, it just fails
pass
```
---
## Configuration Management
### Use Pydantic Settings
```python
# Good ✅ - In app/core/config.py
from pydantic import BaseSettings, validator
class Settings(BaseSettings):
SECRET_KEY: str
DATABASE_URL: str
@validator('SECRET_KEY')
def validate_secret_key(cls, v):
if v == "change-this-to-a-secure-random-secret-key-in-production":
raise ValueError("SECRET_KEY must be changed from default!")
if len(v) < 32:
raise ValueError("SECRET_KEY must be at least 32 characters")
return v
class Config:
env_file = ".env"
# Bad ❌ - Direct os.getenv without validation
SECRET_KEY = os.getenv('SECRET_KEY', 'default_key') # Dangerous default
```
---
## Documentation Patterns
### API Endpoint Documentation
```python
# Good ✅
@router.post(
"/",
response_model=MailAccountResponse,
status_code=status.HTTP_201_CREATED,
summary="Create a new mail account",
description="Creates a new POP3/IMAP mail account for the authenticated user. "
"Credentials are encrypted before storage.",
responses={
201: {"description": "Mail account created successfully"},
403: {"description": "Subscription limit reached"},
422: {"description": "Invalid input data"}
}
)
async def create_mail_account(...):
pass
```
---
## Summary Checklist
Before committing code, ensure:
- [ ] Type hints on all functions
- [ ] Docstrings on public APIs
- [ ] Specific exception handling (no bare `except`)
- [ ] Input validation with Pydantic
- [ ] Credentials encrypted, never logged
- [ ] Async/await used correctly
- [ ] Tests added/updated
- [ ] Error codes documented
- [ ] Security considerations checked
- [ ] Code follows these patterns
---
**Last Updated**: 2026-02-06
**Maintainer**: Development Team
docs/ERRORS.md
+342
View File
@@ -0,0 +1,342 @@
# Error Codes and Messages
This document catalogs all error codes used in the POP3 to Gmail Forwarder application.
## Error Code Format
Error codes follow this pattern: `[DOMAIN]_[NUMBER]`
- **AUTH**: Authentication and authorization errors (001-099)
- **MAIL**: Mail processing errors (100-199)
- **SUB**: Subscription and billing errors (200-299)
- **USER**: User management errors (300-399)
- **NOTIFY**: Notification errors (400-499)
- **SYS**: System and infrastructure errors (500-599)
---
## Authentication & Authorization (AUTH_001-099)
### AUTH_001: Invalid Credentials
- **HTTP Status**: 401 Unauthorized
- **Message**: "Invalid email or password"
- **Cause**: Wrong email/password combination during login
- **Action**: Verify credentials, reset password if needed
### AUTH_002: Token Expired
- **HTTP Status**: 401 Unauthorized
- **Message**: "Authentication token has expired"
- **Cause**: JWT token lifetime exceeded
- **Action**: Refresh token or re-authenticate
### AUTH_003: Token Invalid
- **HTTP Status**: 401 Unauthorized
- **Message**: "Invalid authentication token"
- **Cause**: Malformed or tampered JWT token
- **Action**: Clear tokens and re-authenticate
### AUTH_004: Insufficient Permissions
- **HTTP Status**: 403 Forbidden
- **Message**: "You don't have permission to perform this action"
- **Cause**: User role lacks required permissions
- **Action**: Contact administrator for access
### AUTH_005: Email Already Registered
- **HTTP Status**: 409 Conflict
- **Message**: "An account with this email already exists"
- **Cause**: Registration with existing email
- **Action**: Use different email or login instead
### AUTH_006: OAuth Provider Error
- **HTTP Status**: 502 Bad Gateway
- **Message**: "Failed to authenticate with OAuth provider"
- **Cause**: Google OAuth service unavailable
- **Action**: Retry or use email/password login
### AUTH_007: Invalid OAuth State
- **HTTP Status**: 400 Bad Request
- **Message**: "Invalid OAuth state parameter"
- **Cause**: CSRF token mismatch in OAuth flow
- **Action**: Restart OAuth flow from beginning
### AUTH_008: Email Not Verified
- **HTTP Status**: 403 Forbidden
- **Message**: "Please verify your email address"
- **Cause**: Attempting action before email verification
- **Action**: Check email and click verification link
---
## Mail Processing (MAIL_100-199)
### MAIL_100: Connection Failed
- **HTTP Status**: 502 Bad Gateway
- **Message**: "Failed to connect to POP3/IMAP server"
- **Cause**: Network error, wrong host/port, firewall
- **Action**: Verify host, port, network connectivity
### MAIL_101: Authentication Failed
- **HTTP Status**: 401 Unauthorized
- **Message**: "POP3/IMAP authentication failed"
- **Cause**: Invalid credentials for mail account
- **Action**: Update mail account credentials
### MAIL_102: SSL/TLS Error
- **HTTP Status**: 502 Bad Gateway
- **Message**: "SSL/TLS connection error"
- **Cause**: Certificate issues, SSL not supported
- **Action**: Verify SSL settings, check certificate
### MAIL_103: Mailbox Not Found
- **HTTP Status**: 404 Not Found
- **Message**: "Mailbox or folder not found"
- **Cause**: IMAP folder doesn't exist
- **Action**: Check folder name, create if needed
### MAIL_104: Message Retrieval Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Failed to retrieve email message"
- **Cause**: Corrupt message, server error
- **Action**: Skip message, contact mail provider
### MAIL_105: Forward Failed
- **HTTP Status**: 502 Bad Gateway
- **Message**: "Failed to forward email"
- **Cause**: SMTP error, network issue
- **Action**: Retry, check SMTP settings
### MAIL_106: Rate Limit Exceeded
- **HTTP Status**: 429 Too Many Requests
- **Message**: "Email forwarding rate limit exceeded"
- **Cause**: Too many emails sent too quickly
- **Action**: Wait, upgrade plan, adjust throttling
### MAIL_107: Message Too Large
- **HTTP Status**: 413 Payload Too Large
- **Message**: "Email message exceeds size limit"
- **Cause**: Message larger than allowed size
- **Action**: Filter large messages, upgrade plan
### MAIL_108: Invalid Email Format
- **HTTP Status**: 422 Unprocessable Entity
- **Message**: "Email message format is invalid"
- **Cause**: Malformed email headers or body
- **Action**: Check source email, skip if necessary
### MAIL_109: Encryption Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Failed to encrypt mail credentials"
- **Cause**: Encryption key issue
- **Action**: Check ENCRYPTION_KEY configuration
### MAIL_110: Decryption Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Failed to decrypt mail credentials"
- **Cause**: Wrong encryption key or corrupt data
- **Action**: Re-save credentials with correct key
---
## Subscription & Billing (SUB_200-299)
### SUB_200: Subscription Required
- **HTTP Status**: 402 Payment Required
- **Message**: "This feature requires an active subscription"
- **Cause**: Attempting premium feature without subscription
- **Action**: Subscribe to a plan
### SUB_201: Limit Reached
- **HTTP Status**: 403 Forbidden
- **Message**: "You've reached your plan limit for [resource]"
- **Cause**: Plan limits exceeded (accounts, emails, etc.)
- **Action**: Upgrade plan or remove unused resources
### SUB_202: Payment Failed
- **HTTP Status**: 402 Payment Required
- **Message**: "Payment processing failed"
- **Cause**: Invalid payment method, insufficient funds
- **Action**: Update payment method
### SUB_203: Subscription Expired
- **HTTP Status**: 402 Payment Required
- **Message**: "Your subscription has expired"
- **Cause**: Subscription period ended
- **Action**: Renew subscription
### SUB_204: Invalid Plan
- **HTTP Status**: 404 Not Found
- **Message**: "Subscription plan not found"
- **Cause**: Requesting non-existent plan
- **Action**: Choose valid plan from available options
### SUB_205: Downgrade Not Allowed
- **HTTP Status**: 409 Conflict
- **Message**: "Cannot downgrade: usage exceeds new plan limits"
- **Cause**: Current usage > target plan limits
- **Action**: Reduce usage before downgrading
---
## User Management (USER_300-399)
### USER_300: User Not Found
- **HTTP Status**: 404 Not Found
- **Message**: "User account not found"
- **Cause**: Invalid user ID or deleted account
- **Action**: Verify user ID or create account
### USER_301: Profile Update Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Failed to update user profile"
- **Cause**: Database error, validation failure
- **Action**: Retry, check input data
### USER_302: Password Too Weak
- **HTTP Status**: 422 Unprocessable Entity
- **Message**: "Password does not meet security requirements"
- **Cause**: Password too short or simple
- **Action**: Use stronger password (8+ chars, mixed case, numbers)
### USER_303: Deletion Restricted
- **HTTP Status**: 409 Conflict
- **Message**: "Cannot delete user: active subscription"
- **Cause**: User has active subscription
- **Action**: Cancel subscription first
---
## Notifications (NOTIFY_400-499)
### NOTIFY_400: Notification Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Failed to send notification"
- **Cause**: Notification service error
- **Action**: Check notification service configuration
### NOTIFY_401: Invalid Channel
- **HTTP Status**: 422 Unprocessable Entity
- **Message**: "Invalid notification channel"
- **Cause**: Unsupported notification type
- **Action**: Use supported channel (email, webhook, etc.)
### NOTIFY_402: Channel Not Configured
- **HTTP Status**: 424 Failed Dependency
- **Message**: "Notification channel not configured"
- **Cause**: Required channel settings missing
- **Action**: Configure notification settings
---
## System Errors (SYS_500-599)
### SYS_500: Database Error
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Database operation failed"
- **Cause**: Database connection or query error
- **Action**: Retry, check database status
### SYS_501: Redis Error
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Cache service unavailable"
- **Cause**: Redis connection error
- **Action**: Check Redis service status
### SYS_502: Celery Task Failed
- **HTTP Status**: 500 Internal Server Error
- **Message**: "Background task processing failed"
- **Cause**: Celery worker error
- **Action**: Check worker logs, retry task
### SYS_503: Configuration Error
- **HTTP Status**: 500 Internal Server Error
- **Message**: "System configuration error"
- **Cause**: Invalid or missing configuration
- **Action**: Check environment variables
### SYS_504: External Service Timeout
- **HTTP Status**: 504 Gateway Timeout
- **Message**: "External service request timed out"
- **Cause**: Slow response from external API
- **Action**: Retry, check service status
---
## Usage in Code
### Example: Raising Errors
```python
from fastapi import HTTPException, status
from app.core.errors import ErrorCode, ErrorResponse
# Structured error response
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail=ErrorResponse(
code=ErrorCode.SUB_201,
message="You've reached your plan limit for mail accounts",
details={
"current": 5,
"limit": 5,
"plan": "basic",
"upgrade_url": "/pricing"
}
).dict()
)
```
### Example: Error Response Schema
```python
from pydantic import BaseModel
class ErrorResponse(BaseModel):
code: str # e.g., "MAIL_100"
message: str # Human-readable message
details: dict = {} # Additional context
timestamp: datetime = Field(default_factory=datetime.utcnow)
request_id: str = "" # For tracing
```
### Example: Client Handling
```javascript
// Frontend error handling
try {
const response = await fetch('/api/v1/mail-accounts/', options);
if (!response.ok) {
const error = await response.json();
switch(error.code) {
case 'SUB_201':
showUpgradeModal(error.details);
break;
case 'MAIL_100':
showConnectionErrorDialog(error.message);
break;
default:
showGenericError(error.message);
}
}
} catch (err) {
console.error('Request failed:', err);
}
```
---
## Adding New Error Codes
When adding new error codes:
1. Choose appropriate category (AUTH, MAIL, SUB, USER, NOTIFY, SYS)
2. Assign next available number in that range
3. Document in this file with:
- HTTP status code
- Message template
- Cause
- Recommended action
4. Update `app/core/errors.py` with the code constant
5. Add to API documentation examples
---
**Last Updated**: 2026-02-06
**Maintainer**: Development Team
docs/adr/001-celery-background-tasks.md
+99
View File
@@ -0,0 +1,99 @@
# ADR 001: Use Celery for Background Task Processing
**Status:** Accepted
**Date:** 2026-01-15
**Deciders:** Development Team
## Context
The multi-tenant SaaS version of the POP3 forwarder needs to process emails for multiple users on different schedules. We need a reliable way to:
1. Schedule periodic email checks per mail account
2. Process emails asynchronously without blocking API requests
3. Handle failures and retries gracefully
4. Scale horizontally as user base grows
## Decision
We will use **Celery** with **Redis** as the message broker for background task processing.
## Alternatives Considered
### 1. APScheduler
- **Pros**: Simpler, lightweight, no separate broker needed
- **Cons**: Doesn't scale horizontally, limited monitoring, no distributed task queue
### 2. RQ (Redis Queue)
- **Pros**: Simple Redis-based queue, Pythonic API
- **Cons**: Less mature than Celery, fewer features (no complex routing, less monitoring)
### 3. AWS SQS + Lambda
- **Pros**: Fully managed, auto-scaling
- **Cons**: Cloud vendor lock-in, more expensive, requires AWS infrastructure
### 4. Custom Threading
- **Pros**: No external dependencies
- **Cons**: Complex to implement correctly, hard to scale, no retry logic
## Rationale
Celery was chosen because:
1. **Proven at Scale**: Used by companies like Instagram, Reddit, well-tested
2. **Rich Feature Set**: Built-in retries, rate limiting, task routing, monitoring
3. **Horizontal Scaling**: Add more workers to handle more load
4. **Monitoring**: Flower provides real-time monitoring dashboard
5. **Community**: Large community, extensive documentation
6. **Redis Integration**: Redis already used for caching, can serve dual purpose
## Consequences
### Positive
- Background tasks can scale independently from API
- Automatic retry with exponential backoff
- Task prioritization and routing possible
- Flower provides monitoring and management UI
- Can easily add more task types in future
### Negative
- Additional infrastructure component (Celery workers)
- More complex deployment (workers, beat scheduler)
- Redis becomes critical dependency
- Learning curve for team unfamiliar with Celery
### Neutral
- Need to monitor Redis memory usage
- Task serialization must be considered (use JSON, not pickle)
- Task idempotency should be ensured
## Implementation Notes
```python
# Task definition pattern
@celery_app.task(
bind=True,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 3, 'countdown': 60},
retry_backoff=True
)
def process_mail_account(self: Task, account_id: int) -> dict:
# Implementation
pass
```
## Monitoring
- Use Flower for web-based monitoring: `celery -A app.core.celery_app flower`
- Track metrics: task success/failure rate, execution time, queue length
- Set up alerts for: worker unavailability, high failure rate, queue backlog
## Related Decisions
- See ADR-002 for Redis choice
- See ADR-005 for task retry strategy
## References
- [Celery Documentation](https://docs.celeryq.dev/)
- [Flower Monitoring](https://flower.readthedocs.io/)
- [Celery Best Practices](https://docs.celeryq.dev/en/stable/userguide/tasks.html#best-practices)
docs/adr/002-fernet-encryption.md
+165
View File
@@ -0,0 +1,165 @@
# ADR 002: Use Fernet Encryption for Mail Credentials
**Status:** Accepted
**Date:** 2026-01-20
**Deciders:** Security Team, Development Team
## Context
The application stores POP3/IMAP credentials for user mail accounts. These credentials must be:
1. Encrypted at rest in the database
2. Decryptable when needed for mail operations
3. Protected with industry-standard encryption
4. Simple to implement and maintain
Security requirements:
- Symmetric encryption (need to decrypt for use)
- At least AES-128 bit encryption
- Per-user salt for additional security
- Key rotation capability
## Decision
We will use **Fernet (symmetric encryption)** from the Python `cryptography` library for encrypting mail credentials.
## Alternatives Considered
### 1. AES Directly (PyCrypto/cryptography)
- **Pros**: Full control, widely supported
- **Cons**: Easy to implement incorrectly, need to handle padding, IV, etc.
### 2. Database-Level Encryption (PostgreSQL)
- **Pros**: Transparent to application, secure
- **Cons**: All-or-nothing encryption, harder key rotation, requires DB support
### 3. HashiCorp Vault
- **Pros**: Enterprise-grade secret management, audit logs, key rotation
- **Cons**: Additional infrastructure, complexity, operational overhead
### 4. AWS KMS / Cloud KMS
- **Pros**: Managed service, automatic key rotation
- **Cons**: Cloud vendor lock-in, network latency for each decrypt, cost
## Rationale
Fernet was chosen because:
1. **High-Level API**: Implements encryption best practices by default
2. **Proven Security**: Based on AES-128 in CBC mode with HMAC for authentication
3. **Python Native**: Part of `cryptography` library (PyCA)
4. **Timestamp Validation**: Built-in support for expiring encrypted data
5. **No Complexity**: Handles padding, IV, authentication tag automatically
6. **Battle-Tested**: Used in production by many Python applications
## Implementation Details
### Key Derivation
```python
from cryptography.fernet import Fernet
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2
# Generate key from master secret + per-user salt
kdf = PBKDF2(
algorithm=hashes.SHA256(),
length=32,
salt=salt,
iterations=100_000,
)
key = base64.urlsafe_b64encode(kdf.derive(ENCRYPTION_KEY.encode()))
```
### Encryption/Decryption
```python
def encrypt_password(password: str, user_id: int) -> str:
"""Encrypt password with user-specific salt."""
salt = get_user_salt(user_id)
key = derive_key(ENCRYPTION_KEY, salt)
fernet = Fernet(key)
return fernet.encrypt(password.encode()).decode()
def decrypt_password(encrypted_password: str, user_id: int) -> str:
"""Decrypt password with user-specific salt."""
salt = get_user_salt(user_id)
key = derive_key(ENCRYPTION_KEY, salt)
fernet = Fernet(key)
return fernet.decrypt(encrypted_password.encode()).decode()
```
## Consequences
### Positive
- Simple, secure implementation
- No risk of implementing encryption incorrectly
- Built-in authentication (prevents tampering)
- Can add TTL expiration if needed
- Easy to test and validate
### Negative
- Slower than AES-GCM (includes HMAC overhead)
- Fixed to AES-128 (no AES-256 option without manual implementation)
- All encrypted values become invalid if master key changes (no key rotation)
### Mitigation Strategies
#### For Key Rotation
```python
# Support multiple encryption keys with versioning
ENCRYPTION_KEY_V1 = os.getenv('ENCRYPTION_KEY_V1')
ENCRYPTION_KEY_V2 = os.getenv('ENCRYPTION_KEY_V2') # New key
# Store key version with encrypted data
encrypted_data = f"v2:{fernet_v2.encrypt(data)}"
# Decrypt with appropriate key
version, encrypted = encrypted_data.split(':', 1)
if version == 'v1':
return fernet_v1.decrypt(encrypted)
elif version == 'v2':
return fernet_v2.decrypt(encrypted)
```
#### For Per-User Salt
```python
# Generate unique salt per user (stored in users table)
def get_or_create_user_salt(user_id: int) -> bytes:
# Use deterministic salt based on user_id + global salt
# OR store random salt in database per user
return hashlib.sha256(f'pop3_forwarder_user_{user_id}'.encode()).digest()
```
## Security Best Practices
1. **Never log encryption keys**: Keys only in environment variables
2. **Rotate keys regularly**: Plan for annual key rotation
3. **Secure key storage**: Use secrets manager in production
4. **Strong master key**: Minimum 32 characters, random
5. **Audit access**: Log when credentials are decrypted
6. **Principle of least privilege**: Only workers need decryption
## Monitoring
- Track decryption failures (wrong key indicator)
- Monitor performance impact of encryption
- Alert on unusual decryption volume
- Log credential access for audit
## Future Improvements
1. Migrate to HashiCorp Vault for enterprise deployments
2. Implement automatic key rotation
3. Add encryption key versioning
4. Consider AWS KMS for AWS deployments
5. Add audit trail for credential access
## Related Decisions
- See ADR-006 for key management in production
- See SECURITY_REPORT.md for security analysis
## References
- [Fernet Specification](https://github.com/fernet/spec/blob/master/Spec.md)
- [Python cryptography library](https://cryptography.io/)
- [OWASP Cryptographic Storage Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Cryptographic_Storage_Cheat_Sheet.html)