copilot-swe-agent[bot]
e64f0f2705
- Add SECRET_KEY and ENCRYPTION_KEY validation on startup - Implement security headers middleware (X-Frame-Options, CSP, HSTS) - Add CSRF protection middleware - Create comprehensive GitHub issue templates and PR template - Add Makefile with common development tasks - Configure pre-commit hooks (black, ruff, mypy, bandit, detect-secrets) - Create docs/CODING_PATTERNS.md with best practices - Create docs/ERRORS.md documenting all error codes - Add Architecture Decision Records (ADR) for Celery and Fernet encryption - Create CHANGELOG.md for version tracking - Set up pytest test infrastructure with fixtures and factories - Add sample unit tests for security and config validation - Create CI/CD workflows (test, lint, security) - Add comprehensive TODO.md with milestones and progress tracking Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
3.1 KiB
3.1 KiB
ADR 001: Use Celery for Background Task Processing
Status: Accepted
Date: 2026-01-15
Deciders: Development Team
Context
The multi-tenant SaaS version of the POP3 forwarder needs to process emails for multiple users on different schedules. We need a reliable way to:
- Schedule periodic email checks per mail account
- Process emails asynchronously without blocking API requests
- Handle failures and retries gracefully
- Scale horizontally as user base grows
Decision
We will use Celery with Redis as the message broker for background task processing.
Alternatives Considered
1. APScheduler
- Pros: Simpler, lightweight, no separate broker needed
- Cons: Doesn't scale horizontally, limited monitoring, no distributed task queue
2. RQ (Redis Queue)
- Pros: Simple Redis-based queue, Pythonic API
- Cons: Less mature than Celery, fewer features (no complex routing, less monitoring)
3. AWS SQS + Lambda
- Pros: Fully managed, auto-scaling
- Cons: Cloud vendor lock-in, more expensive, requires AWS infrastructure
4. Custom Threading
- Pros: No external dependencies
- Cons: Complex to implement correctly, hard to scale, no retry logic
Rationale
Celery was chosen because:
- Proven at Scale: Used by companies like Instagram, Reddit, well-tested
- Rich Feature Set: Built-in retries, rate limiting, task routing, monitoring
- Horizontal Scaling: Add more workers to handle more load
- Monitoring: Flower provides real-time monitoring dashboard
- Community: Large community, extensive documentation
- Redis Integration: Redis already used for caching, can serve dual purpose
Consequences
Positive
- Background tasks can scale independently from API
- Automatic retry with exponential backoff
- Task prioritization and routing possible
- Flower provides monitoring and management UI
- Can easily add more task types in future
Negative
- Additional infrastructure component (Celery workers)
- More complex deployment (workers, beat scheduler)
- Redis becomes critical dependency
- Learning curve for team unfamiliar with Celery
Neutral
- Need to monitor Redis memory usage
- Task serialization must be considered (use JSON, not pickle)
- Task idempotency should be ensured
Implementation Notes
# Task definition pattern
@celery_app.task(
bind=True,
autoretry_for=(Exception,),
retry_kwargs={'max_retries': 3, 'countdown': 60},
retry_backoff=True
)
def process_mail_account(self: Task, account_id: int) -> dict:
# Implementation
pass
Monitoring
- Use Flower for web-based monitoring:
celery -A app.core.celery_app flower - Track metrics: task success/failure rate, execution time, queue length
- Set up alerts for: worker unavailability, high failure rate, queue backlog
Related Decisions
- See ADR-002 for Redis choice
- See ADR-005 for task retry strategy