Christian Krakau-Louis
243 lines
6.0 KiB
Markdown
243 lines
6.0 KiB
Markdown
# Architecture
|
|
|
|
This document outlines the architecture of the DMARQ system, explaining its components, data flow, and design decisions.
|
|
|
|
## Overview
|
|
|
|
DMARQ is designed as a modern web application with a clear separation between the backend (API server) and frontend (web interface). The system is built to be scalable, maintainable, and deployable in various environments from single-server setups to containerized cloud deployments.
|
|
|
|
## System Components
|
|
|
|
### Backend Components
|
|
|
|
|
|
|
|
#### API Server
|
|
|
|
The core of DMARQ is a FastAPI application that provides:
|
|
|
|
- REST API endpoints for all functionality
|
|
- Authentication and authorization
|
|
- Business logic for processing DMARC reports
|
|
- Database access layer
|
|
- Background task processing
|
|
|
|
**Key Technologies:**
|
|
- Python 3.9+
|
|
- FastAPI framework
|
|
- Pydantic for data validation
|
|
- SQLAlchemy ORM for database access
|
|
- Alembic for database migrations
|
|
|
|
#### Database
|
|
|
|
DMARQ supports two database options:
|
|
|
|
1. **SQLite** - For small deployments and development
|
|
2. **PostgreSQL** - For production deployments with higher load
|
|
|
|
The database schema is designed to efficiently store:
|
|
- Domain information
|
|
- DMARC reports (aggregate and forensic)
|
|
- User accounts and settings
|
|
- System configuration
|
|
|
|
#### IMAP Client
|
|
|
|
The IMAP client module connects to an email server to automatically fetch DMARC reports. It:
|
|
|
|
- Polls the mailbox at configured intervals
|
|
- Downloads emails with DMARC report attachments
|
|
- Extracts and passes reports to the parser
|
|
- Manages the mailbox (marking as read, moving to folders, etc.)
|
|
|
|
#### DMARC Parser
|
|
|
|
The parser processes DMARC reports in various formats:
|
|
- XML format (direct from email providers)
|
|
- Compressed formats (ZIP, GZ)
|
|
- Email attachments (EML)
|
|
|
|
It extracts all relevant data and stores it in the database for analysis.
|
|
|
|
#### Worker System
|
|
|
|
Background tasks are handled by an asynchronous worker system that processes:
|
|
- Report parsing (which can be time-consuming)
|
|
- Scheduled DNS checks
|
|
- Email notifications
|
|
- Report aggregation and statistics calculation
|
|
|
|
For simpler deployments, this uses FastAPI's built-in background tasks. For production, it can be configured to use Celery with Redis or RabbitMQ as the message broker.
|
|
|
|
### Frontend Components
|
|
|
|
#### Web Interface
|
|
|
|
The DMARQ web interface is built with modern web technologies:
|
|
- HTML5
|
|
- CSS (with TailwindCSS/DaisyUI)
|
|
- JavaScript
|
|
- Jinja2 templates
|
|
|
|
It provides a responsive dashboard and administrative interface accessible on desktop and mobile devices.
|
|
|
|
#### Static Assets
|
|
|
|
Static assets are served directly by the web server or a CDN in production:
|
|
- CSS stylesheets
|
|
- JavaScript files
|
|
- Images and icons
|
|
- Fonts
|
|
|
|
## Data Flow
|
|
|
|
### Report Processing Flow
|
|
|
|
1. Reports are received via:
|
|
- Email (IMAP fetcher)
|
|
- Manual upload (web UI)
|
|
- API endpoint
|
|
|
|
2. The report is queued for processing
|
|
|
|
3. The DMARC parser:
|
|
- Validates the report format
|
|
- Extracts metadata (date range, reporting organization)
|
|
- Extracts authentication results
|
|
- Processes individual records
|
|
|
|
4. Processed data is stored in the database
|
|
|
|
5. Statistics are updated:
|
|
- Domain compliance rates
|
|
- Source IP reputation
|
|
- Authentication success/failure trends
|
|
|
|
6. Notifications are sent if configured thresholds are triggered
|
|
|
|
### User Request Flow
|
|
|
|
1. User makes a request to the web interface
|
|
|
|
2. The request is authenticated:
|
|
- Session cookie for web UI
|
|
- API key for API requests
|
|
|
|
3. The relevant API endpoint processes the request:
|
|
- Validates inputs
|
|
- Queries the database
|
|
- Applies business logic
|
|
- Prepares response
|
|
|
|
4. The response is returned:
|
|
- JSON for API requests
|
|
- HTML for web UI requests
|
|
|
|
## Deployment Architecture
|
|
|
|
### Docker Deployment
|
|
|
|
The recommended deployment method uses Docker Compose with these containers:
|
|
- **backend**: The FastAPI application
|
|
- **db**: PostgreSQL database (when not using SQLite)
|
|
- **nginx**: Web server/reverse proxy (for production)
|
|
|
|
### Traditional Deployment
|
|
|
|
For traditional deployments:
|
|
- FastAPI application served by Uvicorn/Gunicorn
|
|
- Nginx or Apache as reverse proxy
|
|
- PostgreSQL database
|
|
- Systemd services for process management
|
|
|
|
## Security Considerations
|
|
|
|
DMARQ implements several security measures:
|
|
|
|
### Authentication
|
|
|
|
- Password-based authentication for web UI
|
|
- API keys for programmatic access
|
|
- OAuth/OIDC integration (optional)
|
|
|
|
### Authorization
|
|
|
|
- Role-based access control (Admin, Analyst, Viewer)
|
|
- Domain-based permissions
|
|
- API key scoping
|
|
|
|
### Data Protection
|
|
|
|
- TLS for all connections
|
|
- Encrypted storage of sensitive data
|
|
- Secure password hashing
|
|
- API rate limiting
|
|
|
|
## Monitoring and Logging
|
|
|
|
The system includes comprehensive logging:
|
|
|
|
- Application logs (API requests, errors)
|
|
- Audit logs (user actions)
|
|
- Performance metrics
|
|
- Health checks
|
|
|
|
These can be integrated with external monitoring systems through:
|
|
- Prometheus metrics endpoint
|
|
- Structured JSON logs
|
|
- Health check API
|
|
|
|
## Scalability Considerations
|
|
|
|
DMARQ is designed to scale in several ways:
|
|
|
|
### Horizontal Scaling
|
|
|
|
- Stateless API servers can be deployed behind a load balancer
|
|
- Database can be scaled separately
|
|
- Background workers can be scaled independently
|
|
|
|
### Vertical Scaling
|
|
|
|
- Database connection pooling
|
|
- Caching of frequently accessed data
|
|
- Efficient query optimization
|
|
|
|
### Data Volume Handling
|
|
|
|
- Partitioning of report data by date
|
|
- Aggregation of historical data
|
|
- Configurable retention policies
|
|
|
|
## Integration Points
|
|
|
|
DMARQ is designed to integrate with external systems:
|
|
|
|
### DNS Providers
|
|
|
|
- Cloudflare API integration
|
|
- AWS Route 53 integration
|
|
- Generic DNS API support
|
|
|
|
### Notification Systems
|
|
|
|
- Email notifications
|
|
- Webhook callbacks
|
|
- Integration with Slack, Teams, etc.
|
|
|
|
### Authentication Providers
|
|
|
|
- LDAP/Active Directory
|
|
- OAuth providers (Google, GitHub, etc.)
|
|
- SAML for enterprise SSO
|
|
|
|
## Development Architecture
|
|
|
|
For development, the architecture supports:
|
|
|
|
- Fast reload for code changes
|
|
- SQLite database for simplicity
|
|
- Mock data generators
|
|
- Comprehensive test suite
|
|
- CI/CD pipeline integration |