gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive API and architecture documentation for DMARQ

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-04-21 02:05:14 +02:00
parent e6a23c5fc9
commit ae79cca8c1
2 changed files with 659 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/api.md
+416
View File
@@ -0,0 +1,416 @@
# API Reference
DMARQ provides a comprehensive REST API that allows you to integrate with external systems and build custom workflows.
## Authentication
All API requests require authentication using an API key.
### API Keys
To use the API, you need to generate an API key:
1. Navigate to **Settings** > **API Access** in the DMARQ UI
2. Click **Create API Key**
3. Enter a description for the key (e.g., "Integration with Slack")
4. Select the permissions you want to grant to this key
5. Click **Generate Key**
6. Copy the key immediately (it will only be shown once)
### Authentication Header
Include your API key in all requests using the `X-API-Key` header:
```
X-API-Key: your_api_key_here
```
## Base URL
The base URL for all API endpoints is:
```
https://your-dmarq-instance.com/api/v1
```
Replace `your-dmarq-instance.com` with your actual DMARQ hostname.
## API Endpoints
### Domains
#### List Domains
```
GET /domains
```
Returns a list of all domains in your DMARQ instance.
**Example Response:**
```json
{
"domains": [
{
"id": "1",
"name": "example.com",
"added_at": "2025-01-15T14:30:00Z",
"status": "active",
"compliance_rate": 87.5
},
{
"id": "2",
"name": "example.org",
"added_at": "2025-01-16T09:15:00Z",
"status": "active",
"compliance_rate": 95.2
}
],
"total": 2
}
```
#### Get Domain Details
```
GET /domains/{domain_id}
```
Returns details for a specific domain.
**Example Response:**
```json
{
"id": "1",
"name": "example.com",
"added_at": "2025-01-15T14:30:00Z",
"status": "active",
"compliance_rate": 87.5,
"reports_count": 45,
"last_report_date": "2025-04-01T00:00:00Z",
"dns_records": {
"spf": "v=spf1 include:_spf.example.com ~all",
"dmarc": "v=DMARC1; p=none; rua=mailto:dmarc@example.com",
"mx": ["10 mail.example.com"]
}
}
```
#### Add Domain
```
POST /domains
```
Adds a new domain to DMARQ.
**Request Body:**
```json
{
"name": "newdomain.com"
}
```
**Example Response:**
```json
{
"id": "3",
"name": "newdomain.com",
"added_at": "2025-04-21T14:22:00Z",
"status": "active"
}
```
#### Remove Domain
```
DELETE /domains/{domain_id}
```
Removes a domain from DMARQ.
**Example Response:**
```json
{
"success": true,
"message": "Domain removed successfully"
}
```
### Reports
#### List Reports
```
GET /reports
```
Returns a list of DMARC reports.
**Query Parameters:**
- `domain_id` - Filter by domain
- `start_date` - Filter by start date (YYYY-MM-DD)
- `end_date` - Filter by end date (YYYY-MM-DD)
- `source_org` - Filter by source organization
- `page` - Page number for pagination (default: 1)
- `limit` - Number of results per page (default: 25, max: 100)
**Example Response:**
```json
{
"reports": [
{
"id": "1",
"domain": "example.com",
"report_id": "google.com:1234567890",
"date_range": {
"begin": "2025-04-01T00:00:00Z",
"end": "2025-04-01T23:59:59Z"
},
"source_org": "google.com",
"source_email": "noreply-dmarc-support@google.com",
"message_count": 156,
"pass_count": 142,
"fail_count": 14,
"pass_rate": 91.0
}
],
"total": 245,
"page": 1,
"limit": 25,
"total_pages": 10
}
```
#### Get Report Details
```
GET /reports/{report_id}
```
Returns details for a specific report.
**Example Response:**
```json
{
"id": "1",
"domain": "example.com",
"report_id": "google.com:1234567890",
"date_range": {
"begin": "2025-04-01T00:00:00Z",
"end": "2025-04-01T23:59:59Z"
},
"source_org": "google.com",
"source_email": "noreply-dmarc-support@google.com",
"policy_published": {
"domain": "example.com",
"adkim": "r",
"aspf": "r",
"p": "none",
"sp": "none",
"pct": 100
},
"records": [
{
"source_ip": "192.0.2.1",
"count": 34,
"policy_evaluated": {
"disposition": "none",
"dkim": "pass",
"spf": "pass"
},
"identifiers": {
"header_from": "example.com",
"envelope_from": "bounces.example.com"
},
"auth_results": {
"dkim": {
"domain": "example.com",
"selector": "default",
"result": "pass"
},
"spf": {
"domain": "bounces.example.com",
"result": "pass"
}
}
}
]
}
```
#### Upload Report
```
POST /reports/upload
```
Uploads a new DMARC report for processing.
**Request Body:**
- Multipart form with file upload (XML, ZIP, or GZ format)
**Example Response:**
```json
{
"success": true,
"message": "Report uploaded and queued for processing",
"task_id": "abcd1234"
}
```
### Statistics
#### Compliance Summary
```
GET /stats/{domain_id}/compliance
```
Returns compliance statistics for a domain.
**Query Parameters:**
- `start_date` - Start date (YYYY-MM-DD)
- `end_date` - End date (YYYY-MM-DD)
- `interval` - Aggregation interval (day, week, month)
**Example Response:**
```json
{
"domain": "example.com",
"date_range": {
"start": "2025-03-01",
"end": "2025-04-01"
},
"overall": {
"total_messages": 4586,
"pass": 4102,
"fail": 484,
"compliance_rate": 89.4,
"spf_aligned": 4220,
"dkim_aligned": 4315
},
"trend": [
{
"date": "2025-03-01",
"total": 152,
"pass": 132,
"fail": 20,
"rate": 86.8
},
// Additional days...
]
}
```
#### Source Summary
```
GET /stats/{domain_id}/sources
```
Returns statistics about sending sources.
**Example Response:**
```json
{
"domain": "example.com",
"top_sources": [
{
"source_ip": "192.0.2.1",
"source_domain": "mail-server.example.com",
"message_count": 2156,
"pass_count": 2156,
"fail_count": 0,
"pass_rate": 100.0
},
// Additional sources...
]
}
```
### System
#### System Status
```
GET /system/status
```
Returns the status of the DMARQ system.
**Example Response:**
```json
{
"status": "healthy",
"version": "1.2.0",
"uptime": 1234567,
"database": "connected",
"imap": "connected",
"domains_count": 5,
"reports_count": 12543
}
```
## Rate Limiting
The API is rate limited to prevent abuse:
- 60 requests per minute for most endpoints
- 10 requests per minute for upload endpoints
If you exceed these limits, you'll receive a `429 Too Many Requests` response.
## Errors
The API uses standard HTTP status codes to indicate the success or failure of requests:
- `200 OK` - Request succeeded
- `201 Created` - Resource created successfully
- `400 Bad Request` - Invalid request parameters
- `401 Unauthorized` - Missing or invalid API key
- `403 Forbidden` - API key doesn't have sufficient permissions
- `404 Not Found` - Resource not found
- `429 Too Many Requests` - Rate limit exceeded
- `500 Internal Server Error` - Server error
Error responses include a JSON body with details:
```json
{
"error": "validation_error",
"message": "Invalid domain name format",
"details": {
"field": "name",
"reason": "Does not match domain name pattern"
}
}
```
## SDKs and Integrations
DMARQ provides client libraries for common programming languages:
- Python: [dmarq-python](https://github.com/yourusername/dmarq-python)
- JavaScript/Node.js: [dmarq-node](https://github.com/yourusername/dmarq-node)
## API Versioning
The API uses versioning in the URL path (/api/v1/) to ensure backward compatibility. When breaking changes are necessary, we'll introduce a new version (e.g., /api/v2/).
## Webhooks
DMARQ can notify your systems about events via webhooks:
1. Navigate to **Settings** > **API Access** > **Webhooks**
2. Click **Add Webhook**
3. Configure:
- Destination URL
- Secret token (for verification)
- Events to subscribe to
Supported events:
- `report.processed` - When a new report is processed
- `compliance.threshold` - When compliance falls below threshold
- `domain.added` - When a domain is added
- `domain.removed` - When a domain is removed
docs/reference/architecture.md
+243
View File
@@ -0,0 +1,243 @@
# Architecture
This document outlines the architecture of the DMARQ system, explaining its components, data flow, and design decisions.
## Overview
DMARQ is designed as a modern web application with a clear separation between the backend (API server) and frontend (web interface). The system is built to be scalable, maintainable, and deployable in various environments from single-server setups to containerized cloud deployments.
## System Components
### Backend Components
![DMARQ Architecture](../assets/images/architecture_diagram.png)
#### API Server
The core of DMARQ is a FastAPI application that provides:
- REST API endpoints for all functionality
- Authentication and authorization
- Business logic for processing DMARC reports
- Database access layer
- Background task processing
**Key Technologies:**
- Python 3.9+
- FastAPI framework
- Pydantic for data validation
- SQLAlchemy ORM for database access
- Alembic for database migrations
#### Database
DMARQ supports two database options:
1. **SQLite** - For small deployments and development
2. **PostgreSQL** - For production deployments with higher load
The database schema is designed to efficiently store:
- Domain information
- DMARC reports (aggregate and forensic)
- User accounts and settings
- System configuration
#### IMAP Client
The IMAP client module connects to an email server to automatically fetch DMARC reports. It:
- Polls the mailbox at configured intervals
- Downloads emails with DMARC report attachments
- Extracts and passes reports to the parser
- Manages the mailbox (marking as read, moving to folders, etc.)
#### DMARC Parser
The parser processes DMARC reports in various formats:
- XML format (direct from email providers)
- Compressed formats (ZIP, GZ)
- Email attachments (EML)
It extracts all relevant data and stores it in the database for analysis.
#### Worker System
Background tasks are handled by an asynchronous worker system that processes:
- Report parsing (which can be time-consuming)
- Scheduled DNS checks
- Email notifications
- Report aggregation and statistics calculation
For simpler deployments, this uses FastAPI's built-in background tasks. For production, it can be configured to use Celery with Redis or RabbitMQ as the message broker.
### Frontend Components
#### Web Interface
The DMARQ web interface is built with modern web technologies:
- HTML5
- CSS (with TailwindCSS/DaisyUI)
- JavaScript
- Jinja2 templates
It provides a responsive dashboard and administrative interface accessible on desktop and mobile devices.
#### Static Assets
Static assets are served directly by the web server or a CDN in production:
- CSS stylesheets
- JavaScript files
- Images and icons
- Fonts
## Data Flow
### Report Processing Flow
1. Reports are received via:
- Email (IMAP fetcher)
- Manual upload (web UI)
- API endpoint
2. The report is queued for processing
3. The DMARC parser:
- Validates the report format
- Extracts metadata (date range, reporting organization)
- Extracts authentication results
- Processes individual records
4. Processed data is stored in the database
5. Statistics are updated:
- Domain compliance rates
- Source IP reputation
- Authentication success/failure trends
6. Notifications are sent if configured thresholds are triggered
### User Request Flow
1. User makes a request to the web interface
2. The request is authenticated:
- Session cookie for web UI
- API key for API requests
3. The relevant API endpoint processes the request:
- Validates inputs
- Queries the database
- Applies business logic
- Prepares response
4. The response is returned:
- JSON for API requests
- HTML for web UI requests
## Deployment Architecture
### Docker Deployment
The recommended deployment method uses Docker Compose with these containers:
- **backend**: The FastAPI application
- **db**: PostgreSQL database (when not using SQLite)
- **nginx**: Web server/reverse proxy (for production)
### Traditional Deployment
For traditional deployments:
- FastAPI application served by Uvicorn/Gunicorn
- Nginx or Apache as reverse proxy
- PostgreSQL database
- Systemd services for process management
## Security Considerations
DMARQ implements several security measures:
### Authentication
- Password-based authentication for web UI
- API keys for programmatic access
- OAuth/OIDC integration (optional)
### Authorization
- Role-based access control (Admin, Analyst, Viewer)
- Domain-based permissions
- API key scoping
### Data Protection
- TLS for all connections
- Encrypted storage of sensitive data
- Secure password hashing
- API rate limiting
## Monitoring and Logging
The system includes comprehensive logging:
- Application logs (API requests, errors)
- Audit logs (user actions)
- Performance metrics
- Health checks
These can be integrated with external monitoring systems through:
- Prometheus metrics endpoint
- Structured JSON logs
- Health check API
## Scalability Considerations
DMARQ is designed to scale in several ways:
### Horizontal Scaling
- Stateless API servers can be deployed behind a load balancer
- Database can be scaled separately
- Background workers can be scaled independently
### Vertical Scaling
- Database connection pooling
- Caching of frequently accessed data
- Efficient query optimization
### Data Volume Handling
- Partitioning of report data by date
- Aggregation of historical data
- Configurable retention policies
## Integration Points
DMARQ is designed to integrate with external systems:
### DNS Providers
- Cloudflare API integration
- AWS Route 53 integration
- Generic DNS API support
### Notification Systems
- Email notifications
- Webhook callbacks
- Integration with Slack, Teams, etc.
### Authentication Providers
- LDAP/Active Directory
- OAuth providers (Google, GitHub, etc.)
- SAML for enterprise SSO
## Development Architecture
For development, the architecture supports:
- Fast reload for code changes
- SQLite database for simplicity
- Mock data generators
- Comprehensive test suite
- CI/CD pipeline integration