Christian Krakau-Louis
371 lines
14 KiB
Markdown
371 lines
14 KiB
Markdown
# Database Schema
|
|
|
|
This document describes the database schema used by DMARQ, including tables, relationships, and key fields.
|
|
|
|
## Overview
|
|
|
|
DMARQ uses a relational database to store all its data. The schema is designed to efficiently store and query DMARC report data, domain information, and system settings. The system supports both SQLite (for smaller deployments) and PostgreSQL (for production deployments).
|
|
|
|
## Schema Diagram
|
|
|
|
|
|
|
|
## Core Tables
|
|
|
|
### Workspaces
|
|
|
|
The `workspaces` table stores tenant boundaries for multi-organization and MSP
|
|
deployments. Existing single-tenant installs are attached to the default
|
|
workspace during migration.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| slug | VARCHAR | Unique stable workspace slug |
|
|
| name | VARCHAR | Display name |
|
|
| description | TEXT | Optional operator-facing description |
|
|
| active | BOOLEAN | Whether the workspace can be used |
|
|
| created_at | TIMESTAMP | When the workspace was created |
|
|
| updated_at | TIMESTAMP | When the workspace was last updated |
|
|
|
|
### Domains
|
|
|
|
The `domains` table stores information about the domains being monitored.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| workspace_id | INTEGER | Foreign key to workspaces.id |
|
|
| name | VARCHAR(255) | Domain name (e.g., example.com) |
|
|
| created_at | TIMESTAMP | When the domain was added |
|
|
| active | BOOLEAN | Whether the domain is actively monitored |
|
|
| notes | TEXT | Optional notes about the domain |
|
|
| compliance_rate | FLOAT | Cached compliance rate |
|
|
| last_updated | TIMESTAMP | When data was last updated |
|
|
|
|
### Reports
|
|
|
|
The `reports` table stores metadata about DMARC aggregate reports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| domain_id | INTEGER | Foreign key to domains.id |
|
|
| report_id | VARCHAR(255) | Original report ID from the provider |
|
|
| begin_date | TIMESTAMP | Start of report period |
|
|
| end_date | TIMESTAMP | End of report period |
|
|
| org_name | VARCHAR(255) | Organization that sent the report |
|
|
| email | VARCHAR(255) | Email that sent the report |
|
|
| processed_at | TIMESTAMP | When the report was processed |
|
|
| extra_contact_info | VARCHAR(255) | Additional contact info, if provided |
|
|
| error | TEXT | Error information if processing failed |
|
|
| raw_xml | TEXT | Original XML report (optional, can be disabled) |
|
|
|
|
### Report_Records
|
|
|
|
The `report_records` table stores individual authentication results from reports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| report_id | INTEGER | Foreign key to reports.id |
|
|
| source_ip | VARCHAR(45) | Source IP address |
|
|
| count | INTEGER | Count of messages |
|
|
| disposition | VARCHAR(10) | Policy applied (none, quarantine, reject) |
|
|
| dkim_aligned | BOOLEAN | Whether DKIM alignment passed |
|
|
| spf_aligned | BOOLEAN | Whether SPF alignment passed |
|
|
| passed | BOOLEAN | Whether overall DMARC passed |
|
|
| header_from | VARCHAR(255) | Domain in From header |
|
|
| envelope_from | VARCHAR(255) | Domain in envelope From |
|
|
| envelope_to | VARCHAR(255) | Domain in envelope To |
|
|
|
|
### Forensic_Reports
|
|
|
|
The `forensic_reports` table stores DMARC forensic reports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| domain_id | INTEGER | Foreign key to domains.id |
|
|
| report_id | VARCHAR(255) | Original report ID |
|
|
| date | TIMESTAMP | When the report was generated |
|
|
| source_ip | VARCHAR(45) | Source IP address |
|
|
| source_hostname | VARCHAR(255) | Source hostname, if available |
|
|
| failure_type | VARCHAR(20) | Type of auth failure (dkim, spf, both) |
|
|
| auth_failure_detail | TEXT | Detailed reason for failure |
|
|
| delivery_action | VARCHAR(20) | Action taken (delivered, quarantined, rejected) |
|
|
| subject | VARCHAR(255) | Email subject |
|
|
| processed_at | TIMESTAMP | When the report was processed |
|
|
| headers | TEXT | Email headers |
|
|
| original_mail | TEXT | Original email content (if available) |
|
|
|
|
### IP_Info
|
|
|
|
The `ip_info` table caches information about IP addresses.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| ip | VARCHAR(45) | IP address |
|
|
| hostname | VARCHAR(255) | Resolved hostname |
|
|
| country | VARCHAR(2) | Country code |
|
|
| asn | INTEGER | Autonomous System Number |
|
|
| org | VARCHAR(255) | Organization name |
|
|
| last_updated | TIMESTAMP | When the data was last updated |
|
|
|
|
## User and Authentication Tables
|
|
|
|
### Users
|
|
|
|
The `users` table stores user account information.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| workspace_id | INTEGER | Foreign key to workspaces.id |
|
|
| username | VARCHAR(50) | Username |
|
|
| email | VARCHAR(255) | Email address |
|
|
| password_hash | VARCHAR(255) | Hashed password |
|
|
| full_name | VARCHAR(100) | Full name |
|
|
| is_active | BOOLEAN | Whether the account is active |
|
|
| is_admin | BOOLEAN | Whether the user is an administrator |
|
|
| created_at | TIMESTAMP | When the account was created |
|
|
| last_login | TIMESTAMP | When the user last logged in |
|
|
|
|
### API_Tokens
|
|
|
|
The `api_tokens` table stores hashed, scoped API tokens for stable read-only
|
|
automation access. Raw token secrets are returned only once at creation time
|
|
and are never stored.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| name | VARCHAR(120) | Name/description of the token |
|
|
| key_hash | VARCHAR(255) | Bcrypt hash of the token secret |
|
|
| key_prefix | VARCHAR(16) | Non-secret prefix for operator identification |
|
|
| scopes | TEXT | Comma-separated scopes such as `reports:read` |
|
|
| active | BOOLEAN | Whether the token can be used |
|
|
| created_at | TIMESTAMP | When the key was created |
|
|
| updated_at | TIMESTAMP | When the token row was last changed |
|
|
| revoked_at | TIMESTAMP | When the token was revoked |
|
|
| last_used_at | TIMESTAMP | Last successful API use |
|
|
| last_used_ip | VARCHAR(64) | Source IP from the last successful API use |
|
|
| usage_count | INTEGER | Successful API use count |
|
|
|
|
### Webhook_Endpoints
|
|
|
|
The `webhook_endpoints` table stores outbound webhook destinations. Target
|
|
URLs and signing secrets are encrypted at rest.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| name | VARCHAR(120) | Operator-facing endpoint name |
|
|
| url | TEXT | Encrypted destination URL |
|
|
| secret | TEXT | Encrypted signing secret |
|
|
| event_types | TEXT | Comma-separated event subscriptions, or `*` |
|
|
| enabled | BOOLEAN | Whether deliveries can be sent |
|
|
| max_attempts | INTEGER | Maximum attempts before a delivery fails |
|
|
| timeout_seconds | INTEGER | Per-request timeout |
|
|
| last_success_at | TIMESTAMP | Last successful delivery |
|
|
| last_failure_at | TIMESTAMP | Last failed delivery attempt |
|
|
| failure_count | INTEGER | Consecutive endpoint-level failures |
|
|
|
|
### Webhook_Deliveries
|
|
|
|
The `webhook_deliveries` table records delivery attempts and retry state.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| endpoint_id | INTEGER | Foreign key to webhook_endpoints.id |
|
|
| event_type | VARCHAR(80) | Delivered event type |
|
|
| payload | TEXT | Event envelope JSON |
|
|
| idempotency_key | VARCHAR(160) | Stable deduplication key per endpoint |
|
|
| status | VARCHAR(24) | pending, delivered, failed, or abandoned |
|
|
| attempt_count | INTEGER | Attempts already made |
|
|
| max_attempts | INTEGER | Maximum attempts for this delivery |
|
|
| next_attempt_at | TIMESTAMP | Next retry time |
|
|
| last_attempt_at | TIMESTAMP | Last attempt time |
|
|
| delivered_at | TIMESTAMP | Successful delivery time |
|
|
| last_status_code | INTEGER | Last HTTP status code |
|
|
| last_error | TEXT | Last sanitized error |
|
|
| response_excerpt | TEXT | Truncated downstream response |
|
|
|
|
## DNS and Configuration Tables
|
|
|
|
### Mail_Sources
|
|
|
|
The `mail_sources` table stores configured inboxes used to retrieve DMARC,
|
|
forensic, and SMTP TLS reports.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| workspace_id | INTEGER | Foreign key to workspaces.id |
|
|
| name | VARCHAR | Human-readable source name |
|
|
| method | VARCHAR | Source type such as IMAP, Gmail API, or Microsoft Graph |
|
|
| server | VARCHAR | IMAP/POP server hostname |
|
|
| port | INTEGER | IMAP/POP server port |
|
|
| username | VARCHAR | Mailbox username |
|
|
| password | TEXT | Encrypted mailbox password |
|
|
| enabled | BOOLEAN | Whether scheduled imports should poll this source |
|
|
| last_checked | TIMESTAMP | Last polling attempt time |
|
|
| created_at | TIMESTAMP | When the source was created |
|
|
| updated_at | TIMESTAMP | When the source was last updated |
|
|
|
|
### DNS_Records
|
|
|
|
The `dns_records` table stores DNS record information for domains.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| domain_id | INTEGER | Foreign key to domains.id |
|
|
| record_type | VARCHAR(10) | Type of record (SPF, DMARC, DKIM, MX, etc.) |
|
|
| value | TEXT | Value of the DNS record |
|
|
| status | VARCHAR(20) | Status (valid, invalid, warning) |
|
|
| last_checked | TIMESTAMP | When the record was last checked |
|
|
| dkim_selector | VARCHAR(50) | Selector (for DKIM records) |
|
|
|
|
### Settings
|
|
|
|
The `settings` table stores system-wide settings.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| key | VARCHAR(100) | Setting key (primary key) |
|
|
| value | TEXT | Setting value |
|
|
| description | VARCHAR(255) | Description of the setting |
|
|
| type | VARCHAR(20) | Data type (string, integer, boolean, json) |
|
|
| updated_at | TIMESTAMP | When the setting was last updated |
|
|
| updated_by | INTEGER | Foreign key to users.id |
|
|
|
|
## Relationship Tables
|
|
|
|
### User_Domain_Access
|
|
|
|
The `user_domain_access` table manages user permissions for specific domains.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| user_id | INTEGER | Foreign key to users.id |
|
|
| domain_id | INTEGER | Foreign key to domains.id |
|
|
| permission | VARCHAR(20) | Permission level (view, edit, admin) |
|
|
|
|
### Domain_Groups
|
|
|
|
The `domain_groups` table defines groups of domains.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| name | VARCHAR(100) | Group name |
|
|
| description | TEXT | Group description |
|
|
| created_by | INTEGER | Foreign key to users.id |
|
|
| created_at | TIMESTAMP | When the group was created |
|
|
|
|
### Domain_Group_Members
|
|
|
|
The `domain_group_members` table assigns domains to groups.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| group_id | INTEGER | Foreign key to domain_groups.id |
|
|
| domain_id | INTEGER | Foreign key to domains.id |
|
|
|
|
## Logging Tables
|
|
|
|
### Activity_Logs
|
|
|
|
The `activity_logs` table records user actions.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| user_id | INTEGER | Foreign key to users.id (null for system) |
|
|
| action | VARCHAR(50) | Type of action performed |
|
|
| entity_type | VARCHAR(50) | Type of entity affected (domain, report, user) |
|
|
| entity_id | INTEGER | ID of the affected entity |
|
|
| details | TEXT | JSON with additional details |
|
|
| timestamp | TIMESTAMP | When the action occurred |
|
|
| ip_address | VARCHAR(45) | IP address of the user |
|
|
|
|
### System_Logs
|
|
|
|
The `system_logs` table records system events.
|
|
|
|
| Column | Type | Description |
|
|
|--------|------|-------------|
|
|
| id | INTEGER | Primary key |
|
|
| level | VARCHAR(10) | Log level (info, warning, error, debug) |
|
|
| message | TEXT | Log message |
|
|
| component | VARCHAR(50) | Component that generated the log |
|
|
| timestamp | TIMESTAMP | When the event occurred |
|
|
| additional_data | TEXT | JSON with additional data |
|
|
|
|
## Indexes
|
|
|
|
The schema includes several indexes to optimize query performance:
|
|
|
|
- `idx_reports_domain_id`: On reports.domain_id
|
|
- `idx_reports_begin_date`: On reports.begin_date
|
|
- `idx_reports_end_date`: On reports.end_date
|
|
- `idx_report_records_report_id`: On report_records.report_id
|
|
- `idx_report_records_source_ip`: On report_records.source_ip
|
|
- `idx_users_username`: On users.username
|
|
- `idx_users_email`: On users.email
|
|
- `idx_domains_name`: On domains.name
|
|
- `ix_api_tokens_key_hash`: On api_tokens.key_hash
|
|
- `ix_api_tokens_key_prefix`: On api_tokens.key_prefix
|
|
- `ix_api_tokens_active_scope`: On api_tokens.active and api_tokens.scopes
|
|
- `ix_webhook_endpoints_enabled_events`: On webhook_endpoints.enabled and webhook_endpoints.event_types
|
|
- `ix_webhook_delivery_endpoint_idempotency`: Unique on webhook_deliveries.endpoint_id and idempotency_key
|
|
- `ix_webhook_delivery_due`: On webhook_deliveries.status and webhook_deliveries.next_attempt_at
|
|
- `idx_activity_logs_timestamp`: On activity_logs.timestamp
|
|
- `idx_activity_logs_user_id`: On activity_logs.user_id
|
|
- `idx_system_logs_timestamp`: On system_logs.timestamp
|
|
- `idx_system_logs_level`: On system_logs.level
|
|
|
|
## Migrations
|
|
|
|
Database migrations are managed using Alembic, which provides:
|
|
|
|
- Version control for the database schema
|
|
- Automatic schema updates during application upgrades
|
|
- Ability to roll back changes if needed
|
|
- Generation of new migration scripts for schema changes
|
|
|
|
To apply migrations:
|
|
|
|
```bash
|
|
cd backend/app
|
|
python -m alembic upgrade head
|
|
```
|
|
|
|
To create a new migration after schema changes:
|
|
|
|
```bash
|
|
python -m alembic revision --autogenerate -m "Description of changes"
|
|
```
|
|
|
|
## Query Optimization
|
|
|
|
The database schema is designed with query optimization in mind:
|
|
|
|
- Frequently used fields have indexes
|
|
- Historical data can be efficiently queried by date ranges
|
|
- Counters and aggregate data are cached where appropriate
|
|
- Domain-specific data is properly segmented
|
|
|
|
## Database Backup
|
|
|
|
Regular backups of the database should be configured:
|
|
|
|
- For SQLite: Simple file copy or SQLite's `.backup` command
|
|
- For PostgreSQL: `pg_dump` command or continuous archiving with WAL
|
|
|
|
See [Database Backup and Restore](../deployment/backups.md) for backup, restore, and verification commands.
|