gh-christianlouis-dmarq/docs/reference/database.md

Database Schema

This document describes the database schema used by DMARQ, including tables, relationships, and key fields.

Overview

DMARQ uses a relational database to store all its data. The schema is designed to efficiently store and query DMARC report data, domain information, and system settings. The system supports both SQLite (for smaller deployments) and PostgreSQL (for production deployments).

Schema Diagram

Core Tables

Domains

The domains table stores information about the domains being monitored.

Column	Type	Description
id	INTEGER	Primary key
name	VARCHAR(255)	Domain name (e.g., example.com)
created_at	TIMESTAMP	When the domain was added
active	BOOLEAN	Whether the domain is actively monitored
notes	TEXT	Optional notes about the domain
compliance_rate	FLOAT	Cached compliance rate
last_updated	TIMESTAMP	When data was last updated

Reports

The reports table stores metadata about DMARC aggregate reports.

Column	Type	Description
id	INTEGER	Primary key
domain_id	INTEGER	Foreign key to domains.id
report_id	VARCHAR(255)	Original report ID from the provider
begin_date	TIMESTAMP	Start of report period
end_date	TIMESTAMP	End of report period
org_name	VARCHAR(255)	Organization that sent the report
email	VARCHAR(255)	Email that sent the report
processed_at	TIMESTAMP	When the report was processed
extra_contact_info	VARCHAR(255)	Additional contact info, if provided
error	TEXT	Error information if processing failed
raw_xml	TEXT	Original XML report (optional, can be disabled)

Report_Records

The report_records table stores individual authentication results from reports.

Column	Type	Description
id	INTEGER	Primary key
report_id	INTEGER	Foreign key to reports.id
source_ip	VARCHAR(45)	Source IP address
count	INTEGER	Count of messages
disposition	VARCHAR(10)	Policy applied (none, quarantine, reject)
dkim_aligned	BOOLEAN	Whether DKIM alignment passed
spf_aligned	BOOLEAN	Whether SPF alignment passed
passed	BOOLEAN	Whether overall DMARC passed
header_from	VARCHAR(255)	Domain in From header
envelope_from	VARCHAR(255)	Domain in envelope From
envelope_to	VARCHAR(255)	Domain in envelope To

Forensic_Reports

The forensic_reports table stores DMARC forensic reports.

Column	Type	Description
id	INTEGER	Primary key
domain_id	INTEGER	Foreign key to domains.id
report_id	VARCHAR(255)	Original report ID
date	TIMESTAMP	When the report was generated
source_ip	VARCHAR(45)	Source IP address
source_hostname	VARCHAR(255)	Source hostname, if available
failure_type	VARCHAR(20)	Type of auth failure (dkim, spf, both)
auth_failure_detail	TEXT	Detailed reason for failure
delivery_action	VARCHAR(20)	Action taken (delivered, quarantined, rejected)
subject	VARCHAR(255)	Email subject
processed_at	TIMESTAMP	When the report was processed
headers	TEXT	Email headers
original_mail	TEXT	Original email content (if available)

IP_Info

The ip_info table caches information about IP addresses.

Column	Type	Description
id	INTEGER	Primary key
ip	VARCHAR(45)	IP address
hostname	VARCHAR(255)	Resolved hostname
country	VARCHAR(2)	Country code
asn	INTEGER	Autonomous System Number
org	VARCHAR(255)	Organization name
last_updated	TIMESTAMP	When the data was last updated

User and Authentication Tables

Users

The users table stores user account information.

Column	Type	Description
id	INTEGER	Primary key
username	VARCHAR(50)	Username
email	VARCHAR(255)	Email address
password_hash	VARCHAR(255)	Hashed password
full_name	VARCHAR(100)	Full name
is_active	BOOLEAN	Whether the account is active
is_admin	BOOLEAN	Whether the user is an administrator
created_at	TIMESTAMP	When the account was created
last_login	TIMESTAMP	When the user last logged in

API_Tokens

The api_tokens table stores hashed, scoped API tokens for stable read-only automation access. Raw token secrets are returned only once at creation time and are never stored.

Column	Type	Description
id	INTEGER	Primary key
name	VARCHAR(120)	Name/description of the token
key_hash	VARCHAR(255)	Bcrypt hash of the token secret
key_prefix	VARCHAR(16)	Non-secret prefix for operator identification
scopes	TEXT	Comma-separated scopes such as reports:read
active	BOOLEAN	Whether the token can be used
created_at	TIMESTAMP	When the key was created
updated_at	TIMESTAMP	When the token row was last changed
revoked_at	TIMESTAMP	When the token was revoked
last_used_at	TIMESTAMP	Last successful API use
last_used_ip	VARCHAR(64)	Source IP from the last successful API use
usage_count	INTEGER	Successful API use count

Webhook_Endpoints

The webhook_endpoints table stores outbound webhook destinations. Target URLs and signing secrets are encrypted at rest.

Column	Type	Description
id	INTEGER	Primary key
name	VARCHAR(120)	Operator-facing endpoint name
url	TEXT	Encrypted destination URL
secret	TEXT	Encrypted signing secret
event_types	TEXT	Comma-separated event subscriptions, or *
enabled	BOOLEAN	Whether deliveries can be sent
max_attempts	INTEGER	Maximum attempts before a delivery fails
timeout_seconds	INTEGER	Per-request timeout
last_success_at	TIMESTAMP	Last successful delivery
last_failure_at	TIMESTAMP	Last failed delivery attempt
failure_count	INTEGER	Consecutive endpoint-level failures

Webhook_Deliveries

The webhook_deliveries table records delivery attempts and retry state.

Column	Type	Description
id	INTEGER	Primary key
endpoint_id	INTEGER	Foreign key to webhook_endpoints.id
event_type	VARCHAR(80)	Delivered event type
payload	TEXT	Event envelope JSON
idempotency_key	VARCHAR(160)	Stable deduplication key per endpoint
status	VARCHAR(24)	pending, delivered, failed, or abandoned
attempt_count	INTEGER	Attempts already made
max_attempts	INTEGER	Maximum attempts for this delivery
next_attempt_at	TIMESTAMP	Next retry time
last_attempt_at	TIMESTAMP	Last attempt time
delivered_at	TIMESTAMP	Successful delivery time
last_status_code	INTEGER	Last HTTP status code
last_error	TEXT	Last sanitized error
response_excerpt	TEXT	Truncated downstream response

DNS and Configuration Tables

DNS_Records

The dns_records table stores DNS record information for domains.

Column	Type	Description
id	INTEGER	Primary key
domain_id	INTEGER	Foreign key to domains.id
record_type	VARCHAR(10)	Type of record (SPF, DMARC, DKIM, MX, etc.)
value	TEXT	Value of the DNS record
status	VARCHAR(20)	Status (valid, invalid, warning)
last_checked	TIMESTAMP	When the record was last checked
dkim_selector	VARCHAR(50)	Selector (for DKIM records)

Settings

The settings table stores system-wide settings.

Column	Type	Description
key	VARCHAR(100)	Setting key (primary key)
value	TEXT	Setting value
description	VARCHAR(255)	Description of the setting
type	VARCHAR(20)	Data type (string, integer, boolean, json)
updated_at	TIMESTAMP	When the setting was last updated
updated_by	INTEGER	Foreign key to users.id

Relationship Tables

User_Domain_Access

The user_domain_access table manages user permissions for specific domains.

Column	Type	Description
id	INTEGER	Primary key
user_id	INTEGER	Foreign key to users.id
domain_id	INTEGER	Foreign key to domains.id
permission	VARCHAR(20)	Permission level (view, edit, admin)

Domain_Groups

The domain_groups table defines groups of domains.

Column	Type	Description
id	INTEGER	Primary key
name	VARCHAR(100)	Group name
description	TEXT	Group description
created_by	INTEGER	Foreign key to users.id
created_at	TIMESTAMP	When the group was created

Domain_Group_Members

The domain_group_members table assigns domains to groups.

Column	Type	Description
group_id	INTEGER	Foreign key to domain_groups.id
domain_id	INTEGER	Foreign key to domains.id

Logging Tables

Activity_Logs

The activity_logs table records user actions.

Column	Type	Description
id	INTEGER	Primary key
user_id	INTEGER	Foreign key to users.id (null for system)
action	VARCHAR(50)	Type of action performed
entity_type	VARCHAR(50)	Type of entity affected (domain, report, user)
entity_id	INTEGER	ID of the affected entity
details	TEXT	JSON with additional details
timestamp	TIMESTAMP	When the action occurred
ip_address	VARCHAR(45)	IP address of the user

System_Logs

The system_logs table records system events.

Column	Type	Description
id	INTEGER	Primary key
level	VARCHAR(10)	Log level (info, warning, error, debug)
message	TEXT	Log message
component	VARCHAR(50)	Component that generated the log
timestamp	TIMESTAMP	When the event occurred
additional_data	TEXT	JSON with additional data

Indexes

The schema includes several indexes to optimize query performance:

• idx_reports_domain_id: On reports.domain_id
• idx_reports_begin_date: On reports.begin_date
• idx_reports_end_date: On reports.end_date
• idx_report_records_report_id: On report_records.report_id
• idx_report_records_source_ip: On report_records.source_ip
• idx_users_username: On users.username
• idx_users_email: On users.email
• idx_domains_name: On domains.name
• ix_api_tokens_key_hash: On api_tokens.key_hash
• ix_api_tokens_key_prefix: On api_tokens.key_prefix
• ix_api_tokens_active_scope: On api_tokens.active and api_tokens.scopes
• ix_webhook_endpoints_enabled_events: On webhook_endpoints.enabled and webhook_endpoints.event_types
• ix_webhook_delivery_endpoint_idempotency: Unique on webhook_deliveries.endpoint_id and idempotency_key
• ix_webhook_delivery_due: On webhook_deliveries.status and webhook_deliveries.next_attempt_at
• idx_activity_logs_timestamp: On activity_logs.timestamp
• idx_activity_logs_user_id: On activity_logs.user_id
• idx_system_logs_timestamp: On system_logs.timestamp
• idx_system_logs_level: On system_logs.level

Migrations

Database migrations are managed using Alembic, which provides:

• Version control for the database schema
• Automatic schema updates during application upgrades
• Ability to roll back changes if needed
• Generation of new migration scripts for schema changes

To apply migrations:

cd backend/app
python -m alembic upgrade head

To create a new migration after schema changes:

python -m alembic revision --autogenerate -m "Description of changes"

Query Optimization

The database schema is designed with query optimization in mind:

• Frequently used fields have indexes
• Historical data can be efficiently queried by date ranges
• Counters and aggregate data are cached where appropriate
• Domain-specific data is properly segmented

Database Backup

Regular backups of the database should be configured:

• For SQLite: Simple file copy or SQLite's .backup command
• For PostgreSQL: pg_dump command or continuous archiving with WAL

See Database Backup and Restore for backup, restore, and verification commands.