gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive contributing, testing, roadmap, and database schema documentation for DMARQ

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-04-21 02:51:17 +02:00
parent ae79cca8c1
commit a0ddadfaa1
6 changed files with 1284 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/changelog.md
+51
View File
@@ -0,0 +1,51 @@
# Changelog
All notable changes to DMARQ will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [1.0.0] - 2025-04-15
### Added
- Initial release of DMARQ
- Domain management with basic health checks
- DMARC report processing (aggregate and forensic)
- Dashboard with compliance rate visualization
- IMAP integration for automatic report collection
- User authentication and management
- SQLite and PostgreSQL database support
- Docker deployment option
- Basic alert system for compliance issues
- API for third-party integration
- Documentation site
### Security
- Secure password storage with bcrypt
- JWT-based authentication for API
- Rate limiting for API endpoints
- Input validation for all user inputs
## [0.9.0] - 2025-03-01
### Added
- Beta release for early testing
- All core functionality implemented
- Limited to SQLite database only
### Fixed
- Multiple parser bugs for different report formats
- UI responsiveness issues on mobile devices
## [0.8.0] - 2025-02-15
### Added
- Alpha release for internal testing
- Basic DMARC report parsing
- Simple domain management
- Initial dashboard design
### Known Issues
- Limited support for forensic reports
- Missing authentication features
- No alerting capabilities
docs/development/contributing.md
+229
View File
@@ -0,0 +1,229 @@
# Contributing to DMARQ
Thank you for your interest in contributing to DMARQ! This guide will help you get started with the development process.
## Code of Conduct
Please read and follow our [Code of Conduct](https://github.com/yourusername/dmarq/blob/main/CODE_OF_CONDUCT.md) to keep our community approachable and respectable.
## How to Contribute
There are many ways to contribute to DMARQ:
- **Reporting bugs**: Submit detailed bug reports to help us improve
- **Suggesting features**: Propose new features or improvements
- **Writing code**: Contribute code changes or new features
- **Improving docs**: Help make our documentation more comprehensive
- **Translation**: Help translate the interface into other languages
## Development Environment Setup
### Prerequisites
- Python 3.9+
- Node.js 16+ (for frontend assets)
- Docker and Docker Compose (recommended)
- Git
### Setting Up the Project
1. **Fork the repository**
Start by forking the [DMARQ repository](https://github.com/yourusername/dmarq) on GitHub.
2. **Clone your fork**
```bash
git clone https://github.com/YOUR-USERNAME/dmarq.git
cd dmarq
```
3. **Set up the development environment**
Using Docker (recommended):
```bash
docker-compose -f docker-compose.dev.yml up
```
Or manually:
```bash
# Create a virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
cd backend
pip install -r requirements.txt
pip install -r requirements-dev.txt
# Set up the database
cd app
python -m alembic upgrade head
# Start the development server
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
4. **Frontend Assets (if modifying)**
If you're modifying frontend assets:
```bash
cd backend/app/static
npm install
npm run dev
```
## Making Changes
### Branching Strategy
We follow a simple branching strategy:
- `main` branch is the stable release branch
- `develop` branch is for development work
- Feature branches should be created from `develop`
### Creating a Branch
Create a new branch for your changes:
```bash
git checkout develop
git pull origin develop
git checkout -b feature/your-feature-name
```
Use prefixes like:
- `feature/` for new features
- `bugfix/` for bug fixes
- `docs/` for documentation changes
- `test/` for test improvements
### Coding Standards
We follow these standards:
- **Python**: [PEP 8](https://www.python.org/dev/peps/pep-0008/) style guide
- **JavaScript**: ESLint with Airbnb style
- **HTML/CSS**: Follow the project's existing patterns
We use pre-commit hooks to enforce coding standards:
```bash
pip install pre-commit
pre-commit install
```
### Testing
All code changes should include tests:
```bash
# Run the test suite
cd backend
pytest
# With coverage
pytest --cov=app
```
## Submitting a Pull Request
1. **Update your branch**
```bash
git fetch origin
git rebase origin/develop
```
2. **Run tests**
Ensure all tests pass before submitting:
```bash
pytest
```
3. **Commit your changes**
Follow the [Conventional Commits](https://www.conventionalcommits.org/) standard:
```bash
git commit -m "feat: add user authentication"
```
4. **Push to your fork**
```bash
git push origin feature/your-feature-name
```
5. **Submit a pull request**
Go to the [DMARQ repository](https://github.com/yourusername/dmarq) and create a pull request from your branch to the `develop` branch.
Include in your PR description:
- What changes you've made
- Why you've made these changes
- Any relevant issue numbers (e.g., "Fixes #123")
- Screenshots if applicable
6. **Code review**
Maintainers will review your code. You might need to make additional changes based on feedback.
## Pull Request Review Process
Pull requests are reviewed by maintainers who will check:
1. Code quality and style
2. Test coverage
3. Documentation
4. Overall fit with the project goals
## Release Process
We use semantic versioning (MAJOR.MINOR.PATCH):
- MAJOR version for incompatible API changes
- MINOR version for new functionality in a backwards compatible manner
- PATCH version for backwards compatible bug fixes
## Documentation
Please update documentation alongside code changes:
- Update relevant parts of this documentation site
- Add or update docstrings
- Update README.md if needed
To build and preview the documentation:
```bash
# Install mkdocs and requirements
pip install -r docs/readthedocs/requirements.txt
# Serve documentation locally
mkdocs serve
```
## Additional Resources
- [Project Architecture](../reference/architecture.md)
- [Database Schema](../reference/database.md)
- [API Reference](../reference/api.md)
## Getting Help
If you need help with your contribution, you can:
- Open an issue on GitHub
- Join our community channels
- Email the maintainers at maintainers@example.com
## Recognition
All contributors are recognized in our [CONTRIBUTORS.md](https://github.com/yourusername/dmarq/blob/main/CONTRIBUTORS.md) file. We appreciate your help in making DMARQ better!
docs/development/roadmap.md
+201
View File
@@ -0,0 +1,201 @@
# Roadmap
This document outlines the planned development roadmap for DMARQ, including upcoming features, improvements, and long-term goals.
## Current Version: 1.0.0 (April 2025)
The initial release of DMARQ includes:
- Basic DMARC report processing and analysis
- Domain management
- User authentication
- Dashboard with key metrics
- IMAP integration for automatic report collection
- Simple alerting system
- Docker deployment option
## Short-Term Goals (Q2-Q3 2025)
### Version 1.1.0 (June 2025)
- **Advanced Report Filtering**
- Filter reports by IP address
- Filter by authentication result
- Custom date range selection
- Save custom filters
- **Improved Visualizations**
- Interactive charts with drill-down capability
- Geographic IP distribution map
- Timeline view of authentication changes
- **Enhanced DNS Health Checks**
- Automated SPF, DKIM, DMARC syntax validation
- Record monitoring with change detection
- Best practice recommendations
- **API Enhancements**
- Additional endpoints for statistics
- Improved authentication options
- Better documentation and examples
### Version 1.2.0 (August 2025)
- **User Management Improvements**
- Role-based access control
- Domain-specific permissions
- User invitation system
- Activity audit logging
- **Multi-tenant Support**
- Organization-level grouping of domains
- Isolated views for different user groups
- White-labeling options
- **Enhanced IMAP Integration**
- Support for multiple mailboxes
- Advanced filtering options
- Attachment preprocessing rules
- **Forensic Report Analysis**
- Improved parsing for various report formats
- Header analysis tools
- Correlation with aggregate reports
## Mid-Term Goals (Q4 2025 - Q1 2026)
### Version 1.3.0 (November 2025)
- **Integration Ecosystem**
- Slack/Teams notifications
- WebHook support for custom integrations
- Export to BI tools
- SIEM integration
- **Advanced Alerting System**
- Custom alert rules
- Alert severity levels
- Alert acknowledgment workflow
- Historical alert tracking
- **DNS Management**
- Integration with Cloudflare API
- Integration with AWS Route 53
- One-click fix for common DNS issues
- DNS record deployment tracking
- **Report Anomaly Detection**
- Machine learning-based anomaly detection
- Unusual sending pattern identification
- Automatic threat scoring
### Version 2.0.0 (February 2026)
- **Comprehensive Email Authentication Suite**
- SPF record management and monitoring
- DKIM key rotation management
- BIMI record support
- MTA-STS implementation assistance
- **Policy Management**
- DMARC policy transition recommendations
- Automated policy progression
- Impact analysis before policy changes
- Rollback capabilities
- **Reporting Enhancements**
- Scheduled PDF/CSV exports
- Custom report templates
- Executive summary generation
- Trend analysis with predictive insights
- **Multi-Channel Notifications**
- Email notifications
- SMS alerts
- Mobile app push notifications
- Custom notification channels
## Long-Term Goals (Mid 2026+)
### Version 2.x and Beyond
- **Advanced Threat Intelligence**
- Integration with email security platforms
- Shared threat database
- Sender reputation scoring
- Proactive security recommendations
- **Enterprise Features**
- LDAP/Active Directory integration
- SAML/SSO support
- Advanced audit logging
- Custom branding
- **Internationalization**
- Multi-language interface
- Region-specific reporting
- International domain support (IDN)
- Localized documentation
- **AI-Powered Analysis**
- Natural language querying of report data
- Automated root cause analysis
- Predictive compliance modeling
- AI-assisted remediation recommendations
- **Ecosystem Expansion**
- Mobile companion app
- Browser plugins
- Desktop notifications
- Command-line tools
## Feature Requests and Prioritization
We prioritize features based on:
1. **User Impact**: How many users will benefit?
2. **Security Enhancement**: Does it improve email security?
3. **Ease of Implementation**: Can we deliver it quickly?
4. **Strategic Alignment**: Does it align with our vision?
To suggest features:
- Open an issue on our [GitHub repository](https://github.com/yourusername/dmarq)
- Provide details about the feature and why it's valuable
- Include use cases and examples when possible
## Contribution Opportunities
We welcome contributions in these areas:
- **Integrations**: Help build integrations with other services
- **Documentation**: Improve guides, examples, and references
- **UI/UX**: Enhance the user interface and experience
- **Testing**: Add tests and improve test coverage
- **Performance**: Optimize database queries and processing
See our [Contributing Guide](contributing.md) for details on how to contribute.
## Release Schedule
- **Major Releases**: 2 per year (February and August)
- **Minor Releases**: Quarterly (February, May, August, November)
- **Patch Releases**: As needed for bug fixes and security updates
## Deprecation Policy
We maintain backward compatibility where possible, but sometimes need to deprecate features:
1. **Announcement**: We announce deprecations at least 6 months in advance
2. **Alternative**: We provide migration paths to alternative solutions
3. **Support**: We continue supporting deprecated features during the transition period
4. **Removal**: We remove features only in major version updates
## Feedback
We value your feedback on our roadmap! Please share your thoughts:
- Through GitHub issues
- In our community forums
- During community calls
- Via email to roadmap@example.com
docs/development/testing.md
+318
View File
@@ -0,0 +1,318 @@
# Testing
This guide covers the testing methodology for DMARQ, including unit tests, integration tests, and end-to-end testing.
## Testing Philosophy
DMARQ follows a comprehensive testing approach to ensure reliability:
- **Unit Tests**: Test individual functions and classes in isolation
- **Integration Tests**: Test components working together
- **End-to-End Tests**: Test the complete application flow
- **Performance Tests**: Ensure the system can handle expected load
## Test Structure
The test directory structure follows the application structure:
```
backend/app/tests/
├── conftest.py # Pytest fixtures and configuration
├── test_api.py # API endpoint tests
├── test_dmarc_parser.py # DMARC parser tests
├── test_models.py # Database model tests
├── test_reports_api.py # Reports API tests
├── unit/ # Unit tests
│ ├── test_domain_validator.py
│ ├── test_utils.py
│ └── ...
├── integration/ # Integration tests
│ ├── test_database.py
│ ├── test_imap.py
│ └── ...
└── e2e/ # End-to-end tests
├── test_report_flow.py
└── ...
```
## Setting Up the Test Environment
### Prerequisites
- Python 3.9+
- pytest and required plugins
### Installation
```bash
cd backend
pip install -r requirements-dev.txt
```
This will install:
- pytest
- pytest-cov (for coverage reports)
- pytest-mock (for mocking)
- pytest-asyncio (for async tests)
## Running Tests
### All Tests
To run all tests:
```bash
cd backend
pytest
```
### Specific Tests
To run specific test files:
```bash
pytest tests/test_dmarc_parser.py
```
To run tests matching a pattern:
```bash
pytest -k "parser" # Runs tests with "parser" in the name
```
### Test Coverage
To generate a coverage report:
```bash
pytest --cov=app
```
For an HTML coverage report:
```bash
pytest --cov=app --cov-report=html
```
Then open `htmlcov/index.html` to view the report.
## Writing Tests
### Fixtures
We use pytest fixtures for test setup and teardown. Common fixtures are defined in `conftest.py`:
```python
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.models.base import Base
from app.core.database import get_db
@pytest.fixture
def db_engine():
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
return engine
@pytest.fixture
def db_session(db_engine):
Session = sessionmaker(bind=db_engine)
session = Session()
yield session
session.close()
@pytest.fixture
def test_app(db_session):
from app.main import app
app.dependency_overrides[get_db] = lambda: db_session
return app
```
### Unit Tests
Unit tests should focus on testing a single function or class in isolation, using mocks for dependencies:
```python
from app.utils.domain_validator import is_valid_domain
import pytest
def test_is_valid_domain():
# Valid domains
assert is_valid_domain("example.com") is True
assert is_valid_domain("sub.example.com") is True
# Invalid domains
assert is_valid_domain("invalid..com") is False
assert is_valid_domain("a" * 300 + ".com") is False
```
### API Tests
API tests use the FastAPI TestClient:
```python
from fastapi.testclient import TestClient
def test_get_domains(test_app, db_session):
# Add test data to db_session
# ...
client = TestClient(test_app)
response = client.get("/api/v1/domains")
assert response.status_code == 200
data = response.json()
assert len(data["domains"]) == 2 # Assuming 2 domains were added
```
### Mocking
We use pytest-mock for mocking:
```python
def test_imap_client(mocker):
# Mock the imaplib.IMAP4_SSL class
mock_imap = mocker.patch("imaplib.IMAP4_SSL")
mock_imap.return_value.login.return_value = ("OK", [])
mock_imap.return_value.select.return_value = ("OK", [b"10"])
from app.services.imap_client import IMAPClient
client = IMAPClient("imap.example.com", "user", "pass")
result = client.connect()
assert result is True
mock_imap.return_value.login.assert_called_once()
```
### Testing Async Code
For async functions, use pytest-asyncio:
```python
import pytest
@pytest.mark.asyncio
async def test_async_function():
from app.services.report_processor import process_report_async
result = await process_report_async("test_data")
assert result is not None
```
## Testing Database Models
When testing database models, use an in-memory SQLite database:
```python
def test_domain_model(db_session):
from app.models.domain import Domain
domain = Domain(name="example.com")
db_session.add(domain)
db_session.commit()
fetched = db_session.query(Domain).filter_by(name="example.com").first()
assert fetched is not None
assert fetched.name == "example.com"
```
## Test Data
### Sample Files
Sample DMARC report files for testing are stored in:
```
backend/app/tests/data/
```
These include:
- Sample XML reports
- Compressed reports (ZIP, GZ)
- Invalid reports for error testing
### Factories
For generating test data, we use factory_boy:
```python
import factory
from app.models.domain import Domain
from app.models.report import Report
class DomainFactory(factory.Factory):
class Meta:
model = Domain
name = factory.Sequence(lambda n: f"domain-{n}.com")
active = True
class ReportFactory(factory.Factory):
class Meta:
model = Report
domain = factory.SubFactory(DomainFactory)
report_id = factory.Sequence(lambda n: f"report-{n}")
begin_date = factory.LazyFunction(lambda: datetime.now() - timedelta(days=1))
end_date = factory.LazyFunction(lambda: datetime.now())
org_name = "test-org"
```
## Continuous Integration
Tests are automatically run on every pull request using GitHub Actions.
The CI workflow:
1. Sets up the test environment
2. Runs linting checks
3. Runs the test suite
4. Generates coverage reports
5. Reports test results
## Performance Testing
For performance testing, we use Locust:
```bash
cd backend/performance_tests
locust -f locustfile.py
```
This starts a web interface at http://localhost:8089 to configure and run performance tests.
## Debugging Tests
When tests fail, you can use pytest's verbose mode for more details:
```bash
pytest -vv
```
For even more information, add the `-s` flag to show print statements:
```bash
pytest -vvs
```
## Writing Testable Code
To make testing easier:
1. **Dependency Injection**: Pass dependencies rather than creating them inside functions
2. **Single Responsibility**: Keep functions focused on a single task
3. **Pure Functions**: When possible, write pure functions that don't modify state
4. **Testable Units**: Structure code in small, testable units
5. **Configuration**: Make configuration injectable for tests
## Code Coverage Goals
Our coverage goals are:
- Overall coverage: 80%+
- Core modules: 90%+
- API endpoints: 100%
## Reporting Bugs
If you find a bug:
1. Write a failing test that reproduces the issue
2. File an issue describing the bug
3. Link the failing test in the issue
4. If possible, submit a PR with a fix
docs/faq.md
+204
View File
@@ -0,0 +1,204 @@
# Frequently Asked Questions
## General
### What is DMARQ?
DMARQ is a DMARC (Domain-based Message Authentication, Reporting, and Conformance) reporting and analysis tool. It helps organizations monitor their email authentication status, analyze DMARC reports, and improve email deliverability and security.
### Who should use DMARQ?
DMARQ is useful for:
- IT administrators who manage email systems
- Security professionals concerned with email security
- Organizations that want to monitor their DMARC compliance
- Email marketing teams who want to improve deliverability
### Is DMARQ open source?
Yes, DMARQ is open source software licensed under the MIT License. You can freely use, modify, and distribute it according to the terms of the license.
### How much does DMARQ cost?
DMARQ is free to use. As an open source project, there's no license fee. You only need to cover the costs of hosting the application on your own infrastructure.
## Setup and Installation
### What are the system requirements for DMARQ?
The minimum requirements are:
- 1 CPU core (2+ recommended for production)
- 2GB RAM (4GB+ recommended for production)
- 20GB storage
- Docker Engine 20.10.0+ (for Docker installation)
- Python 3.9+ (for manual installation)
### How do I install DMARQ?
There are two main ways to install DMARQ:
1. **Docker**: The recommended method using Docker Compose
2. **Manual Installation**: Traditional installation on a server
See our [Docker Setup](deployment/docker.md) or [Manual Installation](deployment/manual.md) guides for detailed instructions.
### Can I run DMARQ on shared hosting?
DMARQ requires the ability to run Docker containers or a Python application server. Most shared hosting environments don't provide this level of access, so you'll likely need a VPS or dedicated server.
### How do I update DMARQ to a new version?
For Docker installations:
```bash
git pull
docker-compose down
docker-compose up -d --build
```
For manual installations:
```bash
git pull
cd backend
pip install -r requirements.txt
cd app
alembic upgrade head
# Restart your application server
```
## DMARC Reports
### What are DMARC reports?
DMARC reports are feedback reports that email providers send to domain owners about emails they receive that claim to be from your domain. There are two types:
1. **Aggregate Reports (RUA)**: XML files with statistical data about email authentication results
2. **Forensic Reports (RUF)**: Detailed reports about specific authentication failures
### How do I receive DMARC reports?
You need to publish a DMARC record in your domain's DNS that includes your reporting email address. For example:
```
v=DMARC1; p=none; rua=mailto:dmarc@example.com; ruf=mailto:dmarc@example.com
```
### How does DMARQ collect DMARC reports?
DMARQ can collect reports in three ways:
1. **IMAP Integration**: Automatically fetching from your email inbox
2. **Manual Upload**: Uploading reports through the web interface
3. **API Upload**: Sending reports via the API
### How far back can I see DMARC data?
DMARQ stores all the data you import, so you can see historical data from the point you started collecting reports. There's no built-in limit to how far back data can be stored.
## Using DMARQ
### How do I add a domain to monitor?
1. Log in to DMARQ
2. Navigate to the Domains section
3. Click "Add Domain"
4. Enter your domain name and click "Add"
### Can I monitor multiple domains?
Yes, DMARQ supports monitoring an unlimited number of domains. You can add all the domains you want to track in the Domains section.
### How often are reports processed?
If you're using IMAP integration, reports are processed according to your configured polling interval (default is every 60 minutes). Manually uploaded reports are processed immediately.
### What should my DMARC policy be?
DMARQ can help you make this decision by showing your current compliance rate, but typically:
- Start with `p=none` to monitor without affecting delivery
- Move to `p=quarantine` when you reach 90%+ compliance
- Move to `p=reject` when you reach 95%+ compliance
## Technical Questions
### Can I use an external database?
Yes, DMARQ supports both SQLite (for smaller deployments) and PostgreSQL (for production). You can configure an external PostgreSQL database in your environment variables.
### How do I back up DMARQ data?
For SQLite:
```bash
cp data/dmarq.db backup-$(date +%Y%m%d).db
```
For PostgreSQL:
```bash
docker-compose exec db pg_dump -U dmarq dmarq > backup-$(date +%Y%m%d).sql
```
### Can I run DMARQ behind a reverse proxy?
Yes, DMARQ works well behind a reverse proxy like Nginx or Apache. This is recommended for production deployments. See our [Docker Setup](deployment/docker.md) guide for an example Nginx configuration.
### What API endpoints are available?
DMARQ provides a comprehensive REST API. See the [API Reference](reference/api.md) for detailed documentation of all available endpoints.
## Troubleshooting
### Why am I not seeing any reports?
Check the following:
1. Verify your DMARC record is correctly published in DNS
2. Ensure your reporting email is correctly set up
3. Check IMAP settings if using IMAP integration
4. Confirm that enough time has passed (can take 24-48 hours to start receiving reports)
### IMAP integration isn't working. What should I check?
1. Verify your IMAP server and port are correct
2. Check that your username and password are correct
3. Ensure IMAP access is enabled for your email account
4. If using Gmail, you may need to create an app-specific password
5. Check if your email provider requires special settings
### How do I report a bug?
You can report bugs on our [GitHub Issues](https://github.com/yourusername/dmarq/issues) page. Please include:
1. Steps to reproduce the issue
2. Expected behavior
3. Actual behavior
4. DMARQ version
5. Any relevant error messages
### I'm getting database errors. How do I fix them?
For SQLite:
1. Create a backup of your database file
2. Run the database integrity check: `sqlite3 data/dmarq.db "PRAGMA integrity_check;"`
For PostgreSQL:
1. Check database connection parameters
2. Verify the database server is running
3. Check for disk space issues
## Contributing
### How can I contribute to DMARQ?
There are many ways to contribute:
1. Code contributions (features, bug fixes)
2. Documentation improvements
3. Testing and bug reporting
4. Translations
5. Feature suggestions
See our [Contributing Guide](development/contributing.md) for more information.
### Where do I find the source code?
The source code is available on GitHub: [https://github.com/yourusername/dmarq](https://github.com/yourusername/dmarq)
### Is there a community forum?
We're building a community around DMARQ. For now, you can:
1. Join discussions in the GitHub repository
2. Ask questions in the issue tracker
3. Connect with other users in the #dmarq channel on the Email Security Discord
docs/reference/database.md
+281
View File
@@ -0,0 +1,281 @@
# Database Schema
This document describes the database schema used by DMARQ, including tables, relationships, and key fields.
## Overview
DMARQ uses a relational database to store all its data. The schema is designed to efficiently store and query DMARC report data, domain information, and system settings. The system supports both SQLite (for smaller deployments) and PostgreSQL (for production deployments).
## Schema Diagram
![Database Schema](../assets/images/database_schema.png)
## Core Tables
### Domains
The `domains` table stores information about the domains being monitored.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| name | VARCHAR(255) | Domain name (e.g., example.com) |
| created_at | TIMESTAMP | When the domain was added |
| active | BOOLEAN | Whether the domain is actively monitored |
| notes | TEXT | Optional notes about the domain |
| compliance_rate | FLOAT | Cached compliance rate |
| last_updated | TIMESTAMP | When data was last updated |
### Reports
The `reports` table stores metadata about DMARC aggregate reports.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| domain_id | INTEGER | Foreign key to domains.id |
| report_id | VARCHAR(255) | Original report ID from the provider |
| begin_date | TIMESTAMP | Start of report period |
| end_date | TIMESTAMP | End of report period |
| org_name | VARCHAR(255) | Organization that sent the report |
| email | VARCHAR(255) | Email that sent the report |
| processed_at | TIMESTAMP | When the report was processed |
| extra_contact_info | VARCHAR(255) | Additional contact info, if provided |
| error | TEXT | Error information if processing failed |
| raw_xml | TEXT | Original XML report (optional, can be disabled) |
### Report_Records
The `report_records` table stores individual authentication results from reports.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| report_id | INTEGER | Foreign key to reports.id |
| source_ip | VARCHAR(45) | Source IP address |
| count | INTEGER | Count of messages |
| disposition | VARCHAR(10) | Policy applied (none, quarantine, reject) |
| dkim_aligned | BOOLEAN | Whether DKIM alignment passed |
| spf_aligned | BOOLEAN | Whether SPF alignment passed |
| passed | BOOLEAN | Whether overall DMARC passed |
| header_from | VARCHAR(255) | Domain in From header |
| envelope_from | VARCHAR(255) | Domain in envelope From |
| envelope_to | VARCHAR(255) | Domain in envelope To |
### Forensic_Reports
The `forensic_reports` table stores DMARC forensic reports.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| domain_id | INTEGER | Foreign key to domains.id |
| report_id | VARCHAR(255) | Original report ID |
| date | TIMESTAMP | When the report was generated |
| source_ip | VARCHAR(45) | Source IP address |
| source_hostname | VARCHAR(255) | Source hostname, if available |
| failure_type | VARCHAR(20) | Type of auth failure (dkim, spf, both) |
| auth_failure_detail | TEXT | Detailed reason for failure |
| delivery_action | VARCHAR(20) | Action taken (delivered, quarantined, rejected) |
| subject | VARCHAR(255) | Email subject |
| processed_at | TIMESTAMP | When the report was processed |
| headers | TEXT | Email headers |
| original_mail | TEXT | Original email content (if available) |
### IP_Info
The `ip_info` table caches information about IP addresses.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| ip | VARCHAR(45) | IP address |
| hostname | VARCHAR(255) | Resolved hostname |
| country | VARCHAR(2) | Country code |
| asn | INTEGER | Autonomous System Number |
| org | VARCHAR(255) | Organization name |
| last_updated | TIMESTAMP | When the data was last updated |
## User and Authentication Tables
### Users
The `users` table stores user account information.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| username | VARCHAR(50) | Username |
| email | VARCHAR(255) | Email address |
| password_hash | VARCHAR(255) | Hashed password |
| full_name | VARCHAR(100) | Full name |
| is_active | BOOLEAN | Whether the account is active |
| is_admin | BOOLEAN | Whether the user is an administrator |
| created_at | TIMESTAMP | When the account was created |
| last_login | TIMESTAMP | When the user last logged in |
### API_Keys
The `api_keys` table stores API keys for programmatic access.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| key_hash | VARCHAR(255) | Hashed API key |
| user_id | INTEGER | Foreign key to users.id |
| name | VARCHAR(100) | Name/description of the key |
| created_at | TIMESTAMP | When the key was created |
| expires_at | TIMESTAMP | When the key expires (optional) |
| last_used | TIMESTAMP | When the key was last used |
| permissions | TEXT | JSON array of permissions |
## DNS and Configuration Tables
### DNS_Records
The `dns_records` table stores DNS record information for domains.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| domain_id | INTEGER | Foreign key to domains.id |
| record_type | VARCHAR(10) | Type of record (SPF, DMARC, DKIM, MX, etc.) |
| value | TEXT | Value of the DNS record |
| status | VARCHAR(20) | Status (valid, invalid, warning) |
| last_checked | TIMESTAMP | When the record was last checked |
| dkim_selector | VARCHAR(50) | Selector (for DKIM records) |
### Settings
The `settings` table stores system-wide settings.
| Column | Type | Description |
|--------|------|-------------|
| key | VARCHAR(100) | Setting key (primary key) |
| value | TEXT | Setting value |
| description | VARCHAR(255) | Description of the setting |
| type | VARCHAR(20) | Data type (string, integer, boolean, json) |
| updated_at | TIMESTAMP | When the setting was last updated |
| updated_by | INTEGER | Foreign key to users.id |
## Relationship Tables
### User_Domain_Access
The `user_domain_access` table manages user permissions for specific domains.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| user_id | INTEGER | Foreign key to users.id |
| domain_id | INTEGER | Foreign key to domains.id |
| permission | VARCHAR(20) | Permission level (view, edit, admin) |
### Domain_Groups
The `domain_groups` table defines groups of domains.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| name | VARCHAR(100) | Group name |
| description | TEXT | Group description |
| created_by | INTEGER | Foreign key to users.id |
| created_at | TIMESTAMP | When the group was created |
### Domain_Group_Members
The `domain_group_members` table assigns domains to groups.
| Column | Type | Description |
|--------|------|-------------|
| group_id | INTEGER | Foreign key to domain_groups.id |
| domain_id | INTEGER | Foreign key to domains.id |
## Logging Tables
### Activity_Logs
The `activity_logs` table records user actions.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| user_id | INTEGER | Foreign key to users.id (null for system) |
| action | VARCHAR(50) | Type of action performed |
| entity_type | VARCHAR(50) | Type of entity affected (domain, report, user) |
| entity_id | INTEGER | ID of the affected entity |
| details | TEXT | JSON with additional details |
| timestamp | TIMESTAMP | When the action occurred |
| ip_address | VARCHAR(45) | IP address of the user |
### System_Logs
The `system_logs` table records system events.
| Column | Type | Description |
|--------|------|-------------|
| id | INTEGER | Primary key |
| level | VARCHAR(10) | Log level (info, warning, error, debug) |
| message | TEXT | Log message |
| component | VARCHAR(50) | Component that generated the log |
| timestamp | TIMESTAMP | When the event occurred |
| additional_data | TEXT | JSON with additional data |
## Indexes
The schema includes several indexes to optimize query performance:
- `idx_reports_domain_id`: On reports.domain_id
- `idx_reports_begin_date`: On reports.begin_date
- `idx_reports_end_date`: On reports.end_date
- `idx_report_records_report_id`: On report_records.report_id
- `idx_report_records_source_ip`: On report_records.source_ip
- `idx_users_username`: On users.username
- `idx_users_email`: On users.email
- `idx_domains_name`: On domains.name
- `idx_api_keys_key_hash`: On api_keys.key_hash
- `idx_activity_logs_timestamp`: On activity_logs.timestamp
- `idx_activity_logs_user_id`: On activity_logs.user_id
- `idx_system_logs_timestamp`: On system_logs.timestamp
- `idx_system_logs_level`: On system_logs.level
## Migrations
Database migrations are managed using Alembic, which provides:
- Version control for the database schema
- Automatic schema updates during application upgrades
- Ability to roll back changes if needed
- Generation of new migration scripts for schema changes
To apply migrations:
```bash
cd backend/app
python -m alembic upgrade head
```
To create a new migration after schema changes:
```bash
python -m alembic revision --autogenerate -m "Description of changes"
```
## Query Optimization
The database schema is designed with query optimization in mind:
- Frequently used fields have indexes
- Historical data can be efficiently queried by date ranges
- Counters and aggregate data are cached where appropriate
- Domain-specific data is properly segmented
## Database Backup
Regular backups of the database should be configured:
- For SQLite: Simple file copy or SQLite's `.backup` command
- For PostgreSQL: `pg_dump` command or continuous archiving with WAL
See the [Deployment Guide](../deployment/docker.md) for more information on database backup strategies.