gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add comprehensive contributing, testing, roadmap, and database schema documentation for DMARQ

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-04-21 02:51:17 +02:00
parent ae79cca8c1
commit a0ddadfaa1
6 changed files with 1284 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/development/contributing.md
+229
View File
@@ -0,0 +1,229 @@
# Contributing to DMARQ
Thank you for your interest in contributing to DMARQ! This guide will help you get started with the development process.
## Code of Conduct
Please read and follow our [Code of Conduct](https://github.com/yourusername/dmarq/blob/main/CODE_OF_CONDUCT.md) to keep our community approachable and respectable.
## How to Contribute
There are many ways to contribute to DMARQ:
- **Reporting bugs**: Submit detailed bug reports to help us improve
- **Suggesting features**: Propose new features or improvements
- **Writing code**: Contribute code changes or new features
- **Improving docs**: Help make our documentation more comprehensive
- **Translation**: Help translate the interface into other languages
## Development Environment Setup
### Prerequisites
- Python 3.9+
- Node.js 16+ (for frontend assets)
- Docker and Docker Compose (recommended)
- Git
### Setting Up the Project
1. **Fork the repository**
Start by forking the [DMARQ repository](https://github.com/yourusername/dmarq) on GitHub.
2. **Clone your fork**
```bash
git clone https://github.com/YOUR-USERNAME/dmarq.git
cd dmarq
```
3. **Set up the development environment**
Using Docker (recommended):
```bash
docker-compose -f docker-compose.dev.yml up
```
Or manually:
```bash
# Create a virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
cd backend
pip install -r requirements.txt
pip install -r requirements-dev.txt
# Set up the database
cd app
python -m alembic upgrade head
# Start the development server
uvicorn main:app --reload --host 0.0.0.0 --port 8000
```
4. **Frontend Assets (if modifying)**
If you're modifying frontend assets:
```bash
cd backend/app/static
npm install
npm run dev
```
## Making Changes
### Branching Strategy
We follow a simple branching strategy:
- `main` branch is the stable release branch
- `develop` branch is for development work
- Feature branches should be created from `develop`
### Creating a Branch
Create a new branch for your changes:
```bash
git checkout develop
git pull origin develop
git checkout -b feature/your-feature-name
```
Use prefixes like:
- `feature/` for new features
- `bugfix/` for bug fixes
- `docs/` for documentation changes
- `test/` for test improvements
### Coding Standards
We follow these standards:
- **Python**: [PEP 8](https://www.python.org/dev/peps/pep-0008/) style guide
- **JavaScript**: ESLint with Airbnb style
- **HTML/CSS**: Follow the project's existing patterns
We use pre-commit hooks to enforce coding standards:
```bash
pip install pre-commit
pre-commit install
```
### Testing
All code changes should include tests:
```bash
# Run the test suite
cd backend
pytest
# With coverage
pytest --cov=app
```
## Submitting a Pull Request
1. **Update your branch**
```bash
git fetch origin
git rebase origin/develop
```
2. **Run tests**
Ensure all tests pass before submitting:
```bash
pytest
```
3. **Commit your changes**
Follow the [Conventional Commits](https://www.conventionalcommits.org/) standard:
```bash
git commit -m "feat: add user authentication"
```
4. **Push to your fork**
```bash
git push origin feature/your-feature-name
```
5. **Submit a pull request**
Go to the [DMARQ repository](https://github.com/yourusername/dmarq) and create a pull request from your branch to the `develop` branch.
Include in your PR description:
- What changes you've made
- Why you've made these changes
- Any relevant issue numbers (e.g., "Fixes #123")
- Screenshots if applicable
6. **Code review**
Maintainers will review your code. You might need to make additional changes based on feedback.
## Pull Request Review Process
Pull requests are reviewed by maintainers who will check:
1. Code quality and style
2. Test coverage
3. Documentation
4. Overall fit with the project goals
## Release Process
We use semantic versioning (MAJOR.MINOR.PATCH):
- MAJOR version for incompatible API changes
- MINOR version for new functionality in a backwards compatible manner
- PATCH version for backwards compatible bug fixes
## Documentation
Please update documentation alongside code changes:
- Update relevant parts of this documentation site
- Add or update docstrings
- Update README.md if needed
To build and preview the documentation:
```bash
# Install mkdocs and requirements
pip install -r docs/readthedocs/requirements.txt
# Serve documentation locally
mkdocs serve
```
## Additional Resources
- [Project Architecture](../reference/architecture.md)
- [Database Schema](../reference/database.md)
- [API Reference](../reference/api.md)
## Getting Help
If you need help with your contribution, you can:
- Open an issue on GitHub
- Join our community channels
- Email the maintainers at maintainers@example.com
## Recognition
All contributors are recognized in our [CONTRIBUTORS.md](https://github.com/yourusername/dmarq/blob/main/CONTRIBUTORS.md) file. We appreciate your help in making DMARQ better!
docs/development/roadmap.md
+201
View File
@@ -0,0 +1,201 @@
# Roadmap
This document outlines the planned development roadmap for DMARQ, including upcoming features, improvements, and long-term goals.
## Current Version: 1.0.0 (April 2025)
The initial release of DMARQ includes:
- Basic DMARC report processing and analysis
- Domain management
- User authentication
- Dashboard with key metrics
- IMAP integration for automatic report collection
- Simple alerting system
- Docker deployment option
## Short-Term Goals (Q2-Q3 2025)
### Version 1.1.0 (June 2025)
- **Advanced Report Filtering**
- Filter reports by IP address
- Filter by authentication result
- Custom date range selection
- Save custom filters
- **Improved Visualizations**
- Interactive charts with drill-down capability
- Geographic IP distribution map
- Timeline view of authentication changes
- **Enhanced DNS Health Checks**
- Automated SPF, DKIM, DMARC syntax validation
- Record monitoring with change detection
- Best practice recommendations
- **API Enhancements**
- Additional endpoints for statistics
- Improved authentication options
- Better documentation and examples
### Version 1.2.0 (August 2025)
- **User Management Improvements**
- Role-based access control
- Domain-specific permissions
- User invitation system
- Activity audit logging
- **Multi-tenant Support**
- Organization-level grouping of domains
- Isolated views for different user groups
- White-labeling options
- **Enhanced IMAP Integration**
- Support for multiple mailboxes
- Advanced filtering options
- Attachment preprocessing rules
- **Forensic Report Analysis**
- Improved parsing for various report formats
- Header analysis tools
- Correlation with aggregate reports
## Mid-Term Goals (Q4 2025 - Q1 2026)
### Version 1.3.0 (November 2025)
- **Integration Ecosystem**
- Slack/Teams notifications
- WebHook support for custom integrations
- Export to BI tools
- SIEM integration
- **Advanced Alerting System**
- Custom alert rules
- Alert severity levels
- Alert acknowledgment workflow
- Historical alert tracking
- **DNS Management**
- Integration with Cloudflare API
- Integration with AWS Route 53
- One-click fix for common DNS issues
- DNS record deployment tracking
- **Report Anomaly Detection**
- Machine learning-based anomaly detection
- Unusual sending pattern identification
- Automatic threat scoring
### Version 2.0.0 (February 2026)
- **Comprehensive Email Authentication Suite**
- SPF record management and monitoring
- DKIM key rotation management
- BIMI record support
- MTA-STS implementation assistance
- **Policy Management**
- DMARC policy transition recommendations
- Automated policy progression
- Impact analysis before policy changes
- Rollback capabilities
- **Reporting Enhancements**
- Scheduled PDF/CSV exports
- Custom report templates
- Executive summary generation
- Trend analysis with predictive insights
- **Multi-Channel Notifications**
- Email notifications
- SMS alerts
- Mobile app push notifications
- Custom notification channels
## Long-Term Goals (Mid 2026+)
### Version 2.x and Beyond
- **Advanced Threat Intelligence**
- Integration with email security platforms
- Shared threat database
- Sender reputation scoring
- Proactive security recommendations
- **Enterprise Features**
- LDAP/Active Directory integration
- SAML/SSO support
- Advanced audit logging
- Custom branding
- **Internationalization**
- Multi-language interface
- Region-specific reporting
- International domain support (IDN)
- Localized documentation
- **AI-Powered Analysis**
- Natural language querying of report data
- Automated root cause analysis
- Predictive compliance modeling
- AI-assisted remediation recommendations
- **Ecosystem Expansion**
- Mobile companion app
- Browser plugins
- Desktop notifications
- Command-line tools
## Feature Requests and Prioritization
We prioritize features based on:
1. **User Impact**: How many users will benefit?
2. **Security Enhancement**: Does it improve email security?
3. **Ease of Implementation**: Can we deliver it quickly?
4. **Strategic Alignment**: Does it align with our vision?
To suggest features:
- Open an issue on our [GitHub repository](https://github.com/yourusername/dmarq)
- Provide details about the feature and why it's valuable
- Include use cases and examples when possible
## Contribution Opportunities
We welcome contributions in these areas:
- **Integrations**: Help build integrations with other services
- **Documentation**: Improve guides, examples, and references
- **UI/UX**: Enhance the user interface and experience
- **Testing**: Add tests and improve test coverage
- **Performance**: Optimize database queries and processing
See our [Contributing Guide](contributing.md) for details on how to contribute.
## Release Schedule
- **Major Releases**: 2 per year (February and August)
- **Minor Releases**: Quarterly (February, May, August, November)
- **Patch Releases**: As needed for bug fixes and security updates
## Deprecation Policy
We maintain backward compatibility where possible, but sometimes need to deprecate features:
1. **Announcement**: We announce deprecations at least 6 months in advance
2. **Alternative**: We provide migration paths to alternative solutions
3. **Support**: We continue supporting deprecated features during the transition period
4. **Removal**: We remove features only in major version updates
## Feedback
We value your feedback on our roadmap! Please share your thoughts:
- Through GitHub issues
- In our community forums
- During community calls
- Via email to roadmap@example.com
docs/development/testing.md
+318
View File
@@ -0,0 +1,318 @@
# Testing
This guide covers the testing methodology for DMARQ, including unit tests, integration tests, and end-to-end testing.
## Testing Philosophy
DMARQ follows a comprehensive testing approach to ensure reliability:
- **Unit Tests**: Test individual functions and classes in isolation
- **Integration Tests**: Test components working together
- **End-to-End Tests**: Test the complete application flow
- **Performance Tests**: Ensure the system can handle expected load
## Test Structure
The test directory structure follows the application structure:
```
backend/app/tests/
├── conftest.py # Pytest fixtures and configuration
├── test_api.py # API endpoint tests
├── test_dmarc_parser.py # DMARC parser tests
├── test_models.py # Database model tests
├── test_reports_api.py # Reports API tests
├── unit/ # Unit tests
│ ├── test_domain_validator.py
│ ├── test_utils.py
│ └── ...
├── integration/ # Integration tests
│ ├── test_database.py
│ ├── test_imap.py
│ └── ...
└── e2e/ # End-to-end tests
├── test_report_flow.py
└── ...
```
## Setting Up the Test Environment
### Prerequisites
- Python 3.9+
- pytest and required plugins
### Installation
```bash
cd backend
pip install -r requirements-dev.txt
```
This will install:
- pytest
- pytest-cov (for coverage reports)
- pytest-mock (for mocking)
- pytest-asyncio (for async tests)
## Running Tests
### All Tests
To run all tests:
```bash
cd backend
pytest
```
### Specific Tests
To run specific test files:
```bash
pytest tests/test_dmarc_parser.py
```
To run tests matching a pattern:
```bash
pytest -k "parser" # Runs tests with "parser" in the name
```
### Test Coverage
To generate a coverage report:
```bash
pytest --cov=app
```
For an HTML coverage report:
```bash
pytest --cov=app --cov-report=html
```
Then open `htmlcov/index.html` to view the report.
## Writing Tests
### Fixtures
We use pytest fixtures for test setup and teardown. Common fixtures are defined in `conftest.py`:
```python
import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.models.base import Base
from app.core.database import get_db
@pytest.fixture
def db_engine():
engine = create_engine("sqlite:///:memory:")
Base.metadata.create_all(engine)
return engine
@pytest.fixture
def db_session(db_engine):
Session = sessionmaker(bind=db_engine)
session = Session()
yield session
session.close()
@pytest.fixture
def test_app(db_session):
from app.main import app
app.dependency_overrides[get_db] = lambda: db_session
return app
```
### Unit Tests
Unit tests should focus on testing a single function or class in isolation, using mocks for dependencies:
```python
from app.utils.domain_validator import is_valid_domain
import pytest
def test_is_valid_domain():
# Valid domains
assert is_valid_domain("example.com") is True
assert is_valid_domain("sub.example.com") is True
# Invalid domains
assert is_valid_domain("invalid..com") is False
assert is_valid_domain("a" * 300 + ".com") is False
```
### API Tests
API tests use the FastAPI TestClient:
```python
from fastapi.testclient import TestClient
def test_get_domains(test_app, db_session):
# Add test data to db_session
# ...
client = TestClient(test_app)
response = client.get("/api/v1/domains")
assert response.status_code == 200
data = response.json()
assert len(data["domains"]) == 2 # Assuming 2 domains were added
```
### Mocking
We use pytest-mock for mocking:
```python
def test_imap_client(mocker):
# Mock the imaplib.IMAP4_SSL class
mock_imap = mocker.patch("imaplib.IMAP4_SSL")
mock_imap.return_value.login.return_value = ("OK", [])
mock_imap.return_value.select.return_value = ("OK", [b"10"])
from app.services.imap_client import IMAPClient
client = IMAPClient("imap.example.com", "user", "pass")
result = client.connect()
assert result is True
mock_imap.return_value.login.assert_called_once()
```
### Testing Async Code
For async functions, use pytest-asyncio:
```python
import pytest
@pytest.mark.asyncio
async def test_async_function():
from app.services.report_processor import process_report_async
result = await process_report_async("test_data")
assert result is not None
```
## Testing Database Models
When testing database models, use an in-memory SQLite database:
```python
def test_domain_model(db_session):
from app.models.domain import Domain
domain = Domain(name="example.com")
db_session.add(domain)
db_session.commit()
fetched = db_session.query(Domain).filter_by(name="example.com").first()
assert fetched is not None
assert fetched.name == "example.com"
```
## Test Data
### Sample Files
Sample DMARC report files for testing are stored in:
```
backend/app/tests/data/
```
These include:
- Sample XML reports
- Compressed reports (ZIP, GZ)
- Invalid reports for error testing
### Factories
For generating test data, we use factory_boy:
```python
import factory
from app.models.domain import Domain
from app.models.report import Report
class DomainFactory(factory.Factory):
class Meta:
model = Domain
name = factory.Sequence(lambda n: f"domain-{n}.com")
active = True
class ReportFactory(factory.Factory):
class Meta:
model = Report
domain = factory.SubFactory(DomainFactory)
report_id = factory.Sequence(lambda n: f"report-{n}")
begin_date = factory.LazyFunction(lambda: datetime.now() - timedelta(days=1))
end_date = factory.LazyFunction(lambda: datetime.now())
org_name = "test-org"
```
## Continuous Integration
Tests are automatically run on every pull request using GitHub Actions.
The CI workflow:
1. Sets up the test environment
2. Runs linting checks
3. Runs the test suite
4. Generates coverage reports
5. Reports test results
## Performance Testing
For performance testing, we use Locust:
```bash
cd backend/performance_tests
locust -f locustfile.py
```
This starts a web interface at http://localhost:8089 to configure and run performance tests.
## Debugging Tests
When tests fail, you can use pytest's verbose mode for more details:
```bash
pytest -vv
```
For even more information, add the `-s` flag to show print statements:
```bash
pytest -vvs
```
## Writing Testable Code
To make testing easier:
1. **Dependency Injection**: Pass dependencies rather than creating them inside functions
2. **Single Responsibility**: Keep functions focused on a single task
3. **Pure Functions**: When possible, write pure functions that don't modify state
4. **Testable Units**: Structure code in small, testable units
5. **Configuration**: Make configuration injectable for tests
## Code Coverage Goals
Our coverage goals are:
- Overall coverage: 80%+
- Core modules: 90%+
- API endpoints: 100%
## Reporting Bugs
If you find a bug:
1. Write a failing test that reproduces the issue
2. File an issue describing the bug
3. Link the failing test in the issue
4. If possible, submit a PR with a fix