gh-christianlouis-dmarq/docs/development/testing.md

Testing

This guide covers the testing methodology for DMARQ, including unit tests, integration tests, and end-to-end testing.

Testing Philosophy

DMARQ follows a comprehensive testing approach to ensure reliability:

• Unit Tests: Test individual functions and classes in isolation
• Integration Tests: Test components working together
• End-to-End Tests: Test the complete application flow
• Performance Tests: Ensure the system can handle expected load

Test Structure

The test directory structure follows the application structure:

backend/app/tests/
├── conftest.py              # Pytest fixtures and configuration
├── test_api.py              # API endpoint tests
├── test_dmarc_parser.py     # DMARC parser tests
├── test_models.py           # Database model tests
├── test_reports_api.py      # Reports API tests
├── unit/                    # Unit tests
│   ├── test_domain_validator.py
│   ├── test_utils.py
│   └── ...
├── integration/             # Integration tests
│   ├── test_database.py
│   ├── test_imap.py
│   └── ...
└── e2e/                     # End-to-end tests
    ├── test_report_flow.py
    └── ...

Setting Up the Test Environment

Prerequisites

• Python 3.9+
• pytest and required plugins

Installation

cd backend
pip install -r requirements-dev.txt

This will install:

• pytest
• pytest-cov (for coverage reports)
• pytest-mock (for mocking)
• pytest-asyncio (for async tests)

Running Tests

All Tests

To run all tests:

cd backend
pytest

Specific Tests

To run specific test files:

pytest tests/test_dmarc_parser.py

To run tests matching a pattern:

pytest -k "parser"  # Runs tests with "parser" in the name

Test Coverage

To generate a coverage report:

pytest --cov=app

For an HTML coverage report:

pytest --cov=app --cov-report=html

Then open htmlcov/index.html to view the report.

Writing Tests

Fixtures

We use pytest fixtures for test setup and teardown. Common fixtures are defined in conftest.py:

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from app.models.base import Base
from app.core.database import get_db

@pytest.fixture
def db_engine():
    engine = create_engine("sqlite:///:memory:")
    Base.metadata.create_all(engine)
    return engine

@pytest.fixture
def db_session(db_engine):
    Session = sessionmaker(bind=db_engine)
    session = Session()
    yield session
    session.close()

@pytest.fixture
def test_app(db_session):
    from app.main import app
    app.dependency_overrides[get_db] = lambda: db_session
    return app

Unit Tests

Unit tests should focus on testing a single function or class in isolation, using mocks for dependencies:

from app.utils.domain_validator import is_valid_domain
import pytest

def test_is_valid_domain():
    # Valid domains
    assert is_valid_domain("example.com") is True
    assert is_valid_domain("sub.example.com") is True
    
    # Invalid domains
    assert is_valid_domain("invalid..com") is False
    assert is_valid_domain("a" * 300 + ".com") is False

API Tests

API tests use the FastAPI TestClient:

from fastapi.testclient import TestClient

def test_get_domains(test_app, db_session):
    # Add test data to db_session
    # ...
    
    client = TestClient(test_app)
    response = client.get("/api/v1/domains")
    assert response.status_code == 200
    data = response.json()
    assert len(data["domains"]) == 2  # Assuming 2 domains were added

Mocking

We use pytest-mock for mocking:

def test_imap_client(mocker):
    # Mock the imaplib.IMAP4_SSL class
    mock_imap = mocker.patch("imaplib.IMAP4_SSL")
    mock_imap.return_value.login.return_value = ("OK", [])
    mock_imap.return_value.select.return_value = ("OK", [b"10"])
    
    from app.services.imap_client import IMAPClient
    client = IMAPClient("imap.example.com", "user", "pass")
    result = client.connect()
    
    assert result is True
    mock_imap.return_value.login.assert_called_once()

Testing Async Code

For async functions, use pytest-asyncio:

import pytest

@pytest.mark.asyncio
async def test_async_function():
    from app.services.report_processor import process_report_async
    result = await process_report_async("test_data")
    assert result is not None

Testing Database Models

When testing database models, use an in-memory SQLite database:

def test_domain_model(db_session):
    from app.models.domain import Domain
    
    domain = Domain(name="example.com")
    db_session.add(domain)
    db_session.commit()
    
    fetched = db_session.query(Domain).filter_by(name="example.com").first()
    assert fetched is not None
    assert fetched.name == "example.com"

Test Data

Sample Files

Sample DMARC report files for testing are stored in:

backend/app/tests/data/

These include:

• Sample XML reports
• Compressed reports (ZIP, GZ)
• Invalid reports for error testing

Factories

For generating test data, we use factory_boy:

import factory
from app.models.domain import Domain
from app.models.report import Report

class DomainFactory(factory.Factory):
    class Meta:
        model = Domain
    
    name = factory.Sequence(lambda n: f"domain-{n}.com")
    active = True

class ReportFactory(factory.Factory):
    class Meta:
        model = Report
    
    domain = factory.SubFactory(DomainFactory)
    report_id = factory.Sequence(lambda n: f"report-{n}")
    begin_date = factory.LazyFunction(lambda: datetime.now() - timedelta(days=1))
    end_date = factory.LazyFunction(lambda: datetime.now())
    org_name = "test-org"

Continuous Integration

Tests are automatically run on every pull request using GitHub Actions.

The CI workflow:

1. Sets up the test environment
2. Runs linting checks
3. Runs the test suite
4. Generates coverage reports
5. Reports test results

Performance Testing

For performance testing, we use Locust:

cd backend/performance_tests
locust -f locustfile.py

This starts a web interface at http://localhost:8089 to configure and run performance tests.

Debugging Tests

When tests fail, you can use pytest's verbose mode for more details:

pytest -vv

For even more information, add the -s flag to show print statements:

pytest -vvs

Writing Testable Code

To make testing easier:

1. Dependency Injection: Pass dependencies rather than creating them inside functions
2. Single Responsibility: Keep functions focused on a single task
3. Pure Functions: When possible, write pure functions that don't modify state
4. Testable Units: Structure code in small, testable units
5. Configuration: Make configuration injectable for tests

Code Coverage Goals

Our coverage goals are:

• Overall coverage: 80%+
• Core modules: 90%+
• API endpoints: 100%

Reporting Bugs

If you find a bug:

1. Write a failing test that reproduces the issue
2. File an issue describing the bug
3. Link the failing test in the issue
4. If possible, submit a PR with a fix