gh-christianlouis-dmarq/docs/plan.md

Perfect, I’ll start building an MVP for DMARQ with the full backend/frontend integration, IMAP ingestion and DMARC parsing, a web-based config wizard, and a modular design ready for Cloudflare integration and Apprise-based alerting later.

I’ll begin scaffolding the repository and project components. I’ll let you know once the MVP structure is ready for review and iteration.

# DMARQ: DMARC Monitoring Platform MVP

**DMARQ** is a full-stack DMARC monitoring platform designed to help organizations track and improve their email authentication. It processes DMARC reports (aggregate and forensic) and presents compliance insights via a user-friendly dashboard. This MVP will consist of a FastAPI backend to handle data collection, processing, and an authentication API, and a frontend built with Jinja2 templates, HTMX, and Alpine.js for an interactive dashboard and setup wizard. The platform follows clean code practices, uses modern libraries, and adheres to branding guidelines (DMARQ name, color scheme, and fonts) for a professional look.

## Backend (FastAPI) Architecture

The backend is built with **FastAPI**, chosen for its performance and intuitive async support for I/O tasks (like IMAP email fetching). The backend’s responsibilities include user authentication, scheduled retrieval of DMARC reports via IMAP, parsing those reports (both aggregate and forensic per RFC standards), storing results in a PostgreSQL database, and exposing APIs for the frontend. The code is organized into modular components (auth, polling, parsing, storage, alerts, etc.) to ensure maintainability and extensibility.

### Authentication with FastAPI Users

User management is handled by the **FastAPI Users** library, which provides ready-made routes and utilities for authentication. FastAPI Users supports JWT or cookie-based auth and integrates with SQLAlchemy for persistence ([GitHub - fastapi-users/fastapi-users: Ready-to-use and customizable users management for FastAPI](https://github.com/fastapi-users/fastapi-users#:~:text=Add%20quickly%20a%20registration%20and,customizable%20and%20adaptable%20as%20possible)). This gives DMARQ a secure registration and login system out-of-the-box, including endpoints for user signup, login, password reset, and email verification. For example, the FastAPI app can include the library’s routers for auth like so: 

```python
# inside backend/app.py
app = FastAPI()
app.include_router(fastapi_users.get_auth_router(auth_backend), prefix="/auth/jwt", tags=["auth"])
app.include_router(fastapi_users.get_register_router(UserRead, UserCreate), prefix="/auth", tags=["auth"])
app.include_router(fastapi_users.get_reset_password_router(), prefix="/auth", tags=["auth"])
app.include_router(fastapi_users.get_verify_router(UserRead), prefix="/auth", tags=["auth"])
app.include_router(fastapi_users.get_users_router(UserRead, UserUpdate), prefix="/users", tags=["users"])
```

Using this approach, DMARQ’s backend instantly has secure JWT authentication and user management APIs ([Full example - FastAPI Users](https://fastapi-users.github.io/fastapi-users/10.1/configuration/full-example/#:~:text=app.include_router%28%20fastapi_users.get_auth_router%28auth_backend%29%2C%20prefix%3D,%29%20app.include_router)). The user model extends the base model from FastAPI Users (allowing fields like organization or role if needed). This library ensures password hashing, validation, and token generation are implemented following best practices, so the team can focus on DMARC-specific logic.

### IMAP Polling for DMARC Reports

To gather DMARC data, DMARQ polls an IMAP mailbox where aggregate (RUA) and forensic (RUF) reports are sent. Using Python’s IMAP libraries (e.g. `imaplib` or `IMAPClient`), the backend periodically connects to the configured mail server (interval configurable, e.g. every hour) to fetch new emails. It will search for unread messages from known reporter addresses or with subjects indicating DMARC reports (many reports include the domain and date). Each email’s attachments are then processed:

- **Aggregate reports** (RUA) are typically attached as XML files (often zipped). The backend will detect attachments with MIME type “application/zip” or “application/gzip” and unzip them to obtain XML, or directly XML attachments. These XML files contain summary data about many emails’ authentication results.
- **Forensic reports** (RUF) are individual failure reports usually in an Abuse Report Format (ARF, per RFC 6591) – essentially an email message or part of one that failed DMARC. These may be included as `.eml` attachments or in the email body.

The app uses a background task (such as FastAPI’s **BackgroundTasks** or a dedicated Celery/RQ worker if scaling out) to perform IMAP polling so as not to block request handling. Upon retrieving new reports, the emails are marked as read (so they aren’t processed again), and then passed to the parsing module.

### Parsing DMARC Aggregate Reports (RFC 7489)

DMARC **aggregate reports** are XML documents that provide batched statistics about email authentication results ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=DMARC%20aggregate%20reports%20are%20XML,sensitive%20information%20about%20email%20messages)). Each report covers a period (usually 24 hours) and is sent by receiving servers to the domain’s rua address. These reports include fields such as the reporting organization, the sender domain, the DKIM/SPF alignment results, counts of emails, and the DMARC policy applied. Importantly, aggregate reports **do not contain personal email content** – they just summarize authentication data.

The backend will parse the XML according to RFC 7489 structure. Key XML elements include: `<report_metadata>` (report ID, date range, reporter info), `<policy_published>` (the DMARC record of the domain, policy in effect), and multiple `<record>` entries which each contain an `<row>` (source IP, count, disposition, DKIM/SPF pass/fail) and corresponding `<identifiers>` (like the header-from domain) and `<auth_results>` (details on SPF/DKIM checks). 

To implement parsing, the team can either write an XML parser using Python’s `xml.etree` or `defusedxml` (for security) or leverage an existing DMARC parsing library. One great option is **parsedmarc**, an open-source Python module that parses aggregate (and forensic) reports and even handles compressed files transparently ([parsedmarc documentation - Open source DMARC report analyzer and visualizer — parsedmarc 8.18.1 documentation](https://domainaware.github.io/parsedmarc/#:~:text=,standard%20aggregate%2Frua%20reports)). For example, parsedmarc can be used to parse XML content into Python dicts. Using a library like this speeds up development and ensures compliance with DMARC report formats. If using parsedmarc, the flow would be: download attachments -> feed them to parsedmarc’s `parse_report_file` or `parse_aggregate_report_xml` function -> get structured data (Python dict or JSON) back.

After parsing, each aggregate report is stored in the database. The storage can include a **Reports** table for the metadata (reporting org, date range, domain, etc.) and a **Record** table for each source/result record, linked by a foreign key to the report. Storing normalized data allows flexible queries (e.g., calculating compliance rates, finding sources with fails). However, storing the raw JSON from parsedmarc in a JSONB column is another quick approach for MVP, with separate columns for key summary metrics (like total emails, pass count, fail count) to generate the dashboard stats.

### Parsing DMARC Forensic (Failure) Reports (RFC 6591)

**Forensic reports** (failure reports) are sent to the ruf address and contain details of individual emails that failed DMARC. They often use the standard ARF format (Abuse Reporting Format) to convey the original email’s headers and info about the failure ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=Failure%20reports%20go%20to%20the,sources%20that%20need%20further%20configuration)). The backend needs to parse these to extract useful fields such as: the subject, source IP, sending server, headers like From, possibly snippets of the message or attachments that show why it failed (e.g., a DKIM signature that did not align).

Many email receivers no longer send forensic reports due to privacy concerns ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=Often%2C%20email%20services%20don%E2%80%99t%20provide,and%20acting%20on%20aggregate%20reports)). But when they are available, DMARQ will capture them. Using the Python `email` library, the backend can parse the `.eml` content. In ARF, the failure report will have a machine-readable part (with headers like “Feedback-Type: auth-failure”) and the original email attached (or headers thereof). The parser should pull out fields such as the reported domain, the authentication result that failed (SPF or DKIM), and any identifiers (like the header From domain). These are then stored in a **ForensicReport** table in the database, possibly with a reference to the aggregate report (if applicable) or at least to the domain and date.

Again, **parsedmarc** can assist here: it has the capability to parse forensic reports as well ([parsedmarc documentation - Open source DMARC report analyzer and visualizer — parsedmarc 8.18.1 documentation](https://domainaware.github.io/parsedmarc/#:~:text=,standard%20aggregate%2Frua%20reports)), returning standardized fields. This can save time. Either way, DMARQ will organize forensic data so that the frontend can display detailed information per incident, helping admins drill down into specific failures.

### Database Schema and SQLAlchemy Integration

A PostgreSQL database is used via SQLAlchemy ORM for reliable storage of DMARC data. The project will include SQLAlchemy models reflecting the data structure. Key models likely include:

- **User**: extends FastAPI Users’ base model (with fields like id, email, hashed password, is_active, etc.).
- **Domain**: the domain being monitored (in case the platform needs to support multiple domains in the future). Contains domain name and DMARC policy settings, etc. For MVP, this could be a single domain configured in the wizard.
- **AggregateReport**: stores metadata of each aggregate report (report_id, org_name, email, report_date_range, domain, policy_strictness, etc.).
- **AggregateRecord**: stores each source record from aggregate report with fields like source_ip, count, disposition (none/quarantine/reject), SPF_result, DKIM_result, aligned_dkim, aligned_spf, etc., plus a foreign key to AggregateReport ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=,Number%20of%20messages%20sent)).
- **ForensicReport**: stores forensic report details, e.g., original_message_id, source_ip, timestamp, failing_auth (SPF/DKIM), headers (maybe JSON), and a reference to Domain or AggregateReport if useful.

Using SQLAlchemy’s ORM, we can define these as Python classes and create a database schema. FastAPI can use dependency injection to provide a DB session to path operations. When new reports are parsed, the data is inserted: for aggregate, create an AggregateReport entry then bulk insert its AggregateRecord children. Proper indexing (e.g., index on source_ip or domain) will be set to optimize queries like finding all emails from a specific source.

### Configuration Wizard & Initial Setup

DMARQ includes a **web-based configuration wizard** to simplify first-time setup. This wizard will run when the app is first launched (or if a setup flag in the DB is not set). The purpose of the wizard is to collect essential configuration values from the user in a UI, so they don’t have to manually edit environment files. Key settings captured might include:

- **IMAP Credentials**: Mail server, port, username, and password (or an app password) for the mailbox that will receive DMARC reports. This is needed for the backend to start polling.
- **Domain(s) to Monitor**: The primary domain (or domains) for which DMARC reports will be collected. The wizard can prompt for the domain name and possibly suggest adding the necessary DNS records (DMARC, SPF, DKIM) if not already in place.
- **Alerting Preferences**: An option to configure alerts (e.g., email for critical issues). In MVP this might be skipped or minimal, but the structure allows setting up later.
- **Cloudflare API (Optional)**: If the user wants the app to manage DNS via Cloudflare, the wizard can ask for Cloudflare API token and Zone ID for the domain.
- **Admin User**: If no user exists yet, the wizard will also create the first admin account (or prompt to register one via the normal signup flow).

This wizard is implemented on the frontend as a series of forms (discussed later in Frontend section). On the backend, an endpoint (e.g. `/api/setup`) will accept the configuration payload and save it: likely storing to a **Config** table or to environment variables. Storing config in DB (encrypted where sensitive, e.g., IMAP password) allows the backend to retrieve it at runtime (or we inject it via Pydantic settings model). The wizard will only be accessible until setup is completed (after which it is disabled or requires admin rights to re-run, for changing settings).

Notably, these config values can also be **seeded via environment variables** for those who deploy via Docker and want to skip the wizard. For example, if `DMARQ_IMAP_HOST`, `DMARQ_IMAP_USER`, etc., are provided, the app could auto-create the config and mark setup as done. This dual approach (wizard UI or env vars) makes onboarding flexible.

### Modular Alerting Integration (Apprise)

To keep users informed of important events (like DMARC failures from new sources, or a domain moving to enforcement), DMARQ plans to send notifications. The design is modular to accommodate various notification channels. We prepare integration with **Apprise**, a powerful notification library that supports dozens of services (email, Slack, Teams, SMS, etc.) through a simple API ([caronc/apprise: Apprise - Push Notifications that work with just about ...](https://github.com/caronc/apprise#:~:text=caronc%2Fapprise%3A%20Apprise%20,as%3A%20Telegram%2C%20Discord%2C%20Slack%2C)). 

In the backend, an `alerts` module will define functions to send alerts. For MVP, we might implement a basic email alert (using SMTP) for critical issues, but structure the code such that adding a new channel is easy. With Apprise, for example, we can configure it by adding user-supplied URLs (each URL could represent a destination, like an email address, Slack webhook, etc.). The code can load these from config and call `apprise.notify()` with a message. Because Apprise supports *“almost all of the most popular notification services”* in a unified way ([caronc/apprise: Apprise - Push Notifications that work with just about ...](https://github.com/caronc/apprise#:~:text=caronc%2Fapprise%3A%20Apprise%20,as%3A%20Telegram%2C%20Discord%2C%20Slack%2C)), DMARQ users could later choose their preferred methods without the platform having to implement each from scratch.

The modular design means the alerting system is loosely coupled. For instance, whenever a new aggregate report is processed, a function can evaluate conditions (e.g., any DMARC failure from an unknown source? compliance rate dropped below a threshold?) and if so, trigger an alert via the alerts module. Since this is optional, if no alert config is provided, the system can quietly skip it.

### Cloudflare DNS Management (Optional)

Many organizations host their DNS with Cloudflare. DMARQ provides optional **Cloudflare integration** to help manage and monitor DNS records related to email authentication. If enabled (API credentials provided), the backend can use Cloudflare's API to perform tasks such as:

- **DNS Record Retrieval**: On the dashboard's DNS health section, instead of relying on a generic DNS lookup, the backend can directly fetch the DNS records (TXT, MX, etc.) for the domain via Cloudflare API. This can verify the current SPF, DKIM (public keys via selector records), DMARC, MX, and even BIMI records. The results let the UI show a "health check" – e.g., ✅ if a record exists and is correctly formatted, or ❌ if missing or misconfigured.
- **DMARC Policy Updates**: The platform could allow the user to update their DMARC record from the UI. For example, after achieving a high compliance rate, the user might want to change policy from `p=none` to `p=quarantine` or `p=reject`. Through the Cloudflare API, DMARQ can programmatically modify the TXT record. (All such actions would be manual triggers by the user in the UI, with confirmation.)
- **BIMI and Others**: If the user has a BIMI record (brand logo), or needs to add one, the integration could help create that DNS entry as well. Since brand protection is related, having an interface for these records is useful.

This Cloudflare support essentially turns DMARQ into a mini DNS management tool focused on email auth records, streamlining the DMARC deployment journey. It remains optional – if not used, the DNS health check can fall back to direct DNS queries (using `dnspython` library, for example) to still provide record info.

Under the hood, the Cloudflare integration might use the official Cloudflare Python library or direct REST calls. It will be abstracted in a `cloudflare.py` module that the rest of the app can call (e.g., `cloudflare.get_txt_record(domain, name="_dmarc")`). The user's API token is stored securely (in env or DB, marked secret).

## Frontend Implementation (Integrated Approach)

DMARQ uses an **integrated frontend architecture** with FastAPI's built-in Jinja2 templating system, enhanced with modern web technologies. This approach simplifies deployment and improves performance by eliminating the API boundary between frontend and backend. The UI is built with:

1. **Jinja2 Templates**: For server-side HTML rendering directly from FastAPI
2. **Tailwind CSS**: For utility-first styling that aligns with the DMARQ brand palette
3. **shadcn/ui Components**: Adapted from React to work with server-side rendering
4. **HTMX**: For AJAX functionality without writing complex JavaScript
5. **Alpine.js**: For enhanced interactivity and state management in the browser
6. **Chart.js**: For data visualization components throughout the dashboard

This integrated approach offers several advantages:
- Simplified deployment (single container instead of separate frontend/backend)
- Reduced complexity (no need for API contracts between frontend/backend)
- Improved initial page load performance through server-side rendering
- SEO benefits from server-rendered content
- Progressive enhancement for better accessibility

### Dashboard UI Features

The DMARQ dashboard provides an at-a-glance view of email authentication status and recent issues. Key sections of the dashboard include:

- **DMARC Compliance Rate:** A prominent metric showing the percentage of emails passing DMARC (both SPF and/or DKIM aligned) out of total emails. This displays as a large percentage number with a circular gauge visualization. Compliance rate is crucial to track progress – e.g., *"98% compliance"* is often the threshold to move to a stricter policy ([Best Practices: Advancing Your DMARC policy - dmarcian](https://dmarcian.com/advancing-dmarc-policy/#:~:text=that%20these%20domains%20match,mark)). 

  Using Chart.js, we render a line chart showing compliance rate over time (trendline per day/week). Alpine.js manages the date range selector, allowing users to switch between time periods (last 7 days, 30 days, etc.) with the chart updating dynamically. If 100% is not reached, a note indicates how many messages failed and need attention.

- **Policy Enforcement Trends:** This section visualizes how the domain's DMARC policy and enforcement have evolved. A timeline chart created with Chart.js shows the proportion of emails that were quarantined/rejected over time. As compliance improves, organizations typically move to stricter enforcement. DMARQ also displays markers indicating when policy changed from `none → quarantine → reject`. 

  The chart is rendered server-side initially, with Alpine.js handling interactions like tooltips and data filtering. A complementary bar chart shows how many spoofed emails were blocked per month, demonstrating the value of proper DMARC enforcement.

- **DNS Record Health Check:** A panel that lists the essential DNS records for email authentication:
  - **SPF:** Check if a valid SPF TXT record exists for the domain
  - **DKIM:** List the DKIM selectors in use from aggregate reports
  - **DMARC:** Show the domain's DMARC record and key tags (p= policy, rua, ruf, pct, etc)
  - **MX:** Show whether MX records exist and are properly configured
  - **BIMI:** Check for a BIMI record and whether it points to a valid SVG certificate

  Each record is displayed with its actual value and a status icon (✅/❌). With Cloudflare integration enabled, HTMX powers interactive "Update" buttons next to each record, which can trigger server-side modals for editing DNS entries.

- **Alerts Summary:** A section highlighting recent alerts or important notices, implemented as a list of the last N alerts with severity icons (info/warning/critical). Clicking an alert uses HTMX to load more details without a full page reload. The alerts are server-rendered initially, with new alerts fetched periodically using HTMX's polling capabilities.

- **Forensic Report Drilldown:** For detailed investigation, the UI offers a forensic report view implemented as a dynamic data table. Users can filter reports by date, source IP, or sending source through form controls enhanced with Alpine.js for instant filtering. Each entry shows information like the source IP, sending domain, failure reasons, and disposition.

  The detailed view of each forensic report is loaded on-demand via HTMX triggers, preventing the need to load all details at once. For advanced users, a toggle shows full email headers when needed.

All dashboard components are built as reusable Jinja2 macros or includes, ensuring consistency throughout the application. The layout uses Tailwind's grid and flex utilities for responsiveness, automatically adapting to different screen sizes.

### Onboarding Configuration Wizard UI

The frontend includes an **onboarding wizard** that runs on first use, implemented as a multi-step form:

1. **Welcome Step:** Introduces DMARQ and outlines the setup process
2. **User Account Setup:** Collects email and password for the first admin user
3. **Domain & Email Setup:** Form fields for the domain to monitor and IMAP credentials
4. **Optional Services:** Configuration for Cloudflare API and alerting notifications
5. **Completion:** Summary of settings and initial data fetching

The wizard is built with Jinja2 templates styled using Tailwind CSS classes and shadcn/ui components (adapted for server rendering). Form validation happens both client-side (using Alpine.js for immediate feedback) and server-side (for security). Progress through the wizard is maintained in server-side session state, allowing users to resume setup if interrupted.

Each step includes validation with helpful error messages. For example, when testing IMAP credentials, an HTMX request verifies the connection and provides immediate feedback without a full page reload. Upon completion, configurations are saved to the database and the user is redirected to the main dashboard.

### Theming and Branding in the UI

The DMARQ frontend strongly reflects the brand identity:

- **Color Scheme:** The Tailwind configuration extends the default theme with DMARQ's brand colors: deep blue `#1A237E` as the primary color, vibrant teal `#00ACC1` as the secondary/accent color, and bright orange `#FF7043` for warnings and alerts. Light gray `#F5F5F5` serves as the background for panels, while dark gray `#212121` provides contrast for text.

- **Typography:** The application loads **Montserrat** for headers and **Open Sans** for body text through a combination of @font-face declarations and Tailwind's fontFamily configuration. This ensures consistent typography across all pages.

- **Layout and Navigation:** The application features a responsive layout with a top navigation bar displaying the DMARQ logo and main navigation links. The sidebar (on larger screens) provides context-sensitive navigation based on the current section. All UI elements follow shadcn/ui design patterns, adapted to work with Jinja2 templates instead of React components.

- **Charts and Graphics:** Chart.js visualizations are styled to match the theme, using the brand colors for consistency. The charts are rendered server-side for the initial view, with Alpine.js handling interactive features like tooltips, zooming, and filtering.

By implementing these design elements through Jinja2 templates and Tailwind CSS, DMARQ maintains a cohesive visual identity while benefiting from server-side rendering performance.

## Deployment and Project Structure

To make it easy to run the entire platform, DMARQ provides a Docker-based deployment. We use **Docker Compose** to define all required services and ensure they work together out of the box. The repository is organized clearly with separate directories for backend, frontend, and configuration:

- **backend/** – FastAPI application code. This includes submodules like `auth/`, `dmarc/` (for parsing logic), `models/` (SQLAlchemy models), `routes/` (APIRouters for various endpoints such as auth, reports, etc.), and `services/` (e.g., IMAP polling service, alert service). An `app.py` (or `main.py`) at the root creates the FastAPI app, includes routers, and possibly sets up a startup event to kick off the background IMAP polling (or schedules it).
- **frontend/** – Jinja2 templates and static assets. This includes Tailwind CSS configuration, shadcn/ui components adapted for server-side rendering, and JavaScript files for HTMX and Alpine.js interactivity. The templates are organized into directories for pages (e.g., `dashboard.html`, `wizard.html`) and partials (e.g., `header.html`, `footer.html`).
- **config/** – Configuration and deployment files. This includes `docker-compose.yml` to orchestrate containers, Dockerfiles for the backend, an `.env.example` file documenting environment variables (and possibly a `.env` that can be used for local dev). It may also contain a script or instructions for initial setup (like a shell script to run migrations, create a default admin, etc., although we rely on the wizard for admin creation).

- **README.md** – A detailed README in the root explains how to set up and run DMARQ. It will cover prerequisites (Docker installed), how to configure environment (e.g., providing IMAP credentials in `.env` or using the wizard), and how to bring the stack up. It also outlines the project structure for developers, and points to docs or wiki if more info.

- **tests/** – Both backend and frontend tests (could be split into `backend/tests` and `frontend/tests`). This ensures the project includes automated tests for critical functionality.

With this structure, **Docker Compose** can define three main services:
1. **db**: The PostgreSQL database. In `docker-compose.yml`, this uses the official postgres image, with environment for password, and a volume for data persistence.
2. **backend**: The FastAPI app image. We create a Dockerfile in `backend/` that starts from a Python base image, installs dependencies (from a `requirements.txt` or `pyproject.toml`), copies the FastAPI app code, and runs Uvicorn (or Hypercorn) to serve the app (for example: `uvicorn app:app --host 0.0.0.0 --port 8000`). This container would link to `db` for database access (the DB URL set via env such as `DATABASE_URL=postgresql://...`).
3. **frontend**: The static assets served by the backend. The backend serves the Jinja2 templates and static files directly, eliminating the need for a separate frontend container.

The **compose setup** enables anyone to do `docker-compose up -d` and have the system running at `http://localhost` (frontend and API). We will ensure CORS is allowed from the frontend origin in FastAPI settings (or if served under same domain, configure nginx to proxy API requests to backend, e.g., prefix `/api/`).

### Running and Deployment Considerations

After containers are up, the user would typically access the app on the configured host (say `app.dmarq.org`). The first thing they see is the setup wizard (if not configured), or the login page otherwise. We will provide a **setup script** or instructions to initialize the database (running migrations if any, though for MVP we might use SQLAlchemy to create tables on first run automatically). FastAPI can be set to create DB tables at startup by using SQLAlchemy’s `create_all()` in an event handler if not using Alembic yet.

For production deployment beyond dev Docker, the project’s modular design allows scaling: the backend can be scaled to multiple instances (stateless except the background job – which we might eventually offload to a separate worker to avoid duplicate polling), and perhaps a separate worker container for IMAP polling could be introduced if needed. The database is external and could be managed by a cloud provider.

### CI/CD and GitHub Preparation

The repository will be prepared for GitHub with CI in mind:
- A GitHub Actions workflow (YAML) could be included to run backend tests (`pytest`) and frontend tests (`npm test`) on each pull request, ensuring quality.
- The README will encourage contributions and explain the stack.
- Documentation in-code (docstrings and possibly a docs/ directory with more detailed usage or API info) can be included for developers.

By splitting code cleanly and using Docker, the project is easy to set up for anyone reviewing the code on GitHub. The folder structure and documentation reflect a professional, open-source-ready project, where one can run the entire stack or even deploy it to a cloud (for instance, using services like Heroku or Fly.io with minor adjustments, or Kubernetes if scaling up, etc.). 

## Testing and Development Practices

Building a robust platform requires testing and good development standards from the start. DMARQ’s MVP will include:

### Backend Unit Tests (Pytest)

We will write **unit and integration tests** for the FastAPI backend using **pytest**. Important parts to test include:
- **DMARC Parsing Logic:** Provide sample DMARC aggregate report XML files (and zipped variants) in tests to ensure our parsing function or the parsedmarc integration correctly extracts data. For example, test that a known XML yields the expected number of records, and that edge cases (like multiple DKIM identifiers, or a report with a policy override) are handled.
- **IMAP Fetching:** This can be tested by mocking the IMAP server. Using Python’s `imaplib` we might simulate a mailbox with a test email. Alternatively, abstract the email retrieval in a function that we can feed with a prepared email file. The test would verify that an email with an attached report results in correct DB entries after processing.
- **API Endpoints:** Using FastAPI’s TestClient, we can simulate requests to the API. For auth, test that a user can register and login (ensuring FastAPI Users is configured properly). For data endpoints, we may create a fake report in the database and test that the GET endpoint (e.g., `/api/reports/summary`) returns the correct JSON (compliance rate, etc.). Also test security, e.g., that protected endpoints return 401 for anonymous requests.
- **Alerting Module:** Use monkeypatch or dependency override to test that when a certain condition is met, the alert function is called. We might fake the Apprise call to just record that a notification would have been sent.
- **Database Models:** If using an in-memory SQLite for testing (SQLAlchemy can connect to sqlite:///:memory: for speed), we test that `create_all` works and basic CRUD on models functions as expected.

Pytest fixtures can set up a temporary database, perhaps using `sqlite` for quick tests, or a PostgreSQL test container if we want to mimic real environment. We ensure tests are isolated (each test either uses a transaction rollback or a new schema).

### Frontend Testing (Jinja2 and HTMX)

For the Jinja2-based frontend, we use **pytest** and **selenium** for testing:
- **Template Tests:** Test that Jinja2 templates render correctly given context data. For instance, a `ComplianceRateCard` template that takes a percentage should render that number and perhaps an appropriate color (green if ~100%, etc.). We simulate different values and assert the HTML output.
- **Dashboard Page Tests:** Using selenium, render the Dashboard page (with maybe a mock context providing sample data). Ensure that all major sections appear (e.g., "DMARC Compliance Rate" text, an element showing a "%" value, etc.). If there are child components for charts, we might mock the chart library for simplicity or ensure it renders a canvas element.
- **Wizard Flow Tests:** Simulate the multi-step wizard. We can mimic user filling the forms – e.g., fill domain and IMAP fields, click next – and then verify that the next step appears. Also test validation: e.g., leave a required field empty and assert that an error message shows and it doesn’t advance. For the final submission, we’d mock the backend response and ensure the app handles success (redirect to login or dashboard).

These tests help catch regressions as we develop. We include running `pytest` as part of CI.

### Code Quality and Documentation

Throughout development, we emphasize **clean code structure**:
- The Python backend code will follow PEP8 style, and we can include linters (flake8/black) configuration. Complex logic (like parsing) will be broken into smaller functions or classes (e.g., a `DMARCReportParser` class to encapsulate parsing functions, which can be unit-tested independently).
- The Jinja2 templates should be organized into logical components and macros. Avoid large monolithic templates; instead, separate e.g., `DnsHealthCard`, `AlertsList`, `ForensicTable` etc. This not only makes it easier to maintain but also easier to test each piece.
- We will add docstrings and comments in critical sections. For instance, the function that polls IMAP will have a comment explaining the IMAP UID tracking if used, or how often it runs.
- The **README** will document how to run tests, how to run formatting tools, etc. It will also have a high-level overview of the system.

### Extensibility and Future-Proofing

Even as an MVP, DMARQ is designed with **extensibility in mind**:
- New features such as additional alerting methods, support for multiple domains or domains owned by different users, or integration with other email security protocols can be added with minimal refactoring. For example, adding another authentication method (SSO or OAuth) is possible because FastAPI Users supports multiple authentication backends if needed.
- The use of standards (FastAPI, SQLAlchemy, Jinja2) means a broad community support and familiarity, making it easier for others to contribute or for the project to grow.
- By modularizing the backend (each major feature in its own module/router), a developer can navigate the codebase easily. The same goes for frontend: clear separation of concerns (e.g., wizard vs dashboard vs shared components).
- Logging is implemented on the backend to trace the processing of reports. If something fails in parsing an email, it logs an error with details (but without dumping sensitive info). This will greatly help in debugging issues in production.
- Security considerations: storing IMAP credentials securely (if in DB, encrypt them using a key), using HTTPS in production, and making sure secrets (like JWT secret, encryption keys) come from environment vars and are not hardcoded.

By following these practices, we ensure that the MVP is not a throwaway prototype, but a solid foundation that can be built upon for a full-fledged DMARC monitoring service.

## Conclusion

The **DMARQ** MVP as described provides a comprehensive end-to-end solution for DMARC report monitoring and analysis. We have a powerful FastAPI backend handling authentication, data ingestion (IMAP fetching of DMARC reports) and parsing per industry standards (RFC 7489 for aggregate reports and RFC 6591 for forensic reports), with data stored in a structured way on PostgreSQL. The frontend offers a modern, responsive dashboard inspired by top industry solutions, displaying critical metrics like DMARC compliance rates and enforcement progress, and guiding users through setup and issue resolution. The platform is containerized via Docker Compose for easy deployment and follows best practices in testing and code organization, making it maintainable and ready for open-source collaboration.

By adhering to the brand’s guidelines (colors, typography) and focusing on clean UI/UX (with Tailwind, HTMX, and Alpine.js), DMARQ will not only function effectively but also deliver a polished experience. In summary, this MVP achieves the goal of a self-service DMARC monitoring tool – **DMARQ** – that helps organizations improve their email security posture with clarity and confidence. 

**Sources:**

- FastAPI Users documentation (features and JWT auth setup) ([GitHub - fastapi-users/fastapi-users: Ready-to-use and customizable users management for FastAPI](https://github.com/fastapi-users/fastapi-users#:~:text=Add%20quickly%20a%20registration%20and,customizable%20and%20adaptable%20as%20possible)) ([Full example - FastAPI Users](https://fastapi-users.github.io/fastapi-users/10.1/configuration/full-example/#:~:text=app.include_router%28%20fastapi_users.get_auth_router%28auth_backend%29%2C%20prefix%3D,%29%20app.include_router))  
- Parsedmarc library – open source DMARC report parser (supports IMAP, aggregate & forensic) ([parsedmarc documentation - Open source DMARC report analyzer and visualizer — parsedmarc 8.18.1 documentation](https://domainaware.github.io/parsedmarc/#:~:text=,standard%20aggregate%2Frua%20reports))  
- EasyDMARC blog – DMARC aggregate report contents (XML fields and frequency) ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=DMARC%20aggregate%20reports%20are%20XML,sensitive%20information%20about%20email%20messages))  
- EasyDMARC blog – DMARC failure (forensic) reports overview ([EasyDMARC Blog | Understanding DMARC reports](https://easydmarc.com/blog/understanding-dmarc-reports/#:~:text=Failure%20reports%20go%20to%20the,sources%20that%20need%20further%20configuration))  
- Dmarcian guidelines – 98%+ compliance rate recommended before policy enforcement ([Best Practices: Advancing Your DMARC policy - dmarcian](https://dmarcian.com/advancing-dmarc-policy/#:~:text=that%20these%20domains%20match,mark))  
- Bouncebuster DMARC tools review – EasyDMARC noted for user-friendly dashboard ([Top 10 DMARC Monitoring Tools 2025 – Bouncebuster Blog](https://blog.bouncebuster.io/top-10-dmarc-monitoring-tools-2025/#:~:text=%2A%20EasyDMARC%3A%20User,blacklist%20tracking%2C%20and%20Safe%20SPF))  
- ShadCN/UI introduction – open-source Tailwind React components for accessible design