gh-christianlouis-dmarq/docs/milestones.md

Perfect. I'll now create a full milestone and architecture strategy for DMARQ — starting with an MVP for five domains, then expanding toward the complete product vision. Each step will be modular, testable, and deployable within a few hours using GitHub Copilot.

I'll summarize the entire development arc shortly, including clear milestones, architectures per phase, and user stories that guide implementation.

DMARQ Development Roadmap

Project Overview and Positioning

• Purpose: DMARQ is a self-hosted DMARC monitoring and visualization tool. It provides professional visibility into email authentication (DMARC) reports without relying on any third-party services, keeping sensitive email data private.
• Target Users: Designed for IT administrators and security-conscious users who manage one or many domains. It helps them ensure DMARC compliance and email security while maintaining full control of data.
• Technology Stack: Built with a FastAPI backend and server-rendered Jinja2 templates for the frontend. Styling uses Tailwind CSS (leveraging component kits like shadcn/ui or Preline for rapid UI development). No React or separate frontend app is used – the web interface is served directly by FastAPI.
• Deployment: Distributed as a self-contained Docker image, intended to be run via Docker Compose for easy setup. All components (web UI, database, background tasks) run within this service, on-premises.
• Roadmap Scope: The development will start with a Minimal Viable Product (MVP) supporting a few domains, then progress through incremental milestones. Each milestone delivers a working, testable product with new features or improvements, achievable in a few hours of development (especially with GitHub Copilot assistance). Over successive milestones, DMARQ will grow to handle 50–100 domains, add advanced features like alerting, DNS configuration guidance, Cloudflare integration, historical trends, authentication, and more – all while preserving its privacy-centric, self-hosted ethos.

Milestone 1: Minimal Viable Product (MVP) – Basic DMARC Monitoring (up to 5 domains)

• Scope & Features: Implements core DMARC report processing and a simple dashboard for a small number of domains (e.g. up to five). The MVP can ingest DMARC aggregate reports (RUAs) and display summary results per domain. Initially, report files might be provided manually (for example, uploading XML or zip files containing DMARC reports via the web UI). The frontend presents a basic table or list of DMARC statistics for each domain – such as total emails analyzed, percentage that passed DMARC checks, and any failures. This allows an admin to get immediate insight into DMARC compliance for their domains in a local web interface.
• Architecture: The application is a single FastAPI service serving HTML pages. DMARC report parsing is handled in Python (for speed, one could leverage an existing library like parsedmarc, which parses both aggregate and forensic reports into JSON (parsedmarc documentation - Open source DMARC report analyzer and visualizer — parsedmarc 8.18.1 documentation)). Parsed data (e.g. per sending source IP: how many emails, SPF/DKIM pass rates, policy applied) is stored in-memory or in a lightweight local store. The UI is built with Jinja2 templates styled by Tailwind CSS components, ensuring a clean, responsive design without a heavy frontend framework. No database is required at this stage (data can be kept in memory during runtime), which is manageable given the low volume (handful of domains and recent reports). There is no authentication yet (open access locally), simplifying setup.
• User Stories:
  • As an IT admin, I can upload DMARC aggregate report files (XML from my email provider) to DMARQ and see a dashboard of DMARC results for up to five of my domains, so that I can start monitoring email authentication status immediately.
  • As a security-conscious user, I can deploy the DMARQ MVP via Docker Compose on my own server and review DMARC compliance data through a web dashboard, ensuring none of my email report data is sent to any third-party service.
  • As an email administrator, I can view for each domain a list of sending sources (IP addresses or hostnames) and whether they passed or failed DMARC checks, so that I understand which senders are authorized and which might be illegitimate.

Milestone 2: Automated Report Ingestion (IMAP Mailbox Integration)

• Scope & Features: This milestone eliminates manual report handling by introducing automated retrieval of DMARC reports from an email inbox. DMARQ will connect to a configured IMAP mailbox that collects DMARC aggregate reports. It will periodically fetch new emails, extract the DMARC report attachments, parse them, and update the dashboard data. The system can be configured with the IMAP server details (host, port, credentials) via environment variables or a simple config file. This allows continuous monitoring without user intervention – new reports are ingested on schedule (e.g. hourly or daily). The front-end might also get a refresh button or indicator showing the last fetch time and allowing on-demand sync.
• Architecture: A background task or scheduler is added to the FastAPI app to handle periodic mailbox polling (for example, using FastAPI’s background tasks or an APScheduler job started on app startup). On each run, the IMAP client logs in and searches for unread DMARC reports (by sender or subject). For each new report email found, DMARQ downloads the attachment (handling zip or gzip compression), then parses the XML content (using the same parser from MVP). Parsed records are then merged into the in-memory data store. Processed emails can be marked as read or moved to an archive folder to avoid duplication. The rest of the stack remains the same – FastAPI endpoints for display, etc. (If an external library like parsedmarc is used, it can even directly fetch from IMAP and return structured data, simplifying implementation).
• User Stories:
  • As an IT admin, I can configure DMARQ with my DMARC report mailbox credentials and have the system automatically retrieve and process incoming DMARC reports. This means I no longer need to manually upload files – DMARQ stays up-to-date with the latest reports on its own.
  • As a user, when I check the DMARQ dashboard each day, I see that new DMARC statistics have been added automatically (e.g. the previous day’s email stats appear after the daily reports arrive), giving me timely insights without manual effort.
  • As a systems engineer, I want the DMARQ service to run continuously and keep itself updated, so I set it up in Docker Compose with the IMAP settings. The service logs or UI indicates when it last fetched reports, so I have confidence that the data is current.

Milestone 3: Persistence and Multi-Domain Scaling (50–100 Domains Support)

• Scope & Features: This milestone upgrades the backend to handle a larger number of domains and reports reliably. It introduces a database layer for persistence and efficient data querying. With this change, DMARQ can support dozens of domains (50–100) and a higher volume of reports over time. The product now includes domain management capabilities – for example, an interface to add or remove domains being monitored – and ensures data is retained across restarts. The UI may be enhanced to allow filtering or selecting a domain to view (if many domains are present). By the end of this milestone, DMARQ is suitable for an organization with a large domain portfolio, not just a handful.
• Architecture: A lightweight relational database (e.g. SQLite for simplicity, or PostgreSQL if scaling beyond) is integrated via an ORM. Data models are defined for Domain, AggregateReport, and related entities (e.g. per-sender record details). When IMAP-fetched reports are parsed, their data is now saved to the DB. This allows persistent storage of historical records and efficient lookups (instead of keeping everything in memory). The domain management feature means there is a Domain table – the user can explicitly add domains they want to monitor. The system will filter incoming reports to only store data for configured domains (or conversely, auto-populate the domain list from reports, with the option to ignore some). The FastAPI app now uses database queries to populate templates. For example, the dashboard might show a list of domains with summary stats, querying aggregated info per domain from the DB. To handle more data, the UI might introduce pagination or summary views (e.g. show only top 5 sending sources by volume, with an option to view all). The overall architecture is adjusted to ensure that increasing the number of domains or reports doesn’t degrade performance – for instance, heavy parsing could be done in the background job, and the web requests simply read from the prepared database tables.
• User Stories:
  • As an administrator with many domains under management, I can add all my domains to DMARQ and trust that it will track DMARC reports for each domain. The system can handle my 50+ domains without losing data or slowing down, so I have a one-stop dashboard for all domains’ email compliance.
  • As a user, if I restart or update the DMARQ Docker container, all previously collected DMARC data is still there when it comes back up, because the data is stored persistently in a database. I don’t have to worry about backups of XML files or memory – the system retains history automatically.
  • As an IT manager, I can navigate the DMARQ UI to see an overview of all monitored domains and then drill down into a specific domain’s details. Even with 100 domains configured, I can quickly find a particular domain (via a filter or search) and view its DMARC compliance status, which is crucial for managing a large environment.

Milestone 4: Dashboard Enhancements and Historical Visualization

• Scope & Features: Now that DMARQ has robust data storage, this milestone focuses on visualizing trends and enhancing the dashboard UI. It introduces charts and graphs for historical DMARC data and other summary widgets, transforming the dashboard into a more insightful overview of email authentication health. For example, the dashboard may display the DMARC compliance rate (percentage of emails passing DMARC) over the last 7 or 30 days, the enforcement rate (percentage of domains with strict policies), total email volume processed, and active alert counts (once alerts are implemented). Each domain’s page might include a timeline chart of daily email volumes and DMARC pass/fail rates. Additionally, cross-domain summary visuals can be added – e.g. a pie chart of DMARC policies in use across all domains (how many domains have p=none, quarantine, reject), and indicators of how many domains have proper SPF/DKIM/DMARC (this part ties into DNS checks in a later milestone but can be stubbed or updated later).
• Architecture: The front-end now incorporates a charting solution. Since we avoid heavy front-end frameworks, we can use a simple JS library like Chart.js (loaded via CDN) or generate charts server-side as images. The FastAPI endpoints can supply data to charts either by embedding JSON in the page or providing a small API endpoint that returns data for a chart (which the JS code calls via AJAX). Tailwind CSS components from shadcn/ui or Preline can be used to create card-like widgets on the dashboard for key metrics. For example, a card showing “DMARC Compliance Rate: 62%” with a small trend indicator. The system calculates historical metrics from the DB (for instance, each day’s compliance % per domain, or aggregate for all domains) – possibly pre-computing these in the background for efficiency. The UI/UX is refined to be more navigable: a sidebar menu listing sections (Dashboard, Domains, Reports, Alerts, Settings, etc.), consistent with the single-page app feel but implemented with multi-page Jinja templates. This milestone is about presenting existing data in a more user-friendly, insightful way; no major new backend logic beyond queries for time-series data.
• User Stories:
  • As an IT admin, I can view graphs on the DMARQ dashboard that show trends like how my DMARC compliance has improved over the past week and the volume of emails my domains receive daily. This visual context helps me quickly assess whether changes we made (e.g., adding SPF records or changing DMARC policy) are improving email security.
  • As a security engineer, I have a clear dashboard that highlights key metrics: what percentage of our emails are passing DMARC, how many domains are at a reject policy, how many total emails were seen, and if there are any active issues. This at-a-glance information (with charts and colored indicators) allows me to report the organization’s email security posture to management easily.
  • As a user, I find the interface intuitive: I can select a specific domain and see its detailed report history in both tabular and graphical form. For example, I select “example.com” and see a line chart of daily DMARC pass rates and a breakdown of senders – this helps me to pinpoint when a new unauthorized sender started failing DMARC, in a visual way.

Milestone 5: Authentication and Access Control (FastAPI Users Integration)

• Scope & Features: At this stage, DMARQ gains user authentication and access control to secure the interface. Rather than being openly accessible, the application now requires a login. This is important as the tool may be hosted in environments where multiple people or even the internet could access it; we want to ensure only authorized admins can view or change settings. We integrate a standard user management system with features like user registration (for an initial admin), login, and (optionally) password reset. For now, DMARQ might be single-user (one admin account) or a small set of users (if a team of admins), but we use a scalable auth framework to accommodate growth.
• Architecture: We utilize FastAPI Users, a robust plug-and-play authentication library for FastAPI. This provides ready-made routes for authentication and user management (including hashing passwords, JWT or session cookie management, etc.) (GitHub - fastapi-users/fastapi-users: Ready-to-use and customizable users management for FastAPI) (GitHub - fastapi-users/fastapi-users: Ready-to-use and customizable users management for FastAPI). A new database table for users (and perhaps OAuth or verification tokens) is introduced. The initial admin user can be created via environment variables (for example, provide an initial admin email and password hash to auto-create on startup) or through a CLI setup command. FastAPI Users is configured to use the same database (with SQLAlchemy or similar) that DMARQ already uses. We decide on an authentication strategy – likely cookie-based sessions for a web app (so the admin logs in via a form, and a secure session cookie keeps them logged in). All previously open endpoints (dashboard, domain pages, etc.) are now protected by an @login_required dependency, so unauthenticated requests redirect to a login page. The UI gets new pages: Login (and possibly Register or Password Reset if we allow multiple users or self-service account creation). Tailwind UI components are used to style the login form and any auth-related pages in a consistent manner. Access control at this point is straightforward (any logged-in user is considered an admin with full access). If needed, we can tag certain users as read-only vs admin, but that might be overkill for now.
• User Stories:
  • As the system owner, I want DMARQ to be password-protected. Now when I access the DMARQ URL, I am prompted to log in. I use the admin credentials I set up, and then I can see the dashboard. If an unauthorized person tries to access it, they cannot get in without the credentials, ensuring my email report data stays confidential.
  • As an admin user, I can manage my credentials (e.g., change my password) and potentially add another colleague as a user. This way, our security team can have separate logins to the DMARQ dashboard.
  • As a new user (if multiple users allowed), I can sign up or be invited to DMARQ with my email and a password. Once I verify my email (if that feature is enabled) and log in, I can view the DMARC monitoring info. (Initially, we might skip email verification for simplicity, but FastAPI Users supports it if needed.)

Milestone 6: Apprise-Based Notification Alerting (Basic Alerts)

• Scope & Features: This milestone introduces alerting capabilities so that DMARQ can actively notify administrators of important events rather than relying on them to check the dashboard. We implement basic rules for alerts (built-in for now) and use Apprise – a notification library – to send out these alerts via various channels (email, Slack, etc.). Initially, the alert triggers can be simple and fixed: for example, send an alert when a new DMARC report comes in that contains any failure (unauthorized use) above a certain threshold, or send a daily summary report of DMARC statistics to the admin’s email. The system will also have a configuration for the admin to provide their notification endpoints (e.g., an email address or a Slack webhook URL). Using Apprise means DMARQ can support many notification methods through a single integration.
• Architecture: We integrate the Apprise Python library for notifications. Apprise allows sending to almost all popular services (Telegram, Discord, Slack, email, etc.) with a common API (GitHub - caronc/apprise: Apprise - Push Notifications that work with just about every platform!). We add a configuration setting (stored in the DB or .env) where the admin supplies an Apprise “URL” for their desired channel (for example, an SMTP email URL or Slack webhook URL). When new DMARC reports are processed (in the background task from Milestone 2), after updating the database, the system evaluates the built-in alert conditions. A simple condition might be: “if any domain in this batch of reports has DMARC failures > 0, send an alert” or “if a new sending source (never seen before) appears, alert the user.” If the condition is met, DMARQ composes a notification message (e.g., “DMARQ Alert: 5 emails failed DMARC for domain X in today’s report.”) and calls Apprise to dispatch it to the configured channels. This happens asynchronously (so it doesn’t slow down the main thread – Apprise can be called in a background thread or during the fetch routine). We also consider adding an “Alerts” page in the UI where recent alerts are listed for reference. This page would pull from an Alert log table if we store alerts, or it might simply show the last alert time/status. In this basic alerting phase, the rules are not yet user-customizable (that comes next), but the user can at least turn alerts on/off or change the notification endpoint.
• User Stories:
  • As an admin, I want DMARQ to proactively notify me when something potentially problematic occurs. For example, if a DMARC report indicates someone tried to spoof our domain (DMARC failure), I receive an alert via my chosen channel (for instance, a message on Slack or an email). This way, I am aware of issues immediately without logging into the dashboard every day.
  • As a user, I can configure where DMARQ sends alerts. I might provide an email address for notifications or a Slack webhook. The system supports a variety of channels through a unified configuration (thanks to Apprise), so I can choose the channel that best fits my workflow.
  • As an IT security lead, I get a daily summary email from DMARQ each morning that tells me how many emails were processed for each domain and if there were any DMARC failures. This summary ensures I have a daily digest of our email authentication status, aligning with our policy to continuously monitor DMARC compliance.

Milestone 7: Custom Rule-Based Alert Triggers (Advanced Alerting)

• Scope & Features: Building on the basic alerting, this milestone gives administrators the ability to define custom alert rules and triggers. Instead of only having the fixed alerts (e.g., any failure triggers an alert), the user can specify conditions that matter to them. For example: “Trigger an alert if more than 10% of emails for any domain fail DMARC in a day” or “Alert me if a new sending source appears for domain X” or “Notify if no DMARC reports have been received for a domain in 3 days (which could indicate reporting is broken).” We will add a UI for managing these rules – creating, editing, and deleting alert conditions. Each rule will have parameters (domain scope, condition type, threshold values, etc.). This makes the alerting system much more flexible and reduces noise by allowing users to tailor alerts to their environment.
• Architecture: We introduce a Rule/AlertRules model in the database to store these user-defined rules. The UI (perhaps an “Alerts” or “Settings” section) provides a form to add a rule. For example, a rule might be: Type: Failure Rate, Domain: Any or a specific domain, Threshold: 10% (with maybe additional field for minimum volume to avoid small sample noise). Another rule type might be: New Sender Alert with an option to apply to all domains or a specific one. In the background processing loop, after fetching and storing reports, DMARQ will evaluate all active rules against the latest data. We may implement a small rule engine: e.g., for each domain’s stats in the latest report or in the last 24h, check the conditions. If a rule is satisfied, create an Alert event (and send it via Apprise as before). Possibly, to avoid duplicate alerts, we mark rules as triggered for a given day or until conditions normalize. The architecture should consider stateful vs stateless rules (some rules like “new sender” are event-based – once a new sender is seen and alerted, we might not alert again for the same sender). We could maintain state of known senders per domain to detect “new” ones. This adds some complexity, but manageable in a few hours by leveraging the data in the DB (e.g., we know all sources seen so far for a domain, so if in the new report there’s an IP not in that list, trigger the alert and then add it). The UI will display the list of configured rules, and allow toggling them on/off.
• User Stories:
  • As an admin, I want to customize what situations trigger alerts. For example, I create a rule: “If any domain’s DMARC failure rate exceeds 5% in a day, send me an alert.” Later, if one of my domains has a lot of failed emails, DMARQ sends an alert as specified. This flexibility means I only get notified on conditions that I consider important, reducing false alarms.
  • As a security engineer, I add a rule to notify me if a new email source starts sending as one of our domains. I specify the rule for all domains: “Alert on new sender.” DMARQ then tracks known senders, and when an unknown IP appears in a DMARC report, I get an alert with details. This helps me catch potentially malicious actors or new services that we haven’t authorized.
  • As a user, I can adjust or turn off certain alerts. For instance, once we’ve fully deployed DMARC, I might not need daily summaries anymore, so I disable that rule and instead keep only the high-severity alerts. The system’s rule list gives me control to fine-tune notifications easily via the web UI.

Milestone 8: Cloudflare Integration for DNS Health Checks and Guidance

• Scope & Features: This milestone extends DMARQ’s capabilities beyond reading email reports by integrating with DNS to provide DMARC/SPF/DKIM record health checks and guidance. Specifically, if the user’s domains are managed in Cloudflare, DMARQ can use the Cloudflare API to automatically fetch DNS records and detect issues. Even for non-Cloudflare users, DMARQ can perform DNS queries, but the Cloudflare integration streamlines things for those using that platform (and potentially allows easy fixes in the future). Features introduced: On the dashboard (or a dedicated “DNS Health” page), the user can see the status of each domain’s DNS records relevant to email security – whether a valid DMARC record exists, and if so, what the policy is; whether SPF and DKIM records are present and valid; and possibly BIMI or others if relevant. For each domain, DMARQ will highlight any problems (e.g., “No DMARC record found!” or “SPF record exists but includes too many lookups” or “DMARC policy is ‘none’ – consider enforcing if ready”). Additionally, DMARQ will provide recommended DNS record values as guidance. For example, if a domain has no DMARC, it might suggest a baseline DMARC record (v=DMARC1; p=none; rua=… etc.) that the user can adopt. If the domain has DMARC but in monitor mode, and the reports show good compliance, it might suggest upgrading to quarantine/reject. The integration with Cloudflare means DMARQ can (with proper API credentials) pull the exact record values and potentially even offer a one-click update (though actual update could be a later enhancement; initially, we focus on read-only checks and guidance).
• Architecture: We add configuration for Cloudflare API access – e.g., an API token that the user can obtain from Cloudflare. This token is stored (securely) in the DMARQ config. Using Cloudflare’s REST API, DMARQ can list DNS records for a given domain’s zone. The backend will implement a module to call these APIs (using Python requests or a Cloudflare SDK) to retrieve TXT records for DMARC (_dmarc.domain), SPF (the domain’s TXT that starts with v=spf1), and possibly DKIM selectors (which might need user to specify selector names, or we could attempt common names or parse from DNS). For each domain in our database, on a scheduled interval (maybe daily) or when the user clicks “Check DNS,” DMARQ fetches the latest DNS info. The results are stored or cached in a DNS status table. The logic then evaluates each:
  • DMARC: Does a TXT record _dmarc.domain exist? If yes, parse it to see policy (p=) and addresses (rua, ruf). If no, mark as missing.
  • SPF: Check for a TXT on the root domain (or subdomain if that’s what’s used for sending) containing v=spf1. If missing, that’s an issue (though not directly DMARC, it’s related). If present, possibly validate that it’s syntactically correct and not exceeding DNS lookup limits.
  • DKIM: If the user provided their DKIM selector(s), check those DNS records for existence and valid format (public key present). If not provided, we might skip or try to glean from reports which DKIM selector was used successfully/failed and then verify those.
  • We then formulate suggestions: e.g., if DMARC missing => suggestion to add a DMARC record (with a specific example string). If DMARC policy is “none” and say more than 90% of emails are passing, we might suggest “You can consider p=quarantine or p=reject to enforce protection.” If SPF exists but has issues (like includes nonexistent domains or too many includes), flag that (though full SPF analysis might be complex, we can at least note length or existence).
  • The front-end displays these findings in a clear way. For instance, on the Domain Overview page for each domain, show a section “DNS Records Status”: DMARC – Valid/Invalid/Missing (with details), SPF – Valid/Missing, DKIM – Valid/Missing. Possibly use colored badges (green for ok, yellow for warning, red for problem). Additionally, on the main dashboard, we could have a widget summarizing “DNS Record Statuses” across all domains (like “5 domains valid DMARC, 2 warnings, 1 missing” etc.), similar to the example image.
  • We ensure this feature is optional – if no Cloudflare API token is provided, we fallback to direct DNS queries (which can still tell presence/absence). Writing back to DNS (like auto-fixing records) is not done at this stage to keep things read-only and safe.
• User Stories:
  • As an admin, I can connect my Cloudflare account to DMARQ (by providing an API token) so that the tool can automatically check my DNS records for DMARC, SPF, and DKIM. When I open DMARQ, I immediately see which domains have proper records and which need attention – for example, it flags that “example.com” has no DMARC record, alerting me to fix that in DNS.
  • As a user managing email security, I appreciate that DMARQ gives me guidance on DNS configuration. After collecting reports for a while, DMARQ might show a suggestion: “Your domain abc.com is in monitoring mode (p=none) and has high compliance. Consider moving to p=quarantine for stronger enforcement.” This kind of recommendation helps me improve our security posture proactively.
  • As a DevOps engineer, I use Cloudflare for DNS, so having DMARQ tie into it is convenient. On each domain’s page in DMARQ, I can see the actual current DMARC record and SPF record. If something is wrong (like a typo or missing record), DMARQ points it out. It even provides the exact DNS record string I should use. This saves me time – I don’t have to use separate tools to check DNS or recall the syntax for DMARC records.

Milestone 9: Forensic Report (RUF) Support

• Scope & Features: DMARQ will now also handle DMARC forensic (failure) reports (the RUF aspect of DMARC). These forensic reports are detailed, single-incident reports sent when an email fails DMARC (if the DMARC policy’s ruf tag is set). They contain information such as the original email headers, sending source, and why it failed. In this milestone, DMARQ will collect, store, and display these failure reports in a dedicated section. This feature is important for deep dives into specific incidents – it complements the aggregate reports by allowing the admin to examine individual spoofing attempts or misconfigurations in detail.
• Architecture: The IMAP fetching logic is extended to also retrieve forensic report emails. These might come from different senders (depending on receivers sending them) and have different formats (often an attachment in Abuse Report Format (ARF) or just an email with original message attached). If using the parsedmarc library, it can automatically parse forensic reports as well (parsedmarc documentation - Open source DMARC report analyzer and visualizer — parsedmarc 8.18.1 documentation), which we can leverage. Otherwise, we implement parsing: when a non-aggregate report email is encountered, we identify it as a forensic report (perhaps by headers or the presence of an “Feedback-Type: auth-failure” in the content). The parser will extract key fields such as: the domain, the original sender and subject, the source IP, the DKIM/SPF alignment results, and any other info from the report. This data is stored in a new ForensicReport table in the database, with a relationship to the Domain. We likely do not store the full email contents to keep DB lean (or we might store it as a blob for completeness, but not necessary to display all). Instead, store the pertinent fields. The front-end gets a new page or tab, e.g., “Forensic Reports” or on each domain’s page a section for recent forensic reports. The UI lists each forensic incident as a row with columns like Date, Sending IP, From address (or header), Subject (if available), and maybe a short reason (e.g., “SPF fail” or “DKIM fail”). The admin can click or expand a report to see more details (headers or the original DKIM signature that failed, etc.). We also include a note that forensic reports may contain sensitive info (since they can include portions of email). Authentication is already in place from milestone 5, so that adds a layer of protection. Optionally, we provide a toggle in settings to enable/disable forensic report collection in case some users prefer not to store these detailed reports (for privacy reasons), but by default if ruf is used, we collect them.
• User Stories:
  • As a security analyst, when I suspect an email spoofing incident, I can look at the DMARC forensic reports in DMARQ to get detailed information. For example, I see a forensic report for domain X on yesterday’s date, which shows an email claiming to be from our CEO (with his address) was sent from IP 1.2.3.4 and failed both SPF and DKIM. The forensic detail includes the mail headers, so I can see the attacker’s sending server info and the exact failure reason. This helps me investigate the incident thoroughly.
  • As an admin, I have a separate view for “DMARC Failure Reports” (forensics) where each entry corresponds to a specific email that failed DMARC. I can quickly scan through these to see if there are any unusual or alarming incidents (like multiple failures from the same source might indicate a persistent attacker). This complements the aggregate view by providing concrete examples of failures.
  • As a user concerned about data volume, I notice that forensic reports are less frequent but more detailed. DMARQ allows me to toggle collection of these reports. If I turn it on, I get rich data for each failure; if I turn it off, I only rely on aggregate data. By having this choice, I can balance detail vs. storage/privacy as needed.

Milestone 10: Self-Service Setup Wizard (Web-Based Configuration)

• Scope & Features: This milestone greatly improves the initial onboarding of DMARQ by adding a web-based setup wizard that runs on first launch (or when configuration is not yet done). Instead of requiring manual environment variable setup or config file editing, a new user can navigate through a series of guided steps in their browser to configure DMARQ. The wizard will cover all essential configurations: creating the admin account, adding the first domains, setting up the email report mailbox connection, and optional integrations (Cloudflare API, notification settings). It’s essentially a one-time assistant to get DMARQ up and running easily. After completion, DMARQ will use the saved configuration and proceed to operate normally. This caters to less technical users and speeds up deployment, while still allowing power users to bypass it with env vars if they prefer.
• Architecture: We implement a multi-step form flow in the FastAPI app. This might be done by having a distinct set of routes (e.g., /setup path) and templates that render each step. Steps likely include: (1) Welcome – check prerequisites, start setup; (2) Admin User – prompt for email and password for the admin account (unless already provided via env, in which case skip or pre-fill); upon submission, create this user in the DB using FastAPI Users; (3) Domain Configuration – prompt for one or more domain names to monitor (the user can list their domains, maybe up to some number, and we create Domain entries in DB); (4) DMARC Report Email Setup – ask whether they have an email inbox for reports and if yes, gather IMAP server, port, username, password (and possibly the sender addresses or subjects to filter, though not necessary if we handle generically); test the connection if possible (try logging in, and maybe fetch one recent email to verify); (5) Integrations – offer to configure optional features: Cloudflare API token (for DNS checks), Apprise notification URL (test sending a test notification?), and whether to enable forensic reports collection; (6) Review & Save – show a summary of settings entered, and allow confirmation. Once confirmed, save all config to the database (or .env as needed, but likely DB since the app is running inside container without writing to env). Mark a flag (like setup_complete=True in a config table or an environment variable) so that the wizard will not show up again on restart. The wizard pages themselves are protected by a temporary token or simply accessible because initially no config means no auth – we should ensure that once the admin user is created at step 2, subsequent steps are either carried out under that new login or the wizard session continues securely. Since this is first-time setup, we can assume the user has exclusive access at that moment. After setup, the app redirects to the normal login or dashboard. Also, we ensure that if environment variables for all crucial settings are already provided (like in a non-interactive deployment), the wizard will auto-skip entirely. For instance, if an admin email/pass and at least one domain and IMAP settings are in env, DMARQ can detect that and mark setup as done. This satisfies headless setups via Docker Compose. The wizard uses the same styling, with a progress indicator for steps, and forms for input (Tailwind forms components).
• User Stories:
  • As a first-time user of DMARQ, I can set it up without touching any config files. When I launch the app, I’m greeted by a setup wizard. I create my admin login, enter my domain names, and provide the credentials of the email account where my DMARC reports are sent. The wizard is interactive and even tells me if something is wrong (like if it can’t connect to the email server). In a few minutes, I finish the wizard and am taken to the dashboard, which now starts showing data. This guided process makes deployment extremely user-friendly.
  • As an administrator deploying via Docker Compose in a production setting, I have the option to pre-configure DMARQ using environment variables (for automation). I can provide values for the admin account and other settings in the compose file. When the container starts, it recognizes that everything it needs is provided, so it bypasses the wizard and goes straight to normal operation (perhaps logging that setup was auto-configured). This flexibility means I can script deployments, but also have a fallback interactive mode for manual setups.
  • As a user revisiting the configuration, if I ever need to change those initial settings, I have a “Settings” section (or I can rerun the wizard) to update things like adding more domains or changing the IMAP password. The system ensures that the configuration is not a one-time black box; the wizard writes to the same database that the app uses, so I can modify settings later through the normal UI (for example, an “Edit Domain” or “Add Domain” function, which was partly introduced in earlier milestones).

Milestone 11: DNS Record Change History Tracking (Optional Enhancement)

• Scope & Features: The final milestone (optional but valuable for completeness) adds the ability to track historical changes in DNS records for the monitored domains. Since DMARQ is already checking DMARC/SPF/DKIM records (from Milestone 8), extending this to log changes over time provides an audit trail. The system will record whenever a domain’s DMARC, SPF, or DKIM record is changed, along with a timestamp and what changed. This helps administrators see, for example, if someone modified a DMARC policy or if a new SPF include was added, and when that happened. It’s especially useful in teams where DNS changes might be done by different members – it brings visibility into those changes in the context of email security. This feature will likely be toggleable, as it’s not critical for everyone, but when enabled it gives a historical view of DNS config alongside the email report data.
• Architecture: We build on the DNS checking mechanism introduced earlier. Each time DMARQ fetches the DNS records (whether via Cloudflare API or a DNS query), it will compare the current record values to the last known values stored in the database. We maintain a table, say DNSRecordHistory, that stores entries like: (domain, record type [DMARC/SPF/DKIM], timestamp, old value, new value). The first time we capture a record, we might log an “initial value” entry. On subsequent checks, if the value differs from the last logged value, we create a new history entry. For example, if example.com had DMARC p=none last week and now it’s p=quarantine, we add a record: Domain=example.com, Type=DMARC, Time=now, Old Value=v=DMARC1; p=none; ..., New Value=v=DMARC1; p=quarantine; .... Similarly for SPF or DKIM (DKIM might have multiple selectors; we track each selector separately). We might limit history to changes only (not logging every daily check if unchanged, to save space). The UI will present this in a user-friendly way. Perhaps a “DNS Change Log” page or a section under each domain’s detail view. It could look like: a list of change events, or a timeline. For readability, we can show a sentence like “[Date]: DMARC record changed from X to Y” with maybe color highlighting differences. If no changes have happened, it might simply say “No DNS changes detected since [start date]”. This feature can be optional via a setting: if disabled, the system won’t store these changes (some may not want the additional storage or have a very stable DNS and don’t need it). Implementing this is relatively straightforward given the data we already gather – it’s mostly about comparing strings and logging differences.
• User Stories:
  • As a domain administrator, I want to know if and when someone changes a DMARC or SPF record for our domains. With DMARQ’s DNS history tracking, I can go to the “DNS Changes” log and see that “On 2025-04-01, the DMARC policy for domainX was changed from none to reject.” This helps with accountability and troubleshooting (e.g., if email suddenly started failing, I might see a change that caused it).
  • As a security auditor, I can use the DNS change history to verify our compliance processes. For instance, I can confirm that we did implement the planned SPF update last month because DMARQ logs show the SPF record change on that date. If an unexpected change appears (one that I wasn’t aware of), it’s a red flag that prompts me to investigate (maybe someone altered a record without following procedure).
  • As a user of DMARQ who might not need this feature, I have the choice to turn off DNS history tracking if I find it unnecessary. DMARQ respects that by not recording every check. But if I enable it, the additional information is seamlessly captured without any manual work, leveraging the existing Cloudflare/DNS integration for maximum insight into my domain’s email security posture.

Conclusion

Through these milestones, DMARQ evolves from a basic DMARC report viewer into a full-featured email authentication monitoring platform. Each milestone is designed to be completed in a short development burst, producing a functional increment. By the final stages, DMARQ offers a comprehensive suite of features: support for large domain sets, rich visual dashboards, proactive alerting, integration with DNS providers for guidance, detailed forensic analysis, and a user-friendly setup and management experience – all while being self-hosted and privacy-conscious. This roadmap ensures that development can be scoped and prioritized, delivering continuous value to users who need professional DMARC insights without sacrificing control over their data.