Christian Krakau-Louis
165 lines
6.6 KiB
Markdown
165 lines
6.6 KiB
Markdown
# DMARQ Milestones
|
|
|
|
Last updated: 2026-05-22
|
|
|
|
DMARQ is now past the original MVP phase. The current product can parse DMARC aggregate reports, import reports from mailboxes, persist data, and present domain/report summaries through the FastAPI/Jinja application. The next work should focus on making ingestion more reliable at production scale and turning the collected data into clearer operational reports.
|
|
|
|
## Milestone 1: Core DMARC Report Processing - Complete
|
|
|
|
Status: Complete
|
|
|
|
Delivered:
|
|
- DMARC aggregate XML parsing with namespace support.
|
|
- ZIP and GZIP report extraction.
|
|
- Upload endpoint for XML, ZIP, and GZIP reports.
|
|
- Upload safety checks for file type, file size, and compressed archive size.
|
|
- Domain/report summary storage and duplicate upload rejection.
|
|
- Parser and upload tests for valid, invalid, compressed, namespaced, and oversized reports.
|
|
|
|
## Milestone 2: Mailbox Ingestion - Complete
|
|
|
|
Status: Complete
|
|
|
|
Delivered:
|
|
- IMAP mail source configuration and connection testing.
|
|
- IMAP polling for DMARC attachments.
|
|
- Gmail OAuth mail source support.
|
|
- Gmail search query tuned for real DMARC report mail, including Google messages with subjects such as `Report domain: <domain> Submitter: google.com`.
|
|
- Gmail import fixed to use the shared parser path.
|
|
- Duplicate report protection for both Gmail and IMAP imports.
|
|
- Background polling and manual poll hooks for configured mail sources.
|
|
|
|
## Milestone 3: Database Foundation, Domain Management, and Auth Foundation - In Progress
|
|
|
|
Status: In progress
|
|
|
|
Delivered:
|
|
- SQLAlchemy models and Alembic migrations.
|
|
- SQLite/PostgreSQL-compatible database configuration.
|
|
- Domain management APIs and UI.
|
|
- Report and source database models.
|
|
- Settings and mail source persistence.
|
|
- Logto-based auth integration plus an explicit local development auth-disabled mode.
|
|
- Security middleware, safer default secret generation, and security-focused tests.
|
|
|
|
Remaining before this milestone is complete:
|
|
- Persist parsed DMARC reports and report records through the database-backed models.
|
|
- Load or query persisted reports after restart so dashboards do not depend on process memory.
|
|
- Move report/domain summary endpoints from the in-memory `ReportStore` to database queries.
|
|
- Keep duplicate report detection consistent across upload, IMAP, and Gmail after persistence is enabled.
|
|
|
|
## Milestone 4: Reporting Quality and Import Confidence - In Progress
|
|
|
|
Status: In progress
|
|
|
|
Goal: make reports trustworthy for day-to-day administration and make import failures obvious.
|
|
|
|
Recently delivered:
|
|
- Gmail import now handles real inbox metadata patterns and Google-style ZIP filenames.
|
|
- Gmail/IMAP imports skip duplicate report IDs to avoid inflated totals.
|
|
- Tests now cover a real Google-style ZIP attachment path rather than only mocked parser behavior.
|
|
- Mail source imports now persist sanitized import-history records for manual and scheduled polls.
|
|
|
|
Next tasks:
|
|
- Finish Milestone 3 report persistence before expanding report features.
|
|
- Add per-import result details: skipped duplicates, parse failures, unsupported attachments, and imported report IDs.
|
|
- Add a UI import history view for each mail source.
|
|
- Add mailbox search controls for date range/backfill without requiring code changes.
|
|
- Improve source aggregation so each sender IP keeps pass/fail totals instead of only the latest result.
|
|
|
|
Exit criteria:
|
|
- A user can connect a mailbox, run a backfill, see exactly what was imported or skipped, and trust that totals are not double-counted.
|
|
|
|
## Milestone 5: Dashboard and Meaningful Reports - Next
|
|
|
|
Status: Planned
|
|
|
|
Goal: convert raw DMARC data into useful operational reporting.
|
|
|
|
Planned:
|
|
- Dashboard trend charts for volume, compliance rate, and failure rate.
|
|
- Top sender/source reports with pass/fail breakdowns.
|
|
- Per-domain report timeline and daily rollups.
|
|
- Exportable reports for a selected domain and date range.
|
|
- Clear recommendations for common cases: unknown source, SPF-only pass, DKIM-only pass, full fail, and policy not enforced.
|
|
|
|
Exit criteria:
|
|
- A domain owner can answer: who is sending as my domain, what is failing, what changed recently, and what should I fix next?
|
|
|
|
## Milestone 6: Production Secret Handling and Deployment Hardening
|
|
|
|
Status: Planned
|
|
|
|
Goal: make production deployments safer and easier to operate.
|
|
|
|
Planned:
|
|
- Document a 1Password-based secret injection flow for local and containerized deployments.
|
|
- Avoid exposing raw mailbox/OAuth secrets in logs, UI responses, and diagnostics.
|
|
- Add startup checks for production-critical configuration.
|
|
- Add backup/restore guidance for database deployments.
|
|
- Add release checklist covering migrations, tests, and smoke checks.
|
|
|
|
Exit criteria:
|
|
- A self-hosted deployment can be configured without copying secrets into source-controlled files or chat logs.
|
|
|
|
## Milestone 7: Notifications and Alert Rules
|
|
|
|
Status: Planned
|
|
|
|
Goal: notify administrators when action is needed.
|
|
|
|
Planned:
|
|
- Apprise notification integration.
|
|
- Alert rules for new sender source, compliance drop, DMARC failures above threshold, and missing reports.
|
|
- Daily/weekly summary notifications.
|
|
- Alert history and test-notification UI.
|
|
|
|
Exit criteria:
|
|
- A user can receive meaningful alerts without opening the dashboard daily.
|
|
|
|
## Milestone 8: DNS Health and Guidance
|
|
|
|
Status: Planned
|
|
|
|
Goal: connect report findings with DNS configuration guidance.
|
|
|
|
Planned:
|
|
- DMARC/SPF/DKIM DNS checks with cached results.
|
|
- DKIM selector discovery from report data.
|
|
- Per-domain DNS health summary.
|
|
- Suggestions for moving from `p=none` to enforcement when compliance supports it.
|
|
- Optional Cloudflare read-only integration for DNS record inspection.
|
|
|
|
Exit criteria:
|
|
- A user can see whether DNS records match the actual senders observed in DMARC reports.
|
|
|
|
## Milestone 9: Setup and Operations Polish
|
|
|
|
Status: Planned
|
|
|
|
Goal: make first-run setup, maintenance, and troubleshooting straightforward.
|
|
|
|
Planned:
|
|
- Guided setup flow for domains and mail sources.
|
|
- Better mailbox test output and recovery suggestions.
|
|
- Health page for scheduler status, last successful import, and database connectivity.
|
|
- Operator documentation for Docker Compose and manual deployments.
|
|
|
|
Exit criteria:
|
|
- A new user can deploy DMARQ, connect a mailbox, and confirm the system is healthy without reading code.
|
|
|
|
## Milestone 10: Forensic Report Support
|
|
|
|
Status: Backlog
|
|
|
|
Goal: support DMARC RUF/forensic reports for individual failure investigation.
|
|
|
|
Planned:
|
|
- Detect forensic report messages.
|
|
- Parse safe metadata from ARF/attached email formats.
|
|
- Store minimal incident details with privacy controls.
|
|
- Add a dedicated forensic report view.
|
|
|
|
Exit criteria:
|
|
- A security analyst can inspect individual failure reports without mixing them into aggregate statistics.
|