gh-christianlouis-dmarq/docs/todo.md

DMARQ Implementation Todo List

This file tracks the specific implementation tasks for each milestone of the DMARQ project.

Milestone 1: Minimal Viable Product (MVP)

Infrastructure Setup

• Set up FastAPI project structure
• Configure Tailwind CSS
• Create Docker and Docker Compose files
• Set up CI/CD pipeline (optional for MVP)

Core DMARC Parser

• Integrate parsedmarc library
• Create parsing service for DMARC XML reports
• Add support for ZIP/GZ compression extraction
• Implement validation for uploaded reports

Data Models

• Create Domain model
• Create AggregateReport model
• Create ReportRecord model for individual sending sources
• Design in-memory storage for MVP phase

API Endpoints

• Create domain registration endpoint
• Create report upload endpoint
• Create domain summary endpoints
• Create detailed report view endpoints

Frontend

• Create base layout template
• Implement dashboard overview page
• Create domain list component
• Build report upload interface
• Implement domain detail view
• Create report detail view
• Add basic visualization components for report statistics

Testing

• Create unit tests for parser
• Create API tests
• Collect sample DMARC reports for testing
• Manual UI testing

Documentation

• Create user guide for MVP
• Document deployment instructions
• Add sample screenshots

Milestone 2: IMAP Integration

IMAP Client

• Create IMAP connection service
• Implement mailbox search functionality for DMARC reports
• Add attachment extraction capabilities
• Create email filtering logic (by sender, subject)
• Add processed email tracking

Scheduler

• Implement background task system
• Create scheduler for periodic mailbox checking
• Add timestamp tracking for fetched reports
• Create logging for background processes

Configuration

• Create configuration model for IMAP settings
• Build configuration UI
• Implement secure credential storage
• Add connection testing functionality

Frontend Updates

• Add IMAP configuration page
• Create last sync indicator
• Implement manual sync trigger button
• Add status indicators for background processes

Milestone 2A: Gmail and Mail Import Reliability

Gmail API Import

• Create Gmail OAuth mail source flow
• Search Gmail for likely DMARC aggregate report messages
• Match Google DMARC subjects like Report domain: example.com Submitter: google.com
• Parse Google-style ZIP attachments such as google.com!example.com!begin!end.zip
• Use the shared DMARC parser for Gmail imports
• Track ingested Gmail message IDs

Duplicate and Error Handling

• Skip duplicate domain/report IDs during Gmail imports
• Skip duplicate domain/report IDs during IMAP imports
• Persist sanitized import errors for API/UI review
• Count duplicate skips separately from parse failures
• Show recent import history in the Mail Sources UI
• Add manual import trigger per mail source
• Add per-import result details for duplicates, parse failures, unsupported attachments, and imported report IDs
• Add retry/backfill controls per mail source

Milestone 3: Database Integration

Database Setup

• Set up SQLAlchemy ORM
• Create SQLite database (for initial version)
• Design database schema with migrations
• Implement data access layer

Model Migration

• Convert report ingestion and dashboard reads from in-memory storage to database-backed storage
• Create Domain table
• Create AggregateReport table
• Create ReportRecord table for sender details
• Implement relationships between models
• Persist parsed upload reports to dmarc_reports and report_records
• Persist parsed Gmail/IMAP reports to dmarc_reports and report_records
• Load/query persisted reports after app restart

Domain Management

• Create UI for adding/editing domains
• Implement domain validation
• Add domain deletion with data cleanup
• Create domain filtering/search for larger sets

Query Optimization

• Add pagination for large report sets
• Implement efficient queries for dashboard stats
• Create data summarization for performance
• Add database indexes for common queries

Milestone 4: Dashboard Enhancements

Data Visualization

• Integrate Chart.js library
• Create time-series charts for DMARC compliance
• Add volume charts for email traffic
• Implement sender breakdown visualizations
• Create policy distribution charts

Dashboard Widgets

• Create compliance rate summary widget
• Add enforcement rate widget
• Implement email volume trends widget
• Create top sender sources widget
• Add alert status summary (for later integration)

Historical Data

• Implement date range filtering
• Create historical trend analysis
• Add data aggregation for different time periods
• Implement data comparison features

Meaningful Reports

• Add per-domain daily rollups
• Add sender/source pass/fail totals
• Add newly observed source detection
• Add exportable domain reports
• Add actionable recommendations for common DMARC failure patterns

Future Milestones

• Production secret handling guide using 1Password injection
• Redact mailbox/OAuth secrets from diagnostics, logs, import history, and auth contexts
• Add startup checks for production-critical configuration
• Apprise notifications and alert rules
• DNS health guidance and Cloudflare read-only inspection
• Guided setup and operator health pages
• Forensic/RUF report support