diff --git a/docs/architecture.md b/docs/architecture.md index e16ee8a..7314e2d 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,216 +1,173 @@ -# DMARQ – Architecture & Tech Stack Overview +# DMARQ Architecture -**Project:** DMARQ -**Host:** https://app.dmarq.org -**Purpose:** Self-hosted, full-featured DMARC monitoring tool with support for Cloudflare integration, alerting, and visual dashboards. +## System Overview ---- +DMARQ is designed as a self-contained application that processes DMARC reports, stores relevant data, and presents insights through a web interface. The architecture follows a layered approach with clear separation of concerns between components. -## 🧱 System Architecture +## Core Components -DMARQ uses an integrated architecture with: +### Web Application (FastAPI) +- Serves the web interface +- Handles API requests +- Manages user authentication +- Coordinates background tasks +- Renders templates with Jinja2 -- **Unified Backend** (FastAPI with Jinja2 templates) -- **Modern UI** (Jinja2 + Tailwind CSS + shadcn/ui) +### DMARC Processing Engine +- Parses DMARC XML reports +- Extracts meaningful data from reports +- Validates report structure +- Identifies sending sources +- Calculates compliance metrics -The application is deployed via Docker with a PostgreSQL database storing parsed reports, domain configurations, DNS snapshots, and user information. +### Data Storage +- **MVP Phase**: In-memory storage +- **Later Phases**: SQLite or PostgreSQL database +- Stores domain configurations +- Maintains report history +- Tracks sender statistics -Optional services (e.g., Apprise for alerts) are included via container or integrated via API calls. +### Report Acquisition +- **MVP Phase**: Manual file upload +- **Later Phases**: IMAP client for automatic retrieval +- Handles compression formats (ZIP, GZ) +- Deduplicates reports ---- +### Background Processing +- Scheduled report fetching +- Periodic DNS checks +- Alert evaluation +- Data aggregation for dashboards -## 🧩 Tech Stack +## Data Flow -| Component | Stack / Tooling | -|--------------------|---------------------------------------------| -| **Frontend** | Jinja2 Templates + HTMX + Tailwind CSS + shadcn/ui | -| **Charts** | Chart.js with Alpine.js integration | -| **Routing/Auth** | FastAPI routing + JWT auth (FastAPI Users) | -| **Backend** | FastAPI + SQLAlchemy | -| **ORM & DB** | SQLAlchemy ORM, PostgreSQL | -| **IMAP** | `imap-tools`, `aioimaplib` | -| **DMARC Parsing** | `defusedxml`, `lxml`, `zipfile`, `mail-parser` | -| **Cloudflare API** | `cloudflare` Python SDK or raw REST client | -| **DNS Resolution** | `dnspython` | -| **Authentication** | FastAPI Users (JWT + optional OAuth later) | -| **Alerting** | [Apprise](https://github.com/caronc/apprise) | -| **Testing** | `pytest`, `coverage`, `pytest-mock` | -| **CI/CD (optional)**| GitHub Actions, Docker Hub | -| **Deployment** | Docker, Docker Compose | -| **Config Mgmt** | `dynaconf` (ENV + DB integration) | +1. **Report Ingestion** + - Reports arrive via upload or IMAP + - System extracts and validates XML content + - Parser processes report data + - Data is stored in appropriate format ---- +2. **Data Processing** + - Raw report data is transformed into metrics + - System calculates compliance rates + - Identifies new or problematic senders + - Updates historical records -## πŸ“¦ Application Structure +3. **Presentation Layer** + - Dashboard displays key metrics + - Domain details show specific report data + - Charts visualize trends + - Alerts highlight issues requiring attention + +## Architectural Evolution + +The DMARQ architecture is designed to evolve across milestones: + +### Milestone 1: Minimal Architecture +- Single FastAPI service +- In-memory data storage +- Manual file upload +- Basic template rendering ``` -dmarq/ -β”œβ”€β”€ app/ -β”‚ β”œβ”€β”€ api/ # REST API endpoints (v1) -β”‚ β”œβ”€β”€ core/ # App config, security, constants -β”‚ β”œβ”€β”€ models/ # SQLAlchemy ORM models -β”‚ β”œβ”€β”€ services/ # Mail parsing, DNS, CF integrations -β”‚ β”œβ”€β”€ static/ # CSS (Tailwind), JS, images -β”‚ β”‚ β”œβ”€β”€ css/ # Generated Tailwind styles -β”‚ β”‚ β”œβ”€β”€ js/ # Alpine.js and other frontend scripts -β”‚ β”‚ └── img/ # Images and icons -β”‚ β”œβ”€β”€ tasks/ # Scheduled tasks (polling, DNS sync) -β”‚ β”œβ”€β”€ templates/ # Jinja2 templates -β”‚ β”‚ β”œβ”€β”€ components/ # Reusable UI components -β”‚ β”‚ β”œβ”€β”€ dashboard/ # Dashboard views -β”‚ β”‚ β”œβ”€β”€ layouts/ # Base layouts -β”‚ β”‚ β”œβ”€β”€ reports/ # Report-specific templates -β”‚ β”‚ └── wizard/ # Setup wizard templates -β”‚ β”œβ”€β”€ tests/ # Unit + integration tests -β”‚ └── main.py # FastAPI application entrypoint -β”œβ”€β”€ Dockerfile -β”œβ”€β”€ docker-compose.yml -β”œβ”€β”€ config/ -β”‚ └── seed_env.py # Load ENV vars into DB -β”œβ”€β”€ .env.example -β”œβ”€β”€ README.md -└── ARCHITECTURE.md +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Web Browser β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FastAPI Application β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ DMARC Parser β”‚ Templates β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ In-Memory Store β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ---- +### Milestone 2-3: Enhanced Architecture +- FastAPI with background tasks +- Database persistence layer +- IMAP integration +- Expanded web interface -## 🌐 Functional Modules - -### 1. **Config Wizard (Web-Based)** -- First step of app usage -- Collects: - - Admin user creation - - IMAP mailbox login - - Cloudflare API token - - Optional alert channels -- Saves config into DB -- Optionally seeded from `.env` - ---- - -### 2. **Email Processing** -- IMAP polling for inbox (e.g. `dmarc@yourdomain.com`) -- Download zipped aggregate XML or forensic reports -- Parse with validation and deduplication -- Store: - - Reporting org - - Source IPs, volume - - SPF/DKIM result - - Applied disposition (none, quarantine, reject) - - Forensics: failed messages, sample data - ---- - -### 3. **Cloudflare DNS Sync** -- List zones and domains via API -- Pull DNS records: - - DMARC - - SPF - - DKIM - - MX - - BIMI (optional) -- Validate correctness and format -- Generate actionable **fix suggestions** -- DNS updates require manual user approval - ---- - -### 4. **Alerting (via Apprise)** -- Alert on: - - New forensic reports - - New source IPs failing SPF/DKIM - - Compliance drops (configurable) -- Supports: - - Email - - Slack - - Discord - - Webhooks - - Matrix -- Configurable via web wizard and/or user dashboard - ---- - -### 5. **User Authentication** -- FastAPI Users backend -- JWT token auth -- Server-side sessions with secure cookies -- Admin-only access to DNS fix or config modules - ---- - -## πŸ§ͺ Testing Strategy - -- **Unit Tests:** All parsing, validation, config, services -- **Mock External Services:** IMAP, Cloudflare, DNS -- **Frontend:** Testing Jinja templates with pytest-html -- **E2E (later):** Playwright or Selenium - -Run with: -```bash -docker compose exec app pytest +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Web Browser β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FastAPI Application β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚Templatesβ”‚Parser β”‚IMAP β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Background Tasks β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Database Layer β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ SQLite/PostgreSQL DB β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` ---- +### Milestone 5+: Full Architecture +- Authentication layer +- DNS integration +- Alert system +- Visualization enhancements -## πŸ§‘β€πŸŽ¨ UI Implementation +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Web Browser β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FastAPI Application β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚Auth β”‚Templates β”‚Parser β”‚IMAP β”‚Appriseβ”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Background Tasks β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Database Layer β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ SQLite/PostgreSQL DB β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ β”‚ β”‚ +β”Œβ”€β”€β”€β”΄β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β” +β”‚DNS APIβ”‚ β”‚Notif. APIβ”‚ β”‚Email β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` -### Frontend Technology +## Technology Stack Details -DMARQ uses an integrated approach with: +### Backend Framework +- **FastAPI**: Modern, high-performance web framework +- **Pydantic**: Data validation and settings management +- **SQLAlchemy**: ORM for database interactions +- **APScheduler**: Task scheduling for background jobs -1. **Jinja2 Templates**: Server-side rendering of HTML -2. **Tailwind CSS**: Utility-first CSS framework for styling -3. **shadcn/ui**: Component library adapted for server-rendered templates -4. **Alpine.js**: Minimal JavaScript framework for enhanced interactivity -5. **HTMX**: For AJAX requests without writing JavaScript -6. **Chart.js**: For data visualization components +### Frontend +- **Jinja2**: Template engine for rendering HTML +- **Tailwind CSS**: Utility-first CSS framework +- **ShadCN/UI**: Reusable UI component system +- **Chart.js**: Lightweight charting library -This approach offers several advantages: -- Eliminates API-related complexity -- Reduces JavaScript bundle size -- Improves initial page load performance -- Simplifies deployment (single container) -- Server-side rendering improves SEO +### External Libraries +- **parsedmarc**: DMARC report parsing +- **Apprise**: Unified notification system +- **Cloudflare API** (optional): DNS integration +- **FastAPI Users**: Authentication management -### UI Components Structure +### Deployment +- **Docker**: Containerization +- **Docker Compose**: Multi-container orchestration +- **SQLite/PostgreSQL**: Database options -- **Layouts**: Base templates that define the page structure -- **Components**: Reusable UI elements like cards, tables, and forms -- **Pages**: Full page templates for dashboard, reports, settings, etc. +## Security Considerations -The components follow shadcn/ui design patterns but are implemented as Jinja2 macros or includes rather than React components. - ---- - -## πŸ§‘β€πŸŽ¨ Branding & UI Design - -- **Logo:** Stylized shield with "Q" + monogram "D+Q" -- **Colors:** - - Deep Blue `#1A237E` - - Teal `#00ACC1` - - Orange `#FF7043` - - Light Gray `#F5F5F5`, Dark Gray `#212121` -- **Fonts:** Montserrat (headings), Open Sans (body) -- **Style:** Minimal, modern, flat icons β€” inspired by EasyDMARC - ---- - -## 🚧 Known Constraints - -- Single-instance deployment (no horizontal scaling) -- Target performance: 50–100 domains per instance -- Database optimization is secondary -- Multitenancy is not supported (yet) - ---- - -## πŸ“˜ License - -Apache License 2.0 β€” Free for personal or commercial use with attribution. - -**License Compatibility:** -All major dependencies and tools used in DMARQ (including FastAPI, SQLAlchemy, Tailwind CSS, shadcn/ui, Alpine.js, HTMX, Chart.js, Apprise, and others) are distributed under permissive licenses (MIT, BSD, Apache 2.0, ISC, or similar) and are compatible with the Apache 2.0 license. - ---- - -This document serves as the technical foundation for the implementation of DMARQ. \ No newline at end of file +- Sensitive credentials are stored securely +- Authentication protects access to report data +- HTTPS recommended for production deployment +- No external APIs required for core functionality +- Self-hosted approach keeps data private \ No newline at end of file diff --git a/docs/todo.md b/docs/todo.md new file mode 100644 index 0000000..a4cc77c --- /dev/null +++ b/docs/todo.md @@ -0,0 +1,129 @@ +# DMARQ Implementation Todo List + +This file tracks the specific implementation tasks for each milestone of the DMARQ project. + +## Milestone 1: Minimal Viable Product (MVP) + +### Infrastructure Setup +- [ ] Set up FastAPI project structure +- [ ] Configure Tailwind CSS +- [ ] Add ShadCN/UI components library +- [ ] Create Docker and Docker Compose files +- [ ] Set up CI/CD pipeline (optional for MVP) + +### Core DMARC Parser +- [ ] Integrate parsedmarc library +- [ ] Create parsing service for DMARC XML reports +- [ ] Add support for ZIP/GZ compression extraction +- [ ] Implement validation for uploaded reports + +### Data Models +- [ ] Create Domain model +- [ ] Create AggregateReport model +- [ ] Create ReportRecord model for individual sending sources +- [ ] Design in-memory storage for MVP phase + +### API Endpoints +- [ ] Create domain registration endpoint +- [ ] Create report upload endpoint +- [ ] Create domain summary endpoints +- [ ] Create detailed report view endpoints + +### Frontend +- [ ] Create base layout template +- [ ] Implement dashboard overview page +- [ ] Create domain list component +- [ ] Build report upload interface +- [ ] Implement domain detail view +- [ ] Create report detail view +- [ ] Add basic visualization components for report statistics + +### Testing +- [ ] Create unit tests for parser +- [ ] Create API tests +- [ ] Collect sample DMARC reports for testing +- [ ] Manual UI testing + +### Documentation +- [ ] Create user guide for MVP +- [ ] Document deployment instructions +- [ ] Add sample screenshots + +## Milestone 2: IMAP Integration + +### IMAP Client +- [ ] Create IMAP connection service +- [ ] Implement mailbox search functionality for DMARC reports +- [ ] Add attachment extraction capabilities +- [ ] Create email filtering logic (by sender, subject) +- [ ] Add processed email tracking + +### Scheduler +- [ ] Implement background task system +- [ ] Create scheduler for periodic mailbox checking +- [ ] Add timestamp tracking for fetched reports +- [ ] Create logging for background processes + +### Configuration +- [ ] Create configuration model for IMAP settings +- [ ] Build configuration UI +- [ ] Implement secure credential storage +- [ ] Add connection testing functionality + +### Frontend Updates +- [ ] Add IMAP configuration page +- [ ] Create last sync indicator +- [ ] Implement manual sync trigger button +- [ ] Add status indicators for background processes + +## Milestone 3: Database Integration + +### Database Setup +- [ ] Set up SQLAlchemy ORM +- [ ] Create SQLite database (for initial version) +- [ ] Design database schema with migrations +- [ ] Implement data access layer + +### Model Migration +- [ ] Convert in-memory models to database models +- [ ] Create Domain table +- [ ] Create AggregateReport table +- [ ] Create ReportRecord table for sender details +- [ ] Implement relationships between models + +### Domain Management +- [ ] Create UI for adding/editing domains +- [ ] Implement domain validation +- [ ] Add domain deletion with data cleanup +- [ ] Create domain filtering/search for larger sets + +### Query Optimization +- [ ] Add pagination for large report sets +- [ ] Implement efficient queries for dashboard stats +- [ ] Create data summarization for performance +- [ ] Add database indexes for common queries + +## Milestone 4: Dashboard Enhancements + +### Data Visualization +- [ ] Integrate Chart.js library +- [ ] Create time-series charts for DMARC compliance +- [ ] Add volume charts for email traffic +- [ ] Implement sender breakdown visualizations +- [ ] Create policy distribution charts + +### Dashboard Widgets +- [ ] Create compliance rate summary widget +- [ ] Add enforcement rate widget +- [ ] Implement email volume trends widget +- [ ] Create top sender sources widget +- [ ] Add alert status summary (for later integration) + +### Historical Data +- [ ] Implement date range filtering +- [ ] Create historical trend analysis +- [ ] Add data aggregation for different time periods +- [ ] Implement data comparison features + +## Future Milestones +Additional tasks for Milestones 5-11 will be added as we approach those phases of development. \ No newline at end of file