gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add implementation todo list for DMARQ project milestones

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-04-20 18:43:42 +02:00
parent 69f8438a36
commit 1b79ec4f20
2 changed files with 275 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture.md
+146 -189
View File
@@ -1,216 +1,173 @@
# DMARQ – Architecture & Tech Stack Overview # DMARQ Architecture
**Project:** DMARQ ## System Overview
**Host:** https://app.dmarq.org
**Purpose:** Self-hosted, full-featured DMARC monitoring tool with support for Cloudflare integration, alerting, and visual dashboards.
--- DMARQ is designed as a self-contained application that processes DMARC reports, stores relevant data, and presents insights through a web interface. The architecture follows a layered approach with clear separation of concerns between components.
## 🧱 System Architecture ## Core Components
DMARQ uses an integrated architecture with: ### Web Application (FastAPI)
- Serves the web interface
- Handles API requests
- Manages user authentication
- Coordinates background tasks
- Renders templates with Jinja2
- **Unified Backend** (FastAPI with Jinja2 templates) ### DMARC Processing Engine
- **Modern UI** (Jinja2 + Tailwind CSS + shadcn/ui) - Parses DMARC XML reports
- Extracts meaningful data from reports
- Validates report structure
- Identifies sending sources
- Calculates compliance metrics
The application is deployed via Docker with a PostgreSQL database storing parsed reports, domain configurations, DNS snapshots, and user information. ### Data Storage
- **MVP Phase**: In-memory storage
- **Later Phases**: SQLite or PostgreSQL database
- Stores domain configurations
- Maintains report history
- Tracks sender statistics
Optional services (e.g., Apprise for alerts) are included via container or integrated via API calls. ### Report Acquisition
- **MVP Phase**: Manual file upload
- **Later Phases**: IMAP client for automatic retrieval
- Handles compression formats (ZIP, GZ)
- Deduplicates reports
--- ### Background Processing
- Scheduled report fetching
- Periodic DNS checks
- Alert evaluation
- Data aggregation for dashboards
## 🧩 Tech Stack ## Data Flow
| Component | Stack / Tooling | 1. **Report Ingestion**
|--------------------|---------------------------------------------| - Reports arrive via upload or IMAP
| **Frontend** | Jinja2 Templates + HTMX + Tailwind CSS + shadcn/ui | - System extracts and validates XML content
| **Charts** | Chart.js with Alpine.js integration | - Parser processes report data
| **Routing/Auth** | FastAPI routing + JWT auth (FastAPI Users) | - Data is stored in appropriate format
| **Backend** | FastAPI + SQLAlchemy |
| **ORM & DB** | SQLAlchemy ORM, PostgreSQL |
| **IMAP** | `imap-tools`, `aioimaplib` |
| **DMARC Parsing** | `defusedxml`, `lxml`, `zipfile`, `mail-parser` |
| **Cloudflare API** | `cloudflare` Python SDK or raw REST client |
| **DNS Resolution** | `dnspython` |
| **Authentication** | FastAPI Users (JWT + optional OAuth later) |
| **Alerting** | [Apprise](https://github.com/caronc/apprise) |
| **Testing** | `pytest`, `coverage`, `pytest-mock` |
| **CI/CD (optional)**| GitHub Actions, Docker Hub |
| **Deployment** | Docker, Docker Compose |
| **Config Mgmt** | `dynaconf` (ENV + DB integration) |
--- 2. **Data Processing**
- Raw report data is transformed into metrics
- System calculates compliance rates
- Identifies new or problematic senders
- Updates historical records
## 📦 Application Structure 3. **Presentation Layer**
- Dashboard displays key metrics
- Domain details show specific report data
- Charts visualize trends
- Alerts highlight issues requiring attention
## Architectural Evolution
The DMARQ architecture is designed to evolve across milestones:
### Milestone 1: Minimal Architecture
- Single FastAPI service
- In-memory data storage
- Manual file upload
- Basic template rendering
``` ```
dmarq/ ┌─────────────────────────┐
├── app/ │ Web Browser │
│ ├── api/ # REST API endpoints (v1) └───────────┬─────────────┘
│ ├── core/ # App config, security, constants │
│ ├── models/ # SQLAlchemy ORM models ┌───────────┼─────────────┐
│ ├── services/ # Mail parsing, DNS, CF integrations │ FastAPI Application │
│ ├── static/ # CSS (Tailwind), JS, images ├───────────┼─────────────┤
│ │ ├── css/ # Generated Tailwind styles │ DMARC Parser │ Templates │
│ │ ├── js/ # Alpine.js and other frontend scripts ├─────────────┬───────────┤
│ │ └── img/ # Images and icons │ In-Memory Store │
│ ├── tasks/ # Scheduled tasks (polling, DNS sync) └─────────────────────────┘
│ ├── templates/ # Jinja2 templates
│ │ ├── components/ # Reusable UI components
│ │ ├── dashboard/ # Dashboard views
│ │ ├── layouts/ # Base layouts
│ │ ├── reports/ # Report-specific templates
│ │ └── wizard/ # Setup wizard templates
│ ├── tests/ # Unit + integration tests
│ └── main.py # FastAPI application entrypoint
├── Dockerfile
├── docker-compose.yml
├── config/
│ └── seed_env.py # Load ENV vars into DB
├── .env.example
├── README.md
└── ARCHITECTURE.md
``` ```
--- ### Milestone 2-3: Enhanced Architecture
- FastAPI with background tasks
- Database persistence layer
- IMAP integration
- Expanded web interface
## 🌐 Functional Modules ```
┌─────────────────────────┐
### 1. **Config Wizard (Web-Based)** │ Web Browser │
- First step of app usage └───────────┬─────────────┘
- Collects: │
- Admin user creation ┌───────────┼─────────────┐
- IMAP mailbox login │ FastAPI Application │
- Cloudflare API token ├─────────┬───────┬───────┤
- Optional alert channels │Templates│Parser │IMAP │
- Saves config into DB ├─────────┴───────┴───────┤
- Optionally seeded from `.env` │ Background Tasks │
├─────────────────────────┤
--- │ Database Layer │
└───────────┬─────────────┘
### 2. **Email Processing** │
- IMAP polling for inbox (e.g. `dmarc@yourdomain.com`) ┌───────────┼─────────────┐
- Download zipped aggregate XML or forensic reports │ SQLite/PostgreSQL DB │
- Parse with validation and deduplication └─────────────────────────┘
- Store:
- Reporting org
- Source IPs, volume
- SPF/DKIM result
- Applied disposition (none, quarantine, reject)
- Forensics: failed messages, sample data
---
### 3. **Cloudflare DNS Sync**
- List zones and domains via API
- Pull DNS records:
- DMARC
- SPF
- DKIM
- MX
- BIMI (optional)
- Validate correctness and format
- Generate actionable **fix suggestions**
- DNS updates require manual user approval
---
### 4. **Alerting (via Apprise)**
- Alert on:
- New forensic reports
- New source IPs failing SPF/DKIM
- Compliance drops (configurable)
- Supports:
- Email
- Slack
- Discord
- Webhooks
- Matrix
- Configurable via web wizard and/or user dashboard
---
### 5. **User Authentication**
- FastAPI Users backend
- JWT token auth
- Server-side sessions with secure cookies
- Admin-only access to DNS fix or config modules
---
## 🧪 Testing Strategy
- **Unit Tests:** All parsing, validation, config, services
- **Mock External Services:** IMAP, Cloudflare, DNS
- **Frontend:** Testing Jinja templates with pytest-html
- **E2E (later):** Playwright or Selenium
Run with:
```bash
docker compose exec app pytest
``` ```
--- ### Milestone 5+: Full Architecture
- Authentication layer
- DNS integration
- Alert system
- Visualization enhancements
## 🧑‍🎨 UI Implementation ```
┌───────────────────────────────────────────────┐
│ Web Browser │
└───────────────────┬───────────────────────────┘
│
┌───────────────────┼───────────────────────────┐
│ FastAPI Application │
├────────┬──────────┬──────────┬────────┬───────┤
│Auth │Templates │Parser │IMAP │Apprise│
├────────┴──────────┴──────────┴────────┴───────┤
│ Background Tasks │
├───────────────────────────────────────────────┤
│ Database Layer │
└───────────────────┬───────────────────────────┘
│
┌───────────────────┼───────────────────────────┐
│ SQLite/PostgreSQL DB │
└───────────────────────────────────────────────┘
│ │ │
┌───┴───┐ ┌─────┴────┐ ┌─────┴────┐
│DNS API│ │Notif. API│ │Email │
└───────┘ └──────────┘ └──────────┘
```
### Frontend Technology ## Technology Stack Details
DMARQ uses an integrated approach with: ### Backend Framework
- **FastAPI**: Modern, high-performance web framework
- **Pydantic**: Data validation and settings management
- **SQLAlchemy**: ORM for database interactions
- **APScheduler**: Task scheduling for background jobs
1. **Jinja2 Templates**: Server-side rendering of HTML ### Frontend
2. **Tailwind CSS**: Utility-first CSS framework for styling - **Jinja2**: Template engine for rendering HTML
3. **shadcn/ui**: Component library adapted for server-rendered templates - **Tailwind CSS**: Utility-first CSS framework
4. **Alpine.js**: Minimal JavaScript framework for enhanced interactivity - **ShadCN/UI**: Reusable UI component system
5. **HTMX**: For AJAX requests without writing JavaScript - **Chart.js**: Lightweight charting library
6. **Chart.js**: For data visualization components
This approach offers several advantages: ### External Libraries
- Eliminates API-related complexity - **parsedmarc**: DMARC report parsing
- Reduces JavaScript bundle size - **Apprise**: Unified notification system
- Improves initial page load performance - **Cloudflare API** (optional): DNS integration
- Simplifies deployment (single container) - **FastAPI Users**: Authentication management
- Server-side rendering improves SEO
### UI Components Structure ### Deployment
- **Docker**: Containerization
- **Docker Compose**: Multi-container orchestration
- **SQLite/PostgreSQL**: Database options
- **Layouts**: Base templates that define the page structure ## Security Considerations
- **Components**: Reusable UI elements like cards, tables, and forms
- **Pages**: Full page templates for dashboard, reports, settings, etc.
The components follow shadcn/ui design patterns but are implemented as Jinja2 macros or includes rather than React components. - Sensitive credentials are stored securely
- Authentication protects access to report data
--- - HTTPS recommended for production deployment
- No external APIs required for core functionality
## 🧑‍🎨 Branding & UI Design - Self-hosted approach keeps data private
- **Logo:** Stylized shield with "Q" + monogram "D+Q"
- **Colors:**
- Deep Blue `#1A237E`
- Teal `#00ACC1`
- Orange `#FF7043`
- Light Gray `#F5F5F5`, Dark Gray `#212121`
- **Fonts:** Montserrat (headings), Open Sans (body)
- **Style:** Minimal, modern, flat icons — inspired by EasyDMARC
---
## 🚧 Known Constraints
- Single-instance deployment (no horizontal scaling)
- Target performance: 50–100 domains per instance
- Database optimization is secondary
- Multitenancy is not supported (yet)
---
## 📘 License
Apache License 2.0 — Free for personal or commercial use with attribution.
**License Compatibility:**
All major dependencies and tools used in DMARQ (including FastAPI, SQLAlchemy, Tailwind CSS, shadcn/ui, Alpine.js, HTMX, Chart.js, Apprise, and others) are distributed under permissive licenses (MIT, BSD, Apache 2.0, ISC, or similar) and are compatible with the Apache 2.0 license.
---
This document serves as the technical foundation for the implementation of DMARQ.
docs/todo.md
+129
View File
@@ -0,0 +1,129 @@
# DMARQ Implementation Todo List
This file tracks the specific implementation tasks for each milestone of the DMARQ project.
## Milestone 1: Minimal Viable Product (MVP)
### Infrastructure Setup
- [ ] Set up FastAPI project structure
- [ ] Configure Tailwind CSS
- [ ] Add ShadCN/UI components library
- [ ] Create Docker and Docker Compose files
- [ ] Set up CI/CD pipeline (optional for MVP)
### Core DMARC Parser
- [ ] Integrate parsedmarc library
- [ ] Create parsing service for DMARC XML reports
- [ ] Add support for ZIP/GZ compression extraction
- [ ] Implement validation for uploaded reports
### Data Models
- [ ] Create Domain model
- [ ] Create AggregateReport model
- [ ] Create ReportRecord model for individual sending sources
- [ ] Design in-memory storage for MVP phase
### API Endpoints
- [ ] Create domain registration endpoint
- [ ] Create report upload endpoint
- [ ] Create domain summary endpoints
- [ ] Create detailed report view endpoints
### Frontend
- [ ] Create base layout template
- [ ] Implement dashboard overview page
- [ ] Create domain list component
- [ ] Build report upload interface
- [ ] Implement domain detail view
- [ ] Create report detail view
- [ ] Add basic visualization components for report statistics
### Testing
- [ ] Create unit tests for parser
- [ ] Create API tests
- [ ] Collect sample DMARC reports for testing
- [ ] Manual UI testing
### Documentation
- [ ] Create user guide for MVP
- [ ] Document deployment instructions
- [ ] Add sample screenshots
## Milestone 2: IMAP Integration
### IMAP Client
- [ ] Create IMAP connection service
- [ ] Implement mailbox search functionality for DMARC reports
- [ ] Add attachment extraction capabilities
- [ ] Create email filtering logic (by sender, subject)
- [ ] Add processed email tracking
### Scheduler
- [ ] Implement background task system
- [ ] Create scheduler for periodic mailbox checking
- [ ] Add timestamp tracking for fetched reports
- [ ] Create logging for background processes
### Configuration
- [ ] Create configuration model for IMAP settings
- [ ] Build configuration UI
- [ ] Implement secure credential storage
- [ ] Add connection testing functionality
### Frontend Updates
- [ ] Add IMAP configuration page
- [ ] Create last sync indicator
- [ ] Implement manual sync trigger button
- [ ] Add status indicators for background processes
## Milestone 3: Database Integration
### Database Setup
- [ ] Set up SQLAlchemy ORM
- [ ] Create SQLite database (for initial version)
- [ ] Design database schema with migrations
- [ ] Implement data access layer
### Model Migration
- [ ] Convert in-memory models to database models
- [ ] Create Domain table
- [ ] Create AggregateReport table
- [ ] Create ReportRecord table for sender details
- [ ] Implement relationships between models
### Domain Management
- [ ] Create UI for adding/editing domains
- [ ] Implement domain validation
- [ ] Add domain deletion with data cleanup
- [ ] Create domain filtering/search for larger sets
### Query Optimization
- [ ] Add pagination for large report sets
- [ ] Implement efficient queries for dashboard stats
- [ ] Create data summarization for performance
- [ ] Add database indexes for common queries
## Milestone 4: Dashboard Enhancements
### Data Visualization
- [ ] Integrate Chart.js library
- [ ] Create time-series charts for DMARC compliance
- [ ] Add volume charts for email traffic
- [ ] Implement sender breakdown visualizations
- [ ] Create policy distribution charts
### Dashboard Widgets
- [ ] Create compliance rate summary widget
- [ ] Add enforcement rate widget
- [ ] Implement email volume trends widget
- [ ] Create top sender sources widget
- [ ] Add alert status summary (for later integration)
### Historical Data
- [ ] Implement date range filtering
- [ ] Create historical trend analysis
- [ ] Add data aggregation for different time periods
- [ ] Implement data comparison features
## Future Milestones
Additional tasks for Milestones 5-11 will be added as we approach those phases of development.