fix: harden DMARC mail import
This commit is contained in:
Christian Krakau-Louis
+56
-399
@@ -1,418 +1,75 @@
|
||||
# DMARQ Security-Enhanced Roadmap
|
||||
# DMARQ Development Roadmap
|
||||
|
||||
## Document Purpose
|
||||
Last updated: 2026-05-22
|
||||
|
||||
This roadmap outlines the development plan for DMARQ with an enhanced focus on security, code quality, and preparation for agentic coding (AI-assisted development). This document supersedes previous roadmap versions with security milestones integrated throughout.
|
||||
This roadmap tracks implementation status and near-term engineering priorities. The project has moved beyond the initial MVP and now needs production-grade import visibility, better reporting, and hardened operations.
|
||||
|
||||
**Last Updated**: 2026-02-06
|
||||
**Status**: Active Development
|
||||
## Current Status
|
||||
|
||||
---
|
||||
Complete:
|
||||
- Core DMARC aggregate parsing for XML, ZIP, and GZIP reports.
|
||||
- Secure XML parsing with `defusedxml`.
|
||||
- Upload validation and archive safety checks.
|
||||
- IMAP mailbox ingestion.
|
||||
- Gmail OAuth ingestion.
|
||||
- Persistent database models and migrations.
|
||||
- Domain, report, settings, and mail source APIs.
|
||||
- Dashboard and domain detail views.
|
||||
- Logto auth integration and local development auth-disabled mode.
|
||||
- DNS resolver foundation and DNS-related endpoint tests.
|
||||
|
||||
## Current Status (Milestone 1 - COMPLETE ✅)
|
||||
Recently improved:
|
||||
- Gmail ingestion now matches the real Google DMARC report messages found in the connected inbox.
|
||||
- Gmail import now uses the same parser path as uploads and IMAP.
|
||||
- Gmail and IMAP imports now skip duplicate domain/report IDs.
|
||||
- Tests cover Google-style DMARC ZIP attachment imports.
|
||||
|
||||
### Achievements
|
||||
- ✅ Basic DMARC report parsing (XML, ZIP, GZIP)
|
||||
- ✅ In-memory storage for up to 5 domains
|
||||
- ✅ Simple dashboard UI
|
||||
- ✅ Report upload functionality
|
||||
- ✅ Domain overview with compliance stats
|
||||
## Active Milestone: Reporting Quality and Import Confidence
|
||||
|
||||
### Security Status
|
||||
⚠️ **Multiple critical security issues identified** - See [SECURITY.md](../../SECURITY.md) for details
|
||||
Objective: make mailbox imports auditable and make report totals trustworthy.
|
||||
|
||||
---
|
||||
Priority tasks:
|
||||
- Persist import attempts with message ID, source, attachment filename, outcome, and sanitized error details.
|
||||
- Show import history on mail source detail pages.
|
||||
- Report duplicate skips separately from parse failures.
|
||||
- Add backfill controls for Gmail and IMAP sources.
|
||||
- Improve source rollups so a source IP tracks pass/fail counts over time.
|
||||
|
||||
## Security Remediation Sprint (PRIORITY - In Progress)
|
||||
Quality bar:
|
||||
- Importing the same mailbox twice must not change aggregate totals.
|
||||
- Parse failures must be visible and actionable.
|
||||
- The user should be able to tell whether a mail source is healthy without reading logs.
|
||||
|
||||
**Timeline**: Immediate (Next 2-4 weeks)
|
||||
**Status**: 🔄 In Progress
|
||||
## Next Milestone: Meaningful Reports
|
||||
|
||||
### Critical Fixes Required
|
||||
Objective: turn parsed DMARC data into administrator-friendly reports.
|
||||
|
||||
#### 1. Authentication & Authorization (CRITICAL)
|
||||
- [ ] Add authentication middleware to all admin endpoints
|
||||
- [ ] Implement proper user authentication system
|
||||
- [ ] Add authorization checks on sensitive operations
|
||||
- [ ] Add rate limiting to prevent abuse
|
||||
- **Files to Fix**:
|
||||
- `backend/app/main.py` (lines 195-196, 224-225)
|
||||
- `backend/app/api/api_v1/endpoints/imap.py`
|
||||
- `backend/app/api/api_v1/endpoints/domains.py`
|
||||
Priority tasks:
|
||||
- Add time-series charts for volume and compliance.
|
||||
- Add per-domain daily rollups.
|
||||
- Add sender/source breakdowns with SPF, DKIM, and disposition counts.
|
||||
- Add "what changed" summaries for newly observed senders and sudden compliance drops.
|
||||
- Add exportable reports for a domain and date range.
|
||||
|
||||
#### 2. Secret Management (CRITICAL)
|
||||
- [ ] Remove default SECRET_KEY value
|
||||
- [ ] Add SECRET_KEY validation on startup
|
||||
- [ ] Document secret generation in deployment guide
|
||||
- [ ] Add warning if default secret is detected
|
||||
- **Files to Fix**:
|
||||
- `backend/app/core/config.py` (line 24)
|
||||
- Documentation updates
|
||||
Quality bar:
|
||||
- A domain owner can understand who sends mail as their domain, which sources fail, and what to fix next.
|
||||
|
||||
#### 3. XML Parsing Security (HIGH)
|
||||
- [ ] Replace ElementTree with defusedxml
|
||||
- [ ] Add file size limits for uploads
|
||||
- [ ] Implement zip bomb protection
|
||||
- [ ] Add malware scanning hooks (optional)
|
||||
- **Files to Fix**:
|
||||
- `backend/app/services/dmarc_parser.py`
|
||||
## Production Hardening
|
||||
|
||||
#### 4. Input Validation (HIGH)
|
||||
- [ ] Add domain name validation regex
|
||||
- [ ] Implement file type validation (MIME + extension)
|
||||
- [ ] Add parameter validation on all endpoints
|
||||
- [ ] Sanitize error messages
|
||||
- **Files to Fix**:
|
||||
- `backend/app/api/api_v1/endpoints/domains.py`
|
||||
- `backend/app/api/api_v1/endpoints/reports.py`
|
||||
- `backend/app/utils/domain_validator.py`
|
||||
Objective: make self-hosted deployments safer.
|
||||
|
||||
#### 5. Security Headers (MEDIUM)
|
||||
- [ ] Add security headers middleware
|
||||
- [ ] Implement CSP (Content Security Policy)
|
||||
- [ ] Add X-Frame-Options, X-Content-Type-Options
|
||||
- [ ] Configure HSTS for production
|
||||
- **Files to Create/Modify**:
|
||||
- `backend/app/middleware/security.py` (new)
|
||||
- `backend/app/main.py`
|
||||
Priority tasks:
|
||||
- Document a 1Password secret-injection deployment flow.
|
||||
- Keep raw secrets out of diagnostics, logs, and UI responses.
|
||||
- Add startup validation for production settings.
|
||||
- Add backup and restore documentation.
|
||||
- Add a release checklist covering migrations, tests, and smoke checks.
|
||||
|
||||
#### 6. CORS Configuration (MEDIUM)
|
||||
- [ ] Restrict CORS methods and headers
|
||||
- [ ] Remove wildcard configurations
|
||||
- [ ] Document CORS setup for deployments
|
||||
- **Files to Fix**:
|
||||
- `backend/app/main.py` (lines 75-82)
|
||||
## Later Milestones
|
||||
|
||||
#### 7. Error Handling (MEDIUM)
|
||||
- [ ] Implement centralized error handling
|
||||
- [ ] Remove sensitive data from error responses
|
||||
- [ ] Add error logging with request context
|
||||
- [ ] Create user-friendly error messages
|
||||
- **Files to Fix**:
|
||||
- Multiple endpoints across API layer
|
||||
- Notifications and alert rules with Apprise.
|
||||
- DNS health and Cloudflare read-only inspection.
|
||||
- Guided setup and operator health screens.
|
||||
- Forensic/RUF report support.
|
||||
|
||||
### Testing & Validation
|
||||
- [ ] Add security-focused unit tests
|
||||
- [ ] Implement integration tests for auth flow
|
||||
- [ ] Add penetration testing checklist
|
||||
- [ ] Document security testing procedures
|
||||
|
||||
### Documentation
|
||||
- [x] Create SECURITY.md
|
||||
- [ ] Update deployment guides with security best practices
|
||||
- [ ] Create security checklist for contributors
|
||||
- [ ] Add security section to API documentation
|
||||
|
||||
---
|
||||
|
||||
## Milestone 2: IMAP Integration (COMPLETE ✅ - Security Review Needed)
|
||||
|
||||
### Current Features
|
||||
- ✅ IMAP connection and mailbox scanning
|
||||
- ✅ Automated report fetching
|
||||
- ✅ Background task scheduler
|
||||
- ✅ Configuration UI
|
||||
|
||||
### Security Enhancements Needed
|
||||
- [ ] **URGENT**: Remove credentials from URL parameters
|
||||
- [ ] Encrypt IMAP credentials at rest
|
||||
- [ ] Add connection timeout and retry logic
|
||||
- [ ] Implement secure credential storage (vault integration)
|
||||
- [ ] Add audit logging for IMAP operations
|
||||
|
||||
---
|
||||
|
||||
## Milestone 3: Database Integration & Persistence (COMPLETE ✅)
|
||||
|
||||
### Current Features
|
||||
- ✅ SQLAlchemy ORM setup
|
||||
- ✅ SQLite/PostgreSQL support
|
||||
- ✅ Database migrations with Alembic
|
||||
- ✅ Persistent storage
|
||||
|
||||
### Security Enhancements Needed
|
||||
- [ ] Add database encryption at rest
|
||||
- [ ] Implement query audit logging
|
||||
- [ ] Add prepared statement validation
|
||||
- [ ] Review and secure database credentials
|
||||
- [ ] Add database backup encryption
|
||||
|
||||
---
|
||||
|
||||
## Milestone 4: Enhanced Dashboard & Visualization (Next - 4-6 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Historical trend charts (Chart.js integration)
|
||||
- [ ] Compliance rate visualizations
|
||||
- [ ] Volume and sender analytics
|
||||
- [ ] Time-series data displays
|
||||
- [ ] Domain comparison views
|
||||
|
||||
### Security Considerations
|
||||
- [ ] XSS prevention in chart data
|
||||
- [ ] CSP compatibility with Chart.js
|
||||
- [ ] Rate limiting on analytics endpoints
|
||||
- [ ] Data access controls for multi-user scenarios
|
||||
|
||||
### Implementation
|
||||
- **Priority**: Medium
|
||||
- **Dependencies**: Security Sprint completion
|
||||
- **Estimated Effort**: 2-3 weeks
|
||||
|
||||
---
|
||||
|
||||
## Milestone 5: User Authentication & Multi-User Support (8-10 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] FastAPI Users integration
|
||||
- [ ] User registration and management
|
||||
- [ ] JWT-based authentication
|
||||
- [ ] Role-based access control (RBAC)
|
||||
- [ ] Password reset functionality
|
||||
- [ ] Email verification (optional)
|
||||
|
||||
### Security Features
|
||||
- [ ] Strong password policy enforcement
|
||||
- [ ] Multi-factor authentication (MFA)
|
||||
- [ ] Session management
|
||||
- [ ] Account lockout on failed attempts
|
||||
- [ ] Security event logging
|
||||
- [ ] GDPR compliance features
|
||||
|
||||
### Implementation Priority
|
||||
- **Priority**: High
|
||||
- **Security Impact**: Critical
|
||||
- **Dependencies**: Security Sprint, Milestone 4
|
||||
|
||||
---
|
||||
|
||||
## Milestone 6: Alerting & Notifications (10-12 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Apprise integration
|
||||
- [ ] Customizable alert rules
|
||||
- [ ] Multi-channel notifications (Email, Slack, etc.)
|
||||
- [ ] Alert history and management
|
||||
- [ ] Notification preferences per user
|
||||
|
||||
### Security Features
|
||||
- [ ] Secure webhook handling
|
||||
- [ ] Alert rate limiting
|
||||
- [ ] PII filtering in notifications
|
||||
- [ ] Encrypted notification credentials
|
||||
- [ ] Audit trail for alert configuration
|
||||
|
||||
---
|
||||
|
||||
## Milestone 7: Advanced Rule Engine (14-16 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Custom alert conditions
|
||||
- [ ] Threshold-based triggers
|
||||
- [ ] New sender detection
|
||||
- [ ] Anomaly detection
|
||||
- [ ] Scheduled report summaries
|
||||
|
||||
### Security Features
|
||||
- [ ] Rule validation and sandboxing
|
||||
- [ ] Resource limits on rule execution
|
||||
- [ ] Audit logging for rule changes
|
||||
- [ ] Protection against rule abuse
|
||||
|
||||
---
|
||||
|
||||
## Milestone 8: DNS Health & Cloudflare Integration (16-18 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] DNS record health checks
|
||||
- [ ] SPF/DKIM/DMARC validation
|
||||
- [ ] Cloudflare API integration
|
||||
- [ ] Configuration recommendations
|
||||
- [ ] DNS change tracking
|
||||
|
||||
### Security Features
|
||||
- [ ] Secure API credential storage
|
||||
- [ ] DNS query rate limiting
|
||||
- [ ] DNSSEC validation
|
||||
- [ ] Audit logging for DNS operations
|
||||
- [ ] Read-only DNS access (no auto-changes initially)
|
||||
|
||||
---
|
||||
|
||||
## Milestone 9: Forensic Reports (RUF) Support (20-22 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Forensic report parsing
|
||||
- [ ] Failure sample analysis
|
||||
- [ ] PII redaction options
|
||||
- [ ] Detailed authentication failure views
|
||||
- [ ] Sample download/export
|
||||
|
||||
### Security Features
|
||||
- [ ] PII detection and redaction
|
||||
- [ ] Access controls for sensitive data
|
||||
- [ ] Audit logging for forensic data access
|
||||
- [ ] Compliance with privacy regulations
|
||||
- [ ] Secure export with encryption
|
||||
|
||||
---
|
||||
|
||||
## Milestone 10: Advanced Analytics & Reporting (24-26 weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Historical trend analysis
|
||||
- [ ] Comparative reporting
|
||||
- [ ] Export capabilities (PDF, CSV)
|
||||
- [ ] Scheduled reports
|
||||
- [ ] Custom dashboards
|
||||
|
||||
### Security Features
|
||||
- [ ] Export sanitization
|
||||
- [ ] Watermarking for exported reports
|
||||
- [ ] Access logging for exports
|
||||
- [ ] Encrypted export files
|
||||
|
||||
---
|
||||
|
||||
## Milestone 11: Enterprise Features (28-30+ weeks)
|
||||
|
||||
### Planned Features
|
||||
- [ ] Multi-tenant architecture
|
||||
- [ ] API rate limiting
|
||||
- [ ] Advanced RBAC
|
||||
- [ ] SSO integration (SAML, OAuth)
|
||||
- [ ] Compliance reporting (SOC 2, GDPR)
|
||||
- [ ] High availability setup
|
||||
- [ ] Backup and disaster recovery
|
||||
|
||||
### Security Features
|
||||
- [ ] Tenant isolation
|
||||
- [ ] Advanced audit logging
|
||||
- [ ] Security event monitoring
|
||||
- [ ] Compliance automation
|
||||
- [ ] Regular security assessments
|
||||
|
||||
---
|
||||
|
||||
## Continuous Improvements (Ongoing)
|
||||
|
||||
### Code Quality
|
||||
- [ ] Maintain >80% test coverage
|
||||
- [ ] Regular dependency updates
|
||||
- [ ] Code review for all changes
|
||||
- [ ] Performance optimization
|
||||
- [ ] Technical debt reduction
|
||||
|
||||
### Security
|
||||
- [ ] Monthly security audits
|
||||
- [ ] Automated vulnerability scanning (GitHub Actions)
|
||||
- [ ] Dependency security monitoring
|
||||
- [ ] Regular penetration testing
|
||||
- [ ] Security training for contributors
|
||||
|
||||
### Documentation
|
||||
- [ ] Keep documentation current
|
||||
- [ ] API documentation completeness
|
||||
- [ ] Security best practices guide
|
||||
- [ ] Deployment playbooks
|
||||
- [ ] Troubleshooting guides
|
||||
|
||||
### Community
|
||||
- [ ] Issue triage and response
|
||||
- [ ] PR review and merging
|
||||
- [ ] Community engagement
|
||||
- [ ] Feature request evaluation
|
||||
- [ ] Bug fix prioritization
|
||||
|
||||
---
|
||||
|
||||
## Security Milestones Integration
|
||||
|
||||
Each development milestone now includes security considerations:
|
||||
|
||||
| Milestone | Security Priority | Key Security Features |
|
||||
|-----------|------------------|----------------------|
|
||||
| Security Sprint | 🔴 Critical | Fix all critical vulnerabilities |
|
||||
| Milestone 4 | 🟡 Medium | XSS prevention, CSP |
|
||||
| Milestone 5 | 🔴 Critical | Authentication, RBAC, MFA |
|
||||
| Milestone 6 | 🟠 High | Secure webhooks, PII filtering |
|
||||
| Milestone 7 | 🟠 High | Rule sandboxing, audit trails |
|
||||
| Milestone 8 | 🟠 High | API security, DNSSEC |
|
||||
| Milestone 9 | 🔴 Critical | PII redaction, compliance |
|
||||
| Milestone 10 | 🟡 Medium | Export security, watermarking |
|
||||
| Milestone 11 | 🔴 Critical | Enterprise security, SOC 2 |
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Functional
|
||||
- All planned features implemented
|
||||
- Performance meets requirements
|
||||
- User experience is intuitive
|
||||
- Documentation is complete
|
||||
|
||||
### Security
|
||||
- Zero critical vulnerabilities
|
||||
- All high-severity issues resolved
|
||||
- Security tests pass
|
||||
- Regular security audits pass
|
||||
- Compliance requirements met
|
||||
|
||||
### Quality
|
||||
- >80% code coverage
|
||||
- All tests passing
|
||||
- No critical bugs
|
||||
- Performance benchmarks met
|
||||
- Code review approval
|
||||
|
||||
---
|
||||
|
||||
## Risk Management
|
||||
|
||||
### Technical Risks
|
||||
- **Risk**: Complex security implementations
|
||||
- **Mitigation**: Incremental approach, expert review
|
||||
- **Risk**: Performance degradation with security features
|
||||
- **Mitigation**: Performance testing, optimization
|
||||
|
||||
### Resource Risks
|
||||
- **Risk**: Limited security expertise
|
||||
- **Mitigation**: External security audits, community review
|
||||
- **Risk**: Time constraints for security work
|
||||
- **Mitigation**: Prioritize critical issues first
|
||||
|
||||
### Operational Risks
|
||||
- **Risk**: Breaking changes with security fixes
|
||||
- **Mitigation**: Thorough testing, clear documentation
|
||||
- **Risk**: User adoption of security features
|
||||
- **Mitigation**: Clear communication, good UX
|
||||
|
||||
---
|
||||
|
||||
## Contributing to This Roadmap
|
||||
|
||||
This roadmap is a living document. To contribute:
|
||||
|
||||
1. Review current milestones and status
|
||||
2. Propose changes via GitHub Issues
|
||||
3. Discuss in community forums
|
||||
4. Submit PRs for roadmap updates
|
||||
5. Participate in planning discussions
|
||||
|
||||
See [CONTRIBUTING.md](../../CONTRIBUTING.md) for detailed guidelines.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- [SECURITY.md](../../SECURITY.md) - Security policy and vulnerability reporting
|
||||
- [CONTRIBUTING.md](../../CONTRIBUTING.md) - Contribution guidelines
|
||||
- [Agentic Coding Guidelines](agents.md) - AI-assisted development guidelines
|
||||
- [Milestones](../milestones.md) - Detailed milestone specifications
|
||||
- [Todo](../todo.md) - Detailed task tracking
|
||||
|
||||
---
|
||||
|
||||
**Maintained by**: DMARQ Development Team
|
||||
**Contact**: See [SECURITY.md](../../SECURITY.md) for contact information
|
||||
See [milestones.md](../milestones.md) for the full milestone breakdown and exit criteria.
|
||||
|
||||
Reference in New Issue
Block a user