feat: persist imported DMARC reports

docs/development/roadmap.md

@@ -24,24 +24,13 @@ Recently improved:
 - Gmail and IMAP imports now skip duplicate domain/report IDs.
 - Tests cover Google-style DMARC ZIP attachment imports.
 - Mail source imports now create sanitized import-history records for manual and scheduled polls.
+- Parsed upload, Gmail, and IMAP reports are now persisted to `dmarc_reports` and `report_records`.
+- Report/domain API reads can hydrate the dashboard projection from persisted data after restart.
 
-Important gap:
-- Parsed DMARC report data is still served primarily from the in-memory `ReportStore`. The database schema exists, but report upload/import paths and dashboard read paths must be completed before the persistence milestone can be called done.
+Implementation note:
+- The legacy `ReportStore` remains as a projection layer for existing report/dashboard code, but durable report data now lives in the database.
 
-## Active Milestone: Finish Report Persistence
-
-Objective: complete the database-backed report storage promised by Milestone 3.
-
-Priority tasks:
-- Add a report persistence service that converts parsed DMARC report dicts into `Domain`, `DMARCReport`, and `ReportRecord` rows.
-- Load or query persisted reports for dashboard, domain, and report endpoints.
-- Add duplicate report detection against the database.
-- Keep tests covering upload, Gmail import, IMAP import, and restart-style reload behavior.
-
-Quality bar:
-- Uploading or importing a report survives application restart and remains visible in report/domain endpoints.
-
-## Next Milestone: Reporting Quality and Import Confidence
+## Active Milestone: Reporting Quality and Import Confidence
 
 Objective: make mailbox imports auditable and make report totals trustworthy.
 
@@ -57,7 +46,7 @@ Quality bar:
 - Parse failures must be visible and actionable.
 - The user should be able to tell whether a mail source is healthy without reading logs.
 
-## Following Milestone: Meaningful Reports
+## Next Milestone: Meaningful Reports
 
 Objective: turn parsed DMARC data into administrator-friendly reports.
 

docs/milestones.md

@@ -29,24 +29,25 @@ Delivered:
 - Duplicate report protection for both Gmail and IMAP imports.
 - Background polling and manual poll hooks for configured mail sources.
 
-## Milestone 3: Database Foundation, Domain Management, and Auth Foundation - In Progress
+## Milestone 3: Database Foundation, Domain Management, and Auth Foundation - Complete
 
-Status: In progress
+Status: Complete
 
 Delivered:
 - SQLAlchemy models and Alembic migrations.
 - SQLite/PostgreSQL-compatible database configuration.
 - Domain management APIs and UI.
 - Report and source database models.
+- Database-backed persistence for uploaded DMARC reports.
+- Database-backed persistence for Gmail and IMAP imported reports.
+- Duplicate report detection against persisted report data.
+- Report/domain API reads can hydrate their dashboard projection from persisted reports after restart.
 - Settings and mail source persistence.
 - Logto-based auth integration plus an explicit local development auth-disabled mode.
 - Security middleware, safer default secret generation, and security-focused tests.
 
-Remaining before this milestone is complete:
-- Persist parsed DMARC reports and report records through the database-backed models.
-- Load or query persisted reports after restart so dashboards do not depend on process memory.
-- Move report/domain summary endpoints from the in-memory `ReportStore` to database queries.
-- Keep duplicate report detection consistent across upload, IMAP, and Gmail after persistence is enabled.
+Implementation note:
+- The existing `ReportStore` remains as a compatibility projection for dashboard/report code, but persisted database rows are now the durable source for uploads and mailbox imports.
 
 ## Milestone 4: Reporting Quality and Import Confidence - In Progress
 
@@ -59,9 +60,9 @@ Recently delivered:
 - Gmail/IMAP imports skip duplicate report IDs to avoid inflated totals.
 - Tests now cover a real Google-style ZIP attachment path rather than only mocked parser behavior.
 - Mail source imports now persist sanitized import-history records for manual and scheduled polls.
+- Uploaded, Gmail-imported, and IMAP-imported reports are now persisted to report/record tables and can be reloaded into report/domain views.
 
 Next tasks:
-- Finish Milestone 3 report persistence before expanding report features.
 - Add per-import result details: skipped duplicates, parse failures, unsupported attachments, and imported report IDs.
 - Add a UI import history view for each mail source.
 - Add mailbox search controls for date range/backfill without requiring code changes.

docs/todo.md

@@ -101,14 +101,14 @@ This file tracks the specific implementation tasks for each milestone of the DMA
 - [x] Implement data access layer
 
 ### Model Migration
-- [ ] Convert report ingestion and dashboard reads from in-memory storage to database-backed storage
+- [x] Convert report ingestion and dashboard reads from in-memory storage to database-backed storage
 - [x] Create Domain table
 - [x] Create AggregateReport table
 - [x] Create ReportRecord table for sender details
 - [x] Implement relationships between models
-- [ ] Persist parsed upload reports to `dmarc_reports` and `report_records`
-- [ ] Persist parsed Gmail/IMAP reports to `dmarc_reports` and `report_records`
-- [ ] Load/query persisted reports after app restart
+- [x] Persist parsed upload reports to `dmarc_reports` and `report_records`
+- [x] Persist parsed Gmail/IMAP reports to `dmarc_reports` and `report_records`
+- [x] Load/query persisted reports after app restart
 
 ### Domain Management
 - [x] Create UI for adding/editing domains