feat: add mail source import history

docs/development/roadmap.md

@@ -12,7 +12,7 @@ Complete:
 - Upload validation and archive safety checks.
 - IMAP mailbox ingestion.
 - Gmail OAuth ingestion.
-- Persistent database models and migrations.
+- Persistent database models and migrations for domains, reports, records, settings, users, and mail sources.
 - Domain, report, settings, and mail source APIs.
 - Dashboard and domain detail views.
 - Logto auth integration and local development auth-disabled mode.
@@ -23,13 +23,30 @@ Recently improved:
 - Gmail import now uses the same parser path as uploads and IMAP.
 - Gmail and IMAP imports now skip duplicate domain/report IDs.
 - Tests cover Google-style DMARC ZIP attachment imports.
+- Mail source imports now create sanitized import-history records for manual and scheduled polls.
 
-## Active Milestone: Reporting Quality and Import Confidence
+Important gap:
+- Parsed DMARC report data is still served primarily from the in-memory `ReportStore`. The database schema exists, but report upload/import paths and dashboard read paths must be completed before the persistence milestone can be called done.
+
+## Active Milestone: Finish Report Persistence
+
+Objective: complete the database-backed report storage promised by Milestone 3.
+
+Priority tasks:
+- Add a report persistence service that converts parsed DMARC report dicts into `Domain`, `DMARCReport`, and `ReportRecord` rows.
+- Load or query persisted reports for dashboard, domain, and report endpoints.
+- Add duplicate report detection against the database.
+- Keep tests covering upload, Gmail import, IMAP import, and restart-style reload behavior.
+
+Quality bar:
+- Uploading or importing a report survives application restart and remains visible in report/domain endpoints.
+
+## Next Milestone: Reporting Quality and Import Confidence
 
 Objective: make mailbox imports auditable and make report totals trustworthy.
 
 Priority tasks:
-- Persist import attempts with message ID, source, attachment filename, outcome, and sanitized error details.
+- Expand import attempts with message ID, source, attachment filename, outcome, and sanitized error details.
 - Show import history on mail source detail pages.
 - Report duplicate skips separately from parse failures.
 - Add backfill controls for Gmail and IMAP sources.
@@ -40,7 +57,7 @@ Quality bar:
 - Parse failures must be visible and actionable.
 - The user should be able to tell whether a mail source is healthy without reading logs.
 
-## Next Milestone: Meaningful Reports
+## Following Milestone: Meaningful Reports
 
 Objective: turn parsed DMARC data into administrator-friendly reports.
 

docs/milestones.md

@@ -29,19 +29,25 @@ Delivered:
 - Duplicate report protection for both Gmail and IMAP imports.
 - Background polling and manual poll hooks for configured mail sources.
 
-## Milestone 3: Persistence, Domain Management, and Auth Foundation - Complete
+## Milestone 3: Database Foundation, Domain Management, and Auth Foundation - In Progress
 
-Status: Complete
+Status: In progress
 
 Delivered:
 - SQLAlchemy models and Alembic migrations.
 - SQLite/PostgreSQL-compatible database configuration.
 - Domain management APIs and UI.
-- Report and source persistence.
+- Report and source database models.
 - Settings and mail source persistence.
 - Logto-based auth integration plus an explicit local development auth-disabled mode.
 - Security middleware, safer default secret generation, and security-focused tests.
 
+Remaining before this milestone is complete:
+- Persist parsed DMARC reports and report records through the database-backed models.
+- Load or query persisted reports after restart so dashboards do not depend on process memory.
+- Move report/domain summary endpoints from the in-memory `ReportStore` to database queries.
+- Keep duplicate report detection consistent across upload, IMAP, and Gmail after persistence is enabled.
+
 ## Milestone 4: Reporting Quality and Import Confidence - In Progress
 
 Status: In progress
@@ -52,10 +58,11 @@ Recently delivered:
 - Gmail import now handles real inbox metadata patterns and Google-style ZIP filenames.
 - Gmail/IMAP imports skip duplicate report IDs to avoid inflated totals.
 - Tests now cover a real Google-style ZIP attachment path rather than only mocked parser behavior.
+- Mail source imports now persist sanitized import-history records for manual and scheduled polls.
 
 Next tasks:
+- Finish Milestone 3 report persistence before expanding report features.
 - Add per-import result details: skipped duplicates, parse failures, unsupported attachments, and imported report IDs.
-- Store failed import attempts with enough metadata to retry or diagnose them without exposing secrets.
 - Add a UI import history view for each mail source.
 - Add mailbox search controls for date range/backfill without requiring code changes.
 - Improve source aggregation so each sender IP keeps pass/fail totals instead of only the latest result.

docs/todo.md

@@ -88,8 +88,8 @@ This file tracks the specific implementation tasks for each milestone of the DMA
 ### Duplicate and Error Handling
 - [x] Skip duplicate domain/report IDs during Gmail imports
 - [x] Skip duplicate domain/report IDs during IMAP imports
-- [ ] Persist sanitized import errors for UI review
-- [ ] Count duplicate skips separately from parse failures
+- [x] Persist sanitized import errors for API/UI review
+- [x] Count duplicate skips separately from parse failures
 - [ ] Add retry/backfill controls per mail source
 
 ## Milestone 3: Database Integration
@@ -101,11 +101,14 @@ This file tracks the specific implementation tasks for each milestone of the DMA
 - [x] Implement data access layer
 
 ### Model Migration
-- [x] Convert in-memory models to database models
+- [ ] Convert report ingestion and dashboard reads from in-memory storage to database-backed storage
 - [x] Create Domain table
 - [x] Create AggregateReport table
 - [x] Create ReportRecord table for sender details
 - [x] Implement relationships between models
+- [ ] Persist parsed upload reports to `dmarc_reports` and `report_records`
+- [ ] Persist parsed Gmail/IMAP reports to `dmarc_reports` and `report_records`
+- [ ] Load/query persisted reports after app restart
 
 ### Domain Management
 - [x] Create UI for adding/editing domains