Commit Graph

44 Commits

Author SHA1 Message Date
Christian Krakau-Louis 4fc69602b3 feat: add mail source import history 2026-05-22 19:29:49 +02:00
Christian Krakau-Louis 181b43bff9 address copilot review followups (#96) 2026-05-18 19:17:36 +02:00
Christian Krakau-Louis b4191e956c address backend security and test suggestions 2026-05-18 16:44:06 +02:00
copilot-swe-agent[bot] 523b0529f4 fix: use Logto Account Center routes for password change and MFA management
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/d566f8d7-4560-4945-b9e5-388259f626a3

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-04-02 17:05:20 +00:00
Christian Krakau-Louis f8d4f936a7 Merge pull request #90 from christianlouis/copilot/fix-id-token-claims-extraction
fix: remove erroneous await from synchronous getIdTokenClaims() + add callback endpoint tests
2026-03-31 15:59:36 +02:00
copilot-swe-agent[bot] c81173e417 implement password reset on login screen and MFA management for users
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/0419399d-3a03-4f02-a3a8-fc75da7172bc

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-30 19:58:40 +00:00
copilot-swe-agent[bot] 9f1494a090 fix: remove erroneous await from getIdTokenClaims() call
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/2a043a05-25c6-4750-a9e8-da2b1651b55b

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-30 19:51:43 +00:00
copilot-swe-agent[bot] 04931172dd feat: add AUTH_DISABLED no-auth fallback mode
- config.py: AUTH_DISABLED: bool = False setting
- middleware/auth.py: bypass all checks when AUTH_DISABLED=True
- security.py: require_admin_auth returns synthetic context when disabled
- endpoints/auth.py: /me returns synthetic admin; /sign-out → / when disabled
- main.py: startup WARNING when disabled; pass auth_disabled to login.html
- templates/login.html: info banner with Go to dashboard link when disabled
- templates/setup.html: document AUTH_DISABLED option with security warning
- tests/test_auth.py: 4 new AUTH_DISABLED tests (445 total, all pass)

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/18f41bf2-0b68-4b7d-afb5-d2894c212a8f

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-30 11:38:52 +00:00
copilot-swe-agent[bot] 531dc968a8 feat: integrate Logto OIDC for user authentication
- Add Logto OIDC integration (app/core/logto.py): CookieStorage adapter,
  create/decode session token helpers, sync_logto_user upsert
- New auth endpoints (/api/v1/auth): sign-in, callback, sign-out, me
- AuthRedirectMiddleware: protects HTML pages, redirects to /setup when
  Logto is unconfigured, to /login otherwise
- Update require_admin_auth: accepts dmarq_session cookie JWT first,
  then API key, then Bearer JWT (fully backward compatible)
- Update User model: add logto_id, username, picture, created_at, updated_at;
  make hashed_password nullable for Logto-only users; is_superuser default=True
- New Alembic migration d4e5f6a7b8c9 for the above schema changes
- Add LOGTO_ENDPOINT / LOGTO_APP_ID / LOGTO_APP_SECRET / LOGTO_REDIRECT_URI
  settings with logto_configured property
- Create login.html (Sign in with Logto button) and setup.html (step-by-step
  configuration guide)
- Update base.html: user menu with avatar/name and sign-out via Alpine.js
  fetch to /api/v1/auth/me
- Update settings.html: remove localStorage adminApiKey; session cookie is
  sent automatically by browser; add 401 → /login redirect
- Update requirements.txt: replace fastapi-users additions with logto + aiohttp
- Add test_auth.py: 18 new tests covering session tokens, CookieStorage,
  sync_logto_user, /me, /sign-in (503), /sign-out cookie clearing
- Fix test_security_extra.py: pass Request mock to require_admin_auth;
  add new test_valid_session_cookie_returns_auth_context

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/b448f585-7646-40f8-ae2d-9986c361e3fd

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-30 10:09:50 +00:00
copilot-swe-agent[bot] 241713083b Add persistent settings system with database backend and comprehensive UI
- New Setting ORM model (key-value store with category, value_type, audit fields)
- Alembic migration to create the settings table
- Settings API endpoints: GET/PUT /api/v1/settings/{key}, GET /api/v1/settings (list+filter), POST /api/v1/settings/bulk
- Default seeding (17 sensible defaults across general/dmarc/dns/cloudflare/notifications categories)
- Secret redaction for cloudflare.api_token and notifications.smtp_password
- Updated settings.html: General, DMARC Policy Defaults, DNS Resolver, Cloudflare Integration, Email Notifications sections
- All forms wired to the API via Alpine.js with flash feedback
- 12 new tests for the settings model and endpoints

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/19dbc6cd-07cb-406e-b3b6-411f7721f737

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-30 07:58:51 +00:00
Christian Krakau-Louis 06a007d9db Merge pull request #76 from christianlouis/copilot/update-dkim-selectors
Fix DKIM selector extraction: surface all working selectors and auto-discovered report selectors
2026-03-30 01:57:12 +02:00
copilot-swe-agent[bot] 596a9b882e Fix DKIM selector extraction: show all working selectors and report-discovered selectors
- check_dkim now returns ALL matching selectors instead of stopping at first match
- DomainDNSResult.dkim_selectors is now a List[str] instead of a single Optional[str]
- DNSRecordResponse.dkimSelectors is now List[str]
- /selectors endpoint now also returns report_selectors (auto-discovered from DMARC reports)
- Frontend shows all live-check selectors and auto-discovered selectors as read-only
- Updated tests to match new data structures; added tests for multi-selector and report_selectors

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/87d8b8d9-23c3-4d3b-85a3-8e354e62c768

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 23:54:45 +00:00
copilot-swe-agent[bot] de225bc1cd Replace dummy/bogus reports data with real API data on /reports page
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/c6e36fd9-00c9-4000-985f-f7a42e2ba451

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 23:24:40 +00:00
copilot-swe-agent[bot] 58b5755f72 refactor: extract _spf_fix_hint and _safe_ptr_lookup helpers; fix spelling
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/9eaa7749-047c-46bd-8bc0-2851ea02ffe4

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 22:46:49 +00:00
copilot-swe-agent[bot] 30007d21d6 feat: add rDNS hostname, SPF fix hints and auth tooltips to domain sources view
- dns_resolver.py: add _ip_to_arpa_name() helper and lookup_ptr() to
  BaseDNSProvider (no-op default), SystemDNSProvider (dnspython PTR),
  and CloudflareDNSProvider (DoH PTR type=12)
- domains.py: extend SourceEntry with hostname + spf_fix_hint; update
  get_domain_sources to run async PTR lookups and generate ip4:/ip6:
  SPF mechanism hints for failing IPs
- domain_details.html: show rDNS hostname below IP in sources table;
  add DaisyUI tooltip explaining each auth result; add "Fix SPF" popover
  with copy-paste mechanism for IPs that fail SPF
- tests: 15 new tests covering _ip_to_arpa_name, SystemDNSProvider/
  CloudflareDNSProvider PTR lookup, and sources endpoint hostname +
  fix-hint fields

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/9eaa7749-047c-46bd-8bc0-2851ea02ffe4

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 22:44:18 +00:00
Christian Krakau-Louis 1815747fba Merge pull request #69 from christianlouis/copilot/debug-not-found-error
Fix /reports/{report_id} returning 404 Not Found
2026-03-30 00:27:52 +02:00
Christian Krakau-Louis 506346537c Merge pull request #67 from christianlouis/copilot/add-gmail-api-mail-source
feat: add Gmail API & main.py helper tests; fix flake8 lint errors
2026-03-30 00:26:28 +02:00
copilot-swe-agent[bot] af122881a5 Fix /reports/{report_id} returning 404 - add report detail page and API endpoint
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/4a412936-3bf7-4c02-b121-c0c342c34f1b

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 22:18:17 +00:00
copilot-swe-agent[bot] 2d364509bd fix: resolve remaining CodeQL alerts and raise mail_sources coverage to 100%
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/adfb364c-2feb-44fa-b9dd-73b9bdb7fa8f

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 21:39:00 +00:00
copilot-swe-agent[bot] dbbf539182 fix: resolve CodeQL security alerts – log injection, info exposure, incomplete URL checks
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/093a5de2-644b-434d-8ce8-529710ecb40b

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 21:14:02 +00:00
copilot-swe-agent[bot] 11daa15335 Fix domain reports and sources API 500 errors
- Use begin_timestamp/end_timestamp (Unix ints) instead of begin_date/end_date
  (ISO strings) when building ReportEntry, fixing Pydantic int_parsing errors
- Extract policy string from dict (policy["p"]) when the stored value is a dict,
  fixing Pydantic string_type validation error
- Rename _days -> days in ReportStore.get_domain_sources() so the endpoint's
  `days=days` keyword call no longer raises TypeError

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/15ac3521-e81f-42fa-81a2-f6804a423e1d

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 20:03:43 +00:00
copilot-swe-agent[bot] 25b3567414 feat: implement Gmail API mail source with OAuth2, ingestion tracking, and UI
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/0c63851e-a69a-4a76-8dd8-25618e825b8c

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 19:34:03 +00:00
Christian Krakau-Louis 298ca938ec Merge pull request #62 from christianlouis/copilot/check-dns-data-in-dashboard
fix: repair corrupted dns_resolver.py to pass black --check (py310 target)
2026-03-29 21:17:25 +02:00
copilot-swe-agent[bot] a04006f75b fix: address code review — narrow exception types, computed dkimLiveText getter
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/19d17518-732d-4644-889b-cc63256e19b1

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 18:56:08 +00:00
copilot-swe-agent[bot] 0d1a4fdac3 feat: real DNS lookups, manual DKIM selectors, Cloudflare-ready DNS provider architecture
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/19d17518-732d-4644-889b-cc63256e19b1

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 18:53:35 +00:00
copilot-swe-agent[bot] 53d543f49c Add report deduplication and single-report deletion
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/37d68c2c-7cc8-45e4-bac3-e2e6f1611c6a

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 18:28:52 +00:00
copilot-swe-agent[bot] 84117abd13 fix: resolve CodeQL alerts, improve test coverage, fix StaticPool for HTTP tests
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/07b58025-2aeb-4e81-a168-7a6fccdc3569

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 17:08:39 +00:00
copilot-swe-agent[bot] 6d9392e5e7 feat: refactor IMAP config to DB, add multi-account mail sources admin UI
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/f141cd3b-8d0a-4c7e-b479-ee973c209c11

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 16:18:15 +00:00
copilot-swe-agent[bot] 6329d3aa35 Use timezone-aware UTC datetimes for timestamp conversions
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/ea784ce3-2c1c-45f8-ad6d-46e92cf15ed7

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 15:45:26 +00:00
copilot-swe-agent[bot] e51fbe755e Replace mock dashboard data with real database queries and report-derived timelines
- Replace mock statistics in stats_summarizer.py with real SQLAlchemy queries
  against DMARCReport, ReportRecord, and Domain models
- Replace random compliance timeline in domains.py with real data from ReportStore
- Remove unused `import random` from domains.py
- Add comprehensive tests for StatsSummarizer (global/domain/caching)
- Add tests for compliance timeline (deterministic, multi-report aggregation)
- Mark Dashboard Visualizations as complete in TODO.md

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/ea784ce3-2c1c-45f8-ad6d-46e92cf15ed7

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 15:43:33 +00:00
copilot-swe-agent[bot] 01d0331136 Resolve linter contradictions: consolidate config, fix isort first-party, pylint 10/10
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/1e4a1f06-55b9-4040-853e-6aaf9ee574c8

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 11:31:15 +00:00
copilot-swe-agent[bot] 50aa5bd5da Fix pylint warnings: logging, globals, exceptions, imports, duplicates
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/576427d4-4f6a-46d2-b75f-6862ecbcf526

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 11:21:35 +00:00
copilot-swe-agent[bot] 821d9dc863 fix: apply black formatting to reports.py and dmarc_parser.py
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/b108ee4f-3479-43fa-a9e9-23a697718aa3

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 11:05:07 +00:00
copilot-swe-agent[bot] 271b29186a Fix flake8 C901 complexity violations and F824 unused global in 5 files
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/e58d59de-a79d-48e1-a4fc-ffb61b868593

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 10:58:33 +00:00
copilot-swe-agent[bot] bab702a010 Add Docker build & GHCR publish stage, fix CI workflows, update AGENTS.md and testing docs
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/9d256101-34b6-4861-a8cf-7f86f32b54d5

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-29 10:23:49 +00:00
google-labs-jules[bot] 1831c1a079 🔒 security: move IMAP credentials to request body and fix CI/pylint issues
- Moved IMAP credentials from query parameters to the request body for the test-connection endpoint.
- Created missing `__init__.py` files across `backend/app/` to fix CI `ModuleNotFoundError`.
- Reformatted `setup.py` and `user.py` using `black` to pass lint checks.
- Addressed Pylint warnings in `imap.py` (unused arguments, lazy logging, exception chaining).

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-23 18:57:40 +00:00
google-labs-jules[bot] 34c910e131 🔒 security: move IMAP credentials to request body
Moved IMAP credentials from query parameters to the request body
for the test-connection endpoint to prevent cleartext logging.
Implemented a Pydantic model (IMAPTestRequest) for request body validation.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-23 16:18:16 +00:00
copilot-swe-agent[bot] 0e7a01bc5d Address code review feedback
- Fix background_task scope issue (use global consistently)
- Improve IMAP exception handling comment with specific examples
- Move random import to module level with TODO comment
- Add nosec B311 comment for mock data random usage

All code review issues resolved.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 12:16:50 +00:00
copilot-swe-agent[bot] 6ae017b142 Fix code formatting and linting issues
- Auto-format all Python files with black and isort
- Remove unused imports with autoflake
- Fix flake8 issues (missing newlines, blank lines, etc.)
- Fix nonlocal/global scope issues in main.py
- Fix security.py import order (E402)
- Remove f-string without placeholders
- Add nosec comment for intentional exception handling
- Fix test imports to match refactored DMARCParser API

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 12:08:51 +00:00
copilot-swe-agent[bot] 8550799bf8 Address code review feedback: improve XSS prevention, structured errors, API key logging, CSP warnings
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 11:51:58 +00:00
copilot-swe-agent[bot] 03f4eaf724 Implement critical security fixes: secret management, XML parsing, auth, input validation, security headers
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 11:43:22 +00:00
Christian Krakau-Louis 5e8b1f033f Add comprehensive documentation for DMARQ, including user guides, deployment instructions, and feature descriptions
- Created main documentation index and user guide with sections on getting started, dashboard overview, managing domains, and reports.
- Added detailed deployment guide for Docker and manual installation.
- Included user-friendly explanations of DMARC, its benefits, and how to manage domains and reports.
- Implemented visual assets for dashboard, domains, IMAP, and reports.
- Established requirements for documentation build using MkDocs and Material theme.
- Integrated navigation structure for easy access to all documentation sections.
2025-04-21 01:49:34 +02:00
Christian Krakau-Louis 69f8438a36 Add domain management, reports, settings, and upload templates
- Implemented domain management page with a list of monitored domains and their DMARC, SPF, and DKIM statuses.
- Created reports page with filtering options for domain, report type, and date range, displaying DMARC reports.
- Developed settings page for IMAP configuration and DMARC policy management, including form validation and feedback.
- Added upload page for DMARC report files with drag-and-drop functionality and file type validation.
- Integrated Alpine.js for interactivity and dynamic data handling across all templates.
2025-04-20 18:38:29 +02:00
Christian Krakau-Louis f910cb0ba4 Add initial MVP documentation for DMARQ platform, detailing backend architecture, frontend implementation, and deployment structure 2025-04-17 15:20:42 +02:00