gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
283 Commits 2 Branches 0 Tags
4fc69602b328973cfebd090674ffef8a1eabb083
Commit Graph

62 Commits

Author SHA1 Message Date
Christian Krakau-Louis 4fc69602b3 feat: add mail source import history 2026-05-22 19:29:49 +02:00
Christian Krakau-Louis 07b2ea400e fix: harden DMARC mail import 2026-05-22 19:17:32 +02:00
Christian Krakau-Louis 181b43bff9 address copilot review followups (#96) 2026-05-18 19:17:36 +02:00
Christian Krakau-Louis b4191e956c address backend security and test suggestions 2026-05-18 16:44:06 +02:00
copilot-swe-agent[bot] 32351a2134 test: add TestCallbackEndpoint tests to cover /callback auth endpoint 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/0683a057-d74b-4b1a-aeab-47f5bfbe4190

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-31 08:57:05 +00:00
copilot-swe-agent[bot] 3a5b75e964 Fix Logto callback SSL error: extend LOGTO_SKIP_SSL_VERIFY patch to PyJWKClient (JWKS/urllib) 
The 'Fail to fetch data from the url' callback error came from PyJWT's
PyJWKClient.fetch_data() using urllib to retrieve the JWKS, which is not
covered by the existing aiohttp.ClientSession SSL monkey-patch.

Extend _apply_logto_ssl_patch() to also replace PyJWKClient inside
logto.OidcCore with a subclass that injects the non-verifying ssl.SSLContext
via the ssl_context constructor parameter, ensuring both the OIDC discovery/
token requests (aiohttp) and ID-token JWKS verification (urllib) honour
LOGTO_SKIP_SSL_VERIFY=True.

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/14676b1a-3421-4839-9ba3-8229a3e5adf1

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 17:15:58 +00:00
copilot-swe-agent[bot] 3fb90dbcf6 Exclude static asset extensions from auth redirect middleware 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/3e1fe3d2-54b1-4eb4-90d2-173a5e95d376

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 16:36:28 +00:00
copilot-swe-agent[bot] 04931172dd feat: add AUTH_DISABLED no-auth fallback mode 
- config.py: AUTH_DISABLED: bool = False setting
- middleware/auth.py: bypass all checks when AUTH_DISABLED=True
- security.py: require_admin_auth returns synthetic context when disabled
- endpoints/auth.py: /me returns synthetic admin; /sign-out → / when disabled
- main.py: startup WARNING when disabled; pass auth_disabled to login.html
- templates/login.html: info banner with Go to dashboard link when disabled
- templates/setup.html: document AUTH_DISABLED option with security warning
- tests/test_auth.py: 4 new AUTH_DISABLED tests (445 total, all pass)

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/18f41bf2-0b68-4b7d-afb5-d2894c212a8f

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 11:38:52 +00:00
copilot-swe-agent[bot] 531dc968a8 feat: integrate Logto OIDC for user authentication 
- Add Logto OIDC integration (app/core/logto.py): CookieStorage adapter,
  create/decode session token helpers, sync_logto_user upsert
- New auth endpoints (/api/v1/auth): sign-in, callback, sign-out, me
- AuthRedirectMiddleware: protects HTML pages, redirects to /setup when
  Logto is unconfigured, to /login otherwise
- Update require_admin_auth: accepts dmarq_session cookie JWT first,
  then API key, then Bearer JWT (fully backward compatible)
- Update User model: add logto_id, username, picture, created_at, updated_at;
  make hashed_password nullable for Logto-only users; is_superuser default=True
- New Alembic migration d4e5f6a7b8c9 for the above schema changes
- Add LOGTO_ENDPOINT / LOGTO_APP_ID / LOGTO_APP_SECRET / LOGTO_REDIRECT_URI
  settings with logto_configured property
- Create login.html (Sign in with Logto button) and setup.html (step-by-step
  configuration guide)
- Update base.html: user menu with avatar/name and sign-out via Alpine.js
  fetch to /api/v1/auth/me
- Update settings.html: remove localStorage adminApiKey; session cookie is
  sent automatically by browser; add 401 → /login redirect
- Update requirements.txt: replace fastapi-users additions with logto + aiohttp
- Add test_auth.py: 18 new tests covering session tokens, CookieStorage,
  sync_logto_user, /me, /sign-in (503), /sign-out cookie clearing
- Fix test_security_extra.py: pass Request mock to require_admin_auth;
  add new test_valid_session_cookie_returns_auth_context

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/b448f585-7646-40f8-ae2d-9986c361e3fd

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 10:09:50 +00:00
Christian Krakau-Louis 9a331f18c5 Merge pull request #80 from christianlouis/copilot/increase-test-coverage-80 
Increase test coverage to >80% with CI enforcement
 2026-03-30 11:36:28 +02:00
copilot-swe-agent[bot] bd26f3c607 fix: remove unused imports (flake8 F401) and replace hardcoded test password 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/72367edf-35f7-4c7c-a5ff-bbe4909b2139

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 08:32:56 +00:00
copilot-swe-agent[bot] 241713083b Add persistent settings system with database backend and comprehensive UI 
- New Setting ORM model (key-value store with category, value_type, audit fields)
- Alembic migration to create the settings table
- Settings API endpoints: GET/PUT /api/v1/settings/{key}, GET /api/v1/settings (list+filter), POST /api/v1/settings/bulk
- Default seeding (17 sensible defaults across general/dmarc/dns/cloudflare/notifications categories)
- Secret redaction for cloudflare.api_token and notifications.smtp_password
- Updated settings.html: General, DMARC Policy Defaults, DNS Resolver, Cloudflare Integration, Email Notifications sections
- All forms wired to the API via Alpine.js with flash feedback
- 12 new tests for the settings model and endpoints

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/19dbc6cd-07cb-406e-b3b6-411f7721f737

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 07:58:51 +00:00
copilot-swe-agent[bot] dc4e8377ee test: tighten token expiry tolerance in test_security_extra.py 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/752de981-5cf5-4789-a5b9-1f5ff165d202

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 00:38:16 +00:00
copilot-swe-agent[bot] 3524344a4a test: add comprehensive tests for imap, stats, setup, security; enforce 80% coverage gate 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/752de981-5cf5-4789-a5b9-1f5ff165d202

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-30 00:33:00 +00:00
Christian Krakau-Louis 06a007d9db Merge pull request #76 from christianlouis/copilot/update-dkim-selectors 
Fix DKIM selector extraction: surface all working selectors and auto-discovered report selectors
 2026-03-30 01:57:12 +02:00
copilot-swe-agent[bot] 596a9b882e Fix DKIM selector extraction: show all working selectors and report-discovered selectors 
- check_dkim now returns ALL matching selectors instead of stopping at first match
- DomainDNSResult.dkim_selectors is now a List[str] instead of a single Optional[str]
- DNSRecordResponse.dkimSelectors is now List[str]
- /selectors endpoint now also returns report_selectors (auto-discovered from DMARC reports)
- Frontend shows all live-check selectors and auto-discovered selectors as read-only
- Updated tests to match new data structures; added tests for multi-selector and report_selectors

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/87d8b8d9-23c3-4d3b-85a3-8e354e62c768

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 23:54:45 +00:00
Christian Krakau-Louis 14ac9572b2 Merge pull request #73 from christianlouis/copilot/implement-database-backed-key-storage 
Fix CodeQL clear-text logging alerts and improve startup branch coverage
 2026-03-30 01:52:12 +02:00
copilot-swe-agent[bot] 4f9e3b4b4b Fix CodeQL clear-text logging alerts and improve startup test coverage 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/3d5dbbdc-99d5-4dc3-8ca9-a763ba917ae4

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 23:43:14 +00:00
copilot-swe-agent[bot] ba4dcd9f81 Add tests for GET /api/v1/reports endpoint to fix codecov/patch coverage 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/7652d25c-fe9d-4576-a69d-731b091b84ff

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 23:39:16 +00:00
copilot-swe-agent[bot] ccb3f3fb7e Allow admin API key to be configured via ADMIN_API_KEY env var 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/27db6d87-70db-4979-a23c-dd376f1c3b9a

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 23:10:36 +00:00
copilot-swe-agent[bot] 30007d21d6 feat: add rDNS hostname, SPF fix hints and auth tooltips to domain sources view 
- dns_resolver.py: add _ip_to_arpa_name() helper and lookup_ptr() to
  BaseDNSProvider (no-op default), SystemDNSProvider (dnspython PTR),
  and CloudflareDNSProvider (DoH PTR type=12)
- domains.py: extend SourceEntry with hostname + spf_fix_hint; update
  get_domain_sources to run async PTR lookups and generate ip4:/ip6:
  SPF mechanism hints for failing IPs
- domain_details.html: show rDNS hostname below IP in sources table;
  add DaisyUI tooltip explaining each auth result; add "Fix SPF" popover
  with copy-paste mechanism for IPs that fail SPF
- tests: 15 new tests covering _ip_to_arpa_name, SystemDNSProvider/
  CloudflareDNSProvider PTR lookup, and sources endpoint hostname +
  fix-hint fields

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/9eaa7749-047c-46bd-8bc0-2851ea02ffe4

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 22:44:18 +00:00
Christian Krakau-Louis 1815747fba Merge pull request #69 from christianlouis/copilot/debug-not-found-error 
Fix /reports/{report_id} returning 404 Not Found
 2026-03-30 00:27:52 +02:00
Christian Krakau-Louis 8665e39ff4 Merge pull request #70 from christianlouis/copilot/fix-internal-server-error 
Fix 500 on GET /api/v1/domains/{domain}/sources: wrong record keys in ReportStore
 2026-03-30 00:26:51 +02:00
Christian Krakau-Louis 506346537c Merge pull request #67 from christianlouis/copilot/add-gmail-api-mail-source 
feat: add Gmail API & main.py helper tests; fix flake8 lint errors
 2026-03-30 00:26:28 +02:00
copilot-swe-agent[bot] af122881a5 Fix /reports/{report_id} returning 404 - add report detail page and API endpoint 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/4a412936-3bf7-4c02-b121-c0c342c34f1b

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 22:18:17 +00:00
copilot-swe-agent[bot] cd1f82aa79 Fix 500 error in GET /domains/{domain}/sources: use spf_result/dkim_result not spf/dkim 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/59552cb2-df2b-43dd-a22c-bd699a458afb

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 22:17:05 +00:00
copilot-swe-agent[bot] 70c6a70af3 fix: remove unused DMARC_GMAIL_QUERY import and fix trailing blank line/unused variable in test_gmail_client.py 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/21ea643d-3111-45d7-b66b-d90967ced313

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 22:14:01 +00:00
copilot-swe-agent[bot] 69c3bac480 feat: add GmailClient unit tests and main.py helper tests to meet coverage target 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/77ac3e98-81fa-45fa-9cc2-481896ffb48e

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 22:08:38 +00:00
copilot-swe-agent[bot] 2d364509bd fix: resolve remaining CodeQL alerts and raise mail_sources coverage to 100% 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/adfb364c-2feb-44fa-b9dd-73b9bdb7fa8f

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 21:39:00 +00:00
copilot-swe-agent[bot] dbbf539182 fix: resolve CodeQL security alerts – log injection, info exposure, incomplete URL checks 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/093a5de2-644b-434d-8ce8-529710ecb40b

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 21:14:02 +00:00
copilot-swe-agent[bot] 204131ee2a Add tests for domain reports and sources endpoints (fix Codecov patch coverage) 
- test_get_domain_reports_returns_200: basic 200 success path
- test_get_domain_reports_policy_dict_extracted: policy{"p":...} → "reject"
- test_get_domain_reports_policy_string_preserved: string policy unchanged
- test_get_domain_reports_timestamps_are_integers: begin/end_timestamp used
- test_get_domain_reports_unknown_domain_returns_404: 404 guard
- test_get_domain_sources_returns_200: sources returned correctly
- test_get_domain_sources_days_param_accepted: 'days' kwarg no TypeError
- test_get_domain_sources_unknown_domain_returns_404: 404 guard

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/a6f52c82-f239-4910-ac08-50702c154f93

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 21:10:06 +00:00
copilot-swe-agent[bot] 0c05ded74f fix: resolve flake8 C901/F401 errors in main.py, gmail_client.py, test_mail_sources.py 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/38868887-87b6-4032-8713-e90e56ae3318

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 19:50:50 +00:00
copilot-swe-agent[bot] 25b3567414 feat: implement Gmail API mail source with OAuth2, ingestion tracking, and UI 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/0c63851e-a69a-4a76-8dd8-25618e825b8c

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 19:34:03 +00:00
Christian Krakau-Louis 298ca938ec Merge pull request #62 from christianlouis/copilot/check-dns-data-in-dashboard 
fix: repair corrupted dns_resolver.py to pass black --check (py310 target)
 2026-03-29 21:17:25 +02:00
Christian Krakau-Louis 0bdd789944 Merge pull request #61 from christianlouis/copilot/debug-mail-sources-500-error 
fix: GET /mail-sources 500 — unhashable type: 'dict' in Jinja2 template cache
 2026-03-29 21:16:47 +02:00
Christian Krakau-Louis af457e4e48 Merge pull request #60 from christianlouis/copilot/handle-report-management 
Fix Black formatting in report_store.py
 2026-03-29 21:09:09 +02:00
copilot-swe-agent[bot] 2247115a7e test: add coverage for mail_sources_page TemplateResponse fix 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/bf182279-fe6a-4efc-87a5-5eaefabd2e3a

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 19:07:57 +00:00
copilot-swe-agent[bot] 0d1a4fdac3 feat: real DNS lookups, manual DKIM selectors, Cloudflare-ready DNS provider architecture 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/19d17518-732d-4644-889b-cc63256e19b1

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 18:53:35 +00:00
copilot-swe-agent[bot] 423c596c28 fix: ensure data persistence across reinstallations 
- Add entrypoint.sh: creates /app/data, stamps legacy DBs, runs alembic upgrade head
- Update Dockerfile to use new entrypoint and pre-create /app/data
- Change default DATABASE_URL to sqlite:///./data/dmarq.db
- Add _ensure_sqlite_dir() to database.py for automatic directory creation
- Update docker-compose.yml with app_data named volume for /app/data
- Add 6 tests for _ensure_sqlite_dir and updated default URL

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/2bac317b-97a2-450b-84f1-3e316f7ef54c

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 18:53:00 +00:00
copilot-swe-agent[bot] 53d543f49c Add report deduplication and single-report deletion 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/37d68c2c-7cc8-45e4-bac3-e2e6f1611c6a

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 18:28:52 +00:00
copilot-swe-agent[bot] 0844981bc0 fix: handle XML namespaces in DMARCParser for web.de/gmx.net compatibility 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/56611674-6132-435f-ab36-728464bb9157

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 17:59:13 +00:00
copilot-swe-agent[bot] 84117abd13 fix: resolve CodeQL alerts, improve test coverage, fix StaticPool for HTTP tests 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/07b58025-2aeb-4e81-a168-7a6fccdc3569

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 17:08:39 +00:00
copilot-swe-agent[bot] 883d54a428 fix: address code review - Alpine.js $set, SQLAlchemy bool filter style 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/f141cd3b-8d0a-4c7e-b479-ee973c209c11

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 16:20:11 +00:00
copilot-swe-agent[bot] 6d9392e5e7 feat: refactor IMAP config to DB, add multi-account mail sources admin UI 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/f141cd3b-8d0a-4c7e-b479-ee973c209c11

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 16:18:15 +00:00
copilot-swe-agent[bot] 70dc77011a Fix ModuleNotFoundError for asyncpg by normalizing async DB URLs to psycopg2 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/d1890838-1770-4d5f-886f-f210df761fae

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 16:03:18 +00:00
copilot-swe-agent[bot] e51fbe755e Replace mock dashboard data with real database queries and report-derived timelines 
- Replace mock statistics in stats_summarizer.py with real SQLAlchemy queries
  against DMARCReport, ReportRecord, and Domain models
- Replace random compliance timeline in domains.py with real data from ReportStore
- Remove unused `import random` from domains.py
- Add comprehensive tests for StatsSummarizer (global/domain/caching)
- Add tests for compliance timeline (deterministic, multi-report aggregation)
- Mark Dashboard Visualizations as complete in TODO.md

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/ea784ce3-2c1c-45f8-ad6d-46e92cf15ed7

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 15:43:33 +00:00
copilot-swe-agent[bot] 8e6ab244ad fix: remove unused pytest import from test_config.py 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/63e79da9-2002-429a-b696-7147dc579668

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 15:15:46 +00:00
copilot-swe-agent[bot] de4d1b2b34 style: apply black formatting to test_config.py 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/a0201223-b428-496d-90c7-cd268fee7774

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 13:52:28 +00:00
copilot-swe-agent[bot] 9f99b6c6d7 fix: handle empty BACKEND_CORS_ORIGINS env var to prevent JSONDecodeError 
In pydantic_settings v2, List[str] fields are JSON-parsed by the env
source before pydantic validators run. When BACKEND_CORS_ORIGINS=""
(empty string) in Kubernetes, json.loads("") raised JSONDecodeError,
crashing alembic migrations.

Fixes:
- Add env_ignore_empty=True to class Config so pydantic_settings skips
  empty-string env vars and falls back to the field default
- Update assemble_cors_origins validator to explicitly handle empty and
  whitespace-only strings (returns []), JSON array strings (parsed via
  json.loads), and filters empty tokens from comma-separated values
- Add backend/app/tests/test_config.py covering all CORS origins
  parsing scenarios including the empty-string regression case

Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/53bb1064-2a32-4691-8397-9d4663bc18a8

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 12:08:59 +00:00
copilot-swe-agent[bot] 01d0331136 Resolve linter contradictions: consolidate config, fix isort first-party, pylint 10/10 
Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/1e4a1f06-55b9-4040-853e-6aaf9ee574c8

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
 2026-03-29 11:31:15 +00:00