feat: define mail connector framework

docs/deployment/secrets.md

@@ -20,6 +20,8 @@ Store these values in a 1Password Environment for each deployment target:
 
 Non-sensitive values, such as `IMAP_SERVER`, `IMAP_USERNAME`, `LOGTO_ENDPOINT`, `LOGTO_APP_ID`, and `BACKEND_CORS_ORIGINS`, may also live in the Environment so each deployment has one complete configuration bundle.
 
+OAuth mail connectors can also store provider client secrets and refresh/access tokens in encrypted database fields after an administrator authorizes the source. Treat values such as `GMAIL_CLIENT_SECRET`, Microsoft 365 client secrets, refresh tokens, and access tokens as secrets even when they are provider-generated and short-lived.
+
 ## Create the Environment
 
 1. Open 1Password and enable the local MCP server or Environments feature if it is not already enabled.
@@ -89,6 +91,11 @@ Restrict the service user and file permissions so only the DMARQ process and the
 
 - Never commit `.env` files or secret values.
 - Never paste mailbox passwords, OAuth secrets, API tokens, database passwords, or generated session keys into issues, pull requests, logs, or chat.
+- Never include provider tokens, authorization headers, client secrets, raw mailbox payloads, or message bodies in connector diagnostics, import history, webhook payloads, screenshots, or support notes.
 - Keep `AUTH_DISABLED=true` limited to local development or a deployment protected by a separate authentication proxy.
 - Keep `LOGTO_SKIP_SSL_VERIFY=false` in production.
 - Use separate 1Password Environments for development, preprod, and production.
+
+## Connector Development
+
+New mail-source connectors must follow the shared connector contract in [Mail Connector Framework](../development/connectors.md). Use the shared sanitization helpers for provider errors and store only non-secret import context such as mailbox labels, folder labels, search windows, message IDs, filenames, domains, and report IDs.

docs/development/connectors.md

@@ -0,0 +1,56 @@
+# Mail Connector Framework
+
+DMARQ mailbox integrations should share one ingestion contract so new providers do not fork import behavior.
+
+## Connector Contract
+
+New mail-source connectors should implement the `MailSourceConnector` protocol in `backend/app/services/mail_connector.py`:
+
+- `import_context(days=None)` returns safe provider context such as source type, target mailbox, target folder, and search window. Do not include access tokens, refresh tokens, passwords, client secrets, authorization headers, raw provider payloads, or full message bodies.
+- `search_messages(days)` returns provider messages within the bounded search window.
+- `iter_attachments(message)` yields attachments for one provider message.
+- `fetch_reports(days=7)` runs the full ingestion path and returns the shared import-result shape.
+
+Use `ConnectorMessage` and `ConnectorAttachment` when provider data can be normalized cleanly. A connector may keep raw provider objects internally, but API responses and import history must use sanitized context and details only.
+
+## Import Result Shape
+
+Use `initial_import_stats()` to start an import result. The common keys are:
+
+- `success`
+- `processed`
+- `reports_found`
+- `forensic_reports_found`
+- `duplicate_reports`
+- `duplicate_forensic_reports`
+- `new_domains`
+- `errors`
+- `new_ingested_ids`
+- `details`
+
+Use `append_import_detail()` for message and attachment outcomes. Details should make retries understandable with reasons such as `already_ingested_message`, `unsupported_attachment`, `empty_attachment`, `parse_failed`, `duplicate`, or `imported`.
+
+Use `load_ingested_ids()` and `dump_ingested_ids()` for provider message IDs. A connector should mark a message as ingested only after the message was processed or determined to be safely skippable. Retryable message or attachment failures should not add the message ID to the ingested list.
+
+## Error Handling
+
+Provider failures must be mapped to sanitized, operator-readable diagnostics:
+
+- Use `sanitize_connector_error()` before storing or returning provider exception text.
+- Use `connector_failure_stats()` for failed list/search/setup paths.
+- Keep raw provider responses out of logs, import history, API responses, and frontend attributes.
+- Prefer bounded retries with provider backoff hints for throttling or temporary service failures.
+
+## Secret Handling
+
+Connectors may receive secrets from encrypted database fields or environment variables injected by the deployment runtime. They must not print or return those values.
+
+For local, preprod, and production deployments, prefer 1Password Environments or another runner-level secret injection mechanism. The connector code should only read the values it needs at runtime and should keep generated diagnostics safe for GitHub issues, import history, screenshots, and support requests.
+
+When adding a connector, add tests proving that:
+
+- duplicated provider message IDs do not inflate import totals,
+- parse failures and duplicates appear in `details`,
+- provider errors are redacted,
+- search/backfill windows are bounded,
+- secret-bearing strings are redacted before storage or API return.

docs/milestones.md

@@ -207,7 +207,7 @@ Planned:
 - Shared mailbox and folder selection support for DMARC report collection. Delivered with shared mailbox targeting, Microsoft Graph folder listing, folder-id based imports, UI selection, and mailbox/folder context in import history.
 - Import-history parity with existing sources (auditable attachment outcomes, duplicates, parse failures). Delivered for Microsoft 365 imports.
 - Backfill support with safe throttling and progressive search windows. Delivered with days-based Graph `receivedDateTime` filters, duplicate-safe reruns, and retry/backoff for throttled or temporarily unavailable Graph requests.
-- Secret handling mirrors existing guidance (no raw secrets in logs; 1Password-friendly).
+- Secret handling mirrors existing guidance (no raw secrets in logs; 1Password-friendly). Delivered with a shared connector protocol, sanitized import-result helpers, duplicate ID serialization helpers, and connector development guidance for future sources.
 
 Exit criteria:
 - A user can connect an Exchange Online mailbox, run an initial backfill, and then run scheduled polls with visible and trustworthy import history.