diff --git a/docs/deployment/configuration.md b/docs/deployment/configuration.md index 68267c8..13a20fb 100644 --- a/docs/deployment/configuration.md +++ b/docs/deployment/configuration.md @@ -13,6 +13,7 @@ DMARQ can be configured through: For production secrets, use the [1Password secret handling guide](secrets.md) so sensitive values are injected into the DMARQ process without being committed or copied into deployment notes. For database operations, use the [Database Backup and Restore](backups.md) guide before upgrades and migrations. +For deployment verification, upgrades, rollback, and routine checks, use the [Operator Runbook](operations.md). For incident response, use [Troubleshooting Playbooks](troubleshooting.md). ## Core Settings diff --git a/docs/deployment/docker.md b/docs/deployment/docker.md index 8bda247..2a33e96 100644 --- a/docs/deployment/docker.md +++ b/docs/deployment/docker.md @@ -147,6 +147,8 @@ The Docker Compose setup uses these volumes: For production deployments, consider these additional steps: +For an operator-focused sequence that covers Docker Compose, Coolify, verification, upgrades, and rollback, use the [Operator Runbook](operations.md). For incident response and ingestion/auth troubleshooting, use [Troubleshooting Playbooks](troubleshooting.md). + ### Using a Reverse Proxy In production, it's recommended to use a reverse proxy like Nginx or Traefik in front of DMARQ: @@ -224,6 +226,7 @@ docker-compose --profile production up -d ## Updating DMARQ Before updating a production deployment, follow the [Release Checklist](release-checklist.md) and create a database backup. +The full operator upgrade sequence is documented in [Operator Runbook](operations.md). To update to a newer version: diff --git a/docs/deployment/manual.md b/docs/deployment/manual.md index 138a2e2..734e1d7 100644 --- a/docs/deployment/manual.md +++ b/docs/deployment/manual.md @@ -2,6 +2,8 @@ This guide covers how to deploy DMARQ without Docker, using a traditional installation method. +For production operations after installation, use the [Operator Runbook](operations.md) for verification, upgrades, rollback, and routine checks. Use [Troubleshooting Playbooks](troubleshooting.md) when auth, ingestion, DNS, database, or notification workflows fail. + ## Prerequisites Before proceeding with a manual installation, ensure you have: @@ -241,6 +243,7 @@ sudo systemctl enable dmarq-celery ## Updating DMARQ Before updating a production deployment, follow the [Release Checklist](release-checklist.md) and create a database backup. +The full operator upgrade sequence is documented in [Operator Runbook](operations.md). To update to a newer version: diff --git a/docs/deployment/operations.md b/docs/deployment/operations.md new file mode 100644 index 0000000..7ed43b8 --- /dev/null +++ b/docs/deployment/operations.md @@ -0,0 +1,126 @@ +# Operator Runbook + +This runbook is for people who deploy and operate DMARQ. It focuses on repeatable actions: deploy, verify, upgrade, back up, and recover. Keep secrets in 1Password or your deployment secret manager; do not paste raw values into tickets, pull requests, chat, or deployment notes. + +## Deployment Modes + +### Docker Compose + +Use Docker Compose for a single-host deployment or a small production instance. + +1. Put configuration in the project `.env` file, preferably mounted from a 1Password Environment. +2. Use PostgreSQL for production. SQLite is acceptable for local or low-volume personal deployments. +3. Start the stack: + + ```bash + docker compose up -d + ``` + +4. Verify: + + ```bash + docker compose ps + docker compose logs --tail=100 backend + curl -fsS http://localhost:8000/health + curl -fsS http://localhost:8000/api/v1/health + ``` + +### Coolify + +Use Coolify when you want Git-based Docker deployment with managed environment variables. + +1. Create a Docker Compose application from the DMARQ repository. +2. Configure persistent storage for: + - PostgreSQL data if Coolify manages the database. + - `/app/data` when using SQLite or file-backed runtime data. +3. Add environment variables in Coolify or inject them from 1Password. Keep `SECRET_KEY`, database credentials, mailbox passwords, OAuth secrets, `WEBHOOK_SECRET`, and Cloudflare tokens concealed. +4. Deploy the app and confirm the backend container stays healthy. +5. Verify the public URL with `/health`, `/api/v1/health`, the dashboard, and Mail Sources. + +Do not mount the source tree over the production container. Production containers should run the image contents, not a local development volume. + +### Manual Systemd + +Use manual installation when Docker is not available. + +1. Install Python 3.13 and dependencies in a virtual environment. +2. Configure a systemd unit with a restricted service user. +3. Reference an environment file mounted from 1Password or managed by the host secret store. +4. Run behind Nginx or another TLS reverse proxy. +5. Verify: + + ```bash + sudo systemctl status dmarq + sudo journalctl -u dmarq -n 100 + curl -fsS http://127.0.0.1:8000/health + ``` + +### Kubernetes Or GitOps + +Use Kubernetes when DMARQ is deployed through a cluster-state repository. + +1. Store secrets in the cluster secret manager, not in manifests. +2. Pin the image tag to the release or commit being promoted. +3. Apply database migrations through the normal release workflow before routing traffic. +4. Watch rollout status and pod logs. +5. Confirm the preprod manifest update from CI before promoting the same image to production. + +## First Deployment Verification + +After a new deployment: + +1. Open `/health` and `/api/v1/health`. +2. Complete the initial setup flow or confirm login works. +3. Add one monitored domain. +4. Upload a known-good DMARC aggregate report. +5. Confirm the dashboard shows domain totals and compliance. +6. Open the domain details page and verify DNS Health Summary loads. +7. Add or connect a Mail Source, then run **Test connection**. +8. Trigger a manual fetch in a safe mailbox and confirm import history records the attempt. +9. Check logs for repeated database, authentication, DNS, or mailbox polling errors. + +## Upgrade Procedure + +Use this sequence for every production upgrade: + +1. Read the release notes and check for migration or configuration changes. +2. Follow [Database Backup and Restore](backups.md) to create and verify a fresh backup. +3. Record the current image tag, commit SHA, database type, and deployment target. +4. Pull or deploy the new image. +5. Apply database migrations if the release includes them. +6. Restart the app. +7. Run the smoke checks in [Release Checklist](release-checklist.md). +8. Confirm background polling still works by checking Mail Sources import history. +9. Keep the previous image tag and backup available until the deployment has been stable through at least one polling interval. + +## Routine Operations + +### Daily + +- Check the dashboard for report volume and compliance drops. +- Review Mail Sources import history for repeated failures. +- Confirm notifications are being delivered if alerts are enabled. + +### Weekly + +- Verify the most recent database backup. +- Review unresolved DNS health warnings for monitored domains. +- Check app logs for repeated provider throttling, authentication, or database errors. + +### Monthly + +- Rotate credentials according to your policy. +- Confirm the restore procedure still works in a temporary environment. +- Review `ALLOWED_HOSTS`, Logto redirect URLs, Cloudflare tokens, and mailbox access. + +## Rollback + +Rollback is safest when the database schema did not change. If a migration ran, prefer a full restore into the previous application version unless the release notes explicitly say downgrade is supported. + +1. Stop DMARQ. +2. Restore the pre-upgrade database backup if needed. +3. Repoint the deployment to the previous image tag or commit. +4. Start DMARQ. +5. Run health checks, dashboard checks, and Mail Sources checks. +6. Record the rollback cause and open a follow-up issue. + diff --git a/docs/deployment/release-checklist.md b/docs/deployment/release-checklist.md index 901d549..b2d7362 100644 --- a/docs/deployment/release-checklist.md +++ b/docs/deployment/release-checklist.md @@ -2,6 +2,8 @@ Use this checklist for every DMARQ release or production upgrade. It focuses on the work that most often causes avoidable downtime: database migrations, test coverage, release automation, and smoke checks. +For the full deploy, verification, rollback, and routine operations runbook, see [Operator Runbook](operations.md). For incident-specific recovery steps, see [Troubleshooting Playbooks](troubleshooting.md). + ## Before Merging Confirm the release candidate is ready: diff --git a/docs/deployment/troubleshooting.md b/docs/deployment/troubleshooting.md new file mode 100644 index 0000000..d1757ce --- /dev/null +++ b/docs/deployment/troubleshooting.md @@ -0,0 +1,175 @@ +# Troubleshooting Playbooks + +Use these playbooks when DMARQ is running but an operator workflow is not behaving as expected. Keep sensitive values redacted. When you need to test credentials, inject them into the authorized process from 1Password or your deployment secret manager instead of copying them into commands or notes. + +## Quick Triage + +Start with these checks: + +```bash +curl -fsS https://your-dmarq-host.example.com/health +curl -fsS https://your-dmarq-host.example.com/api/v1/health +``` + +For Docker Compose: + +```bash +docker compose ps +docker compose logs --tail=100 backend +``` + +For systemd: + +```bash +sudo systemctl status dmarq +sudo journalctl -u dmarq -n 100 +``` + +Then check the UI: + +1. Dashboard loads and shows domains. +2. Domain details load DNS health, reports, and sending sources. +3. Mail Sources shows recent import history. +4. Settings saves and test notifications work if enabled. + +## App Does Not Start + +Likely causes: + +- Missing or weak `SECRET_KEY`. +- Database connection failure. +- PostgreSQL credentials or host changed. +- SQLite path is not writable. +- Production startup checks blocked unsafe settings. + +Actions: + +1. Check the last 100 log lines. +2. Confirm required environment variables are present in the deployment process. +3. Confirm secrets are injected, not present only in your shell. +4. For PostgreSQL, verify network access from the app host to the database host. +5. For SQLite, verify the data directory exists and is writable by the app user. +6. Restart after fixing configuration. + +## Login Or Setup Fails + +Likely causes: + +- Logto redirect URL does not match the deployed public URL. +- `ALLOWED_HOSTS` does not include the public hostname. +- `SECRET_KEY` changed unexpectedly, invalidating sessions. +- `AUTH_DISABLED` or `LOGTO_SKIP_SSL_VERIFY` is set incorrectly for the environment. + +Actions: + +1. Confirm `ALLOWED_HOSTS` includes the host users open in the browser. +2. Confirm Logto callback URLs match the DMARQ callback URL exactly. +3. Confirm `SECRET_KEY` is stable across restarts. +4. Clear browser cookies after intentional auth changes. +5. Keep `AUTH_DISABLED=false` and `LOGTO_SKIP_SSL_VERIFY=false` in production. + +## Mailbox Test Fails + +Open **Mail Sources**, run **Test connection**, and use the diagnostic category shown in the result. + +| Diagnostic | What it means | Recovery | +|------------|---------------|----------| +| `auth_required` | Gmail OAuth was not completed. | Use **Connect Gmail** and authorize the mailbox that receives DMARC reports. | +| `auth_expired` | OAuth token was revoked, expired, or rejected. | Reconnect the source and approve the requested scope again. | +| `authentication` | Username, password, app password, or token was rejected. | Verify credentials. For IMAP providers with MFA, create a provider app password. | +| `permissions` | The account is connected but lacks mailbox access. | Grant mailbox read access or reconnect Gmail with the requested read-only scope. | +| `connectivity` | DMARQ cannot reach the provider. | Check hostname, port, TLS setting, firewall, DNS, and provider availability. | +| `mailbox_not_found` | The configured folder cannot be opened. | Use one of the returned mailbox names and match capitalization/separators exactly. | +| `throttling` | Provider is rate limiting requests. | Wait, retry, and increase the polling interval if failures repeat. | +| `missing_config` | Required settings are absent. | Fill in server, username, password, or complete OAuth authorization. | + +Never paste mailbox passwords or OAuth tokens into logs or issue comments. If a provider error contains a token-like value, redact it before sharing. + +## Reports Are Not Importing + +Likely causes: + +- DMARC reports are sent to a different mailbox than the configured source. +- The folder setting points to the wrong mailbox. +- Provider search is throttled or delayed. +- Messages have already been imported and are skipped as duplicates. +- Attachments are malformed, empty, encrypted, or not XML/ZIP/GZIP. + +Actions: + +1. Run **Test connection** and verify the selected folder exists. +2. Run a manual fetch from Mail Sources. +3. Open import history and inspect processed count, reports found, duplicates, errors, and details. +4. Search the mailbox manually for recent DMARC aggregate report subjects. +5. Upload a known-good report through the Reports page to confirm parsing still works. +6. Check logs for attachment parse errors or provider throttling. + +## Gmail API Source Is Connected But Fetch Fails + +Actions: + +1. Reconnect the Gmail source. +2. Confirm the connected account email is the mailbox receiving reports. +3. Confirm the Google OAuth app still has the Gmail read-only scope configured. +4. Check whether the Google project or OAuth consent configuration changed. +5. Retry after a few minutes if the failure looks like quota or throttling. + +## Webhook Imports Fail + +Likely causes: + +- `WEBHOOK_SECRET` mismatch between DMARQ and the email worker. +- Worker sends malformed base64 or raw message payload. +- The email has no supported DMARC attachment. +- Public URL or routing changed. + +Actions: + +1. Confirm `WEBHOOK_SECRET` is set in both systems through the secret manager. +2. Check DMARQ logs for webhook rejection reasons. +3. Confirm the worker points to the current DMARQ public URL. +4. Send a test message with a known-good DMARC aggregate attachment. +5. Rotate the webhook secret if it may have been exposed. + +## DNS Health Looks Wrong + +Actions: + +1. Open the domain details page and refresh DNS health. +2. Check whether the DNS result is cached. Wait 15 minutes or use a refresh action after DNS changes. +3. Confirm the domain exists as a monitored domain even if no reports have arrived yet. +4. Add missing DKIM selectors manually when reports do not reveal all selectors. +5. For Cloudflare-managed domains, confirm the API token can read the zone. + +## Database Or Migration Problems + +Actions: + +1. Stop the app before making manual database changes. +2. Confirm the latest backup exists and passes the relevant validation check. +3. For PostgreSQL, test connection from the app host using the same injected environment. +4. For SQLite, check file ownership and disk space. +5. Restore from [Database Backup and Restore](backups.md) if the app cannot safely continue. + +## Notifications Do Not Send + +Actions: + +1. Open **Settings** > **Notifications** and send a test notification. +2. Confirm Apprise URLs are saved through the UI and are redacted in API responses. +3. Check logs for provider rejection or rate limiting. +4. Confirm outbound network access from the DMARQ host. +5. Rotate notification credentials if a target was exposed. + +## When To Escalate + +Open a follow-up issue with: + +- Deployment mode and image tag or commit SHA. +- Health check result. +- Sanitized log excerpt. +- Mail source diagnostic category, if ingestion is involved. +- Whether a recent upgrade, migration, credential rotation, or DNS change happened. + +Do not include raw secrets, mailbox contents, OAuth tokens, API tokens, database passwords, or full report files unless they have been sanitized. + diff --git a/docs/index.md b/docs/index.md index d441ec1..696acbf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -21,4 +21,4 @@ DMARQ is a full-stack DMARC monitoring platform designed to help organizations t To get started with DMARQ, please see the [Getting Started](user_guide/getting_started.md) guide. -For installation instructions, check the [Docker Setup](deployment/docker.md) or [Manual Installation](deployment/manual.md) guides. For production secrets, use [Secret Handling with 1Password](deployment/secrets.md). For database operations, use [Database Backup and Restore](deployment/backups.md). For upgrades, use the [Release Checklist](deployment/release-checklist.md). +For installation instructions, check the [Docker Setup](deployment/docker.md) or [Manual Installation](deployment/manual.md) guides. Operators should use the [Operator Runbook](deployment/operations.md) for deployment modes, verification, upgrades, and rollback, and the [Troubleshooting Playbooks](deployment/troubleshooting.md) for ingestion, authentication, DNS, database, and notification failures. For production secrets, use [Secret Handling with 1Password](deployment/secrets.md). For database operations, use [Database Backup and Restore](deployment/backups.md). For upgrades, use the [Release Checklist](deployment/release-checklist.md). diff --git a/docs/milestones.md b/docs/milestones.md index 1f2e5d5..2fa6ad4 100644 --- a/docs/milestones.md +++ b/docs/milestones.md @@ -144,7 +144,7 @@ Exit criteria: ## Milestone 9: Setup and Operations Polish -Status: In Progress +Status: Delivered Goal: make first-run setup, maintenance, and troubleshooting straightforward. @@ -154,10 +154,9 @@ Delivered: - Monitored domains can be created before any DMARC report has arrived. - Domain summaries and detail pages include manually configured domains with no report history yet. - Health page and API show database connectivity, scheduler state, report totals, latest import, and latest successful import. - -Remaining: -- Better mailbox test output and recovery suggestions. -- Operator documentation refresh for Docker Compose and manual deployments. +- Mailbox connection tests return sanitized diagnostic categories and recovery suggestions. +- Mail Sources UI shows actionable recovery steps for common ingestion/auth failures. +- Operator runbooks cover Docker Compose, Coolify, manual systemd, Kubernetes/GitOps, upgrades, rollback, routine checks, and troubleshooting. Exit criteria: - A new user can deploy DMARQ, connect a mailbox, and confirm the system is healthy without reading code. diff --git a/docs/user_guide/deployment_guide.md b/docs/user_guide/deployment_guide.md index 616c117..c99b66d 100644 --- a/docs/user_guide/deployment_guide.md +++ b/docs/user_guide/deployment_guide.md @@ -11,6 +11,8 @@ This guide provides step-by-step instructions for deploying DMARQ in various env 5. [Production Best Practices](#production-best-practices) 6. [Upgrading](#upgrading) +For day-to-day production operation, use the [Operator Runbook](../deployment/operations.md). For failure recovery, use [Troubleshooting Playbooks](../deployment/troubleshooting.md). + ## Docker Deployment (Recommended) The easiest way to deploy DMARQ is using Docker and Docker Compose. This approach packages all dependencies and provides a consistent environment. @@ -333,4 +335,4 @@ docker-compose down docker-compose up -d ``` -Always check the release notes for any specific upgrade instructions or breaking changes. \ No newline at end of file +Always check the release notes for any specific upgrade instructions or breaking changes.