Merge branch 'main' into copilot/replace-mock-dns-data-with-actual-lookups

TODO.md

@@ -170,5 +170,7 @@ have no working implementation in the codebase yet.
 - [ ] Remove or wire up `fastapi-users` (currently installed but unused)
 - [x] Replace mock data in stats endpoints with real database queries
 - [x] Replace mock DNS data with actual DNS lookups
-- [ ] Add CI/CD pipeline
+- [x] Add CI/CD pipeline — GitHub Actions workflows in `.github/workflows/ci.yml`
+        (lint → test/security/CodeQL/dependency-review → Docker build/push → GitOps)
+        and `.github/workflows/release.yml` (semantic versioning)
 - [ ] Reach >80% test coverage

backend/app/templates/report_detail.html

@@ -51,6 +51,10 @@
                     <a :href="'/domains/' + report.domain" class="btn btn-outline btn-sm">
                         Back to Domain
                     </a>
+                    <button class="btn btn-error btn-sm"
+                            @click="deleteReport(report.domain, report.report_id)">
+                        Delete Report
+                    </button>
                 </div>
             </div>
 
@@ -247,6 +251,27 @@ function reportDetailApp(reportId) {
             }
         },
 
+        async deleteReport(domain, reportId) {
+            if (!confirm(`Delete report "${reportId}" for domain "${domain}"?\n\nThis will remove the report from the system. You can re-import it afterwards.`)) {
+                return;
+            }
+            try {
+                const response = await fetch(
+                    `/api/v1/reports/domain/${encodeURIComponent(domain)}/reports/${encodeURIComponent(reportId)}`,
+                    { method: 'DELETE' }
+                );
+                if (response.ok) {
+                    window.location.href = '/reports';
+                } else {
+                    const data = await response.json().catch(() => ({}));
+                    alert('Failed to delete report: ' + (data.detail || response.statusText));
+                }
+            } catch (error) {
+                console.error('Error deleting report:', error);
+                alert('Network error — could not delete report.');
+            }
+        },
+
         formatDate(timestamp) {
             if (!timestamp) return '—';
             return new Date(timestamp * 1000).toLocaleString();

backend/app/templates/reports.html

@@ -95,11 +95,17 @@
                                     </div>
                                 {% endcall %}
                                 {% call td("text-right") %}
+                                    <div class="inline-flex gap-2 justify-end">
                                         <a :href="`/reports/${report.report_id}`">
                                             {% call button(variant="outline", size="sm") %}
                                                 View Details
                                             {% endcall %}
                                         </a>
+                                        <button class="btn btn-error btn-sm"
+                                                @click="deleteReport(report.domain, report.report_id)">
+                                            Delete
+                                        </button>
+                                    </div>
                                 {% endcall %}
                             {% endcall %}
                         </template>
@@ -174,6 +180,32 @@ function reportsApp() {
             } finally {
                 this.loading = false;
             }
+        },
+
+        async deleteReport(domain, reportId) {
+            if (!confirm(`Delete report "${reportId}" for domain "${domain}"?\n\nThis will remove the report from the system. You can re-import it afterwards.`)) {
+                return;
+            }
+            try {
+                const response = await fetch(
+                    `/api/v1/reports/domain/${encodeURIComponent(domain)}/reports/${encodeURIComponent(reportId)}`,
+                    { method: 'DELETE' }
+                );
+                if (response.ok) {
+                    this.reports = this.reports.filter(
+                        r => !(r.domain === domain && r.report_id === reportId)
+                    );
+                    if (!this.reports.some(r => r.domain === domain)) {
+                        this.domains = this.domains.filter(d => d !== domain);
+                    }
+                } else {
+                    const data = await response.json().catch(() => ({}));
+                    alert('Failed to delete report: ' + (data.detail || response.statusText));
+                }
+            } catch (error) {
+                console.error('Error deleting report:', error);
+                alert('Network error — could not delete report.');
+            }
         }
     }
 }

docs/development/ci-cd.md

@@ -0,0 +1,175 @@
+# CI/CD Pipeline
+
+DMARQ uses GitHub Actions for all continuous integration and delivery tasks.
+The pipeline is defined in two workflow files:
+
+| File | Purpose | Triggers |
+|------|---------|----------|
+| `.github/workflows/ci.yml` | Lint → Test → Security → Docker → GitOps | Push to `main`/`develop`, pull requests, weekly schedule |
+| `.github/workflows/release.yml` | Semantic versioning & changelog | Push to `main` |
+
+---
+
+## Pipeline Stages
+
+### Stage 1 — Lint (blocking gate)
+
+All subsequent jobs depend on this stage succeeding.
+
+| Tool | What it checks |
+|------|---------------|
+| **Black** | Code formatting (line length 100, target Python 3.13) |
+| **isort** | Import ordering (Black-compatible profile) |
+| **Flake8** | Style and complexity (`max-complexity=10`, E203/W503/E501 ignored) |
+| **Pylint** | Deeper static analysis (`continue-on-error` — advisory only) |
+
+The lint job auto-formats with Black and isort before running the `--check`
+step, so a failing lint job is usually caused by a Flake8 or Pylint issue.
+
+### Stage 2 — Parallel quality gates
+
+These jobs run in parallel once lint passes.
+
+#### Test
+
+```bash
+cd backend
+pytest --cov=app --cov-report=xml --cov-report=term-missing
+```
+
+Coverage results are uploaded to [Codecov](https://codecov.io).
+
+#### Security Scan
+
+- **Bandit** — Python security linter; the JSON report is uploaded as a
+  workflow artifact (`bandit-security-report`) for every run.
+- **pip-audit** — checks all packages in `backend/requirements.txt` against
+  known CVE databases.
+
+Both steps use `continue-on-error: true` so they are advisory; a finding will
+not block the build but will appear in the workflow summary.
+
+#### CodeQL Analysis
+
+GitHub's semantic code analysis scans the Python source for known vulnerability
+patterns (security and quality queries).  Runs on push and PR events only —
+skipped on the weekly scheduled scan.
+
+#### Dependency Review
+
+Runs on pull requests only.  Fails if any new dependency introduces a
+vulnerability of `moderate` severity or higher.
+
+### Stage 3 — Docker Build & Publish
+
+Runs only on pushes to `main` after both **Test** and **Security** pass.
+
+- Builds from `./backend/Dockerfile`
+- Pushes to `ghcr.io/<owner>/dmarq` with three tags:
+  - `latest` (default branch only)
+  - branch name (e.g. `main`)
+  - short commit SHA (e.g. `a1b2c3d`)
+- Layer cache is stored via GitHub Actions cache (`type=gha`).
+
+### Stage 4 — GitOps (Update K8s Manifest)
+
+Runs only on pushes to `main` after the Docker stage succeeds.
+
+Updates the image tag in the preprod Kubernetes manifest at
+`apps/dmarq/preprod/dmarq-stack.yaml` in the `christianlouis/k8s-cluster-state`
+repository.
+
+!!! note "Optional"
+    This stage requires a `GH_PAT` repository secret with write access to the
+    k8s-cluster-state repo.  If the secret is absent or lacks access the step
+    emits a warning and skips gracefully — it will never fail the pipeline.
+
+---
+
+## Release Workflow
+
+`release.yml` uses
+[python-semantic-release](https://python-semantic-release.readthedocs.io/)
+to automate versioning from
+[Conventional Commits](https://www.conventionalcommits.org/):
+
+- Bumps `version` in `pyproject.toml`
+- Generates/updates `CHANGELOG.md`
+- Creates a Git tag and a GitHub Release
+
+Commit prefixes that trigger a release:
+
+| Prefix | Version bump |
+|--------|-------------|
+| `fix:` | Patch (0.0.**x**) |
+| `feat:` | Minor (0.**x**.0) |
+| `feat!:` / `BREAKING CHANGE` | Major (**x**.0.0) |
+
+---
+
+## Scheduled Security Scan
+
+A cron job runs every Monday at 00:00 UTC and executes the **Lint**,
+**Security Scan**, and **Test** stages against the latest `main` code.
+CodeQL and Dependency Review are skipped on schedule events.
+
+---
+
+## Troubleshooting
+
+### Lint failure
+
+1. Pull the latest branch and run locally:
+   ```bash
+   black backend/app
+   isort backend/app
+   flake8 backend/app
+   ```
+2. Commit the formatted files and push.
+
+Pylint failures are advisory (`continue-on-error: true`) and will not block
+the pipeline, but should still be investigated.
+
+### Test failure
+
+1. Reproduce locally:
+   ```bash
+   cd backend
+   pytest --cov=app --cov-report=term-missing -x
+   ```
+2. Check the test output in the **Test** job log for the failing assertion and
+   stack trace.
+3. The coverage XML is not uploaded as an artifact — run locally to inspect
+   `htmlcov/index.html`.
+
+### Security scan failure
+
+- **Bandit**: Download the `bandit-security-report` artifact from the workflow
+  run's *Artifacts* panel and review `bandit-report.json`.
+- **pip-audit**: The finding is printed directly to the job log.  Update or pin
+  the affected dependency in `backend/requirements.txt`.
+
+### Docker build failure
+
+Common causes:
+
+| Symptom | Fix |
+|---------|-----|
+| `pip install` error | Check `backend/requirements.txt` for unpinned or incompatible versions |
+| `COPY` file not found | Ensure the file exists and is not listed in `.dockerignore` |
+| Registry auth failure | Verify the workflow has `packages: write` permission |
+
+### Dependency Review failure (PR only)
+
+The PR introduced a dependency with a known vulnerability.  Either:
+
+- Remove the dependency, or
+- Upgrade to a patched version, or
+- If the finding is a false positive, discuss with a maintainer — the severity
+  threshold (`moderate`) can be adjusted in the workflow.
+
+### Release workflow not triggering
+
+Ensure your merge commit message follows Conventional Commits.  The release
+workflow only creates a new version when `python-semantic-release` detects a
+`fix:`, `feat:`, or breaking-change commit since the last tag.

docs/development/testing.md

@@ -140,10 +140,14 @@ isort backend/app
 
 ## Continuous Integration
 
-Tests run automatically on every push and PR via GitHub Actions (`.github/workflows/test.yml`).
+Tests run automatically on every push and PR via GitHub Actions
+(`.github/workflows/ci.yml`).  See [CI/CD Pipeline](ci-cd.md) for a full
+description of every stage.
 
-The CI workflow:
+The pipeline runs in four stages:
-1. Installs dependencies (Python 3.13)
+
-2. Runs `pytest` with coverage
+1. **Lint** (blocking gate) — Black, isort, Flake8, Pylint
-3. Runs linting checks (Black, isort, Flake8, Pylint)
+2. **Parallel quality gates** — Test (pytest + Codecov), Security (Bandit + pip-audit),
-4. Uploads coverage to Codecov
+   CodeQL analysis, and Dependency Review (PRs only)
+3. **Docker** — builds and pushes to `ghcr.io` (main branch only)
+4. **GitOps** — updates the preprod Kubernetes manifest (main branch only)

mkdocs.yml

@@ -44,6 +44,7 @@ nav:
   - Development:
     - Contributing: development/contributing.md
     - Testing: development/testing.md
+    - CI/CD Pipeline: development/ci-cd.md
     - Roadmap: development/roadmap.md
     - Agentic Coding: development/agents.md
     - Issue Generation: development/issue_generation.md