From 172d6465ab1fef6a8e7af1fdc0283ed9d29e62c9 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Sun, 29 Mar 2026 23:59:43 +0000 Subject: [PATCH 1/5] Initial plan From 7955dac8e193259742245a1169e33cdb4ed0b192 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 30 Mar 2026 00:05:16 +0000 Subject: [PATCH 2/5] docs: mark CI/CD pipeline done, add ci-cd.md, fix testing.md reference Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/db6908f8-a99f-426a-97f3-947d5c02944b Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> --- TODO.md | 4 +- docs/development/ci-cd.md | 175 ++++++++++++++++++++++++++++++++++++ docs/development/testing.md | 17 ++-- mkdocs.yml | 1 + 4 files changed, 190 insertions(+), 7 deletions(-) create mode 100644 docs/development/ci-cd.md diff --git a/TODO.md b/TODO.md index 6ae6b67..200aa26 100644 --- a/TODO.md +++ b/TODO.md @@ -160,5 +160,7 @@ have no working implementation in the codebase yet. - [ ] Remove or wire up `fastapi-users` (currently installed but unused) - [x] Replace mock data in stats endpoints with real database queries - [ ] Replace mock DNS data with actual DNS lookups -- [ ] Add CI/CD pipeline +- [x] Add CI/CD pipeline — GitHub Actions workflows in `.github/workflows/ci.yml` + (lint → test/security/CodeQL/dependency-review → Docker build/push → GitOps) + and `.github/workflows/release.yml` (semantic versioning) - [ ] Reach >80% test coverage diff --git a/docs/development/ci-cd.md b/docs/development/ci-cd.md new file mode 100644 index 0000000..fb94227 --- /dev/null +++ b/docs/development/ci-cd.md @@ -0,0 +1,175 @@ +# CI/CD Pipeline + +DMARQ uses GitHub Actions for all continuous integration and delivery tasks. +The pipeline is defined in two workflow files: + +| File | Purpose | Triggers | +|------|---------|----------| +| `.github/workflows/ci.yml` | Lint → Test → Security → Docker → GitOps | Push to `main`/`develop`, pull requests, weekly schedule | +| `.github/workflows/release.yml` | Semantic versioning & changelog | Push to `main` | + +--- + +## Pipeline Stages + +### Stage 1 — Lint (blocking gate) + +All subsequent jobs depend on this stage succeeding. + +| Tool | What it checks | +|------|---------------| +| **Black** | Code formatting (line length 100, target Python 3.13) | +| **isort** | Import ordering (Black-compatible profile) | +| **Flake8** | Style and complexity (`max-complexity=10`, E203/W503/E501 ignored) | +| **Pylint** | Deeper static analysis (`continue-on-error` — advisory only) | + +The lint job auto-formats with Black and isort before running the `--check` +step, so a failing lint job is usually caused by a Flake8 or Pylint issue. + +### Stage 2 — Parallel quality gates + +These jobs run in parallel once lint passes. + +#### Test + +```bash +cd backend +pytest --cov=app --cov-report=xml --cov-report=term-missing +``` + +Coverage results are uploaded to [Codecov](https://codecov.io). + +#### Security Scan + +- **Bandit** — Python security linter; the JSON report is uploaded as a + workflow artifact (`bandit-security-report`) for every run. +- **pip-audit** — checks all packages in `backend/requirements.txt` against + known CVE databases. + +Both steps use `continue-on-error: true` so they are advisory; a finding will +not block the build but will appear in the workflow summary. + +#### CodeQL Analysis + +GitHub's semantic code analysis scans the Python source for known vulnerability +patterns (security and quality queries). Runs on push and PR events only — +skipped on the weekly scheduled scan. + +#### Dependency Review + +Runs on pull requests only. Fails if any new dependency introduces a +vulnerability of `moderate` severity or higher. + +### Stage 3 — Docker Build & Publish + +Runs only on pushes to `main` after both **Test** and **Security** pass. + +- Builds from `./backend/Dockerfile` +- Pushes to `ghcr.io//dmarq` with three tags: + - `latest` (default branch only) + - branch name (e.g. `main`) + - short commit SHA (e.g. `a1b2c3d`) +- Layer cache is stored via GitHub Actions cache (`type=gha`). + +### Stage 4 — GitOps (Update K8s Manifest) + +Runs only on pushes to `main` after the Docker stage succeeds. + +Updates the image tag in the preprod Kubernetes manifest at +`apps/dmarq/preprod/dmarq-stack.yaml` in the `christianlouis/k8s-cluster-state` +repository. + +!!! note "Optional" + This stage requires a `GH_PAT` repository secret with write access to the + k8s-cluster-state repo. If the secret is absent or lacks access the step + emits a warning and skips gracefully — it will never fail the pipeline. + +--- + +## Release Workflow + +`release.yml` uses +[python-semantic-release](https://python-semantic-release.readthedocs.io/) +to automate versioning from +[Conventional Commits](https://www.conventionalcommits.org/): + +- Bumps `version` in `pyproject.toml` +- Generates/updates `CHANGELOG.md` +- Creates a Git tag and a GitHub Release + +Commit prefixes that trigger a release: + +| Prefix | Version bump | +|--------|-------------| +| `fix:` | Patch (0.0.**x**) | +| `feat:` | Minor (0.**x**.0) | +| `feat!:` / `BREAKING CHANGE` | Major (**x**.0.0) | + +--- + +## Scheduled Security Scan + +A cron job runs every Monday at 00:00 UTC and executes the **Lint**, +**Security Scan**, and **Test** stages against the latest `main` code. +CodeQL and Dependency Review are skipped on schedule events. + +--- + +## Troubleshooting + +### Lint failure + +1. Pull the latest branch and run locally: + ```bash + black backend/app + isort backend/app + flake8 backend/app + ``` +2. Commit the formatted files and push. + +Pylint failures are advisory (`continue-on-error: true`) and will not block +the pipeline, but should still be investigated. + +### Test failure + +1. Reproduce locally: + ```bash + cd backend + pytest --cov=app --cov-report=term-missing -x + ``` +2. Check the test output in the **Test** job log for the failing assertion and + stack trace. +3. The coverage XML is not uploaded as an artifact — run locally to inspect + `htmlcov/index.html`. + +### Security scan failure + +- **Bandit**: Download the `bandit-security-report` artifact from the workflow + run's *Artifacts* panel and review `bandit-report.json`. +- **pip-audit**: The finding is printed directly to the job log. Update or pin + the affected dependency in `backend/requirements.txt`. + +### Docker build failure + +Common causes: + +| Symptom | Fix | +|---------|-----| +| `pip install` error | Check `backend/requirements.txt` for unpinned or incompatible versions | +| `COPY` file not found | Ensure the file exists and is not listed in `.dockerignore` | +| Registry auth failure | Verify the workflow has `packages: write` permission | + +### Dependency Review failure (PR only) + +The PR introduced a dependency with a known vulnerability. Either: + +- Remove the dependency, or +- Upgrade to a patched version, or +- If the finding is a false positive, discuss with a maintainer — the severity + threshold (`moderate`) can be adjusted in the workflow. + +### Release workflow not triggering + +Ensure your merge commit message follows Conventional Commits. The release +workflow only creates a new version when `python-semantic-release` detects a +`fix:`, `feat:`, or breaking-change commit since the last tag. diff --git a/docs/development/testing.md b/docs/development/testing.md index a5605d9..13be491 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -140,10 +140,15 @@ isort backend/app ## Continuous Integration -Tests run automatically on every push and PR via GitHub Actions (`.github/workflows/test.yml`). +Tests run automatically on every push and PR via GitHub Actions +(`.github/workflows/ci.yml`). See [CI/CD Pipeline](ci-cd.md) for a full +description of every stage. -The CI workflow: -1. Installs dependencies (Python 3.13) -2. Runs `pytest` with coverage -3. Runs linting checks (Black, isort, Flake8, Pylint) -4. Uploads coverage to Codecov \ No newline at end of file +The pipeline runs in four stages: + +1. **Lint** (blocking gate) — Black, isort, Flake8, Pylint +2. **Test** — `pytest` with coverage; report uploaded to Codecov +3. **Security** — Bandit static analysis + pip-audit dependency scan +4. **CodeQL** — GitHub's semantic code analysis for Python +5. **Docker** — builds and pushes to `ghcr.io` (main branch only) +6. **GitOps** — updates the preprod Kubernetes manifest (main branch only) \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 85bbe73..f2f8e01 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -44,6 +44,7 @@ nav: - Development: - Contributing: development/contributing.md - Testing: development/testing.md + - CI/CD Pipeline: development/ci-cd.md - Roadmap: development/roadmap.md - Agentic Coding: development/agents.md - Issue Generation: development/issue_generation.md From 7b932bb41dc388de9579e58bde8402e88367fdbf Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 30 Mar 2026 00:06:31 +0000 Subject: [PATCH 3/5] docs: fix stage count in testing.md CI section Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/db6908f8-a99f-426a-97f3-947d5c02944b Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> --- docs/development/testing.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/development/testing.md b/docs/development/testing.md index 13be491..246e9e0 100644 --- a/docs/development/testing.md +++ b/docs/development/testing.md @@ -147,8 +147,7 @@ description of every stage. The pipeline runs in four stages: 1. **Lint** (blocking gate) — Black, isort, Flake8, Pylint -2. **Test** — `pytest` with coverage; report uploaded to Codecov -3. **Security** — Bandit static analysis + pip-audit dependency scan -4. **CodeQL** — GitHub's semantic code analysis for Python -5. **Docker** — builds and pushes to `ghcr.io` (main branch only) -6. **GitOps** — updates the preprod Kubernetes manifest (main branch only) \ No newline at end of file +2. **Parallel quality gates** — Test (pytest + Codecov), Security (Bandit + pip-audit), + CodeQL analysis, and Dependency Review (PRs only) +3. **Docker** — builds and pushes to `ghcr.io` (main branch only) +4. **GitOps** — updates the preprod Kubernetes manifest (main branch only) \ No newline at end of file From e75fde4311f3e65237812d5b0c4a674c0dd397a8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 30 Mar 2026 00:07:01 +0000 Subject: [PATCH 4/5] Initial plan From 61cb428f5476e027d427be02fa38dbf7505039b8 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 30 Mar 2026 00:37:53 +0000 Subject: [PATCH 5/5] feat: add delete buttons to reports list and report detail pages - Add "Delete" button to reports list table (reports.html): clicking shows a confirmation dialog then calls the existing DELETE API. On success, the row is removed from the table without a page reload. Domain filter dropdown is pruned if the domain has no remaining reports. - Add "Delete Report" button to report detail page (report_detail.html): clicking shows a confirmation dialog then calls the DELETE API. On success, redirects to /reports. The backend DELETE endpoint and re-upload-after-delete (deduplication exemption) were already in place and covered by existing tests. Agent-Logs-Url: https://github.com/christianlouis/dmarq/sessions/d6c0078b-9338-49f0-87bd-32238fbde237 Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> --- backend/app/templates/report_detail.html | 25 ++++++++++++++ backend/app/templates/reports.html | 42 +++++++++++++++++++++--- 2 files changed, 62 insertions(+), 5 deletions(-) diff --git a/backend/app/templates/report_detail.html b/backend/app/templates/report_detail.html index c623f6a..acc6813 100644 --- a/backend/app/templates/report_detail.html +++ b/backend/app/templates/report_detail.html @@ -51,6 +51,10 @@ Back to Domain + @@ -247,6 +251,27 @@ function reportDetailApp(reportId) { } }, + async deleteReport(domain, reportId) { + if (!confirm(`Delete report "${reportId}" for domain "${domain}"?\n\nThis will remove the report from the system. You can re-import it afterwards.`)) { + return; + } + try { + const response = await fetch( + `/api/v1/reports/domain/${encodeURIComponent(domain)}/reports/${encodeURIComponent(reportId)}`, + { method: 'DELETE' } + ); + if (response.ok) { + window.location.href = '/reports'; + } else { + const data = await response.json().catch(() => ({})); + alert('Failed to delete report: ' + (data.detail || response.statusText)); + } + } catch (error) { + console.error('Error deleting report:', error); + alert('Network error — could not delete report.'); + } + }, + formatDate(timestamp) { if (!timestamp) return '—'; return new Date(timestamp * 1000).toLocaleString(); diff --git a/backend/app/templates/reports.html b/backend/app/templates/reports.html index ed0bb71..087d70d 100644 --- a/backend/app/templates/reports.html +++ b/backend/app/templates/reports.html @@ -95,11 +95,17 @@ {% endcall %} {% call td("text-right") %} - - {% call button(variant="outline", size="sm") %} - View Details - {% endcall %} - +
+ + {% call button(variant="outline", size="sm") %} + View Details + {% endcall %} + + +
{% endcall %} {% endcall %} @@ -174,6 +180,32 @@ function reportsApp() { } finally { this.loading = false; } + }, + + async deleteReport(domain, reportId) { + if (!confirm(`Delete report "${reportId}" for domain "${domain}"?\n\nThis will remove the report from the system. You can re-import it afterwards.`)) { + return; + } + try { + const response = await fetch( + `/api/v1/reports/domain/${encodeURIComponent(domain)}/reports/${encodeURIComponent(reportId)}`, + { method: 'DELETE' } + ); + if (response.ok) { + this.reports = this.reports.filter( + r => !(r.domain === domain && r.report_id === reportId) + ); + if (!this.reports.some(r => r.domain === domain)) { + this.domains = this.domains.filter(d => d !== domain); + } + } else { + const data = await response.json().catch(() => ({})); + alert('Failed to delete report: ' + (data.detail || response.statusText)); + } + } catch (error) { + console.error('Error deleting report:', error); + alert('Network error — could not delete report.'); + } } } }