gh-christianlouis-dmarq/docs/development/ci-cd.md

CI/CD Pipeline

DMARQ uses GitHub Actions for all continuous integration and delivery tasks. The pipeline is defined in two workflow files:

File	Purpose	Triggers
.github/workflows/ci.yml	Lint → Test → Security → Docker → GitOps	Push to main/develop, pull requests, weekly schedule
.github/workflows/release.yml	Semantic versioning & changelog	Push to main

Use the Release Checklist before merging production changes and after the deployment updates.

---

Pipeline Stages

Stage 1 — Lint (blocking gate)

All subsequent jobs depend on this stage succeeding.

Tool	What it checks
Black	Code formatting (line length 100, target Python 3.13)
isort	Import ordering (Black-compatible profile)
Flake8	Style and complexity (max-complexity=10, E203/W503/E501 ignored)
Pylint	Deeper static analysis (continue-on-error — advisory only)

The lint job auto-formats with Black and isort before running the --check step, so a failing lint job is usually caused by a Flake8 or Pylint issue.

Stage 2 — Parallel quality gates

These jobs run in parallel once lint passes.

Test

cd backend
pytest --cov=app --cov-report=xml --cov-report=term-missing

Coverage results are uploaded to Codecov.

Security Scan

• Bandit — Python security linter; the JSON report is uploaded as a workflow artifact (bandit-security-report) for every run.
• pip-audit — checks all packages in backend/requirements.txt against known CVE databases.

Both steps use continue-on-error: true so they are advisory; a finding will not block the build but will appear in the workflow summary.

CodeQL Analysis

GitHub's semantic code analysis scans the Python source for known vulnerability patterns (security and quality queries). Runs on push and PR events only — skipped on the weekly scheduled scan.

Dependency Review

Runs on pull requests only. Fails if any new dependency introduces a vulnerability of moderate severity or higher.

Stage 3 — Docker Build & Publish

Runs only on pushes to main after both Test and Security pass.

• Builds from ./backend/Dockerfile
• Pushes to ghcr.io/<owner>/dmarq with three tags:
  • latest (default branch only)
  • branch name (e.g. main)
  • short commit SHA (e.g. a1b2c3d)
• Layer cache is stored via GitHub Actions cache (type=gha).

Stage 4 — GitOps (Update K8s Manifest)

Runs only on pushes to main after the Docker stage succeeds.

Updates the image tag in the preprod Kubernetes manifest at apps/dmarq/preprod/dmarq-stack.yaml in the christianlouis/k8s-cluster-state repository.

!!! note "Optional" This stage requires a GH_PAT repository secret with write access to the k8s-cluster-state repo. If the secret is absent or lacks access the step emits a warning and skips gracefully — it will never fail the pipeline.

---

Release Workflow

release.yml uses python-semantic-release to automate versioning from Conventional Commits:

• Bumps version in pyproject.toml
• Generates/updates CHANGELOG.md
• Creates a Git tag and a GitHub Release

Commit prefixes that trigger a release:

Prefix	Version bump
fix:	Patch (0.0.x)
feat:	Minor (0.x.0)
feat!: / BREAKING CHANGE	Major (x.0.0)

---

Scheduled Security Scan

A cron job runs every Monday at 00:00 UTC and executes the Lint, Security Scan, and Test stages against the latest main code. CodeQL and Dependency Review are skipped on schedule events.

---

Troubleshooting

Lint failure

1. Pull the latest branch and run locally:

   black backend/app
   isort backend/app
   flake8 backend/app
   

2. Commit the formatted files and push.

Pylint failures are advisory (continue-on-error: true) and will not block the pipeline, but should still be investigated.

Test failure

1. Reproduce locally:

   cd backend
   pytest --cov=app --cov-report=term-missing -x
   

2. Check the test output in the Test job log for the failing assertion and stack trace.
3. The coverage XML is not uploaded as an artifact — run locally to inspect htmlcov/index.html.

Security scan failure

• Bandit: Download the bandit-security-report artifact from the workflow run's Artifacts panel and review bandit-report.json.
• pip-audit: The finding is printed directly to the job log. Update or pin the affected dependency in backend/requirements.txt.

Docker build failure

Common causes:

Symptom	Fix
pip install error	Check backend/requirements.txt for unpinned or incompatible versions
COPY file not found	Ensure the file exists and is not listed in .dockerignore
Registry auth failure	Verify the workflow has packages: write permission

Dependency Review failure (PR only)

The PR introduced a dependency with a known vulnerability. Either:

• Remove the dependency, or
• Upgrade to a patched version, or
• If the finding is a false positive, discuss with a maintainer — the severity threshold (moderate) can be adjusted in the workflow.

Release workflow not triggering

Ensure your merge commit message follows Conventional Commits. The release workflow only creates a new version when python-semantic-release detects a fix:, feat:, or breaking-change commit since the last tag.