feat(ci): integrate Codecov for test coverage analysis and reporting
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
+71
@@ -0,0 +1,71 @@
|
|||||||
|
# Codecov Configuration for DocuElevate
|
||||||
|
# Documentation: https://docs.codecov.com/docs/codecov-yaml
|
||||||
|
#
|
||||||
|
# SETUP INSTRUCTIONS:
|
||||||
|
# 1. Go to https://app.codecov.io and sign in with GitHub
|
||||||
|
# 2. Add the DocuElevate repository
|
||||||
|
# 3. Copy the repository upload token
|
||||||
|
# 4. Add it as a GitHub Actions secret named CODECOV_TOKEN
|
||||||
|
# (Settings → Secrets and variables → Actions → New repository secret)
|
||||||
|
# 5. Push to trigger CI — Codecov will begin reporting coverage
|
||||||
|
|
||||||
|
# ── Coverage thresholds ────────────────────────────────────────────────────
|
||||||
|
coverage:
|
||||||
|
# Minimum acceptable overall project coverage
|
||||||
|
# CI will fail if the total drops below this percentage
|
||||||
|
status:
|
||||||
|
project:
|
||||||
|
default:
|
||||||
|
# Fail if overall project coverage drops below 60%
|
||||||
|
target: 60%
|
||||||
|
# Allow up to 2% drop compared to the base branch before failing
|
||||||
|
threshold: 2%
|
||||||
|
# Branches to enforce the threshold against
|
||||||
|
branches:
|
||||||
|
- main
|
||||||
|
- develop
|
||||||
|
patch:
|
||||||
|
default:
|
||||||
|
# Fail if the lines changed in a PR are covered below 70%
|
||||||
|
target: 70%
|
||||||
|
# Allow up to 5% slack on patch coverage (newly added/changed lines)
|
||||||
|
threshold: 5%
|
||||||
|
|
||||||
|
# Lines that are never counted toward coverage (mirrors pyproject.toml)
|
||||||
|
ignore:
|
||||||
|
- "migrations/**"
|
||||||
|
- "tests/**"
|
||||||
|
- "frontend/**"
|
||||||
|
- "docs/**"
|
||||||
|
- "scripts/**"
|
||||||
|
- "**/__pycache__/**"
|
||||||
|
- "**/conftest.py"
|
||||||
|
|
||||||
|
# ── Pull-request comments ─────────────────────────────────────────────────
|
||||||
|
comment:
|
||||||
|
# Post a coverage summary comment on every PR
|
||||||
|
layout: "condensed_header, condensed_files, condensed_footer"
|
||||||
|
behavior: default # update the existing comment instead of posting a new one
|
||||||
|
require_changes: false # always post, even if coverage hasn't changed
|
||||||
|
require_base: false # post even when there is no base report to compare against
|
||||||
|
require_head: true # only post when a head report is available
|
||||||
|
hide_project_coverage: false
|
||||||
|
|
||||||
|
# ── Upload settings ───────────────────────────────────────────────────────
|
||||||
|
# Flags let you split coverage by test type and track each independently
|
||||||
|
# Each flag maps to the `flags` parameter in the codecov/codecov-action step
|
||||||
|
flag_management:
|
||||||
|
individual_flags:
|
||||||
|
- name: unittests
|
||||||
|
paths:
|
||||||
|
- app/
|
||||||
|
carryforward: true
|
||||||
|
- name: integration
|
||||||
|
paths:
|
||||||
|
- app/
|
||||||
|
carryforward: true
|
||||||
|
|
||||||
|
# ── Miscellaneous ─────────────────────────────────────────────────────────
|
||||||
|
github_checks:
|
||||||
|
# Annotate individual lines in PR diffs with coverage status
|
||||||
|
annotations: true
|
||||||
+103
-9
@@ -128,18 +128,112 @@ pytest -m integration
|
|||||||
#### Codecov - Coverage Tracking
|
#### Codecov - Coverage Tracking
|
||||||
|
|
||||||
**What it does:**
|
**What it does:**
|
||||||
- Visualizes test coverage trends
|
- Visualizes test coverage trends over time
|
||||||
- Comments on PRs with coverage changes
|
- Comments on PRs with a per-file coverage diff
|
||||||
- Tracks coverage over time
|
- Enforces project-level and patch-level coverage thresholds
|
||||||
- Provides coverage badges
|
- Provides embeddable coverage badges
|
||||||
|
- Annotates PR diff lines with coverage status
|
||||||
|
|
||||||
**Why we chose it:**
|
**Why we chose it:**
|
||||||
- Free for open-source
|
- Free for open-source projects
|
||||||
- Excellent PR integration
|
- Excellent GitHub PR integration
|
||||||
- Clear coverage visualization
|
- Clear coverage visualization and trend graphs
|
||||||
- Industry standard
|
- Industry standard with broad toolchain support
|
||||||
|
|
||||||
**Configuration:** Integrated in `.github/workflows/tests.yaml`
|
**Configuration files:**
|
||||||
|
- `codecov.yml` — repository-level settings (thresholds, flags, PR comments)
|
||||||
|
- `.github/workflows/ci.yml` (`run-tests` job) — uploads `coverage.xml` via `codecov/codecov-action@v5`
|
||||||
|
- `pyproject.toml` (`[tool.coverage.*]`) — what pytest-cov measures and omits
|
||||||
|
|
||||||
|
##### Initial Setup (Repository Admins)
|
||||||
|
|
||||||
|
1. **Sign in to Codecov** at <https://app.codecov.io> using your GitHub account.
|
||||||
|
2. **Add the repository**: click *Add new repository* and select `DocuElevate`.
|
||||||
|
3. **Copy the upload token** shown on the repository settings page.
|
||||||
|
4. **Store the token as a GitHub Actions secret**:
|
||||||
|
- Navigate to *Settings → Secrets and variables → Actions → New repository secret*
|
||||||
|
- Name: `CODECOV_TOKEN`
|
||||||
|
- Value: paste the token copied in step 3
|
||||||
|
5. **Push a commit** to trigger CI. The `run-tests` job will upload `coverage.xml` and Codecov will begin reporting.
|
||||||
|
|
||||||
|
##### Adjusting Coverage Thresholds
|
||||||
|
|
||||||
|
Edit `codecov.yml` at the repository root:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
coverage:
|
||||||
|
status:
|
||||||
|
project:
|
||||||
|
default:
|
||||||
|
target: 60% # Minimum overall project coverage
|
||||||
|
threshold: 2% # Allowed drop compared to base branch
|
||||||
|
patch:
|
||||||
|
default:
|
||||||
|
target: 70% # Minimum coverage of lines changed in a PR
|
||||||
|
threshold: 5% # Allowed slack on patch coverage
|
||||||
|
```
|
||||||
|
|
||||||
|
Commit and push the change — Codecov picks it up automatically.
|
||||||
|
|
||||||
|
##### Using Coverage Flags
|
||||||
|
|
||||||
|
Flags let you track unit and integration coverage separately.
|
||||||
|
The `run-tests` job in `ci.yml` uploads a single unified report; to split it add a
|
||||||
|
`flags` parameter to the upload step for each job:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
- name: Upload Unified Coverage to Codecov
|
||||||
|
uses: codecov/codecov-action@v5
|
||||||
|
with:
|
||||||
|
token: ${{ secrets.CODECOV_TOKEN }}
|
||||||
|
files: ./coverage.xml
|
||||||
|
flags: unittests # matches flag_management in codecov.yml
|
||||||
|
fail_ci_if_error: true
|
||||||
|
```
|
||||||
|
|
||||||
|
The flag names (`unittests`, `integration`) are defined in `codecov.yml` under
|
||||||
|
`flag_management.individual_flags`.
|
||||||
|
|
||||||
|
##### Embedding the Coverage Badge
|
||||||
|
|
||||||
|
Add the following Markdown to `README.md`, replacing `<owner>` and `<repo>`:
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
<!-- Public repository (no token needed) -->
|
||||||
|
[](https://codecov.io/gh/<owner>/<repo>)
|
||||||
|
|
||||||
|
<!-- Private repository (include token for badge access) -->
|
||||||
|
[](https://codecov.io/gh/<owner>/<repo>)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exact badge snippet (with the correct token pre-filled) is shown on the Codecov repository overview page under *Settings → Badge*.
|
||||||
|
|
||||||
|
##### Accessing Coverage Reports
|
||||||
|
|
||||||
|
| Where | What you see |
|
||||||
|
|-------|-------------|
|
||||||
|
| Codecov dashboard (<https://app.codecov.io>) | Full report, trend graphs, file explorer |
|
||||||
|
| GitHub PR comment | Per-file diff, overall Δ, patch coverage |
|
||||||
|
| GitHub Checks tab | Pass/fail status for project and patch thresholds |
|
||||||
|
| CI artifacts | `coverage.xml` (machine-readable), HTML report (human-readable) |
|
||||||
|
|
||||||
|
To view the HTML report locally:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pytest --cov=app --cov-report=html
|
||||||
|
open htmlcov/index.html
|
||||||
|
```
|
||||||
|
|
||||||
|
##### Troubleshooting
|
||||||
|
|
||||||
|
| Symptom | Likely cause | Fix |
|
||||||
|
|---------|-------------|-----|
|
||||||
|
| "Token is invalid" upload error | `CODECOV_TOKEN` secret missing or wrong | Re-copy the token from Codecov settings and update the secret |
|
||||||
|
| No PR comment posted | Repository is private and token not set | Set `CODECOV_TOKEN` — required for private repos |
|
||||||
|
| Coverage always 0% | `coverage.xml` not generated | Confirm pytest runs with `--cov=app --cov-report=xml:coverage.xml` |
|
||||||
|
| CI fails with "coverage decreased" | Patch coverage dropped below threshold | Improve test coverage for the changed lines, or adjust `threshold` in `codecov.yml` |
|
||||||
|
| Codecov not picking up `codecov.yml` | YAML syntax error | Validate the file locally with `python3 -c "import yaml; yaml.safe_load(open('codecov.yml'))"`, or use the web validator at the Codecov dashboard (*Settings → YAML*) |
|
||||||
|
| Flags not appearing in dashboard | Flag name mismatch | Ensure the `flags:` value in the upload step matches an entry in `codecov.yml` |
|
||||||
|
|
||||||
#### Pre-commit - Local Quality Gates
|
#### Pre-commit - Local Quality Gates
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user