gh-christianlouis-docuelevate/docs/CIToolsGuide.md

# CI/CD Tools Guide

This document provides a comprehensive overview of the CI/CD tools used in DocuElevate, including the rationale for tool selection and the de-duplication strategy that was applied.

## Overview

DocuElevate uses a **streamlined, non-redundant set of CI/CD tools** to ensure code quality, security, and reliability without bloat. The tool selection prioritizes:

1. **No duplication**: Each tool serves a unique purpose
2. **Performance**: Fast feedback in CI runs
3. **Developer experience**: Clear, actionable feedback
4. **Modern tooling**: Active development and support
5. **Cost-effectiveness**: Preference for free/open-source tools

## Current Tool Stack

### Core CI Tools

| Tool | Purpose | Frequency | Status |
|------|---------|-----------|--------|
| **Ruff** | Linting, formatting, security (Python) | Every push/PR | ✅ Active |
| **Mypy** | Static type checking (Python) | Every push/PR | ✅ Active |
| **pytest** | Unit and integration testing | Every push/PR | ✅ Active |
| **CodeQL** | Advanced security scanning | Push to main, PR, weekly | ✅ Active |
| **Codecov** | Coverage tracking and reporting | Every push/PR | ✅ Active |
| **Pre-commit** | Local checks before commit | Pre-commit hook | ✅ Active |
| **Dependabot** | Dependency security updates | Daily | ✅ Active |

### Tool Details

#### Ruff - All-in-One Python Linter

**What it does:**
- PEP 8 style checking (replaces Flake8)
- Code formatting (replaces Black)
- Import sorting (replaces isort)
- Security vulnerability detection (replaces Bandit)
- Code quality checks (replaces parts of Pylint)

**Why we chose it:**
- 10-100x faster than traditional tools
- Single configuration file (`pyproject.toml`)
- Written in Rust, actively maintained
- Auto-fix capability for most issues
- Comprehensive rule set (1000+ rules)

**Configuration:** `pyproject.toml` → `[tool.ruff]`

**Commands:**
```bash
# Lint code
ruff check app/ tests/

# Auto-fix issues
ruff check app/ tests/ --fix

# Format code
ruff format app/ tests/
```

#### Mypy - Type Checking

**What it does:**
- Static type analysis for Python code
- Catches type-related bugs before runtime
- Enforces type hint usage

**Why we chose it:**
- Industry standard for Python type checking
- Unique value - no other tool provides this
- Excellent IDE integration
- Configurable strictness levels

**Configuration:** `pyproject.toml` → `[tool.mypy]`

**Commands:**
```bash
mypy app/
```

#### pytest - Testing Framework

**What it does:**
- Runs unit and integration tests
- Generates coverage reports
- Provides test result reporting

**Why we chose it:**
- Modern Python testing standard
- Rich plugin ecosystem
- Excellent fixture support
- Built-in parameterization

**Configuration:** `pyproject.toml` → `[tool.pytest.ini_options]`

**Commands:**
```bash
# Run all tests
pytest tests/ -v

# Run with coverage
pytest tests/ -v --cov=app --cov-report=term-missing

# Run specific test markers
pytest -m unit
pytest -m integration
```

#### CodeQL - Advanced Security Scanning

**What it does:**
- Deep semantic code analysis
- Detects security vulnerabilities
- Finds complex code quality issues
- Scans Python, JavaScript, GitHub Actions

**Why we chose it:**
- Free for open-source projects
- Native GitHub integration
- Enterprise-grade scanning
- GitHub Advanced Security features
- Results appear in Security tab

**Configuration:** `.github/workflows/codeql.yml`

**Frequency:** Push to main, PRs, weekly scheduled scan

#### Codecov - Coverage Tracking

**What it does:**
- Visualizes test coverage trends over time
- Comments on PRs with a per-file coverage diff
- Enforces project-level and patch-level coverage thresholds
- Provides embeddable coverage badges
- Annotates PR diff lines with coverage status

**Why we chose it:**
- Free for open-source projects
- Excellent GitHub PR integration
- Clear coverage visualization and trend graphs
- Industry standard with broad toolchain support

**Configuration files:**
- `codecov.yml` — repository-level settings (thresholds, flags, PR comments)
- `.github/workflows/ci.yml` (`run-tests` job) — uploads `coverage.xml` via `codecov/codecov-action@v5`
- `pyproject.toml` (`[tool.coverage.*]`) — what pytest-cov measures and omits

##### Initial Setup (Repository Admins)

1. **Sign in to Codecov** at <https://app.codecov.io> using your GitHub account.
2. **Add the repository**: click *Add new repository* and select `DocuElevate`.
3. **Copy the upload token** shown on the repository settings page.
4. **Store the token as a GitHub Actions secret**:
   - Navigate to *Settings → Secrets and variables → Actions → New repository secret*
   - Name: `CODECOV_TOKEN`
   - Value: paste the token copied in step 3
5. **Push a commit** to trigger CI. The `run-tests` job will upload `coverage.xml` and Codecov will begin reporting.

##### Adjusting Coverage Thresholds

Edit `codecov.yml` at the repository root:

```yaml
coverage:
  status:
    project:
      default:
        target: 60%      # Minimum overall project coverage
        threshold: 2%    # Allowed drop compared to base branch
    patch:
      default:
        target: 70%      # Minimum coverage of lines changed in a PR
        threshold: 5%    # Allowed slack on patch coverage
```

Commit and push the change — Codecov picks it up automatically.

##### Using Coverage Flags

Flags let you track unit and integration coverage separately.
The `run-tests` job in `ci.yml` uploads a single unified report; to split it add a
`flags` parameter to the upload step for each job:

```yaml
- name: Upload Unified Coverage to Codecov
  uses: codecov/codecov-action@v5
  with:
    token: ${{ secrets.CODECOV_TOKEN }}
    files: ./coverage.xml
    flags: unittests       # matches flag_management in codecov.yml
    fail_ci_if_error: true
```

The flag names (`unittests`, `integration`) are defined in `codecov.yml` under
`flag_management.individual_flags`.

##### Embedding the Coverage Badge

Add the following Markdown to `README.md`, replacing `<owner>` and `<repo>`:

```markdown
<!-- Public repository (no token needed) -->
[![codecov](https://codecov.io/gh/<owner>/<repo>/graph/badge.svg)](https://codecov.io/gh/<owner>/<repo>)

<!-- Private repository (include token for badge access) -->
[![codecov](https://codecov.io/gh/<owner>/<repo>/graph/badge.svg?token=<TOKEN>)](https://codecov.io/gh/<owner>/<repo>)
```

The exact badge snippet (with the correct token pre-filled) is shown on the Codecov repository overview page under *Settings → Badge*.

##### Accessing Coverage Reports

| Where | What you see |
|-------|-------------|
| Codecov dashboard (<https://app.codecov.io>) | Full report, trend graphs, file explorer |
| GitHub PR comment | Per-file diff, overall Δ, patch coverage |
| GitHub Checks tab | Pass/fail status for project and patch thresholds |
| CI artifacts | `coverage.xml` (machine-readable), HTML report (human-readable) |

To view the HTML report locally:

```bash
pytest --cov=app --cov-report=html
open htmlcov/index.html
```

##### Troubleshooting

| Symptom | Likely cause | Fix |
|---------|-------------|-----|
| "Token is invalid" upload error | `CODECOV_TOKEN` secret missing or wrong | Re-copy the token from Codecov settings and update the secret |
| No PR comment posted | Repository is private and token not set | Set `CODECOV_TOKEN` — required for private repos |
| Coverage always 0% | `coverage.xml` not generated | Confirm pytest runs with `--cov=app --cov-report=xml:coverage.xml` |
| CI fails with "coverage decreased" | Patch coverage dropped below threshold | Improve test coverage for the changed lines, or adjust `threshold` in `codecov.yml` |
| Codecov not picking up `codecov.yml` | YAML syntax error | Validate the file locally with `python3 -c "import yaml; yaml.safe_load(open('codecov.yml'))"`, or use the web validator at the Codecov dashboard (*Settings → YAML*) |
| Flags not appearing in dashboard | Flag name mismatch | Ensure the `flags:` value in the upload step matches an entry in `codecov.yml` |

#### Pre-commit - Local Quality Gates

**What it does:**
- Runs checks before git commit
- Prevents committing bad code
- Enforces conventional commits
- Detects secrets and security issues

**Why we chose it:**
- Catches issues before CI
- Fast local feedback
- Configurable hook selection
- Wide ecosystem support

**Configuration:** `.pre-commit-config.yaml`

**Hooks included:**
- Ruff linting and formatting
- Mypy type checking
- Secret detection (detect-secrets)
- Conventional commit validation
- YAML/JSON validation
- Trailing whitespace removal
- Large file detection

**Commands:**
```bash
# Install hooks
pre-commit install

# Run on all files
pre-commit run --all-files

# Run on staged files
pre-commit run
```

## De-duplication Strategy

### Removed Tools

#### DeepSource ❌ Removed

**Reason for removal:** Redundant with Ruff + CodeQL

DeepSource provided:
- Static analysis → Now covered by Ruff
- Security scanning → Now covered by Ruff + CodeQL
- Code quality metrics → Now covered by Ruff

**Why it was redundant:**
- Overlapped 90% with Ruff's capabilities
- CodeQL provides superior security scanning
- No unique value proposition
- Added CI complexity without benefit

**Action taken:** Removed `.deepsource.toml` configuration file

### Tools Not Configured (No Action Needed)

#### SonarQube ❌ Not configured

**Status:** No configuration found in repository

**Analysis:** 
- Enterprise-focused tool
- Best for large organizations needing quality gates and dashboards
- Would duplicate Ruff + CodeQL capabilities
- Not needed for this project's scale

#### Snyk ❌ Not configured

**Status:** No configuration found in repository

**Analysis:**
- Would duplicate CodeQL for security scanning
- Would duplicate Dependabot + pip-audit for dependency vulnerabilities
- Current tools provide adequate coverage
- Not needed at this time

### Tool Overlap Analysis (Before De-duplication)

| Capability | Old Setup | New Setup | Status |
|------------|-----------|-----------|--------|
| PEP 8 Style | Flake8, Pylint | Ruff | ✅ Consolidated |
| Code Formatting | Black | Ruff | ✅ Consolidated |
| Import Sorting | isort | Ruff | ✅ Consolidated |
| Security Linting | Bandit | Ruff | ✅ Consolidated |
| Code Quality | Pylint, DeepSource | Ruff | ✅ Consolidated |
| Security Scanning | CodeQL, DeepSource | CodeQL | ✅ De-duplicated |
| Dependency Vulnerabilities | None | pip-audit | ✅ Added |
| Type Checking | Mypy | Mypy | ✅ Kept (unique) |
| Testing | pytest | pytest | ✅ Kept (unique) |
| Coverage | Codecov | Codecov | ✅ Kept (unique) |

## Workflow Structure

### Tests & Linting Workflow (`.github/workflows/tests.yaml`)

Runs on every push and pull request.

**Jobs (run in parallel):**

1. **test**
   - Runs pytest with coverage
   - Uploads results to Codecov
   - Provides artifacts (junit.xml, coverage.xml)

2. **lint**
   - Runs Ruff check (linting)
   - Runs Ruff format (formatting validation)

3. **mypy**
   - Runs type checking

**Result:** All three jobs complete independently, providing comprehensive feedback even if one fails.

### CodeQL Workflow (`.github/workflows/codeql.yml`)

Runs on:
- Push to `main` branch
- Pull requests to `main`
- Weekly schedule (Mondays at 1:37 AM UTC)

**Languages scanned:**
- Python
- JavaScript/TypeScript
- GitHub Actions

### Release Workflow (`.github/workflows/release.yml`)

Runs on push to `main` branch.

**Actions:**
- Generates version based on conventional commits
- Updates CHANGELOG.md
- Creates Git tags
- Triggers Docker builds

### Docker Build Workflows

- `docker-ci.yml` - Builds and pushes Docker images on main branch
- `docker-build.yaml` - Builds on tags and branches

## Best Practices

### For Contributors

1. **Install pre-commit hooks** (recommended):
   ```bash
   pip install pre-commit
   pre-commit install
   ```

2. **Run checks locally before pushing**:
   ```bash
   # Quick check
   ruff check app/ tests/
   mypy app/
   pytest tests/ -m unit

   # Full check (what CI runs)
   ruff check app/ tests/
   ruff format --check app/ tests/
   mypy app/
   pytest tests/ -v --cov=app -m "not e2e"
   ```

3. **Use Ruff auto-fix** to resolve most issues automatically:
   ```bash
   ruff check app/ tests/ --fix
   ruff format app/ tests/
   ```

4. **Follow conventional commits** (enforced by pre-commit):
   - `feat:` for new features
   - `fix:` for bug fixes
   - `docs:` for documentation
   - `refactor:`, `test:`, `chore:`, etc.

### For Maintainers

1. **Review CodeQL security alerts** in the Security tab regularly
2. **Monitor Codecov reports** to ensure coverage doesn't drop
3. **Update dependencies** via Dependabot PRs promptly
4. **Review CI failures** for patterns indicating needed tool configuration changes

## Performance Metrics

### CI Run Time (Typical)

| Job | Duration | Status |
|-----|----------|--------|
| test | ~2-3 min | ✅ Fast |
| lint (Ruff) | ~10-15 sec | ✅ Very Fast |
| mypy | ~30-45 sec | ✅ Fast |
| **Total (parallel)** | **~2-3 min** | ✅ Fast |

### Before De-duplication
- Total jobs: 6+ (Flake8, Black, isort, Pylint, Bandit, tests, Mypy)
- Total run time: ~5-7 minutes
- Tool overlap: High

### After De-duplication
- Total jobs: 3 (Ruff, Mypy, tests)
- Total run time: ~2-3 minutes
- Tool overlap: None

**Improvement:** 40-50% faster CI runs with zero functionality loss.

## Future Considerations

### Potential Additions (Only if Needed)

1. **Performance monitoring** (if performance becomes an issue)
   - Tool: Lighthouse CI for frontend
   - Tool: py-spy for Python profiling

2. **End-to-end testing** (if integration testing is insufficient)
   - Tool: Playwright or Selenium
   - Currently handled by pytest with testcontainers locally

3. **Dependency license scanning** (if needed for compliance)
   - Tool: licensee or similar

### Tools to Avoid (Redundant)

- ❌ SonarQube (overlaps with Ruff + CodeQL)
- ❌ Snyk (overlaps with CodeQL + Dependabot)
- ❌ Additional Python linters (Ruff is comprehensive)
- ❌ Additional formatters (Ruff format is sufficient)

## Related Documentation

- [CI Workflow Guide](CIWorkflow.md) - Detailed workflow documentation
- [Contributing Guide](../CONTRIBUTING.md) - Contribution guidelines including testing
- [AGENTIC_CODING.md](../AGENTIC_CODING.md) - Development guide for AI agents
- [pyproject.toml](../pyproject.toml) - Tool configurations

## Summary

DocuElevate's CI/CD pipeline is designed to be:
- **Lean**: No redundant tools
- **Fast**: Parallel execution, fast tools (Ruff)
- **Comprehensive**: Security, quality, testing all covered
- **Developer-friendly**: Clear feedback, auto-fix capabilities
- **Maintainable**: Single configuration file, modern tools

The de-duplication effort removed DeepSource and consolidated 6 separate linting tools into Ruff, resulting in faster CI runs without sacrificing code quality or security coverage.