gh-christianlouis-docuelevate/.github/copilot-instructions.md

Copilot Instructions for DocuElevate

Project Overview

DocuElevate is an intelligent document processing system that automates handling, extraction, and processing of documents. It integrates with multiple cloud storage providers (Dropbox, Google Drive, OneDrive, S3, Nextcloud) and uses AI services (OpenAI, Azure Document Intelligence) for metadata extraction and OCR.

Tech Stack

• Backend: FastAPI, SQLAlchemy, Celery, Redis
• Frontend: Jinja2 templates, Tailwind CSS
• AI/ML: OpenAI API, Azure Document Intelligence
• Auth: Authentik (OAuth2), Basic Auth
• Infrastructure: Docker, Docker Compose, Alembic (migrations)
• Testing: Pytest, pytest-asyncio, httpx

Supported Runtimes

• Python: 3.11+ (3.11 and 3.12 specified in pyproject.toml)
• Docker: Production images use python:3.14.1 / python:3.14.1-slim
• Redis: Alpine-based (redis:alpine)
• Gotenberg: gotenberg/gotenberg:latest for PDF conversion

Build Commands

# Install production dependencies
pip install -r requirements.txt

# Install development dependencies (includes linters, test tools)
pip install -r requirements-dev.txt

# Run the FastAPI development server
uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

# Run the Celery worker (requires Redis)
celery -A app.celery_worker worker -B --loglevel=info -Q document_processor,default,celery

# Docker build and run
docker compose up -d

# Database migrations
alembic upgrade head                              # Apply all migrations
alembic revision --autogenerate -m "description"  # Create new migration

Test Commands

# Run all tests with coverage (default via pyproject.toml addopts)
pytest

# Run tests by marker
pytest -m unit
pytest -m integration
pytest -m "not requires_external"

# Run a specific test file or test
pytest tests/test_api.py -v
pytest tests/test_api.py::test_function_name -v

# Coverage report
pytest --cov=app --cov-report=term-missing
pytest --cov=app --cov-report=html

Lint / Format Commands

# Format and lint with Ruff (replaces Black, isort, Flake8, Bandit — all-in-one)
ruff format app/ tests/
ruff check app/ tests/ --fix

# Type checking with mypy
mypy app/

# Check for dependency vulnerabilities
safety check

# Run all pre-commit hooks at once (recommended — runs ruff, mypy, secret detection, etc.)
pre-commit run --all-files

Agent Workflow (Follow for Every Task)

Follow these steps in order for every task — do not skip any:

1. Understand — read the issue/request in full before writing any code
2. Explore — search the codebase for existing patterns and relevant implementations
3. Plan — outline your changes as a checklist before starting
4. Implement — make the smallest correct change that solves the problem
5. Test — write or update tests; new code requires 100% test coverage
6. Document — update all relevant docs in docs/; this is mandatory, not optional
7. Quality Gate — run the single gate command below and fix every failure before committing:

ruff format app/ tests/ && \
ruff check app/ tests/ --fix && \
safety check && \
pytest --tb=short -q

8. Review — re-read your own diff; confirm it is clean, secure, minimal, and well-documented

All commands in the quality gate must exit with code 0. Never submit with failures.

Core Principles

Code Quality

• Security first: treat security as a non-negotiable requirement, not an afterthought — review SECURITY_AUDIT.md for every change
• Write clean, modern, well-documented code — prioritize readability, maintainability, and idiomatic Python
• Use Ruff for all formatting, linting, import sorting, and security scanning — ruff format + ruff check --fix (replaces Black, isort, Flake8, Bandit)
• Line length: 120 characters (configured in pyproject.toml)
• Use type hints for all function parameters and return values
• Write docstrings for all public functions, classes, and modules
• Maintain 100% test coverage for new code

Python Conventions

• Use descriptive variable names (e.g., user_document_path, not udp)
• Follow PEP 8 naming: snake_case for functions/variables, PascalCase for classes
• Use modern Python 3.10+ type hints: list[str], dict[str, Any], str | None — avoid List, Dict, Optional from typing
• Only import from typing for Any, Callable, TypeVar, Protocol, and other constructs unavailable natively
• Prefer pathlib.Path over string paths for file operations
• Use f-strings for string formatting, not .format() or %
• Handle exceptions explicitly - avoid bare except: clauses

Security Best Practices

• Never commit secrets or credentials to the repository
• Use environment variables for sensitive configuration (see .env.demo)
• Validate and sanitize all user inputs
• Use parameterized queries with SQLAlchemy (never raw SQL with user input)
• Review SECURITY_AUDIT.md before making security-related changes
• Security linting is built into Ruff via S rules — runs automatically with ruff check; fix all S-prefixed findings
• Run safety check to scan dependencies for known CVEs before submitting any PR

FastAPI Patterns

• Organize endpoints by feature in app/api/ directory
• Use dependency injection for database sessions and authentication
• Return Pydantic models from endpoints for automatic validation
• Use proper HTTP status codes (200, 201, 400, 401, 403, 404, 500)
• Document endpoints with docstrings for OpenAPI documentation
• Use async def for I/O-bound operations

Database (SQLAlchemy)

• All models are defined in app/models.py
• Use Alembic for schema migrations (create migration for any model change)
• Use declarative base for models
• Define relationships with relationship() and proper back_populates
• Use database sessions from app.database.get_db() dependency
• Always close sessions in finally blocks or use context managers

Celery Tasks

• Define tasks in app/tasks/ directory, organized by feature
• Use descriptive task names: module.action (e.g., document.process_ocr)
• Set appropriate retry policies and error handling
• Log progress and errors using Python's logging module
• Use bind=True for tasks that need access to task instance
• Keep tasks idempotent when possible

Frontend

• Templates are in frontend/templates/ using Jinja2
• Static files (CSS, JS, images) in frontend/static/
• Use Tailwind CSS utility classes (already configured)
• Keep JavaScript minimal - prefer server-side rendering
• Follow existing template structure and patterns

Internationalization (i18n) & Localization (l10n)

• Always use the _("key") helper in Jinja2 templates and translate("key", locale) in Python for every user-visible string — never hardcode UI text.
• Only add new keys to frontend/translations/en.json — that is the one and only file you must touch when introducing new UI strings.
• Do not manually edit any non-English translation file (de.json, fr.json, etc.). An external automation script syncs all other language files from en.json automatically.
• Key naming convention: <section>.<descriptor> in snake_case, e.g. language.search_placeholder, nav.help, common.cancel.
• The test_all_languages_have_same_keys check has been intentionally removed — key completeness across locales is enforced by the external sync script, not by the test suite.

Testing

• Write tests in tests/ directory, mirroring app/ structure
• Use pytest markers: @pytest.mark.unit, @pytest.mark.integration, etc.
• Mock external services (OpenAI, Azure, cloud storage) in tests
• Use pytest.fixture for test setup and teardown
• Run tests with: pytest -v
• Check coverage with: pytest --cov=app --cov-report=term-missing
• All tests must pass before submitting changes — never leave failing tests
• All linters must pass before submitting — run pre-commit run --all-files

Configuration

• All configuration is in app/config.py using Pydantic Settings
• Use environment variables for configuration (12-factor app)
• Provide sensible defaults when possible
• Document all configuration options in docs/ConfigurationGuide.md

Documentation

• Keep documentation in docs/ directory in Markdown format
• Always update relevant docs when adding or changing any feature — documentation updates are mandatory, never optional
• User-facing documentation should be clear and include examples
• Reference existing docs: docs/UserGuide.md, docs/API.md, docs/DeploymentGuide.md
• See AGENTIC_CODING.md for detailed development guide

Error Handling

• Use custom exceptions defined in application (follow existing patterns)
• Log errors with context using Python's logging module
• Return user-friendly error messages in API responses
• Include error details in development, sanitize in production
• In API endpoints, raise HTTPException with appropriate status codes (400, 401, 403, 404, 500)
• In Celery tasks, use self.retry(exc=e, countdown=60) for transient errors; log and return error dict for permanent errors
• Never use bare except: — always catch specific exception types
• Wrap database operations in try/except with db.rollback() in the except block

Logging Conventions

• Use Python's built-in logging module: import logging; logger = logging.getLogger(__name__)
• Log levels:
  • logger.debug() — detailed diagnostic information
  • logger.info() — general operational events (document processed, task started)
  • logger.warning() — recoverable issues (retrying, fallback used)
  • logger.error() — errors that need attention (failed operations)
  • logger.critical() — system-level failures requiring immediate action
• Always include context in log messages: logger.info(f"Processing document: {file_id}, user: {user_id}")
• Never log sensitive data: passwords, tokens, API keys, personal information
• Use f-strings in log messages (consistent with project style)

Architectural Boundaries

• app/api/ — REST API endpoints only; organize by feature
• app/tasks/ — Celery background tasks only; keep idempotent
• app/views/ — UI routes serving Jinja2 templates
• app/utils/ — Shared utility functions and helpers
• app/routes/ — Deprecated; being migrated to app/api/ — do not add new code here
• app/models.py — All SQLAlchemy models (single file)
• app/config.py — All configuration via Pydantic Settings (single file)
• app/database.py — Database engine and session setup (single file)
• app/auth.py — Authentication logic (single file)
• frontend/templates/ — Jinja2 templates; do not mix backend logic
• frontend/static/ — CSS, JS, images; keep JavaScript minimal
• tests/ — Test files mirroring app/ structure
• migrations/ — Alembic migration scripts; always auto-generate with alembic revision --autogenerate

Don't Change Rules

These files and directories are managed by automation or are critical infrastructure — do not manually edit:

• VERSION — Managed by python-semantic-release; updated automatically on merge to main
• CHANGELOG.md — Auto-generated from conventional commit messages by semantic-release
• migrations/ — Do not manually edit existing migration files; only create new ones via alembic revision --autogenerate
• Git tags and GitHub Releases — Created automatically by semantic-release; never create manually
• .pre-commit-config.yaml — Only change if adding/updating linting tools; do not remove existing hooks
• pyproject.toml [tool.semantic_release] — Release configuration; do not modify without explicit approval
• docker-compose.yaml service names — External systems depend on api, worker, redis, gotenberg names

Dependencies

• Add new dependencies to requirements.txt (production) or requirements-dev.txt (development)
• Document any new dependencies and their licenses in README.md
• Check for security vulnerabilities with safety check
• Pin major versions, allow minor updates (e.g., fastapi>=0.100.0,<1.0.0)

Git Workflow

• Write clear, descriptive commit messages
• ALWAYS follow Conventional Commits format (see below)
• Keep commits focused and atomic
• All tests must pass before committing — pytest must succeed with no failures
• All linters must pass before committing — pre-commit run --all-files must succeed
• Pre-commit hooks are configured (.pre-commit-config.yaml)

Conventional Commits (REQUIRED)

All commit messages MUST follow the Conventional Commits specification.

Format

<type>(<scope>): <subject>

<body>

<footer>

Commit Types and Version Impact

• feat: New feature → minor version bump (0.5.0 → 0.6.0)
• fix: Bug fix → patch version bump (0.5.0 → 0.5.1)
• perf: Performance improvement → patch version bump
• docs: Documentation only → no version bump
• style: Formatting changes → no version bump
• refactor: Code refactoring → no version bump
• test: Test changes → no version bump
• build: Build system changes → no version bump
• ci: CI/CD changes → no version bump
• chore: Other changes → no version bump

Breaking Changes

For breaking changes (major version bump), add ! after type or include BREAKING CHANGE: in footer:

feat(api)!: redesign authentication endpoints

BREAKING CHANGE: OAuth2 tokens now required instead of API keys.

Result: 0.5.0 → 1.0.0

Scope Examples

• api - REST API changes
• ui - Frontend changes
• auth - Authentication
• storage - Storage providers
• ocr - OCR processing
• tasks - Celery tasks
• config - Configuration
• docs - Documentation

Commit Examples

feat(storage): add Amazon S3 storage provider
fix(ocr): handle PDFs without text layer
docs: update deployment guide with Docker setup
refactor(tasks): consolidate duplicate code
test: add integration tests for upload API
chore: update dependencies for security fixes

Semantic Release Process

Automated Versioning

DocuElevate uses python-semantic-release for automated version management:

1. On merge to main: semantic-release analyzes commit messages
2. Automatic actions:
   • Determines next version from commit types
   • Updates VERSION file
   • Generates/updates CHANGELOG.md
   • Creates Git tag with v prefix (e.g., v0.6.0)
   • Creates GitHub Release with auto-generated notes
   • Triggers Docker image builds with version tag

Agent Rules for Versioning

• ✅ DO: Write conventional commit messages
• ✅ DO: Use appropriate commit types for your changes
• ✅ DO: Mark breaking changes explicitly
• ❌ DON'T: Manually edit VERSION file
• ❌ DON'T: Manually edit CHANGELOG.md
• ❌ DON'T: Create version tags or GitHub Releases manually

These files are managed entirely by the semantic-release automation.

File Organization

• Place API endpoints in app/api/ organized by feature
• Background tasks go in app/tasks/
• Utility functions in app/utils/
• UI routes in app/views/
• Database models in app/models.py
• Configuration in app/config.py

Common Patterns

• Use modern Python 3.10+ type hints: list[str], dict[str, Any], str | None; only import from typing for Any, Callable, TypeVar, Protocol
• Import FastAPI dependencies: from fastapi import Depends, HTTPException, status
• Get DB session: db: Session = Depends(get_db)
• Current user: current_user: User = Depends(get_current_user)
• Logger: import logging; logger = logging.getLogger(__name__)

Resources

• AGENTIC_CODING.md - Comprehensive development guide
• CONTRIBUTING.md - Contribution guidelines
• SECURITY_AUDIT.md - Security considerations
• README.md - Project overview and quickstart