docs(copilot): overhaul instructions — Ruff toolchain, 100% coverage, agent workflow, modern Python

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>

.github/copilot-instructions.md

@@ -66,36 +66,50 @@ pytest --cov=app --cov-report=html
 ## Lint / Format Commands
 
 ```bash
-# Format code with Black (line length 120)
-black app/ tests/
-
-# Sort imports with isort (Black-compatible profile)
-isort app/ tests/
-
-# Lint with flake8 (max line length 120, ignores E203/W503)
-flake8 app/ --max-line-length=120
+# Format and lint with Ruff (replaces Black, isort, Flake8, Bandit — all-in-one)
+ruff format app/ tests/
+ruff check app/ tests/ --fix
 
 # Type checking with mypy
 mypy app/
 
-# Security lint with bandit (excludes tests)
-bandit -r app/
-
 # Check for dependency vulnerabilities
 safety check
 
-# Run all pre-commit hooks at once
+# Run all pre-commit hooks at once (recommended — runs ruff, mypy, secret detection, etc.)
 pre-commit run --all-files
 ```
 
+## Agent Workflow (Follow for Every Task)
+
+Follow these steps **in order** for every task — do not skip any:
+
+1. **Understand** — read the issue/request in full before writing any code
+2. **Explore** — search the codebase for existing patterns and relevant implementations
+3. **Plan** — outline your changes as a checklist before starting
+4. **Implement** — make the smallest correct change that solves the problem
+5. **Test** — write or update tests; new code requires 100% test coverage
+6. **Document** — update all relevant docs in `docs/`; this is mandatory, not optional
+7. **Quality Gate** — run the single gate command below and fix every failure before committing:
+
+```bash
+ruff format app/ tests/ && \
+ruff check app/ tests/ --fix && \
+safety check && \
+pytest --tb=short -q
+```
+
+8. **Review** — re-read your own diff; confirm it is clean, secure, minimal, and well-documented
+
+> All commands in the quality gate must exit with code 0. Never submit with failures.
+
 ## Core Principles
 
 ### Code Quality
 - **Security first**: treat security as a non-negotiable requirement, not an afterthought — review [SECURITY_AUDIT.md](../SECURITY_AUDIT.md) for every change
 - Write **clean, modern, well-documented code** — prioritize readability, maintainability, and idiomatic Python
-- Always use **Black** for formatting (line length: 120)
-- Use **isort** with Black profile for import sorting
-- Use **flake8** for linting (ignore E203, W503)
+- Use **Ruff** for all formatting, linting, import sorting, and security scanning — `ruff format` + `ruff check --fix` (replaces Black, isort, Flake8, Bandit)
+- Line length: 120 characters (configured in `pyproject.toml`)
 - Use **type hints** for all function parameters and return values
 - Write **docstrings** for all public functions, classes, and modules
 - Maintain **100% test coverage** for new code
@@ -103,7 +117,8 @@ pre-commit run --all-files
 ### Python Conventions
 - Use descriptive variable names (e.g., `user_document_path`, not `udp`)
 - Follow PEP 8 naming: `snake_case` for functions/variables, `PascalCase` for classes
-- Use type hints from `typing` module (Dict, List, Optional, etc.)
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None` — avoid `List`, `Dict`, `Optional` from `typing`
+- Only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol`, and other constructs unavailable natively
 - Prefer `pathlib.Path` over string paths for file operations
 - Use f-strings for string formatting, not `.format()` or `%`
 - Handle exceptions explicitly - avoid bare `except:` clauses
@@ -114,7 +129,8 @@ pre-commit run --all-files
 - Validate and sanitize all user inputs
 - Use parameterized queries with SQLAlchemy (never raw SQL with user input)
 - Review [SECURITY_AUDIT.md](../SECURITY_AUDIT.md) before making security-related changes
-- Run `bandit` to check for security issues in Python code
+- Security linting is built into Ruff via `S` rules — runs automatically with `ruff check`; fix all `S`-prefixed findings
+- Run `safety check` to scan dependencies for known CVEs before submitting any PR
 
 ### FastAPI Patterns
 - Organize endpoints by feature in `app/api/` directory
@@ -318,7 +334,7 @@ These files are managed entirely by the semantic-release automation.
 - Configuration in `app/config.py`
 
 ### Common Patterns
-- Use `from typing import Optional, Dict, List, Any` for type hints
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None`; only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol`
 - Import FastAPI dependencies: `from fastapi import Depends, HTTPException, status`
 - Get DB session: `db: Session = Depends(get_db)`
 - Current user: `current_user: User = Depends(get_current_user)`

.github/instructions/documentation.instructions.md

@@ -230,12 +230,15 @@ For API details, refer to the [API Documentation](./API.md).
 ```
 
 ## Updating Documentation
+Documentation updates are **mandatory** — every PR that changes code must include matching documentation updates in the same PR. There are no exceptions.
+
 When making code changes:
-1. Update relevant documentation in the same PR
-2. Check for outdated information
+1. **Update relevant documentation** in the same PR — never defer docs to a follow-up
+2. Check for outdated information in existing docs
 3. Add new sections for new features
 4. Update examples if behavior changes
 5. Review related documentation for consistency
+6. Update `docs/ConfigurationGuide.md` and `.env.demo` for any new or changed configuration options
 
 ## Screenshots and Diagrams
 - Use clear, high-quality images

.github/instructions/python-backend.instructions.md

@@ -7,16 +7,18 @@ applyTo: "app/**/*.py"
 These instructions apply to all Python code in the `app/` directory.
 
 ## Code Style
-- Use **Ruff** for linting and formatting with 120 character line length
+- Use **Ruff** for all formatting, linting, import sorting, and security scanning — `ruff format app/ tests/ && ruff check app/ tests/ --fix`
+- Line length: 120 characters (configured in `pyproject.toml` `[tool.ruff]`)
 - All functions must have type hints for parameters and return values
-- Use `from typing import Optional, Dict, List, Any, Union` as needed
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None`
+- Only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol` (not `Dict`, `List`, `Optional`, `Union`)
 
-## Import Order (Ruff enforces isort-compatible ordering)
+## Import Order (enforced by Ruff `I` rules)
 ```python
 # Standard library imports
 import os
 from pathlib import Path
-from typing import Optional, Dict, List
+from typing import Any  # Only for Any, Callable, TypeVar, Protocol
 
 # Third-party imports
 from fastapi import APIRouter, Depends, HTTPException
@@ -33,7 +35,7 @@ from app.models import Document, User
 def process_document(
     file_path: Path,
     user_id: int,
-    metadata: Optional[Dict[str, Any]] = None
+    metadata: dict[str, Any] | None = None
 ) -> DocumentMetadata:
     """
     Process a document and extract metadata.
@@ -127,7 +129,7 @@ import logging
 logger = logging.getLogger(__name__)
 
 @shared_task(bind=True, max_retries=3)
-def process_ocr(self, document_id: int) -> Dict[str, Any]:
+def process_ocr(self, document_id: int) -> dict[str, Any]:
     """Process OCR for a document."""
     try:
         # Processing logic
@@ -153,10 +155,11 @@ class Settings(BaseSettings):
         env_file = ".env"
 ```
 
-## Security
-- Never commit secrets
-- Validate all user inputs
-- Use parameterized queries
-- Sanitize file paths
-- Check file permissions
-- Review SECURITY_AUDIT.md for guidelines
+## Security (First and Foremost)
+- **Security first**: treat every change as a potential attack surface — review `SECURITY_AUDIT.md` before making any security-related change
+- Never commit secrets, tokens, or credentials
+- Validate and sanitize all user inputs
+- Use parameterized queries — never raw SQL with user data
+- Sanitize file paths; check file permissions before access
+- Security linting is built into Ruff via `S` rules — fix all `S`-prefixed findings before committing
+- Run `safety check` before submitting any PR to catch dependency CVEs

.github/instructions/testing.instructions.md

@@ -174,7 +174,8 @@ def test_create_document(db_session):
 ```
 
 ## Test Coverage Goals
-- Aim for **80% code coverage** for all new code
+- Achieve **100% test coverage** for all new code — use `# pragma: no cover` only for genuinely unreachable or platform-specific branches, with an inline comment explaining why
+- Enforce the threshold: `pytest --cov=app --cov-fail-under=100`
 - Focus on critical paths and error handling
 - Test both success and failure scenarios
 - Don't test third-party library code