Merge branch 'main' into copilot/add-self-hosted-ocr-support

.github/copilot-instructions.md

@@ -66,42 +66,59 @@ pytest --cov=app --cov-report=html
 ## Lint / Format Commands
 
 ```bash
-# Format code with Black (line length 120)
+# Format and lint with Ruff (replaces Black, isort, Flake8, Bandit — all-in-one)
-black app/ tests/
+ruff format app/ tests/
-
+ruff check app/ tests/ --fix
-# Sort imports with isort (Black-compatible profile)
-isort app/ tests/
-
-# Lint with flake8 (max line length 120, ignores E203/W503)
-flake8 app/ --max-line-length=120
 
 # Type checking with mypy
 mypy app/
 
-# Security lint with bandit (excludes tests)
-bandit -r app/
-
 # Check for dependency vulnerabilities
 safety check
 
-# Run all pre-commit hooks at once
+# Run all pre-commit hooks at once (recommended — runs ruff, mypy, secret detection, etc.)
 pre-commit run --all-files
 ```
 
+## Agent Workflow (Follow for Every Task)
+
+Follow these steps **in order** for every task — do not skip any:
+
+1. **Understand** — read the issue/request in full before writing any code
+2. **Explore** — search the codebase for existing patterns and relevant implementations
+3. **Plan** — outline your changes as a checklist before starting
+4. **Implement** — make the smallest correct change that solves the problem
+5. **Test** — write or update tests; new code requires 100% test coverage
+6. **Document** — update all relevant docs in `docs/`; this is mandatory, not optional
+7. **Quality Gate** — run the single gate command below and fix every failure before committing:
+
+```bash
+ruff format app/ tests/ && \
+ruff check app/ tests/ --fix && \
+safety check && \
+pytest --tb=short -q
+```
+
+8. **Review** — re-read your own diff; confirm it is clean, secure, minimal, and well-documented
+
+> All commands in the quality gate must exit with code 0. Never submit with failures.
+
 ## Core Principles
 
 ### Code Quality
-- Always use **Black** for formatting (line length: 120)
+- **Security first**: treat security as a non-negotiable requirement, not an afterthought — review [SECURITY_AUDIT.md](../SECURITY_AUDIT.md) for every change
-- Use **isort** with Black profile for import sorting
+- Write **clean, modern, well-documented code** — prioritize readability, maintainability, and idiomatic Python
-- Use **flake8** for linting (ignore E203, W503)
+- Use **Ruff** for all formatting, linting, import sorting, and security scanning — `ruff format` + `ruff check --fix` (replaces Black, isort, Flake8, Bandit)
+- Line length: 120 characters (configured in `pyproject.toml`)
 - Use **type hints** for all function parameters and return values
 - Write **docstrings** for all public functions, classes, and modules
-- Maintain **80% test coverage** for new code
+- Maintain **100% test coverage** for new code
 
 ### Python Conventions
 - Use descriptive variable names (e.g., `user_document_path`, not `udp`)
 - Follow PEP 8 naming: `snake_case` for functions/variables, `PascalCase` for classes
-- Use type hints from `typing` module (Dict, List, Optional, etc.)
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None` — avoid `List`, `Dict`, `Optional` from `typing`
+- Only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol`, and other constructs unavailable natively
 - Prefer `pathlib.Path` over string paths for file operations
 - Use f-strings for string formatting, not `.format()` or `%`
 - Handle exceptions explicitly - avoid bare `except:` clauses
@@ -112,7 +129,8 @@ pre-commit run --all-files
 - Validate and sanitize all user inputs
 - Use parameterized queries with SQLAlchemy (never raw SQL with user input)
 - Review [SECURITY_AUDIT.md](../SECURITY_AUDIT.md) before making security-related changes
-- Run `bandit` to check for security issues in Python code
+- Security linting is built into Ruff via `S` rules — runs automatically with `ruff check`; fix all `S`-prefixed findings
+- Run `safety check` to scan dependencies for known CVEs before submitting any PR
 
 ### FastAPI Patterns
 - Organize endpoints by feature in `app/api/` directory
@@ -152,6 +170,8 @@ pre-commit run --all-files
 - Use `pytest.fixture` for test setup and teardown
 - Run tests with: `pytest -v`
 - Check coverage with: `pytest --cov=app --cov-report=term-missing`
+- **All tests must pass** before submitting changes — never leave failing tests
+- **All linters must pass** before submitting — run `pre-commit run --all-files`
 
 ### Configuration
 - All configuration is in `app/config.py` using Pydantic Settings
@@ -161,7 +181,7 @@ pre-commit run --all-files
 
 ### Documentation
 - Keep documentation in `docs/` directory in Markdown format
-- Update relevant docs when adding features or changing behavior
+- **Always update** relevant docs when adding or changing any feature — documentation updates are mandatory, never optional
 - User-facing documentation should be clear and include examples
 - Reference existing docs: `docs/UserGuide.md`, `docs/API.md`, `docs/DeploymentGuide.md`
 - See [AGENTIC_CODING.md](../AGENTIC_CODING.md) for detailed development guide
@@ -223,7 +243,8 @@ These files and directories are managed by automation or are critical infrastruc
 - Write clear, descriptive commit messages
 - **ALWAYS follow Conventional Commits format** (see below)
 - Keep commits focused and atomic
-- Run tests and linters before committing
+- **All tests must pass** before committing — `pytest` must succeed with no failures
+- **All linters must pass** before committing — `pre-commit run --all-files` must succeed
 - Pre-commit hooks are configured (`.pre-commit-config.yaml`)
 
 ## Conventional Commits (REQUIRED)
@@ -313,7 +334,7 @@ These files are managed entirely by the semantic-release automation.
 - Configuration in `app/config.py`
 
 ### Common Patterns
-- Use `from typing import Optional, Dict, List, Any` for type hints
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None`; only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol`
 - Import FastAPI dependencies: `from fastapi import Depends, HTTPException, status`
 - Get DB session: `db: Session = Depends(get_db)`
 - Current user: `current_user: User = Depends(get_current_user)`

.github/copilot.yml

@@ -29,6 +29,10 @@ network:
     - api.dropboxapi.com
     - content.dropboxapi.com
     
+    # Example/test domains - Used in test fixtures and SMTP configuration tests
+    - example.com
+    - smtp.example.com
+
     # Package registries (if needed for dependency installation during tests)
     - pypi.org
     - files.pythonhosted.org

.github/instructions/documentation.instructions.md

@@ -230,12 +230,15 @@ For API details, refer to the [API Documentation](./API.md).
 ```
 
 ## Updating Documentation
+Documentation updates are **mandatory** — every PR that changes code must include matching documentation updates in the same PR. There are no exceptions.
+
 When making code changes:
-1. Update relevant documentation in the same PR
+1. **Update relevant documentation** in the same PR — never defer docs to a follow-up
-2. Check for outdated information
+2. Check for outdated information in existing docs
 3. Add new sections for new features
 4. Update examples if behavior changes
 5. Review related documentation for consistency
+6. Update `docs/ConfigurationGuide.md` and `.env.demo` for any new or changed configuration options
 
 ## Screenshots and Diagrams
 - Use clear, high-quality images

.github/instructions/python-backend.instructions.md

@@ -7,16 +7,18 @@ applyTo: "app/**/*.py"
 These instructions apply to all Python code in the `app/` directory.
 
 ## Code Style
-- Use **Ruff** for linting and formatting with 120 character line length
+- Use **Ruff** for all formatting, linting, import sorting, and security scanning — `ruff format app/ tests/ && ruff check app/ tests/ --fix`
+- Line length: 120 characters (configured in `pyproject.toml` `[tool.ruff]`)
 - All functions must have type hints for parameters and return values
-- Use `from typing import Optional, Dict, List, Any, Union` as needed
+- Use modern Python 3.10+ type hints: `list[str]`, `dict[str, Any]`, `str | None`
+- Only import from `typing` for `Any`, `Callable`, `TypeVar`, `Protocol` (not `Dict`, `List`, `Optional`, `Union`)
 
-## Import Order (Ruff enforces isort-compatible ordering)
+## Import Order (enforced by Ruff `I` rules)
 ```python
 # Standard library imports
 import os
 from pathlib import Path
-from typing import Optional, Dict, List
+from typing import Any  # Only for Any, Callable, TypeVar, Protocol
 
 # Third-party imports
 from fastapi import APIRouter, Depends, HTTPException
@@ -33,7 +35,7 @@ from app.models import Document, User
 def process_document(
     file_path: Path,
     user_id: int,
-    metadata: Optional[Dict[str, Any]] = None
+    metadata: dict[str, Any] | None = None
 ) -> DocumentMetadata:
     """
     Process a document and extract metadata.
@@ -127,7 +129,7 @@ import logging
 logger = logging.getLogger(__name__)
 
 @shared_task(bind=True, max_retries=3)
-def process_ocr(self, document_id: int) -> Dict[str, Any]:
+def process_ocr(self, document_id: int) -> dict[str, Any]:
     """Process OCR for a document."""
     try:
         # Processing logic
@@ -153,10 +155,11 @@ class Settings(BaseSettings):
         env_file = ".env"
 ```
 
-## Security
+## Security (First and Foremost)
-- Never commit secrets
+- **Security first**: treat every change as a potential attack surface — review `SECURITY_AUDIT.md` before making any security-related change
-- Validate all user inputs
+- Never commit secrets, tokens, or credentials
-- Use parameterized queries
+- Validate and sanitize all user inputs
-- Sanitize file paths
+- Use parameterized queries — never raw SQL with user data
-- Check file permissions
+- Sanitize file paths; check file permissions before access
-- Review SECURITY_AUDIT.md for guidelines
+- Security linting is built into Ruff via `S` rules — fix all `S`-prefixed findings before committing
+- Run `safety check` before submitting any PR to catch dependency CVEs

.github/instructions/testing.instructions.md

@@ -174,7 +174,8 @@ def test_create_document(db_session):
 ```
 
 ## Test Coverage Goals
-- Aim for **80% code coverage** for all new code
+- Achieve **100% test coverage** for all new code — use `# pragma: no cover` only for genuinely unreachable or platform-specific branches, with an inline comment explaining why
+- Enforce the threshold: `pytest --cov=app --cov-fail-under=100`
 - Focus on critical paths and error handling
 - Test both success and failure scenarios
 - Don't test third-party library code

BUILD_DATE

@@ -1 +1 @@
-2026-02-23T20:35:58Z
+2026-02-23T22:30:53Z

CHANGELOG.md

@@ -10,6 +10,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 <!-- version list -->
 
+## v0.47.0 (2026-02-23)
+
+### Documentation
+
+- **changelog**: Update changelog [skip ci]
+  ([`d70fffd`](https://github.com/christianlouis/DocuElevate/commit/d70fffd4d21214900fdb018066f10a60019b5706))
+
+- **copilot**: Checkpoint before comprehensive instructions overhaul
+  ([`dbea61d`](https://github.com/christianlouis/DocuElevate/commit/dbea61dc0b5808cb25db15553a950ca849a3228d))
+
+- **copilot**: Overhaul instructions — Ruff toolchain, 100% coverage, agent workflow, modern Python
+  ([`156e0ba`](https://github.com/christianlouis/DocuElevate/commit/156e0ba524000a1063c0cb316063f1e28e5f0182))
+
+### Features
+
+- **ai**: Handle temperature incompatibility for gpt-5 and o-series models, add model picker UI
+  ([`a94b52e`](https://github.com/christianlouis/DocuElevate/commit/a94b52ee144e6e1f5a54ebc85d548bf5f860626b))
+
+
+## Unreleased
+
+### Documentation
+
+- **copilot**: Checkpoint before comprehensive instructions overhaul
+  ([`dbea61d`](https://github.com/christianlouis/DocuElevate/commit/dbea61dc0b5808cb25db15553a950ca849a3228d))
+
+- **copilot**: Overhaul instructions — Ruff toolchain, 100% coverage, agent workflow, modern Python
+  ([`156e0ba`](https://github.com/christianlouis/DocuElevate/commit/156e0ba524000a1063c0cb316063f1e28e5f0182))
+
+
+## v0.46.0 (2026-02-23)
+
+### Bug Fixes
+
+- **api**: Swap parameter order in test_ai_extraction to fix 500 error
+  ([`ce4700a`](https://github.com/christianlouis/DocuElevate/commit/ce4700ae44abf55a87bc94b83c4d41e4cac2278e))
+
+### Features
+
+- **ui**: Add copy button to text modals in file detail view
+  ([`219e03e`](https://github.com/christianlouis/DocuElevate/commit/219e03ed3dc5cd80c1ed073167669db49dc01512))
+
+
+## v0.45.0 (2026-02-23)
+
+### Chores
+
+- Add example.com and smtp.example.com to copilot agent network allowlist
+  ([`d93cd96`](https://github.com/christianlouis/DocuElevate/commit/d93cd96b6205021c4739765090fa55641e23e2ee))
+
+### Documentation
+
+- **changelog**: Update changelog [skip ci]
+  ([`d5e4a74`](https://github.com/christianlouis/DocuElevate/commit/d5e4a74f693b6a458dcdc2978d834d41c5b4cb4d))
+
+### Features
+
+- **api**: Add POST /api/ai/test-extraction endpoint and Test Extraction UI button
+  ([`ce73f23`](https://github.com/christianlouis/DocuElevate/commit/ce73f23a8981ffae94594461362df3834481ced7))
+
+### Testing
+
+- **api**: Add comprehensive tests for AI extraction endpoint reaching 100% coverage
+  ([`3262d65`](https://github.com/christianlouis/DocuElevate/commit/3262d65bfe38d0669ba7eb6c0c51961ff6817976))
+
+
+## Unreleased
+
+### Chores
+
+- Add example.com and smtp.example.com to copilot agent network allowlist
+  ([`d93cd96`](https://github.com/christianlouis/DocuElevate/commit/d93cd96b6205021c4739765090fa55641e23e2ee))
+
+
 ## v0.44.0 (2026-02-23)
 
 ### Code Style

GIT_SHA

@@ -1 +1 @@
-5a64a3e
+7224e0b

RUNTIME_INFO

@@ -1,10 +1,10 @@
 DocuElevate Build Information
 ==============================
-Version: 0.44.0
+Version: 0.47.0
-Build Date: 2026-02-23T20:35:58Z
+Build Date: 2026-02-23T22:30:53Z
-Git Commit: 5a64a3ec71d2e4cc788e2bc822023f879fef4370
+Git Commit: 7224e0be7de0eb258777f0817875aa772eaa459b
-Git Short SHA: 5a64a3e
+Git Short SHA: 7224e0b
 Git Branch: main
-Commit Date: 2026-02-23T21:35:35+01:00
+Commit Date: 2026-02-23T23:30:33+01:00
-Build Timestamp: 2026-02-23T20:35:58Z
+Build Timestamp: 2026-02-23T22:30:53Z
 ==============================

VERSION

@@ -1 +1 @@
-0.44.0
+0.47.0

app/api/openai.py

@@ -1,14 +1,20 @@
 """
 AI provider and OpenAI API endpoints.
 
-Exposes two endpoints:
+Exposes three endpoints:
 - GET /api/ai/test             – tests the currently configured AI provider (generic, provider-agnostic)
 - GET /api/openai/test         – backward-compatible alias that tests the OpenAI API specifically
+- POST /api/ai/test-extraction – runs the metadata-extraction prompt against the configured AI provider
+                                 with caller-supplied plaintext and returns the raw response, parsed JSON,
+                                 and extracted tags so operators can evaluate model quality.
 """
 
+import json
 import logging
+import re
 
 from fastapi import APIRouter, Request
+from pydantic import BaseModel, Field
 
 from app.auth import require_login
 from app.config import settings
@@ -18,6 +24,10 @@ logger = logging.getLogger(__name__)
 
 router = APIRouter()
 
+# Maximum number of characters accepted for a test-extraction request.
+# Keeps individual requests reasonable without blocking any real-world document.
+_MAX_EXTRACTION_TEXT_LEN = 50_000
+
 
 def _get_exception_chain_detail(exc: Exception) -> str:
     """
@@ -202,3 +212,123 @@ async def test_ai_provider_connection(request: Request):
             "message": f"Connection failed: {detail}",
             "provider": provider_name,
         }
+
+
+class ExtractionTestRequest(BaseModel):
+    """Request body for the AI extraction test endpoint."""
+
+    text: str = Field(..., min_length=1, max_length=_MAX_EXTRACTION_TEXT_LEN, description="Plain-text document content")
+
+
+def _build_extraction_prompt(text: str) -> str:
+    """Return the metadata-extraction prompt used in the standard processing pipeline."""
+    return (
+        "You are a specialized document analyzer trained to extract structured metadata from documents.\n"
+        "Your task is to analyze the given text and return a well-structured JSON object.\n\n"
+        "Extract and return the following fields:\n"
+        "1. **filename**: Machine-readable filename "
+        "(YYYY-MM-DD_DescriptiveTitle, use only letters, numbers, periods, and underscores).\n"
+        '2. **empfaenger**: The recipient, or "Unknown" if not found.\n'
+        '3. **absender**: The sender, or "Unknown" if not found.\n'
+        "4. **correspondent**: The entity or company that issued the document "
+        '(shortest possible name, e.g., "Amazon" instead of "Amazon EU SARL, German branch").\n'
+        "5. **kommunikationsart**: One of [Behoerdlicher_Brief, Rechnung, Kontoauszug, Vertrag, "
+        "Quittung, Privater_Brief, Einladung, Gewerbliche_Korrespondenz, Newsletter, Werbung, Sonstiges].\n"
+        "6. **kommunikationskategorie**: One of [Amtliche_Postbehoerdliche_Dokumente, "
+        "Finanz_und_Vertragsdokumente, Geschaeftliche_Kommunikation, "
+        "Private_Korrespondenz, Sonstige_Informationen].\n"
+        "7. **document_type**: Precise classification (e.g., Invoice, Contract, Information, Unknown).\n"
+        "8. **tags**: A list of up to 4 relevant thematic keywords.\n"
+        '9. **language**: Detected document language (ISO 639-1 code, e.g., "de" or "en").\n'
+        "10. **title**: A human-readable title summarizing the document content.\n"
+        "11. **confidence_score**: A numeric value (0-100) indicating the confidence level "
+        "of the extracted metadata.\n"
+        "12. **reference_number**: Extracted invoice/order/reference number if available.\n"
+        "13. **monetary_amounts**: A list of key monetary values detected in the document.\n\n"
+        "### Important Rules:\n"
+        "- **OCR Correction**: Assume the text has been corrected for OCR errors.\n"
+        "- **Tagging**: Max 4 tags, avoiding generic or overly specific terms.\n"
+        "- **Title**: Concise, no addresses, and contains key identifying features.\n"
+        "- **Date Selection**: Use the most relevant date if multiple are found.\n"
+        "- **Output Language**: Maintain the document's original language.\n\n"
+        f"Extracted text:\n{text}\n\n"
+        "Return only valid JSON with no additional commentary.\n"
+    )
+
+
+def _extract_json_from_text(text: str):
+    """Try to extract a JSON object from the LLM response text."""
+    pattern = r"```(?:json)?\s*(\{.*?\})\s*```"
+    match = re.search(pattern, text, re.DOTALL)
+    if match:
+        return match.group(1)
+    start = text.find("{")
+    end = text.rfind("}")
+    if start != -1 and end != -1 and end > start:
+        return text[start : end + 1]
+    return None
+
+
+@router.post("/ai/test-extraction")
+@require_login
+async def test_ai_extraction(request: Request, body: ExtractionTestRequest):
+    """
+    Run the metadata-extraction prompt against the configured AI provider.
+
+    Accepts plain-text document content, sends it through the same prompt used
+    by the background processing pipeline, and returns:
+    - ``raw_response``:  verbatim LLM output
+    - ``parsed_json``:   the extracted JSON object (null when parsing fails)
+    - ``tags``:          the ``tags`` list from the parsed JSON (empty list on failure)
+    - ``provider`` / ``model``: which provider / model was used
+    """
+    from app.utils.ai_provider import get_ai_provider
+
+    provider_name = settings.ai_provider
+    model = settings.ai_model or settings.openai_model
+
+    logger.info(f"AI extraction test requested: provider={provider_name}, model={model}")
+
+    try:
+        provider = get_ai_provider()
+        prompt = _build_extraction_prompt(body.text)
+        raw_response = provider.chat_completion(
+            messages=[
+                {"role": "system", "content": "You are an intelligent document classifier."},
+                {"role": "user", "content": prompt},
+            ],
+            model=model,
+            temperature=0,
+        )
+    except ValueError as e:
+        logger.warning(f"AI extraction test – configuration error: {e}")
+        return {"status": "error", "message": str(e), "provider": provider_name}
+    except Exception as e:
+        detail = _get_exception_chain_detail(e)
+        logger.error(f"AI extraction test failed for provider '{provider_name}': {detail}", exc_info=True)
+        return {"status": "error", "message": f"AI call failed: {detail}", "provider": provider_name}
+
+    # Attempt to parse JSON from the response
+    parsed_json = None
+    tags: list = []
+    parse_error = None
+    json_text = _extract_json_from_text(raw_response)
+    if json_text:
+        try:
+            parsed_json = json.loads(json_text)
+            tags = parsed_json.get("tags", [])
+        except json.JSONDecodeError as exc:
+            parse_error = str(exc)
+            logger.warning(f"AI extraction test: JSON parse error: {exc}")
+    else:
+        parse_error = "No JSON object found in response"
+
+    return {
+        "status": "success",
+        "provider": provider_name,
+        "model": model,
+        "raw_response": raw_response,
+        "parsed_json": parsed_json,
+        "tags": tags,
+        "parse_error": parse_error,
+    }

app/utils/ai_provider.py

@@ -11,6 +11,7 @@ See the Configuration Guide for full details on each provider's settings.
 """
 
 import logging
+import re
 from abc import ABC, abstractmethod
 from typing import Any, Dict, List, Optional
 
@@ -19,6 +20,50 @@ from app.config import settings
 logger = logging.getLogger(__name__)
 
 
+def _resolve_temperature(model: str, requested: float) -> Optional[float]:
+    """Return a temperature value compatible with the given model, or ``None`` to omit it.
+
+    Certain model families have restrictions on the ``temperature`` parameter:
+
+    * **o-series reasoning models** (``o1``, ``o3``, ``o4``, …) – do not accept
+      a ``temperature`` argument at all.  Return ``None`` so callers can skip the
+      parameter entirely.
+    * **gpt-5 family** (``gpt-5``, ``gpt-5-nano``, ``gpt-5-codex``, …) – only
+      ``temperature=1`` is accepted; passing ``0`` raises a 400 error.  Return
+      ``1`` and emit a debug log so the caller is aware of the coercion.
+    * All other models – return the requested value unchanged.
+
+    The model string may include a provider prefix (e.g. ``openai/gpt-4o``);
+    only the part after the last ``/`` is examined.
+
+    Args:
+        model: Model identifier (may include a provider prefix).
+        requested: The temperature the caller wants to use.
+
+    Returns:
+        A compatible temperature float, or ``None`` if temperature should be
+        omitted from the API call.
+    """
+    bare = model.lower().split("/")[-1]
+
+    # o-series reasoning models (o1, o3, o4 …) do not support temperature
+    if re.match(r"^o\d+(-|$)", bare):
+        logger.debug("Dropping temperature parameter for reasoning model '%s' (not supported)", model)
+        return None
+
+    # gpt-5 family only supports temperature=1
+    if bare.startswith("gpt-5"):
+        if requested != 1.0:
+            logger.debug(
+                "Coercing temperature from %s to 1 for model '%s' (only temperature=1 is supported)",
+                requested,
+                model,
+            )
+        return 1.0
+
+    return requested
+
+
 def _require_text_content(content: Optional[str]) -> str:
     """Raise a clear error if the AI response contains no text content.
 
@@ -101,12 +146,12 @@ class OpenAIProvider(AIProvider):
         temperature: float = 0,
         **kwargs: Any,
     ) -> str:
-        completion = self._client.chat.completions.create(
+        call_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            model=model,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            **kwargs,
+        call_kwargs.update(kwargs)
-        )
+        completion = self._client.chat.completions.create(**call_kwargs)
         _content = completion.choices[0].message.content
         return _require_text_content(_content)
 
@@ -130,12 +175,12 @@ class AzureOpenAIProvider(AIProvider):
         temperature: float = 0,
         **kwargs: Any,
     ) -> str:
-        completion = self._client.chat.completions.create(
+        call_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            model=model,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            **kwargs,
+        call_kwargs.update(kwargs)
-        )
+        completion = self._client.chat.completions.create(**call_kwargs)
         _content = completion.choices[0].message.content
         return _require_text_content(_content)
 
@@ -161,13 +206,12 @@ class AnthropicProvider(AIProvider):
         import litellm
 
         model_name = model if model.startswith("anthropic/") else f"anthropic/{model}"
-        response = litellm.completion(
+        call_kwargs: Dict[str, Any] = {"model": model_name, "messages": messages, "api_key": self._api_key}
-            model=model_name,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            api_key=self._api_key,
+        call_kwargs.update(kwargs)
-            **kwargs,
+        response = litellm.completion(**call_kwargs)
-        )
         _content = response.choices[0].message.content
         return _require_text_content(_content)
 
@@ -193,13 +237,12 @@ class GeminiProvider(AIProvider):
         import litellm
 
         model_name = model if model.startswith("gemini/") else f"gemini/{model}"
-        response = litellm.completion(
+        call_kwargs: Dict[str, Any] = {"model": model_name, "messages": messages, "api_key": self._api_key}
-            model=model_name,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            api_key=self._api_key,
+        call_kwargs.update(kwargs)
-            **kwargs,
+        response = litellm.completion(**call_kwargs)
-        )
         _content = response.choices[0].message.content
         return _require_text_content(_content)
 
@@ -235,12 +278,12 @@ class OllamaProvider(AIProvider):
         temperature: float = 0,
         **kwargs: Any,
     ) -> str:
-        completion = self._client.chat.completions.create(
+        call_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            model=model,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            **kwargs,
+        call_kwargs.update(kwargs)
-        )
+        completion = self._client.chat.completions.create(**call_kwargs)
         _content = completion.choices[0].message.content
         return _require_text_content(_content)
 
@@ -269,12 +312,12 @@ class OpenRouterProvider(AIProvider):
         temperature: float = 0,
         **kwargs: Any,
     ) -> str:
-        completion = self._client.chat.completions.create(
+        call_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            model=model,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            **kwargs,
+        call_kwargs.update(kwargs)
-        )
+        completion = self._client.chat.completions.create(**call_kwargs)
         _content = completion.choices[0].message.content
         return _require_text_content(_content)
 
@@ -335,12 +378,12 @@ class PortkeyProvider(AIProvider):
         temperature: float = 0,
         **kwargs: Any,
     ) -> str:
-        completion = self._client.chat.completions.create(
+        call_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            model=model,
+        safe_temp = _resolve_temperature(model, temperature)
-            messages=messages,
+        if safe_temp is not None:
-            temperature=temperature,
+            call_kwargs["temperature"] = safe_temp
-            **kwargs,
+        call_kwargs.update(kwargs)
-        )
+        completion = self._client.chat.completions.create(**call_kwargs)
         _content = completion.choices[0].message.content
         return _require_text_content(_content)
 
@@ -371,11 +414,10 @@ class LiteLLMProvider(AIProvider):
     ) -> str:
         import litellm
 
-        completion_kwargs: Dict[str, Any] = {
+        completion_kwargs: Dict[str, Any] = {"model": model, "messages": messages}
-            "model": model,
+        safe_temp = _resolve_temperature(model, temperature)
-            "messages": messages,
+        if safe_temp is not None:
-            "temperature": temperature,
+            completion_kwargs["temperature"] = safe_temp
-        }
         if self._api_key:
             completion_kwargs["api_key"] = self._api_key
         if self._api_base:

app/utils/settings_service.py

@@ -154,10 +154,33 @@ SETTING_METADATA = {
     "openai_model": {
         "category": "AI Services",
         "description": "Fallback model name used when AI_MODEL is not set (e.g. gpt-4o-mini)",
-        "type": "string",
+        "type": "model_picker",
         "sensitive": False,
         "required": False,
         "restart_required": False,
+        "suggested_models": [
+            "gpt-4o",
+            "gpt-4o-mini",
+            "gpt-4-turbo",
+            "gpt-4",
+            "gpt-3.5-turbo",
+            "o1",
+            "o1-mini",
+            "o3",
+            "o3-mini",
+            "gpt-5",
+            "gpt-5-nano",
+            "claude-3-5-sonnet-20241022",
+            "claude-3-5-haiku-20241022",
+            "claude-3-opus-20240229",
+            "gemini-1.5-pro",
+            "gemini-1.5-flash",
+            "gemini-2.0-flash-exp",
+            "llama3.2",
+            "qwen2.5:7b",
+            "phi3",
+            "mistral",
+        ],
     },
     "ai_provider": {
         "category": "AI Services",
@@ -171,10 +194,33 @@ SETTING_METADATA = {
     "ai_model": {
         "category": "AI Services",
         "description": "Model name for the selected provider (overrides OPENAI_MODEL). E.g. gpt-4o, claude-3-5-sonnet-20241022, gemini-1.5-pro, llama3.2",
-        "type": "string",
+        "type": "model_picker",
         "sensitive": False,
         "required": False,
         "restart_required": False,
+        "suggested_models": [
+            "gpt-4o",
+            "gpt-4o-mini",
+            "gpt-4-turbo",
+            "gpt-4",
+            "gpt-3.5-turbo",
+            "o1",
+            "o1-mini",
+            "o3",
+            "o3-mini",
+            "gpt-5",
+            "gpt-5-nano",
+            "claude-3-5-sonnet-20241022",
+            "claude-3-5-haiku-20241022",
+            "claude-3-opus-20240229",
+            "gemini-1.5-pro",
+            "gemini-1.5-flash",
+            "gemini-2.0-flash-exp",
+            "llama3.2",
+            "qwen2.5:7b",
+            "phi3",
+            "mistral",
+        ],
     },
     "anthropic_api_key": {
         "category": "AI Services",

frontend/templates/file_detail.html

@@ -802,6 +802,50 @@
       }
     }
 
+    function showCopyFeedback(type, success) {
+      const btn = document.getElementById(type + '-copy-btn');
+      if (!btn) return;
+      const original = btn.innerHTML;
+      if (success) {
+        btn.innerHTML = '<i class="fas fa-check"></i> Copied!';
+        btn.style.backgroundColor = '#48bb78';
+      } else {
+        btn.innerHTML = '<i class="fas fa-exclamation-triangle"></i> Failed';
+        btn.style.backgroundColor = '#f56565';
+      }
+      setTimeout(() => {
+        btn.innerHTML = original;
+        btn.style.backgroundColor = '#4299e1';
+      }, 2000);
+    }
+
+    function copyTextToClipboard(type) {
+      const contentId = type + '-text-content';
+      const content = document.getElementById(contentId);
+      const text = content ? content.textContent : '';
+
+      if (!text) return;
+
+      navigator.clipboard.writeText(text).then(() => {
+        showCopyFeedback(type, true);
+      }).catch(() => {
+        // Fallback for older browsers
+        try {
+          const textarea = document.createElement('textarea');
+          textarea.value = text;
+          textarea.style.position = 'fixed';
+          textarea.style.opacity = '0';
+          document.body.appendChild(textarea);
+          textarea.select();
+          const ok = document.execCommand('copy');
+          document.body.removeChild(textarea);
+          showCopyFeedback(type, ok);
+        } catch (e) {
+          showCopyFeedback(type, false);
+        }
+      });
+    }
+
     // Close modal when clicking outside the content
     window.onclick = function(event) {
       const modals = document.querySelectorAll('.text-modal');
@@ -1210,6 +1254,14 @@
     <div class="text-modal-content">
       <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 1rem; padding-bottom: 1rem; border-bottom: 2px solid #e2e8f0;">
         <h3 style="margin: 0; color: #2d3748;">Extracted Text (Original)</h3>
+        <div style="display: flex; gap: 0.5rem;">
+          <button
+            id="original-copy-btn"
+            onclick="copyTextToClipboard('original')"
+            style="background-color: #4299e1; color: white; padding: 0.5rem 1rem; border: none; border-radius: 0.25rem; cursor: pointer; font-weight: 600;"
+          >
+            <i class="fas fa-copy"></i> Copy
+          </button>
           <button
             onclick="toggleTextModal('original-text-modal')"
             style="background-color: #f56565; color: white; padding: 0.5rem 1rem; border: none; border-radius: 0.25rem; cursor: pointer; font-weight: 600;"
@@ -1217,6 +1269,7 @@
             <i class="fas fa-times"></i> Close
           </button>
         </div>
+      </div>
       <div id="original-text-loading" style="text-align: center; padding: 3rem; color: #4299e1;">
         <i class="fas fa-spinner fa-spin" style="font-size: 2rem; margin-bottom: 1rem;"></i>
         <p>Extracting text from PDF...</p>
@@ -1230,6 +1283,14 @@
     <div class="text-modal-content">
       <div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 1rem; padding-bottom: 1rem; border-bottom: 2px solid #e2e8f0;">
         <h3 style="margin: 0; color: #2d3748;">Extracted Text (Processed)</h3>
+        <div style="display: flex; gap: 0.5rem;">
+          <button
+            id="processed-copy-btn"
+            onclick="copyTextToClipboard('processed')"
+            style="background-color: #4299e1; color: white; padding: 0.5rem 1rem; border: none; border-radius: 0.25rem; cursor: pointer; font-weight: 600;"
+          >
+            <i class="fas fa-copy"></i> Copy
+          </button>
           <button
             onclick="toggleTextModal('processed-text-modal')"
             style="background-color: #f56565; color: white; padding: 0.5rem 1rem; border: none; border-radius: 0.25rem; cursor: pointer; font-weight: 600;"
@@ -1237,6 +1298,7 @@
             <i class="fas fa-times"></i> Close
           </button>
         </div>
+      </div>
       <div id="processed-text-loading" style="text-align: center; padding: 3rem; color: #48bb78;">
         <i class="fas fa-spinner fa-spin" style="font-size: 2rem; margin-bottom: 1rem;"></i>
         <p>Extracting text from PDF...</p>

frontend/templates/settings.html

@@ -139,6 +139,7 @@
                   Enable {{ setting.key.replace('_', ' ').title() }}
                 </label>
               </div>
+<<<<<<< copilot/add-self-hosted-ocr-support
               {% elif setting.metadata.type == 'multiselect' and setting.metadata.options %}
               <!-- Multi-select checkboxes: stored as comma-separated string -->
               <div class="flex flex-wrap gap-2 mt-1">
@@ -160,6 +161,31 @@
                 {% endfor %}
               </div>
               <p class="mt-1 text-xs text-gray-400">Effective value: <code x-text="formData['{{ setting.key }}'] || '(none)'"></code></p>
+=======
+              {% elif setting.metadata.type == 'model_picker' %}
+              <!-- Model Picker: free-text input with datalist of common models -->
+              <div class="relative">
+                <input
+                  type="text"
+                  id="{{ setting.key }}"
+                  name="{{ setting.key }}"
+                  list="{{ setting.key }}_models"
+                  x-model="formData['{{ setting.key }}']"
+                  class="setting-input w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-blue-500 focus:border-blue-500"
+                  placeholder="Select a common model or type a custom name…"
+                  autocomplete="off"
+                />
+                <datalist id="{{ setting.key }}_models">
+                  {% for m in setting.metadata.suggested_models %}
+                  <option value="{{ m }}">{{ m }}</option>
+                  {% endfor %}
+                </datalist>
+                <p class="text-xs text-gray-400 mt-1">
+                  <i class="fas fa-info-circle mr-1"></i>
+                  Pick from the list or type any model name supported by your provider.
+                </p>
+              </div>
+>>>>>>> main
               {% elif setting.metadata.options %}
               <!-- Dropdown Select for fields with a fixed list of values -->
               <select

frontend/templates/status_dashboard.html

@@ -181,6 +181,12 @@
                       data-provider="ai_provider">
                       Test Connection
                     </button>
+                    <button
+                      id="testAiExtractionBtn"
+                      class="inline-flex items-center px-2.5 py-1.5 border border-transparent text-xs font-medium rounded text-indigo-700 bg-indigo-100 hover:bg-indigo-200 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500">
+                      <i class="fa-solid fa-flask mr-1"></i>
+                      Test Extraction
+                    </button>
                   {% endif %}
                 {% elif name == "Azure AI" %}
                   {% if provider.configured %}
@@ -273,6 +279,85 @@
       </div>
     </div>
   </div>
+
+  <!-- AI Extraction Test Modal -->
+  <div id="aiExtractionModal" class="fixed inset-0 bg-gray-600 bg-opacity-50 hidden overflow-y-auto h-full w-full z-50" aria-modal="true" role="dialog">
+    <div class="relative top-10 mx-auto p-5 border w-11/12 md:w-3/4 lg:w-2/3 xl:w-1/2 shadow-lg rounded-md bg-white">
+      <div class="absolute top-0 right-0 pt-4 pr-4">
+        <button type="button" id="closeAiExtractionModal" class="text-gray-400 hover:text-gray-500">
+          <span class="sr-only">Close</span>
+          <i class="fa-solid fa-xmark h-6 w-6"></i>
+        </button>
+      </div>
+      <div class="mt-3">
+        <h3 class="text-lg leading-6 font-medium text-gray-900 mb-1">
+          <i class="fa-solid fa-flask mr-2 text-indigo-600"></i>AI Extraction Test
+        </h3>
+        <p class="text-sm text-gray-500 mb-4">
+          Paste the plain-text content of a document below and run it through the configured AI provider
+          to inspect the raw response, extracted JSON, and tags.
+        </p>
+
+        <!-- Input area -->
+        <div id="aiExtractionInput">
+          <label for="aiExtractionText" class="block text-sm font-medium text-gray-700 mb-1">Document Text</label>
+          <textarea
+            id="aiExtractionText"
+            rows="10"
+            class="w-full px-3 py-2 border border-gray-300 rounded-md shadow-sm text-sm font-mono focus:outline-none focus:ring-indigo-500 focus:border-indigo-500"
+            placeholder="Paste the plain-text content of your document here…"></textarea>
+          <div class="mt-3 flex justify-end space-x-2">
+            <button type="button" id="cancelAiExtractionBtn"
+              class="px-4 py-2 border border-gray-300 text-sm font-medium rounded-md text-gray-700 bg-white hover:bg-gray-50 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500">
+              Cancel
+            </button>
+            <button type="button" id="runAiExtractionBtn"
+              class="inline-flex items-center px-4 py-2 border border-transparent text-sm font-medium rounded-md shadow-sm text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500">
+              <i class="fa-solid fa-play mr-2"></i>Run Extraction
+            </button>
+          </div>
+        </div>
+
+        <!-- Results area (hidden until extraction completes) -->
+        <div id="aiExtractionResults" class="hidden mt-4 space-y-4">
+          <!-- Provider / model badge -->
+          <div id="aiExtractionMeta" class="text-xs text-gray-500"></div>
+
+          <!-- Parse warning -->
+          <div id="aiParseWarning" class="hidden bg-yellow-50 border-l-4 border-yellow-400 p-3 text-sm text-yellow-800"></div>
+
+          <!-- Raw response -->
+          <div>
+            <h4 class="text-sm font-medium text-gray-900 mb-1">Raw LLM Response</h4>
+            <pre id="aiRawResponse" class="bg-gray-50 border border-gray-200 rounded p-3 text-xs overflow-auto max-h-48 whitespace-pre-wrap break-words"></pre>
+          </div>
+
+          <!-- Parsed JSON -->
+          <div id="aiParsedJsonSection">
+            <h4 class="text-sm font-medium text-gray-900 mb-1">Parsed JSON</h4>
+            <pre id="aiParsedJson" class="bg-gray-50 border border-gray-200 rounded p-3 text-xs overflow-auto max-h-64 whitespace-pre-wrap break-words"></pre>
+          </div>
+
+          <!-- Tags -->
+          <div id="aiTagsSection">
+            <h4 class="text-sm font-medium text-gray-900 mb-1">Extracted Tags</h4>
+            <div id="aiTags" class="flex flex-wrap gap-2"></div>
+          </div>
+
+          <div class="flex justify-end space-x-2 pt-2">
+            <button type="button" id="aiExtractionBackBtn"
+              class="px-4 py-2 border border-gray-300 text-sm font-medium rounded-md text-gray-700 bg-white hover:bg-gray-50 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500">
+              <i class="fa-solid fa-arrow-left mr-1"></i>Back
+            </button>
+            <button type="button" id="aiExtractionCloseBtn"
+              class="px-4 py-2 bg-indigo-600 text-white text-sm font-medium rounded-md shadow-sm hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-600">
+              Close
+            </button>
+          </div>
+        </div>
+      </div>
+    </div>
+  </div>
 </div>
 {% endblock %}
 
@@ -573,6 +658,152 @@ document.addEventListener('DOMContentLoaded', function() {
         });
     });
   });
+
+  // ── AI Extraction Test Modal ──────────────────────────────────────────────
+
+  const aiExtractionModal    = document.getElementById('aiExtractionModal');
+  const aiExtractionText     = document.getElementById('aiExtractionText');
+  const aiExtractionInput    = document.getElementById('aiExtractionInput');
+  const aiExtractionResults  = document.getElementById('aiExtractionResults');
+  const aiExtractionMeta     = document.getElementById('aiExtractionMeta');
+  const aiRawResponse        = document.getElementById('aiRawResponse');
+  const aiParsedJson         = document.getElementById('aiParsedJson');
+  const aiParsedJsonSection  = document.getElementById('aiParsedJsonSection');
+  const aiTags               = document.getElementById('aiTags');
+  const aiTagsSection        = document.getElementById('aiTagsSection');
+  const aiParseWarning       = document.getElementById('aiParseWarning');
+  const testAiExtractionBtn  = document.getElementById('testAiExtractionBtn');
+  const runAiExtractionBtn   = document.getElementById('runAiExtractionBtn');
+  const cancelAiExtractionBtn= document.getElementById('cancelAiExtractionBtn');
+  const aiExtractionBackBtn  = document.getElementById('aiExtractionBackBtn');
+  const aiExtractionCloseBtn = document.getElementById('aiExtractionCloseBtn');
+  const closeAiExtractionModal = document.getElementById('closeAiExtractionModal');
+
+  function openAiExtractionModal() {
+    // Reset to input view
+    aiExtractionText.value = '';
+    aiExtractionInput.classList.remove('hidden');
+    aiExtractionResults.classList.add('hidden');
+    aiExtractionModal.classList.remove('hidden');
+  }
+
+  function closeAiExtractionModalFn() {
+    aiExtractionModal.classList.add('hidden');
+  }
+
+  function showAiExtractionResults(data) {
+    // Provider / model meta
+    aiExtractionMeta.textContent = `Provider: ${data.provider || '—'}  |  Model: ${data.model || '—'}`;
+
+    // Raw response
+    aiRawResponse.textContent = data.raw_response || '(empty)';
+
+    // Parse warning
+    if (data.parse_error) {
+      aiParseWarning.textContent = `JSON parse issue: ${data.parse_error}`;
+      aiParseWarning.classList.remove('hidden');
+    } else {
+      aiParseWarning.classList.add('hidden');
+    }
+
+    // Parsed JSON
+    if (data.parsed_json) {
+      aiParsedJson.textContent = JSON.stringify(data.parsed_json, null, 2);
+      aiParsedJsonSection.classList.remove('hidden');
+    } else {
+      aiParsedJsonSection.classList.add('hidden');
+    }
+
+    // Tags
+    aiTags.innerHTML = '';
+    const tags = Array.isArray(data.tags) ? data.tags : [];
+    if (tags.length > 0) {
+      tags.forEach(tag => {
+        const span = document.createElement('span');
+        span.className = 'inline-flex items-center px-2.5 py-0.5 rounded-full text-xs font-medium bg-indigo-100 text-indigo-800';
+        span.textContent = tag;
+        aiTags.appendChild(span);
+      });
+      aiTagsSection.classList.remove('hidden');
+    } else {
+      aiTagsSection.classList.add('hidden');
+    }
+
+    // Switch view
+    aiExtractionInput.classList.add('hidden');
+    aiExtractionResults.classList.remove('hidden');
+  }
+
+  if (testAiExtractionBtn) {
+    testAiExtractionBtn.addEventListener('click', openAiExtractionModal);
+  }
+
+  if (closeAiExtractionModal) {
+    closeAiExtractionModal.addEventListener('click', closeAiExtractionModalFn);
+  }
+
+  if (cancelAiExtractionBtn) {
+    cancelAiExtractionBtn.addEventListener('click', closeAiExtractionModalFn);
+  }
+
+  if (aiExtractionCloseBtn) {
+    aiExtractionCloseBtn.addEventListener('click', closeAiExtractionModalFn);
+  }
+
+  if (aiExtractionBackBtn) {
+    aiExtractionBackBtn.addEventListener('click', function() {
+      aiExtractionResults.classList.add('hidden');
+      aiExtractionInput.classList.remove('hidden');
+    });
+  }
+
+  // Close when clicking the backdrop
+  if (aiExtractionModal) {
+    aiExtractionModal.addEventListener('click', function(e) {
+      if (e.target === aiExtractionModal) {
+        closeAiExtractionModalFn();
+      }
+    });
+  }
+
+  if (runAiExtractionBtn) {
+    runAiExtractionBtn.addEventListener('click', function() {
+      const text = aiExtractionText.value.trim();
+      if (!text) {
+        aiExtractionText.focus();
+        return;
+      }
+
+      const originalHTML = runAiExtractionBtn.innerHTML;
+      runAiExtractionBtn.innerHTML = '<i class="fa-solid fa-spinner fa-spin mr-2"></i>Running…';
+      runAiExtractionBtn.disabled = true;
+      cancelAiExtractionBtn.disabled = true;
+
+      fetch('/api/ai/test-extraction', {
+        method: 'POST',
+        headers: { 'Content-Type': 'application/json' },
+        body: JSON.stringify({ text: text }),
+      })
+        .then(response => response.json())
+        .then(data => {
+          if (data.status === 'success') {
+            showAiExtractionResults(data);
+          } else {
+            closeAiExtractionModalFn();
+            showModal('error', 'AI Extraction Failed', data.message || 'Unknown error');
+          }
+        })
+        .catch(error => {
+          closeAiExtractionModalFn();
+          showModal('error', 'Connection Error', 'Error running extraction: ' + error.message);
+        })
+        .finally(() => {
+          runAiExtractionBtn.innerHTML = originalHTML;
+          runAiExtractionBtn.disabled = false;
+          cancelAiExtractionBtn.disabled = false;
+        });
+    });
+  }
 });
 </script>
 {% endblock %}

tests/test_ai_provider.py

@@ -16,6 +16,7 @@ from app.utils.ai_provider import (
     OpenRouterProvider,
     PortkeyProvider,
     _require_text_content,
+    _resolve_temperature,
     get_ai_provider,
 )
 
@@ -74,6 +75,77 @@ class TestRequireTextContent:
             _require_text_content(None)
 
 
+# ---------------------------------------------------------------------------
+# _resolve_temperature helper
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.unit
+class TestResolveTemperature:
+    """Tests for the _resolve_temperature compatibility helper."""
+
+    def test_regular_model_returns_requested_temperature(self):
+        """Standard models return the temperature unchanged."""
+        assert _resolve_temperature("gpt-4o", 0) == 0
+        assert _resolve_temperature("gpt-4o-mini", 0.7) == 0.7
+
+    def test_gpt4_model_returns_requested_temperature(self):
+        """gpt-4 models are not gpt-5, so temperature is returned as-is."""
+        assert _resolve_temperature("gpt-4-turbo", 0) == 0
+
+    def test_gpt5_model_forces_temperature_1(self):
+        """gpt-5 models only accept temperature=1; any other value is coerced."""
+        assert _resolve_temperature("gpt-5", 0) == 1.0
+
+    def test_gpt5_nano_forces_temperature_1(self):
+        """gpt-5-nano (gpt-5 variant) gets temperature coerced to 1."""
+        assert _resolve_temperature("gpt-5-nano", 0) == 1.0
+
+    def test_gpt5_codex_forces_temperature_1(self):
+        """gpt-5-codex (gpt-5 variant) gets temperature coerced to 1."""
+        assert _resolve_temperature("gpt-5-codex", 0) == 1.0
+
+    def test_gpt5_already_at_1_unchanged(self):
+        """gpt-5 with temperature=1 returns 1 (no unnecessary log noise)."""
+        assert _resolve_temperature("gpt-5", 1.0) == 1.0
+
+    def test_o1_returns_none(self):
+        """o1 reasoning model does not support temperature; None is returned."""
+        assert _resolve_temperature("o1", 0) is None
+
+    def test_o1_mini_returns_none(self):
+        """o1-mini returns None (temperature not supported)."""
+        assert _resolve_temperature("o1-mini", 0) is None
+
+    def test_o1_preview_returns_none(self):
+        """o1-preview returns None (temperature not supported)."""
+        assert _resolve_temperature("o1-preview", 0) is None
+
+    def test_o3_returns_none(self):
+        """o3 reasoning model does not support temperature."""
+        assert _resolve_temperature("o3", 0) is None
+
+    def test_o3_mini_returns_none(self):
+        """o3-mini returns None (temperature not supported)."""
+        assert _resolve_temperature("o3-mini", 0) is None
+
+    def test_o4_mini_returns_none(self):
+        """o4-mini returns None (temperature not supported)."""
+        assert _resolve_temperature("o4-mini", 0) is None
+
+    def test_provider_prefix_is_stripped(self):
+        """Provider prefix (e.g. 'openai/') is ignored when matching."""
+        assert _resolve_temperature("openai/gpt-5-nano", 0) == 1.0
+        assert _resolve_temperature("openai/o1-mini", 0) is None
+        assert _resolve_temperature("openai/gpt-4o", 0) == 0
+
+    def test_model_names_are_case_insensitive(self):
+        """Model matching is case-insensitive."""
+        assert _resolve_temperature("GPT-5", 0) == 1.0
+        assert _resolve_temperature("O1", 0) is None
+        assert _resolve_temperature("O3-Mini", 0) is None
+
+
 # ---------------------------------------------------------------------------
 # Abstract base class
 # ---------------------------------------------------------------------------

tests/test_api_openai.py

@@ -309,3 +309,363 @@ class TestOpenAIConnectionErrors:
 
         assert data["status"] == "error"
         assert data.get("is_auth_error") is False
+
+
+@pytest.mark.unit
+class TestAiExtractionEndpoint:
+    """Tests for POST /api/ai/test-extraction endpoint."""
+
+    @patch("app.api.openai.settings")
+    def test_extraction_missing_text_returns_422(self, mock_settings, client):
+        """Test that missing text body returns 422 validation error."""
+        response = client.post("/api/ai/test-extraction", json={})
+        assert response.status_code == 422
+
+    @patch("app.api.openai.settings")
+    def test_extraction_empty_text_returns_422(self, mock_settings, client):
+        """Test that empty string text returns 422 validation error."""
+        response = client.post("/api/ai/test-extraction", json={"text": ""})
+        assert response.status_code == 422
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_returns_raw_response_and_parsed_json(self, mock_settings, mock_get_provider, client):
+        """Test successful extraction returns raw response, parsed JSON, and tags."""
+        import json as json_module
+
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        expected_json = {
+            "filename": "2024-01-01_Invoice",
+            "tags": ["invoice", "payment"],
+            "title": "January Invoice",
+            "document_type": "Invoice",
+        }
+        raw = "```json\n" + json_module.dumps(expected_json) + "\n```"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = raw
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Invoice content here"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["raw_response"] == raw
+        assert data["parsed_json"]["filename"] == "2024-01-01_Invoice"
+        assert data["tags"] == ["invoice", "payment"]
+        assert data["parse_error"] is None
+        assert data["provider"] == "openai"
+        assert data["model"] == "gpt-4o-mini"
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_handles_invalid_json_in_response(self, mock_settings, mock_get_provider, client):
+        """Test that invalid JSON in LLM response is reported via parse_error."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = "Sorry, I cannot help with that."
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Some document text"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["parsed_json"] is None
+        assert data["tags"] == []
+        assert data["parse_error"] is not None
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_provider_config_error(self, mock_settings, mock_get_provider, client):
+        """Test that configuration errors (missing keys) are returned as error status."""
+        mock_settings.ai_provider = "anthropic"
+        mock_settings.ai_model = "claude-3"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_get_provider.side_effect = ValueError("ANTHROPIC_API_KEY must be set")
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Some document text"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "error"
+        assert "ANTHROPIC_API_KEY" in data["message"]
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_provider_runtime_error(self, mock_settings, mock_get_provider, client):
+        """Test that runtime errors during AI call are returned as error status."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.side_effect = Exception("Connection refused")
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Some document text"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "error"
+        assert "Connection refused" in data["message"]
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_plain_json_without_code_fences(self, mock_settings, mock_get_provider, client):
+        """Test that JSON returned without code fences is still parsed correctly."""
+        import json as json_module
+
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        expected_json = {"tags": ["contract"], "title": "Service Agreement"}
+        raw = json_module.dumps(expected_json)
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = raw
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Contract content"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["tags"] == ["contract"]
+        assert data["parse_error"] is None
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_json_found_but_invalid_reports_parse_error(self, mock_settings, mock_get_provider, client):
+        """Test JSONDecodeError branch: response contains '{...}' but is not valid JSON."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        # Looks like JSON (has { and }) but is NOT parseable
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = "{this is not: valid json!!}"
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Some document text"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["parsed_json"] is None
+        assert data["tags"] == []
+        assert data["parse_error"] is not None
+        assert "raw_response" in data
+
+    @patch("app.api.openai.settings")
+    def test_extraction_text_too_long_returns_422(self, mock_settings, client):
+        """Test that text exceeding max length returns 422 validation error."""
+        from app.api.openai import _MAX_EXTRACTION_TEXT_LEN
+
+        oversized_text = "x" * (_MAX_EXTRACTION_TEXT_LEN + 1)
+        response = client.post("/api/ai/test-extraction", json={"text": oversized_text})
+        assert response.status_code == 422
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_extraction_response_with_no_tags_key(self, mock_settings, mock_get_provider, client):
+        """Test extraction where parsed JSON has no 'tags' key returns empty tags list."""
+        import json as json_module
+
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        # Valid JSON but no 'tags' key
+        payload = {"title": "Report", "document_type": "Report"}
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = json_module.dumps(payload)
+        mock_get_provider.return_value = mock_provider
+
+        response = client.post("/api/ai/test-extraction", json={"text": "Report content"})
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["tags"] == []
+        assert data["parsed_json"]["title"] == "Report"
+        assert data["parse_error"] is None
+
+
+@pytest.mark.unit
+class TestAiProviderTestEndpoint:
+    """Tests for GET /api/ai/test endpoint."""
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_ai_test_success(self, mock_settings, mock_get_provider, client):
+        """Test successful AI provider connection."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = "ok"
+        mock_get_provider.return_value = mock_provider
+
+        response = client.get("/api/ai/test")
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert "reachable" in data["message"].lower()
+        assert data["provider"] == "openai"
+        assert data["model"] == "gpt-4o-mini"
+        assert "response_preview" in data
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_ai_test_value_error(self, mock_settings, mock_get_provider, client):
+        """Test GET /api/ai/test returns error on provider configuration ValueError."""
+        mock_settings.ai_provider = "anthropic"
+        mock_settings.ai_model = None
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_get_provider.side_effect = ValueError("ANTHROPIC_API_KEY must be set")
+
+        response = client.get("/api/ai/test")
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "error"
+        assert "ANTHROPIC_API_KEY" in data["message"]
+        assert data["provider"] == "anthropic"
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_ai_test_connection_exception(self, mock_settings, mock_get_provider, client):
+        """Test GET /api/ai/test returns error on provider runtime exception."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = "gpt-4o-mini"
+        mock_settings.openai_model = "gpt-4o-mini"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.side_effect = Exception("Connection refused")
+        mock_get_provider.return_value = mock_provider
+
+        response = client.get("/api/ai/test")
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "error"
+        assert "Connection refused" in data["message"]
+        assert data["provider"] == "openai"
+
+    @patch("app.utils.ai_provider.get_ai_provider")
+    @patch("app.api.openai.settings")
+    def test_ai_test_uses_openai_model_fallback(self, mock_settings, mock_get_provider, client):
+        """Test that ai_model=None falls back to openai_model."""
+        mock_settings.ai_provider = "openai"
+        mock_settings.ai_model = None
+        mock_settings.openai_model = "gpt-3.5-turbo"
+
+        mock_provider = MagicMock()
+        mock_provider.chat_completion.return_value = "ok"
+        mock_get_provider.return_value = mock_provider
+
+        response = client.get("/api/ai/test")
+        assert response.status_code == 200
+        data = response.json()
+
+        assert data["status"] == "success"
+        assert data["model"] == "gpt-3.5-turbo"
+
+
+@pytest.mark.unit
+class TestExceptionChainDetail:
+    """Unit tests for the _get_exception_chain_detail helper."""
+
+    def test_single_exception_no_chain(self):
+        """Test with a simple exception that has no cause."""
+        from app.api.openai import _get_exception_chain_detail
+
+        exc = ValueError("root cause")
+        result = _get_exception_chain_detail(exc)
+        assert result == "root cause"
+
+    def test_exception_with_cause(self):
+        """Test that chained exceptions are surfaced in the message."""
+        from app.api.openai import _get_exception_chain_detail
+
+        inner = OSError("DNS resolution failed")
+        outer = ConnectionError("Connection failed")
+        outer.__cause__ = inner
+
+        result = _get_exception_chain_detail(outer)
+        assert "Connection failed" in result
+        assert "DNS resolution failed" in result
+        assert "caused by" in result
+
+    def test_empty_cause_string_skipped(self):
+        """Test that a cause with empty string representation is not appended."""
+        from app.api.openai import _get_exception_chain_detail
+
+        inner = Exception("")  # str() returns ""
+        outer = RuntimeError("outer error")
+        outer.__cause__ = inner
+
+        result = _get_exception_chain_detail(outer)
+        # The empty-string cause should be skipped (branch 47->49)
+        assert result == "outer error"
+
+    def test_duplicate_cause_string_skipped(self):
+        """Test that a cause whose str() is already in parts is not duplicated."""
+        from app.api.openai import _get_exception_chain_detail
+
+        outer = RuntimeError("same message")
+        inner = RuntimeError("same message")  # same text as outer
+        outer.__cause__ = inner
+
+        result = _get_exception_chain_detail(outer)
+        # "same message" should appear only once (branch: cause_str already in parts)
+        assert result.count("same message") == 1
+
+
+@pytest.mark.unit
+class TestExtractJsonFromText:
+    """Unit tests for the _extract_json_from_text helper."""
+
+    def test_json_in_code_fence(self):
+        """Test extraction from markdown code fence."""
+        from app.api.openai import _extract_json_from_text
+
+        text = '```json\n{"key": "value"}\n```'
+        result = _extract_json_from_text(text)
+        assert result == '{"key": "value"}'
+
+    def test_json_in_plain_code_fence(self):
+        """Test extraction from plain (non-json) code fence."""
+        from app.api.openai import _extract_json_from_text
+
+        text = '```\n{"key": "value"}\n```'
+        result = _extract_json_from_text(text)
+        assert result == '{"key": "value"}'
+
+    def test_bare_json_object(self):
+        """Test extraction of a bare JSON object without code fence."""
+        from app.api.openai import _extract_json_from_text
+
+        text = 'Here is the result: {"title": "Invoice"} done.'
+        result = _extract_json_from_text(text)
+        assert result == '{"title": "Invoice"}'
+
+    def test_no_json_returns_none(self):
+        """Test that text with no JSON object returns None."""
+        from app.api.openai import _extract_json_from_text
+
+        result = _extract_json_from_text("No JSON here at all.")
+        assert result is None