Initial plan

test: increase watch_folder_tasks.py coverage from 37% to 98%

.env.demo

@@ -16,6 +16,12 @@ ALLOW_FILE_DELETE=true  # Allow deletion of file records
 PROCESSALL_THROTTLE_THRESHOLD=20  # Number of files above which throttling is applied (default: 20)
 PROCESSALL_THROTTLE_DELAY=3  # Delay in seconds between each task submission when throttling (default: 3)
 
+# **Task Retry Settings**
+# Failed tasks are automatically retried with exponential backoff and jitter.
+# TASK_RETRY_MAX_RETRIES=3          # Max retry attempts per task (default: 3)
+# TASK_RETRY_DELAYS=60,300,900      # Countdown (seconds) before each retry; 1 min, 5 min, 15 min
+# TASK_RETRY_JITTER=true            # Add ±20% random jitter to prevent thundering-herd (default: true)
+
 # **Client-Side Upload Throttling**
 # Controls pacing when the browser uploads files (especially large directory drops).
 # The browser auto-detects rate-limit (HTTP 429) responses and backs off accordingly.
@@ -123,6 +129,29 @@ ADMIN_USERNAME=admin
 ADMIN_PASSWORD=your_secure_password
 ADMIN_GROUP_NAME=admin
 
+# **Multi-User Mode**
+# When enabled, each user has their own document space with isolated uploads,
+# search, and file management. Requires AUTH_ENABLED=true.
+MULTI_USER_ENABLED=false
+# Allow users to self-register with an email address and password.
+# Set to true to enable the /signup page. Requires MULTI_USER_ENABLED=true.
+# When SMTP is configured, a verification email is sent before the account is activated.
+# Without SMTP, accounts are activated immediately upon registration.
+# ALLOW_LOCAL_SIGNUP=false
+# Default upload limit per user per day (0 = unlimited)
+DEFAULT_DAILY_UPLOAD_LIMIT=0
+# Show unowned documents (owner_id=NULL) to all users (true) or only admins (false)
+UNOWNED_DOCS_VISIBLE_TO_ALL=true
+# Auto-assign this owner ID to documents ingested without a session (e.g. IMAP, API)
+# Leave empty/unset to keep them unowned until claimed.
+# DEFAULT_OWNER_ID=
+
+# **Subscription / Quota Settings**
+# Soft-limit overage buffer in percent (0–200). Announced quota is multiplied by (1 + percent/100)
+# for actual enforcement. E.g. 20 means a 150-doc/month plan enforces at 180. 0 = enforce exactly.
+# Per-plan overage_percent set in the Plan Designer overrides this global default.
+# SUBSCRIPTION_OVERAGE_PERCENT=20
+
 # **OpenID Connect/Authentik Settings**
 AUTHENTIK_CLIENT_ID=<yourAuthentikAppClientID>
 AUTHENTIK_CLIENT_SECRET=<yourAuthentikClientSecret>
@@ -168,16 +197,81 @@ OPENAI_MODEL=gpt-4o-mini
 # AI_MODEL=gpt-4o   # deployment name in Azure
 
 # Azure Document Intelligence (OCR – separate from AI provider above)
-# **Email Settings**
+# **Email Settings (shared SMTP – password reset, verification, and system notifications)**
 EMAIL_HOST=smtp.example.com
 EMAIL_PORT=587
 EMAIL_USERNAME=docuelevate@example.com
 EMAIL_PASSWORD=your_secure_email_password
 EMAIL_USE_TLS=True
 EMAIL_SENDER=DocuElevate System <docuelevate@example.com>
-EMAIL_DEFAULT_RECIPIENT=recipient@example.com
+# EMAIL_DEFAULT_RECIPIENT is not used for document delivery (see DEST_EMAIL_* below)
+
+# **Email Destination Settings (dedicated SMTP for document delivery)**
+# These settings are intentionally separate from the shared EMAIL_* settings above.
+# Configuring EMAIL_HOST for password reset / notifications does NOT automatically
+# enable the email destination – you must set DEST_EMAIL_HOST to activate it.
+DEST_EMAIL_HOST=smtp.example.com
+DEST_EMAIL_PORT=587
+DEST_EMAIL_USERNAME=docuelevate@example.com
+DEST_EMAIL_PASSWORD=your_secure_email_password
+DEST_EMAIL_USE_TLS=True
+DEST_EMAIL_SENDER=DocuElevate Delivery <docuelevate@example.com>
+DEST_EMAIL_DEFAULT_RECIPIENT=recipient@example.com
+
+# **Watch Folder Ingestion**
+# DocuElevate can automatically monitor directories (local, FTP, SFTP, and cloud providers) for new files.
+#
+# Local watch folders — works with any mounted path (SMB/CIFS, NFS, local disk, etc.)
+# Set WATCH_FOLDERS to a comma-separated list of absolute paths inside the container.
+WATCH_FOLDERS=
+WATCH_FOLDER_POLL_INTERVAL=1
+WATCH_FOLDER_DELETE_AFTER_PROCESS=false
+
+# FTP ingest — poll an FTP directory for new files (uses FTP connection settings above)
+FTP_INGEST_ENABLED=false
+FTP_INGEST_FOLDER=
+FTP_INGEST_DELETE_AFTER_PROCESS=false
+
+# SFTP ingest — poll an SFTP directory for new files (uses SFTP connection settings above)
+SFTP_INGEST_ENABLED=false
+SFTP_INGEST_FOLDER=
+SFTP_INGEST_DELETE_AFTER_PROCESS=false
+
+# Dropbox ingest — poll a Dropbox folder (uses Dropbox OAuth credentials above)
+DROPBOX_INGEST_ENABLED=false
+DROPBOX_INGEST_FOLDER=
+DROPBOX_INGEST_DELETE_AFTER_PROCESS=false
+
+# Google Drive ingest — poll a Google Drive folder (uses Google Drive credentials above)
+GOOGLE_DRIVE_INGEST_ENABLED=false
+GOOGLE_DRIVE_INGEST_FOLDER_ID=
+GOOGLE_DRIVE_INGEST_DELETE_AFTER_PROCESS=false
+
+# OneDrive ingest — poll a OneDrive folder (uses OneDrive MSAL credentials above)
+ONEDRIVE_INGEST_ENABLED=false
+ONEDRIVE_INGEST_FOLDER_PATH=
+ONEDRIVE_INGEST_DELETE_AFTER_PROCESS=false
+
+# Nextcloud ingest — poll a Nextcloud folder (uses Nextcloud WebDAV credentials above)
+NEXTCLOUD_INGEST_ENABLED=false
+NEXTCLOUD_INGEST_FOLDER=
+NEXTCLOUD_INGEST_DELETE_AFTER_PROCESS=false
+
+# Amazon S3 ingest — poll an S3 prefix (uses S3/AWS credentials above)
+S3_INGEST_ENABLED=false
+S3_INGEST_PREFIX=
+S3_INGEST_DELETE_AFTER_PROCESS=false
+
+# WebDAV ingest — poll a WebDAV folder (uses WebDAV credentials above)
+WEBDAV_INGEST_ENABLED=false
+WEBDAV_INGEST_FOLDER=
+WEBDAV_INGEST_DELETE_AFTER_PROCESS=false
 
 # **IMAP Settings**
+# DocuElevate polls these mailboxes for new email attachments and automatically ingests them.
+# No manual forwarding required — DocuElevate acts as an IMAP *client*.
+# For HP Scanners / Scan-to-Email: configure the scanner to send to a dedicated mailbox,
+# then point DocuElevate at that mailbox using the settings below.
 IMAP1_HOST=mail.example.com
 IMAP1_PORT=993
 IMAP1_USERNAME=<IMAP1_USERNAME>
@@ -313,9 +407,27 @@ NOTIFY_ON_STARTUP=True
 NOTIFY_ON_SHUTDOWN=False
 NOTIFY_ON_FILE_PROCESSED=True
 
+# Webhooks – Notify external systems via HTTP POST on document events.
+# Individual webhooks (URL, events, secret) are managed via /api/webhooks/.
+WEBHOOK_ENABLED=True
+
 # Uptime Kuma
 UPTIME_KUMA_URL=https://status.example.com/api/push/abcdef123456?status=up
-UPTIME_KUMA_PING_INTERVAL=5
+UPTIME_KUMA_PING_INTERVAL=5
+
+# Backup & Restore
+# Enable automatic scheduled backups (hourly, daily, weekly)
+BACKUP_ENABLED=True
+# Directory for local backup archives (defaults to <WORKDIR>/backups)
+# BACKUP_DIR=/data/backups
+# Optional remote destination: s3, dropbox, google_drive, onedrive, nextcloud, webdav, ftp, sftp, email
+# BACKUP_REMOTE_DESTINATION=s3
+# Sub-folder used when uploading backup archives to the remote destination
+BACKUP_REMOTE_FOLDER=backups
+# Retention: number of snapshots to keep per tier
+BACKUP_RETAIN_HOURLY=96   # 4 days of hourly snapshots
+BACKUP_RETAIN_DAILY=21    # 3 weeks of daily snapshots
+BACKUP_RETAIN_WEEKLY=13   # ~3 months of weekly snapshots
 
 # **Full-Text Search (Meilisearch)**
 # URL for the Meilisearch instance.
@@ -326,4 +438,43 @@ MEILISEARCH_URL=http://meilisearch:7700
 # Optional master/API key for secured Meilisearch instances
 # MEILISEARCH_API_KEY=your_master_key_here
 MEILISEARCH_INDEX_NAME=documents
-ENABLE_SEARCH=True
+ENABLE_SEARCH=True
+
+# **Duplicate Detection**
+# Exact duplicate detection (SHA-256) is always on during document processing.
+# The settings below control near-duplicate detection (same scanned content,
+# different hash) and the visibility of deduplication steps.
+ENABLE_DEDUPLICATION=True
+SHOW_DEDUPLICATION_STEP=True
+# Minimum cosine similarity score (0–1) for two documents to be flagged as
+# near-duplicates.  0.85 means 85 % semantic overlap.  Lower = more matches.
+NEAR_DUPLICATE_THRESHOLD=0.85
+
+# **PDF/A Archival Conversion**
+# When enabled, PDF/A copies of both the original ingested file and the processed
+# file are created and saved alongside the standard copies.  This may double or
+# triple storage but provides better legal coverage with time-stamped archival copies.
+# Uses ocrmypdf with Ghostscript for the conversion.
+ENABLE_PDFA_CONVERSION=false
+# PDF/A format variant: 1 = PDF/A-1b, 2 = PDF/A-2b (default), 3 = PDF/A-3b
+PDFA_FORMAT=2
+# Upload original-file PDF/A variant to all configured storage providers
+PDFA_UPLOAD_ORIGINAL=false
+# Upload processed-file PDF/A variant to all configured storage providers
+PDFA_UPLOAD_PROCESSED=false
+# Subfolder name appended to each provider's folder for PDF/A uploads
+# e.g. if Dropbox folder is '/Documents' this puts PDF/A files into '/Documents/pdfa'
+PDFA_UPLOAD_FOLDER=pdfa
+# Google Drive folder ID for PDF/A uploads (uses folder IDs, not paths)
+# Leave empty to use the same folder as regular uploads
+GOOGLE_DRIVE_PDFA_FOLDER_ID=
+# RFC 3161 timestamping of PDF/A files (creates .tsr proof-of-existence files)
+PDFA_TIMESTAMP_ENABLED=false
+# Timestamp Authority URL (default: FreeTSA, a free RFC 3161 TSA)
+PDFA_TIMESTAMP_URL=https://freetsa.org/tsr
+# Model used to generate text embeddings for document similarity.
+# Must be supported by your OpenAI-compatible API endpoint.
+EMBEDDING_MODEL=text-embedding-3-small
+# Maximum tokens to send to the embedding model.  Set below the model's
+# context window (e.g. 8000 for an 8192-token model).
+EMBEDDING_MAX_TOKENS=8000

.github/workflows/ci.yml

@@ -2,15 +2,10 @@ name: CI Pipeline
 
 on:
   push:
-    branches:
-      - main
-      - develop
-    tags:
-      - 'v*'
-      - '[0-9]+.*'
+    branches: [main, develop]
+    tags: ['v*', '[0-9]+.*']
   pull_request:
-    branches:
-      - main
+    branches: [main]
 
 permissions:
   contents: read
@@ -25,269 +20,145 @@ env:
 
 jobs:
   # ══════════════════════════════════════════════════════════════════════════
-  # Stage 1: Ruff Lint & Format (runs first to catch style issues early)
+  # Stage 1: Static Analysis (Fast Fail Gates)
   # ══════════════════════════════════════════════════════════════════════════
-
   lint:
     name: Ruff Lint & Format
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-
+      - uses: actions/checkout@v4
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: "3.11"
-
+          cache: 'pip'
       - name: Install Ruff
         run: pip install ruff
-
-      - name: Show Ruff version (debug)
-        run: ruff --version
-
       - name: Check for merge conflict markers
         run: |
           if git grep -rn -E '^(<{7} |>{7} |={7}$)' -- '.'; then
-            echo "ERROR: Merge conflict markers found in tracked files."
+            echo "ERROR: Merge conflict markers found."
             exit 1
           fi
-
-      - name: Run Ruff Lint (check)
-        # ruff check can --fix locally, but CI should only check (no modifications)
-        run: ruff check app/ tests/
-
-      - name: Run Ruff Format check
-        # ruff format only supports --check; do not pass --fix here
-        run: ruff format --check app/ tests/
-
-  # ══════════════════════════════════════════════════════════════════════════
-  # Stage 1b: HTML Accessibility Lint (catches a11y regressions early)
-  # ══════════════════════════════════════════════════════════════════════════
+      - run: ruff check app/ tests/
+      - run: ruff format --check app/ tests/
 
   html-lint:
     name: HTML Accessibility Lint
     runs-on: ubuntu-latest
     steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-
+      - uses: actions/checkout@v4
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: "3.11"
-
-      - name: Install djLint
-        run: pip install djlint>=1.36.0
-
-      - name: Lint HTML templates for accessibility
-        run: djlint frontend/templates/ --lint
+          cache: 'pip'
+      - run: pip install djlint>=1.36.0
+      - run: djlint frontend/templates/ --lint
 
   # ══════════════════════════════════════════════════════════════════════════
-  # Stage 2a: Dependency Vulnerability Scan (runs in parallel with lint)
+  # Stage 2: Parallel Heavy Lifters (Consolidated for Efficiency)
   # ══════════════════════════════════════════════════════════════════════════
+  
+  mypy:
+    name: Mypy Type Check
+    runs-on: ubuntu-latest
+    needs: [lint]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: 'pip'
+      - name: Install Dependencies
+        run: pip install -r requirements-dev.txt
+      - run: mypy app/
 
   dependency-scan:
-    name: Dependency Vulnerability Scan
+    name: Dependency Scan
     runs-on: ubuntu-latest
+    needs: [lint]
     steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-
+      - uses: actions/checkout@v4
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: "3.11"
+          cache: 'pip'
+      - run: pip install pip-audit>=2.7.0
+      - run: pip-audit -r requirements.txt --desc on
 
-      - name: Install pip-audit
-        run: pip install pip-audit>=2.7.0
-
-      - name: Run pip-audit on production dependencies
-        run: pip-audit -r requirements.txt --desc on
-
-      - name: Run pip-audit on dev dependencies
-        run: pip-audit -r requirements-dev.txt --desc on
-
-  # ══════════════════════════════════════════════════════════════════════════
-  # Stage 2b: Quick Tests (unit + basic integration — fast fail gate)
-  # ══════════════════════════════════════════════════════════════════════════
-
-  test-quick:
-    name: Quick Tests
+  run-tests:
+    name: Execute All Tests (Quick + Integration)
     runs-on: ubuntu-latest
-    timeout-minutes: 15
-    needs: [lint, html-lint, dependency-scan]
+    needs: [lint]
     services:
       redis:
         image: redis:7
-        ports:
-          - 6379:6379
-        options: >-
-          --health-cmd "redis-cli ping"
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
+        ports: ["6379:6379"]
+        options: --health-cmd "redis-cli ping" --health-interval 10s --health-timeout 5s --health-retries 5
+      rabbitmq:
+        image: rabbitmq:3-management
+        ports: ["5672:5672", "15672:15672"]
+        options: --health-cmd "rabbitmq-diagnostics -q ping" --health-interval 10s --health-timeout 5s --health-retries 5
     steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
+      - uses: actions/checkout@v4
 
       - name: Set up Python
         uses: actions/setup-python@v5
         with:
           python-version: "3.11"
+          cache: 'pip'
 
       - name: Install Dependencies
         run: |
           python -m pip install --upgrade pip
           pip install -r requirements-dev.txt
 
-      - name: Run Quick Tests
+      - name: Run Tests
         run: >
-          pytest tests/ -v
-          --timeout=120
-          --cov=app --cov-report=xml --cov-report=term
+          pytest tests/ -v --timeout=300
+          --cov=app --cov-report=xml:coverage.xml
           --junitxml=junit.xml -o junit_family=legacy
-          -m "not e2e and not requires_docker and not requires_external and not slow"
+          -m "not e2e"
 
-      - name: Upload coverage reports to Codecov
-        if: ${{ !cancelled() }}
+      - name: Upload Unified Coverage to Codecov
+        if: always()
         uses: codecov/codecov-action@v5
         with:
           token: ${{ secrets.CODECOV_TOKEN }}
           files: ./coverage.xml
-          fail_ci_if_error: false
-
-      - name: Upload test results to Codecov
-        if: ${{ !cancelled() }}
-        uses: codecov/codecov-action@v5
-        with:
-          token: ${{ secrets.CODECOV_TOKEN }}
-          files: ./junit.xml
-          report_type: test_results
-          fail_ci_if_error: false
-
-      - name: Upload test artifacts
-        if: ${{ !cancelled() }}
-        uses: actions/upload-artifact@v4
-        with:
-          name: test-results-quick
-          path: |
-            junit.xml
-            coverage.xml
+          fail_ci_if_error: true
 
   # ══════════════════════════════════════════════════════════════════════════
-  # Stage 2c: Integration Tests (Docker containers, external services)
+  # Stage 3: Build & Push (Quality Gate)
   # ══════════════════════════════════════════════════════════════════════════
-
-  test-integration:
-    name: Integration Tests
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    needs: [test-quick]  # Only run after quick tests pass (fail early)
-    services:
-      redis:
-        image: redis:7
-        ports:
-          - 6379:6379
-        options: >-
-          --health-cmd "redis-cli ping"
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-      rabbitmq:
-        image: rabbitmq:3-management
-        ports:
-          - 5672:5672
-          - 15672:15672
-        options: >-
-          --health-cmd "rabbitmq-diagnostics -q ping"
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      - name: Install Dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install -r requirements-dev.txt
-
-      - name: Run Integration Tests
-        run: >
-          pytest tests/ -v
-          --timeout=300
-          --junitxml=junit-integration.xml -o junit_family=legacy
-          -m "(requires_docker or requires_external or slow) and not e2e"
-
-      - name: Upload integration test results
-        if: ${{ !cancelled() }}
-        uses: actions/upload-artifact@v4
-        with:
-          name: test-results-integration
-          path: junit-integration.xml
-
-  mypy:
-    name: Mypy
-    runs-on: ubuntu-latest
-    needs: [lint, html-lint, dependency-scan]  # Wait for lint, HTML a11y lint, and dependency scan before running type checks
-    steps:
-      - name: Checkout Code
-        uses: actions/checkout@v4
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.11"
-
-      - name: Install Dependencies
-        run: |
-          python -m pip install --upgrade pip
-          pip install -r requirements-dev.txt
-
-      - name: Run Mypy
-        run: mypy app/
-
-  # ══════════════════════════════════════════════════════════════════════════
-  # Stage 3: Build & Push Docker Image (only after all Stage 2 jobs pass)
-  # ══════════════════════════════════════════════════════════════════════════
-
   build:
     name: Build & Push Docker Image
     runs-on: ubuntu-latest
-    needs: [test-quick, test-integration, lint, html-lint, mypy, dependency-scan]
+    needs: [run-tests, mypy, dependency-scan, html-lint]
     if: github.event_name == 'push'
-
     steps:
       - name: Checkout Code
         uses: actions/checkout@v4
-
       - name: Generate Build Metadata
         run: |
           chmod +x scripts/generate_build_metadata.sh
           ./scripts/generate_build_metadata.sh
-
       - name: Set up Docker Buildx
         uses: docker/setup-buildx-action@v3
-
       - name: Log in to Docker Hub
         uses: docker/login-action@v3
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
-
       - name: Log in to GitHub Container Registry
         uses: docker/login-action@v3
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
           password: ${{ secrets.GITHUB_TOKEN }}
-
       - name: Extract metadata for tags
         id: meta
         uses: docker/metadata-action@v5
@@ -299,48 +170,40 @@ jobs:
             type=ref,event=branch
             type=sha,prefix={{branch}}-
             type=semver,pattern={{version}}
-            type=semver,pattern={{major}}.{{minor}}
             type=raw,value=latest,enable={{is_default_branch}}
-
       - name: Build and Push Docker Image
         uses: docker/build-push-action@v6
         with:
           context: .
-          file: Dockerfile
-          platforms: linux/amd64
           push: true
-          sbom: true
-          provenance: mode=max
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
           cache-from: type=gha
           cache-to: type=gha,mode=max
+          sbom: true
+          provenance: mode=max
 
   # ══════════════════════════════════════════════════════════════════════════
-  # Stage 4: Update preprod K8s manifest (ArgoCD GitOps, only on main)
+  # Stage 4: GitOps Update
   # ══════════════════════════════════════════════════════════════════════════
-
   update-k8s-manifest:
     name: Update Preprod K8s Manifest
     runs-on: ubuntu-latest
     needs: [build]
     if: github.ref == 'refs/heads/main' && github.event_name == 'push'
-
     steps:
       - name: Compute image tag
         id: tag
         run: |
           SHORT_SHA=$(echo "${{ github.sha }}" | cut -c1-7)
-          echo "image=ghcr.io/${{ github.repository_owner }}/docuelevate:main-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
           echo "tag=main-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
-
+          echo "image=ghcr.io/${{ github.repository_owner }}/docuelevate:main-${SHORT_SHA}" >> "$GITHUB_OUTPUT"
       - name: Checkout k8s-cluster-state
         uses: actions/checkout@v4
         with:
           repository: christianlouis/k8s-cluster-state
           token: ${{ secrets.GH_PAT }}
           path: k8s-cluster-state
-
       - name: Update image tag in preprod manifest
         uses: mikefarah/yq@v4.44.6
         env:
@@ -349,7 +212,6 @@ jobs:
           cmd: |
             yq -i '(.. | select(tag == "!!str") | select(test("^(ghcr\\.io/christianlouis/docuelevate|christianlouis/docuelevate):"))) = strenv(IMAGE)' \
               k8s-cluster-state/apps/docuelevate/preprod/docuelevate-stack.yaml
-
       - name: Commit and push
         run: |
           cd k8s-cluster-state
@@ -357,7 +219,7 @@ jobs:
           git config user.email "github-actions[bot]@users.noreply.github.com"
           git add apps/docuelevate/preprod/docuelevate-stack.yaml
           if git diff --staged --quiet; then
-            echo "No changes to commit -- image tag already up to date"
+            echo "No changes to commit"
           else
             git commit -m "chore(preprod): update docuelevate image to ${{ steps.tag.outputs.tag }}"
             git push

.github/workflows/codeql.yml

@@ -24,7 +24,7 @@ jobs:
         include:
         - language: actions
           build-mode: none
-        - language: javascript-typescript
+        - language: javascript
           build-mode: none
         - language: python
           build-mode: none

.gitignore

@@ -171,6 +171,7 @@ venv.bak/
 
 # mkdocs documentation
 /site
+/docs_build
 
 # mypy
 .mypy_cache/

BUILD_DATE

@@ -1 +1 @@
-2026-03-01T17:23:25Z
+2026-03-08T12:15:34Z

Dockerfile

@@ -7,6 +7,22 @@ WORKDIR /app
 COPY requirements.txt /app/
 RUN pip install --no-cache-dir -r requirements.txt
 
+# ── Documentation build stage ───────────────────────────────────────────────
+FROM python:3.14.1-slim AS docs-builder
+
+WORKDIR /docs
+
+# Install MkDocs Material and its dependencies
+COPY docs/requirements.txt /docs/requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+# Copy documentation sources
+COPY docs /docs/docs
+COPY mkdocs.yml /docs/mkdocs.yml
+
+# Build the static documentation site
+RUN mkdocs build --config-file /docs/mkdocs.yml --site-dir /docs/docs_build
+
 # Second stage for the actual runtime
 FROM python:3.14.3-slim
 
@@ -33,6 +49,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # Copy application code
 COPY ./app /app/app
 COPY ./frontend /app/frontend
+COPY ./migrations /app/migrations
+COPY ./alembic.ini /app/alembic.ini
 COPY ./LICENSE /app/LICENSE
 
 # Copy build metadata files (generated at build time)
@@ -41,6 +59,9 @@ COPY ./BUILD_DATE /app/BUILD_DATE
 COPY ./GIT_SHA /app/GIT_SHA
 COPY ./RUNTIME_INFO /app/RUNTIME_INFO
 
+# Copy the pre-built MkDocs documentation site (served at /help)
+COPY --from=docs-builder /docs/docs_build /app/docs_build
+
 # Create runtime_info directory
 RUN mkdir -p /app/runtime_info
 

Dockerfile.local

@@ -6,6 +6,19 @@ WORKDIR /app
 COPY requirements.txt /app/
 RUN pip install --no-cache-dir -r requirements.txt
 
+# ── Documentation build stage ───────────────────────────────────────────────
+FROM python:3.14.1-slim AS docs-builder
+
+WORKDIR /docs
+
+COPY docs/requirements.txt /docs/requirements.txt
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY docs /docs/docs
+COPY mkdocs.yml /docs/mkdocs.yml
+
+RUN mkdocs build --config-file /docs/mkdocs.yml --site-dir /docs/docs_build
+
 FROM python:3.14.1-slim
 
 WORKDIR /app
@@ -27,10 +40,15 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 
 COPY ./app /app/app
 COPY ./frontend /app/frontend
+COPY ./migrations /app/migrations
+COPY ./alembic.ini /app/alembic.ini
 COPY ./LICENSE /app/LICENSE
 COPY ./VERSION /app/VERSION
 COPY ./BUILD_DATE /app/BUILD_DATE
 
+# Copy the pre-built MkDocs documentation site (served at /help)
+COPY --from=docs-builder /docs/docs_build /app/docs_build
+
 # Local fallbacks for build metadata
 RUN echo "local" > /app/GIT_SHA \
     && echo "local" > /app/RUNTIME_INFO

GIT_SHA

@@ -1 +1 @@
-0134ed3
+640662e

ROADMAP.md

@@ -7,7 +7,20 @@
 
 DocuElevate aims to be the premier open-source intelligent document processing platform, providing seamless integration with cloud storage providers, advanced AI-powered metadata extraction, and enterprise-grade security and scalability.
 
-## Current Status (v0.5.0)
+## Release Naming
+
+Each major milestone release carries a codename to anchor key project moments. These names appear in the status dashboard, build metadata, and changelog. For details, see [docs/ReleaseNaming.md](docs/ReleaseNaming.md).
+
+| Version Range | Codename      | Theme                                           |
+|---------------|---------------|--------------------------------------------------|
+| 0.5.x         | **Foundation** | Core platform, multi-provider storage, AI, UI   |
+| 0.6.x         | **Clarity**    | Enhanced search, filtering, UI/UX improvements  |
+| 0.7.x         | **Conductor**  | Workflow automation, pipelines, rule-based logic |
+| 1.0.x         | **Summit**     | Enterprise features, multi-tenancy, RBAC        |
+| 1.1.x         | **Bridge**     | Collaboration, sharing, analytics                |
+| 2.0.x         | **Horizon**    | On-premise AI, platform expansion               |
+
+## Current Status (v0.5.0 "Foundation")
 
 ### Core Features ✅
 - Multi-provider document storage (Dropbox, Google Drive, OneDrive, Nextcloud, S3, etc.)
@@ -23,7 +36,7 @@ DocuElevate aims to be the premier open-source intelligent document processing p
 - Celery-based async task processing
 - OAuth2 authentication via Authentik with admin group support
 
-## Short-term Goals (Q1-Q2 2026) - v0.4.x to v0.5.x
+## Short-term Goals (Q1-Q2 2026) - v0.4.x to v0.5.x "Foundation"
 
 ### Quality & Stability 🎯
 - **Test Coverage** (High Priority)
@@ -53,7 +66,7 @@ DocuElevate aims to be the premier open-source intelligent document processing p
   - [x] Integrate Docker builds with releases
 
 ### Features - v0.4.0
-- **Enhanced Search & Filtering**
+- **Enhanced Search & Filtering** → _preparing for v0.6.0 "Clarity"_
   - [ ] Full-text search across documents
   - [ ] Advanced filtering by metadata, tags, date ranges
   - [ ] Saved search queries
@@ -67,8 +80,8 @@ DocuElevate aims to be the premier open-source intelligent document processing p
   - [ ] Progress indicators for long-running tasks
   - [ ] Real-time notifications via WebSocket
 
-### Features - v0.5.0
-- **Workflow Automation**
+### Features - v0.5.0 "Foundation"
+- **Workflow Automation** → _evolving into v0.7.0 "Conductor"_
   - [ ] Custom processing pipelines
   - [ ] Conditional routing based on document type
   - [ ] Scheduled batch processing
@@ -82,9 +95,9 @@ DocuElevate aims to be the premier open-source intelligent document processing p
   - [ ] Automatic duplicate detection
   - [ ] Intelligent document splitting
 
-## Medium-term Goals (Q3-Q4 2026) - v1.0.x
+## Medium-term Goals (Q3-Q4 2026) - v1.0.x "Summit"
 
-### Enterprise Features - v1.0.0
+### Enterprise Features - v1.0.0 "Summit"
 - **Multi-tenancy**
   - [ ] Organization/team management
   - [ ] Role-based access control (RBAC)
@@ -106,7 +119,7 @@ DocuElevate aims to be the premier open-source intelligent document processing p
   - [ ] Custom webhook receivers
   - [ ] GraphQL API
 
-### Features - v1.1.0
+### Features - v1.1.0 "Bridge"
 - **Collaboration**
   - [ ] Document sharing with expiring links
   - [ ] Comments and annotations
@@ -121,7 +134,7 @@ DocuElevate aims to be the premier open-source intelligent document processing p
   - [ ] Cost analysis per provider
   - [ ] Export reports (PDF, CSV, Excel)
 
-## Long-term Goals (2027+) - v2.0+
+## Long-term Goals (2027+) - v2.0+ "Horizon"
 
 ### Strategic Initiatives
 - **On-Premise AI Models**

RUNTIME_INFO

@@ -1,10 +1,10 @@
 DocuElevate Build Information
 ==============================
-Version: 0.67.2
-Build Date: 2026-03-01T17:23:25Z
-Git Commit: 0134ed37d5c602faf5b10cc6a7229263ba2f6aa1
-Git Short SHA: 0134ed3
+Version: 0.93.1
+Build Date: 2026-03-08T12:15:34Z
+Git Commit: 640662e9b265d22885adf9cab5699399624bc0a1
+Git Short SHA: 640662e
 Git Branch: main
-Commit Date: 2026-03-01T18:23:06+01:00
-Build Timestamp: 2026-03-01T17:23:25Z
+Commit Date: 2026-03-08T13:15:14+01:00
+Build Timestamp: 2026-03-08T12:15:34Z
 ==============================

VERSION

@@ -1 +1 @@
-0.67.2
+0.93.1

app/api/__init__.py

@@ -6,23 +6,34 @@ import logging
 
 from fastapi import APIRouter
 
+from app.api.admin_users import router as admin_users_router
 from app.api.azure import router as azure_router
+from app.api.backup import router as backup_router
+from app.api.billing import router as billing_router
+from app.api.database import router as database_router
 from app.api.diagnostic import router as diagnostic_router
 from app.api.dropbox import router as dropbox_router
+from app.api.duplicates import router as duplicates_router
 from app.api.files import router as files_router
 from app.api.google_drive import router as google_drive_router
 from app.api.logs import router as logs_router
+from app.api.onboarding import router as onboarding_router
 from app.api.onedrive import router as onedrive_router
 from app.api.openai import router as openai_router
+from app.api.pipelines import router as pipelines_router
+from app.api.plans import router as plans_router
 from app.api.process import router as process_router
 from app.api.queue import router as queue_router
 from app.api.saved_searches import router as saved_searches_router
 from app.api.search import router as search_router
 from app.api.settings import router as settings_router
+from app.api.similarity import router as similarity_router
+from app.api.subscriptions import router as subscriptions_router
 from app.api.url_upload import router as url_upload_router
 
 # Import all the individual routers
 from app.api.user import router as user_router
+from app.api.webhooks import router as webhooks_router
 
 # Set up logging
 logger = logging.getLogger(__name__)
@@ -31,7 +42,9 @@ logger = logging.getLogger(__name__)
 router = APIRouter()
 
 # Include all the routers
+router.include_router(admin_users_router)
 router.include_router(user_router)
+router.include_router(backup_router)
 router.include_router(files_router)
 router.include_router(process_router)
 router.include_router(diagnostic_router)
@@ -46,3 +59,12 @@ router.include_router(url_upload_router)
 router.include_router(search_router)
 router.include_router(queue_router)
 router.include_router(saved_searches_router)
+router.include_router(similarity_router)
+router.include_router(duplicates_router)
+router.include_router(webhooks_router)
+router.include_router(database_router)
+router.include_router(subscriptions_router)
+router.include_router(plans_router)
+router.include_router(onboarding_router)
+router.include_router(billing_router)
+router.include_router(pipelines_router)

app/api/admin_users.py

@@ -0,0 +1,668 @@
+"""API endpoints for admin user management.
+
+Provides CRUD operations for user profiles and aggregate statistics so that
+administrators can inspect, configure, and manage users in multi-user mode.
+Also provides endpoints for admins to create and manage local (email/password)
+user accounts directly, without requiring email verification.
+"""
+
+import logging
+from datetime import datetime, timezone
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from pydantic import BaseModel, Field
+from sqlalchemy import func
+from sqlalchemy.orm import Session
+
+from app.config import settings
+from app.database import get_db
+from app.models import FileRecord, LocalUser, UserProfile
+from app.utils.local_auth import generate_token, hash_password, send_password_reset_email
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/admin/users", tags=["admin-users"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+# ---------------------------------------------------------------------------
+# Auth helper
+# ---------------------------------------------------------------------------
+
+
+def _require_admin(request: Request) -> dict:
+    """Ensure the caller is an admin. Raises 403 otherwise."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+    return user
+
+
+AdminUser = Annotated[dict, Depends(_require_admin)]
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class UserProfileUpsert(BaseModel):
+    """Body for creating or updating a user profile."""
+
+    display_name: str | None = Field(default=None, max_length=255, description="Human-readable display name")
+    daily_upload_limit: int | None = Field(
+        default=None, ge=0, description="Per-user daily upload cap; null = use global default"
+    )
+    notes: str | None = Field(default=None, max_length=4096, description="Admin notes about this user")
+    is_blocked: bool = Field(default=False, description="Block this user from uploading")
+    subscription_tier: str | None = Field(
+        default="free",
+        description="Subscription tier: free | starter | professional | business",
+    )
+    subscription_billing_cycle: str = Field(default="monthly", pattern="^(monthly|yearly)$")
+    subscription_period_start: datetime | None = None
+    allow_overage: bool = False
+    is_complimentary: bool = Field(
+        default=False,
+        description="When True the user is on a complimentary (uncharged) plan — they keep all tier "
+        "quota benefits but are never billed via Stripe.",
+    )
+
+
+class PaymentIssueBody(BaseModel):
+    """Body for reporting a payment issue for a user."""
+
+    issue: str = Field(..., min_length=1, max_length=2048, description="Description of the payment issue")
+
+
+class UserProfileResponse(BaseModel):
+    """Response schema for a user profile record."""
+
+    id: int
+    user_id: str
+    display_name: str | None
+    daily_upload_limit: int | None
+    notes: str | None
+    is_blocked: bool
+    subscription_tier: str | None
+    subscription_billing_cycle: str
+    subscription_period_start: str | None
+    allow_overage: bool
+    is_complimentary: bool
+    created_at: str | None
+    updated_at: str | None
+
+    model_config = {"from_attributes": True}
+
+
+class UserSummary(BaseModel):
+    """Per-user summary combining profile data with document statistics."""
+
+    user_id: str
+    display_name: str | None
+    daily_upload_limit: int | None
+    notes: str | None
+    is_blocked: bool
+    subscription_tier: str | None
+    subscription_billing_cycle: str | None
+    subscription_period_start: str | None
+    allow_overage: bool
+    is_complimentary: bool
+    profile_id: int | None
+    document_count: int
+    last_upload: str | None
+
+
+class LocalUserCreate(BaseModel):
+    """Body for admin-creating a local (email/password) user account."""
+
+    email: str = Field(..., max_length=255, description="Email address for the new user")
+    username: str = Field(..., min_length=3, max_length=64, pattern=r"^[a-zA-Z0-9_-]+$")
+    display_name: str | None = Field(default=None, max_length=255)
+    password: str = Field(..., min_length=8, max_length=128)
+    is_admin: bool = Field(default=False, description="Grant admin privileges")
+
+
+class LocalUserUpdate(BaseModel):
+    """Body for admin-updating a local (email/password) user account."""
+
+    email: str | None = Field(default=None, max_length=255, description="New email address")
+    display_name: str | None = Field(default=None, max_length=255, description="New display name")
+    is_admin: bool | None = Field(default=None, description="Grant or revoke admin privileges")
+    is_active: bool | None = Field(default=None, description="Activate or deactivate the account")
+
+
+class LocalUserSetPassword(BaseModel):
+    """Body for admin setting a temporary password for a local user."""
+
+    password: str = Field(..., min_length=8, max_length=128, description="New temporary password")
+
+
+class LocalUserResponse(BaseModel):
+    """Summary of a local user account."""
+
+    id: int
+    email: str
+    username: str
+    display_name: str | None
+    is_active: bool
+    is_admin: bool
+    created_at: str | None
+
+    model_config = {"from_attributes": True}
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _get_or_none(db: Session, user_id: str) -> UserProfile | None:
+    """Return the UserProfile row for *user_id*, or None if it doesn't exist."""
+    return db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+
+
+def _profile_to_dict(profile: UserProfile) -> dict[str, Any]:
+    return {
+        "id": profile.id,
+        "user_id": profile.user_id,
+        "display_name": profile.display_name,
+        "daily_upload_limit": profile.daily_upload_limit,
+        "notes": profile.notes,
+        "is_blocked": profile.is_blocked,
+        "subscription_tier": profile.subscription_tier or "free",
+        "subscription_billing_cycle": profile.subscription_billing_cycle or "monthly",
+        "subscription_period_start": profile.subscription_period_start.isoformat()
+        if profile.subscription_period_start
+        else None,
+        "allow_overage": bool(profile.allow_overage),
+        "is_complimentary": bool(profile.is_complimentary),
+        "created_at": profile.created_at.isoformat() if profile.created_at else None,
+        "updated_at": profile.updated_at.isoformat() if profile.updated_at else None,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/", summary="List all known users with statistics")
+def list_users(
+    db: DbSession,
+    _admin: AdminUser,
+    q: str = Query("", description="Filter by user_id substring (case-insensitive)"),
+    page: int = Query(1, ge=1, description="Page number"),
+    per_page: int = Query(25, ge=1, le=100, description="Items per page"),
+) -> dict[str, Any]:
+    """Return every distinct user_id that has at least one document or an explicit profile,
+    enriched with aggregate document statistics and the admin-managed profile.
+
+    Supports substring filtering (``q``) and pagination.
+    """
+    # 1. Collect every distinct owner_id from documents
+    doc_stats_query = (
+        db.query(
+            FileRecord.owner_id.label("user_id"),
+            func.count(FileRecord.id).label("doc_count"),
+            func.max(FileRecord.created_at).label("last_upload"),
+        )
+        .filter(FileRecord.owner_id.isnot(None))
+        .group_by(FileRecord.owner_id)
+    )
+
+    # 2. Collect all user_ids that have explicit profiles (may not have docs yet)
+    profile_query = db.query(UserProfile)
+
+    # Build a unified set of user_ids
+    doc_rows = {row.user_id: row for row in doc_stats_query.all()}
+    profile_rows = {p.user_id: p for p in profile_query.all()}
+
+    all_user_ids = set(doc_rows.keys()) | set(profile_rows.keys())
+
+    # Apply optional substring filter
+    if q.strip():
+        q_lower = q.strip().lower()
+        all_user_ids = {uid for uid in all_user_ids if q_lower in uid.lower()}
+
+    # Sort and paginate
+    sorted_ids = sorted(all_user_ids)
+    total = len(sorted_ids)
+    start = (page - 1) * per_page
+    page_ids = sorted_ids[start : start + per_page]
+
+    users: list[dict[str, Any]] = []
+    for uid in page_ids:
+        doc_row = doc_rows.get(uid)
+        profile = profile_rows.get(uid)
+        users.append(
+            {
+                "user_id": uid,
+                "display_name": profile.display_name if profile else None,
+                "daily_upload_limit": profile.daily_upload_limit if profile else None,
+                "notes": profile.notes if profile else None,
+                "is_blocked": profile.is_blocked if profile else False,
+                "subscription_tier": (profile.subscription_tier or "free") if profile else "free",
+                "subscription_billing_cycle": (profile.subscription_billing_cycle or "monthly")
+                if profile
+                else "monthly",
+                "subscription_period_start": profile.subscription_period_start.isoformat()
+                if (profile and profile.subscription_period_start)
+                else None,
+                "allow_overage": bool(profile.allow_overage) if profile else False,
+                "is_complimentary": bool(profile.is_complimentary) if profile else False,
+                "profile_id": profile.id if profile else None,
+                "document_count": doc_row.doc_count if doc_row else 0,
+                "last_upload": doc_row.last_upload.isoformat() if (doc_row and doc_row.last_upload) else None,
+            }
+        )
+
+    return {
+        "users": users,
+        "total": total,
+        "page": page,
+        "per_page": per_page,
+        "pages": max(1, (total + per_page - 1) // per_page),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Local user management (admin-only)
+# ---------------------------------------------------------------------------
+# NOTE: These routes MUST be defined before /{user_id:path} to avoid being
+# swallowed by the catch-all path parameter.
+# ---------------------------------------------------------------------------
+
+
+@router.get("/local", summary="List all local (email/password) user accounts")
+def list_local_users(db: DbSession, _admin: AdminUser) -> list[dict[str, Any]]:
+    """Return every local user account with basic metadata."""
+    users = db.query(LocalUser).order_by(LocalUser.created_at.desc()).all()
+    return [
+        {
+            "id": u.id,
+            "email": u.email,
+            "username": u.username,
+            "display_name": u.display_name,
+            "is_active": u.is_active,
+            "is_admin": u.is_admin,
+            "created_at": u.created_at.isoformat() if u.created_at else None,
+        }
+        for u in users
+    ]
+
+
+@router.post("/local", status_code=status.HTTP_201_CREATED, summary="Create a local user account")
+def create_local_user(body: LocalUserCreate, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Create a new local (email/password) user account.
+
+    The account is immediately active — no email verification is required when
+    created by an administrator.  A matching UserProfile row is also created.
+
+    Raises:
+        409: Email or username already registered.
+    """
+    if db.query(LocalUser).filter(LocalUser.email == body.email).first():
+        raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Email already registered.")
+    if db.query(LocalUser).filter(LocalUser.username == body.username).first():
+        raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Username already taken.")
+
+    user = LocalUser(
+        email=body.email,
+        username=body.username,
+        display_name=body.display_name,
+        hashed_password=hash_password(body.password),
+        is_active=True,
+        is_admin=body.is_admin,
+    )
+    db.add(user)
+
+    # Ensure a UserProfile exists for the new user
+    if not db.query(UserProfile).filter(UserProfile.user_id == body.email).first():
+        db.add(UserProfile(user_id=body.email, display_name=body.display_name or body.username))
+
+    try:
+        db.commit()
+        db.refresh(user)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Admin created local user account: %s", body.email)
+    return {
+        "id": user.id,
+        "email": user.email,
+        "username": user.username,
+        "display_name": user.display_name,
+        "is_active": user.is_active,
+        "is_admin": user.is_admin,
+        "created_at": user.created_at.isoformat() if user.created_at else None,
+    }
+
+
+@router.delete(
+    "/local/{local_user_id}",
+    status_code=status.HTTP_204_NO_CONTENT,
+    summary="Delete a local user account",
+)
+def delete_local_user(local_user_id: int, db: DbSession, _admin: AdminUser) -> None:
+    """Delete a local user account by its numeric ID.
+
+    The associated UserProfile is also removed.  Documents owned by this user
+    are **not** deleted.
+    """
+    user = db.query(LocalUser).filter(LocalUser.id == local_user_id).first()
+    if not user:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Local user not found.")
+
+    # Remove associated profile if present
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user.email).first()
+    if profile:
+        db.delete(profile)
+
+    try:
+        db.delete(user)
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Admin deleted local user account: %s", user.email)
+
+
+@router.patch("/local/{local_user_id}", summary="Update a local user account")
+def update_local_user(local_user_id: int, body: LocalUserUpdate, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Update the email address, display name, admin flag, or active status of a local user account.
+
+    Only fields explicitly provided (non-None) are modified.  If the email is changed
+    the associated UserProfile row is also updated to keep ``user_id`` in sync.
+
+    Raises:
+        404: Local user not found.
+        409: The new email is already taken by another account.
+    """
+    user = db.query(LocalUser).filter(LocalUser.id == local_user_id).first()
+    if not user:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Local user not found.")
+
+    old_email = user.email
+
+    if body.email is not None and body.email != user.email:
+        if db.query(LocalUser).filter(LocalUser.email == body.email, LocalUser.id != local_user_id).first():
+            raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Email already registered.")
+        user.email = body.email
+
+    if body.display_name is not None:
+        # Normalise empty string to None so that clearing the field removes the display name
+        user.display_name = body.display_name or None
+
+    if body.is_admin is not None:
+        user.is_admin = body.is_admin
+
+    if body.is_active is not None:
+        user.is_active = body.is_active
+
+    try:
+        db.flush()
+        # Keep UserProfile.user_id in sync when email changes
+        if body.email is not None and body.email != old_email:
+            profile = db.query(UserProfile).filter(UserProfile.user_id == old_email).first()
+            if profile:
+                profile.user_id = body.email
+        db.commit()
+        db.refresh(user)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Admin updated local user %s (id=%d)", user.email, user.id)
+    return {
+        "id": user.id,
+        "email": user.email,
+        "username": user.username,
+        "display_name": user.display_name,
+        "is_active": user.is_active,
+        "is_admin": user.is_admin,
+        "created_at": user.created_at.isoformat() if user.created_at else None,
+    }
+
+
+@router.post(
+    "/local/{local_user_id}/send-password-reset",
+    status_code=status.HTTP_200_OK,
+    summary="Send a password reset email to a local user",
+)
+def admin_send_password_reset(local_user_id: int, request: Request, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Generate a password reset token and email the reset link to the local user.
+
+    This is a last-resort tool for admins to help users who are locked out.
+    Returns ``{"sent": true}`` on success and ``{"sent": false, "reason": "..."}`` when
+    SMTP is not configured or sending fails.
+
+    Raises:
+        404: Local user not found.
+    """
+    user = db.query(LocalUser).filter(LocalUser.id == local_user_id).first()
+    if not user:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Local user not found.")
+
+    if not settings.email_host:
+        logger.warning("Admin requested password reset for %s but SMTP is not configured", user.email)
+        return {"sent": False, "reason": "SMTP is not configured on this server."}
+
+    token = generate_token()
+    user.password_reset_token = token
+    user.password_reset_sent_at = datetime.now(tz=timezone.utc)
+    db.commit()
+
+    base_url = str(request.base_url).rstrip("/")
+    try:
+        send_password_reset_email(user.email, user.username, token, base_url)
+    except Exception as exc:
+        logger.warning("Admin-triggered password reset email failed for %s: %s", user.email, exc)
+        return {"sent": False, "reason": str(exc)}
+
+    logger.info("[SECURITY] ADMIN_PASSWORD_RESET_EMAIL user=%s admin=%s", user.email, _admin.get("email", "unknown"))
+    return {"sent": True, "email": user.email}
+
+
+@router.post(
+    "/local/{local_user_id}/set-password",
+    status_code=status.HTTP_200_OK,
+    summary="Set a temporary password for a local user account",
+)
+def admin_set_password(
+    local_user_id: int, body: LocalUserSetPassword, db: DbSession, _admin: AdminUser
+) -> dict[str, Any]:
+    """Directly set a new password for a local user without requiring an email token.
+
+    Use this as a last resort when email delivery is unavailable.  The user
+    should be advised to change their password after logging in.
+
+    Raises:
+        404: Local user not found.
+    """
+    user = db.query(LocalUser).filter(LocalUser.id == local_user_id).first()
+    if not user:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Local user not found.")
+
+    user.hashed_password = hash_password(body.password)
+    # Clear any outstanding reset tokens and activate the account so the user
+    # can log in immediately after an admin sets their password.
+    user.password_reset_token = None
+    user.password_reset_sent_at = None
+    user.is_active = True
+
+    try:
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("[SECURITY] ADMIN_SET_PASSWORD user=%s admin=%s", user.email, _admin.get("email", "unknown"))
+    return {"updated": True, "email": user.email}
+
+
+@router.get("/{user_id:path}", summary="Get details for a single user")
+def get_user(user_id: str, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Return profile and document statistics for a specific user."""
+    doc_count = db.query(func.count(FileRecord.id)).filter(FileRecord.owner_id == user_id).scalar() or 0
+    last_row = (
+        db.query(FileRecord.created_at)
+        .filter(FileRecord.owner_id == user_id)
+        .order_by(FileRecord.created_at.desc())
+        .first()
+    )
+    last_upload = last_row[0].isoformat() if last_row and last_row[0] else None
+
+    profile = _get_or_none(db, user_id)
+
+    return {
+        "user_id": user_id,
+        "display_name": profile.display_name if profile else None,
+        "daily_upload_limit": profile.daily_upload_limit if profile else None,
+        "notes": profile.notes if profile else None,
+        "is_blocked": profile.is_blocked if profile else False,
+        "subscription_tier": (profile.subscription_tier or "free") if profile else "free",
+        "subscription_billing_cycle": (profile.subscription_billing_cycle or "monthly") if profile else "monthly",
+        "subscription_period_start": profile.subscription_period_start.isoformat()
+        if (profile and profile.subscription_period_start)
+        else None,
+        "allow_overage": bool(profile.allow_overage) if profile else False,
+        "is_complimentary": bool(profile.is_complimentary) if profile else False,
+        "profile_id": profile.id if profile else None,
+        "document_count": doc_count,
+        "last_upload": last_upload,
+        "profile": _profile_to_dict(profile) if profile else None,
+    }
+
+
+@router.put("/{user_id:path}", summary="Create or update a user profile")
+def upsert_user_profile(
+    user_id: str,
+    body: UserProfileUpsert,
+    db: DbSession,
+    _admin: AdminUser,
+) -> dict[str, Any]:
+    """Create a new profile or update an existing one for *user_id*.
+
+    Returns the persisted profile.
+    """
+    profile = _get_or_none(db, user_id)
+    if profile is None:
+        profile = UserProfile(user_id=user_id)
+        db.add(profile)
+
+    old_tier = (profile.subscription_tier or "free") if profile.id else None  # None means brand-new profile
+    profile.display_name = body.display_name
+    profile.daily_upload_limit = body.daily_upload_limit
+    profile.notes = body.notes
+    profile.is_blocked = body.is_blocked
+    profile.subscription_billing_cycle = body.subscription_billing_cycle
+    profile.subscription_period_start = body.subscription_period_start
+    profile.allow_overage = body.allow_overage
+    profile.is_complimentary = body.is_complimentary
+    tier_changed = False
+    new_tier: str | None = None
+    if body.subscription_tier is not None:
+        from app.utils.subscription import TIERS
+
+        if body.subscription_tier not in TIERS:
+            raise HTTPException(
+                status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+                detail=f"Invalid subscription_tier '{body.subscription_tier}'. Valid values: {list(TIERS.keys())}",
+            )
+        # Detect a real change only for existing profiles (old_tier is not None)
+        if old_tier is not None and old_tier != body.subscription_tier:
+            tier_changed = True
+            new_tier = body.subscription_tier
+        profile.subscription_tier = body.subscription_tier
+
+    try:
+        db.commit()
+        db.refresh(profile)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Admin upserted profile for user %s", user_id)
+
+    # Notify admins and fire webhook when plan is changed by an admin
+    if tier_changed and new_tier is not None:
+        try:
+            from app.utils.notification import notify_plan_changed
+            from app.utils.webhook import dispatch_webhook_event
+
+            notify_plan_changed(user_id, old_tier=old_tier, new_tier=new_tier, changed_by="admin")  # type: ignore[arg-type]
+            dispatch_webhook_event(
+                "user.plan_changed",
+                {
+                    "user_id": user_id,
+                    "old_tier": old_tier,
+                    "new_tier": new_tier,
+                    "billing_cycle": body.subscription_billing_cycle,
+                    "changed_by": "admin",
+                },
+            )
+        except Exception:
+            logger.exception("Failed to send plan-change notification/webhook for user %s", user_id)
+
+    return _profile_to_dict(profile)
+
+
+@router.post(
+    "/{user_id:path}/payment-issue", status_code=status.HTTP_200_OK, summary="Report a payment issue for a user"
+)
+def report_payment_issue(user_id: str, body: PaymentIssueBody, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Notify admins and fire a webhook for a payment issue reported against *user_id*.
+
+    The user profile must exist.  Use this endpoint when a payment processor
+    webhook or manual review identifies a billing problem (e.g. failed charge,
+    expired card, disputed transaction).
+
+    Returns the user profile dict alongside an acknowledgement flag.
+    """
+    profile = _get_or_none(db, user_id)
+    if not profile:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User profile not found")
+
+    logger.warning("Payment issue reported for user %s: %s", user_id, body.issue)
+
+    try:
+        from app.utils.notification import notify_payment_issue
+        from app.utils.webhook import dispatch_webhook_event
+
+        notify_payment_issue(user_id, issue=body.issue)
+        dispatch_webhook_event(
+            "user.payment_issue",
+            {
+                "user_id": user_id,
+                "issue": body.issue,
+            },
+        )
+    except Exception:
+        logger.exception("Failed to send payment-issue notification/webhook for user %s", user_id)
+
+    return {"acknowledged": True, "user_id": user_id, "profile": _profile_to_dict(profile)}
+
+
+@router.delete("/{user_id:path}", status_code=status.HTTP_204_NO_CONTENT, summary="Delete a user profile")
+def delete_user_profile(user_id: str, db: DbSession, _admin: AdminUser) -> None:
+    """Delete the admin-managed profile for *user_id*.
+
+    Documents owned by this user are **not** removed; only the profile record
+    is deleted.  To reassign or purge documents use the files API.
+    """
+    profile = _get_or_none(db, user_id)
+    if not profile:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="User profile not found")
+
+    try:
+        db.delete(profile)
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Admin deleted profile for user %s", user_id)

app/api/backup.py

@@ -0,0 +1,253 @@
+"""
+Backup and restore API endpoints for DocuElevate.
+
+Provides REST endpoints for:
+- Listing existing backups
+- Triggering a manual backup
+- Downloading a backup archive
+- Restoring from an uploaded backup file
+- Deleting a backup record
+- Running retention cleanup
+"""
+
+import logging
+import os
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, HTTPException, Request, UploadFile, status
+from fastapi.responses import FileResponse
+from sqlalchemy.orm import Session
+
+from app.database import get_db
+from app.models import BackupRecord
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/admin/backup", tags=["backup"])
+
+
+def _require_admin(request: Request) -> dict:
+    """Ensure the caller is an admin.  Raises 403 otherwise."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+    return user
+
+
+# Annotated shorthand so FastAPI can resolve and tests can override it.
+AdminUser = Annotated[dict, Depends(_require_admin)]
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/")
+async def list_backups(
+    _admin: AdminUser,
+    db: Session = Depends(get_db),
+) -> list[dict]:
+    """Return all backup records, newest first."""
+    records = db.query(BackupRecord).order_by(BackupRecord.created_at.desc()).all()
+    return [
+        {
+            "id": r.id,
+            "filename": r.filename,
+            "backup_type": r.backup_type,
+            "size_bytes": r.size_bytes,
+            "checksum": r.checksum,
+            "status": r.status,
+            "local_path": r.local_path,
+            "remote_destination": r.remote_destination,
+            "remote_path": r.remote_path,
+            "created_at": r.created_at.isoformat() if r.created_at else None,
+            "local_available": bool(r.local_path and os.path.exists(r.local_path)),
+        }
+        for r in records
+    ]
+
+
+@router.post("/create")
+async def trigger_backup(
+    _admin: AdminUser,
+    backup_type: str = "hourly",
+) -> dict:
+    """Trigger a manual backup immediately.
+
+    Query parameter ``backup_type`` accepts ``hourly``, ``daily``, or
+    ``weekly`` (default: ``hourly``).
+    """
+    if backup_type not in ("hourly", "daily", "weekly"):
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid backup_type")
+
+    from app.tasks.backup_tasks import create_backup
+
+    task = create_backup.delay(backup_type=backup_type)
+    return {"task_id": task.id, "status": "queued", "backup_type": backup_type}
+
+
+@router.get("/{backup_id}/download")
+async def download_backup(
+    backup_id: int,
+    _admin: AdminUser,
+    db: Session = Depends(get_db),
+) -> FileResponse:
+    """Stream the backup archive to the client."""
+    rec = db.get(BackupRecord, backup_id)
+    if rec is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Backup not found")
+    if not rec.local_path or not os.path.exists(rec.local_path):
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="Local archive file is not available (may have been pruned)",
+        )
+    return FileResponse(
+        path=rec.local_path,
+        filename=rec.filename,
+        media_type="application/gzip",
+    )
+
+
+@router.post("/restore")
+async def restore_backup(
+    _admin: AdminUser,
+    file: UploadFile,
+    db: Session = Depends(get_db),
+) -> dict:
+    """Restore the database from an uploaded gzip-compressed SQL dump.
+
+    **Warning**: This overwrites the current database contents.
+
+    Supported formats (must match the currently configured database backend):
+
+    - ``*.db.gz``    – gzip-compressed SQLite ``.dump()`` SQL script (SQLite backend)
+    - ``*.pgsql.gz`` – gzip-compressed ``pg_dump --format=plain`` output (PostgreSQL backend)
+    - ``*.mysql.gz`` – gzip-compressed ``mysqldump`` output (MySQL / MariaDB backend)
+    """
+    import tempfile
+    from pathlib import Path
+
+    from sqlalchemy.engine.url import make_url
+
+    from app.config import settings as app_settings
+    from app.tasks.backup_tasks import (
+        _archive_ext_for_backend,
+        _db_path,
+        _restore_mysql,
+        _restore_postgresql,
+        _restore_sqlite,
+    )
+
+    url = make_url(app_settings.database_url)
+    backend = url.get_backend_name()
+    expected_ext = _archive_ext_for_backend(backend)
+
+    if not file.filename or not file.filename.endswith(expected_ext):
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=(
+                f"Uploaded file must be a '{expected_ext}' backup archive for the current database backend ({backend})."
+            ),
+        )
+
+    # Write upload to a temp file
+    with tempfile.NamedTemporaryFile(suffix=expected_ext, delete=False) as tmp:
+        tmp_path = Path(tmp.name)
+        content = await file.read()
+        tmp.write(content)
+
+    try:
+        if backend == "sqlite":
+            db_path = _db_path()
+            if db_path is None:
+                raise HTTPException(
+                    status_code=status.HTTP_400_BAD_REQUEST,
+                    detail="Restore is only supported for file-based SQLite databases.",
+                )
+            # Close the application DB session before replacing the file
+            db.close()
+            try:
+                _restore_sqlite(db_path, tmp_path)
+            except ValueError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_400_BAD_REQUEST,
+                    detail=str(exc),
+                ) from exc
+            except RuntimeError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+                    detail=str(exc),
+                ) from exc
+
+        elif backend == "postgresql":
+            db.close()
+            try:
+                _restore_postgresql(app_settings.database_url, tmp_path)
+            except FileNotFoundError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+                    detail=f"psql binary not found – is PostgreSQL client installed? ({exc})",
+                ) from exc
+            except RuntimeError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+                    detail=f"PostgreSQL restore failed: {exc}",
+                ) from exc
+
+        elif backend == "mysql":
+            db.close()
+            try:
+                _restore_mysql(app_settings.database_url, tmp_path)
+            except FileNotFoundError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+                    detail=f"mysql binary not found – is MySQL client installed? ({exc})",
+                ) from exc
+            except RuntimeError as exc:
+                raise HTTPException(
+                    status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+                    detail=f"MySQL restore failed: {exc}",
+                ) from exc
+
+        else:
+            raise HTTPException(
+                status_code=status.HTTP_400_BAD_REQUEST,
+                detail=f"Database backend '{backend}' does not support restore.",
+            )
+
+    finally:
+        tmp_path.unlink(missing_ok=True)
+
+    logger.info(f"Database restored from uploaded backup: {file.filename}")
+    return {"status": "restored", "filename": file.filename}
+
+
+@router.delete("/{backup_id}")
+async def delete_backup(
+    backup_id: int,
+    _admin: AdminUser,
+    db: Session = Depends(get_db),
+) -> dict:
+    """Delete a backup record (and local file if present)."""
+    rec = db.get(BackupRecord, backup_id)
+    if rec is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Backup not found")
+
+    if rec.local_path and os.path.exists(rec.local_path):
+        try:
+            os.remove(rec.local_path)
+        except OSError as exc:
+            logger.warning(f"Could not remove local backup file {rec.local_path}: {exc}")
+
+    db.delete(rec)
+    db.commit()
+    return {"status": "deleted", "id": backup_id}
+
+
+@router.post("/cleanup")
+async def run_cleanup(_admin: AdminUser) -> dict:
+    """Manually trigger the retention cleanup for all backup tiers."""
+    from app.tasks.backup_tasks import cleanup_old_backups
+
+    task = cleanup_old_backups.delay()
+    return {"task_id": task.id, "status": "queued"}

app/api/billing.py

@@ -0,0 +1,418 @@
+"""Stripe billing integration for DocuElevate.
+
+Provides three endpoints:
+- POST /api/billing/create-checkout-session — starts Stripe Checkout for a plan upgrade
+- POST /api/billing/create-portal-session   — opens Stripe Customer Portal (manage/cancel)
+- POST /api/billing/webhook                 — handles Stripe webhook events
+- GET  /api/billing/success                 — success landing page after checkout
+
+Stripe Python SDK license: MIT (compatible with this project's Apache 2.0 license).
+
+GDPR:  Stripe acts as a data processor under a Data Processing Agreement (DPA).
+       Stripe is SOC 2 Type II certified and supports EU data residency.
+SOC2:  Stripe is SOC 2 Type II certified.
+EU VAT: Configure Stripe Tax in the Stripe Dashboard for automatic VAT collection.
+"""
+
+import json
+import logging
+import pathlib
+from datetime import datetime, timezone
+from typing import Any
+
+import stripe
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from fastapi.templating import Jinja2Templates
+from pydantic import BaseModel
+from sqlalchemy.orm import Session
+
+from app.auth import require_login
+from app.config import settings
+from app.database import get_db
+from app.models import SubscriptionPlan, UserProfile
+from app.utils.user_scope import get_current_owner_id
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/billing", tags=["billing"])
+
+_templates_dir = pathlib.Path(__file__).parents[2] / "frontend" / "templates"
+_templates = Jinja2Templates(directory=str(_templates_dir))
+
+
+def _get_stripe() -> stripe.StripeClient | None:
+    """Return a configured Stripe client, or None when not configured."""
+    if not settings.stripe_secret_key:
+        return None
+    return stripe.StripeClient(settings.stripe_secret_key)
+
+
+def _get_or_create_stripe_customer(
+    client: stripe.StripeClient,
+    db: Session,
+    owner_id: str,
+    email: str | None,
+    name: str | None,
+) -> str:
+    """Return the Stripe customer_id for *owner_id*, creating one if needed.
+
+    Args:
+        client: Configured Stripe client.
+        db: Database session.
+        owner_id: Stable user identifier.
+        email: User's email for the Stripe customer record.
+        name: User's display name for the Stripe customer record.
+
+    Returns:
+        The Stripe customer ID string.
+    """
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if profile and profile.stripe_customer_id:
+        return profile.stripe_customer_id
+
+    customer = client.customers.create(
+        params={
+            "email": email or "",
+            "name": name or "",
+            "metadata": {"docuelevate_user_id": owner_id},
+        }
+    )
+    if profile:
+        profile.stripe_customer_id = customer.id
+        db.commit()
+    return customer.id
+
+
+class CheckoutSessionBody(BaseModel):
+    """Request body for creating a Stripe Checkout session."""
+
+    plan_id: str
+    billing_cycle: str = "monthly"  # "monthly" | "yearly"
+
+
+class PortalSessionBody(BaseModel):
+    """Request body for creating a Stripe Customer Portal session."""
+
+    return_url: str | None = None
+
+
+@router.post("/create-checkout-session", summary="Create a Stripe Checkout session for a plan upgrade")
+@require_login
+async def create_checkout_session(
+    request: Request,
+    body: CheckoutSessionBody,
+    db: Session = Depends(get_db),
+) -> dict[str, Any]:
+    """Create a Stripe Checkout session.
+
+    The client should redirect the user to the returned ``checkout_url``.
+
+    Raises:
+        503: Stripe is not configured.
+        404: Plan not found or has no Stripe price configured.
+    """
+    client = _get_stripe()
+    if not client:
+        raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Billing is not configured.")
+
+    plan = db.query(SubscriptionPlan).filter(SubscriptionPlan.plan_id == body.plan_id).first()
+    if plan is None:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Plan {body.plan_id!r} not found.")
+
+    price_id = plan.stripe_price_id_yearly if body.billing_cycle == "yearly" else plan.stripe_price_id_monthly
+    if not price_id:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=(
+                f"Stripe price ID not configured for plan {body.plan_id!r} ({body.billing_cycle}). "
+                "Please set it in the Admin Plan Designer."
+            ),
+        )
+
+    user = request.session.get("user") or {}
+    owner_id = get_current_owner_id(request) or user.get("email") or ""
+    email = user.get("email")
+    name = user.get("name")
+
+    customer_id = _get_or_create_stripe_customer(client, db, owner_id, email, name)
+
+    base = str(request.base_url).rstrip("/")
+    success_url = settings.stripe_success_url or f"{base}/api/billing/success"
+    cancel_url = settings.stripe_cancel_url or f"{base}/pricing"
+
+    trial_days = plan.trial_days if plan.trial_days > 0 else None
+
+    session_params: dict[str, Any] = {
+        "customer": customer_id,
+        "mode": "subscription",
+        "line_items": [{"price": price_id, "quantity": 1}],
+        "success_url": success_url + "?session_id={CHECKOUT_SESSION_ID}",
+        "cancel_url": cancel_url,
+        "subscription_data": {
+            "metadata": {
+                "docuelevate_user_id": owner_id,
+                "plan_id": body.plan_id,
+                "billing_cycle": body.billing_cycle,
+            },
+        },
+        "metadata": {"docuelevate_user_id": owner_id, "plan_id": body.plan_id},
+        "allow_promotion_codes": True,
+        "billing_address_collection": "auto",
+        "tax_id_collection": {"enabled": True},
+        "automatic_tax": {"enabled": True},
+    }
+    if trial_days:
+        session_params["subscription_data"]["trial_period_days"] = trial_days
+
+    checkout_session = client.checkout.sessions.create(params=session_params)
+
+    logger.info(
+        "Created Stripe checkout session %s for user %s plan %s",
+        checkout_session.id,
+        owner_id,
+        body.plan_id,
+    )
+    return {"checkout_url": checkout_session.url, "session_id": checkout_session.id}
+
+
+@router.post("/create-portal-session", summary="Create a Stripe Customer Portal session")
+@require_login
+async def create_portal_session(
+    request: Request,
+    body: PortalSessionBody,
+    db: Session = Depends(get_db),
+) -> dict[str, Any]:
+    """Create a Stripe Customer Portal session for subscription self-management.
+
+    Raises:
+        503: Stripe not configured.
+        404: No Stripe customer found for this user.
+    """
+    client = _get_stripe()
+    if not client:
+        raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Billing is not configured.")
+
+    user = request.session.get("user") or {}
+    owner_id = get_current_owner_id(request) or user.get("email") or ""
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if not profile or not profile.stripe_customer_id:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail="No billing account found. Please subscribe to a plan first.",
+        )
+
+    base = str(request.base_url).rstrip("/")
+    return_url = body.return_url or f"{base}/subscription"
+
+    portal = client.billing_portal.sessions.create(
+        params={
+            "customer": profile.stripe_customer_id,
+            "return_url": return_url,
+        }
+    )
+
+    logger.info("Created Stripe portal session for user %s", owner_id)
+    return {"portal_url": portal.url}
+
+
+@router.post("/webhook", include_in_schema=False)
+async def stripe_webhook(request: Request, db: Session = Depends(get_db)) -> dict[str, str]:
+    """Handle Stripe webhook events.
+
+    Syncs subscription status to UserProfile.subscription_tier.
+
+    Events handled:
+
+    - ``checkout.session.completed``    — activate subscription after payment
+    - ``customer.subscription.updated`` — sync tier change
+    - ``customer.subscription.deleted`` — downgrade to free on cancellation
+    - ``invoice.payment_failed``        — log failed payment
+    """
+    if not settings.stripe_secret_key:
+        raise HTTPException(status_code=status.HTTP_503_SERVICE_UNAVAILABLE, detail="Billing not configured.")
+
+    payload = await request.body()
+    sig_header = request.headers.get("stripe-signature", "")
+
+    try:
+        if settings.stripe_webhook_secret:
+            event = stripe.Webhook.construct_event(payload, sig_header, settings.stripe_webhook_secret)
+        else:
+            logger.warning(
+                "[SECURITY] STRIPE_WEBHOOK_SECRET is not configured. "
+                "Webhook events are accepted without signature verification. "
+                "Set STRIPE_WEBHOOK_SECRET in production to prevent spoofed events."
+            )
+            event = stripe.Event.construct_from(json.loads(payload), stripe.api_key)
+    except stripe.SignatureVerificationError:
+        logger.warning("[SECURITY] Stripe webhook signature verification failed")
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid webhook signature.")
+    except Exception as exc:
+        logger.warning("Failed to parse Stripe webhook: %s", exc)
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail="Invalid webhook payload.")
+
+    _handle_stripe_event(db, event)
+    return {"status": "ok"}
+
+
+@router.get("/success", include_in_schema=False)
+@require_login
+async def billing_success(request: Request) -> Any:
+    """Show a success page after a completed Stripe Checkout."""
+    return _templates.TemplateResponse("billing_success.html", {"request": request})
+
+
+def _handle_stripe_event(db: Session, event: Any) -> None:
+    """Dispatch Stripe event to the appropriate handler.
+
+    Args:
+        db: Database session.
+        event: Parsed Stripe event object.
+    """
+    etype = event.get("type", "") if isinstance(event, dict) else getattr(event, "type", "")
+    data_obj = (
+        event.get("data", {}).get("object", {})
+        if isinstance(event, dict)
+        else getattr(getattr(event, "data", None), "object", {})
+    )
+
+    if etype == "checkout.session.completed":
+        _on_checkout_completed(db, data_obj)
+    elif etype == "customer.subscription.updated":
+        _on_subscription_updated(db, data_obj)
+    elif etype == "customer.subscription.deleted":
+        _on_subscription_deleted(db, data_obj)
+    elif etype == "invoice.payment_failed":
+        customer_id = data_obj.get("customer", "") if isinstance(data_obj, dict) else getattr(data_obj, "customer", "")
+        logger.warning("Stripe invoice payment failed for customer %s", customer_id)
+    else:
+        logger.debug("Unhandled Stripe event type: %s", etype)
+
+
+def _resolve_user_id_from_customer(db: Session, customer_id: str) -> str | None:
+    """Look up the DocuElevate user_id for a Stripe customer_id.
+
+    Args:
+        db: Database session.
+        customer_id: Stripe customer ID.
+
+    Returns:
+        The matching ``UserProfile.user_id``, or ``None`` if not found.
+    """
+    profile = db.query(UserProfile).filter(UserProfile.stripe_customer_id == customer_id).first()
+    return profile.user_id if profile else None
+
+
+def _resolve_plan_id_from_price(db: Session, price_id: str) -> str | None:
+    """Map a Stripe price_id to a DocuElevate plan_id via SubscriptionPlan.
+
+    Args:
+        db: Database session.
+        price_id: Stripe price ID.
+
+    Returns:
+        The matching ``SubscriptionPlan.plan_id``, or ``None`` if not found.
+    """
+    plan = (
+        db.query(SubscriptionPlan)
+        .filter(
+            (SubscriptionPlan.stripe_price_id_monthly == price_id)
+            | (SubscriptionPlan.stripe_price_id_yearly == price_id)
+        )
+        .first()
+    )
+    return plan.plan_id if plan else None
+
+
+def _on_checkout_completed(db: Session, data: Any) -> None:
+    """Activate a subscription after a successful checkout.
+
+    Args:
+        db: Database session.
+        data: Stripe ``checkout.session`` object.
+    """
+    meta = data.get("metadata") or {} if isinstance(data, dict) else getattr(data, "metadata", {}) or {}
+    user_id = meta.get("docuelevate_user_id") if isinstance(meta, dict) else getattr(meta, "docuelevate_user_id", None)
+    plan_id = meta.get("plan_id") if isinstance(meta, dict) else getattr(meta, "plan_id", None)
+    billing_cycle = (
+        meta.get("billing_cycle", "monthly") if isinstance(meta, dict) else getattr(meta, "billing_cycle", "monthly")
+    )
+    if not user_id:
+        return
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+    if profile and plan_id:
+        profile.subscription_tier = plan_id
+        profile.subscription_billing_cycle = billing_cycle
+        profile.subscription_period_start = datetime.now(tz=timezone.utc)
+        customer_id = data.get("customer", "") if isinstance(data, dict) else getattr(data, "customer", "")
+        if customer_id:
+            profile.stripe_customer_id = customer_id
+        db.commit()
+        logger.info("Activated plan %s/%s after checkout", plan_id, billing_cycle)
+
+
+def _on_subscription_updated(db: Session, data: Any) -> None:
+    """Sync tier change when a subscription is updated.
+
+    Args:
+        db: Database session.
+        data: Stripe ``customer.subscription`` object.
+    """
+    customer_id = data.get("customer", "") if isinstance(data, dict) else getattr(data, "customer", "")
+    user_id = _resolve_user_id_from_customer(db, customer_id)
+    if not user_id:
+        return
+
+    items_data = data.get("items") or {} if isinstance(data, dict) else getattr(data, "items", None) or {}
+    items = items_data.get("data") or [] if isinstance(items_data, dict) else getattr(items_data, "data", []) or []
+    if not items:
+        return
+
+    first_item = items[0]
+    price_obj = (
+        first_item.get("price") or {} if isinstance(first_item, dict) else getattr(first_item, "price", {}) or {}
+    )
+    price_id = price_obj.get("id") if isinstance(price_obj, dict) else getattr(price_obj, "id", None)
+    if not price_id:
+        return
+
+    plan_id = _resolve_plan_id_from_price(db, price_id)
+    if not plan_id:
+        logger.warning("Unknown Stripe price_id %s on subscription.updated", price_id)
+        return
+
+    recurring = (
+        price_obj.get("recurring", {}) if isinstance(price_obj, dict) else getattr(price_obj, "recurring", {}) or {}
+    )
+    interval = (
+        recurring.get("interval", "month") if isinstance(recurring, dict) else getattr(recurring, "interval", "month")
+    )
+    billing_cycle = "yearly" if interval == "year" else "monthly"
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+    if profile:
+        profile.subscription_tier = plan_id
+        profile.subscription_billing_cycle = billing_cycle
+        db.commit()
+        logger.info("Updated subscription to %s/%s", plan_id, billing_cycle)
+
+
+def _on_subscription_deleted(db: Session, data: Any) -> None:
+    """Downgrade user to free tier after subscription cancellation.
+
+    Args:
+        db: Database session.
+        data: Stripe ``customer.subscription`` object.
+    """
+    customer_id = data.get("customer", "") if isinstance(data, dict) else getattr(data, "customer", "")
+    user_id = _resolve_user_id_from_customer(db, customer_id)
+    if not user_id:
+        return
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+    if profile:
+        profile.subscription_tier = "free"
+        profile.subscription_billing_cycle = "monthly"
+        db.commit()
+        logger.info("Downgraded user %s to free tier after subscription cancellation", user_id)

app/api/database.py

@@ -0,0 +1,170 @@
+"""
+API endpoints for the database configuration wizard and migration tool.
+
+Provides REST endpoints for:
+- Testing database connections
+- Building connection strings from form components
+- Previewing and executing data migrations between databases
+"""
+
+import logging
+
+from fastapi import APIRouter, HTTPException, Request, status
+from pydantic import BaseModel, Field
+
+from app.utils.db_migrate import migrate_data, preview_migration
+from app.utils.db_wizard import (
+    build_connection_string,
+    get_supported_backends,
+    parse_connection_string,
+    test_connection,
+    validate_url_format,
+)
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/database", tags=["database"])
+
+
+def _require_admin(request: Request) -> dict:
+    """Ensure the caller is an admin.  Raises 403 otherwise."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+    return user
+
+
+# ---------------------------------------------------------------------------
+# Request / Response models
+# ---------------------------------------------------------------------------
+
+
+class ConnectionStringRequest(BaseModel):
+    """Request body for building a connection string."""
+
+    backend: str = Field(..., description="Database backend: sqlite, postgresql, mysql")
+    host: str = Field("", description="Database server hostname")
+    port: int | None = Field(None, description="Database server port")
+    database: str = Field("", description="Database name")
+    username: str = Field("", description="Authentication username")
+    password: str = Field("", description="Authentication password")
+    ssl_mode: str = Field("", description="SSL mode (e.g. require, verify-full)")
+    extra_options: str = Field("", description="Additional query-string options")
+    sqlite_path: str = Field("", description="File path for SQLite databases")
+
+
+class TestConnectionRequest(BaseModel):
+    """Request body for testing a database connection."""
+
+    url: str = Field(..., description="Full SQLAlchemy connection URL to test")
+
+
+class MigrateRequest(BaseModel):
+    """Request body for data migration."""
+
+    source_url: str = Field(..., description="Source database connection URL")
+    target_url: str = Field(..., description="Target database connection URL")
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/backends")
+async def list_backends() -> list[dict]:
+    """List all supported database backends with metadata."""
+    return get_supported_backends()
+
+
+@router.post("/build-url")
+async def build_url(body: ConnectionStringRequest, request: Request) -> dict:
+    """Build a SQLAlchemy connection string from individual components.
+
+    Returns the assembled URL string.
+    """
+    _require_admin(request)
+    try:
+        url = build_connection_string(
+            backend=body.backend,
+            host=body.host,
+            port=body.port,
+            database=body.database,
+            username=body.username,
+            password=body.password,
+            ssl_mode=body.ssl_mode,
+            extra_options=body.extra_options,
+            sqlite_path=body.sqlite_path,
+        )
+        return {"url": url}
+    except ValueError as exc:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
+
+
+@router.post("/parse-url")
+async def parse_url(body: TestConnectionRequest, request: Request) -> dict:
+    """Parse a connection string into its components."""
+    _require_admin(request)
+    return parse_connection_string(body.url)
+
+
+@router.post("/validate-url")
+async def validate_url(body: TestConnectionRequest, request: Request) -> dict:
+    """Validate a connection string format without connecting."""
+    _require_admin(request)
+    return validate_url_format(body.url)
+
+
+@router.post("/test-connection")
+async def test_db_connection(body: TestConnectionRequest, request: Request) -> dict:
+    """Test connectivity to a database and return status info.
+
+    This creates a temporary engine, executes ``SELECT 1``, and disposes
+    of the engine.  It does **not** modify any global application state.
+    """
+    _require_admin(request)
+    return test_connection(body.url)
+
+
+@router.post("/preview-migration")
+async def preview_db_migration(body: TestConnectionRequest, request: Request) -> dict:
+    """Preview what a migration from the given source would include.
+
+    Returns a table-by-table row count without actually copying data.
+    """
+    _require_admin(request)
+    return preview_migration(body.url)
+
+
+@router.post("/migrate")
+async def execute_migration(body: MigrateRequest, request: Request) -> dict:
+    """Execute a full data migration from source to target database.
+
+    **Warning:** This copies all data from the source database into the
+    target.  The target schema is created from the current application
+    models.  Existing data in the target is **not** deleted first — use
+    on an empty target database.
+    """
+    _require_admin(request)
+
+    # Validate both URLs first
+    src_check = validate_url_format(body.source_url)
+    if not src_check.get("valid"):
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Invalid source URL: {src_check.get('error', 'unknown')}",
+        )
+    tgt_check = validate_url_format(body.target_url)
+    if not tgt_check.get("valid"):
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Invalid target URL: {tgt_check.get('error', 'unknown')}",
+        )
+
+    result = migrate_data(body.source_url, body.target_url)
+    if not result["success"]:
+        error_summary = "; ".join(result.get("errors", ["Unknown error"]))
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail=f"Migration completed with errors: {error_summary}",
+        )
+    return result

app/api/diagnostic.py

@@ -2,19 +2,112 @@
 Diagnostic API endpoints
 """
 
+import datetime
 import logging
 
+import redis as redis_lib
 from fastapi import APIRouter, Request
+from fastapi.responses import JSONResponse
+from sqlalchemy import text
 
 from app.auth import require_login
 from app.config import settings
+from app.database import engine
 
 # Set up logging
 logger = logging.getLogger(__name__)
 
+_DEFAULT_REDIS_URL = "redis://localhost:6379/0"
+
 router = APIRouter()
 
 
+@router.get("/diagnostic/health")
+@require_login
+async def health_check(request: Request):
+    """
+    System health endpoint for monitoring tools (Grafana, Uptime Kuma, etc.).
+
+    Checks database connectivity and Redis availability and returns a
+    machine-readable summary that monitoring systems can scrape.
+
+    **Authentication:** Required (no-op when AUTH_ENABLED=False)
+
+    **Response (200 OK) – all subsystems healthy:**
+    ```json
+    {
+      "status": "healthy",
+      "version": "1.2.3",
+      "timestamp": "2024-01-15T10:30:00+00:00",
+      "checks": {
+        "database": {"status": "ok"},
+        "redis":    {"status": "ok"}
+      }
+    }
+    ```
+
+    **Response (200 OK) – one or more subsystems degraded:**
+    ```json
+    {
+      "status": "degraded",
+      "version": "1.2.3",
+      "timestamp": "2024-01-15T10:30:00+00:00",
+      "checks": {
+        "database": {"status": "ok"},
+        "redis":    {"status": "error", "detail": "Connection refused"}
+      }
+    }
+    ```
+
+    The outer ``status`` field is always one of:
+    - ``"healthy"``  – all checks passed
+    - ``"degraded"`` – at least one non-critical check failed
+    - ``"unhealthy"`` – a critical check failed (currently: database)
+    """
+    timestamp = datetime.datetime.now(datetime.timezone.utc).isoformat()
+    checks: dict[str, dict[str, str]] = {}
+
+    # ── Database check ─────────────────────────────────────────────────────
+    db_ok = False
+    try:
+        with engine.connect() as conn:
+            conn.execute(text("SELECT 1"))
+        checks["database"] = {"status": "ok"}
+        db_ok = True
+    except Exception as exc:
+        logger.warning("Health check: database probe failed: %s", exc)
+        checks["database"] = {"status": "error", "detail": str(exc)}
+
+    # ── Redis check ────────────────────────────────────────────────────────
+    try:
+        redis_url = settings.redis_url or _DEFAULT_REDIS_URL
+        r = redis_lib.from_url(redis_url, socket_connect_timeout=2, socket_timeout=2)
+        r.ping()
+        checks["redis"] = {"status": "ok"}
+    except Exception as exc:
+        logger.warning("Health check: Redis probe failed: %s", exc)
+        checks["redis"] = {"status": "error", "detail": str(exc)}
+
+    # ── Overall status ─────────────────────────────────────────────────────
+    if not db_ok:
+        overall = "unhealthy"
+    elif any(v.get("status") != "ok" for v in checks.values()):
+        overall = "degraded"
+    else:
+        overall = "healthy"
+
+    http_status = 503 if overall == "unhealthy" else 200
+
+    payload = {
+        "status": overall,
+        "version": settings.version,
+        "timestamp": timestamp,
+        "checks": checks,
+    }
+
+    return JSONResponse(content=payload, status_code=http_status)
+
+
 @router.post("/diagnostic/test-notification")
 @require_login
 async def test_notification(request: Request):

app/api/duplicates.py

@@ -0,0 +1,230 @@
+"""Duplicate document detection and management API endpoints.
+
+Provides endpoints for listing all duplicate groups (exact SHA-256 duplicates) and
+for retrieving both exact and near-duplicate matches for a specific document.
+
+Near-duplicate detection is powered by the same text-embedding cosine-similarity
+engine used by the ``/api/files/{id}/similar`` endpoint
+(see ``app/utils/similarity.py``).
+"""
+
+import logging
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from sqlalchemy.orm import Session
+
+from app.auth import require_login
+from app.config import settings
+from app.database import get_db
+from app.models import FileRecord
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter()
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+@router.get("/duplicates")
+@require_login
+def list_duplicate_groups(
+    request: Request,
+    db: DbSession,
+    page: int = Query(1, ge=1, description="Page number"),
+    per_page: int = Query(25, ge=1, le=200, description="Items per page"),
+):
+    """List all groups of exact-duplicate documents (same SHA-256 hash).
+
+    Returns one entry per duplicate group showing the original document and all
+    files that were detected as copies of it.  Groups are sorted by descending
+    duplicate count.
+
+    Example:
+    ```
+    GET /api/duplicates
+    ```
+
+    Response:
+    ```json
+    {
+      "groups": [
+        {
+          "filehash": "abc123...",
+          "original": {"id": 1, "original_filename": "invoice.pdf", ...},
+          "duplicates": [{"id": 5, "original_filename": "invoice_copy.pdf", ...}],
+          "duplicate_count": 1
+        }
+      ],
+      "total_groups": 1,
+      "total_duplicate_files": 1,
+      "pagination": {...}
+    }
+    ```
+    """
+    # Find all hashes that have at least one duplicate record
+    dup_hashes_query = db.query(FileRecord.filehash).filter(FileRecord.is_duplicate.is_(True)).distinct()
+    total_groups = dup_hashes_query.count()
+
+    # Paginate hash groups
+    offset = (page - 1) * per_page
+    dup_hashes = [row.filehash for row in dup_hashes_query.offset(offset).limit(per_page).all()]
+
+    groups = []
+    total_duplicate_files = 0
+
+    for filehash in dup_hashes:
+        # Find the original (non-duplicate) record with this hash
+        original = (
+            db.query(FileRecord)
+            .filter(FileRecord.filehash == filehash, FileRecord.is_duplicate.is_(False))
+            .order_by(FileRecord.id.asc())
+            .first()
+        )
+
+        # Find all duplicate records for this hash
+        duplicates = (
+            db.query(FileRecord)
+            .filter(FileRecord.filehash == filehash, FileRecord.is_duplicate.is_(True))
+            .order_by(FileRecord.id.asc())
+            .all()
+        )
+
+        total_duplicate_files += len(duplicates)
+
+        groups.append(
+            {
+                "filehash": filehash,
+                "original": _file_record_to_dict(original) if original else None,
+                "duplicates": [_file_record_to_dict(d) for d in duplicates],
+                "duplicate_count": len(duplicates),
+            }
+        )
+
+    total_pages = (total_groups + per_page - 1) // per_page if total_groups > 0 else 1
+
+    return {
+        "groups": groups,
+        "total_groups": total_groups,
+        "total_duplicate_files": total_duplicate_files,
+        "pagination": {
+            "page": page,
+            "per_page": per_page,
+            "total": total_groups,
+            "pages": total_pages,
+            "next": str(request.url.include_query_params(page=page + 1)) if page < total_pages else None,
+            "previous": str(request.url.include_query_params(page=page - 1)) if page > 1 else None,
+        },
+    }
+
+
+@router.get("/files/{file_id}/duplicates")
+@require_login
+def get_file_duplicates(
+    request: Request,
+    file_id: int,
+    db: DbSession,
+    near_duplicate_limit: int = Query(5, ge=1, le=20, description="Maximum near-duplicates to return"),
+    near_duplicate_threshold: float = Query(
+        -1.0,
+        ge=-1.0,
+        le=1.0,
+        description="Minimum similarity score for near-duplicates; -1 uses the configured default",
+    ),
+):
+    """Get exact and near-duplicate documents for the specified file.
+
+    **Exact duplicates** share the same SHA-256 hash.
+    **Near-duplicates** have a text-embedding cosine similarity score ≥
+    ``NEAR_DUPLICATE_THRESHOLD`` (configurable; default 0.85).
+
+    Near-duplicate detection requires OCR text to be available for both the
+    target file and candidate files.  Files without OCR text are excluded.
+
+    Example:
+    ```
+    GET /api/files/42/duplicates
+    ```
+
+    Response:
+    ```json
+    {
+      "file_id": 42,
+      "exact_duplicates": [
+        {"id": 7, "original_filename": "invoice.pdf", "is_duplicate": true, "duplicate_of_id": 42, ...}
+      ],
+      "near_duplicates": [
+        {"file_id": 15, "original_filename": "invoice_jan.pdf", "similarity_score": 0.92, ...}
+      ],
+      "near_duplicate_threshold": 0.85
+    }
+    ```
+    """
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    # --- Exact duplicates ---
+    # Case 1: This file is the original — find all records that are duplicates of it
+    exact_duplicates_of_this = (
+        db.query(FileRecord)
+        .filter(FileRecord.filehash == file_record.filehash, FileRecord.id != file_id)
+        .order_by(FileRecord.id.asc())
+        .all()
+    )
+
+    # Case 2: This file itself is a duplicate — find the original
+    is_self_duplicate = file_record.is_duplicate
+    duplicate_of_original: FileRecord | None = None
+    if is_self_duplicate and file_record.duplicate_of_id:
+        duplicate_of_original = db.query(FileRecord).filter(FileRecord.id == file_record.duplicate_of_id).first()
+
+    exact_duplicate_dicts = [_file_record_to_dict(f) for f in exact_duplicates_of_this]
+
+    # --- Near-duplicates (embedding-based) ---
+    effective_threshold = (
+        near_duplicate_threshold if near_duplicate_threshold >= 0.0 else settings.near_duplicate_threshold
+    )
+
+    near_duplicates: list[dict] = []
+    if file_record.ocr_text and file_record.ocr_text.strip():
+        try:
+            from app.utils.similarity import find_similar_documents
+
+            near_duplicates = find_similar_documents(
+                db,
+                file_id,
+                limit=near_duplicate_limit,
+                threshold=effective_threshold,
+            )
+        except Exception as e:
+            logger.warning(f"Near-duplicate detection failed for file {file_id}: {e}")
+            near_duplicates = []
+
+    return {
+        "file_id": file_id,
+        "is_duplicate": is_self_duplicate,
+        "duplicate_of": _file_record_to_dict(duplicate_of_original) if duplicate_of_original else None,
+        "exact_duplicates": exact_duplicate_dicts,
+        "near_duplicates": near_duplicates,
+        "near_duplicate_threshold": effective_threshold,
+        "exact_duplicate_count": len(exact_duplicate_dicts),
+        "near_duplicate_count": len(near_duplicates),
+    }
+
+
+def _file_record_to_dict(file_record: FileRecord | None) -> dict | None:
+    """Serialise a ``FileRecord`` to a plain dict for JSON responses."""
+    if file_record is None:
+        return None
+    return {
+        "id": file_record.id,
+        "original_filename": file_record.original_filename,
+        "filehash": file_record.filehash,
+        "file_size": file_record.file_size,
+        "mime_type": file_record.mime_type,
+        "is_duplicate": file_record.is_duplicate,
+        "duplicate_of_id": file_record.duplicate_of_id,
+        "document_title": file_record.document_title,
+        "created_at": file_record.created_at.isoformat() if file_record.created_at else None,
+    }

app/api/files.py

@@ -11,7 +11,7 @@ import zipfile
 from datetime import datetime, timezone
 from typing import Annotated, List, Optional
 
-from fastapi import APIRouter, Depends, File, HTTPException, Query, Request, UploadFile
+from fastapi import APIRouter, Depends, File, HTTPException, Query, Request, UploadFile, status
 from fastapi.responses import StreamingResponse
 from sqlalchemy import asc, desc
 from sqlalchemy.orm import Session
@@ -23,10 +23,12 @@ from app.models import FileProcessingStep, FileRecord, ProcessingLog
 from app.tasks.convert_to_pdf import convert_to_pdf
 from app.tasks.process_document import process_document
 from app.utils.allowed_types import ALLOWED_EXTENSIONS, ALLOWED_MIME_TYPES, IMAGE_MIME_TYPES
+from app.utils.file_operations import hash_file
 from app.utils.file_queries import apply_status_filter
 from app.utils.file_status import get_files_processing_status
 from app.utils.filename_utils import sanitize_filename
 from app.utils.input_validation import validate_search_query, validate_sort_field, validate_sort_order
+from app.utils.user_scope import apply_owner_filter, get_current_owner_id
 
 # Set up logging
 logger = logging.getLogger(__name__)
@@ -49,7 +51,7 @@ def list_files_api(
     request: Request,
     db: DbSession,
     page: int = Query(1, ge=1, description="Page number"),
-    per_page: int = Query(50, ge=1, le=200, description="Items per page"),
+    per_page: int = Query(25, ge=1, le=200, description="Items per page"),
     sort_by: str = Query(
         "created_at",
         description="Sort field: id, original_filename, file_size, mime_type, created_at, status",
@@ -69,7 +71,7 @@ def list_files_api(
 
     Query Parameters:
     - page: Page number (default: 1)
-    - per_page: Items per page (default: 50, max: 200)
+    - per_page: Items per page (default: 25, max: 200)
     - sort_by: Field to sort by (default: created_at)
     - sort_order: asc or desc (default: desc)
     - search: Search in filename
@@ -85,9 +87,11 @@ def list_files_api(
       "files": [...],
       "pagination": {
         "page": 1,
-        "per_page": 50,
-        "total_items": 150,
-        "total_pages": 3
+        "per_page": 25,
+        "total": 150,
+        "pages": 6,
+        "next": "http://host/api/files?page=2",
+        "previous": null
       }
     }
     """
@@ -96,8 +100,9 @@ def list_files_api(
     validate_sort_order(sort_order)
     search = validate_search_query(search)
 
-    # Start with base query
+    # Start with base query, scoped to the current user in multi-user mode
     query = db.query(FileRecord)
+    query = apply_owner_filter(query, request)
 
     # Apply search filter
     if search:
@@ -205,13 +210,19 @@ def list_files_api(
     # Calculate pagination info
     total_pages = (total_items + per_page - 1) // per_page
 
+    # Build next / previous page URLs by replacing the page query parameter
+    next_url = str(request.url.include_query_params(page=page + 1)) if page < total_pages else None
+    previous_url = str(request.url.include_query_params(page=page - 1)) if page > 1 else None
+
     return {
         "files": result,
         "pagination": {
             "page": page,
             "per_page": per_page,
-            "total_items": total_items,
-            "total_pages": total_pages,
+            "total": total_items,
+            "pages": total_pages,
+            "next": next_url,
+            "previous": previous_url,
         },
     }
 
@@ -232,8 +243,10 @@ def get_file_details(request: Request, file_id: int, db: DbSession):
     """
     Get detailed information about a specific file including processing history.
     """
-    # Find the file record
-    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    # Find the file record, scoped to the current user in multi-user mode
+    query = db.query(FileRecord).filter(FileRecord.id == file_id)
+    query = apply_owner_filter(query, request)
+    file_record = query.first()
 
     if not file_record:
         raise HTTPException(status_code=404, detail=f"File record with ID {file_id} not found")
@@ -291,8 +304,10 @@ def delete_file_record(request: Request, file_id: int, db: DbSession):
         raise HTTPException(status_code=403, detail="File deletion is disabled in the configuration")
 
     try:
-        # Find the file record
-        file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+        # Find the file record, scoped to the current user in multi-user mode
+        query = db.query(FileRecord).filter(FileRecord.id == file_id)
+        query = apply_owner_filter(query, request)
+        file_record = query.first()
 
         if not file_record:
             raise HTTPException(status_code=404, detail=f"File record with ID {file_id} not found")
@@ -1204,7 +1219,7 @@ def download_file(
 
 @router.post("/ui-upload")
 @require_login
-async def ui_upload(request: Request, file: UploadFile = File(...)):
+async def ui_upload(request: Request, db: DbSession, file: UploadFile = File(...)):
     """Endpoint to accept a user-uploaded file and enqueue it for processing."""
     workdir = settings.workdir
 
@@ -1241,6 +1256,23 @@ async def ui_upload(request: Request, file: UploadFile = File(...)):
     # Store both the safe original name and the unique name
     target_path = os.path.join(workdir, target_filename)
 
+    # Determine the owner_id for multi-user document isolation
+    upload_owner_id = get_current_owner_id(request) if settings.multi_user_enabled else None
+
+    # Enforce subscription tier upload quotas (multi-user mode only) BEFORE writing the file
+    # so that users who have exceeded their quota do not waste bandwidth or disk I/O.
+    if settings.multi_user_enabled and upload_owner_id:
+        from app.utils.subscription import QuotaExceeded, check_upload_allowed, get_user_tier_id
+
+        tier_id = get_user_tier_id(db, upload_owner_id)
+        try:
+            check_upload_allowed(db, upload_owner_id, tier_id)
+        except QuotaExceeded as qe:
+            raise HTTPException(
+                status_code=status.HTTP_402_PAYMENT_REQUIRED,
+                detail=str(qe),
+            )
+
     # Read file in chunks to avoid loading the entire body into memory at once,
     # enforcing the size limit during the read so memory usage stays bounded.
     try:
@@ -1301,7 +1333,7 @@ async def ui_upload(request: Request, file: UploadFile = File(...)):
             task_ids = []
             for split_file in split_files:
                 split_filename = os.path.basename(split_file)
-                task = process_document.delay(split_file, original_filename=split_filename)
+                task = process_document.delay(split_file, original_filename=split_filename, owner_id=upload_owner_id)
                 task_ids.append(task.id)
                 logger.info(f"Enqueued split PDF part for processing: {split_file}")
 
@@ -1324,7 +1356,7 @@ async def ui_upload(request: Request, file: UploadFile = File(...)):
 
     if is_pdf and not should_split:
         # If it's a PDF, process directly
-        task = process_document.delay(target_path, original_filename=safe_filename)
+        task = process_document.delay(target_path, original_filename=safe_filename, owner_id=upload_owner_id)
         logger.info(f"Enqueued PDF for processing: {target_path}")
     elif mime_type in IMAGE_MIME_TYPES or file_ext in {
         ".jpg",
@@ -1338,20 +1370,260 @@ async def ui_upload(request: Request, file: UploadFile = File(...)):
         ".svg",
     }:
         # If it's an image, convert to PDF first
-        task = convert_to_pdf.delay(target_path, original_filename=safe_filename)
+        task = convert_to_pdf.delay(target_path, original_filename=safe_filename, owner_id=upload_owner_id)
         logger.info(f"Enqueued image for PDF conversion: {target_path}")
     elif mime_type in ALLOWED_MIME_TYPES or file_ext in ALLOWED_EXTENSIONS:
         # Office document, HTML, Markdown, or other Gotenberg-supported format
-        task = convert_to_pdf.delay(target_path, original_filename=safe_filename)
+        task = convert_to_pdf.delay(target_path, original_filename=safe_filename, owner_id=upload_owner_id)
         logger.info(f"Enqueued document for PDF conversion: {target_path}")
     else:
         # For any other file type, attempt conversion but log a warning
         logger.warning(f"Unsupported MIME type {mime_type} for {target_path}, attempting conversion")
-        task = convert_to_pdf.delay(target_path, original_filename=safe_filename)
+        task = convert_to_pdf.delay(target_path, original_filename=safe_filename, owner_id=upload_owner_id)
 
-    return {
+    # Check for exact duplicates (same SHA-256 hash) before returning.
+    # This gives the caller an immediate warning without waiting for the pipeline.
+    # Only performed when deduplication is enabled in settings.
+    exact_duplicate_warning = None
+    if settings.enable_deduplication:
+        try:
+            filehash = hash_file(target_path)
+            existing = (
+                db.query(FileRecord)
+                .filter(FileRecord.filehash == filehash, FileRecord.is_duplicate.is_(False))
+                .order_by(FileRecord.id.asc())
+                .first()
+            )
+            if existing:
+                exact_duplicate_warning = {
+                    "duplicate_type": "exact",
+                    "original_file_id": existing.id,
+                    "original_filename": existing.original_filename,
+                    "message": (
+                        "This file appears to be an exact duplicate of an already-processed document. "
+                        "It will still be queued but will be flagged as a duplicate."
+                    ),
+                }
+                logger.info(f"Exact duplicate detected on upload: '{safe_filename}' matches file ID {existing.id}")
+        except Exception as e:
+            logger.warning(f"Duplicate check failed for uploaded file '{safe_filename}': {e}")
+
+    response: dict = {
         "task_id": task.id,
         "status": "queued",
         "original_filename": safe_filename,
         "stored_filename": target_filename,
     }
+    if exact_duplicate_warning:
+        response["duplicate_warning"] = exact_duplicate_warning
+    return response
+
+
+# ---------------------------------------------------------------------------
+# Document ownership / claim endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.post("/files/{file_id}/claim")
+@require_login
+def claim_file(request: Request, file_id: int, db: DbSession):
+    """
+    Claim an unowned document for the current user.
+
+    Only documents with ``owner_id IS NULL`` can be claimed.  The requesting
+    user's identifier is written into ``owner_id``.  In single-user mode
+    the endpoint is a no-op (returns the file unchanged).
+    """
+    if not settings.multi_user_enabled:
+        raise HTTPException(status_code=400, detail="Multi-user mode is not enabled")
+
+    owner_id = get_current_owner_id(request)
+    if owner_id is None:
+        raise HTTPException(status_code=401, detail="Authentication required to claim a document")
+
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=404, detail=f"File record with ID {file_id} not found")
+
+    if file_record.owner_id is not None:
+        if file_record.owner_id == owner_id:
+            return {"status": "already_owned", "message": "You already own this document", "file_id": file_id}
+        raise HTTPException(status_code=403, detail="This document is already owned by another user")
+
+    file_record.owner_id = owner_id
+    try:
+        db.commit()
+    except Exception as e:
+        db.rollback()
+        logger.exception(f"Error claiming file {file_id}: {e}")
+        raise HTTPException(status_code=500, detail="Failed to claim document")
+
+    logger.info(f"File {file_id} claimed by user '{owner_id}'")
+    return {"status": "success", "message": "Document claimed successfully", "file_id": file_id, "owner_id": owner_id}
+
+
+@router.post("/files/bulk-claim")
+@require_login
+def bulk_claim_files(request: Request, file_ids: list[int], db: DbSession):
+    """
+    Claim multiple unowned documents for the current user.
+
+    Only documents with ``owner_id IS NULL`` will be claimed.  Documents
+    already owned (by anyone) are skipped and reported in ``skipped``.
+    """
+    if not settings.multi_user_enabled:
+        raise HTTPException(status_code=400, detail="Multi-user mode is not enabled")
+
+    owner_id = get_current_owner_id(request)
+    if owner_id is None:
+        raise HTTPException(status_code=401, detail="Authentication required to claim documents")
+
+    file_records = db.query(FileRecord).filter(FileRecord.id.in_(file_ids)).all()
+    if not file_records:
+        raise HTTPException(status_code=404, detail="No files found with the provided IDs")
+
+    claimed = []
+    skipped = []
+    for rec in file_records:
+        if rec.owner_id is None:
+            rec.owner_id = owner_id
+            claimed.append(rec.id)
+        else:
+            skipped.append({"file_id": rec.id, "reason": "already owned"})
+
+    try:
+        db.commit()
+    except Exception as e:
+        db.rollback()
+        logger.exception(f"Error during bulk claim: {e}")
+        raise HTTPException(status_code=500, detail="Failed to claim documents")
+
+    logger.info(f"Bulk claim by '{owner_id}': claimed={claimed}, skipped={[s['file_id'] for s in skipped]}")
+    return {
+        "status": "success",
+        "claimed_count": len(claimed),
+        "claimed_ids": claimed,
+        "skipped": skipped,
+        "owner_id": owner_id,
+    }
+
+
+@router.post("/files/assign-owner")
+@require_login
+def assign_owner(request: Request, db: DbSession, owner_id: str = Query(...), file_ids: list[int] | None = None):
+    """
+    Admin-only: assign an owner to documents.
+
+    If ``file_ids`` is provided, only those files are updated.  If omitted,
+    **all** currently unowned documents (``owner_id IS NULL``) are assigned
+    to the given ``owner_id``.
+    """
+    if not settings.multi_user_enabled:
+        raise HTTPException(status_code=400, detail="Multi-user mode is not enabled")
+
+    user = request.session.get("user")
+    if not isinstance(user, dict) or not user.get("is_admin"):
+        raise HTTPException(status_code=403, detail="Only admins can assign document owners")
+
+    if not owner_id or not owner_id.strip():
+        raise HTTPException(status_code=422, detail="owner_id must be a non-empty string")
+    owner_id = owner_id.strip()
+
+    if file_ids is not None:
+        # Assign to specific files
+        updated = (
+            db.query(FileRecord)
+            .filter(FileRecord.id.in_(file_ids))
+            .update({FileRecord.owner_id: owner_id}, synchronize_session="fetch")
+        )
+    else:
+        # Assign to all currently unowned documents
+        updated = (
+            db.query(FileRecord)
+            .filter(FileRecord.owner_id.is_(None))
+            .update({FileRecord.owner_id: owner_id}, synchronize_session="fetch")
+        )
+
+    try:
+        db.commit()
+    except Exception as e:
+        db.rollback()
+        logger.exception(f"Error assigning owner: {e}")
+        raise HTTPException(status_code=500, detail="Failed to assign owner")
+
+    admin_name = get_current_owner_id(request) or "admin"
+    logger.info(f"Admin '{admin_name}' assigned owner_id='{owner_id}' to {updated} file(s)")
+    return {
+        "status": "success",
+        "message": f"Assigned owner to {updated} document(s)",
+        "updated_count": updated,
+        "owner_id": owner_id,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Pipeline assignment
+# ---------------------------------------------------------------------------
+
+
+@router.post("/files/{file_id}/assign-pipeline")
+@require_login
+def assign_pipeline_to_file(
+    request: Request,
+    file_id: int,
+    db: DbSession,
+    pipeline_id: int | None = None,
+):
+    """Assign (or remove) a processing pipeline from a file.
+
+    Path Parameters:
+        file_id: The file to update.
+
+    Query / Body Parameters:
+        pipeline_id: The pipeline to assign.  Pass ``null`` or omit to clear the
+            assignment (the system default will be used for future processing).
+
+    Returns:
+        A summary dict with the file_id and updated pipeline_id.
+
+    Raises:
+        HTTPException 404: If the file or pipeline does not exist / is not
+            accessible to the current user.
+    """
+    from app.auth import get_current_user, get_current_user_id
+    from app.models import Pipeline
+
+    user = get_current_user(request)
+    user_id: str = get_current_user_id(request)
+
+    is_admin_user = bool(user and user.get("is_admin"))
+
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    # Non-admins may only update files they own (or unowned files in single-user mode)
+    owner_id = get_current_owner_id(request)
+    if not is_admin_user and file_record.owner_id is not None and file_record.owner_id != owner_id:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    if pipeline_id is not None:
+        pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+        if not pipeline:
+            raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+        # Check access: users can only assign their own pipelines or system pipelines (owner_id=None)
+        if not is_admin_user and pipeline.owner_id is not None and pipeline.owner_id != user_id:
+            raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    file_record.pipeline_id = pipeline_id
+
+    try:
+        db.commit()
+        db.refresh(file_record)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to assign pipeline to file id={file_id}: {exc}")
+        raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Failed to assign pipeline")
+
+    logger.info(f"Pipeline {pipeline_id!r} assigned to file id={file_id}")
+    return {"file_id": file_id, "pipeline_id": file_record.pipeline_id}

app/api/local_auth.py

@@ -0,0 +1,391 @@
+"""Local user authentication API — signup, email verification, password reset.
+
+Provides the REST endpoints and page routes for the self-registration flow:
+
+- GET  /signup                           — signup page (HTML)
+- POST /api/auth/signup                  — create account + send verification email
+- GET  /verify-email                     — activate account from email link (redirect)
+- GET  /verify-email-sent                — confirmation landing page (HTML)
+- POST /api/auth/resend-verification     — re-send verification email
+- POST /api/auth/request-password-reset  — start password reset
+- POST /api/auth/reset-password          — set new password using token
+- GET  /reset-password                   — password reset form page (HTML)
+"""
+
+import logging
+import pathlib
+from datetime import datetime, timezone
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from fastapi.templating import Jinja2Templates
+from pydantic import BaseModel, Field
+from sqlalchemy.orm import Session
+from starlette.responses import RedirectResponse
+
+from app.config import settings
+from app.database import get_db
+from app.models import LocalUser, UserProfile
+from app.utils.local_auth import (
+    build_session_user,
+    generate_token,
+    hash_password,
+    is_token_expired,
+    send_forgot_username_email,
+    send_password_reset_email,
+    send_verification_email,
+)
+
+logger = logging.getLogger(__name__)
+router = APIRouter(tags=["local-auth"])
+
+_templates_dir = pathlib.Path(__file__).parents[2] / "frontend" / "templates"
+templates = Jinja2Templates(directory=str(_templates_dir))
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class SignupBody(BaseModel):
+    """Body for the signup endpoint."""
+
+    email: str = Field(..., max_length=255)
+    username: str = Field(..., min_length=3, max_length=64, pattern=r"^[a-zA-Z0-9_-]+$")
+    display_name: str | None = Field(default=None, max_length=255)
+    password: str = Field(..., min_length=8, max_length=128)
+    password_confirm: str
+
+
+class ResendVerificationBody(BaseModel):
+    """Body for the resend-verification endpoint."""
+
+    email: str
+
+
+class PasswordResetRequestBody(BaseModel):
+    """Body for the request-password-reset endpoint."""
+
+    email: str
+
+
+class PasswordResetBody(BaseModel):
+    """Body for the reset-password endpoint."""
+
+    token: str
+    new_password: str = Field(..., min_length=8, max_length=128)
+    new_password_confirm: str
+
+
+class ForgotUsernameBody(BaseModel):
+    """Body for the forgot-username endpoint."""
+
+    email: str
+
+
+# ---------------------------------------------------------------------------
+# Page routes (return HTML)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/signup", include_in_schema=False)
+async def signup_page(request: Request) -> Any:
+    """Render the signup page, or redirect to login when multi-user / signup is disabled."""
+    if not settings.multi_user_enabled:
+        return RedirectResponse(url="/login?error=Multi-user+mode+is+not+enabled", status_code=302)
+    if not settings.allow_local_signup:
+        return RedirectResponse(url="/login?error=Registration+is+not+enabled", status_code=302)
+    return templates.TemplateResponse(
+        "signup.html",
+        {
+            "request": request,
+            "csrf_token": getattr(request.state, "csrf_token", ""),
+            "app_version": settings.version,
+        },
+    )
+
+
+@router.get("/verify-email-sent", include_in_schema=False)
+async def verify_email_sent_page(request: Request) -> Any:
+    """Render the verify-email-sent confirmation page."""
+    return templates.TemplateResponse("verify_email_sent.html", {"request": request})
+
+
+@router.get("/forgot-username", include_in_schema=False)
+async def forgot_username_page(request: Request) -> Any:
+    """Render the forgot-username page where users can request a username reminder email."""
+    return templates.TemplateResponse(
+        "forgot_username.html",
+        {
+            "request": request,
+            "csrf_token": getattr(request.state, "csrf_token", ""),
+            "app_version": settings.version,
+        },
+    )
+
+
+@router.get("/forgot-password", include_in_schema=False)
+async def forgot_password_page(request: Request) -> Any:
+    """Render the forgot-password page where users can request a reset email."""
+    return templates.TemplateResponse(
+        "forgot_password.html",
+        {
+            "request": request,
+            "csrf_token": getattr(request.state, "csrf_token", ""),
+            "app_version": settings.version,
+        },
+    )
+
+
+@router.get("/reset-password", include_in_schema=False)
+async def reset_password_page(request: Request) -> Any:
+    """Render the password reset form page."""
+    token = request.query_params.get("token", "")
+    return templates.TemplateResponse(
+        "password_reset_form.html",
+        {
+            "request": request,
+            "token": token,
+            "csrf_token": getattr(request.state, "csrf_token", ""),
+            "app_version": settings.version,
+        },
+    )
+
+
+# ---------------------------------------------------------------------------
+# API endpoints (return JSON or redirect)
+# ---------------------------------------------------------------------------
+
+
+@router.post("/api/auth/signup", status_code=status.HTTP_201_CREATED)
+async def signup(request: Request, body: SignupBody, db: DbSession) -> dict[str, str | bool]:
+    """Create a new local user account.
+
+    When SMTP is configured the account is inactive until the user clicks the
+    verification link sent to their email.  When SMTP is **not** configured the
+    account is activated immediately so that deployments without email can still
+    use the self-registration flow.
+
+    Both ``MULTI_USER_ENABLED`` and ``ALLOW_LOCAL_SIGNUP`` must be ``True``.
+
+    Raises:
+        403: Multi-user mode or local signup is disabled.
+        422: Passwords do not match.
+        409: Email or username already registered.
+    """
+    if not settings.multi_user_enabled:
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Multi-user mode is not enabled.")
+    if not settings.allow_local_signup:
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Registration is not enabled.")
+    if body.password != body.password_confirm:
+        raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail="Passwords do not match.")
+
+    if db.query(LocalUser).filter(LocalUser.email == body.email).first():
+        raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Email already registered.")
+    if db.query(LocalUser).filter(LocalUser.username == body.username).first():
+        raise HTTPException(status_code=status.HTTP_409_CONFLICT, detail="Username already taken.")
+
+    smtp_configured = bool(settings.email_host)
+
+    if smtp_configured:
+        token = generate_token()
+        user = LocalUser(
+            email=body.email,
+            username=body.username,
+            display_name=body.display_name,
+            hashed_password=hash_password(body.password),
+            is_active=False,
+            email_verification_token=token,
+            email_verification_sent_at=datetime.now(tz=timezone.utc),
+        )
+    else:
+        # No SMTP configured — activate the account immediately.
+        token = None
+        user = LocalUser(
+            email=body.email,
+            username=body.username,
+            display_name=body.display_name,
+            hashed_password=hash_password(body.password),
+            is_active=True,
+        )
+
+    db.add(user)
+
+    profile = UserProfile(
+        user_id=body.email,
+        display_name=body.display_name or body.username,
+    )
+    db.add(profile)
+
+    # Flush to the DB so constraint violations (duplicate key etc.) surface NOW,
+    # before we attempt to send the email. We do NOT commit yet — the commit only
+    # happens after the email is sent successfully so that a failed email leaves
+    # no orphan records in the database.
+    try:
+        db.flush()
+    except Exception:
+        db.rollback()
+        raise
+
+    if smtp_configured and token:
+        base_url = str(request.base_url).rstrip("/")
+        try:
+            send_verification_email(body.email, body.username, token, base_url)
+        except Exception as exc:
+            # Email failed — roll back so no unverifiable user row persists.
+            # The user can simply try registering again once SMTP is fixed.
+            db.rollback()
+            logger.warning("Signup email failed for %s: %s", body.email, exc)
+            raise HTTPException(
+                status_code=status.HTTP_503_SERVICE_UNAVAILABLE,
+                detail=(
+                    "Failed to send verification email. Please check that SMTP is correctly configured and try again."
+                ),
+            ) from exc
+
+    db.commit()
+    logger.info("New local user registered: %s", body.email)
+
+    if smtp_configured:
+        return {"message": "Verification email sent. Please check your inbox.", "email_verification_required": True}
+    return {"message": "Account created successfully. You can now log in.", "email_verification_required": False}
+
+
+@router.get("/verify-email", include_in_schema=False)
+async def verify_email(request: Request, db: DbSession) -> Any:
+    """Activate a local user account from the email verification link.
+
+    Redirects to the login page on failure, or to onboarding/upload on success.
+    """
+    token = request.query_params.get("token", "")
+    user = db.query(LocalUser).filter(LocalUser.email_verification_token == token).first()
+
+    if not user:
+        return RedirectResponse(
+            url="/login?error=Invalid+or+expired+verification+link",
+            status_code=302,
+        )
+    if is_token_expired(user.email_verification_sent_at):
+        return RedirectResponse(
+            url="/login?error=Verification+link+has+expired.+Please+request+a+new+one",
+            status_code=302,
+        )
+
+    user.is_active = True
+    user.email_verification_token = None
+    user.email_verification_sent_at = None
+
+    # Ensure profile exists
+    if not db.query(UserProfile).filter(UserProfile.user_id == user.email).first():
+        db.add(UserProfile(user_id=user.email, display_name=user.display_name or user.username))
+
+    db.commit()
+
+    request.session["user"] = build_session_user(user)
+    logger.info("[SECURITY] EMAIL_VERIFIED user=%s", user.email)
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user.email).first()
+    if profile and not profile.onboarding_completed:
+        post_onboarding = request.session.pop("redirect_after_login", "/upload")
+        request.session["post_onboarding_redirect"] = post_onboarding
+        return RedirectResponse(url="/onboarding", status_code=302)
+    return RedirectResponse(url="/upload", status_code=302)
+
+
+@router.post("/api/auth/resend-verification")
+async def resend_verification(request: Request, body: ResendVerificationBody, db: DbSession) -> dict[str, str]:
+    """Re-send the verification email for a pending account.
+
+    Always returns 200 to avoid leaking whether an email is registered.
+    """
+    user = db.query(LocalUser).filter(LocalUser.email == body.email).first()
+    if not user or user.is_active:
+        return {"message": "Verification email resent if account exists."}
+
+    token = generate_token()
+    user.email_verification_token = token
+    user.email_verification_sent_at = datetime.now(tz=timezone.utc)
+    db.commit()
+
+    base_url = str(request.base_url).rstrip("/")
+    try:
+        send_verification_email(user.email, user.username, token, base_url)
+    except Exception as exc:
+        logger.warning("Failed to resend verification email to %s: %s", user.email, exc)
+
+    return {"message": "Verification email resent if account exists."}
+
+
+@router.post("/api/auth/request-password-reset")
+async def request_password_reset(request: Request, body: PasswordResetRequestBody, db: DbSession) -> dict[str, str]:
+    """Send a password reset email.
+
+    Always returns 200 to avoid leaking whether an email is registered.
+    """
+    user = db.query(LocalUser).filter(LocalUser.email == body.email).first()
+    if not user:
+        return {"message": "Password reset email sent if account exists."}
+
+    token = generate_token()
+    user.password_reset_token = token
+    user.password_reset_sent_at = datetime.now(tz=timezone.utc)
+    db.commit()
+
+    base_url = str(request.base_url).rstrip("/")
+    try:
+        send_password_reset_email(user.email, user.username, token, base_url)
+    except Exception as exc:
+        logger.warning("Failed to send password reset email to %s: %s", user.email, exc)
+
+    return {"message": "Password reset email sent if account exists."}
+
+
+@router.post("/api/auth/reset-password")
+async def reset_password(body: PasswordResetBody, db: DbSession) -> dict[str, str]:
+    """Set a new password using a valid reset token.
+
+    Raises:
+        400: Token is invalid or expired.
+        422: Passwords do not match.
+    """
+    user = db.query(LocalUser).filter(LocalUser.password_reset_token == body.token).first()
+    if not user or is_token_expired(user.password_reset_sent_at):
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Invalid or expired reset token.",
+        )
+    if body.new_password != body.new_password_confirm:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="Passwords do not match.",
+        )
+
+    user.hashed_password = hash_password(body.new_password)
+    user.password_reset_token = None
+    user.password_reset_sent_at = None
+    # Activate the account in case it was still pending email verification.
+    # A valid password-reset token proves control of the registered email address.
+    user.is_active = True
+    db.commit()
+
+    logger.info("[SECURITY] PASSWORD_RESET_SUCCESS user=%s", user.email)
+    return {"message": "Password updated successfully."}
+
+
+@router.post("/api/auth/forgot-username")
+async def forgot_username(body: ForgotUsernameBody, db: DbSession) -> dict[str, str]:
+    """Send a username reminder email.
+
+    Always returns 200 to avoid leaking whether an email is registered.
+    """
+    user = db.query(LocalUser).filter(LocalUser.email == body.email).first()
+    if user:
+        try:
+            send_forgot_username_email(user.email, user.username)
+        except Exception as exc:
+            logger.warning("Failed to send forgot-username email to %s: %s", user.email, exc)
+
+    return {"message": "Username reminder sent if account exists."}

app/api/onboarding.py

@@ -0,0 +1,253 @@
+"""API endpoints for the user onboarding wizard.
+
+Provides a REST interface for the multi-step onboarding flow, allowing
+authenticated users to set their profile, choose a subscription plan,
+select a storage destination, and mark onboarding as complete.
+"""
+
+import logging
+from datetime import datetime, timezone
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from pydantic import BaseModel, Field
+from sqlalchemy.orm import Session
+
+from app.database import get_db
+from app.models import UserProfile
+from app.utils.subscription import TIERS
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/onboarding", tags=["onboarding"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+# ---------------------------------------------------------------------------
+# Auth helper
+# ---------------------------------------------------------------------------
+
+
+def _get_current_user_id(request: Request) -> str:
+    """Extract the stable user_id from the session using the same priority as _ensure_user_profile.
+
+    Priority: sub → preferred_username → email → id.
+
+    Raises:
+        HTTPException: 401 if the user is not authenticated.
+    """
+    user = request.session.get("user")
+    if not user:
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Not authenticated")
+    user_id = user.get("sub") or user.get("preferred_username") or user.get("email") or user.get("id")
+    if not user_id:
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Not authenticated")
+    return user_id
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class ProfileBody(BaseModel):
+    """Body for the profile step of the onboarding wizard."""
+
+    display_name: str | None = Field(default=None, max_length=255)
+    contact_email: str | None = Field(default=None, max_length=255)
+
+
+class PlanBody(BaseModel):
+    """Body for the plan step of the onboarding wizard."""
+
+    subscription_tier: str
+    billing_cycle: str = Field(pattern="^(monthly|yearly)$")
+
+
+class StorageBody(BaseModel):
+    """Body for the storage step of the onboarding wizard."""
+
+    preferred_destination: str | None = Field(default=None, max_length=50)
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _profile_to_dict(profile: UserProfile) -> dict[str, Any]:
+    """Serialize a UserProfile to a plain dict for API responses."""
+    return {
+        "user_id": profile.user_id,
+        "display_name": profile.display_name,
+        "contact_email": profile.contact_email,
+        "subscription_tier": profile.subscription_tier or "free",
+        "subscription_billing_cycle": profile.subscription_billing_cycle or "monthly",
+        "preferred_destination": profile.preferred_destination,
+        "onboarding_completed": bool(profile.onboarding_completed),
+        "onboarding_completed_at": profile.onboarding_completed_at.isoformat()
+        if profile.onboarding_completed_at
+        else None,
+    }
+
+
+def _get_or_create_profile(db: Session, user_id: str) -> UserProfile:
+    """Return the UserProfile for *user_id*, creating one if it does not exist."""
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+    if profile is None:
+        profile = UserProfile(user_id=user_id)
+        db.add(profile)
+        db.flush()
+    return profile
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/status", summary="Get onboarding status for the current user")
+def get_onboarding_status(request: Request, db: DbSession) -> dict[str, Any]:
+    """Return whether onboarding has been completed and the current step.
+
+    The ``step`` field is a best-effort estimate: 1 for brand-new profiles,
+    further along when partial data has already been saved.
+    """
+    user_id = _get_current_user_id(request)
+    profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+
+    if profile is None:
+        return {"completed": False, "step": 1, "profile": None}
+
+    # Derive a sensible current step from saved data so the wizard can resume.
+    step = 1
+    if profile.display_name or profile.contact_email:
+        step = 2
+    if profile.subscription_tier and profile.subscription_tier != "free":
+        step = 3
+    if profile.preferred_destination:
+        step = 4
+    if profile.onboarding_completed:
+        step = 5
+
+    return {
+        "completed": bool(profile.onboarding_completed),
+        "step": step,
+        "profile": _profile_to_dict(profile),
+    }
+
+
+@router.post("/profile", summary="Save profile step during onboarding")
+def save_profile(request: Request, body: ProfileBody, db: DbSession) -> dict[str, Any]:
+    """Persist the user's display name and contact email from the profile step."""
+    user_id = _get_current_user_id(request)
+    profile = _get_or_create_profile(db, user_id)
+
+    if body.display_name is not None:
+        profile.display_name = body.display_name
+    if body.contact_email is not None:
+        profile.contact_email = body.contact_email
+
+    try:
+        db.commit()
+        db.refresh(profile)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Onboarding: saved profile for user %s", user_id)
+    return _profile_to_dict(profile)
+
+
+@router.post("/plan", summary="Save plan selection during onboarding")
+def save_plan(request: Request, body: PlanBody, db: DbSession) -> dict[str, Any]:
+    """Persist the chosen subscription tier and billing cycle from the plan step.
+
+    Raises:
+        HTTPException: 422 if the tier is not a recognised value.
+    """
+    user_id = _get_current_user_id(request)
+
+    if body.subscription_tier not in TIERS:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail=f"Invalid subscription_tier '{body.subscription_tier}'. Valid values: {list(TIERS.keys())}",
+        )
+
+    profile = _get_or_create_profile(db, user_id)
+    old_tier = profile.subscription_tier or "free"
+    profile.subscription_tier = body.subscription_tier
+    profile.subscription_billing_cycle = body.billing_cycle
+
+    try:
+        db.commit()
+        db.refresh(profile)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Onboarding: saved plan %s/%s", body.subscription_tier, body.billing_cycle)
+
+    # Notify admins and fire webhook when the plan actually changes
+    if old_tier != body.subscription_tier:
+        try:
+            from app.utils.notification import notify_plan_changed
+            from app.utils.webhook import dispatch_webhook_event
+
+            notify_plan_changed(user_id, old_tier=old_tier, new_tier=body.subscription_tier, changed_by="user")
+            dispatch_webhook_event(
+                "user.plan_changed",
+                {
+                    "user_id": user_id,
+                    "old_tier": old_tier,
+                    "new_tier": body.subscription_tier,
+                    "billing_cycle": body.billing_cycle,
+                    "changed_by": "user",
+                },
+            )
+        except Exception:
+            logger.exception("Failed to send plan-change notification/webhook for user %s", user_id)
+
+    return _profile_to_dict(profile)
+
+
+@router.post("/storage", summary="Save storage preference during onboarding")
+def save_storage(request: Request, body: StorageBody, db: DbSession) -> dict[str, Any]:
+    """Persist the user's preferred storage destination from the storage step."""
+    user_id = _get_current_user_id(request)
+    profile = _get_or_create_profile(db, user_id)
+    profile.preferred_destination = body.preferred_destination
+
+    try:
+        db.commit()
+        db.refresh(profile)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Onboarding: saved storage preference '%s' for user %s", body.preferred_destination, user_id)
+    return _profile_to_dict(profile)
+
+
+@router.post("/complete", summary="Mark onboarding as completed")
+def complete_onboarding(request: Request, db: DbSession) -> dict[str, Any]:
+    """Set onboarding_completed=True, record the completion timestamp, and return the post-onboarding redirect URL.
+
+    The redirect URL is read from ``request.session["post_onboarding_redirect"]`` (stored by
+    ``oauth_callback`` when it reroutes a first-time user to the wizard) and defaults to
+    ``/upload`` when the session key is absent.
+    """
+    user_id = _get_current_user_id(request)
+    profile = _get_or_create_profile(db, user_id)
+    profile.onboarding_completed = True
+    profile.onboarding_completed_at = datetime.now(tz=timezone.utc)
+
+    try:
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+
+    redirect_url = request.session.pop("post_onboarding_redirect", "/upload")
+    logger.info("Onboarding: completed for user %s, redirecting to %s", user_id, redirect_url)
+    return {"success": True, "redirect_url": redirect_url}

app/api/pipelines.py

@@ -0,0 +1,886 @@
+"""
+Pipelines API endpoints.
+
+Provides full CRUD for processing pipelines and their steps.  Pipelines are
+user-specific: regular users can only manage their own pipelines, while admins
+can also create and manage *system default* pipelines (owner_id = NULL) that
+are visible to all users.
+
+Built-in step types are exposed via GET /api/pipelines/step-types so that UIs
+can render the correct configuration form without hard-coding the catalogue.
+"""
+
+import json
+import logging
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Body, Depends, HTTPException, Request, status
+from pydantic import BaseModel, Field
+from sqlalchemy.orm import Session
+
+from app.auth import get_current_user, get_current_user_id, require_login
+from app.database import get_db
+from app.models import Pipeline, PipelineStep
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/pipelines", tags=["pipelines"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+# ---------------------------------------------------------------------------
+# Built-in step type catalogue
+# ---------------------------------------------------------------------------
+
+PIPELINE_STEP_TYPES: dict[str, dict[str, Any]] = {
+    "convert_to_pdf": {
+        "label": "Convert to PDF",
+        "description": "Convert non-PDF documents to PDF format using Gotenberg.",
+        "config_schema": {},
+    },
+    "check_duplicates": {
+        "label": "Check for Duplicates",
+        "description": "Compare file hash against existing documents to detect duplicates.",
+        "config_schema": {},
+    },
+    "ocr": {
+        "label": "OCR Processing",
+        "description": "Extract text using Azure Document Intelligence or local Tesseract.",
+        "config_schema": {
+            "force_cloud_ocr": {
+                "type": "boolean",
+                "default": False,
+                "description": "Always use cloud OCR even if the PDF already has embedded text.",
+            }
+        },
+    },
+    "extract_metadata": {
+        "label": "Metadata Extraction",
+        "description": "Extract structured metadata (document type, sender, recipient, tags) using AI.",
+        "config_schema": {},
+    },
+    "embed_metadata": {
+        "label": "Embed Metadata into PDF",
+        "description": "Write the extracted metadata into the PDF document properties.",
+        "config_schema": {},
+    },
+    "compute_embedding": {
+        "label": "Compute Text Embedding",
+        "description": "Compute semantic text embeddings for full-text and similarity search.",
+        "config_schema": {},
+    },
+    "send_to_destinations": {
+        "label": "Send to Storage Destinations",
+        "description": "Upload the processed document to all configured storage destinations.",
+        "config_schema": {},
+    },
+    "classify": {
+        "label": "Document Classification",
+        "description": "Classify the document type using AI without full metadata extraction.",
+        "config_schema": {},
+    },
+}
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+MAX_STEPS_PER_PIPELINE = 50
+MAX_NAME_LENGTH = 255
+
+
+def _get_user_id(request: Request) -> str:
+    """Return a stable user identifier from the session.
+
+    Delegates to :func:`app.auth.get_current_user_id` so the same fallback
+    logic ("anonymous") is used consistently throughout the application.
+    """
+    return get_current_user_id(request)
+
+
+def _is_admin(request: Request) -> bool:
+    """Return True if the current session user is an admin."""
+    user = get_current_user(request)
+    return bool(user and user.get("is_admin"))
+
+
+def _can_access_pipeline(pipeline: Pipeline, user_id: str, admin: bool) -> bool:
+    """Return True if the user may read or write this pipeline."""
+    # System pipelines (owner_id=NULL) are readable by everyone; only admins can write
+    if pipeline.owner_id is None:
+        return True
+    # Own pipeline
+    return pipeline.owner_id == user_id or admin
+
+
+def _can_write_pipeline(pipeline: Pipeline, user_id: str, admin: bool) -> bool:
+    """Return True if the user may create/update/delete this pipeline."""
+    if pipeline.owner_id is None:
+        return admin
+    return pipeline.owner_id == user_id or admin
+
+
+def _serialize_step(step: PipelineStep) -> dict[str, Any]:
+    return {
+        "id": step.id,
+        "pipeline_id": step.pipeline_id,
+        "position": step.position,
+        "step_type": step.step_type,
+        "label": step.label,
+        "config": json.loads(step.config) if step.config else {},
+        "enabled": step.enabled,
+        "created_at": step.created_at.isoformat() if step.created_at else None,
+        "updated_at": step.updated_at.isoformat() if step.updated_at else None,
+    }
+
+
+def _serialize_pipeline(pipeline: Pipeline, include_steps: bool = False, db: Session | None = None) -> dict[str, Any]:
+    data: dict[str, Any] = {
+        "id": pipeline.id,
+        "owner_id": pipeline.owner_id,
+        "name": pipeline.name,
+        "description": pipeline.description,
+        "is_default": pipeline.is_default,
+        "is_active": pipeline.is_active,
+        "created_at": pipeline.created_at.isoformat() if pipeline.created_at else None,
+        "updated_at": pipeline.updated_at.isoformat() if pipeline.updated_at else None,
+    }
+    if include_steps and db is not None:
+        steps = (
+            db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline.id).order_by(PipelineStep.position).all()
+        )
+        data["steps"] = [_serialize_step(s) for s in steps]
+    return data
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class PipelineCreate(BaseModel):
+    """Body for creating a pipeline."""
+
+    name: str = Field(..., max_length=MAX_NAME_LENGTH, description="Human-readable pipeline name")
+    description: str | None = Field(default=None, max_length=4096)
+    is_default: bool = Field(default=False)
+    is_active: bool = Field(default=True)
+
+
+class PipelineUpdate(BaseModel):
+    """Body for updating a pipeline (all fields optional)."""
+
+    name: str | None = Field(default=None, max_length=MAX_NAME_LENGTH)
+    description: str | None = Field(default=None, max_length=4096)
+    is_default: bool | None = None
+    is_active: bool | None = None
+
+
+class PipelineStepCreate(BaseModel):
+    """Body for adding a step to a pipeline."""
+
+    step_type: str = Field(..., description="One of the recognised step type keys")
+    label: str | None = Field(default=None, max_length=MAX_NAME_LENGTH)
+    config: dict[str, Any] = Field(default_factory=dict)
+    enabled: bool = Field(default=True)
+    position: int | None = Field(default=None, ge=0, description="Insertion position; appended at end if omitted")
+
+
+class PipelineStepUpdate(BaseModel):
+    """Body for updating a pipeline step (all fields optional)."""
+
+    step_type: str | None = None
+    label: str | None = Field(default=None, max_length=MAX_NAME_LENGTH)
+    config: dict[str, Any] | None = None
+    enabled: bool | None = None
+    position: int | None = Field(default=None, ge=0)
+
+
+# ---------------------------------------------------------------------------
+# Step-types catalogue endpoint (no auth required — it's public metadata)
+# ---------------------------------------------------------------------------
+
+
+@router.get("/step-types")
+def list_step_types() -> dict[str, Any]:
+    """Return the catalogue of built-in pipeline step types.
+
+    Returns:
+        A mapping of step_type key → metadata (label, description, config_schema).
+    """
+    return PIPELINE_STEP_TYPES
+
+
+# ---------------------------------------------------------------------------
+# Pipeline CRUD
+# ---------------------------------------------------------------------------
+
+
+@router.get("")
+@require_login
+def list_pipelines(request: Request, db: DbSession) -> list[dict[str, Any]]:
+    """List pipelines visible to the current user.
+
+    Regular users see: their own pipelines + system pipelines (owner_id=NULL).
+    Admins see: all pipelines from all users.
+
+    Returns:
+        A list of pipeline objects (without steps — use GET /pipelines/{id} for steps).
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    if admin:
+        pipelines = db.query(Pipeline).order_by(Pipeline.owner_id.nullsfirst(), Pipeline.name).all()
+    else:
+        pipelines = (
+            db.query(Pipeline)
+            .filter((Pipeline.owner_id == user_id) | (Pipeline.owner_id.is_(None)))
+            .order_by(Pipeline.owner_id.nullsfirst(), Pipeline.name)
+            .all()
+        )
+
+    return [_serialize_pipeline(p) for p in pipelines]
+
+
+@router.post("", status_code=status.HTTP_201_CREATED)
+@require_login
+def create_pipeline(request: Request, db: DbSession, body: PipelineCreate) -> dict[str, Any]:
+    """Create a new pipeline for the current user.
+
+    Admins can create system default pipelines by passing ``owner_id=null``
+    via the body — however, that is handled implicitly: to create a system
+    pipeline, call ``POST /api/admin/pipelines`` (admin endpoint) instead.
+    Regular users always get their own user_id as owner.
+
+    Returns:
+        The created pipeline object.
+
+    Raises:
+        HTTPException 409: If a pipeline with the same name already exists for this owner.
+    """
+    user_id = _get_user_id(request)
+
+    name = body.name.strip() if body.name else ""
+    if not name:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="name is required",
+        )
+
+    # Enforce unique name per owner
+    existing = db.query(Pipeline).filter(Pipeline.owner_id == user_id, Pipeline.name == name).first()
+    if existing:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"A pipeline named '{name}' already exists",
+        )
+
+    # If this pipeline is marked as default, unset the existing default for this user
+    if body.is_default:
+        _unset_default(db, user_id)
+
+    pipeline = Pipeline(
+        owner_id=user_id,
+        name=name,
+        description=body.description,
+        is_default=body.is_default,
+        is_active=body.is_active,
+    )
+    try:
+        db.add(pipeline)
+        db.commit()
+        db.refresh(pipeline)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to create pipeline user={user_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to create pipeline",
+        )
+
+    logger.info(f"Pipeline created: id={pipeline.id}, owner={user_id}, name={name!r}")
+    return _serialize_pipeline(pipeline)
+
+
+@router.get("/{pipeline_id}")
+@require_login
+def get_pipeline(pipeline_id: int, request: Request, db: DbSession) -> dict[str, Any]:
+    """Return a single pipeline with its steps.
+
+    Path Parameters:
+        pipeline_id: The ID of the pipeline.
+
+    Returns:
+        The pipeline object including its ordered steps.
+
+    Raises:
+        HTTPException 404: If the pipeline does not exist or is not accessible.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    return _serialize_pipeline(pipeline, include_steps=True, db=db)
+
+
+@router.put("/{pipeline_id}")
+@require_login
+def update_pipeline(pipeline_id: int, request: Request, db: DbSession, body: PipelineUpdate) -> dict[str, Any]:
+    """Update a pipeline's metadata.
+
+    Path Parameters:
+        pipeline_id: The ID of the pipeline to update.
+
+    Returns:
+        The updated pipeline object.
+
+    Raises:
+        HTTPException 403: If the caller does not own this pipeline.
+        HTTPException 404: If the pipeline does not exist.
+        HTTPException 409: If the new name conflicts with an existing pipeline.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot modify this pipeline")
+
+    if body.name is not None:
+        new_name = body.name.strip()
+        if not new_name:
+            raise HTTPException(
+                status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+                detail="name must not be empty",
+            )
+        if new_name != pipeline.name:
+            conflict = (
+                db.query(Pipeline)
+                .filter(Pipeline.owner_id == pipeline.owner_id, Pipeline.name == new_name, Pipeline.id != pipeline_id)
+                .first()
+            )
+            if conflict:
+                raise HTTPException(
+                    status_code=status.HTTP_409_CONFLICT,
+                    detail=f"A pipeline named '{new_name}' already exists",
+                )
+        pipeline.name = new_name
+
+    if body.description is not None:
+        pipeline.description = body.description
+
+    if body.is_active is not None:
+        pipeline.is_active = body.is_active
+
+    if body.is_default is not None:
+        if body.is_default and not pipeline.is_default:
+            _unset_default(db, pipeline.owner_id)
+        pipeline.is_default = body.is_default
+
+    try:
+        db.commit()
+        db.refresh(pipeline)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to update pipeline id={pipeline_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to update pipeline",
+        )
+
+    logger.info(f"Pipeline updated: id={pipeline_id}, user={user_id}")
+    return _serialize_pipeline(pipeline, include_steps=True, db=db)
+
+
+@router.delete("/{pipeline_id}", status_code=status.HTTP_204_NO_CONTENT)
+@require_login
+def delete_pipeline(pipeline_id: int, request: Request, db: DbSession) -> None:
+    """Delete a pipeline and all its steps.
+
+    Path Parameters:
+        pipeline_id: The ID of the pipeline to delete.
+
+    Raises:
+        HTTPException 403: If the caller does not own this pipeline.
+        HTTPException 404: If the pipeline does not exist.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot delete this pipeline")
+
+    try:
+        db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline_id).delete()
+        db.delete(pipeline)
+        db.commit()
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to delete pipeline id={pipeline_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to delete pipeline",
+        )
+
+    logger.info(f"Pipeline deleted: id={pipeline_id}, user={user_id}")
+
+
+# ---------------------------------------------------------------------------
+# Admin-only: create system (owner_id=NULL) pipeline
+# ---------------------------------------------------------------------------
+
+
+@router.post("/admin/system", status_code=status.HTTP_201_CREATED, tags=["admin-pipelines"])
+@require_login
+def create_system_pipeline(request: Request, db: DbSession, body: PipelineCreate) -> dict[str, Any]:
+    """Create a system-level (owner_id=NULL) default pipeline.  Admin only.
+
+    System pipelines are visible to all users and can be set as the global
+    default.  Only admins may create them.
+
+    Returns:
+        The created system pipeline.
+
+    Raises:
+        HTTPException 403: If the caller is not an admin.
+        HTTPException 409: If a system pipeline with the same name already exists.
+    """
+    if not _is_admin(request):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+
+    name = body.name.strip() if body.name else ""
+    if not name:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="name is required",
+        )
+
+    existing = db.query(Pipeline).filter(Pipeline.owner_id.is_(None), Pipeline.name == name).first()
+    if existing:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"A system pipeline named '{name}' already exists",
+        )
+
+    if body.is_default:
+        _unset_default(db, None)
+
+    pipeline = Pipeline(
+        owner_id=None,
+        name=name,
+        description=body.description,
+        is_default=body.is_default,
+        is_active=body.is_active,
+    )
+    try:
+        db.add(pipeline)
+        db.commit()
+        db.refresh(pipeline)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to create system pipeline: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to create system pipeline",
+        )
+
+    logger.info(f"System pipeline created: id={pipeline.id}, name={name!r}")
+    return _serialize_pipeline(pipeline)
+
+
+# ---------------------------------------------------------------------------
+# Step management
+# ---------------------------------------------------------------------------
+
+
+@router.post("/{pipeline_id}/steps", status_code=status.HTTP_201_CREATED)
+@require_login
+def add_step(pipeline_id: int, request: Request, db: DbSession, body: PipelineStepCreate) -> dict[str, Any]:
+    """Add a step to a pipeline.
+
+    Steps are automatically appended at the end unless an explicit ``position``
+    is supplied.  All existing steps at or after the insertion position are
+    shifted forward by one.
+
+    Path Parameters:
+        pipeline_id: The pipeline to add the step to.
+
+    Returns:
+        The created step object.
+
+    Raises:
+        HTTPException 403: If the caller cannot modify this pipeline.
+        HTTPException 404: If the pipeline does not exist.
+        HTTPException 422: If the step_type is not recognised.
+        HTTPException 409: If the maximum number of steps per pipeline is reached.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot modify this pipeline")
+
+    if body.step_type not in PIPELINE_STEP_TYPES:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail=f"Unknown step type '{body.step_type}'. Valid types: {sorted(PIPELINE_STEP_TYPES)}",
+        )
+
+    current_count = db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline_id).count()
+    if current_count >= MAX_STEPS_PER_PIPELINE:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"Maximum of {MAX_STEPS_PER_PIPELINE} steps per pipeline reached",
+        )
+
+    # Determine insertion position
+    if body.position is None:
+        max_pos = (
+            db.query(PipelineStep.position)
+            .filter(PipelineStep.pipeline_id == pipeline_id)
+            .order_by(PipelineStep.position.desc())
+            .first()
+        )
+        insert_pos = (max_pos[0] + 1) if max_pos else 0
+    else:
+        insert_pos = body.position
+        # Shift existing steps
+        steps_to_shift = (
+            db.query(PipelineStep)
+            .filter(PipelineStep.pipeline_id == pipeline_id, PipelineStep.position >= insert_pos)
+            .all()
+        )
+        for s in steps_to_shift:
+            s.position += 1
+
+    step = PipelineStep(
+        pipeline_id=pipeline_id,
+        position=insert_pos,
+        step_type=body.step_type,
+        label=body.label,
+        config=json.dumps(body.config) if body.config else None,
+        enabled=body.enabled,
+    )
+    try:
+        db.add(step)
+        db.commit()
+        db.refresh(step)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to add step to pipeline id={pipeline_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to add step",
+        )
+
+    logger.info(f"Step added: pipeline={pipeline_id}, step_type={body.step_type!r}, pos={insert_pos}")
+    return _serialize_step(step)
+
+
+@router.put("/{pipeline_id}/steps/reorder")
+@require_login
+def reorder_steps(
+    pipeline_id: int,
+    request: Request,
+    db: DbSession,
+    step_ids: list[int] = Body(..., description="Ordered list of step IDs representing the new order"),
+) -> list[dict[str, Any]]:
+    """Replace the step order for a pipeline.
+
+    Provide a complete ordered list of *all* step IDs.  Their ``position``
+    values will be reassigned 0, 1, 2, … in the given order.
+
+    Path Parameters:
+        pipeline_id: The pipeline whose steps are being reordered.
+
+    Returns:
+        The updated, ordered list of step objects.
+
+    Raises:
+        HTTPException 422: If the provided list does not contain exactly the
+            current set of step IDs for this pipeline.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot modify this pipeline")
+
+    existing_steps = db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline_id).all()
+    existing_ids = {s.id for s in existing_steps}
+
+    if set(step_ids) != existing_ids or len(step_ids) != len(existing_ids):
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail="step_ids must contain exactly the current step IDs for this pipeline",
+        )
+
+    step_map = {s.id: s for s in existing_steps}
+    for pos, sid in enumerate(step_ids):
+        step_map[sid].position = pos
+
+    try:
+        db.commit()
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to reorder steps for pipeline id={pipeline_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to reorder steps",
+        )
+
+    updated = (
+        db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline_id).order_by(PipelineStep.position).all()
+    )
+    return [_serialize_step(s) for s in updated]
+
+
+@router.put("/{pipeline_id}/steps/{step_id}")
+@require_login
+def update_step(
+    pipeline_id: int, step_id: int, request: Request, db: DbSession, body: PipelineStepUpdate
+) -> dict[str, Any]:
+    """Update an existing pipeline step.
+
+    Path Parameters:
+        pipeline_id: The owning pipeline.
+        step_id: The step to update.
+
+    Returns:
+        The updated step object.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot modify this pipeline")
+
+    step = db.query(PipelineStep).filter(PipelineStep.id == step_id, PipelineStep.pipeline_id == pipeline_id).first()
+    if not step:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Step not found")
+
+    if body.step_type is not None:
+        if body.step_type not in PIPELINE_STEP_TYPES:
+            raise HTTPException(
+                status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+                detail=f"Unknown step type '{body.step_type}'",
+            )
+        step.step_type = body.step_type
+
+    if body.label is not None:
+        step.label = body.label
+
+    if body.config is not None:
+        step.config = json.dumps(body.config)
+
+    if body.enabled is not None:
+        step.enabled = body.enabled
+
+    if body.position is not None and body.position != step.position:
+        old_pos = step.position
+        new_pos = body.position
+        if new_pos > old_pos:
+            # Moving down: shift intervening steps up
+            db.query(PipelineStep).filter(
+                PipelineStep.pipeline_id == pipeline_id,
+                PipelineStep.position > old_pos,
+                PipelineStep.position <= new_pos,
+                PipelineStep.id != step_id,
+            ).update({"position": PipelineStep.position - 1})
+        else:
+            # Moving up: shift intervening steps down
+            db.query(PipelineStep).filter(
+                PipelineStep.pipeline_id == pipeline_id,
+                PipelineStep.position >= new_pos,
+                PipelineStep.position < old_pos,
+                PipelineStep.id != step_id,
+            ).update({"position": PipelineStep.position + 1})
+        step.position = new_pos
+
+    try:
+        db.commit()
+        db.refresh(step)
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to update step id={step_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to update step",
+        )
+
+    return _serialize_step(step)
+
+
+@router.delete("/{pipeline_id}/steps/{step_id}", status_code=status.HTTP_204_NO_CONTENT)
+@require_login
+def delete_step(pipeline_id: int, step_id: int, request: Request, db: DbSession) -> None:
+    """Delete a step from a pipeline.
+
+    Path Parameters:
+        pipeline_id: The owning pipeline.
+        step_id: The step to delete.
+    """
+    user_id = _get_user_id(request)
+    admin = _is_admin(request)
+
+    pipeline = db.query(Pipeline).filter(Pipeline.id == pipeline_id).first()
+    if not pipeline or not _can_access_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Pipeline not found")
+
+    if not _can_write_pipeline(pipeline, user_id, admin):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Cannot modify this pipeline")
+
+    step = db.query(PipelineStep).filter(PipelineStep.id == step_id, PipelineStep.pipeline_id == pipeline_id).first()
+    if not step:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Step not found")
+
+    deleted_pos = step.position
+    try:
+        db.delete(step)
+        # Compact remaining step positions
+        db.query(PipelineStep).filter(
+            PipelineStep.pipeline_id == pipeline_id,
+            PipelineStep.position > deleted_pos,
+        ).update({"position": PipelineStep.position - 1})
+        db.commit()
+    except Exception as exc:
+        db.rollback()
+        logger.exception(f"Failed to delete step id={step_id}: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to delete step",
+        )
+
+    logger.info(f"Step deleted: id={step_id}, pipeline={pipeline_id}")
+
+
+# ---------------------------------------------------------------------------
+# Helper: unset default flag for an owner
+# ---------------------------------------------------------------------------
+
+
+def _unset_default(db: Session, owner_id: str | None) -> None:
+    """Clear the is_default flag on all pipelines for the given owner."""
+    if owner_id is None:
+        db.query(Pipeline).filter(Pipeline.owner_id.is_(None), Pipeline.is_default.is_(True)).update(
+            {"is_default": False}
+        )
+    else:
+        db.query(Pipeline).filter(Pipeline.owner_id == owner_id, Pipeline.is_default.is_(True)).update(
+            {"is_default": False}
+        )
+
+
+# ---------------------------------------------------------------------------
+# Default system pipeline seeding
+# ---------------------------------------------------------------------------
+
+# The steps that make up the standard document-processing workflow.  The order
+# here mirrors what the existing Celery-based pipeline executes for every
+# uploaded file.
+_DEFAULT_PIPELINE_STEPS: list[tuple[str, str]] = [
+    ("convert_to_pdf", "Convert to PDF"),
+    ("check_duplicates", "Check for Duplicates"),
+    ("ocr", "OCR Processing"),
+    ("extract_metadata", "Extract Metadata"),
+    ("embed_metadata", "Embed Metadata into PDF"),
+    ("compute_embedding", "Compute Text Embedding"),
+    ("send_to_destinations", "Send to Storage Destinations"),
+]
+
+#: Human-readable name shown in the management UI for the auto-seeded pipeline.
+DEFAULT_PIPELINE_NAME = "Standard Processing Pipeline"
+
+
+def seed_default_pipeline(db: Session) -> int:
+    """Ensure a system-owned default pipeline exists in the database.
+
+    This function is idempotent — it is a no-op when any system pipeline
+    (``owner_id IS NULL``) already exists.  It is intended to be called once
+    at application startup (in ``app.main.lifespan``) so that the pipeline
+    management UI always shows the default workflow that mirrors the existing
+    Celery-based processing steps.
+
+    The created pipeline:
+
+    * ``owner_id = None`` — owned by the system, visible to all users
+    * ``is_default = True`` — selected automatically for new documents
+    * Steps (in order): convert_to_pdf → check_duplicates → ocr →
+      extract_metadata → embed_metadata → compute_embedding →
+      send_to_destinations
+
+    Args:
+        db: An active SQLAlchemy session.
+
+    Returns:
+        ``1`` if a new pipeline was created, ``0`` if one already existed.
+    """
+    try:
+        if db.query(Pipeline).filter(Pipeline.owner_id.is_(None)).count() > 0:
+            return 0
+    except Exception:
+        # Table may not exist yet during the very first migration run.
+        return 0
+
+    pipeline = Pipeline(
+        owner_id=None,
+        name=DEFAULT_PIPELINE_NAME,
+        description=(
+            "The standard document processing workflow: PDF conversion, "
+            "duplicate detection, OCR, metadata extraction and embedding, "
+            "semantic embeddings, and final distribution to storage destinations."
+        ),
+        is_default=True,
+        is_active=True,
+    )
+    db.add(pipeline)
+    try:
+        db.flush()  # Assign pipeline.id without committing yet
+    except Exception as exc:  # pragma: no cover
+        db.rollback()
+        logger.error(f"Failed to create default pipeline: {exc}")
+        return 0
+
+    for pos, (step_type, label) in enumerate(_DEFAULT_PIPELINE_STEPS):
+        db.add(
+            PipelineStep(
+                pipeline_id=pipeline.id,
+                position=pos,
+                step_type=step_type,
+                label=label,
+                enabled=True,
+            )
+        )
+
+    try:
+        db.commit()
+        logger.info("Seeded default system pipeline: '%s' (id=%d)", DEFAULT_PIPELINE_NAME, pipeline.id)
+    except Exception as exc:  # pragma: no cover
+        db.rollback()
+        logger.error(f"Failed to seed default pipeline steps: {exc}")
+        return 0
+
+    return 1

app/api/plans.py

@@ -0,0 +1,273 @@
+"""REST API for subscription plan CRUD.
+
+Endpoints:
+  GET  /api/plans/          — list active plans (public)
+  GET  /api/plans/admin     — list all plans inc. inactive (admin only)
+  POST /api/plans/          — create plan (admin only)
+  GET  /api/plans/{plan_id} — get single active plan (public)
+  PUT  /api/plans/{plan_id} — update plan (admin only)
+  DELETE /api/plans/{plan_id} — delete plan (admin only)
+  POST /api/plans/seed      — seed default plans (admin only)
+  POST /api/plans/reorder   — set sort_order for multiple plans (admin only)
+"""
+
+import json
+import logging
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from pydantic import BaseModel, Field
+from sqlalchemy.orm import Session
+
+from app.database import get_db
+from app.models import SubscriptionPlan
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/plans", tags=["plans"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+# ---------------------------------------------------------------------------
+# Auth helper (admin-only)
+# ---------------------------------------------------------------------------
+
+
+def _require_admin(request: Request) -> dict:
+    """Ensure the caller is an admin. Raises 403 otherwise."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+    return user
+
+
+AdminUser = Annotated[dict, Depends(_require_admin)]
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class PlanUpsert(BaseModel):
+    """Body for creating or updating a subscription plan."""
+
+    name: str
+    tagline: str | None = None
+    price_monthly: float = 0.0
+    price_yearly: float = 0.0
+    trial_days: int = 0
+    lifetime_file_limit: int = 0
+    daily_upload_limit: int = 0
+    monthly_upload_limit: int = 0
+    max_storage_destinations: int = 0
+    max_ocr_pages_monthly: int = 0
+    max_file_size_mb: int = 0
+    max_mailboxes: int = 0
+    overage_percent: int = Field(default=20, ge=0, le=200)
+    allow_overage_billing: bool = False
+    overage_price_per_doc: float | None = None
+    overage_price_per_ocr_page: float | None = None
+    is_active: bool = True
+    is_highlighted: bool = False
+    badge_text: str | None = None
+    cta_text: str = "Get started"
+    sort_order: int = 0
+    features: list[str] = []
+    api_access: bool = False
+
+
+class ReorderBody(BaseModel):
+    """Body for reordering plans."""
+
+    order: list[str]
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _plan_to_response(plan: SubscriptionPlan) -> dict[str, Any]:
+    features: list[str] = []
+    if plan.features:
+        try:
+            features = json.loads(plan.features)
+        except (json.JSONDecodeError, TypeError):
+            features = []
+    return {
+        "id": plan.id,
+        "plan_id": plan.plan_id,
+        "name": plan.name,
+        "tagline": plan.tagline,
+        "price_monthly": plan.price_monthly,
+        "price_yearly": plan.price_yearly,
+        "trial_days": plan.trial_days,
+        "lifetime_file_limit": plan.lifetime_file_limit,
+        "daily_upload_limit": plan.daily_upload_limit,
+        "monthly_upload_limit": plan.monthly_upload_limit,
+        "max_storage_destinations": plan.max_storage_destinations,
+        "max_ocr_pages_monthly": plan.max_ocr_pages_monthly,
+        "max_file_size_mb": plan.max_file_size_mb,
+        "max_mailboxes": plan.max_mailboxes,
+        "overage_percent": plan.overage_percent,
+        "allow_overage_billing": plan.allow_overage_billing,
+        "overage_price_per_doc": plan.overage_price_per_doc,
+        "overage_price_per_ocr_page": plan.overage_price_per_ocr_page,
+        "is_active": plan.is_active,
+        "is_highlighted": plan.is_highlighted,
+        "badge_text": plan.badge_text,
+        "cta_text": plan.cta_text,
+        "sort_order": plan.sort_order,
+        "features": features,
+        "api_access": plan.api_access,
+        "created_at": plan.created_at.isoformat() if plan.created_at else None,
+        "updated_at": plan.updated_at.isoformat() if plan.updated_at else None,
+    }
+
+
+def _apply_body(plan: SubscriptionPlan, body: PlanUpsert) -> None:
+    """Apply PlanUpsert fields onto a SubscriptionPlan ORM object."""
+    plan.name = body.name
+    plan.tagline = body.tagline
+    plan.price_monthly = body.price_monthly
+    plan.price_yearly = body.price_yearly
+    plan.trial_days = body.trial_days
+    plan.lifetime_file_limit = body.lifetime_file_limit
+    plan.daily_upload_limit = body.daily_upload_limit
+    plan.monthly_upload_limit = body.monthly_upload_limit
+    plan.max_storage_destinations = body.max_storage_destinations
+    plan.max_ocr_pages_monthly = body.max_ocr_pages_monthly
+    plan.max_file_size_mb = body.max_file_size_mb
+    plan.max_mailboxes = body.max_mailboxes
+    plan.overage_percent = body.overage_percent
+    plan.allow_overage_billing = body.allow_overage_billing
+    plan.overage_price_per_doc = body.overage_price_per_doc
+    plan.overage_price_per_ocr_page = body.overage_price_per_ocr_page
+    plan.is_active = body.is_active
+    plan.is_highlighted = body.is_highlighted
+    plan.badge_text = body.badge_text
+    plan.cta_text = body.cta_text
+    plan.sort_order = body.sort_order
+    plan.features = json.dumps(body.features)
+    plan.api_access = body.api_access
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/", summary="List active plans (public)")
+def list_active_plans(db: DbSession) -> dict[str, Any]:
+    """Return all active plans in sort order. Public endpoint — no auth required."""
+    plans = (
+        db.query(SubscriptionPlan)
+        .filter(SubscriptionPlan.is_active.is_(True))
+        .order_by(SubscriptionPlan.sort_order)
+        .all()
+    )
+    return {"plans": [_plan_to_response(p) for p in plans]}
+
+
+@router.get("/admin", summary="List all plans including inactive (admin only)")
+def list_all_plans(db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Return all plans (active and inactive) in sort order. Admin only."""
+    plans = db.query(SubscriptionPlan).order_by(SubscriptionPlan.sort_order).all()
+    return {"plans": [_plan_to_response(p) for p in plans]}
+
+
+@router.post("/seed", summary="Seed default plans (admin only)", status_code=status.HTTP_200_OK)
+def seed_plans(db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Seed the subscription_plans table from TIER_DEFAULTS. No-op if plans already exist."""
+    from app.utils.subscription import seed_default_plans
+
+    inserted = seed_default_plans(db)
+    return {"inserted": inserted, "message": f"Seeded {inserted} default plan(s)."}
+
+
+@router.post("/reorder", summary="Reorder plans (admin only)")
+def reorder_plans(body: ReorderBody, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Update sort_order for each plan_id in *body.order* (position = index in list)."""
+    updated = 0
+    for sort_order, plan_id in enumerate(body.order):
+        plan = db.query(SubscriptionPlan).filter(SubscriptionPlan.plan_id == plan_id).first()
+        if plan:
+            plan.sort_order = sort_order
+            updated += 1
+    try:
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="Failed to reorder plans")
+    return {"updated": updated}
+
+
+@router.post("/", summary="Create a new plan (admin only)", status_code=status.HTTP_201_CREATED)
+def create_plan(plan_id: str, body: PlanUpsert, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Create a new subscription plan with the given *plan_id* slug."""
+    existing = db.query(SubscriptionPlan).filter(SubscriptionPlan.plan_id == plan_id).first()
+    if existing:
+        raise HTTPException(
+            status_code=status.HTTP_409_CONFLICT,
+            detail=f"Plan '{plan_id}' already exists.",
+        )
+    plan = SubscriptionPlan(plan_id=plan_id)
+    _apply_body(plan, body)
+    db.add(plan)
+    try:
+        db.commit()
+        db.refresh(plan)
+    except Exception:
+        db.rollback()
+        raise
+    logger.info("Admin created subscription plan '%s'", plan_id)
+    return _plan_to_response(plan)
+
+
+@router.get("/{plan_id}", summary="Get a single active plan (public)")
+def get_plan(plan_id: str, db: DbSession) -> dict[str, Any]:
+    """Return a single active plan by plan_id. Public endpoint."""
+    plan = (
+        db.query(SubscriptionPlan)
+        .filter(
+            SubscriptionPlan.plan_id == plan_id,
+            SubscriptionPlan.is_active.is_(True),
+        )
+        .first()
+    )
+    if not plan:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Plan '{plan_id}' not found.")
+    return _plan_to_response(plan)
+
+
+@router.put("/{plan_id}", summary="Update an existing plan (admin only)")
+def update_plan(plan_id: str, body: PlanUpsert, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Update an existing subscription plan. Admin only."""
+    plan = db.query(SubscriptionPlan).filter(SubscriptionPlan.plan_id == plan_id).first()
+    if not plan:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Plan '{plan_id}' not found.")
+    _apply_body(plan, body)
+    try:
+        db.commit()
+        db.refresh(plan)
+    except Exception:
+        db.rollback()
+        raise
+    logger.info("Admin updated subscription plan '%s'", plan_id)
+    return _plan_to_response(plan)
+
+
+@router.delete("/{plan_id}", summary="Delete a plan (admin only)", status_code=status.HTTP_204_NO_CONTENT)
+def delete_plan(plan_id: str, db: DbSession, _admin: AdminUser) -> None:
+    """Delete a subscription plan. Admin only."""
+    plan = db.query(SubscriptionPlan).filter(SubscriptionPlan.plan_id == plan_id).first()
+    if not plan:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=f"Plan '{plan_id}' not found.")
+    try:
+        db.delete(plan)
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+    logger.info("Admin deleted subscription plan '%s'", plan_id)

app/api/settings.py

@@ -446,6 +446,42 @@ async def install_ocr_languages(request: Request, admin: AdminUser):
         )
 
 
+@router.get("/{key}/suggestions")
+async def get_setting_suggestions(
+    key: str,
+    request: Request,
+    q: str = "",
+    limit: int = 10,
+):
+    """
+    Return autocomplete suggestions for a setting key.
+
+    Fetches values dynamically from cloud SDKs, installed tools, or
+    curated static lists depending on the setting.  Results are filtered
+    by case-insensitive substring match on the ``q`` parameter.
+
+    This endpoint does **not** require admin privileges so that the
+    autocomplete widget works for any authenticated user viewing settings.
+    """
+    from app.utils.suggestion_providers import SUGGESTION_PROVIDERS, get_suggestions  # noqa: PLC0415
+
+    if key not in SUGGESTION_PROVIDERS:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"No suggestions available for setting '{key}'",
+        )
+
+    try:
+        suggestions = get_suggestions(key, query=q, limit=max(1, min(limit, 50)))
+        return {"key": key, "suggestions": suggestions}
+    except Exception as e:
+        logger.error(f"Error fetching suggestions for {key}: {e}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to fetch suggestions",
+        )
+
+
 @router.get("/{key}/history")
 async def get_key_history(key: str, request: Request, db: DbSession, admin: AdminUser):
     """

app/api/similarity.py

@@ -0,0 +1,473 @@
+"""Document similarity API endpoints.
+
+Provides endpoints to find documents similar to a given file based on
+text embeddings and cosine similarity scoring, plus debug/diagnostic
+endpoints for inspecting and triggering embedding computation.
+"""
+
+import json
+import logging
+from typing import Annotated
+
+from fastapi import APIRouter, Depends, HTTPException, Query, Request, status
+from sqlalchemy.orm import Session
+
+from app.auth import require_login
+from app.config import settings
+from app.database import get_db
+from app.models import FileRecord
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter()
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+@router.get("/files/{file_id}/similar")
+@require_login
+def get_similar_documents(
+    request: Request,
+    file_id: int,
+    db: DbSession,
+    limit: int = Query(5, ge=1, le=20, description="Maximum number of similar documents to return"),
+    threshold: float = Query(0.3, ge=0.0, le=1.0, description="Minimum similarity score (0–1)"),
+):
+    """Find documents similar to the specified file.
+
+    Uses text embeddings generated from OCR-extracted text and cosine
+    similarity to rank documents by relevance.  Similarity scores range
+    from 0 (completely different) to 1 (identical content).
+
+    Embeddings are generated on first access and cached for subsequent
+    requests.  Documents without OCR text are excluded.
+
+    Query Parameters:
+        - limit: Maximum results to return (default: 5, max: 20)
+        - threshold: Minimum similarity score to include (default: 0.3)
+
+    Example:
+    ```
+    GET /api/files/42/similar?limit=5&threshold=0.5
+    ```
+
+    Response:
+    ```json
+    {
+      "file_id": 42,
+      "similar_documents": [
+        {
+          "file_id": 15,
+          "original_filename": "Invoice_2026-01.pdf",
+          "document_title": "January Invoice",
+          "similarity_score": 0.8934,
+          "mime_type": "application/pdf",
+          "created_at": "2026-01-15T10:30:00+00:00"
+        }
+      ],
+      "count": 1
+    }
+    ```
+    """
+    # Verify the file exists
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    if not file_record.ocr_text or not file_record.ocr_text.strip():
+        return {
+            "file_id": file_id,
+            "similar_documents": [],
+            "count": 0,
+            "message": "No OCR text available for similarity comparison",
+        }
+
+    # Check whether an embedding has been computed yet
+    if not file_record.embedding:
+        return {
+            "file_id": file_id,
+            "similar_documents": [],
+            "count": 0,
+            "message": (
+                "Embedding not yet computed for this file. "
+                "It will be generated automatically during processing or via the backfill task. "
+                "You can also trigger it manually with POST /api/files/{file_id}/compute-embedding."
+            ),
+        }
+
+    try:
+        from app.utils.similarity import find_similar_documents
+
+        similar = find_similar_documents(db, file_id, limit=limit, threshold=threshold)
+
+        return {
+            "file_id": file_id,
+            "similar_documents": similar,
+            "count": len(similar),
+        }
+    except Exception as e:
+        logger.error(f"Error finding similar documents for file {file_id}: {e}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to compute document similarity",
+        )
+
+
+# ---------------------------------------------------------------------------
+# Debug / diagnostic endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/files/{file_id}/embedding-status")
+@require_login
+def get_embedding_status(
+    request: Request,
+    file_id: int,
+    db: DbSession,
+):
+    """Return the embedding status for a single file.
+
+    Useful for debugging whether the embedding has been computed
+    and cached for a given document.
+
+    Response:
+    ```json
+    {
+      "file_id": 42,
+      "has_embedding": true,
+      "embedding_dimensions": 1536,
+      "has_ocr_text": true,
+      "ocr_text_length": 4200,
+      "embedding_model": "text-embedding-3-small"
+    }
+    ```
+    """
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    has_embedding = False
+    embedding_dimensions = None
+    if file_record.embedding:
+        try:
+            parsed = json.loads(file_record.embedding)
+            has_embedding = True
+            embedding_dimensions = len(parsed)
+        except (json.JSONDecodeError, TypeError):
+            pass
+
+    has_ocr_text = bool(file_record.ocr_text and file_record.ocr_text.strip())
+
+    return {
+        "file_id": file_id,
+        "has_embedding": has_embedding,
+        "embedding_dimensions": embedding_dimensions,
+        "has_ocr_text": has_ocr_text,
+        "ocr_text_length": len(file_record.ocr_text) if file_record.ocr_text else 0,
+        "embedding_model": settings.embedding_model,
+    }
+
+
+@router.post("/files/{file_id}/compute-embedding")
+@require_login
+def trigger_compute_embedding(
+    request: Request,
+    file_id: int,
+    db: DbSession,
+):
+    """Trigger embedding computation for a single file.
+
+    If the file already has a cached embedding it will be recomputed.
+    The computation happens synchronously so the caller receives the
+    result immediately.
+
+    Response:
+    ```json
+    {
+      "file_id": 42,
+      "status": "success",
+      "embedding_dimensions": 1536
+    }
+    ```
+    """
+    file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not file_record:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="File not found")
+
+    if not file_record.ocr_text or not file_record.ocr_text.strip():
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="File has no OCR text — cannot generate embedding",
+        )
+
+    try:
+        from app.utils.similarity import generate_embedding
+
+        # Clear cached embedding to force recomputation
+        file_record.embedding = None
+        db.flush()
+
+        embedding = generate_embedding(file_record.ocr_text)
+        file_record.embedding = json.dumps(embedding)
+        db.commit()
+
+        return {
+            "file_id": file_id,
+            "status": "success",
+            "embedding_dimensions": len(embedding),
+        }
+    except Exception as e:
+        db.rollback()
+        logger.error(f"Failed to compute embedding for file {file_id}: {e}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail=f"Embedding computation failed: {e}",
+        )
+
+
+@router.get("/diagnostic/embeddings")
+@require_login
+def get_embeddings_overview(
+    request: Request,
+    db: DbSession,
+):
+    """Return an overview of embedding status across all files.
+
+    Provides aggregate counts as well as a per-file breakdown so an
+    administrator can quickly identify documents that are missing
+    embeddings.
+
+    Response:
+    ```json
+    {
+      "total_files": 120,
+      "files_with_ocr_text": 95,
+      "files_with_embedding": 42,
+      "files_missing_embedding": 53,
+      "embedding_model": "text-embedding-3-small",
+      "files": [
+        {
+          "file_id": 1,
+          "original_filename": "invoice.pdf",
+          "has_ocr_text": true,
+          "has_embedding": true,
+          "embedding_dimensions": 1536
+        }
+      ]
+    }
+    ```
+    """
+    # Use column-only query to avoid loading full ORM objects into memory
+    all_files = (
+        db.query(
+            FileRecord.id,
+            FileRecord.original_filename,
+            FileRecord.ocr_text,
+            FileRecord.embedding,
+        )
+        .order_by(FileRecord.id.desc())
+        .all()
+    )
+
+    files_info = []
+    total_with_ocr = 0
+    total_with_embedding = 0
+
+    for f in all_files:
+        has_ocr = bool(f.ocr_text and f.ocr_text.strip())
+        has_emb = False
+        emb_dims = None
+
+        if f.embedding:
+            try:
+                parsed = json.loads(f.embedding)
+                has_emb = True
+                emb_dims = len(parsed)
+            except (json.JSONDecodeError, TypeError):
+                pass
+
+        if has_ocr:
+            total_with_ocr += 1
+        if has_emb:
+            total_with_embedding += 1
+
+        files_info.append(
+            {
+                "file_id": f.id,
+                "original_filename": f.original_filename,
+                "has_ocr_text": has_ocr,
+                "has_embedding": has_emb,
+                "embedding_dimensions": emb_dims,
+            }
+        )
+
+    return {
+        "total_files": len(all_files),
+        "files_with_ocr_text": total_with_ocr,
+        "files_with_embedding": total_with_embedding,
+        "files_missing_embedding": total_with_ocr - total_with_embedding,
+        "embedding_model": settings.embedding_model,
+        "files": files_info,
+    }
+
+
+@router.post("/diagnostic/compute-all-embeddings")
+@require_login
+def trigger_compute_all_embeddings(
+    request: Request,
+    db: DbSession,
+):
+    """Queue embedding computation for all files that have OCR text but no embedding.
+
+    Each file is processed as a separate Celery task so the endpoint
+    returns immediately.
+
+    Response:
+    ```json
+    {
+      "status": "queued",
+      "files_queued": 53
+    }
+    ```
+    """
+    candidates = (
+        db.query(FileRecord)
+        .filter(
+            FileRecord.ocr_text.isnot(None),
+            FileRecord.ocr_text != "",
+            (FileRecord.embedding.is_(None)) | (FileRecord.embedding == ""),
+        )
+        .all()
+    )
+
+    queued = 0
+    for f in candidates:
+        try:
+            from app.tasks.compute_embedding import compute_document_embedding
+
+            compute_document_embedding.delay(f.id)
+            queued += 1
+        except Exception as e:
+            logger.warning(f"Could not queue embedding for file {f.id}: {e}")
+
+    return {
+        "status": "queued",
+        "files_queued": queued,
+    }
+
+
+@router.get("/similarity/pairs")
+@require_login
+def get_similarity_pairs(
+    request: Request,
+    db: DbSession,
+    threshold: float = Query(0.7, ge=0.0, le=1.0, description="Minimum similarity score for a pair"),
+    limit: int = Query(50, ge=1, le=200, description="Maximum number of pairs to return"),
+    page: int = Query(1, ge=1, description="Page number"),
+):
+    """Return pairs of documents with high similarity across the entire corpus.
+
+    Unlike the per-file ``/files/{id}/similar`` endpoint, this scans every
+    document that has a pre-computed embedding and returns **all** pairs
+    whose cosine similarity exceeds ``threshold``, sorted by descending
+    score.
+
+    To keep memory bounded the query loads only the columns needed for
+    scoring and streams results in chunks.
+
+    Response:
+    ```json
+    {
+      "pairs": [
+        {
+          "file_a": {"file_id": 1, "original_filename": "invoice_jan.pdf", ...},
+          "file_b": {"file_id": 5, "original_filename": "invoice_feb.pdf", ...},
+          "similarity_score": 0.94
+        }
+      ],
+      "total_pairs": 12,
+      "threshold": 0.7,
+      "page": 1,
+      "pages": 1,
+      "embedding_coverage": {"total_files": 120, "files_with_embedding": 95}
+    }
+    ```
+    """
+    from app.utils.similarity import cosine_similarity
+
+    # Load all files that have embeddings (columns only for efficiency)
+    rows = (
+        db.query(
+            FileRecord.id,
+            FileRecord.original_filename,
+            FileRecord.document_title,
+            FileRecord.mime_type,
+            FileRecord.created_at,
+            FileRecord.embedding,
+        )
+        .filter(
+            FileRecord.embedding.isnot(None),
+            FileRecord.embedding != "",
+        )
+        .order_by(FileRecord.id)
+        .all()
+    )
+
+    # Parse embeddings upfront
+    parsed: list[tuple] = []
+    for row in rows:
+        try:
+            vec = json.loads(row.embedding)
+            parsed.append((row, vec))
+        except (json.JSONDecodeError, TypeError):
+            continue
+
+    # Pairwise comparison (triangle: i < j avoids duplicating A↔B / B↔A)
+    all_pairs: list[dict] = []
+    for i in range(len(parsed)):
+        row_a, vec_a = parsed[i]
+        for j in range(i + 1, len(parsed)):
+            row_b, vec_b = parsed[j]
+            score = cosine_similarity(vec_a, vec_b)
+            if score >= threshold:
+                all_pairs.append(
+                    {
+                        "file_a": _row_to_dict(row_a),
+                        "file_b": _row_to_dict(row_b),
+                        "similarity_score": round(score, 4),
+                    }
+                )
+
+    # Sort by score descending
+    all_pairs.sort(key=lambda p: p["similarity_score"], reverse=True)
+
+    total_pairs = len(all_pairs)
+    total_pages = max(1, (total_pairs + limit - 1) // limit)
+    offset = (page - 1) * limit
+    page_pairs = all_pairs[offset : offset + limit]
+
+    total_files = db.query(FileRecord).count()
+
+    return {
+        "pairs": page_pairs,
+        "total_pairs": total_pairs,
+        "threshold": threshold,
+        "page": page,
+        "pages": total_pages,
+        "per_page": limit,
+        "embedding_coverage": {
+            "total_files": total_files,
+            "files_with_embedding": len(parsed),
+        },
+    }
+
+
+def _row_to_dict(row) -> dict:
+    """Serialise a column-only query row to a dict for JSON responses."""
+    return {
+        "file_id": row.id,
+        "original_filename": row.original_filename,
+        "document_title": row.document_title,
+        "mime_type": row.mime_type,
+        "created_at": row.created_at.isoformat() if row.created_at else None,
+    }

app/api/subscriptions.py

@@ -0,0 +1,259 @@
+"""API endpoints for subscription tiers and usage statistics.
+
+Public endpoints:
+  GET  /api/subscriptions/tiers        — list all available plans
+  GET  /api/subscriptions/my           — current user's plan + usage (auth required)
+  POST /api/subscriptions/change       — request a plan change (auth required)
+  DELETE /api/subscriptions/change     — cancel a pending plan change (auth required)
+  GET  /api/subscriptions/platform     — platform-wide stats (admin only)
+"""
+
+import logging
+from datetime import datetime, timezone
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from pydantic import BaseModel
+from sqlalchemy import func
+from sqlalchemy.orm import Session
+
+from app.api.admin_users import _require_admin
+from app.database import get_db
+from app.utils.subscription import (
+    TIER_ORDER,
+    TIERS,
+    SubscriptionChangeError,
+    apply_pending_subscription_changes,
+    cancel_pending_subscription_change,
+    get_all_tiers,
+    get_tier,
+    get_user_tier_id,
+    get_user_usage,
+    request_subscription_change,
+)
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter(prefix="/subscriptions", tags=["subscriptions"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+AdminUser = Annotated[dict, Depends(_require_admin)]
+
+
+# ---------------------------------------------------------------------------
+# Request / response models
+# ---------------------------------------------------------------------------
+
+
+class SubscriptionChangeRequest(BaseModel):
+    """Request body for a subscription plan change."""
+
+    plan_id: str
+    billing_cycle: str = "monthly"  # "monthly" | "yearly"
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _get_owner_id(request: Request) -> str:
+    """Extract the authenticated user's owner_id from the session."""
+    user = request.session.get("user") or {}
+    return user.get("username") or user.get("email") or user.get("sub") or ""
+
+
+def _require_authenticated(request: Request) -> str:
+    """Return the owner_id or raise 401."""
+    owner_id = _get_owner_id(request)
+    if not owner_id:
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
+    return owner_id
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/tiers", summary="List all subscription tiers")
+def list_tiers() -> dict[str, Any]:
+    """Return the full list of subscription plans in display order."""
+    return {
+        "tiers": get_all_tiers(),
+        "order": TIER_ORDER,
+        "default": "free",
+    }
+
+
+@router.get("/my", summary="Get current user's subscription and usage")
+def my_subscription(request: Request, db: DbSession) -> dict[str, Any]:
+    """Return the authenticated user's subscription tier and current usage counts.
+
+    Also applies any pending subscription changes that have become due.
+    """
+    from app.config import settings
+    from app.models import UserProfile
+
+    user = request.session.get("user")
+
+    if not settings.multi_user_enabled:
+        # In single-user mode there is no concept of a subscription plan
+        return {
+            "multi_user_mode": False,
+            "tier": TIERS["business"],  # unrestricted
+            "usage": None,
+        }
+
+    if not user:
+        raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="Authentication required")
+
+    owner_id: str = user.get("username") or user.get("email") or user.get("sub") or ""
+
+    # Apply any pending change that has become due
+    apply_pending_subscription_changes(db, owner_id)
+
+    tier_id = get_user_tier_id(db, owner_id)
+    tier = get_tier(tier_id, db)
+    usage = get_user_usage(db, owner_id)
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    pending_tier_id: str | None = profile.subscription_change_pending_tier if profile else None
+    pending_date: str | None = (
+        profile.subscription_change_pending_date.isoformat()
+        if profile and profile.subscription_change_pending_date
+        else None
+    )
+    period_start: str | None = (
+        profile.subscription_period_start.isoformat() if profile and profile.subscription_period_start else None
+    )
+
+    return {
+        "multi_user_mode": True,
+        "owner_id": owner_id,
+        "tier": tier,
+        "usage": usage,
+        "period_start": period_start,
+        "pending_change": (
+            {
+                "tier_id": pending_tier_id,
+                "tier": get_tier(pending_tier_id, db),
+                "effective_date": pending_date,
+            }
+            if pending_tier_id
+            else None
+        ),
+    }
+
+
+@router.post("/change", summary="Request a subscription plan change", status_code=status.HTTP_200_OK)
+def change_subscription(request: Request, body: SubscriptionChangeRequest, db: DbSession) -> dict[str, Any]:
+    """Request a subscription tier change.
+
+    **Upgrades** (moving to a higher-ranked plan) take effect immediately.
+
+    **Downgrades** (moving to a lower-ranked plan) are scheduled for the end
+    of the current billing period to prevent gaming.  The user keeps their
+    current plan benefits until the scheduled date.
+
+    Requesting the currently active tier while a downgrade is pending cancels
+    that pending change.
+    """
+    from app.config import settings
+
+    if not settings.multi_user_enabled:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Subscription management is not available in single-user mode.",
+        )
+
+    owner_id = _require_authenticated(request)
+
+    try:
+        result = request_subscription_change(db, owner_id, body.plan_id, body.billing_cycle)
+    except SubscriptionChangeError as exc:
+        raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(exc)) from exc
+
+    return result
+
+
+@router.delete("/change", summary="Cancel a pending subscription change", status_code=status.HTTP_200_OK)
+def cancel_subscription_change(request: Request, db: DbSession) -> dict[str, Any]:
+    """Cancel a scheduled future subscription change.
+
+    Only downgrades can be pending; upgrades always take effect immediately.
+    Returns 404 when there is no pending change to cancel.
+    """
+    from app.config import settings
+
+    if not settings.multi_user_enabled:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail="Subscription management is not available in single-user mode.",
+        )
+
+    owner_id = _require_authenticated(request)
+
+    cancelled = cancel_pending_subscription_change(db, owner_id)
+    if not cancelled:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="No pending subscription change found.")
+
+    return {"cancelled": True, "message": "Your pending subscription change has been cancelled."}
+
+
+@router.get("/platform", summary="Platform-wide usage statistics (admin only)")
+def platform_stats(request: Request, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Return aggregate statistics across all users and tiers (admin only)."""
+    from app.models import FileRecord, UserProfile
+
+    today = datetime.now(timezone.utc).date()
+
+    # Total files
+    total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0
+
+    # Files today
+    files_today: int = (
+        db.query(func.count(FileRecord.id)).filter(func.date(FileRecord.created_at) == today).scalar() or 0
+    )
+
+    # Files this month
+    files_this_month: int = (
+        db.query(func.count(FileRecord.id))
+        .filter(func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"))
+        .scalar()
+        or 0
+    )
+
+    # Files with OCR text (proxy for pages OCRed — approximation)
+    files_with_ocr: int = db.query(func.count(FileRecord.id)).filter(FileRecord.ocr_text.isnot(None)).scalar() or 0
+
+    # Unique active users (ever uploaded)
+    unique_users: int = (
+        db.query(func.count(func.distinct(FileRecord.owner_id))).filter(FileRecord.owner_id.isnot(None)).scalar() or 0
+    )
+
+    # Users per subscription tier
+    profiles = (
+        db.query(UserProfile.subscription_tier, func.count(UserProfile.id))
+        .group_by(UserProfile.subscription_tier)
+        .all()
+    )
+    tier_distribution: dict[str, int] = {row[0] or "free": row[1] for row in profiles}
+
+    # Fill in zeros for tiers with no users
+    for tid in TIER_ORDER:
+        tier_distribution.setdefault(tid, 0)
+
+    return {
+        "files": {
+            "total": total_files,
+            "today": files_today,
+            "this_month": files_this_month,
+            "with_ocr": files_with_ocr,
+        },
+        "users": {
+            "unique_uploaders": unique_users,
+            "tier_distribution": tier_distribution,
+        },
+        "generated_at": datetime.now(timezone.utc).isoformat(),
+    }

app/api/user.py

@@ -4,14 +4,23 @@ User-related API endpoints
 
 import logging
 from hashlib import md5
+from typing import Annotated
 
-from fastapi import APIRouter, HTTPException, Request
+from fastapi import APIRouter, Depends, HTTPException, Query, Request
+from sqlalchemy import func
+from sqlalchemy.orm import Session
+
+from app.auth import require_login
+from app.database import get_db
+from app.models import FileRecord
 
 # Set up logging
 logger = logging.getLogger(__name__)
 
 router = APIRouter()
 
+DbSession = Annotated[Session, Depends(get_db)]
+
 
 async def whoami_handler(request: Request):
     """
@@ -46,3 +55,30 @@ async def whoami(request: Request):
 @router.get("/auth/whoami")
 async def auth_whoami(request: Request):
     return await whoami_handler(request)
+
+
+@router.get("/users/search")
+@require_login
+def search_known_users(
+    db: DbSession,
+    q: str = Query("", description="Substring to match against known owner IDs"),
+    limit: int = Query(5, ge=1, le=20, description="Maximum number of results"),
+):
+    """
+    Search known user identifiers (owner_ids) from existing documents.
+
+    Returns distinct ``owner_id`` values from the files table that contain
+    the query string as a case-insensitive substring.  Results are limited
+    to at most ``limit`` entries (default 5).
+
+    This powers the autocomplete widget on the settings page for the
+    ``default_owner_id`` field.
+    """
+    base_query = db.query(FileRecord.owner_id).filter(FileRecord.owner_id.isnot(None)).distinct()
+
+    if q.strip():
+        base_query = base_query.filter(func.lower(FileRecord.owner_id).contains(q.strip().lower()))
+
+    results = base_query.order_by(FileRecord.owner_id).limit(limit).all()
+
+    return {"users": [row[0] for row in results]}

app/api/webhooks.py

@@ -0,0 +1,206 @@
+"""API endpoints for managing webhook configurations.
+
+Provides CRUD operations for webhook configs that notify external systems
+when document events occur (``document.uploaded``, ``document.processed``,
+``document.failed``).
+"""
+
+import json
+import logging
+from typing import Annotated, Any
+
+from fastapi import APIRouter, Depends, HTTPException, Request, status
+from pydantic import BaseModel, Field
+from sqlalchemy.orm import Session
+
+from app.database import get_db
+from app.models import WebhookConfig
+from app.utils.webhook import VALID_EVENTS
+
+logger = logging.getLogger(__name__)
+router = APIRouter(prefix="/webhooks", tags=["webhooks"])
+
+DbSession = Annotated[Session, Depends(get_db)]
+
+
+# ---------------------------------------------------------------------------
+# Auth helper (reuse the pattern from settings API)
+# ---------------------------------------------------------------------------
+
+
+def _require_admin(request: Request) -> dict:
+    """Ensure the caller is an admin. Raises 403 otherwise."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
+    return user
+
+
+AdminUser = Annotated[dict, Depends(_require_admin)]
+
+
+# ---------------------------------------------------------------------------
+# Pydantic schemas
+# ---------------------------------------------------------------------------
+
+
+class WebhookCreate(BaseModel):
+    """Schema for creating a new webhook configuration."""
+
+    url: str = Field(..., min_length=1, max_length=2048, description="Target URL for webhook delivery")
+    secret: str | None = Field(default=None, max_length=512, description="Shared secret for HMAC-SHA256 signatures")
+    events: list[str] = Field(..., min_length=1, description="List of events to subscribe to")
+    is_active: bool = Field(default=True, description="Whether the webhook is active")
+    description: str | None = Field(default=None, max_length=500, description="Optional human-readable description")
+
+
+class WebhookUpdate(BaseModel):
+    """Schema for updating an existing webhook configuration."""
+
+    url: str | None = Field(default=None, min_length=1, max_length=2048)
+    secret: str | None = Field(default=None, max_length=512)
+    events: list[str] | None = Field(default=None, min_length=1)
+    is_active: bool | None = None
+    description: str | None = Field(default=None, max_length=500)
+
+
+class WebhookResponse(BaseModel):
+    """Schema returned to clients (secret is never exposed)."""
+
+    id: int
+    url: str
+    events: list[str]
+    is_active: bool
+    description: str | None
+    has_secret: bool
+
+    model_config = {"from_attributes": True}
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+
+def _validate_events(events: list[str]) -> None:
+    """Raise 422 if any event name is not recognised."""
+    invalid = set(events) - VALID_EVENTS
+    if invalid:
+        raise HTTPException(
+            status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
+            detail=f"Invalid event(s): {', '.join(sorted(invalid))}. Valid events: {', '.join(sorted(VALID_EVENTS))}",
+        )
+
+
+def _to_response(cfg: WebhookConfig) -> dict[str, Any]:
+    """Convert a DB model instance to a response dict."""
+    try:
+        events = json.loads(cfg.events)
+    except (json.JSONDecodeError, TypeError):
+        events = []
+    return {
+        "id": cfg.id,
+        "url": cfg.url,
+        "events": events,
+        "is_active": cfg.is_active,
+        "description": cfg.description,
+        "has_secret": cfg.secret is not None and len(cfg.secret) > 0,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Endpoints
+# ---------------------------------------------------------------------------
+
+
+@router.get("/", summary="List all webhook configurations")
+def list_webhooks(db: DbSession, _admin: AdminUser) -> list[dict[str, Any]]:
+    """Return all webhook configurations. Secrets are never included."""
+    configs = db.query(WebhookConfig).order_by(WebhookConfig.id).all()
+    return [_to_response(c) for c in configs]
+
+
+@router.get("/{webhook_id}", summary="Get a single webhook configuration")
+def get_webhook(webhook_id: int, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Return a single webhook configuration by ID."""
+    cfg = db.query(WebhookConfig).filter(WebhookConfig.id == webhook_id).first()
+    if not cfg:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Webhook not found")
+    return _to_response(cfg)
+
+
+@router.post("/", status_code=status.HTTP_201_CREATED, summary="Create a webhook configuration")
+def create_webhook(body: WebhookCreate, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Create a new webhook configuration."""
+    _validate_events(body.events)
+
+    cfg = WebhookConfig(
+        url=body.url,
+        secret=body.secret,
+        events=json.dumps(sorted(body.events)),
+        is_active=body.is_active,
+        description=body.description,
+    )
+    try:
+        db.add(cfg)
+        db.commit()
+        db.refresh(cfg)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Webhook %d created for events %s", cfg.id, body.events)
+    return _to_response(cfg)
+
+
+@router.put("/{webhook_id}", summary="Update a webhook configuration")
+def update_webhook(webhook_id: int, body: WebhookUpdate, db: DbSession, _admin: AdminUser) -> dict[str, Any]:
+    """Update an existing webhook configuration. Only supplied fields are changed."""
+    cfg = db.query(WebhookConfig).filter(WebhookConfig.id == webhook_id).first()
+    if not cfg:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Webhook not found")
+
+    if body.url is not None:
+        cfg.url = body.url
+    if body.secret is not None:
+        cfg.secret = body.secret
+    if body.events is not None:
+        _validate_events(body.events)
+        cfg.events = json.dumps(sorted(body.events))
+    if body.is_active is not None:
+        cfg.is_active = body.is_active
+    if body.description is not None:
+        cfg.description = body.description
+
+    try:
+        db.commit()
+        db.refresh(cfg)
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Webhook %d updated", cfg.id)
+    return _to_response(cfg)
+
+
+@router.delete("/{webhook_id}", status_code=status.HTTP_204_NO_CONTENT, summary="Delete a webhook configuration")
+def delete_webhook(webhook_id: int, db: DbSession, _admin: AdminUser) -> None:
+    """Delete a webhook configuration."""
+    cfg = db.query(WebhookConfig).filter(WebhookConfig.id == webhook_id).first()
+    if not cfg:
+        raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Webhook not found")
+
+    try:
+        db.delete(cfg)
+        db.commit()
+    except Exception:
+        db.rollback()
+        raise
+
+    logger.info("Webhook %d deleted", cfg.id)
+
+
+@router.get("/events/", summary="List valid webhook event types")
+def list_events(_admin: AdminUser) -> list[str]:
+    """Return the list of valid event types that can be subscribed to."""
+    return sorted(VALID_EVENTS)

app/auth.py

@@ -3,18 +3,30 @@ import inspect
 import logging
 import pathlib
 from functools import wraps
+from urllib.parse import urlparse
 
 from authlib.integrations.starlette_client import OAuth
-from fastapi import APIRouter, Request, status
+from fastapi import APIRouter, Depends, Request, status
+from fastapi.responses import JSONResponse
 from fastapi.templating import Jinja2Templates
+from sqlalchemy.orm import Session
 from starlette.responses import RedirectResponse
 
 from app.config import settings
+from app.database import get_db
 
-oauth = OAuth()
+# Conditional imports: only used when multi_user_enabled=True. Imported here at
+# module level (not inside auth()) so they don't incur repeated import overhead.
+# Guards at call-sites ensure they are never *called* in single-user mode.
+from app.models import LocalUser as _LocalUser
+from app.models import UserProfile as _UserProfile
+from app.utils.local_auth import build_session_user as _build_session_user
+from app.utils.local_auth import verify_password as _verify_password
 
 logger = logging.getLogger(__name__)
 
+oauth = OAuth()
+
 AUTH_ENABLED = settings.auth_enabled
 
 # Set up templates for authentication
@@ -43,6 +55,27 @@ def get_current_user(request: Request):
     return request.session.get("user")
 
 
+def get_current_user_id(request: Request) -> str:
+    """Return a stable string identifier for the authenticated user.
+
+    Falls back to ``"anonymous"`` when no user is in the session (e.g. when
+    AUTH_ENABLED=False in single-user mode).  The returned value is consistent
+    with how pipeline and file ownership is stored in the database.
+
+    Priority order: ``preferred_username`` → ``email`` → ``id`` → ``"anonymous"``.
+
+    Args:
+        request: The current FastAPI request.
+
+    Returns:
+        A non-empty string identifying the current user.
+    """
+    user = get_current_user(request)
+    if not user or not isinstance(user, dict):
+        return "anonymous"
+    return user.get("preferred_username") or user.get("email") or user.get("id") or "anonymous"
+
+
 def require_login(func):
     if not AUTH_ENABLED:
         return func  # no-op
@@ -50,13 +83,27 @@ def require_login(func):
     @wraps(func)
     async def wrapper(request: Request, *args, **kwargs):
         if not request.session.get("user"):
+            # For API endpoints return 401 instead of storing the URL in the session
+            # and redirecting to /login.  Without this guard, the /api/auth/whoami
+            # probe issued by common.js on every page load would overwrite
+            # redirect_after_login with the API URL, causing the post-login redirect
+            # to land on a JSON endpoint rather than the original page.
+            url_path = urlparse(str(request.url)).path
+            if url_path.startswith("/api/"):
+                return JSONResponse(
+                    status_code=status.HTTP_401_UNAUTHORIZED,
+                    content={"error": "Not authenticated"},
+                )
             request.session["redirect_after_login"] = str(request.url)
             return RedirectResponse(url="/login", status_code=status.HTTP_302_FOUND)
-        # Check if the wrapped function is a coroutine function
+        # Pass request as a keyword argument so that endpoints whose first
+        # parameter is a path variable (e.g. pipeline_id) are not accidentally
+        # bound to the request object when FastAPI supplies all arguments as
+        # keyword arguments.
         if inspect.iscoroutinefunction(func):
-            return await func(request, *args, **kwargs)
+            return await func(*args, request=request, **kwargs)
         else:
-            return func(request, *args, **kwargs)
+            return func(*args, request=request, **kwargs)
 
     return wrapper
 
@@ -70,7 +117,7 @@ def get_gravatar_url(email):
 
 
 async def login(request: Request):
-    """Show login page with appropriate authentication options"""
+    """Show login page with appropriate authentication options."""
     return templates.TemplateResponse(
         "login.html",
         {
@@ -79,8 +126,10 @@ async def login(request: Request):
             "message": request.query_params.get("message"),
             "show_oauth": OAUTH_CONFIGURED,
             "oauth_provider_name": OAUTH_PROVIDER_NAME,
-            "app_version": settings.version,  # Changed from app_version to version
+            "app_version": settings.version,
             "csrf_token": getattr(request.state, "csrf_token", ""),
+            # "Create account" link is only shown when multi-user mode AND local signup are both enabled
+            "allow_signup": settings.multi_user_enabled and settings.allow_local_signup,
         },
     )
 
@@ -94,7 +143,103 @@ async def oauth_login(request: Request):
     return await oauth.authentik.authorize_redirect(request, redirect_uri)
 
 
-async def oauth_callback(request: Request):
+def _ensure_user_profile(db: Session, user_data: dict, is_admin: bool = False) -> None:
+    """Create or update a UserProfile row for *user_data*.
+
+    Uses the same identifier priority as ``get_current_owner_id`` (sub →
+    preferred_username → email → id) so that the profile's ``user_id`` matches
+    ``FileRecord.owner_id`` for every document the user uploads.
+
+    For regular users, an existing profile is left unchanged so that
+    admin-managed settings (tier, limits, etc.) are preserved across logins.
+
+    For admin users (*is_admin=True*) the following rules apply:
+    - If no profile exists: one is created with the highest subscription tier,
+      ``is_complimentary=True``, and ``onboarding_completed=True`` so that
+      admins skip the first-time setup wizard.
+    - If a profile already exists: ``is_complimentary`` is set to ``True``
+      and, when the current tier is ``"free"``, the tier is upgraded to the
+      highest available plan.  Other admin-managed settings are left intact.
+
+    Args:
+        db: Active database session.
+        user_data: Mapping of user attributes as returned by the OAuth provider
+            or built by :func:`app.utils.local_auth.build_session_user`.
+        is_admin: When ``True``, apply admin-specific defaults on first login
+            and ensure the complimentary flag is always set.
+    """
+    from app.models import UserProfile
+    from app.utils.subscription import TIER_ORDER
+
+    highest_tier = TIER_ORDER[-1]
+
+    user_id = (
+        user_data.get("sub") or user_data.get("preferred_username") or user_data.get("email") or user_data.get("id")
+    )
+    if not user_id:
+        logger.warning("Cannot create UserProfile: no stable user identifier in OAuth userinfo")
+        return
+
+    try:
+        existing = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+        if existing is None:
+            display_name = user_data.get("name") or user_data.get("preferred_username") or user_data.get("email")
+            email = user_data.get("email")
+            profile = UserProfile(
+                user_id=user_id,
+                display_name=display_name,
+                subscription_tier=highest_tier if is_admin else "free",
+                is_complimentary=is_admin,
+                onboarding_completed=is_admin,
+            )
+            db.add(profile)
+            db.commit()
+            logger.info(
+                "Auto-created UserProfile for user_id=%s (admin=%s, tier=%s)",
+                user_id,
+                is_admin,
+                highest_tier if is_admin else "free",
+            )
+            # Notify admins and fire webhook for new (non-admin) user signup
+            if not is_admin:
+                try:
+                    from app.utils.notification import notify_user_signup
+                    from app.utils.webhook import dispatch_webhook_event
+
+                    notify_user_signup(user_id, display_name=display_name, email=email)
+                    dispatch_webhook_event(
+                        "user.signup",
+                        {
+                            "user_id": user_id,
+                            "display_name": display_name,
+                            "email": email,
+                        },
+                    )
+                except Exception:
+                    logger.exception("Failed to send signup notification/webhook for user_id=%s", user_id)
+        elif is_admin:
+            # Ensure existing admin profiles always have complimentary flag set.
+            # Also upgrade from free tier to highest if still on default.
+            changed = False
+            if not existing.is_complimentary:
+                existing.is_complimentary = True
+                changed = True
+            if (existing.subscription_tier or "free") == "free":
+                existing.subscription_tier = highest_tier
+                changed = True
+            if changed:
+                db.commit()
+                logger.info(
+                    "Updated admin UserProfile for user_id=%s (complimentary=True, tier=%s)",
+                    user_id,
+                    existing.subscription_tier,
+                )
+    except Exception:
+        db.rollback()
+        logger.exception("Failed to auto-create/update UserProfile for user_id=%s", user_id)
+
+
+async def oauth_callback(request: Request, db: Session = Depends(get_db)):
     """Handle OAuth callback from provider"""
     try:
         token = await oauth.authentik.authorize_access_token(request)
@@ -126,8 +271,22 @@ async def oauth_callback(request: Request):
 
         request.session["user"] = user_data
 
+        # Auto-create or update UserProfile so the user appears in admin user management
+        _ensure_user_profile(db, user_data, is_admin=is_admin)
+
         # Log the successful authentication
-        logger.info(f"[SECURITY] OAUTH_LOGIN_SUCCESS user={user_data.get('email', 'unknown')} admin={is_admin}")
+        logger.info("[SECURITY] OAUTH_LOGIN_SUCCESS user=%s admin=%s", user_data.get("email", "unknown"), is_admin)
+
+        # Redirect first-time users to onboarding
+        user_id = (
+            user_data.get("sub") or user_data.get("preferred_username") or user_data.get("email") or user_data.get("id")
+        )
+        if user_id:
+            profile = db.query(_UserProfile).filter(_UserProfile.user_id == user_id).first()
+            if profile and not profile.onboarding_completed:
+                post_onboarding = request.session.pop("redirect_after_login", "/upload")
+                request.session["post_onboarding_redirect"] = post_onboarding
+                return RedirectResponse(url="/onboarding", status_code=status.HTTP_302_FOUND)
 
         # Redirect to original destination or default
         redirect_url = request.session.pop("redirect_after_login", "/upload")
@@ -137,15 +296,62 @@ async def oauth_callback(request: Request):
         return RedirectResponse(url=f"/login?error=Authentication+failed:+{str(e)}", status_code=status.HTTP_302_FOUND)
 
 
-async def auth(request: Request):
-    """Handle local username/password authentication"""
+async def auth(request: Request, db: Session = Depends(get_db)):
+    """Handle local username/password authentication.
+
+    In multi-user mode (``MULTI_USER_ENABLED=True``) local registered users are
+    checked first; if no matching LocalUser is found the request falls through to
+    the single admin-credential check so that single-user deployments continue to
+    work without any database involvement.
+
+    In single-user mode (``MULTI_USER_ENABLED=False``, the default) the LocalUser
+    table is never queried — only the configured ADMIN_USERNAME / ADMIN_PASSWORD
+    are accepted, preserving full backward compatibility.
+    """
     form_data = await request.form()
     username = form_data.get("username")
     password = form_data.get("password")
 
-    if username == settings.admin_username and password == settings.admin_password:
-        # Create user session
-        request.session["user"] = {
+    # --- LocalUser check (multi-user mode only) ---
+    if settings.multi_user_enabled:
+        local_user = (
+            db.query(_LocalUser).filter((_LocalUser.username == username) | (_LocalUser.email == username)).first()
+        )
+        if local_user is not None:
+            if not local_user.is_active:
+                logger.warning("[SECURITY] LOCAL_LOGIN_UNVERIFIED user=%s", username)
+                return RedirectResponse(
+                    url="/login?error=Please+verify+your+email+address+before+logging+in",
+                    status_code=302,
+                )
+            if not _verify_password(password or "", local_user.hashed_password):
+                logger.warning("[SECURITY] LOCAL_LOGIN_FAILURE user=%s", username)
+                return RedirectResponse(url="/login?error=Invalid+username+or+password", status_code=302)
+            user_data = _build_session_user(local_user)
+            request.session["user"] = user_data
+            logger.info("[SECURITY] LOCAL_LOGIN_SUCCESS user=%s", local_user.email)
+            _ensure_user_profile(db, user_data, is_admin=bool(local_user.is_admin))
+            profile = db.query(_UserProfile).filter(_UserProfile.user_id == local_user.email).first()
+            if profile and not profile.onboarding_completed:
+                post_onboarding = request.session.pop("redirect_after_login", "/upload")
+                request.session["post_onboarding_redirect"] = post_onboarding
+                return RedirectResponse(url="/onboarding", status_code=302)
+            redirect_url = request.session.pop("redirect_after_login", "/upload")
+            return RedirectResponse(url=redirect_url, status_code=302)
+
+    # --- Admin credentials (always available as a fallback / single-user mode) ---
+    # Guard: only attempt the match when credentials are actually configured.
+    # Without this guard, Python's `None == None` would be True when neither
+    # ADMIN_USERNAME nor ADMIN_PASSWORD is set, allowing any request that omits
+    # those form fields to be authenticated as an admin — creating a phantom
+    # "None@local.docuelevate" admin profile with full privileges.
+    if (
+        settings.admin_username
+        and settings.admin_password
+        and username == settings.admin_username
+        and password == settings.admin_password
+    ):
+        admin_user_data = {
             "id": "admin",
             "name": "Administrator",
             "email": f"{username}@local.docuelevate",
@@ -153,12 +359,13 @@ async def auth(request: Request):
             "picture": "/static/images/default-avatar.svg",
             "is_admin": True,
         }
-        logger.info(f"[SECURITY] LOCAL_LOGIN_SUCCESS user={username}")
-        # Redirect to original destination or default
+        request.session["user"] = admin_user_data
+        logger.info("[SECURITY] LOCAL_LOGIN_SUCCESS user=%s", username)
+        _ensure_user_profile(db, admin_user_data, is_admin=True)
         redirect_url = request.session.pop("redirect_after_login", "/upload")
         return RedirectResponse(url=redirect_url, status_code=302)
     else:
-        logger.warning(f"[SECURITY] LOCAL_LOGIN_FAILURE user={username}")
+        logger.warning("[SECURITY] LOCAL_LOGIN_FAILURE user=%s", username)
         return RedirectResponse(url="/login?error=Invalid+username+or+password", status_code=302)
 
 

app/celery_worker.py

@@ -8,19 +8,25 @@ from app import tasks  # noqa: F401 - Imports app/tasks.py so Celery can registe
 # Import the shared Celery instance
 from app.celery_app import celery
 from app.config import settings
+from app.tasks.backup_tasks import cleanup_old_backups, create_backup  # noqa: F401
 from app.tasks.check_credentials import check_credentials
+from app.tasks.compute_embedding import backfill_missing_embeddings, compute_document_embedding  # noqa: F401
 from app.tasks.convert_to_pdf import convert_to_pdf  # noqa: F401
+from app.tasks.convert_to_pdfa import convert_to_pdfa  # noqa: F401
 from app.tasks.embed_metadata_into_pdf import embed_metadata_into_pdf  # noqa: F401
 from app.tasks.extract_metadata_with_gpt import extract_metadata_with_gpt  # noqa: F401
+from app.tasks.finalize_document_storage import finalize_document_storage  # noqa: F401
 from app.tasks.imap_tasks import pull_all_inboxes  # noqa: F401
 from app.tasks.monitor_stalled_steps import monitor_stalled_steps  # noqa: F401
 
 # **Ensure all tasks are imported before Celery starts**
 from app.tasks.process_document import process_document  # noqa: F401
 from app.tasks.process_with_azure_document_intelligence import process_with_azure_document_intelligence  # noqa: F401
+from app.tasks.process_with_ocr import process_with_ocr  # noqa: F401
 from app.tasks.refine_text_with_gpt import refine_text_with_gpt  # noqa: F401
 from app.tasks.rotate_pdf_pages import rotate_pdf_pages  # noqa: F401
 from app.tasks.send_to_all import send_to_all_destinations  # noqa: F401
+from app.tasks.subscription_tasks import apply_pending_subscription_changes_all  # noqa: F401
 
 # Import new send tasks
 from app.tasks.upload_to_dropbox import upload_to_dropbox  # noqa: F401
@@ -33,7 +39,10 @@ from app.tasks.upload_to_paperless import upload_to_paperless  # noqa: F401
 from app.tasks.upload_to_s3 import upload_to_s3  # noqa: F401
 from app.tasks.upload_to_sftp import upload_to_sftp  # noqa: F401
 from app.tasks.upload_to_webdav import upload_to_webdav  # noqa: F401
+from app.tasks.upload_with_rclone import send_to_all_rclone_destinations, upload_with_rclone  # noqa: F401
 from app.tasks.uptime_kuma_tasks import ping_uptime_kuma  # noqa: F401
+from app.tasks.watch_folder_tasks import scan_all_watch_folders  # noqa: F401
+from app.tasks.webhook_tasks import deliver_webhook_task  # noqa: F401
 
 # Register the settings reload signal handler so workers pick up config changes
 from app.utils.settings_sync import register_settings_reload_signal
@@ -91,6 +100,74 @@ celery.conf.beat_schedule = {
         "schedule": crontab(minute="*/1"),  # Every minute
         "options": {"expires": 55},  # Must complete within 55 seconds
     },
+    # Watch folder scanning — polls local paths, FTP, SFTP, and cloud ingest folders.
+    # Schedule is controlled by WATCH_FOLDER_POLL_INTERVAL (default: 1 minute).
+    "scan-watch-folders": (
+        {
+            "task": "app.tasks.watch_folder_tasks.scan_all_watch_folders",
+            "schedule": crontab(minute=f"*/{max(1, settings.watch_folder_poll_interval)}"),
+            "options": {"expires": 55},
+        }
+        if (
+            settings.watch_folders
+            or settings.ftp_ingest_enabled
+            or settings.sftp_ingest_enabled
+            or settings.dropbox_ingest_enabled
+            or settings.google_drive_ingest_enabled
+            or settings.onedrive_ingest_enabled
+            or settings.nextcloud_ingest_enabled
+            or settings.s3_ingest_enabled
+            or settings.webdav_ingest_enabled
+        )
+        else None
+    ),
+    # Backfill embeddings for files that were processed before the
+    # embedding pipeline was enabled, or where the embedding task failed.
+    "backfill-missing-embeddings": {
+        "task": "backfill_missing_embeddings",
+        "schedule": crontab(minute="*/5"),  # Every 5 minutes
+        "options": {"expires": 240},  # 4 minutes expiry
+    },
+    # Apply scheduled subscription downgrades daily at 00:05 UTC
+    "apply-pending-subscription-changes": {
+        "task": "app.tasks.subscription_tasks.apply_pending_subscription_changes_all",
+        "schedule": crontab(hour="0", minute="5"),  # 00:05 UTC daily
+        "options": {"expires": 3600},
+    },
+    # ── Database backup tasks ──────────────────────────────────────────────
+    # Hourly backup (kept for 4 days)
+    "backup-hourly": (
+        {
+            "task": "app.tasks.backup_tasks.create_backup",
+            "schedule": crontab(minute="0"),  # top of every hour
+            "kwargs": {"backup_type": "hourly"},
+            "options": {"expires": 3300},
+        }
+        if settings.backup_enabled
+        else None
+    ),
+    # Daily backup (kept for 3 weeks) – runs at 02:30 UTC
+    "backup-daily": (
+        {
+            "task": "app.tasks.backup_tasks.create_backup",
+            "schedule": crontab(hour="2", minute="30"),
+            "kwargs": {"backup_type": "daily"},
+            "options": {"expires": 3600},
+        }
+        if settings.backup_enabled
+        else None
+    ),
+    # Weekly backup (kept for 13 weeks) – runs every Sunday at 03:00 UTC
+    "backup-weekly": (
+        {
+            "task": "app.tasks.backup_tasks.create_backup",
+            "schedule": crontab(hour="3", minute="0", day_of_week="0"),
+            "kwargs": {"backup_type": "weekly"},
+            "options": {"expires": 3600},
+        }
+        if settings.backup_enabled
+        else None
+    ),
 }
 
 # Remove None entries from beat_schedule

app/config.py

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 
+import json
 import os
 from typing import Any, List, Optional, Union
 
@@ -115,12 +116,265 @@ class Settings(BaseSettings):
     session_secret: Optional[str] = None
     admin_group_name: str = "admin"
 
+    # Multi-user settings
+    multi_user_enabled: bool = Field(
+        default=False,
+        description=(
+            "Enable multi-user mode with individual document spaces per user. "
+            "When enabled, each authenticated user sees only their own documents, "
+            "uploads, and search results. Shared settings (AI, OCR) remain global. "
+            "Requires auth_enabled=True. Default: False (single-user/shared mode)."
+        ),
+    )
+    default_daily_upload_limit: int = Field(
+        default=0,
+        description=(
+            "Default maximum number of document uploads allowed per user per day "
+            "in multi-user mode. Set to 0 for unlimited. "
+            "Individual user limits can override this default. Default: 0 (unlimited)."
+        ),
+    )
+    unowned_docs_visible_to_all: bool = Field(
+        default=True,
+        description=(
+            "In multi-user mode, controls whether documents without an owner (owner_id is NULL) "
+            "are visible to all authenticated users. When True, unowned documents appear in every "
+            "user's file list alongside their own files. When False, only admins can see unowned "
+            "documents. Default: True."
+        ),
+    )
+    default_owner_id: Optional[str] = Field(
+        default=None,
+        description=(
+            "When set, automatically assigns this owner ID to newly ingested documents that would "
+            "otherwise have no owner (e.g. documents from IMAP, API without session, or legacy imports). "
+            "Use the admin /api/files/assign-owner endpoint to bulk-assign existing unclaimed documents. "
+            "Default: None (documents remain unowned until claimed)."
+        ),
+    )
+
+    subscription_overage_percent: int = Field(
+        default=20,
+        ge=0,
+        le=200,
+        description=(
+            "Soft-limit overage buffer in percent (0–200). The announced monthly quota is "
+            "increased by this percentage for actual enforcement. E.g. 20 means a 150-doc/month "
+            "plan enforces at 180 docs (150 × 1.20). Set 0 to enforce exactly at the announced "
+            "limit. Per-plan overage_percent (set in Plan Designer) overrides this global default. "
+            "Default: 20."
+        ),
+    )
+
     # Authentik
     authentik_client_id: Optional[str] = None
     authentik_client_secret: Optional[str] = None
     authentik_config_url: Optional[str] = None
     oauth_provider_name: Optional[str] = None  # Name to display for the OAuth provider
 
+    # Local user signup
+    allow_local_signup: bool = Field(
+        default=False,
+        description=(
+            "Allow users to self-register with email and password. "
+            "Has no effect unless MULTI_USER_ENABLED is also True. "
+            "Requires SMTP to be configured so verification emails can be sent. "
+            "Default: False (registration disabled — admin creates users manually)."
+        ),
+    )
+
+    # Stripe billing
+    stripe_secret_key: Optional[str] = None
+    stripe_publishable_key: Optional[str] = None
+    stripe_webhook_secret: Optional[str] = None
+    stripe_success_url: Optional[str] = None  # e.g. https://app.example.com/billing/success
+    stripe_cancel_url: Optional[str] = None  # e.g. https://app.example.com/pricing
+
+    # ---------------------------------------------------------------------------
+    # Watch Folder Ingestion
+    # ---------------------------------------------------------------------------
+    # Local filesystem watch folders (comma-separated list of absolute paths).
+    # DocuElevate will poll each path for new files and automatically ingest them.
+    # Works with any mounted path, including SMB/CIFS (via system mount), NFS, etc.
+    # Example: /watchfolders/scanner,/mnt/shared/inbox
+    watch_folders: Optional[str] = Field(
+        default=None,
+        description=(
+            "Comma-separated list of local filesystem paths (absolute) that DocuElevate will "
+            "poll for new files to ingest. Each file found is enqueued for document processing. "
+            "Works with any mounted path including SMB/CIFS (mounted via system) and NFS. "
+            "Example: /watchfolders/scanner,/mnt/shared/inbox"
+        ),
+    )
+    watch_folder_poll_interval: int = Field(
+        default=1,
+        description=("Poll interval in minutes for local watch folder scanning. Default: 1 minute."),
+    )
+    watch_folder_delete_after_process: bool = Field(
+        default=False,
+        description=(
+            "Delete files from local watch folders after they have been successfully enqueued "
+            "for processing. When False (default), files are left in place and tracked via a "
+            "cache file to avoid re-ingesting them."
+        ),
+    )
+
+    # FTP Ingest / Watch Folder
+    # Uses the existing FTP credentials (ftp_host, ftp_username, ftp_password) to poll
+    # a source folder on the FTP server for new files to ingest.
+    ftp_ingest_folder: Optional[str] = Field(
+        default=None,
+        description=(
+            "FTP folder path to monitor for new files to ingest. "
+            "Uses the existing FTP connection settings (FTP_HOST, FTP_USERNAME, FTP_PASSWORD). "
+            "When set, DocuElevate will periodically poll this folder and download new files for processing."
+        ),
+    )
+    ftp_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable FTP watch folder ingestion. Requires FTP_INGEST_FOLDER and FTP connection settings.",
+    )
+    ftp_ingest_delete_after_process: bool = Field(
+        default=False,
+        description=(
+            "Delete files from the FTP ingest folder after they have been successfully downloaded "
+            "and enqueued for processing. Default: False (files are left in place)."
+        ),
+    )
+
+    # SFTP Ingest / Watch Folder
+    # Uses the existing SFTP credentials (sftp_host, sftp_username, sftp_password/sftp_private_key)
+    # to poll a source folder on the SFTP server for new files to ingest.
+    sftp_ingest_folder: Optional[str] = Field(
+        default=None,
+        description=(
+            "SFTP folder path to monitor for new files to ingest. "
+            "Uses the existing SFTP connection settings (SFTP_HOST, SFTP_USERNAME, SFTP_PASSWORD/SFTP_PRIVATE_KEY). "
+            "When set, DocuElevate will periodically poll this folder and download new files for processing."
+        ),
+    )
+    sftp_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable SFTP watch folder ingestion. Requires SFTP_INGEST_FOLDER and SFTP connection settings.",
+    )
+    sftp_ingest_delete_after_process: bool = Field(
+        default=False,
+        description=(
+            "Delete files from the SFTP ingest folder after they have been successfully downloaded "
+            "and enqueued for processing. Default: False (files are left in place)."
+        ),
+    )
+
+    # ---------------------------------------------------------------------------
+    # Cloud Provider Watch Folders
+    # ---------------------------------------------------------------------------
+    # Each cloud provider has three settings:
+    #   <provider>_ingest_enabled  — enable the watch-folder for this provider
+    #   <provider>_ingest_folder   — the remote path / folder ID to poll
+    #   <provider>_ingest_delete_after_process — delete from cloud after download
+
+    # Dropbox ingest — reuses existing Dropbox OAuth credentials
+    dropbox_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable Dropbox watch folder ingestion. Requires Dropbox OAuth credentials.",
+    )
+    dropbox_ingest_folder: Optional[str] = Field(
+        default=None,
+        description=(
+            "Dropbox folder path to poll for new files to ingest (e.g. /Inbox/Scanner). "
+            "Uses the existing Dropbox OAuth credentials."
+        ),
+    )
+    dropbox_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete files from Dropbox ingest folder after download and enqueue.",
+    )
+
+    # Google Drive ingest — reuses existing Google Drive credentials
+    google_drive_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable Google Drive watch folder ingestion. Requires Google Drive credentials.",
+    )
+    google_drive_ingest_folder_id: Optional[str] = Field(
+        default=None,
+        description=(
+            "Google Drive folder ID to poll for new files to ingest. "
+            "Uses the existing Google Drive service-account or OAuth credentials."
+        ),
+    )
+    google_drive_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete files from Google Drive ingest folder after download and enqueue.",
+    )
+
+    # OneDrive ingest — reuses existing OneDrive MSAL credentials
+    onedrive_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable OneDrive watch folder ingestion. Requires OneDrive MSAL credentials.",
+    )
+    onedrive_ingest_folder_path: Optional[str] = Field(
+        default=None,
+        description=(
+            "OneDrive folder path to poll for new files to ingest (e.g. /Inbox/Scanner). "
+            "Uses the existing OneDrive client credentials."
+        ),
+    )
+    onedrive_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete files from OneDrive ingest folder after download and enqueue.",
+    )
+
+    # Nextcloud ingest — reuses existing Nextcloud WebDAV credentials
+    nextcloud_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable Nextcloud watch folder ingestion. Requires Nextcloud WebDAV credentials.",
+    )
+    nextcloud_ingest_folder: Optional[str] = Field(
+        default=None,
+        description=(
+            "Nextcloud folder path to poll for new files to ingest (e.g. /Scans/Inbox). "
+            "Uses the existing Nextcloud upload URL and credentials."
+        ),
+    )
+    nextcloud_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete files from Nextcloud ingest folder after download and enqueue.",
+    )
+
+    # S3 ingest — reuses existing AWS/S3 credentials
+    s3_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable Amazon S3 watch folder (prefix) ingestion. Requires S3 credentials.",
+    )
+    s3_ingest_prefix: Optional[str] = Field(
+        default=None,
+        description=(
+            "S3 key prefix to poll for new objects to ingest (e.g. inbox/scanner/). "
+            "Uses the existing S3 bucket and AWS credentials."
+        ),
+    )
+    s3_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete objects from S3 ingest prefix after download and enqueue.",
+    )
+
+    # WebDAV ingest — reuses existing WebDAV credentials
+    webdav_ingest_enabled: bool = Field(
+        default=False,
+        description="Enable WebDAV watch folder ingestion. Requires WebDAV URL and credentials.",
+    )
+    webdav_ingest_folder: Optional[str] = Field(
+        default=None,
+        description=(
+            "WebDAV folder path to poll for new files to ingest (e.g. /remote.php/webdav/Inbox). "
+            "Uses the existing WebDAV URL and credentials."
+        ),
+    )
+    webdav_ingest_delete_after_process: bool = Field(
+        default=False,
+        description="Delete files from WebDAV ingest folder after download and enqueue.",
+    )
+
     # IMAP 1
     imap1_host: Optional[str] = None
     imap1_port: Optional[int] = 993
@@ -178,7 +432,7 @@ class Settings(BaseSettings):
     # In development/testing, set to True to disable verification (not recommended)
     sftp_disable_host_key_verification: bool = False  # Default enforces host key verification
 
-    # Email settings
+    # Email settings (shared SMTP – used for password reset, verification emails, etc.)
     email_host: Optional[str] = None
     email_port: Optional[int] = 587
     email_username: Optional[str] = None
@@ -187,6 +441,15 @@ class Settings(BaseSettings):
     email_sender: Optional[str] = None  # From address, defaults to email_username if not set
     email_default_recipient: Optional[str] = None
 
+    # Email destination settings (dedicated SMTP for document delivery – decoupled from shared email above)
+    dest_email_host: Optional[str] = None
+    dest_email_port: Optional[int] = 587
+    dest_email_username: Optional[str] = None
+    dest_email_password: Optional[str] = None
+    dest_email_use_tls: bool = True
+    dest_email_sender: Optional[str] = None  # From address for delivered documents
+    dest_email_default_recipient: Optional[str] = None  # Fallback recipient for document delivery
+
     # OneDrive settings
     onedrive_client_id: Optional[str] = None
     onedrive_client_secret: Optional[str] = None
@@ -221,6 +484,69 @@ class Settings(BaseSettings):
 
     # Feature flags
     allow_file_delete: bool = True  # Default to allowing file deletion from database
+
+    # PDF/A archival conversion settings
+    enable_pdfa_conversion: bool = Field(
+        default=False,
+        description=(
+            "Enable PDF/A archival variant generation. When enabled, PDF/A copies of both the "
+            "original ingested file and the processed file are created and saved alongside the "
+            "standard copies. Uses ocrmypdf with Ghostscript for the conversion. "
+            "This may double or triple storage but provides better legal coverage. Default: False."
+        ),
+    )
+    pdfa_format: str = Field(
+        default="2",
+        description=(
+            "PDF/A format variant to produce. Passed to ocrmypdf --output-type pdfa-N. "
+            "Valid values: '1' (PDF/A-1b), '2' (PDF/A-2b), '3' (PDF/A-3b). Default: '2'."
+        ),
+    )
+    pdfa_upload_original: bool = Field(
+        default=False,
+        description=(
+            "Upload the original-file PDF/A variant to all configured storage providers. "
+            "Files are placed in the provider's folder + PDFA_UPLOAD_FOLDER subfolder. Default: False."
+        ),
+    )
+    pdfa_upload_processed: bool = Field(
+        default=False,
+        description=(
+            "Upload the processed-file PDF/A variant to all configured storage providers. "
+            "Files are placed in the provider's folder + PDFA_UPLOAD_FOLDER subfolder. Default: False."
+        ),
+    )
+    pdfa_upload_folder: str = Field(
+        default="pdfa",
+        description=(
+            "Subfolder name appended to each storage provider's configured folder for PDF/A uploads. "
+            "For example if Dropbox folder is '/Documents' and this is 'pdfa', PDF/A files go to "
+            "'/Documents/pdfa'. Set to empty string to upload into the same folder. Default: 'pdfa'."
+        ),
+    )
+    google_drive_pdfa_folder_id: str = Field(
+        default="",
+        description=(
+            "Google Drive folder ID for PDF/A uploads. Since Google Drive uses IDs not paths, "
+            "this must be set separately. If empty, uses the standard google_drive_folder_id."
+        ),
+    )
+    pdfa_timestamp_enabled: bool = Field(
+        default=False,
+        description=(
+            "Enable RFC 3161 timestamping of PDF/A files via a Timestamp Authority (TSA). "
+            "Creates a .tsr file alongside each PDF/A file for legal proof of existence. "
+            "Requires openssl binary on PATH. Default: False."
+        ),
+    )
+    pdfa_timestamp_url: str = Field(
+        default="https://freetsa.org/tsr",
+        description=(
+            "URL of the RFC 3161 Timestamp Authority. Default: FreeTSA (https://freetsa.org/tsr). "
+            "Other options: GlobalSign, DigiStamp, or any RFC 3161-compliant TSA."
+        ),
+    )
+
     imap_readonly_mode: bool = Field(
         default=False,
         description=(
@@ -272,6 +598,66 @@ class Settings(BaseSettings):
         default=True,
         description="Send notifications when files are successfully processed",
     )
+    notify_on_user_signup: bool = Field(
+        default=True,
+        description="Send admin notifications when a new user signs up",
+    )
+    notify_on_plan_change: bool = Field(
+        default=True,
+        description="Send admin notifications when a user changes their subscription plan",
+    )
+    notify_on_payment_issue: bool = Field(
+        default=True,
+        description="Send admin notifications when a payment issue is reported for a user",
+    )
+
+    # Webhook settings
+    webhook_enabled: bool = Field(
+        default=True,
+        description="Enable webhook delivery for document events",
+    )
+
+    # ── Backup / restore settings ──────────────────────────────────────────────
+    backup_enabled: bool = Field(
+        default=True,
+        description=(
+            "Enable automatic scheduled database backups. "
+            "When enabled, hourly, daily, and weekly backups are created automatically. Default: True."
+        ),
+    )
+    backup_dir: Optional[str] = Field(
+        default=None,
+        description=("Directory where local backup archives are stored. Defaults to <workdir>/backups when not set."),
+    )
+    # Remote destination: one of s3, dropbox, google_drive, onedrive, nextcloud,
+    # webdav, ftp, sftp, email, or empty/None for local-only.
+    backup_remote_destination: Optional[str] = Field(
+        default=None,
+        description=(
+            "Storage provider to upload remote backup copies to. "
+            "Accepted values: s3, dropbox, google_drive, onedrive, nextcloud, webdav, ftp, sftp, email. "
+            "Leave empty to keep backups local only."
+        ),
+    )
+    backup_remote_folder: str = Field(
+        default="backups",
+        description=(
+            "Sub-folder / key prefix used when uploading backup archives to the remote destination. Default: 'backups'."
+        ),
+    )
+    # Retention counts (number of snapshots to keep per tier)
+    backup_retain_hourly: int = Field(
+        default=96,
+        description="Number of hourly backups to retain (default 96 = 4 days × 24 h).",
+    )
+    backup_retain_daily: int = Field(
+        default=21,
+        description="Number of daily backups to retain (default 21 = 3 weeks).",
+    )
+    backup_retain_weekly: int = Field(
+        default=13,
+        description="Number of weekly backups to retain (default 13 ≈ 3 months / 91 days).",
+    )
 
     # File upload size limits (for security - see SECURITY_AUDIT.md)
     max_upload_size: int = Field(
@@ -309,6 +695,36 @@ class Settings(BaseSettings):
             " If False, the check is still performed but not displayed. Default: True."
         ),
     )
+    near_duplicate_threshold: float = Field(
+        default=0.85,
+        description=(
+            "Minimum cosine similarity score (0–1) between two documents' text embeddings to consider "
+            "them near-duplicates.  Higher values require closer content matches.  Default: 0.85."
+        ),
+    )
+    embedding_model: str = Field(
+        default="text-embedding-3-small",
+        description=(
+            "Model name used for generating text embeddings via the OpenAI-compatible API.  "
+            "Embeddings drive the document similarity feature.  Default: text-embedding-3-small."
+        ),
+    )
+    embedding_max_tokens: int = Field(
+        default=8000,
+        description=(
+            "Maximum number of tokens to send to the embedding model.  "
+            "Text is truncated to approximately this many tokens (using a "
+            "conservative 3-chars-per-token estimate) before calling the API.  "
+            "Set this below the model's context window (e.g. 8000 for an 8192-token model)."
+        ),
+    )
+    embedding_backfill_batch_size: int = Field(
+        default=50,
+        description=(
+            "Maximum number of files to queue for embedding computation per "
+            "backfill run.  Keeps the worker and embedding API load bounded."
+        ),
+    )
 
     # Text quality check - AI-based assessment of embedded PDF text
     enable_text_quality_check: bool = Field(
@@ -339,6 +755,32 @@ class Settings(BaseSettings):
         ),
     )
 
+    # ---------------------------------------------------------------------------
+    # Task retry settings (see app/tasks/retry_config.py)
+    # ---------------------------------------------------------------------------
+    task_retry_max_retries: int = Field(
+        default=3,
+        description=("Maximum number of automatic retry attempts for failed Celery tasks. Default: 3."),
+    )
+    task_retry_delays: Union[List[int], str] = Field(
+        default_factory=lambda: [60, 300, 900],
+        description=(
+            "Comma-separated list of retry countdown values in seconds. "
+            "Each value is the delay before the corresponding retry attempt. "
+            "If a task fails more times than entries in this list, the last delay "
+            "is doubled for each additional attempt. "
+            "Default: 60,300,900 (1 min, 5 min, 15 min)."
+        ),
+    )
+    task_retry_jitter: bool = Field(
+        default=True,
+        description=(
+            "Apply ±20 % random jitter to retry countdowns to prevent "
+            "thundering-herd problems when many tasks fail simultaneously. "
+            "Default: True (enabled)."
+        ),
+    )
+
     # Processing step timeout - prevents files from getting stuck in "in_progress" state
     step_timeout: int = Field(
         default=600,
@@ -527,6 +969,15 @@ class Settings(BaseSettings):
             return []
         return v
 
+    @field_validator("task_retry_delays", mode="before")
+    @classmethod
+    def parse_task_retry_delays(cls, v: str | list) -> list[int]:
+        """Parse task retry delays from comma-separated string or list of ints."""
+        if isinstance(v, str):
+            parts = [p.strip() for p in v.split(",") if p.strip()]
+            return [int(p) for p in parts]
+        return [int(item) for item in v]
+
     @field_validator("session_secret")
     @classmethod
     def validate_session_secret(cls, v: str | None, info: object) -> str | None:
@@ -599,5 +1050,51 @@ class Settings(BaseSettings):
         # Return basic info if file not found
         return f"Version: {self.version}\nBuild Date: {self.build_date}\nGit SHA: {self.git_sha}"
 
+    @property
+    def release_name(self) -> str | None:
+        """Get the release codename for the current version from release_names.json.
+
+        Looks up the current version's minor version prefix (e.g., '0.5' for '0.5.3')
+        in release_names.json to find the associated codename. Returns None if no
+        codename is defined for the current version.
+
+        Returns:
+            The release codename string, or None if not found.
+        """
+        version = self.version
+        if not version or version == "unknown":
+            return None
+
+        release_names_file = os.path.join(os.path.dirname(os.path.dirname(__file__)), "release_names.json")
+        if not os.path.exists(release_names_file):
+            return None
+
+        try:
+            with open(release_names_file, "r") as f:
+                data = json.load(f)
+
+            releases = data.get("releases", {})
+
+            # Try exact version match first (e.g., "0.5.0")
+            if version in releases:
+                return releases[version].get("codename")
+
+            # Try minor version prefix (e.g., "0.5" for "0.5.3")
+            parts = version.split(".")
+            if len(parts) >= 2:
+                minor_prefix = f"{parts[0]}.{parts[1]}"
+                if minor_prefix in releases:
+                    return releases[minor_prefix].get("codename")
+
+            # Try major version prefix (e.g., "1" for "1.0.0")
+            if len(parts) >= 1:
+                major_prefix = parts[0]
+                if major_prefix in releases:
+                    return releases[major_prefix].get("codename")
+
+            return None
+        except (json.JSONDecodeError, KeyError, IndexError):
+            return None
+
 
 settings = Settings()

app/database.py

@@ -26,8 +26,14 @@ SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
 def init_db() -> None:
     """
     Ensures the SQLite database file and its parent directory exist (if using sqlite).
-    Then runs Base.metadata.create_all(bind=engine) to initialize tables and
-    applies any pending Alembic migrations.
+    Then initializes tables and applies any pending Alembic migrations:
+
+    - Fresh/legacy databases (no ``alembic_version`` table): creates all tables via
+      ``Base.metadata.create_all()``, then stamps the Alembic version to ``head``.
+    - Alembic-tracked databases (``alembic_version`` present): skips ``create_all()``
+      and applies pending migrations via ``alembic upgrade head``.  Skipping
+      ``create_all()`` prevents an ``OperationalError`` when the ORM model defines a
+      table (e.g. ``webhook_configs``) that a pending migration also tries to create.
     """
     # 1. Parse the DB URL to see if it's sqlite
     url = make_url(DB_URL)
@@ -47,12 +53,19 @@ def init_db() -> None:
                 logger.info(f"Creating new SQLite database file at {database_path}")
                 open(database_path, "a").close()
 
-    # 5. Now create tables if they don't exist yet
+    # 5. Create tables only for fresh/legacy databases not yet tracked by Alembic.
+    #    For Alembic-tracked databases, skip create_all to avoid conflicts where
+    #    the ORM model would create a table (e.g. webhook_configs) that a pending
+    #    Alembic migration also tries to create, causing an OperationalError.
     try:
-        Base.metadata.create_all(bind=engine)
-        logger.info("Database initialization complete (tables created if not exist).")
+        from sqlalchemy import inspect
 
-        # 6. Run Alembic migrations for existing databases
+        table_names = inspect(engine).get_table_names()
+        if "alembic_version" not in table_names:
+            Base.metadata.create_all(bind=engine)
+            logger.info("Database initialization complete (tables created if not exist).")
+
+        # 6. Run Alembic migrations (stamps fresh/legacy DBs to head, upgrades tracked DBs)
         _run_alembic_upgrade(engine)
     except exc.SQLAlchemyError as e:
         logger.error(f"Error initializing database: {e}")

app/main.py

@@ -16,6 +16,7 @@ from starlette.middleware.trustedhost import TrustedHostMiddleware
 from uvicorn.middleware.proxy_headers import ProxyHeadersMiddleware
 
 from app.api import router as api_router
+from app.api.local_auth import router as local_auth_router
 from app.auth import router as auth_router
 from app.config import settings
 from app.database import init_db
@@ -99,6 +100,33 @@ async def lifespan(app: FastAPI):
     # Send startup notification
     notify_startup()
 
+    # Seed default subscription plans if none exist
+    try:
+        from app.database import SessionLocal as _SessionLocal
+        from app.utils.subscription import seed_default_plans as _seed_plans
+
+        _db_seed = _SessionLocal()
+        try:
+            _seed_plans(_db_seed)
+        finally:
+            _db_seed.close()
+    except Exception:
+        logging.debug("Subscription plan seeding skipped — DB may not be ready yet")  # noqa: S110
+
+    # Seed the default system pipeline (mirrors the current hardcoded processing
+    # workflow) so it is immediately visible in the Pipelines management UI.
+    try:
+        from app.api.pipelines import seed_default_pipeline as _seed_pipeline
+        from app.database import SessionLocal as _SessionLocal  # noqa: F811 (re-import for clarity)
+
+        _db_pipeline = _SessionLocal()
+        try:
+            _seed_pipeline(_db_pipeline)
+        finally:
+            _db_pipeline.close()
+    except Exception:
+        logging.debug("Default pipeline seeding skipped — DB may not be ready yet")  # noqa: S110
+
     # Application is now running
     yield
 
@@ -174,6 +202,15 @@ if os.path.exists(static_dir):
 else:
     print(f"WARNING: Static directory not found at {static_dir}. Static files will not be served.")
 
+# Mount the built MkDocs documentation site at /help/
+# The docs are pre-built into docs_build/ during the Docker image build.
+# When running locally, run `mkdocs build` from the repo root first.
+docs_build_dir = pathlib.Path(__file__).parents[1] / "docs_build"
+if os.path.exists(docs_build_dir):
+    app.mount("/help", StaticFiles(directory=str(docs_build_dir), html=True), name="help_docs")
+else:
+    print(f"INFO: Help docs not found at {docs_build_dir}. Run 'mkdocs build' to generate them.")
+
 
 # Custom exception handlers that return JSON for API routes and HTML for frontend routes
 @app.exception_handler(HTTPException)
@@ -233,4 +270,5 @@ def test_500():
 app.include_router(frontend_router)
 app.include_router(files_router)  # Explicitly include the files router
 app.include_router(auth_router)
+app.include_router(local_auth_router)
 app.include_router(api_router, prefix="/api")

app/models.py

@@ -1,11 +1,12 @@
 # app/models.py
 
-from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Integer, String, Text, UniqueConstraint, func
+from sqlalchemy import Boolean, Column, DateTime, Float, ForeignKey, Integer, String, Text, UniqueConstraint, func
 
 from app.database import Base
 
 # Foreign key constants
 _FILES_ID_FK = "files.id"
+_PIPELINES_ID_FK = "pipelines.id"
 
 
 class DocumentMetadata(Base):
@@ -24,6 +25,11 @@ class FileRecord(Base):
 
     id = Column(Integer, primary_key=True, index=True)
 
+    # Owner identifier for multi-user mode.
+    # Stores the user's unique identifier (e.g. email or OAuth sub claim).
+    # NULL means the file belongs to the shared/global space (single-user mode).
+    owner_id = Column(String, nullable=True, index=True)
+
     # Hash of the file content (e.g. SHA-256)
     # Note: duplicates are allowed so filehash is not unique
     filehash = Column(String, index=True, nullable=False)
@@ -67,6 +73,16 @@ class FileRecord(Base):
     # Human-readable document title from AI metadata
     document_title = Column(String, nullable=True)
 
+    # PDF/A archival variant paths (generated when ENABLE_PDFA_CONVERSION is True)
+    original_pdfa_path = Column(String, nullable=True)  # PDF/A copy of the original ingested file
+    processed_pdfa_path = Column(String, nullable=True)  # PDF/A copy of the processed file
+
+    # Pre-computed text embedding vector stored as JSON array of floats
+    embedding = Column(Text, nullable=True)
+
+    # Processing pipeline assigned to this file (NULL = use system default)
+    pipeline_id = Column(Integer, ForeignKey(_PIPELINES_ID_FK), nullable=True, index=True)
+
     # Timestamp when we inserted this record
     created_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)
 
@@ -143,3 +159,262 @@ class SavedSearch(Base):
     updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
 
     __table_args__ = (UniqueConstraint("user_id", "name", name="unique_user_search_name"),)
+
+
+class WebhookConfig(Base):
+    """Webhook configuration for notifying external systems of document events."""
+
+    __tablename__ = "webhook_configs"
+
+    id = Column(Integer, primary_key=True, index=True)
+    url = Column(String, nullable=False)  # Target URL for webhook delivery
+    secret = Column(String, nullable=True)  # Shared secret for HMAC-SHA256 signature
+    events = Column(Text, nullable=False)  # JSON list of subscribed events
+    is_active = Column(Boolean, default=True, nullable=False)  # Whether the webhook is active
+    description = Column(String, nullable=True)  # Optional human-readable description
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class LocalUser(Base):
+    """A locally-registered user authenticated by email and bcrypt password.
+
+    Created during the self-registration flow when ``allow_local_signup`` is
+    enabled.  The account is inactive (``is_active=False``) until the user
+    clicks the verification link sent to their email address.
+    """
+
+    __tablename__ = "local_users"
+
+    id = Column(Integer, primary_key=True, index=True)
+    email = Column(String(255), unique=True, nullable=False, index=True)
+    username = Column(String(64), unique=True, nullable=False, index=True)
+    display_name = Column(String(255), nullable=True)
+    hashed_password = Column(String(255), nullable=False)
+    is_active = Column(Boolean, nullable=False, default=False, server_default="0")
+    is_admin = Column(Boolean, nullable=False, default=False, server_default="0")
+    email_verification_token = Column(String(128), nullable=True)
+    email_verification_sent_at = Column(DateTime(timezone=True), nullable=True)
+    password_reset_token = Column(String(128), nullable=True)
+    password_reset_sent_at = Column(DateTime(timezone=True), nullable=True)
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class UserProfile(Base):
+    """Per-user profile for admin-managed settings in multi-user mode.
+
+    Each row corresponds to one authenticated user (identified by their
+    ``user_id``, which matches ``FileRecord.owner_id``).  The admin can
+    create or update profiles to override global defaults such as the
+    daily upload limit and to attach notes or block a user.
+    """
+
+    __tablename__ = "user_profiles"
+
+    id = Column(Integer, primary_key=True, index=True)
+
+    # Stable user identifier — matches FileRecord.owner_id (OAuth sub / email / username)
+    user_id = Column(String, unique=True, nullable=False, index=True)
+
+    # Optional human-readable display name set by the admin
+    display_name = Column(String, nullable=True)
+
+    # Per-user daily upload limit; NULL means "use global default"
+    daily_upload_limit = Column(Integer, nullable=True)
+
+    # Admin-only free-text notes about this user
+    notes = Column(Text, nullable=True)
+
+    # When True the user is prevented from uploading new documents
+    is_blocked = Column(Boolean, default=False, nullable=False)
+
+    # Subscription tier: "free" | "starter" | "professional" | "business"
+    # NULL is treated as "free" by the subscription utility.
+    subscription_tier = Column(String(50), nullable=True, default="free")
+
+    # Billing cycle and overage settings (added in migration 016)
+    subscription_billing_cycle = Column(String(10), nullable=False, default="monthly", server_default="monthly")
+    subscription_period_start = Column(DateTime(timezone=True), nullable=True)
+    allow_overage = Column(Boolean, nullable=False, default=False, server_default="0")
+
+    # Pending subscription change (added in migration 020_add_subscription_change_pending)
+    # When a user requests a downgrade, the new tier is stored here and the
+    # change is applied on `subscription_change_pending_date`.  Upgrades are
+    # applied immediately and these fields are left NULL.
+    subscription_change_pending_tier = Column(String(50), nullable=True)
+    subscription_change_pending_date = Column(DateTime(timezone=True), nullable=True)
+
+    # When True, the user is on a complimentary (uncharged) plan — they keep all tier
+    # quota benefits but are never billed via Stripe.  Automatically set for admin users.
+    is_complimentary = Column(Boolean, nullable=False, default=False, server_default="0")
+
+    # Onboarding tracking (added in migration 017)
+    onboarding_completed = Column(Boolean, nullable=False, default=False, server_default="0")
+    onboarding_completed_at = Column(DateTime(timezone=True), nullable=True)
+    contact_email = Column(String(255), nullable=True)
+    preferred_destination = Column(String(50), nullable=True)
+    stripe_customer_id = Column(String(64), nullable=True)
+
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class SubscriptionPlan(Base):
+    """Dynamically configurable subscription plan stored in the database.
+
+    Plans are shown on the public /pricing page and assigned to users via
+    UserProfile.subscription_tier (which stores plan_id).  On first start the
+    four default plans are seeded from TIER_DEFAULTS in app/utils/subscription.py.
+    """
+
+    __tablename__ = "subscription_plans"
+
+    id = Column(Integer, primary_key=True, index=True)
+    plan_id = Column(String(50), unique=True, nullable=False, index=True)
+    name = Column(String(100), nullable=False)
+    tagline = Column(String(255), nullable=True)
+
+    # Pricing
+    price_monthly = Column(Float, nullable=False, default=0.0)
+    price_yearly = Column(Float, nullable=False, default=0.0)
+    trial_days = Column(Integer, nullable=False, default=0)
+
+    # Volume limits (0 = unlimited)
+    lifetime_file_limit = Column(Integer, nullable=False, default=0)
+    daily_upload_limit = Column(Integer, nullable=False, default=0)
+    monthly_upload_limit = Column(Integer, nullable=False, default=0)
+    max_storage_destinations = Column(Integer, nullable=False, default=0)
+    max_ocr_pages_monthly = Column(Integer, nullable=False, default=0)
+    max_file_size_mb = Column(Integer, nullable=False, default=0)
+    max_mailboxes = Column(Integer, nullable=False, default=0)
+
+    # Overage configuration
+    overage_percent = Column(Integer, nullable=False, default=20)
+    allow_overage_billing = Column(Boolean, nullable=False, default=False)
+    overage_price_per_doc = Column(Float, nullable=True)
+    overage_price_per_ocr_page = Column(Float, nullable=True)
+
+    # Display / marketing
+    is_active = Column(Boolean, nullable=False, default=True)
+    is_highlighted = Column(Boolean, nullable=False, default=False)
+    badge_text = Column(String(50), nullable=True)
+    cta_text = Column(String(100), nullable=False, default="Get started")
+    sort_order = Column(Integer, nullable=False, default=0)
+    features = Column(Text, nullable=True)  # JSON-encoded list[str]
+    api_access = Column(Boolean, nullable=False, default=False)
+    stripe_price_id_monthly = Column(String(128), nullable=True)
+    stripe_price_id_yearly = Column(String(128), nullable=True)
+
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class Pipeline(Base):
+    """User-defined processing pipeline: an ordered set of steps.
+
+    Pipelines are user-specific.  A pipeline with ``owner_id = NULL`` is a
+    *system default* pipeline that only admins may create.  Regular users
+    create pipelines under their own ``owner_id``.  When a file has no
+    explicit pipeline assigned, the active system default is used.
+    """
+
+    __tablename__ = "pipelines"
+
+    id = Column(Integer, primary_key=True, index=True)
+
+    # Owner of this pipeline.  NULL = system/admin pipeline visible to everyone.
+    owner_id = Column(String, nullable=True, index=True)
+
+    # Human-readable name (unique per owner)
+    name = Column(String(255), nullable=False)
+
+    # Optional description
+    description = Column(Text, nullable=True)
+
+    # When True this pipeline is the default for new files belonging to the owner
+    # (or the global default when owner_id is NULL).  Only one pipeline per
+    # owner may be active default at a time — enforced at the application level.
+    is_default = Column(Boolean, nullable=False, default=False)
+
+    # Soft-disable without deleting
+    is_active = Column(Boolean, nullable=False, default=True)
+
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class PipelineStep(Base):
+    """A single step in a processing pipeline.
+
+    Steps are executed in ascending ``position`` order.  Each step has a
+    ``step_type`` that maps to a built-in processing action and an optional
+    ``config`` JSON blob with step-specific parameters.
+    """
+
+    __tablename__ = "pipeline_steps"
+
+    id = Column(Integer, primary_key=True, index=True)
+    pipeline_id = Column(Integer, ForeignKey(_PIPELINES_ID_FK), nullable=False, index=True)
+
+    # Execution order within the pipeline (lower = earlier)
+    position = Column(Integer, nullable=False, default=0)
+
+    # One of the recognised step types (see PIPELINE_STEP_TYPES in pipelines.py)
+    step_type = Column(String(100), nullable=False)
+
+    # Optional human-readable label override (defaults to step_type label)
+    label = Column(String(255), nullable=True)
+
+    # JSON-encoded step-specific configuration dict
+    config = Column(Text, nullable=True)
+
+    # When False this step is skipped during execution
+    enabled = Column(Boolean, nullable=False, default=True)
+
+    created_at = Column(DateTime(timezone=True), server_default=func.now())
+    updated_at = Column(DateTime(timezone=True), server_default=func.now(), onupdate=func.now())
+
+
+class BackupRecord(Base):
+    """Tracks database backup files and their retention metadata.
+
+    Each row represents one backup archive (a gzipped SQLite dump).
+    ``backup_type`` classifies the backup for retention purposes:
+    - ``hourly``  – kept for up to 4 days (96 snapshots)
+    - ``daily``   – kept for up to 3 weeks (21 snapshots)
+    - ``weekly``  – kept for up to 13 weeks (≈ 90 days)
+    ``local_path`` is the full filesystem path of the local copy (``None``
+    once pruned).  ``remote_destination`` and ``remote_path`` describe the
+    remote copy when one has been uploaded to a storage provider or e-mailed.
+    """
+
+    __tablename__ = "backup_records"
+
+    id = Column(Integer, primary_key=True, index=True)
+
+    # Human-readable archive filename (e.g. backup_hourly_2026-03-07T12-00-00.db.gz)
+    filename = Column(String(255), nullable=False, unique=True)
+
+    # Full path on the local filesystem (may be NULL for remote-only backups)
+    local_path = Column(String(1024), nullable=True)
+
+    # Classification used by the retention policy
+    backup_type = Column(String(20), nullable=False, index=True)  # hourly | daily | weekly
+
+    # Archive size in bytes (0 if unknown)
+    size_bytes = Column(Integer, nullable=False, default=0)
+
+    # Checksum of the archive for integrity verification (SHA-256 hex)
+    checksum = Column(String(64), nullable=True)
+
+    # Whether the backup was successfully created
+    status = Column(String(20), nullable=False, default="ok")  # ok | failed
+
+    # Storage destination where a remote copy was uploaded (e.g. "s3", "dropbox", "email")
+    remote_destination = Column(String(50), nullable=True)
+
+    # Path / key of the remote copy (bucket key, folder path, etc.)
+    remote_path = Column(String(1024), nullable=True)
+
+    created_at = Column(DateTime(timezone=True), server_default=func.now(), index=True)

app/tasks/backup_tasks.py

@@ -0,0 +1,741 @@
+"""
+Backup and restore tasks for DocuElevate.
+
+Retention strategy
+------------------
+- **hourly** backups  – retained for 4 days  (``backup_retain_hourly``, default 96)
+- **daily** backups   – retained for 3 weeks (``backup_retain_daily``,  default 21)
+- **weekly** backups  – retained for 13 weeks (``backup_retain_weekly``, default 13)
+
+Three separate Celery-beat entries call ``create_backup`` with the appropriate
+``backup_type`` argument:
+- every hour  → ``create_backup("hourly")``
+- every day   → ``create_backup("daily")``
+- every week  → ``create_backup("weekly")``
+
+After each backup is created ``_apply_retention`` prunes old local backups for
+that tier.  Remote copies are pruned by ``_prune_remote_backups`` which mirrors
+the same retention limits.
+
+Supported database backends
+----------------------------
+- **SQLite**    – dumped via Python's built-in ``sqlite3.iterdump()``; archive extension ``.db.gz``
+- **PostgreSQL** – dumped via ``pg_dump --format=plain``; archive extension ``.pgsql.gz``
+- **MySQL / MariaDB** – dumped via ``mysqldump --single-transaction``; archive extension ``.mysql.gz``
+"""
+
+import gzip
+import hashlib
+import logging
+import os
+import subprocess
+from datetime import datetime, timezone
+from pathlib import Path
+
+from app.celery_app import celery
+from app.config import settings
+from app.database import SessionLocal
+from app.models import BackupRecord
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Internal helpers
+# ---------------------------------------------------------------------------
+
+_BACKUP_TYPE_RETAIN: dict[str, str] = {
+    "hourly": "backup_retain_hourly",
+    "daily": "backup_retain_daily",
+    "weekly": "backup_retain_weekly",
+}
+
+#: Map of backend name → archive file extension.
+_BACKEND_EXTENSIONS: dict[str, str] = {
+    "sqlite": ".db.gz",
+    "postgresql": ".pgsql.gz",
+    "mysql": ".mysql.gz",
+}
+
+
+def _backup_dir() -> Path:
+    """Return (and create) the local backup directory."""
+    raw = getattr(settings, "backup_dir", None) or os.path.join(settings.workdir, "backups")
+    path = Path(raw)
+    path.mkdir(parents=True, exist_ok=True)
+    return path
+
+
+def _db_backend() -> str:
+    """Return the database backend name (e.g. ``'sqlite'``, ``'postgresql'``, ``'mysql'``)."""
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(settings.database_url)
+    return url.get_backend_name()
+
+
+def _db_path() -> Path | None:
+    """Return the SQLite database file path, or None for non-SQLite databases."""
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(settings.database_url)
+    if url.get_backend_name() != "sqlite":
+        return None
+    db = url.database
+    if not db or db == ":memory:":
+        return None
+    return Path(db)
+
+
+def _archive_ext_for_backend(backend: str) -> str:
+    """Return the archive file extension for the given database backend.
+
+    Args:
+        backend: Backend name as returned by
+            ``sqlalchemy.engine.url.URL.get_backend_name()`` (e.g. ``'sqlite'``).
+
+    Returns:
+        File extension string including the leading dot, e.g. ``'.db.gz'``.
+        Falls back to ``'.sql.gz'`` for unknown backends.
+    """
+    return _BACKEND_EXTENSIONS.get(backend, ".sql.gz")
+
+
+def _sha256(path: Path) -> str:
+    """Return the SHA-256 hex digest of *path*."""
+    h = hashlib.sha256()
+    with open(path, "rb") as fh:
+        for chunk in iter(lambda: fh.read(65536), b""):
+            h.update(chunk)
+    return h.hexdigest()
+
+
+def _dump_sqlite(db_path: Path, dest: Path) -> None:
+    """Write a gzip-compressed SQL dump of *db_path* to *dest*."""
+    import sqlite3
+
+    conn = sqlite3.connect(str(db_path))
+    try:
+        with gzip.open(str(dest), "wt", encoding="utf-8") as gz:
+            for line in conn.iterdump():
+                gz.write(line + "\n")
+    finally:
+        conn.close()
+
+
+def _dump_postgresql(db_url: str, dest: Path) -> None:
+    """Write a gzip-compressed ``pg_dump`` of the PostgreSQL database to *dest*.
+
+    Uses ``PGPASSWORD`` environment variable so the password is never exposed on
+    the process command line.
+
+    Args:
+        db_url: Full SQLAlchemy database URL (e.g. ``postgresql://user:pass@host/db``).
+        dest: Destination path for the ``.pgsql.gz`` archive.
+
+    Raises:
+        RuntimeError: If ``pg_dump`` exits with a non-zero return code.
+        FileNotFoundError: If the ``pg_dump`` binary is not found.
+    """
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(db_url)
+    env = os.environ.copy()
+    if url.password:
+        env["PGPASSWORD"] = str(url.password)
+
+    # Command arguments are built from the SQLAlchemy URL (admin-configured DATABASE_URL),
+    # not from user-controlled input.  shell=False (the default when passing a list) is used
+    # so there is no shell interpretation of the argument values.
+    cmd: list[str] = ["pg_dump", "--format=plain", "--no-password"]
+    if url.host:
+        cmd.extend(["-h", url.host])
+    if url.port:
+        cmd.extend(["-p", str(url.port)])
+    if url.username:
+        cmd.extend(["-U", url.username])
+    if url.database:
+        cmd.append(url.database)
+
+    with gzip.open(str(dest), "wb") as gz:
+        proc = subprocess.Popen(  # noqa: S603
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            env=env,
+        )
+        stdout = proc.stdout
+        if stdout is None:  # pragma: no cover – guaranteed by stdout=PIPE
+            raise RuntimeError("pg_dump produced no stdout pipe")
+        try:
+            while True:
+                chunk = stdout.read(65536)
+                if not chunk:
+                    break
+                gz.write(chunk)
+        finally:
+            stdout.close()
+            stderr_bytes = proc.stderr.read() if proc.stderr else b""
+            proc.wait()
+
+    if proc.returncode != 0:
+        dest.unlink(missing_ok=True)
+        raise RuntimeError(
+            f"pg_dump exited with code {proc.returncode}: {stderr_bytes.decode(errors='replace').strip()}"
+        )
+
+
+def _dump_mysql(db_url: str, dest: Path) -> None:
+    """Write a gzip-compressed ``mysqldump`` of the MySQL database to *dest*.
+
+    Uses the ``MYSQL_PWD`` environment variable so the password is never exposed
+    on the process command line.
+
+    Args:
+        db_url: Full SQLAlchemy database URL
+            (e.g. ``mysql+pymysql://user:pass@host/db``).
+        dest: Destination path for the ``.mysql.gz`` archive.
+
+    Raises:
+        RuntimeError: If ``mysqldump`` exits with a non-zero return code.
+        FileNotFoundError: If the ``mysqldump`` binary is not found.
+    """
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(db_url)
+    env = os.environ.copy()
+    if url.password:
+        env["MYSQL_PWD"] = str(url.password)
+
+    # Command arguments are built from the SQLAlchemy URL (admin-configured DATABASE_URL).
+    # shell=False (list form) prevents shell interpretation of argument values.
+    cmd: list[str] = ["mysqldump", "--single-transaction", "--routines", "--triggers"]
+    if url.host:
+        cmd.extend(["-h", url.host])
+    if url.port:
+        cmd.extend(["-P", str(url.port)])
+    if url.username:
+        cmd.extend(["-u", url.username])
+    if url.database:
+        cmd.append(url.database)
+
+    with gzip.open(str(dest), "wb") as gz:
+        proc = subprocess.Popen(  # noqa: S603
+            cmd,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            env=env,
+        )
+        stdout = proc.stdout
+        if stdout is None:  # pragma: no cover – guaranteed by stdout=PIPE
+            raise RuntimeError("mysqldump produced no stdout pipe")
+        try:
+            while True:
+                chunk = stdout.read(65536)
+                if not chunk:
+                    break
+                gz.write(chunk)
+        finally:
+            stdout.close()
+            stderr_bytes = proc.stderr.read() if proc.stderr else b""
+            proc.wait()
+
+    if proc.returncode != 0:
+        dest.unlink(missing_ok=True)
+        raise RuntimeError(
+            f"mysqldump exited with code {proc.returncode}: {stderr_bytes.decode(errors='replace').strip()}"
+        )
+
+
+def _restore_sqlite(db_path: Path, archive_path: Path) -> None:
+    """Restore a SQLite database from a gzip-compressed SQL dump archive.
+
+    Validates the SQL by replaying it on an in-memory database before touching
+    the live file.  Saves a ``<db_path>.pre_restore`` rollback copy first.
+
+    Args:
+        db_path: Path to the live SQLite database file to overwrite.
+        archive_path: Path to the ``.db.gz`` gzip-compressed SQL dump.
+
+    Raises:
+        ValueError: If the archive cannot be decompressed or contains invalid SQL.
+        RuntimeError: If writing the restored database fails.
+    """
+    import shutil
+    import sqlite3
+
+    # Decompress and read SQL statements
+    try:
+        with gzip.open(str(archive_path), "rt", encoding="utf-8") as gz:
+            sql_script = gz.read()
+    except Exception as exc:
+        raise ValueError(f"Failed to decompress backup file: {exc}") from exc
+
+    # Validate by replaying on an in-memory database
+    try:
+        mem_conn = sqlite3.connect(":memory:")
+        mem_conn.executescript(sql_script)
+        mem_conn.close()
+    except sqlite3.Error as exc:
+        raise ValueError(f"Backup file contains invalid SQL: {exc}") from exc
+
+    # Preserve the current DB before overwriting
+    bak = str(db_path) + ".pre_restore"
+    try:
+        shutil.copy2(str(db_path), bak)
+    except OSError as exc:
+        logger.warning(f"Could not create pre-restore backup at {bak}: {exc}")
+
+    try:
+        restore_conn = sqlite3.connect(str(db_path))
+        restore_conn.executescript(sql_script)
+        restore_conn.close()
+    except sqlite3.Error as exc:
+        # Attempt rollback to the pre-restore copy
+        try:
+            if os.path.exists(bak):
+                shutil.copy2(bak, str(db_path))
+        except OSError as rollback_exc:
+            logger.error(f"Rollback failed; database may be corrupted: {rollback_exc}")
+        raise RuntimeError(f"SQLite restore failed: {exc}") from exc
+
+
+def _restore_postgresql(db_url: str, archive_path: Path) -> None:
+    """Restore a PostgreSQL database from a gzip-compressed SQL dump archive.
+
+    Pipes the decompressed dump to ``psql``.  Uses ``PGPASSWORD`` so the
+    password is never exposed on the process command line.
+
+    Args:
+        db_url: Full SQLAlchemy database URL.
+        archive_path: Path to the ``.pgsql.gz`` gzip-compressed ``pg_dump`` archive.
+
+    Raises:
+        RuntimeError: If ``psql`` exits with a non-zero return code.
+        FileNotFoundError: If the ``psql`` binary is not found.
+    """
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(db_url)
+    env = os.environ.copy()
+    if url.password:
+        env["PGPASSWORD"] = str(url.password)
+
+    # Command arguments are built from the SQLAlchemy URL (admin-configured DATABASE_URL).
+    # shell=False (list form) prevents shell interpretation of argument values.
+    cmd: list[str] = ["psql", "--no-password"]
+    if url.host:
+        cmd.extend(["-h", url.host])
+    if url.port:
+        cmd.extend(["-p", str(url.port)])
+    if url.username:
+        cmd.extend(["-U", url.username])
+    if url.database:
+        cmd.append(url.database)
+
+    with gzip.open(str(archive_path), "rb") as gz:
+        proc = subprocess.Popen(  # noqa: S603
+            cmd,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            env=env,
+        )
+        _, stderr_bytes = proc.communicate(input=gz.read())
+
+    if proc.returncode != 0:
+        raise RuntimeError(f"psql exited with code {proc.returncode}: {stderr_bytes.decode(errors='replace').strip()}")
+
+
+def _restore_mysql(db_url: str, archive_path: Path) -> None:
+    """Restore a MySQL database from a gzip-compressed SQL dump archive.
+
+    Pipes the decompressed dump to ``mysql``.  Uses the ``MYSQL_PWD``
+    environment variable so the password is never exposed on the command line.
+
+    Args:
+        db_url: Full SQLAlchemy database URL.
+        archive_path: Path to the ``.mysql.gz`` gzip-compressed ``mysqldump`` archive.
+
+    Raises:
+        RuntimeError: If ``mysql`` exits with a non-zero return code.
+        FileNotFoundError: If the ``mysql`` binary is not found.
+    """
+    from sqlalchemy.engine.url import make_url
+
+    url = make_url(db_url)
+    env = os.environ.copy()
+    if url.password:
+        env["MYSQL_PWD"] = str(url.password)
+
+    # Command arguments are built from the SQLAlchemy URL (admin-configured DATABASE_URL).
+    # shell=False (list form) prevents shell interpretation of argument values.
+    cmd: list[str] = ["mysql"]
+    if url.host:
+        cmd.extend(["-h", url.host])
+    if url.port:
+        cmd.extend(["-P", str(url.port)])
+    if url.username:
+        cmd.extend(["-u", url.username])
+    if url.database:
+        cmd.append(url.database)
+
+    with gzip.open(str(archive_path), "rb") as gz:
+        proc = subprocess.Popen(  # noqa: S603
+            cmd,
+            stdin=subprocess.PIPE,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.PIPE,
+            env=env,
+        )
+        _, stderr_bytes = proc.communicate(input=gz.read())
+
+    if proc.returncode != 0:
+        raise RuntimeError(f"mysql exited with code {proc.returncode}: {stderr_bytes.decode(errors='replace').strip()}")
+
+
+def _apply_retention(backup_type: str, db: object) -> None:
+    """Delete local backups beyond the retention limit for *backup_type*.
+
+    Args:
+        backup_type: One of ``hourly``, ``daily``, ``weekly``.
+        db: Active SQLAlchemy session.
+    """
+    retain_attr = _BACKUP_TYPE_RETAIN.get(backup_type, "backup_retain_hourly")
+    retain = int(getattr(settings, retain_attr, 96))
+
+    # Query ALL records for this tier (with or without a local file) so that
+    # remote-only and already-pruned records still count toward the retention window.
+    records = (
+        db.query(BackupRecord)
+        .filter(BackupRecord.backup_type == backup_type)
+        .order_by(BackupRecord.created_at.desc())
+        .all()
+    )
+
+    to_prune = records[retain:]
+    for rec in to_prune:
+        if rec.local_path and os.path.exists(rec.local_path):
+            try:
+                os.remove(rec.local_path)
+                logger.info(f"Pruned local backup: {rec.local_path}")
+            except OSError as exc:
+                logger.warning(f"Failed to remove local backup {rec.local_path}: {exc}")
+        rec.local_path = None
+        # If no remote copy either, delete the record entirely
+        if not rec.remote_path:
+            db.delete(rec)
+
+    db.commit()
+
+
+def _prune_remote_backups(backup_type: str, db: object) -> None:
+    """Prune remote backup records beyond the retention limit.
+
+    The actual remote deletion is best-effort (logged but not fatal).
+
+    Args:
+        backup_type: One of ``hourly``, ``daily``, ``weekly``.
+        db: Active SQLAlchemy session.
+    """
+    retain_attr = _BACKUP_TYPE_RETAIN.get(backup_type, "backup_retain_hourly")
+    retain = int(getattr(settings, retain_attr, 96))
+
+    # Query ALL records for this tier so that already-pruned local records
+    # still count toward the retention window.
+    records = (
+        db.query(BackupRecord)
+        .filter(BackupRecord.backup_type == backup_type)
+        .order_by(BackupRecord.created_at.desc())
+        .all()
+    )
+
+    to_prune = [r for r in records[retain:] if r.remote_path]
+    for rec in to_prune:
+        _delete_remote_copy(rec)
+        rec.remote_path = None
+        rec.remote_destination = None
+        if not rec.local_path:
+            db.delete(rec)
+
+    db.commit()
+
+
+def _delete_remote_copy(rec: BackupRecord) -> None:  # noqa: C901
+    """Best-effort deletion of the remote copy described by *rec*."""
+    dest = rec.remote_destination
+    remote_path = rec.remote_path
+    if not dest or not remote_path:
+        return
+
+    try:
+        if dest == "s3":
+            import boto3
+
+            s3 = boto3.client(
+                "s3",
+                region_name=settings.aws_region,
+                aws_access_key_id=settings.aws_access_key_id,
+                aws_secret_access_key=settings.aws_secret_access_key,
+            )
+            s3.delete_object(Bucket=settings.s3_bucket_name, Key=remote_path)
+            logger.info(f"Deleted remote S3 backup: s3://{settings.s3_bucket_name}/{remote_path}")
+
+        elif dest == "dropbox":
+            import dropbox as dbx_module
+
+            dbx = dbx_module.Dropbox(settings.dropbox_refresh_token)
+            dbx.files_delete_v2(remote_path)
+            logger.info(f"Deleted remote Dropbox backup: {remote_path}")
+
+        elif dest in ("ftp", "sftp", "nextcloud", "webdav", "google_drive", "onedrive", "email"):
+            # For other providers best-effort is logged only – deletion not implemented yet.
+            logger.debug(f"Remote deletion not implemented for destination '{dest}', skipping {remote_path}")
+
+    except Exception as exc:
+        logger.warning(f"Failed to delete remote backup {remote_path} from {dest}: {exc}")
+
+
+def _upload_remote(archive_path: Path, filename: str) -> tuple[str, str] | None:  # noqa: C901
+    """Upload *archive_path* to the configured remote destination.
+
+    Returns:
+        ``(destination, remote_path)`` on success, ``None`` on failure or when
+        no remote destination is configured.
+    """
+    dest = getattr(settings, "backup_remote_destination", None)
+    if not dest:
+        return None
+
+    remote_folder = getattr(settings, "backup_remote_folder", "backups") or "backups"
+    remote_key = f"{remote_folder}/{filename}"
+
+    try:
+        if dest == "s3":
+            import boto3
+
+            s3 = boto3.client(
+                "s3",
+                region_name=settings.aws_region,
+                aws_access_key_id=settings.aws_access_key_id,
+                aws_secret_access_key=settings.aws_secret_access_key,
+            )
+            with open(archive_path, "rb") as fh:
+                s3.upload_fileobj(fh, settings.s3_bucket_name, remote_key)
+            logger.info(f"Uploaded backup to S3: s3://{settings.s3_bucket_name}/{remote_key}")
+            return (dest, remote_key)
+
+        elif dest == "dropbox":
+            import dropbox as dbx_module
+
+            dbx = dbx_module.Dropbox(settings.dropbox_refresh_token)
+            dropbox_path = f"/{remote_key}"
+            with open(archive_path, "rb") as fh:
+                dbx.files_upload(fh.read(), dropbox_path, mode=dbx_module.files.WriteMode("overwrite"))
+            logger.info(f"Uploaded backup to Dropbox: {dropbox_path}")
+            return (dest, dropbox_path)
+
+        elif dest == "email":
+            _email_backup(archive_path, filename)
+            return (dest, f"email:{filename}")
+
+        elif dest == "nextcloud":
+            import requests
+
+            url = f"{settings.nextcloud_upload_url}/{remote_key}"
+            with open(archive_path, "rb") as fh:
+                resp = requests.put(
+                    url,
+                    data=fh,
+                    auth=(settings.nextcloud_username, settings.nextcloud_password),
+                    timeout=120,
+                )
+            resp.raise_for_status()
+            logger.info(f"Uploaded backup to Nextcloud: {url}")
+            return (dest, url)
+
+        elif dest == "webdav":
+            import requests
+
+            url = f"{settings.webdav_url}/{remote_key}"
+            with open(archive_path, "rb") as fh:
+                resp = requests.put(
+                    url,
+                    data=fh,
+                    auth=(settings.webdav_username, settings.webdav_password),
+                    verify=settings.webdav_verify_ssl,
+                    timeout=120,
+                )
+            resp.raise_for_status()
+            logger.info(f"Uploaded backup to WebDAV: {url}")
+            return (dest, url)
+
+        else:
+            logger.warning(f"Backup remote destination '{dest}' upload not implemented; keeping local only.")
+            return None
+
+    except Exception as exc:
+        logger.error(f"Failed to upload backup to {dest}: {exc}", exc_info=True)
+        return None
+
+
+def _email_backup(archive_path: Path, filename: str) -> None:
+    """Send *archive_path* as an e-mail attachment to the default recipient."""
+    import smtplib
+    from email.mime.application import MIMEApplication
+    from email.mime.multipart import MIMEMultipart
+    from email.mime.text import MIMEText
+
+    recipient = settings.email_default_recipient
+    if not recipient:
+        raise ValueError("email_default_recipient is not configured")
+
+    msg = MIMEMultipart()
+    msg["Subject"] = f"[DocuElevate] Database backup – {filename}"
+    msg["From"] = settings.email_sender or settings.email_username or "docuelevate@localhost"
+    msg["To"] = recipient
+
+    body = MIMEText(f"Automated database backup from DocuElevate.\n\nFile: {filename}\n", "plain")
+    msg.attach(body)
+
+    with open(archive_path, "rb") as fh:
+        part = MIMEApplication(fh.read(), Name=filename)
+        part["Content-Disposition"] = f'attachment; filename="{filename}"'
+        msg.attach(part)
+
+    with smtplib.SMTP(settings.email_host, settings.email_port, timeout=60) as server:
+        if settings.email_use_tls:
+            server.starttls()
+        if settings.email_username and settings.email_password:
+            server.login(settings.email_username, settings.email_password)
+        server.sendmail(msg["From"], [recipient], msg.as_string())
+
+    logger.info(f"Backup e-mailed to {recipient}: {filename}")
+
+
+# ---------------------------------------------------------------------------
+# Public Celery tasks
+# ---------------------------------------------------------------------------
+
+
+@celery.task(name="app.tasks.backup_tasks.create_backup", bind=True)
+def create_backup(self, backup_type: str = "hourly") -> dict:
+    """Create a database backup archive and apply retention.
+
+    Supports SQLite (``.db.gz``), PostgreSQL (``.pgsql.gz``), and
+    MySQL / MariaDB (``.mysql.gz``) databases.  The native dump tool for the
+    configured backend (``sqlite3``, ``pg_dump``, or ``mysqldump``) must be
+    available on the worker's ``PATH``.
+
+    Args:
+        backup_type: ``"hourly"``, ``"daily"``, or ``"weekly"``.
+
+    Returns:
+        A dict with ``filename``, ``size_bytes``, and ``status``.
+    """
+    if backup_type not in _BACKUP_TYPE_RETAIN:
+        backup_type = "hourly"
+
+    if not getattr(settings, "backup_enabled", True):
+        logger.debug("Backup is disabled; skipping create_backup task.")
+        return {"status": "disabled"}
+
+    backend = _db_backend()
+    ext = _archive_ext_for_backend(backend)
+
+    ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H-%M-%S")
+    filename = f"backup_{backup_type}_{ts}{ext}"
+    archive_path = _backup_dir() / filename
+
+    # SQLite: verify the database file exists before attempting to dump it
+    db_path: Path | None = None
+    if backend == "sqlite":
+        db_path = _db_path()
+        if db_path is None:
+            logger.warning("Backup task skipped: in-memory SQLite databases are not supported.")
+            return {"status": "unsupported_db"}
+        if not db_path.exists():
+            logger.error(f"Database file not found: {db_path}")
+            return {"status": "error", "detail": f"DB file missing: {db_path}"}
+    elif backend not in ("postgresql", "mysql"):
+        logger.warning(f"Backup task skipped: unsupported database backend '{backend}'.")
+        return {"status": "unsupported_db"}
+
+    status = "ok"
+    checksum: str | None = None
+    size_bytes = 0
+    remote_destination: str | None = None
+    remote_path: str | None = None
+
+    try:
+        if backend == "sqlite":
+            # db_path is guaranteed non-None: we returned early if it were None
+            if db_path is None:  # pragma: no cover
+                return {"status": "error", "detail": "db_path unexpectedly None"}
+            _dump_sqlite(db_path, archive_path)
+        elif backend == "postgresql":
+            _dump_postgresql(settings.database_url, archive_path)
+        elif backend == "mysql":
+            _dump_mysql(settings.database_url, archive_path)
+        size_bytes = archive_path.stat().st_size
+        checksum = _sha256(archive_path)
+        logger.info(f"Created {backup_type} backup: {archive_path} ({size_bytes:,} bytes)")
+    except Exception as exc:
+        logger.error(f"Failed to create backup archive {filename}: {exc}", exc_info=True)
+        status = "failed"
+        # Record the failure so it is visible in the dashboard
+        with SessionLocal() as db:
+            rec = BackupRecord(
+                filename=filename,
+                local_path=None,
+                backup_type=backup_type,
+                size_bytes=0,
+                checksum=None,
+                status="failed",
+            )
+            db.add(rec)
+            db.commit()
+        return {"status": "error", "detail": str(exc)}
+
+    # Optional remote upload
+    result = _upload_remote(archive_path, filename)
+    if result:
+        remote_destination, remote_path = result
+
+    with SessionLocal() as db:
+        rec = BackupRecord(
+            filename=filename,
+            local_path=str(archive_path),
+            backup_type=backup_type,
+            size_bytes=size_bytes,
+            checksum=checksum,
+            status=status,
+            remote_destination=remote_destination,
+            remote_path=remote_path,
+        )
+        db.add(rec)
+        db.commit()
+
+        # Apply retention policy for this tier
+        _apply_retention(backup_type, db)
+        if remote_destination:
+            _prune_remote_backups(backup_type, db)
+
+    return {
+        "filename": filename,
+        "size_bytes": size_bytes,
+        "status": status,
+        "remote_destination": remote_destination,
+    }
+
+
+@celery.task(name="app.tasks.backup_tasks.cleanup_old_backups")
+def cleanup_old_backups() -> dict:
+    """Manually trigger retention clean-up for all backup tiers.
+
+    This is also called automatically after each ``create_backup`` run.
+    """
+    with SessionLocal() as db:
+        for btype in ("hourly", "daily", "weekly"):
+            _apply_retention(btype, db)
+            _prune_remote_backups(btype, db)
+    return {"status": "ok"}

app/tasks/compute_embedding.py

@@ -0,0 +1,174 @@
+"""Celery task for pre-computing document text embeddings.
+
+Runs after document processing to ensure embeddings are available for
+the similarity feature without requiring a user to trigger them on first
+access.
+"""
+
+import logging
+from datetime import datetime, timezone
+
+from app.celery_app import celery
+from app.config import settings
+from app.database import SessionLocal
+from app.models import FileRecord
+from app.tasks.retry_config import BaseTaskWithRetry
+from app.utils import log_task_progress
+from app.utils.step_manager import update_step_status
+
+logger = logging.getLogger(__name__)
+
+
+@celery.task(base=BaseTaskWithRetry, bind=True, name="compute_document_embedding")
+def compute_document_embedding(self, file_id: int) -> dict:
+    """Compute and cache the text embedding for a single document.
+
+    Skips silently when the file has no OCR text or already has a cached
+    embedding.  The result is stored in ``FileRecord.embedding`` for
+    subsequent similarity queries.
+
+    Args:
+        file_id: Primary key of the :class:`~app.models.FileRecord`.
+
+    Returns:
+        A dict with ``status`` (``"success"`` / ``"skipped"`` / ``"error"``)
+        and optional ``detail`` message.
+    """
+    task_id = self.request.id
+    logger.info("[%s] Computing embedding for file %s", task_id, file_id)
+    log_task_progress(
+        task_id,
+        "compute_embedding",
+        "in_progress",
+        f"Computing text embedding for file {file_id}",
+        file_id=file_id,
+    )
+
+    with SessionLocal() as db:
+        file_record = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+        if not file_record:
+            logger.warning("[%s] File %s not found, skipping embedding", task_id, file_id)
+            return {"status": "skipped", "detail": "File not found"}
+
+        now = datetime.now(timezone.utc)
+        update_step_status(db, file_id, "compute_embedding", "in_progress", started_at=now)
+
+        # Already has a cached embedding – nothing to do
+        if file_record.embedding:
+            logger.info("[%s] File %s already has a cached embedding", task_id, file_id)
+            log_task_progress(
+                task_id,
+                "compute_embedding",
+                "success",
+                "Embedding already cached",
+                file_id=file_id,
+            )
+            update_step_status(db, file_id, "compute_embedding", "success", completed_at=now)
+            return {"status": "skipped", "detail": "Embedding already cached"}
+
+        if not file_record.ocr_text or not file_record.ocr_text.strip():
+            logger.info("[%s] File %s has no OCR text, skipping embedding", task_id, file_id)
+            log_task_progress(
+                task_id,
+                "compute_embedding",
+                "skipped",
+                "No OCR text available",
+                file_id=file_id,
+            )
+            update_step_status(db, file_id, "compute_embedding", "skipped", completed_at=now)
+            return {"status": "skipped", "detail": "No OCR text available"}
+
+        try:
+            from app.utils.similarity import compute_and_store_embedding
+
+            embedding = compute_and_store_embedding(db, file_record)
+            completed = datetime.now(timezone.utc)
+            if embedding:
+                log_task_progress(
+                    task_id,
+                    "compute_embedding",
+                    "success",
+                    f"Embedding computed ({len(embedding)} dimensions)",
+                    file_id=file_id,
+                )
+                update_step_status(db, file_id, "compute_embedding", "success", completed_at=completed)
+                return {
+                    "status": "success",
+                    "detail": f"Embedding computed ({len(embedding)} dimensions)",
+                }
+            else:
+                log_task_progress(
+                    task_id,
+                    "compute_embedding",
+                    "failure",
+                    "Embedding computation returned None",
+                    file_id=file_id,
+                )
+                update_step_status(
+                    db,
+                    file_id,
+                    "compute_embedding",
+                    "failure",
+                    error_message="Embedding computation returned None",
+                    completed_at=completed,
+                )
+                return {"status": "error", "detail": "Embedding computation returned None"}
+        except Exception as exc:
+            logger.exception("[%s] Embedding computation failed for file %s: %s", task_id, file_id, exc)
+            log_task_progress(
+                task_id,
+                "compute_embedding",
+                "failure",
+                f"Exception: {exc}",
+                file_id=file_id,
+            )
+            update_step_status(
+                db,
+                file_id,
+                "compute_embedding",
+                "failure",
+                error_message=str(exc),
+                completed_at=datetime.now(timezone.utc),
+            )
+            return {"status": "error", "detail": str(exc)}
+
+
+@celery.task(bind=True, name="backfill_missing_embeddings")
+def backfill_missing_embeddings(self) -> dict:
+    """Periodic task that computes embeddings for documents that lack them.
+
+    Iterates over all ``FileRecord`` rows that have OCR text but no
+    cached embedding and queues a :func:`compute_document_embedding`
+    task for each one.  A configurable ``batch_size`` caps the number
+    of tasks queued per run to avoid overwhelming the worker or the
+    embedding API.
+
+    Returns:
+        A dict with the number of tasks ``queued``.
+    """
+    batch_size = settings.embedding_backfill_batch_size
+    task_id = self.request.id
+    logger.info("[%s] Backfill: scanning for files missing embeddings (batch_size=%d)", task_id, batch_size)
+
+    with SessionLocal() as db:
+        candidates = (
+            db.query(FileRecord.id)
+            .filter(
+                FileRecord.ocr_text.isnot(None),
+                FileRecord.ocr_text != "",
+                (FileRecord.embedding.is_(None)) | (FileRecord.embedding == ""),
+            )
+            .limit(batch_size)
+            .all()
+        )
+
+    queued = 0
+    for (file_id,) in candidates:
+        try:
+            compute_document_embedding.delay(file_id)
+            queued += 1
+        except Exception as exc:
+            logger.warning("[%s] Could not queue embedding for file %s: %s", task_id, file_id, exc)
+
+    logger.info("[%s] Backfill: queued %d embedding tasks", task_id, queued)
+    return {"queued": queued}

app/tasks/convert_to_pdf.py

@@ -124,7 +124,9 @@ def _build_filename(file_path: str, original_filename: Optional[str], file_ext:
 
 
 @shared_task(bind=True)
-def convert_to_pdf(self, file_path: str, original_filename: Optional[str] = None) -> Optional[str]:
+def convert_to_pdf(
+    self, file_path: str, original_filename: Optional[str] = None, owner_id: Optional[str] = None
+) -> Optional[str]:
     """
     Converts a file to PDF using Gotenberg's API.
     Determines the appropriate Gotenberg endpoint based on the file's MIME type.
@@ -133,6 +135,7 @@ def convert_to_pdf(self, file_path: str, original_filename: Optional[str] = None
     Args:
         file_path: Path to the file to convert
         original_filename: Optional original filename (if different from path basename)
+        owner_id: Optional user identifier forwarded to process_document for multi-user mode.
     """
     task_id = self.request.id
     logger.info(f"[{task_id}] Starting PDF conversion: {file_path}")
@@ -332,9 +335,9 @@ def convert_to_pdf(self, file_path: str, original_filename: Optional[str] = None
                 # Change extension to .pdf for the original filename
                 original_base = os.path.splitext(original_filename)[0]
                 pdf_original_filename = f"{original_base}.pdf"
-                process_document.delay(converted_file_path, original_filename=pdf_original_filename)
+                process_document.delay(converted_file_path, original_filename=pdf_original_filename, owner_id=owner_id)
             else:
-                process_document.delay(converted_file_path)
+                process_document.delay(converted_file_path, owner_id=owner_id)
 
             return converted_file_path
         else:

app/tasks/convert_to_pdfa.py

@@ -0,0 +1,453 @@
+"""PDF/A archival conversion task.
+
+Converts PDF files to PDF/A format using ocrmypdf (which relies on Ghostscript
+internally).  Two variants are produced when enabled:
+
+1. **Original PDF/A** – an archival copy of the ingested file, providing a
+   time-stamped record of the document as it was upon ingestion.
+2. **Processed PDF/A** – an archival copy of the processed file with embedded
+   metadata.
+
+Both are saved under ``workdir/pdfa/`` and referenced in the database via
+``FileRecord.original_pdfa_path`` and ``FileRecord.processed_pdfa_path``.
+
+When ``PDFA_TIMESTAMP_ENABLED`` is True, each PDF/A file also gets an RFC 3161
+timestamp response (``.tsr``) from a configurable Timestamp Authority (default:
+FreeTSA).  This provides cryptographic proof of the file's existence at a given
+point in time.
+
+.. note::
+
+   PDF/A conversion may alter font rendering (especially OCR text overlays
+   produced by Microsoft Azure Document Intelligence).  This is expected –
+   the PDF/A copies are parallel archival variants, not replacements.
+"""
+
+import logging
+import os
+import shutil
+import subprocess
+
+import requests as http_requests
+
+from app.celery_app import celery
+from app.config import settings
+from app.database import SessionLocal
+from app.models import FileRecord
+from app.tasks.retry_config import BaseTaskWithRetry
+from app.utils import get_unique_filepath_with_counter, log_task_progress
+
+logger = logging.getLogger(__name__)
+
+# Subdirectory structure under workdir for PDF/A copies
+PDFA_ORIGINAL_SUBDIR = os.path.join("pdfa", "original")
+PDFA_PROCESSED_SUBDIR = os.path.join("pdfa", "processed")
+
+
+def _convert_pdf_to_pdfa(input_path: str, output_path: str, pdfa_format: str = "2") -> bool:
+    """Convert a PDF file to PDF/A using ocrmypdf.
+
+    Uses ``ocrmypdf --skip-text --output-type pdfa-N`` so that existing text
+    layers are preserved (not re-OCR'd) while the output is converted to
+    PDF/A via Ghostscript.
+
+    Args:
+        input_path: Absolute path to the source PDF file.
+        output_path: Absolute path for the PDF/A output file.
+        pdfa_format: PDF/A variant ('1', '2', or '3'). Defaults to '2' for PDF/A-2b.
+
+    Returns:
+        True if conversion succeeded, False otherwise.
+    """
+    # Validate format to prevent argument injection via output-type
+    if pdfa_format not in ("1", "2", "3"):
+        logger.error(f"[convert_to_pdfa] Invalid pdfa_format: {pdfa_format}")
+        return False
+
+    ocrmypdf_bin = shutil.which("ocrmypdf")
+    if not ocrmypdf_bin:
+        logger.error("[convert_to_pdfa] ocrmypdf binary not found on PATH")
+        return False
+
+    output_type = f"pdfa-{pdfa_format}"
+
+    cmd = [
+        ocrmypdf_bin,
+        "--skip-text",
+        "--output-type",
+        output_type,
+        "--quiet",
+        "--invalidate-digital-signatures",
+        input_path,
+        output_path,
+    ]
+
+    logger.info(f"[convert_to_pdfa] Running: {' '.join(cmd)}")
+
+    try:
+        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=600, check=False)  # noqa: S603
+    except subprocess.TimeoutExpired:
+        logger.warning("[convert_to_pdfa] ocrmypdf timed out after 600s")
+        return False
+
+    if proc.returncode != 0:
+        stderr_snippet = proc.stderr.strip()[:500] if proc.stderr else ""
+        logger.warning(f"[convert_to_pdfa] ocrmypdf exited with code {proc.returncode}: {stderr_snippet}")
+        return False
+
+    logger.info(f"[convert_to_pdfa] PDF/A file written to {output_path}")
+    return True
+
+
+def _timestamp_file(file_path: str, tsa_url: str) -> str | None:
+    """Create an RFC 3161 timestamp for a file using a Timestamp Authority.
+
+    Uses ``openssl ts`` to create a timestamp request (TSQ) from the file's
+    SHA-256 hash, submits it to the TSA via HTTP POST, and saves the timestamp
+    response (TSR) alongside the file.
+
+    Args:
+        file_path: Absolute path to the file to timestamp.
+        tsa_url: URL of the RFC 3161 Timestamp Authority.
+
+    Returns:
+        Path to the ``.tsr`` file if successful, None otherwise.
+    """
+    openssl_bin = shutil.which("openssl")
+    if not openssl_bin:
+        logger.error("[timestamp] openssl binary not found on PATH")
+        return None
+
+    tsr_path = file_path + ".tsr"
+    tsq_path = file_path + ".tsq"
+
+    try:
+        # Step 1: Create timestamp request
+        cmd = [openssl_bin, "ts", "-query", "-data", file_path, "-sha256", "-no_nonce", "-out", tsq_path]
+        proc = subprocess.run(cmd, capture_output=True, text=True, timeout=30, check=False)  # noqa: S603
+        if proc.returncode != 0:
+            logger.warning(f"[timestamp] openssl ts -query failed: {proc.stderr.strip()[:200]}")
+            return None
+
+        # Step 2: Submit TSQ to the Timestamp Authority
+        with open(tsq_path, "rb") as f:
+            tsq_data = f.read()
+
+        response = http_requests.post(
+            tsa_url,
+            data=tsq_data,
+            headers={"Content-Type": "application/timestamp-query"},
+            timeout=30,
+        )
+
+        if response.status_code != 200:
+            logger.warning(f"[timestamp] TSA returned HTTP {response.status_code} from {tsa_url}")
+            return None
+
+        # Step 3: Save the timestamp response
+        with open(tsr_path, "wb") as f:
+            f.write(response.content)
+
+        logger.info(f"[timestamp] RFC 3161 timestamp saved to {tsr_path}")
+        return tsr_path
+
+    except http_requests.RequestException as e:
+        logger.warning(f"[timestamp] Failed to contact TSA at {tsa_url}: {e}")
+        return None
+    except subprocess.TimeoutExpired:
+        logger.warning("[timestamp] openssl ts timed out")
+        return None
+    finally:
+        # Always clean up the TSQ file
+        if os.path.exists(tsq_path):
+            os.remove(tsq_path)
+
+
+def _compute_pdfa_folder_overrides() -> dict[str, str]:
+    """Compute per-provider folder overrides for PDF/A uploads.
+
+    Appends ``settings.pdfa_upload_folder`` to each provider's configured
+    folder.  For Google Drive (which uses folder IDs), uses the dedicated
+    ``google_drive_pdfa_folder_id`` setting.
+
+    Returns:
+        Dictionary mapping provider names to folder override strings.
+    """
+    subfolder = settings.pdfa_upload_folder
+    overrides: dict[str, str] = {}
+
+    if not subfolder:
+        return overrides
+
+    # Path-based providers: append subfolder
+    for provider, folder_attr in [
+        ("dropbox", "dropbox_folder"),
+        ("nextcloud", "nextcloud_folder"),
+        ("webdav", "webdav_folder"),
+        ("ftp", "ftp_folder"),
+        ("sftp", "sftp_folder"),
+        ("onedrive", "onedrive_folder_path"),
+    ]:
+        base = getattr(settings, folder_attr, "") or ""
+        overrides[provider] = f"{base.rstrip('/')}/{subfolder}" if base else subfolder
+
+    # S3: append subfolder to prefix (trailing slash is required by S3 convention
+    # where "folder" paths are key prefixes, unlike path-based providers above)
+    s3_prefix = getattr(settings, "s3_folder_prefix", "") or ""
+    overrides["s3"] = f"{s3_prefix.rstrip('/')}/{subfolder}/"
+
+    # Google Drive: use dedicated folder ID or fall back to default
+    gdrive_pdfa_id = settings.google_drive_pdfa_folder_id
+    if gdrive_pdfa_id:
+        overrides["google_drive"] = gdrive_pdfa_id
+
+    return overrides
+
+
+@celery.task(base=BaseTaskWithRetry, bind=True)
+def convert_to_pdfa(self, file_id: int) -> dict:
+    """Generate PDF/A archival copies for a processed document.
+
+    Creates PDF/A variants of both the original ingested file and the
+    processed file (with embedded metadata).  Files are saved under
+    ``workdir/pdfa/original/`` and ``workdir/pdfa/processed/`` respectively.
+
+    When timestamping is enabled, each PDF/A file also gets an RFC 3161
+    ``.tsr`` timestamp from the configured TSA.
+
+    Upload of each variant to storage providers is controlled independently
+    by ``pdfa_upload_original`` and ``pdfa_upload_processed``.
+
+    Args:
+        file_id: ID of the FileRecord to create PDF/A copies for.
+
+    Returns:
+        Dictionary with status and file paths.
+    """
+    task_id = self.request.id
+    logger.info(f"[{task_id}] Starting PDF/A conversion for file_id={file_id}")
+    log_task_progress(
+        task_id,
+        "convert_to_pdfa",
+        "in_progress",
+        "Starting PDF/A archival conversion",
+        file_id=file_id,
+    )
+
+    # Fetch file record
+    with SessionLocal() as db:
+        file_record = db.query(FileRecord).filter_by(id=file_id).first()
+        if not file_record:
+            logger.error(f"[{task_id}] FileRecord {file_id} not found")
+            log_task_progress(task_id, "convert_to_pdfa", "failure", "File record not found", file_id=file_id)
+            return {"error": "File record not found", "file_id": file_id}
+
+        original_path = file_record.original_file_path
+        processed_path = file_record.processed_file_path
+
+    pdfa_format = settings.pdfa_format
+    timestamp_enabled = settings.pdfa_timestamp_enabled
+    timestamp_url = settings.pdfa_timestamp_url
+    results = {}
+
+    # --- Convert original file to PDF/A ---
+    if original_path and os.path.exists(original_path):
+        original_pdfa_dir = os.path.join(settings.workdir, PDFA_ORIGINAL_SUBDIR)
+        os.makedirs(original_pdfa_dir, exist_ok=True)
+
+        base_name = os.path.splitext(os.path.basename(original_path))[0]
+        original_pdfa_path = get_unique_filepath_with_counter(original_pdfa_dir, base_name, ".pdf")
+
+        logger.info(f"[{task_id}] Converting original to PDF/A: {original_path} -> {original_pdfa_path}")
+        log_task_progress(
+            task_id,
+            "convert_original_to_pdfa",
+            "in_progress",
+            f"Converting original to PDF/A: {os.path.basename(original_path)}",
+            file_id=file_id,
+        )
+
+        success = _convert_pdf_to_pdfa(original_path, original_pdfa_path, pdfa_format)
+        if success:
+            results["original_pdfa_path"] = original_pdfa_path
+            log_task_progress(
+                task_id,
+                "convert_original_to_pdfa",
+                "success",
+                f"Original PDF/A saved: {os.path.basename(original_pdfa_path)}",
+                file_id=file_id,
+            )
+            # Timestamp the original PDF/A
+            if timestamp_enabled:
+                tsr = _timestamp_file(original_pdfa_path, timestamp_url)
+                if tsr:
+                    log_task_progress(
+                        task_id,
+                        "timestamp_original_pdfa",
+                        "success",
+                        f"Timestamped: {os.path.basename(tsr)}",
+                        file_id=file_id,
+                    )
+                else:
+                    log_task_progress(
+                        task_id,
+                        "timestamp_original_pdfa",
+                        "failure",
+                        "Failed to timestamp original PDF/A",
+                        file_id=file_id,
+                    )
+        else:
+            log_task_progress(
+                task_id,
+                "convert_original_to_pdfa",
+                "failure",
+                "Failed to convert original to PDF/A",
+                file_id=file_id,
+            )
+    else:
+        logger.warning(f"[{task_id}] Original file not found, skipping original PDF/A conversion")
+        log_task_progress(
+            task_id,
+            "convert_original_to_pdfa",
+            "skipped",
+            "Original file not available",
+            file_id=file_id,
+        )
+
+    # --- Convert processed file to PDF/A ---
+    if processed_path and os.path.exists(processed_path):
+        processed_pdfa_dir = os.path.join(settings.workdir, PDFA_PROCESSED_SUBDIR)
+        os.makedirs(processed_pdfa_dir, exist_ok=True)
+
+        base_name = os.path.splitext(os.path.basename(processed_path))[0]
+        processed_pdfa_path = get_unique_filepath_with_counter(processed_pdfa_dir, f"{base_name}-PDFA", ".pdf")
+
+        logger.info(f"[{task_id}] Converting processed to PDF/A: {processed_path} -> {processed_pdfa_path}")
+        log_task_progress(
+            task_id,
+            "convert_processed_to_pdfa",
+            "in_progress",
+            f"Converting processed to PDF/A: {os.path.basename(processed_path)}",
+            file_id=file_id,
+        )
+
+        success = _convert_pdf_to_pdfa(processed_path, processed_pdfa_path, pdfa_format)
+        if success:
+            results["processed_pdfa_path"] = processed_pdfa_path
+            log_task_progress(
+                task_id,
+                "convert_processed_to_pdfa",
+                "success",
+                f"Processed PDF/A saved: {os.path.basename(processed_pdfa_path)}",
+                file_id=file_id,
+            )
+            # Timestamp the processed PDF/A
+            if timestamp_enabled:
+                tsr = _timestamp_file(processed_pdfa_path, timestamp_url)
+                if tsr:
+                    log_task_progress(
+                        task_id,
+                        "timestamp_processed_pdfa",
+                        "success",
+                        f"Timestamped: {os.path.basename(tsr)}",
+                        file_id=file_id,
+                    )
+                else:
+                    log_task_progress(
+                        task_id,
+                        "timestamp_processed_pdfa",
+                        "failure",
+                        "Failed to timestamp processed PDF/A",
+                        file_id=file_id,
+                    )
+        else:
+            log_task_progress(
+                task_id,
+                "convert_processed_to_pdfa",
+                "failure",
+                "Failed to convert processed to PDF/A",
+                file_id=file_id,
+            )
+    else:
+        logger.warning(f"[{task_id}] Processed file not found, skipping processed PDF/A conversion")
+        log_task_progress(
+            task_id,
+            "convert_processed_to_pdfa",
+            "skipped",
+            "Processed file not available",
+            file_id=file_id,
+        )
+
+    # --- Update database with PDF/A paths ---
+    with SessionLocal() as db:
+        file_record = db.query(FileRecord).filter_by(id=file_id).first()
+        if file_record:
+            if "original_pdfa_path" in results:
+                file_record.original_pdfa_path = results["original_pdfa_path"]
+            if "processed_pdfa_path" in results:
+                file_record.processed_pdfa_path = results["processed_pdfa_path"]
+            db.commit()
+            logger.info(f"[{task_id}] Updated database with PDF/A paths")
+
+    # --- Upload PDF/A variants to storage providers ---
+    folder_overrides = _compute_pdfa_folder_overrides()
+
+    if settings.pdfa_upload_original and "original_pdfa_path" in results:
+        from app.tasks.send_to_all import send_to_all_destinations
+
+        logger.info(f"[{task_id}] Uploading original PDF/A to storage providers")
+        log_task_progress(
+            task_id,
+            "upload_original_pdfa",
+            "in_progress",
+            "Uploading original PDF/A to storage providers",
+            file_id=file_id,
+        )
+        send_to_all_destinations.delay(
+            results["original_pdfa_path"],
+            True,
+            file_id,
+            folder_overrides=folder_overrides if folder_overrides else None,
+        )
+        log_task_progress(
+            task_id,
+            "upload_original_pdfa",
+            "success",
+            "Original PDF/A queued for upload",
+            file_id=file_id,
+        )
+
+    if settings.pdfa_upload_processed and "processed_pdfa_path" in results:
+        from app.tasks.send_to_all import send_to_all_destinations
+
+        logger.info(f"[{task_id}] Uploading processed PDF/A to storage providers")
+        log_task_progress(
+            task_id,
+            "upload_processed_pdfa",
+            "in_progress",
+            "Uploading processed PDF/A to storage providers",
+            file_id=file_id,
+        )
+        send_to_all_destinations.delay(
+            results["processed_pdfa_path"],
+            True,
+            file_id,
+            folder_overrides=folder_overrides if folder_overrides else None,
+        )
+        log_task_progress(
+            task_id,
+            "upload_processed_pdfa",
+            "success",
+            "Processed PDF/A queued for upload",
+            file_id=file_id,
+        )
+
+    # --- Final status ---
+    has_any = bool(results)
+    status = "success" if has_any else "failure"
+    message = (
+        f"PDF/A conversion complete ({len(results)} variant(s) created)" if has_any else "No PDF/A variants created"
+    )
+    log_task_progress(task_id, "convert_to_pdfa", status, message, file_id=file_id)
+
+    return {"status": status, "file_id": file_id, **results}

app/tasks/finalize_document_storage.py

@@ -32,7 +32,7 @@ def finalize_document_storage(self, original_file: str, processed_file: str, met
     task_id = self.request.id
     logger.info(f"[{task_id}] Finalizing document storage for {processed_file}")
 
-    # 1. Update Database Status (From Main)
+    # 1. Update Database Status
     log_task_progress(
         task_id,
         "finalize_document_storage",
@@ -41,7 +41,7 @@ def finalize_document_storage(self, original_file: str, processed_file: str, met
         file_id=file_id,
     )
 
-    # Get file_id from database if not provided (fallback logic from Main)
+    # Get file_id from database if not provided (fallback logic)
     if file_id is None:
         with SessionLocal() as db:
             # Only as a last resort, try to find by exact match on local_filename
@@ -50,7 +50,7 @@ def finalize_document_storage(self, original_file: str, processed_file: str, met
             if file_record:
                 file_id = file_record.id
 
-    # 2. Determine Configured Destinations (From Copilot)
+    # 2. Determine Configured Destinations
     # This is needed for the notification message later
     configured_destinations = []
     try:
@@ -65,20 +65,36 @@ def finalize_document_storage(self, original_file: str, processed_file: str, met
         logger.warning(f"[WARNING] Could not determine configured destinations: {e}")
         configured_destinations = ["configured destinations"]
 
-    # 3. Queue Uploads (Merged)
-    # Uses Main branch signature to ensure file_id is passed, but keeps logic structure
+    # 3. Queue Uploads
     logger.info(f"[{task_id}] Queueing uploads to all destinations")
     log_task_progress(
         task_id, "finalize_document_storage", "success", "Queuing uploads to destinations", file_id=file_id
     )
 
     # Note: send_to_all_destinations is asynchronous and queues upload tasks
-    # We pass 'True' (delete_after) and 'file_id' as per Main branch requirements
     send_to_all_destinations.delay(processed_file, True, file_id)
 
-    # 4. Send Notification (From Copilot)
-    # Note: This notification is sent after processing is complete but while uploads
-    # are being queued.
+    # 3a. Trigger PDF/A archival conversion if enabled (from feature branch)
+    if settings.enable_pdfa_conversion:
+        try:
+            from app.tasks.convert_to_pdfa import convert_to_pdfa
+
+            logger.info(f"[{task_id}] PDF/A conversion enabled, queueing archival conversion")
+            convert_to_pdfa.delay(file_id)
+        except Exception as e:
+            logger.warning(f"[{task_id}] Could not queue PDF/A conversion: {e}")
+
+    # 3b. Queue embedding computation (from main branch)
+    if file_id is not None:
+        try:
+            from app.tasks.compute_embedding import compute_document_embedding
+
+            compute_document_embedding.delay(file_id)
+            logger.info(f"[{task_id}] Queued embedding computation for file {file_id}")
+        except Exception as e:
+            logger.warning(f"[{task_id}] Could not queue embedding task: {e}")
+
+    # 4. Send Notification
     try:
         # Get file information
         file_size = os.path.getsize(processed_file) if os.path.exists(processed_file) else 0

app/tasks/process_document.py

@@ -25,7 +25,12 @@ logger = logging.getLogger(__name__)
 
 @celery.task(base=BaseTaskWithRetry, bind=True)
 def process_document(
-    self, original_local_file: str, original_filename: str = None, file_id: int = None, force_cloud_ocr: bool = False
+    self,
+    original_local_file: str,
+    original_filename: str = None,
+    file_id: int = None,
+    force_cloud_ocr: bool = False,
+    owner_id: str = None,
 ):
     """
     Process a document file and trigger appropriate text extraction.
@@ -37,6 +42,8 @@ def process_document(
                  detection and reuses the existing record (used for reprocessing).
         force_cloud_ocr: If True, forces Azure Document Intelligence OCR processing
                         regardless of embedded text quality. Used for re-processing.
+        owner_id: Optional user identifier for multi-user mode. When provided, the
+                  created FileRecord is associated with this user.
 
     Steps:
       1. Check if we have a FileRecord entry (via SHA-256 hash). If found, skip re-processing.
@@ -48,6 +55,11 @@ def process_document(
          - Otherwise, queue Azure Document Intelligence processing
       3. If force_cloud_ocr is True, skip local text extraction and use cloud OCR
     """
+    # Fall back to the configured default_owner_id when no explicit owner was provided
+    default_owner_id = settings.default_owner_id
+    if owner_id is None and isinstance(default_owner_id, str) and default_owner_id.strip():
+        owner_id = default_owner_id
+
     task_id = self.request.id
     logger.info(f"[{task_id}] Starting document processing: {original_local_file}")
     log_task_progress(
@@ -141,6 +153,7 @@ def process_document(
                     mime_type=mime_type,
                     is_duplicate=True,
                     duplicate_of_id=existing.id,
+                    owner_id=owner_id,
                 )
                 db.add(duplicate_record)
                 db.commit()
@@ -191,6 +204,7 @@ def process_document(
                 file_size=file_size,
                 mime_type=mime_type,
                 is_duplicate=False,
+                owner_id=owner_id,
             )
             db.add(new_record)
             db.commit()

app/tasks/process_with_azure_document_intelligence.py

@@ -9,7 +9,7 @@ from azure.core.credentials import AzureKeyCredential
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import OcrTaskWithRetry
 from app.tasks.rotate_pdf_pages import rotate_pdf_pages
 from app.utils import log_task_progress
 
@@ -81,7 +81,7 @@ def check_page_rotation(result, filename, task_id=None):
     return rotation_data
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=OcrTaskWithRetry, bind=True)
 def process_with_azure_document_intelligence(self, filename: str, file_id: int = None):
     """
     Processes a PDF document using Azure Document Intelligence and overlays OCR text onto

app/tasks/process_with_ocr.py

@@ -23,7 +23,7 @@ from app.celery_app import celery
 from app.config import settings
 from app.database import SessionLocal
 from app.models import FileRecord
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import OcrTaskWithRetry
 from app.tasks.rotate_pdf_pages import rotate_pdf_pages
 from app.utils import log_task_progress
 from app.utils.ocr_provider import OCRResult, embed_text_layer, get_ocr_providers, merge_ocr_results
@@ -32,7 +32,7 @@ from app.utils.text_quality import TextSource, check_text_quality, compare_text_
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=OcrTaskWithRetry, bind=True)
 def process_with_ocr(self, filename: str, file_id: Optional[int] = None, original_text: Optional[str] = None):
     """Run the configured OCR providers on *filename* and continue the pipeline.
 

app/tasks/retry_config.py

@@ -1,9 +1,232 @@
 #!/usr/bin/env python3
+"""Retry configuration for Celery tasks with exponential backoff and jitter.
+
+Provides a :class:`BaseTaskWithRetry` Celery task base class that implements
+configurable retry logic with exponential backoff and optional ±20 % random
+jitter.  Pre-defined subclasses offer task-type-specific retry policies:
+
+* :class:`BaseTaskWithRetry` – general default (3 retries: 60 s, 300 s, 900 s)
+* :class:`OcrTaskWithRetry` – longer waits for OCR / AI API calls
+* :class:`UploadTaskWithRetry` – standard waits for cloud-storage uploads
+
+Usage::
+
+    from app.tasks.retry_config import BaseTaskWithRetry, OcrTaskWithRetry
+
+    @celery.task(base=OcrTaskWithRetry, bind=True)
+    def my_ocr_task(self, ...):
+        ...
+"""
+
+import logging
+import random
+from typing import Any
 
 from celery import Task
 
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Defaults
+# ---------------------------------------------------------------------------
+
+#: Default per-retry countdowns in seconds (1 min, 5 min, 15 min).
+DEFAULT_RETRY_DELAYS: list[int] = [60, 300, 900]
+
+
+def _parse_delay_string(value: str) -> list[int]:
+    """Parse a comma-separated string of integers into a list.
+
+    Args:
+        value: Comma-separated integer string, e.g. ``"60,300,900"``.
+
+    Returns:
+        Parsed list of integers, e.g. ``[60, 300, 900]``.
+    """
+    return [int(v.strip()) for v in value.split(",") if v.strip()]
+
+
+def compute_countdown(
+    retries: int,
+    base_delays: list[int] | None = None,
+    jitter: bool = True,
+) -> int:
+    """Compute the countdown in seconds for the next retry attempt.
+
+    Selects the appropriate base delay for the given retry number.  When all
+    defined delays are exhausted the last delay is doubled for each additional
+    attempt.  An optional ±20 % jitter is then applied to spread retry storms.
+
+    Args:
+        retries: Current retry count (0-based; 0 = first retry attempt).
+        base_delays: Ordered list of base countdown values (in seconds) for
+            each retry attempt.  ``None`` uses :data:`DEFAULT_RETRY_DELAYS`.
+        jitter: When ``True``, apply ±20 % random jitter to the countdown.
+
+    Returns:
+        Countdown in seconds (minimum 1 s).
+
+    Examples::
+
+        >>> compute_countdown(0, [60, 300, 900], jitter=False)
+        60
+        >>> compute_countdown(1, [60, 300, 900], jitter=False)
+        300
+        >>> compute_countdown(3, [60, 300, 900], jitter=False)  # beyond list
+        1800
+    """
+    delays = base_delays if base_delays is not None else DEFAULT_RETRY_DELAYS
+
+    if not delays:
+        base = 60
+    elif retries < len(delays):
+        base = delays[retries]
+    else:
+        # Exhausted defined delays – double the last value for each extra attempt.
+        extra = retries - len(delays) + 1
+        base = delays[-1] * (2**extra)
+
+    if jitter:
+        # ±20 % uniform jitter – not cryptographic, S311 is intentional.
+        jitter_factor = 1.0 + random.uniform(-0.2, 0.2)  # noqa: S311
+        base = int(base * jitter_factor)
+
+    return max(base, 1)
+
 
 class BaseTaskWithRetry(Task):
+    """Celery task base class with exponential backoff and optional jitter.
+
+    Automatically retries on any :class:`Exception` using delays derived from
+    :attr:`retry_delays`.  When :attr:`retry_delays` is ``None`` the value is
+    read from ``TASK_RETRY_DELAYS`` (env-var / settings); if that is also
+    unset :data:`DEFAULT_RETRY_DELAYS` (``[60, 300, 900]`` seconds) is used.
+
+    Override class attributes in subclasses to customise per-task-type policy:
+
+    * ``max_retries`` (``int``) – maximum retry attempts; default ``3``.
+    * ``retry_delays`` (``list[int] | None``) – per-retry countdowns in
+      seconds; ``None`` falls back to settings / :data:`DEFAULT_RETRY_DELAYS`.
+    * ``retry_jitter`` (``bool``) – add ±20 % jitter; default ``True``.
+    """
+
+    #: Retry on any exception raised inside the task body.
     autoretry_for = (Exception,)
-    retry_kwargs = {"max_retries": 3, "countdown": 10}  # 3 retries, 10s delay
-    retry_backoff = True  # Exponential backoff
+
+    #: Maximum number of retry attempts.
+    max_retries: int = 3
+
+    #: Pass max_retries through autoretry_for; no countdown override here
+    #: (our retry() method injects the countdown instead).
+    retry_kwargs: dict = {"max_retries": 3}
+
+    #: Per-retry countdown values (seconds).  ``None`` → settings / DEFAULT.
+    retry_delays: list[int] | None = None
+
+    #: Apply ±20 % random jitter to prevent thundering-herd problems.
+    retry_jitter: bool = True
+
+    # ------------------------------------------------------------------
+    # Public API
+    # ------------------------------------------------------------------
+
+    def retry(
+        self,
+        args: Any = None,
+        kwargs: Any = None,
+        exc: BaseException | None = None,
+        throw: bool = True,
+        eta: Any = None,
+        countdown: int | None = None,
+        max_retries: int | None = None,
+        **options: Any,
+    ) -> Any:
+        """Retry the task, injecting the backoff countdown when not supplied.
+
+        If *countdown* is not explicitly provided (and *eta* is not set) the
+        countdown is computed via :func:`compute_countdown` using this task's
+        :attr:`retry_delays` and :attr:`retry_jitter` settings.
+        """
+        if countdown is None and eta is None:
+            countdown = compute_countdown(
+                retries=self.request.retries,
+                base_delays=self._effective_retry_delays(),
+                jitter=self.retry_jitter,
+            )
+            logger.debug(
+                "Retry %d/%d for task %s in %d s",
+                self.request.retries + 1,
+                max_retries if max_retries is not None else self.max_retries,
+                self.name,
+                countdown,
+            )
+
+        return super().retry(
+            args=args,
+            kwargs=kwargs,
+            exc=exc,
+            throw=throw,
+            eta=eta,
+            countdown=countdown,
+            max_retries=max_retries,
+            **options,
+        )
+
+    # ------------------------------------------------------------------
+    # Internal helpers
+    # ------------------------------------------------------------------
+
+    def _effective_retry_delays(self) -> list[int]:
+        """Return the retry delays to use, with settings-level override support.
+
+        Priority (highest first):
+
+        1. Explicit class-level ``retry_delays`` attribute (not ``None``).
+        2. ``TASK_RETRY_DELAYS`` environment variable / setting.
+        3. :data:`DEFAULT_RETRY_DELAYS` module-level constant.
+        """
+        if self.retry_delays is not None:
+            return self.retry_delays
+
+        # Lazily read from settings to avoid circular imports at module load.
+        try:
+            from app.config import settings  # noqa: PLC0415
+
+            raw = getattr(settings, "task_retry_delays", None)
+            if raw:
+                if isinstance(raw, list):
+                    return [int(v) for v in raw]
+                if isinstance(raw, str):
+                    return _parse_delay_string(raw)
+        except Exception as exc:  # pragma: no cover
+            logger.debug("Could not read task_retry_delays from settings: %s", exc)
+
+        return DEFAULT_RETRY_DELAYS
+
+
+# ---------------------------------------------------------------------------
+# Task-type-specific retry policies
+# ---------------------------------------------------------------------------
+
+
+class OcrTaskWithRetry(BaseTaskWithRetry):
+    """Retry policy for OCR and document-intelligence API tasks.
+
+    Uses longer initial delays to allow transient API rate-limit windows to
+    clear before the next attempt.
+
+    Default: 3 retries at 120 s, 600 s, 1800 s.
+    """
+
+    retry_delays: list[int] = [120, 600, 1800]
+
+
+class UploadTaskWithRetry(BaseTaskWithRetry):
+    """Retry policy for cloud-storage upload tasks.
+
+    Uses the standard default delays (60 s, 300 s, 900 s) which are
+    appropriate for most transient upload failures (network blips, rate
+    limits, temporary service outages).
+    """
+
+    # Inherits DEFAULT_RETRY_DELAYS via retry_delays = None.

app/tasks/send_to_all.py

@@ -64,7 +64,10 @@ def _should_upload_to_sftp():
 
 def _should_upload_to_email():
     return bool(
-        settings.email_host and settings.email_username and settings.email_password and settings.email_default_recipient
+        settings.dest_email_host
+        and settings.dest_email_username
+        and settings.dest_email_password
+        and settings.dest_email_default_recipient
     )
 
 
@@ -106,7 +109,7 @@ def get_configured_services_from_validator():
 
 
 @celery.task(base=BaseTaskWithRetry, bind=True)
-def send_to_all_destinations(self, file_path: str, use_validator=True, file_id: int = None):
+def send_to_all_destinations(self, file_path: str, use_validator=True, file_id: int = None, folder_overrides=None):
     """
     Distribute a file to all configured storage destinations.
 
@@ -115,6 +118,10 @@ def send_to_all_destinations(self, file_path: str, use_validator=True, file_id:
         use_validator: Whether to use the config validator to determine enabled services
                        (if False, falls back to individual checks)
         file_id: Optional file ID to associate with logs
+        folder_overrides: Optional dict mapping provider names to folder override strings.
+                          When set, the override is passed to the upload task which uses it
+                          instead of the provider's default folder.  Example:
+                          {"dropbox": "/Documents/pdfa", "s3": "docs/pdfa/"}
     """
     task_id = self.request.id
 
@@ -236,7 +243,10 @@ def send_to_all_destinations(self, file_path: str, use_validator=True, file_id:
                 task_id, f"queue_{service_name}", "in_progress", f"Queueing upload to {service_name}", file_id=file_id
             )
             try:
-                task = service["upload_func"].delay(file_path, file_id=file_id)
+                kwargs = {"file_id": file_id}
+                if folder_overrides and service_name in folder_overrides:
+                    kwargs["folder_override"] = folder_overrides[service_name]
+                task = service["upload_func"].delay(file_path, **kwargs)
                 results[f"{service_name}_task_id"] = task.id
                 queued_count += 1
                 log_task_progress(

app/tasks/subscription_tasks.py

@@ -0,0 +1,50 @@
+"""Celery task to apply pending subscription changes that have become due.
+
+Runs daily to ensure that scheduled downgrades are applied on time.
+"""
+
+import logging
+
+from app.celery_app import celery
+
+logger = logging.getLogger(__name__)
+
+
+@celery.task(name="app.tasks.subscription_tasks.apply_pending_subscription_changes_all")
+def apply_pending_subscription_changes_all() -> dict[str, int]:
+    """Apply all pending subscription changes whose effective date has arrived.
+
+    Iterates over every ``UserProfile`` that has a pending change and calls
+    :func:`app.utils.subscription.apply_pending_subscription_changes` for
+    each one.
+
+    Returns:
+        A dict with ``{"applied": <count>, "checked": <count>}``.
+    """
+    from app.database import SessionLocal
+    from app.models import UserProfile
+    from app.utils.subscription import apply_pending_subscription_changes
+
+    applied = 0
+    checked = 0
+    db = SessionLocal()
+    try:
+        profiles = (
+            db.query(UserProfile)
+            .filter(
+                UserProfile.subscription_change_pending_tier.isnot(None),
+                UserProfile.subscription_change_pending_date.isnot(None),
+            )
+            .all()
+        )
+        for profile in profiles:
+            checked += 1
+            if apply_pending_subscription_changes(db, profile.user_id):
+                applied += 1
+    except Exception as exc:
+        logger.error("Error in apply_pending_subscription_changes_all: %s", exc)
+    finally:
+        db.close()
+
+    logger.info("apply_pending_subscription_changes_all: checked=%d applied=%d", checked, applied)
+    return {"checked": checked, "applied": applied}

app/tasks/upload_to_dropbox.py

@@ -9,7 +9,7 @@ from dropbox.exceptions import ApiError, AuthError
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 from app.utils.filename_utils import extract_remote_path, get_unique_filename
 
@@ -102,8 +102,8 @@ def get_dropbox_client():
         raise
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_dropbox(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_dropbox(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Upload a file to Dropbox.
 
@@ -147,7 +147,7 @@ def upload_to_dropbox(self, file_path: str, file_id: int = None):
         dbx = get_dropbox_client()
 
         # Calculate remote path based on local file structure
-        remote_base = settings.dropbox_folder or ""
+        remote_base = folder_override if folder_override is not None else (settings.dropbox_folder or "")
         remote_path = extract_remote_path(file_path, settings.workdir, remote_base)
 
         # Function to check if file exists in Dropbox

app/tasks/upload_to_email.py

@@ -15,7 +15,7 @@ from jinja2 import Environment, FileSystemLoader, select_autoescape
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
@@ -125,11 +125,11 @@ def attach_logo(msg):
 def _prepare_recipients(recipients):
     """Helper function to prepare email recipients list."""
     if not recipients:
-        if not settings.email_default_recipient:
+        if not settings.dest_email_default_recipient:
             error_msg = "No recipients specified and no default recipient configured"
             logger.error(error_msg)
             return None, error_msg
-        return [settings.email_default_recipient], None
+        return [settings.dest_email_default_recipient], None
     elif isinstance(recipients, str):
         return [recipients], None  # Convert single email to list
     return recipients, None
@@ -139,17 +139,17 @@ def _send_email_with_smtp(msg, filename, recipients):
     """Helper function to handle SMTP connection and sending."""
     try:
         # First try to resolve the hostname
-        socket.gethostbyname(settings.email_host)
+        socket.gethostbyname(settings.dest_email_host)
 
         # Connect to the SMTP server
-        with smtplib.SMTP(settings.email_host, settings.email_port, timeout=30) as server:
+        with smtplib.SMTP(settings.dest_email_host, settings.dest_email_port, timeout=30) as server:
             # Use TLS if specified
-            if settings.email_use_tls:
+            if settings.dest_email_use_tls:
                 server.starttls()
 
             # Login if credentials are provided
-            if settings.email_username and settings.email_password:
-                server.login(settings.email_username, settings.email_password)
+            if settings.dest_email_username and settings.dest_email_password:
+                server.login(settings.dest_email_username, settings.dest_email_password)
 
             # Send the email
             server.send_message(msg)
@@ -157,16 +157,16 @@ def _send_email_with_smtp(msg, filename, recipients):
         logger.info(f"Successfully sent {filename} via email to {', '.join(recipients)}")
         return None
     except socket.gaierror as e:
-        error_msg = f"Failed to resolve email host: {settings.email_host} - {str(e)}"
+        error_msg = f"Failed to resolve email host: {settings.dest_email_host} - {str(e)}"
         logger.error(error_msg)
         return {"status": "Failed", "reason": error_msg, "error": str(e)}
     except (ConnectionRefusedError, TimeoutError) as e:
-        error_msg = f"Connection error to SMTP server {settings.email_host}:{settings.email_port} - {str(e)}"
+        error_msg = f"Connection error to SMTP server {settings.dest_email_host}:{settings.dest_email_port} - {str(e)}"
         logger.error(error_msg)
         return {"status": "Failed", "reason": error_msg, "error": str(e)}
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=UploadTaskWithRetry, bind=True)
 def upload_to_email(
     self,
     file_path: str,
@@ -205,17 +205,17 @@ def upload_to_email(
     # Extract filename
     filename = os.path.basename(file_path)
 
-    # Check if email settings are configured
-    if not settings.email_host:
-        error_msg = "Email host is not configured"
+    # Check if email destination settings are configured
+    if not settings.dest_email_host:
+        error_msg = "Email destination host is not configured (DEST_EMAIL_HOST)"
         logger.error(f"[{task_id}] {error_msg}")
         log_task_progress(task_id, "upload_to_email", "skipped", error_msg, file_id=file_id)
         return {"status": "Skipped", "reason": error_msg}
 
     # Log email configuration for debugging
     logger.debug(
-        f"[{task_id}] Email config - Host: {settings.email_host}, Port: {settings.email_port}, "
-        f"Username: {settings.email_username}, TLS: {settings.email_use_tls}"
+        f"[{task_id}] Email destination config - Host: {settings.dest_email_host}, Port: {settings.dest_email_port}, "
+        f"Username: {settings.dest_email_username}, TLS: {settings.dest_email_use_tls}"
     )
 
     # Process recipients
@@ -236,7 +236,7 @@ def upload_to_email(
     try:
         # Create the email
         msg = MIMEMultipart("related")
-        msg["From"] = settings.email_sender or settings.email_username
+        msg["From"] = settings.dest_email_sender or settings.dest_email_username
         msg["To"] = ", ".join(recipients)
         msg["Subject"] = subject
 

app/tasks/upload_to_ftp.py

@@ -8,14 +8,14 @@ import os
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_ftp(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_ftp(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Uploads a file to an FTP server in the configured folder.
 
@@ -97,10 +97,11 @@ def upload_to_ftp(self, file_path: str, file_id: int = None):
             ftp.login(user=settings.ftp_username, passwd=settings.ftp_password)
 
         # Change to target directory if specified
-        if settings.ftp_folder:
+        ftp_folder_setting = folder_override if folder_override is not None else settings.ftp_folder
+        if ftp_folder_setting:
             try:
                 # Try to navigate to the directory, create if it doesn't exist
-                ftp_folder = settings.ftp_folder
+                ftp_folder = ftp_folder_setting
                 # Remove leading slash if present
                 if ftp_folder.startswith("/"):
                     ftp_folder = ftp_folder[1:]
@@ -138,7 +139,7 @@ def upload_to_ftp(self, file_path: str, file_id: int = None):
             "status": "Completed",
             "file": file_path,
             "ftp_host": settings.ftp_host,
-            "ftp_path": f"{settings.ftp_folder}/{filename}" if settings.ftp_folder else filename,
+            "ftp_path": f"{ftp_folder_setting}/{filename}" if ftp_folder_setting else filename,
             "used_tls": used_tls,
         }
 

app/tasks/upload_to_google_drive.py

@@ -15,7 +15,7 @@ from googleapiclient.http import MediaFileUpload
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
@@ -152,8 +152,10 @@ def truncate_property_value(key, value, max_bytes=100):
     return str_value
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_google_drive(self, file_path: str, include_metadata=True, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_google_drive(
+    self, file_path: str, include_metadata=True, file_id: int = None, folder_override: str = None
+):
     """
     Uploads a file to Google Drive in the configured folder with optional metadata.
 
@@ -201,8 +203,9 @@ def upload_to_google_drive(self, file_path: str, include_metadata=True, file_id:
         }
 
         # If folder ID is specified, set parent folder
-        if settings.google_drive_folder_id:
-            file_metadata["parents"] = [settings.google_drive_folder_id]
+        gdrive_folder_id = folder_override if folder_override is not None else settings.google_drive_folder_id
+        if gdrive_folder_id:
+            file_metadata["parents"] = [gdrive_folder_id]
 
         # Add custom properties if metadata exists
         if metadata:

app/tasks/upload_to_nextcloud.py

@@ -8,15 +8,15 @@ from requests.auth import HTTPBasicAuth
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 from app.utils.filename_utils import extract_remote_path, get_unique_filename
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_nextcloud(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_nextcloud(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Upload a file to Nextcloud WebDAV.
 
@@ -60,7 +60,9 @@ def upload_to_nextcloud(self, file_path: str, file_id: int = None):
             webdav_url += "/"
 
         # Calculate remote path based on local file structure
-        remote_base = getattr(settings, "nextcloud_folder", "") or ""
+        remote_base = (
+            folder_override if folder_override is not None else (getattr(settings, "nextcloud_folder", "") or "")
+        )
         remote_path = extract_remote_path(file_path, settings.workdir, remote_base)
         full_url = f"{webdav_url}/{remote_path}"
 

app/tasks/upload_to_onedrive.py

@@ -10,7 +10,7 @@ import requests
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
@@ -209,8 +209,8 @@ def upload_large_file(file_path, upload_url):
     return response.json()
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_onedrive(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_onedrive(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Uploads a file to OneDrive in the configured folder.
 
@@ -248,15 +248,17 @@ def upload_to_onedrive(self, file_path: str, file_id: int = None):
         # Get access token
         access_token = get_onedrive_token()
 
+        onedrive_folder = folder_override if folder_override is not None else settings.onedrive_folder_path
+
         # Create upload session
-        upload_url = create_upload_session(filename, settings.onedrive_folder_path, access_token)
+        upload_url = create_upload_session(filename, onedrive_folder, access_token)
 
         # Upload the file
         result = upload_large_file(file_path, upload_url)
 
         # Log success
         web_url = result.get("webUrl", "Not available")
-        logger.info(f"[{task_id}] Successfully uploaded {filename} to OneDrive at path {settings.onedrive_folder_path}")
+        logger.info(f"[{task_id}] Successfully uploaded {filename} to OneDrive at path {onedrive_folder}")
         logger.info(f"[{task_id}] File accessible at: {web_url}")
         log_task_progress(
             task_id, "upload_to_onedrive", "success", f"Uploaded to OneDrive: {filename}", file_id=file_id
@@ -265,7 +267,7 @@ def upload_to_onedrive(self, file_path: str, file_id: int = None):
         return {
             "status": "Completed",
             "file_path": file_path,
-            "onedrive_path": f"{settings.onedrive_folder_path}/{filename}",
+            "onedrive_path": f"{onedrive_folder}/{filename}",
             "web_url": web_url,
         }
 

app/tasks/upload_to_paperless.py

@@ -10,7 +10,7 @@ import requests
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
@@ -199,7 +199,7 @@ def set_document_custom_fields(doc_id: int, custom_fields: dict, task_id: str) -
         logger.error(f"[{task_id}] Response: {getattr(exc.response, 'text', '<no response>')}")
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=UploadTaskWithRetry, bind=True)
 def upload_to_paperless(self, file_path: str, file_id: int = None):
     """
     Uploads a file to Paperless-ngx and sets custom fields from metadata.

app/tasks/upload_to_s3.py

@@ -8,14 +8,14 @@ from botocore.exceptions import ClientError
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_s3(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_s3(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Uploads a file to Amazon S3 in the configured bucket and folder.
 
@@ -61,9 +61,10 @@ def upload_to_s3(self, file_path: str, file_id: int = None):
         )
 
         # Construct the S3 key (path within the bucket)
-        if settings.s3_folder_prefix:
+        s3_folder = folder_override if folder_override is not None else settings.s3_folder_prefix
+        if s3_folder:
             # Ensure folder prefix ends with a slash
-            folder_prefix = settings.s3_folder_prefix
+            folder_prefix = s3_folder
             if not folder_prefix.endswith("/"):
                 folder_prefix += "/"
             s3_key = f"{folder_prefix}{filename}"

app/tasks/upload_to_sftp.py

@@ -7,15 +7,15 @@ import paramiko
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 from app.utils.filename_utils import extract_remote_path, get_unique_filename, sanitize_filename
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_sftp(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_sftp(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Upload a file to an SFTP server.
 
@@ -95,7 +95,7 @@ def upload_to_sftp(self, file_path: str, file_id: int = None):
         sftp = ssh.open_sftp()
 
         # Calculate remote path based on local file structure
-        remote_base = settings.sftp_folder or ""
+        remote_base = folder_override if folder_override is not None else (settings.sftp_folder or "")
         remote_path = extract_remote_path(file_path, settings.workdir, remote_base)
 
         # Ensure the remote path starts with a slash if the base folder does

app/tasks/upload_to_webdav.py

@@ -8,14 +8,14 @@ import requests
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
-def upload_to_webdav(self, file_path: str, file_id: int = None):
+@celery.task(base=UploadTaskWithRetry, bind=True)
+def upload_to_webdav(self, file_path: str, file_id: int = None, folder_override: str = None):
     """
     Uploads a file to a WebDAV server in the configured folder.
 
@@ -50,7 +50,7 @@ def upload_to_webdav(self, file_path: str, file_id: int = None):
         raise ValueError(error_msg)
 
     # Construct the full upload URL
-    webdav_folder = settings.webdav_folder or ""
+    webdav_folder = folder_override if folder_override is not None else (settings.webdav_folder or "")
     # Ensure folder doesn't have leading slash if we're joining it to the base URL
     if webdav_folder and webdav_folder.startswith("/"):
         webdav_folder = webdav_folder[1:]

app/tasks/upload_with_rclone.py

@@ -6,13 +6,13 @@ import subprocess
 
 from app.celery_app import celery
 from app.config import settings
-from app.tasks.retry_config import BaseTaskWithRetry
+from app.tasks.retry_config import UploadTaskWithRetry
 from app.utils import log_task_progress
 
 logger = logging.getLogger(__name__)
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=UploadTaskWithRetry, bind=True)
 def upload_with_rclone(self, file_path: str, destination: str):
     """
     Uploads a file using rclone to the specified destination.
@@ -107,7 +107,7 @@ def upload_with_rclone(self, file_path: str, destination: str):
         raise RuntimeError(error_msg) from e
 
 
-@celery.task(base=BaseTaskWithRetry, bind=True)
+@celery.task(base=UploadTaskWithRetry, bind=True)
 def send_to_all_rclone_destinations(self, file_path: str):
     """
     Uploads a file to all configured rclone destinations.

app/tasks/webhook_tasks.py

@@ -0,0 +1,39 @@
+"""Celery task for asynchronous webhook delivery with retry and backoff.
+
+Uses :class:`~app.tasks.retry_config.BaseTaskWithRetry` so failed deliveries
+are automatically retried with exponential backoff (default: 60 s, 300 s,
+900 s) and ±20 % jitter.
+"""
+
+import logging
+from typing import Any
+
+from app.celery_app import celery
+from app.tasks.retry_config import BaseTaskWithRetry
+from app.utils.webhook import deliver_webhook
+
+logger = logging.getLogger(__name__)
+
+
+@celery.task(base=BaseTaskWithRetry, bind=True, name="webhook.deliver")
+def deliver_webhook_task(self, url: str, payload: dict[str, Any], secret: str | None = None) -> dict[str, Any]:
+    """Deliver a webhook payload to *url* with automatic retries.
+
+    Args:
+        url: Target webhook URL.
+        payload: The full webhook payload envelope.
+        secret: Optional shared secret for HMAC-SHA256 signing.
+
+    Returns:
+        A dict with ``status`` and ``url`` on success.
+
+    Raises:
+        RuntimeError: Re-raised to trigger Celery retry on delivery failure.
+    """
+    logger.info("Delivering webhook to %s (attempt %d/%d)", url, self.request.retries + 1, self.max_retries + 1)
+
+    success = deliver_webhook(url, payload, secret)
+    if success:
+        return {"status": "delivered", "url": url}
+
+    raise RuntimeError(f"Webhook delivery to {url} failed")

app/templates/email/default.html

@@ -0,0 +1,63 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8">
+  <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>{{ filename }} – DocuElevate</title>
+  <style>
+    body { margin: 0; padding: 0; background-color: #f4f4f5; font-family: Arial, Helvetica, sans-serif; }
+    .wrapper { max-width: 600px; margin: 32px auto; background: #ffffff; border-radius: 8px; box-shadow: 0 2px 8px rgba(0,0,0,.08); overflow: hidden; }
+    .header { background: linear-gradient(135deg, #4f46e5 0%, #7c3aed 100%); padding: 32px 40px; text-align: center; }
+    .header img { max-height: 48px; }
+    .header h1 { color: #ffffff; font-size: 22px; margin: 16px 0 0; }
+    .body { padding: 32px 40px; }
+    .body p { color: #374151; font-size: 15px; line-height: 1.6; margin: 0 0 16px; }
+    .attachment-box { background: #f9fafb; border: 1px solid #e5e7eb; border-radius: 6px; padding: 16px 20px; margin: 24px 0; }
+    .attachment-box .label { font-size: 11px; font-weight: bold; color: #6b7280; text-transform: uppercase; letter-spacing: .05em; margin-bottom: 6px; }
+    .attachment-box .filename { color: #111827; font-size: 15px; font-weight: bold; word-break: break-all; }
+    .metadata-table { width: 100%; border-collapse: collapse; margin-top: 8px; font-size: 13px; }
+    .metadata-table td { padding: 6px 0; color: #374151; vertical-align: top; }
+    .metadata-table td:first-child { font-weight: bold; color: #6b7280; width: 40%; padding-right: 12px; }
+    .footer { background: #f9fafb; border-top: 1px solid #e5e7eb; padding: 20px 40px; text-align: center; color: #9ca3af; font-size: 12px; }
+    .footer a { color: #4f46e5; text-decoration: none; }
+  </style>
+</head>
+<body>
+  <div class="wrapper">
+    <div class="header">
+      {% if has_logo %}
+      <img src="cid:logo" alt="{{ app_name }} logo">
+      {% endif %}
+      <h1>Document Delivery</h1>
+    </div>
+    <div class="body">
+      <p>{{ message }}</p>
+
+      <div class="attachment-box">
+        <div class="label">Attached file</div>
+        <div class="filename">📎 {{ filename }}</div>
+      </div>
+
+      {% if has_metadata and metadata %}
+      <p style="font-weight:bold; color:#374151; margin-bottom:8px;">Document metadata</p>
+      <table class="metadata-table">
+        {% for key, value in metadata.items() %}
+        <tr>
+          <td>{{ key }}</td>
+          <td>{{ value }}</td>
+        </tr>
+        {% endfor %}
+      </table>
+      {% endif %}
+
+      <p style="margin-top:24px; color:#6b7280; font-size:13px;">
+        This document was sent automatically by {{ app_name }}.{% if app_url %} Visit <a href="{{ app_url }}" style="color:#4f46e5;">{{ app_url }}</a> to manage your documents.{% endif %}
+      </p>
+    </div>
+    <div class="footer">
+      &copy; {{ current_year }} {{ app_name }} &middot; Intelligent Document Processing
+      {% if app_url %}&middot; <a href="{{ app_url }}">{{ app_url }}</a>{% endif %}
+    </div>
+  </div>
+</body>
+</html>

app/utils/config_validator/providers.py

@@ -154,23 +154,23 @@ def get_provider_status() -> dict[str, dict[str, object]]:
         },
     }
 
-    # Add Email configuration
+    # Add Email destination configuration (dedicated settings for document delivery)
     providers["Email"] = {
         "name": "Email",
         "icon": "fa-solid fa-envelope",
         "configured": bool(
-            getattr(settings, "email_host", None) and getattr(settings, "email_default_recipient", None)
+            getattr(settings, "dest_email_host", None) and getattr(settings, "dest_email_default_recipient", None)
         ),
         "enabled": True,
         "description": "Send documents via email",
         "details": {
-            "host": getattr(settings, "email_host", "Not set"),
-            "port": getattr(settings, "email_port", "Not set"),
-            "username": getattr(settings, "email_username", "Not set"),
-            "password": mask_sensitive_value(getattr(settings, "email_password", None)),
-            "use_tls": getattr(settings, "email_use_tls", "Not set"),
-            "sender": getattr(settings, "email_sender", "Not set"),
-            "default_recipient": getattr(settings, "email_default_recipient", "Not set"),
+            "host": getattr(settings, "dest_email_host", "Not set"),
+            "port": getattr(settings, "dest_email_port", "Not set"),
+            "username": getattr(settings, "dest_email_username", "Not set"),
+            "password": mask_sensitive_value(getattr(settings, "dest_email_password", None)),
+            "use_tls": getattr(settings, "dest_email_use_tls", "Not set"),
+            "sender": getattr(settings, "dest_email_sender", "Not set"),
+            "default_recipient": getattr(settings, "dest_email_default_recipient", "Not set"),
         },
     }
 

app/utils/config_validator/settings_display.py

@@ -106,6 +106,35 @@ def get_settings_for_display(show_values: bool = False) -> dict[str, list[dict[s
             "email_sender",
             "email_default_recipient",
         ],
+        "Watch Folders": [
+            "watch_folders",
+            "watch_folder_poll_interval",
+            "watch_folder_delete_after_process",
+            "ftp_ingest_enabled",
+            "ftp_ingest_folder",
+            "ftp_ingest_delete_after_process",
+            "sftp_ingest_enabled",
+            "sftp_ingest_folder",
+            "sftp_ingest_delete_after_process",
+            "dropbox_ingest_enabled",
+            "dropbox_ingest_folder",
+            "dropbox_ingest_delete_after_process",
+            "google_drive_ingest_enabled",
+            "google_drive_ingest_folder_id",
+            "google_drive_ingest_delete_after_process",
+            "onedrive_ingest_enabled",
+            "onedrive_ingest_folder_path",
+            "onedrive_ingest_delete_after_process",
+            "nextcloud_ingest_enabled",
+            "nextcloud_ingest_folder",
+            "nextcloud_ingest_delete_after_process",
+            "s3_ingest_enabled",
+            "s3_ingest_prefix",
+            "s3_ingest_delete_after_process",
+            "webdav_ingest_enabled",
+            "webdav_ingest_folder",
+            "webdav_ingest_delete_after_process",
+        ],
         "IMAP": [
             "imap1_host",
             "imap1_port",

app/utils/config_validator/validators.py

@@ -107,12 +107,12 @@ def validate_storage_configs() -> dict[str, list[str]]:
 
     issues["sftp"] = sftp_issues
 
-    # Validate Email sending
+    # Validate Email sending (destination-specific settings)
     email_issues = []
-    if not getattr(settings, "email_host", None):
-        email_issues.append("EMAIL_HOST is not configured")
-    if not getattr(settings, "email_default_recipient", None):
-        email_issues.append("EMAIL_DEFAULT_RECIPIENT is not configured")
+    if not getattr(settings, "dest_email_host", None):
+        email_issues.append("DEST_EMAIL_HOST is not configured")
+    if not getattr(settings, "dest_email_default_recipient", None):
+        email_issues.append("DEST_EMAIL_DEFAULT_RECIPIENT is not configured")
     issues["email"] = email_issues
 
     # Validate S3

app/utils/db_migrate.py

@@ -0,0 +1,251 @@
+"""
+Database migration utility for transferring data between databases.
+
+Copies all table rows from a *source* SQLAlchemy database to a *target*
+database.  This is designed for the common scenario of migrating from the
+built-in SQLite database to an external PostgreSQL / MySQL instance.
+
+The utility:
+1. Creates the schema in the target via ``Base.metadata.create_all``.
+2. Copies rows table-by-table in dependency order.
+3. Stamps the Alembic version in the target to ``head``.
+"""
+
+import logging
+from typing import Any
+
+from sqlalchemy import MetaData, create_engine, inspect, text
+from sqlalchemy.engine import Engine
+from sqlalchemy.engine.url import make_url
+from sqlalchemy.orm import sessionmaker
+
+logger = logging.getLogger(__name__)
+
+# Tables to skip during migration (Alembic manages its own state).
+_SKIP_TABLES = {"alembic_version"}
+
+# Ordered list — parent tables first to respect foreign-key constraints.
+_TABLE_ORDER = [
+    "documents",
+    "files",
+    "file_processing_steps",
+    "processing_logs",
+    "application_settings",
+    "settings_audit_log",
+    "saved_searches",
+    "webhook_configs",
+]
+
+
+def _make_engine(url: str) -> Engine:
+    """Create a SQLAlchemy engine from *url* with sensible defaults."""
+    parsed = make_url(url)
+    connect_args: dict[str, Any] = {}
+    if parsed.get_backend_name() == "sqlite":
+        connect_args["check_same_thread"] = False
+    return create_engine(url, connect_args=connect_args)
+
+
+def _ordered_tables(inspector: Any) -> list[str]:
+    """Return table names in safe insertion order.
+
+    Tables listed in ``_TABLE_ORDER`` come first (in that order); any
+    remaining tables are appended alphabetically.
+    """
+    existing = set(inspector.get_table_names())
+    ordered: list[str] = []
+    for name in _TABLE_ORDER:
+        if name in existing and name not in _SKIP_TABLES:
+            ordered.append(name)
+    for name in sorted(existing):
+        if name not in ordered and name not in _SKIP_TABLES:
+            ordered.append(name)
+    return ordered
+
+
+def preview_migration(source_url: str) -> dict[str, Any]:
+    """Preview what a migration would do without actually copying data.
+
+    Args:
+        source_url: Connection string for the source database.
+
+    Returns:
+        Dict with ``tables`` (list of dicts with ``name`` and ``row_count``)
+        and ``total_rows``.
+    """
+    try:
+        src_engine = _make_engine(source_url)
+        src_inspector = inspect(src_engine)
+        tables = _ordered_tables(src_inspector)
+
+        result: list[dict[str, Any]] = []
+        total = 0
+        with src_engine.connect() as conn:
+            for table_name in tables:
+                # table_name is safe — sourced from inspect().get_table_names(), not user input
+                row = conn.execute(text(f'SELECT COUNT(*) FROM "{table_name}"')).fetchone()  # noqa: S608
+                count = row[0] if row else 0
+                result.append({"name": table_name, "row_count": count})
+                total += count
+
+        src_engine.dispose()
+        return {"tables": result, "total_rows": total, "success": True}
+    except Exception as exc:
+        logger.error(f"Migration preview failed: {exc}")
+        return {"success": False, "error": str(exc), "tables": [], "total_rows": 0}
+
+
+def migrate_data(
+    source_url: str,
+    target_url: str,
+    *,
+    batch_size: int = 500,
+    progress_callback: Any | None = None,
+) -> dict[str, Any]:
+    """Copy all data from *source_url* to *target_url*.
+
+    The target schema is created automatically from the application models.
+    Alembic is stamped to ``head`` in the target after a successful copy.
+
+    Args:
+        source_url: SQLAlchemy connection string for the source DB.
+        target_url: SQLAlchemy connection string for the target DB.
+        batch_size: Number of rows to insert per batch.
+        progress_callback: Optional ``callable(table_name, copied, total)``
+            invoked after each batch.
+
+    Returns:
+        Dict with ``success`` (bool), ``tables_copied`` (int),
+        ``rows_copied`` (int), and ``errors`` (list of str).
+    """
+    errors: list[str] = []
+    tables_copied = 0
+    rows_copied = 0
+
+    try:
+        src_engine = _make_engine(source_url)
+        tgt_engine = _make_engine(target_url)
+
+        # ------------------------------------------------------------------
+        # 1.  Create schema in target from application models
+        # ------------------------------------------------------------------
+        from app.database import Base  # local import to avoid circular deps
+
+        Base.metadata.create_all(bind=tgt_engine)
+        logger.info("Target schema created from application models.")
+
+        # ------------------------------------------------------------------
+        # 2.  Reflect source schema & determine copy order
+        # ------------------------------------------------------------------
+        src_meta = MetaData()
+        src_meta.reflect(bind=src_engine)
+
+        src_inspector = inspect(src_engine)
+        table_names = _ordered_tables(src_inspector)
+
+        SrcSession = sessionmaker(bind=src_engine)
+        TgtSession = sessionmaker(bind=tgt_engine)
+
+        # ------------------------------------------------------------------
+        # 3.  Copy data table-by-table
+        # ------------------------------------------------------------------
+        for table_name in table_names:
+            try:
+                src_session = SrcSession()
+                tgt_session = TgtSession()
+
+                src_table = src_meta.tables.get(table_name)
+                if src_table is None:
+                    continue
+
+                # Read all rows from source
+                rows = src_session.execute(src_table.select()).fetchall()
+                column_names = [c.name for c in src_table.columns]
+
+                if not rows:
+                    logger.info(f"Skipping empty table: {table_name}")
+                    tables_copied += 1
+                    src_session.close()
+                    tgt_session.close()
+                    continue
+
+                # Reflect the target table to insert into
+                tgt_meta = MetaData()
+                tgt_meta.reflect(bind=tgt_engine, only=[table_name])
+                tgt_table = tgt_meta.tables.get(table_name)
+                if tgt_table is None:
+                    errors.append(f"Target table {table_name} not found after schema creation")
+                    src_session.close()
+                    tgt_session.close()
+                    continue
+
+                # Batch insert
+                total_for_table = len(rows)
+                for i in range(0, total_for_table, batch_size):
+                    batch = rows[i : i + batch_size]
+                    # strict=False: column count should always match, but tolerate
+                    # minor schema drift (e.g. extra columns) to avoid crashing mid-migration.
+                    insert_data = [dict(zip(column_names, row, strict=False)) for row in batch]
+                    tgt_session.execute(tgt_table.insert(), insert_data)
+                    tgt_session.commit()
+
+                    rows_copied += len(batch)
+                    if progress_callback:
+                        progress_callback(table_name, min(i + batch_size, total_for_table), total_for_table)
+
+                tables_copied += 1
+                logger.info(f"Copied {total_for_table} rows from {table_name}")
+                src_session.close()
+                tgt_session.close()
+
+            except Exception as exc:
+                msg = f"Error copying table {table_name}: {exc}"
+                logger.error(msg)
+                errors.append(msg)
+
+        # ------------------------------------------------------------------
+        # 4.  Stamp Alembic to head in the target
+        # ------------------------------------------------------------------
+        try:
+            _stamp_alembic_head(tgt_engine)
+            logger.info("Alembic version stamped to head in target database.")
+        except Exception as exc:
+            msg = f"Failed to stamp Alembic version: {exc}"
+            logger.error(msg)
+            errors.append(msg)
+
+        src_engine.dispose()
+        tgt_engine.dispose()
+
+        return {
+            "success": len(errors) == 0,
+            "tables_copied": tables_copied,
+            "rows_copied": rows_copied,
+            "errors": errors,
+        }
+
+    except Exception as exc:
+        logger.error(f"Migration failed: {exc}")
+        return {
+            "success": False,
+            "tables_copied": tables_copied,
+            "rows_copied": rows_copied,
+            "errors": errors + [str(exc)],
+        }
+
+
+def _stamp_alembic_head(engine: Engine) -> None:
+    """Stamp the Alembic version table to ``head`` in the given engine."""
+    from pathlib import Path
+
+    from alembic import command
+    from alembic.config import Config
+
+    migrations_dir = str(Path(__file__).resolve().parent.parent.parent / "migrations")
+    alembic_cfg = Config()
+    alembic_cfg.set_main_option("script_location", migrations_dir)
+    alembic_cfg.set_main_option("sqlalchemy.url", "")
+
+    with engine.begin() as connection:
+        alembic_cfg.attributes["connection"] = connection
+        command.stamp(alembic_cfg, "head")

app/utils/db_wizard.py

@@ -0,0 +1,257 @@
+"""
+Database configuration wizard utilities.
+
+Provides helpers for building, validating, and testing database connection
+strings.  Used by both the interactive wizard UI and the REST API.
+"""
+
+import logging
+from typing import Any
+
+from sqlalchemy import create_engine, text
+from sqlalchemy.engine.url import make_url
+
+logger = logging.getLogger(__name__)
+
+# Supported database backends with human-readable labels and defaults.
+SUPPORTED_BACKENDS: list[dict[str, Any]] = [
+    {
+        "id": "sqlite",
+        "label": "SQLite (Development)",
+        "driver": "",
+        "default_port": None,
+        "description": "File-based database. Best for development and single-user setups.",
+        "requires_host": False,
+    },
+    {
+        "id": "postgresql",
+        "label": "PostgreSQL (Recommended for Production)",
+        "driver": "",
+        "default_port": 5432,
+        "description": "Robust, full-featured database. Recommended for production.",
+        "requires_host": True,
+    },
+    {
+        "id": "mysql",
+        "label": "MySQL / MariaDB",
+        "driver": "pymysql",
+        "default_port": 3306,
+        "description": "Popular open-source database. Requires pymysql driver.",
+        "requires_host": True,
+    },
+]
+
+
+def get_supported_backends() -> list[dict[str, Any]]:
+    """Return the list of supported database backends with metadata.
+
+    Returns:
+        List of backend descriptor dicts.
+    """
+    return SUPPORTED_BACKENDS
+
+
+def build_connection_string(
+    backend: str,
+    host: str = "",
+    port: int | None = None,
+    database: str = "",
+    username: str = "",
+    password: str = "",
+    ssl_mode: str = "",
+    extra_options: str = "",
+    sqlite_path: str = "",
+) -> str:
+    """Build a SQLAlchemy connection string from individual components.
+
+    Args:
+        backend: Database backend identifier (``sqlite``, ``postgresql``, ``mysql``).
+        host: Database server hostname or IP.
+        port: Database server port (uses backend default when ``None``).
+        database: Database / schema name.
+        username: Authentication username.
+        password: Authentication password.
+        ssl_mode: SSL mode (e.g. ``require``, ``verify-full``).  PostgreSQL only.
+        extra_options: Additional query-string options appended to the URL.
+        sqlite_path: File path for SQLite databases.
+
+    Returns:
+        A SQLAlchemy-compatible connection URL string.
+
+    Raises:
+        ValueError: If required fields are missing for the chosen backend.
+    """
+    if backend == "sqlite":
+        path = sqlite_path.strip() if sqlite_path else ""
+        if not path:
+            path = "./app/database.db"
+        return f"sqlite:///{path}"
+
+    # Resolve driver prefix
+    backend_info = next((b for b in SUPPORTED_BACKENDS if b["id"] == backend), None)
+    if backend_info is None:
+        raise ValueError(f"Unsupported backend: {backend}")
+
+    if not host:
+        raise ValueError("Host is required for non-SQLite backends")
+    if not database:
+        raise ValueError("Database name is required for non-SQLite backends")
+    if not username:
+        raise ValueError("Username is required for non-SQLite backends")
+
+    driver_suffix = f"+{backend_info['driver']}" if backend_info["driver"] else ""
+    scheme = f"{backend}{driver_suffix}"
+
+    resolved_port = port if port else backend_info["default_port"]
+
+    # Build query parameters
+    params: list[str] = []
+    if ssl_mode:
+        params.append(f"sslmode={ssl_mode}")
+    if extra_options:
+        params.append(extra_options)
+    if backend == "mysql" and "charset=" not in extra_options:
+        params.append("charset=utf8mb4")
+
+    query_string = "&".join(params)
+
+    # Construct URL
+    auth = username
+    if password:
+        auth = f"{username}:{password}"
+
+    url = f"{scheme}://{auth}@{host}:{resolved_port}/{database}"
+    if query_string:
+        url = f"{url}?{query_string}"
+
+    return url
+
+
+def parse_connection_string(url: str) -> dict[str, Any]:
+    """Parse a SQLAlchemy connection string into its components.
+
+    Args:
+        url: A SQLAlchemy database URL string.
+
+    Returns:
+        Dict with keys: ``backend``, ``host``, ``port``, ``database``,
+        ``username``, ``password``, ``ssl_mode``, ``is_sqlite``.
+    """
+    try:
+        parsed = make_url(url)
+        backend_name = parsed.get_backend_name()
+        return {
+            "backend": backend_name,
+            "host": parsed.host or "",
+            "port": parsed.port,
+            "database": parsed.database or "",
+            "username": parsed.username or "",
+            "password": parsed.password or "",
+            "ssl_mode": "",
+            "is_sqlite": backend_name == "sqlite",
+            "valid": True,
+        }
+    except Exception as exc:
+        logger.warning(f"Failed to parse connection string: {exc}")
+        return {"valid": False, "error": str(exc)}
+
+
+def test_connection(url: str, timeout: int = 10) -> dict[str, Any]:
+    """Attempt to connect to a database and return status information.
+
+    The function creates a short-lived engine, executes a simple ``SELECT 1``
+    query, and disposes the engine.  It does **not** modify any global state.
+
+    Args:
+        url: SQLAlchemy database URL to test.
+        timeout: Connection timeout in seconds.
+
+    Returns:
+        Dict with ``success`` (bool), ``message`` (str), and optional
+        ``server_version`` (str).
+    """
+    try:
+        parsed = make_url(url)
+        backend = parsed.get_backend_name()
+
+        connect_args: dict[str, Any] = {}
+        kwargs: dict[str, Any] = {"pool_pre_ping": True}
+
+        if backend == "sqlite":
+            connect_args["check_same_thread"] = False
+        else:
+            kwargs["pool_timeout"] = timeout
+
+        test_engine = create_engine(
+            url,
+            connect_args=connect_args,
+            **kwargs,
+        )
+
+        with test_engine.connect() as conn:
+            result = conn.execute(text("SELECT 1"))
+            result.fetchone()
+
+            # Try to fetch server version for informational display
+            server_version = _get_server_version(conn, backend)
+
+        test_engine.dispose()
+
+        return {
+            "success": True,
+            "message": "Connection successful",
+            "backend": backend,
+            "server_version": server_version,
+        }
+    except Exception as exc:
+        logger.warning(f"Connection test failed: {exc}")
+        return {
+            "success": False,
+            "message": str(exc),
+            "backend": "",
+            "server_version": "",
+        }
+
+
+def _get_server_version(conn: Any, backend: str) -> str:
+    """Retrieve a human-readable server version string.
+
+    Args:
+        conn: An active SQLAlchemy connection.
+        backend: Backend identifier (``sqlite``, ``postgresql``, ``mysql``).
+
+    Returns:
+        Server version string, or empty string on failure.
+    """
+    try:
+        if backend == "postgresql":
+            row = conn.execute(text("SELECT version()")).fetchone()
+            return str(row[0]) if row else ""
+        elif backend == "mysql":
+            row = conn.execute(text("SELECT version()")).fetchone()
+            return str(row[0]) if row else ""
+        elif backend == "sqlite":
+            row = conn.execute(text("SELECT sqlite_version()")).fetchone()
+            return f"SQLite {row[0]}" if row else ""
+    except Exception:
+        logger.debug("Could not retrieve server version")
+    return ""
+
+
+def validate_url_format(url: str) -> dict[str, Any]:
+    """Validate that a connection string is syntactically correct.
+
+    Args:
+        url: The connection string to validate.
+
+    Returns:
+        Dict with ``valid`` (bool) and optional ``error`` (str).
+    """
+    try:
+        parsed = make_url(url)
+        backend = parsed.get_backend_name()
+        if backend not in ("sqlite", "postgresql", "mysql"):
+            return {"valid": False, "error": f"Unsupported backend: {backend}"}
+        return {"valid": True, "backend": backend}
+    except Exception as exc:
+        return {"valid": False, "error": str(exc)}

app/utils/local_auth.py

@@ -0,0 +1,222 @@
+"""Utilities for local (email/password) user authentication.
+
+Provides password hashing (bcrypt), secure token generation, and
+synchronous SMTP email helpers for account verification and password
+reset flows.  No external dependencies beyond bcrypt (already in
+requirements.txt) and Python stdlib.
+"""
+
+import logging
+import secrets
+import smtplib
+import socket
+from datetime import datetime, timedelta, timezone
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+
+import bcrypt
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+TOKEN_BYTES = 32  # 256 bits of entropy
+TOKEN_EXPIRY_HOURS = 24  # verification + reset tokens expire after 24 h
+
+
+def hash_password(plain: str) -> str:
+    """Return a bcrypt hash of *plain*. Stores result as a UTF-8 string."""
+    return bcrypt.hashpw(plain.encode("utf-8"), bcrypt.gensalt(rounds=12)).decode("utf-8")
+
+
+def verify_password(plain: str, hashed: str) -> bool:
+    """Return True when *plain* matches the stored bcrypt *hashed* string."""
+    try:
+        return bcrypt.checkpw(plain.encode("utf-8"), hashed.encode("utf-8"))
+    except Exception:
+        return False
+
+
+def generate_token() -> str:
+    """Return a 256-bit URL-safe random token string."""
+    return secrets.token_urlsafe(TOKEN_BYTES)
+
+
+def is_token_expired(sent_at: datetime | None) -> bool:
+    """Return True when *sent_at* is None or older than TOKEN_EXPIRY_HOURS."""
+    if sent_at is None:
+        return True
+    return datetime.now(tz=timezone.utc) > sent_at.astimezone(timezone.utc) + timedelta(hours=TOKEN_EXPIRY_HOURS)
+
+
+def _smtp_send(subject: str, html_body: str, plain_body: str, recipient: str) -> None:
+    """Send an HTML email via the configured SMTP server.
+
+    Args:
+        subject: Email subject line.
+        html_body: HTML version of the email body.
+        plain_body: Plain-text version of the email body.
+        recipient: Recipient email address.
+
+    Raises:
+        RuntimeError: When SMTP is not configured or sending fails.
+    """
+    if not settings.email_host:
+        raise RuntimeError("SMTP is not configured (EMAIL_HOST missing). Cannot send email.")
+
+    sender = settings.email_sender or settings.email_username or "noreply@docuelevate.local"
+
+    msg = MIMEMultipart("alternative")
+    msg["Subject"] = subject
+    msg["From"] = sender
+    msg["To"] = recipient
+    msg.attach(MIMEText(plain_body, "plain", "utf-8"))
+    msg.attach(MIMEText(html_body, "html", "utf-8"))
+
+    try:
+        socket.gethostbyname(settings.email_host)
+    except socket.gaierror as exc:
+        raise RuntimeError(f"Cannot resolve SMTP host {settings.email_host!r}: {exc}") from exc
+
+    with smtplib.SMTP(settings.email_host, settings.email_port or 587, timeout=30) as server:
+        if settings.email_use_tls:
+            server.starttls()
+        if settings.email_username and settings.email_password:
+            server.login(settings.email_username, settings.email_password)
+        server.send_message(msg)
+
+    logger.info("Sent %r to %s", subject, recipient)
+
+
+def send_verification_email(email: str, username: str, token: str, base_url: str) -> None:
+    """Send a double opt-in verification email to *email*.
+
+    Args:
+        email: Recipient email address.
+        username: The user's chosen username (used in greeting).
+        token: The verification token to embed in the link.
+        base_url: The base URL of the application (e.g. https://app.example.com).
+    """
+    verify_url = f"{base_url}/verify-email?token={token}"
+    subject = "Verify your DocuElevate account"
+    html_body = f"""<!DOCTYPE html>
+<html lang="en">
+<head><meta charset="UTF-8"></head>
+<body style="font-family:Arial,sans-serif;background:#f4f4f5;margin:0;padding:32px;">
+  <div style="max-width:480px;margin:0 auto;background:#fff;border-radius:12px;padding:32px;box-shadow:0 2px 8px rgba(0,0,0,.08);">
+    <h1 style="color:#4f46e5;font-size:24px;margin-bottom:8px;">Welcome to DocuElevate, {username}!</h1>
+    <p style="color:#374151;">Thanks for signing up. Please confirm your email address to activate your account.</p>
+    <div style="text-align:center;margin:32px 0;">
+      <a href="{verify_url}"
+         style="display:inline-block;background:#4f46e5;color:#fff;text-decoration:none;padding:14px 32px;border-radius:8px;font-weight:600;font-size:16px;">
+        Confirm my email address
+      </a>
+    </div>
+    <p style="color:#6b7280;font-size:13px;">This link expires in 24 hours. If you did not create an account, you can safely ignore this email.</p>
+    <hr style="border:none;border-top:1px solid #e5e7eb;margin:24px 0;">
+    <p style="color:#9ca3af;font-size:12px;text-align:center;">DocuElevate &middot; Intelligent Document Processing</p>
+  </div>
+</body>
+</html>"""
+    plain_body = (
+        f"Welcome to DocuElevate, {username}!\n\n"
+        f"Please verify your email address by visiting:\n{verify_url}\n\n"
+        "This link expires in 24 hours."
+    )
+    _smtp_send(subject, html_body, plain_body, email)
+
+
+def send_password_reset_email(email: str, username: str, token: str, base_url: str) -> None:
+    """Send a password reset email to *email*.
+
+    Args:
+        email: Recipient email address.
+        username: The user's username (used in greeting).
+        token: The password reset token to embed in the link.
+        base_url: The base URL of the application.
+    """
+    reset_url = f"{base_url}/reset-password?token={token}"
+    subject = "Reset your DocuElevate password"
+    html_body = f"""<!DOCTYPE html>
+<html lang="en">
+<head><meta charset="UTF-8"></head>
+<body style="font-family:Arial,sans-serif;background:#f4f4f5;margin:0;padding:32px;">
+  <div style="max-width:480px;margin:0 auto;background:#fff;border-radius:12px;padding:32px;box-shadow:0 2px 8px rgba(0,0,0,.08);">
+    <h1 style="color:#4f46e5;font-size:24px;margin-bottom:8px;">Password Reset</h1>
+    <p style="color:#374151;">Hi {username}, you requested a password reset for your DocuElevate account.</p>
+    <div style="text-align:center;margin:32px 0;">
+      <a href="{reset_url}"
+         style="display:inline-block;background:#4f46e5;color:#fff;text-decoration:none;padding:14px 32px;border-radius:8px;font-weight:600;font-size:16px;">
+        Reset my password
+      </a>
+    </div>
+    <p style="color:#6b7280;font-size:13px;">This link expires in 24 hours. If you did not request a password reset, you can safely ignore this email.</p>
+    <hr style="border:none;border-top:1px solid #e5e7eb;margin:24px 0;">
+    <p style="color:#9ca3af;font-size:12px;text-align:center;">DocuElevate &middot; Intelligent Document Processing</p>
+  </div>
+</body>
+</html>"""
+    plain_body = (
+        f"Hi {username},\n\n"
+        f"You requested a password reset. Visit the link below:\n{reset_url}\n\n"
+        "This link expires in 24 hours. If you did not request this, ignore this email."
+    )
+    _smtp_send(subject, html_body, plain_body, email)
+
+
+def send_forgot_username_email(email: str, username: str) -> None:
+    """Send an email reminding the user of their username.
+
+    Args:
+        email: Recipient email address.
+        username: The user's username to include in the message.
+    """
+    subject = "Your DocuElevate username"
+    html_body = f"""<!DOCTYPE html>
+<html lang="en">
+<head><meta charset="UTF-8"></head>
+<body style="font-family:Arial,sans-serif;background:#f4f4f5;margin:0;padding:32px;">
+  <div style="max-width:480px;margin:0 auto;background:#fff;border-radius:12px;padding:32px;box-shadow:0 2px 8px rgba(0,0,0,.08);">
+    <h1 style="color:#4f46e5;font-size:24px;margin-bottom:8px;">Your Username</h1>
+    <p style="color:#374151;">You requested a reminder of your DocuElevate username.</p>
+    <div style="text-align:center;margin:32px 0;background:#f3f4f6;border-radius:8px;padding:20px;">
+      <p style="color:#6b7280;font-size:13px;margin-bottom:4px;">Your username is:</p>
+      <p style="color:#111827;font-size:22px;font-weight:700;font-family:monospace;">{username}</p>
+    </div>
+    <p style="color:#374151;font-size:14px;">You can sign in using your username <strong>or</strong> your email address.</p>
+    <p style="color:#6b7280;font-size:13px;margin-top:16px;">If you did not request this reminder, you can safely ignore this email.</p>
+    <hr style="border:none;border-top:1px solid #e5e7eb;margin:24px 0;">
+    <p style="color:#9ca3af;font-size:12px;text-align:center;">DocuElevate &middot; Intelligent Document Processing</p>
+  </div>
+</body>
+</html>"""
+    plain_body = (
+        f"You requested a reminder of your DocuElevate username.\n\n"
+        f"Your username is: {username}\n\n"
+        "You can sign in using your username or your email address.\n\n"
+        "If you did not request this, please ignore this email."
+    )
+    _smtp_send(subject, html_body, plain_body, email)
+
+
+def build_session_user(user: object) -> dict:
+    """Build the session user dict for a LocalUser, matching the OAuth session format.
+
+    Args:
+        user: A ``LocalUser`` ORM instance.
+
+    Returns:
+        Dict suitable for storing in ``request.session["user"]``.
+    """
+    from app.auth import get_gravatar_url
+
+    return {
+        "sub": user.email,  # type: ignore[attr-defined]
+        "id": user.email,  # type: ignore[attr-defined]
+        "email": user.email,  # type: ignore[attr-defined]
+        "preferred_username": user.username,  # type: ignore[attr-defined]
+        "name": user.display_name or user.username,  # type: ignore[attr-defined]
+        "picture": get_gravatar_url(user.email),  # type: ignore[attr-defined]
+        "is_admin": bool(user.is_admin),  # type: ignore[attr-defined]
+        "auth_method": "local",
+    }

app/utils/notification.py

@@ -204,3 +204,104 @@ The file has been successfully processed and is being uploaded to all configured
     return send_notification(
         title=title, message=message.strip(), notification_type="success", tags=["document", "processed", "success"]
     )
+
+
+def notify_user_signup(user_id: str, display_name: str | None = None, email: str | None = None) -> bool:
+    """Send a notification to admins when a new user signs up.
+
+    Args:
+        user_id: The stable user identifier (preferred_username / email / sub).
+        display_name: Optional human-readable name for the user.
+        email: Optional email address for the user.
+
+    Returns:
+        bool: True if the notification was sent successfully.
+    """
+    if not settings.notify_on_user_signup:
+        return False
+
+    name_str = display_name or user_id
+    email_str = email or "N/A"
+
+    title = f"New User Signup: {name_str}"
+    message = f"""A new user has signed up for DocuElevate.
+
+User ID: {user_id}
+Display Name: {name_str}
+Email: {email_str}
+
+Review the new account in the admin panel."""
+
+    return send_notification(
+        title=title,
+        message=message.strip(),
+        notification_type="info",
+        tags=["user", "signup"],
+    )
+
+
+def notify_plan_changed(
+    user_id: str,
+    old_tier: str,
+    new_tier: str,
+    changed_by: str = "user",
+) -> bool:
+    """Send a notification to admins when a user changes their subscription plan.
+
+    Args:
+        user_id: The stable user identifier.
+        old_tier: The previous subscription tier.
+        new_tier: The new subscription tier.
+        changed_by: Who initiated the change (``"user"`` or ``"admin"``).
+
+    Returns:
+        bool: True if the notification was sent successfully.
+    """
+    if not settings.notify_on_plan_change:
+        return False
+
+    title = f"Plan Changed: {user_id}"
+    message = f"""A user's subscription plan has changed.
+
+User ID: {user_id}
+Previous Plan: {old_tier}
+New Plan: {new_tier}
+Changed By: {changed_by}
+
+Review the account in the admin panel."""
+
+    return send_notification(
+        title=title,
+        message=message.strip(),
+        notification_type="info",
+        tags=["user", "plan", "subscription"],
+    )
+
+
+def notify_payment_issue(user_id: str, issue: str) -> bool:
+    """Send a notification to admins when a payment issue is reported for a user.
+
+    Args:
+        user_id: The stable user identifier.
+        issue: A human-readable description of the payment issue.
+
+    Returns:
+        bool: True if the notification was sent successfully.
+    """
+    if not settings.notify_on_payment_issue:
+        return False
+
+    title = f"Payment Issue: {user_id}"
+    message = f"""A payment issue has been reported for a user.
+
+User ID: {user_id}
+Issue: {issue}
+
+Please review the account in the admin panel and follow up with the user."""
+
+    return send_notification(
+        title=title,
+        message=message.strip(),
+        notification_type="warning",
+        tags=["user", "payment", "billing"],
+    )

app/utils/similarity.py

@@ -0,0 +1,270 @@
+"""Document similarity detection using text embeddings and cosine similarity.
+
+Provides functions to generate text embeddings via the configured AI provider
+(OpenAI-compatible) and compute cosine similarity scores between documents.
+Embeddings are cached in the ``FileRecord.embedding`` column to avoid
+redundant API calls.
+"""
+
+import json
+import logging
+import math
+from typing import Any
+
+from sqlalchemy.orm import Session
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+
+def _get_embedding_client() -> Any:
+    """Create an OpenAI client for embedding generation.
+
+    Returns:
+        An ``openai.OpenAI`` client instance configured from application settings.
+
+    Raises:
+        RuntimeError: If the ``openai`` package is not installed.
+    """
+    try:
+        import openai
+    except ImportError as exc:
+        raise RuntimeError("The 'openai' package is required for embedding generation") from exc
+
+    return openai.OpenAI(
+        api_key=settings.openai_api_key,
+        base_url=settings.openai_base_url,
+    )
+
+
+def generate_embedding(text: str, model: str | None = None) -> list[float]:
+    """Generate a text embedding vector using the OpenAI-compatible API.
+
+    Args:
+        text: The input text to embed.  Truncated to stay within the
+            model's context window based on ``settings.embedding_max_tokens``
+            (default 8 000 tokens ≈ 24 000 characters).
+        model: The embedding model to use.  When ``None`` (the default), the
+            value of ``settings.embedding_model`` is used.
+
+    Returns:
+        A list of floats representing the embedding vector.
+
+    Raises:
+        RuntimeError: If the OpenAI client cannot be created.
+        Exception: If the API call fails.
+    """
+    if model is None:
+        model = settings.embedding_model
+
+    # Truncate to stay within the model's context window.
+    # Conservative 3 chars/token estimate (actual ratio varies by language;
+    # English averages ~4 chars/token but 3 gives a safety margin).
+    max_chars = settings.embedding_max_tokens * 3
+    if len(text) > max_chars:
+        logger.debug(
+            "Truncating text from %d to %d chars (~%d tokens) for model %s",
+            len(text),
+            max_chars,
+            settings.embedding_max_tokens,
+            model,
+        )
+        text = text[:max_chars]
+
+    client = _get_embedding_client()
+    logger.debug("Generating embedding for %d chars using model=%s", len(text), model)
+    response = client.embeddings.create(input=text, model=model)
+    return response.data[0].embedding
+
+
+def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
+    """Compute cosine similarity between two vectors.
+
+    Args:
+        vec_a: First embedding vector.
+        vec_b: Second embedding vector.
+
+    Returns:
+        A similarity score between 0 and 1.  Returns 0.0 if either vector
+        has zero magnitude.
+    """
+    if len(vec_a) != len(vec_b):
+        return 0.0
+
+    dot_product = sum(a * b for a, b in zip(vec_a, vec_b, strict=True))
+    magnitude_a = math.sqrt(sum(a * a for a in vec_a))
+    magnitude_b = math.sqrt(sum(b * b for b in vec_b))
+
+    if magnitude_a == 0.0 or magnitude_b == 0.0:
+        return 0.0
+
+    similarity = dot_product / (magnitude_a * magnitude_b)
+    # Clamp to [0, 1] to handle floating-point drift
+    return max(0.0, min(1.0, similarity))
+
+
+def _get_cached_embedding(file_record: Any) -> list[float] | None:
+    """Return the cached embedding for a file record, or ``None``.
+
+    This is a **read-only** helper — it never triggers an API call.  Use
+    :func:`compute_and_store_embedding` when you need to generate a new
+    embedding.
+
+    Args:
+        file_record: A ``FileRecord`` instance (or any object with ``id``
+            and ``embedding`` attributes).
+
+    Returns:
+        The parsed embedding vector, or ``None`` if no valid cached
+        embedding exists.
+    """
+    raw = file_record.embedding if hasattr(file_record, "embedding") else None
+    if not raw:
+        return None
+    try:
+        cached = json.loads(raw)
+        logger.debug("Using cached embedding for file %s (%d dimensions)", file_record.id, len(cached))
+        return cached
+    except (json.JSONDecodeError, TypeError):
+        logger.warning("Invalid cached embedding for file %s", file_record.id)
+        return None
+
+
+def compute_and_store_embedding(db: Session, file_record: Any) -> list[float] | None:
+    """Generate an embedding for a file and persist it in the database.
+
+    Called during document ingestion (Celery task) or via the manual
+    ``POST /api/files/{id}/compute-embedding`` debug endpoint.  The
+    similarity query path (:func:`find_similar_documents`) intentionally
+    does **not** call this — it only reads pre-computed embeddings so
+    that it returns instantly without blocking on external API calls.
+
+    Args:
+        db: Active database session.
+        file_record: A ``FileRecord`` instance.
+
+    Returns:
+        The embedding vector, or ``None`` if the document has no OCR text
+        or embedding generation fails.
+    """
+    # Return cached embedding if already present
+    if file_record.embedding:
+        try:
+            cached = json.loads(file_record.embedding)
+            logger.debug("Embedding already cached for file %s (%d dims)", file_record.id, len(cached))
+            return cached
+        except (json.JSONDecodeError, TypeError):
+            logger.warning("Invalid cached embedding for file %s, recomputing", file_record.id)
+
+    # Need OCR text to generate an embedding
+    if not file_record.ocr_text or not file_record.ocr_text.strip():
+        logger.debug("No OCR text for file %s, cannot generate embedding", file_record.id)
+        return None
+
+    try:
+        logger.info("Computing embedding for file %s (%d chars of OCR text)", file_record.id, len(file_record.ocr_text))
+        embedding = generate_embedding(file_record.ocr_text)
+        # Persist in the database
+        file_record.embedding = json.dumps(embedding)
+        db.commit()
+        logger.info("Embedding computed and cached for file %s (%d dimensions)", file_record.id, len(embedding))
+        return embedding
+    except Exception as e:
+        db.rollback()
+        logger.error("Failed to generate embedding for file %s: %s", file_record.id, e)
+        return None
+
+
+# Keep the legacy alias so that existing callers (e.g. tests) keep working.
+_get_or_compute_embedding = compute_and_store_embedding
+
+
+def find_similar_documents(
+    db: Session,
+    file_id: int,
+    limit: int = 5,
+    threshold: float = 0.3,
+) -> list[dict[str, Any]]:
+    """Find documents similar to the given file using **pre-computed** embeddings.
+
+    Only documents whose embeddings were already generated (during
+    ingestion or via the debug endpoint) are considered.  No external API
+    calls are made — the function reads cached vectors from the database
+    and computes cosine similarity in-process.
+
+    To keep memory usage bounded for large corpora (100 k+ documents) the
+    candidate query fetches only the columns needed for scoring and
+    iterates in chunks via ``yield_per``.
+
+    Args:
+        db: Active database session.
+        file_id: The ID of the target ``FileRecord``.
+        limit: Maximum number of similar documents to return.
+        threshold: Minimum similarity score (0–1) to include in results.
+
+    Returns:
+        A list of dicts, each containing:
+        - ``file_id``: The similar document's ID.
+        - ``original_filename``: The document's original filename.
+        - ``document_title``: The document's AI-extracted title (may be None).
+        - ``similarity_score``: Cosine similarity (0–1, rounded to 4 decimals).
+        - ``mime_type``: The document's MIME type.
+        - ``created_at``: ISO-formatted creation timestamp.
+    """
+    from app.models import FileRecord
+
+    # Get the target document's cached embedding (read-only, no API call)
+    target = db.query(FileRecord).filter(FileRecord.id == file_id).first()
+    if not target:
+        return []
+
+    target_embedding = _get_cached_embedding(target)
+    if not target_embedding:
+        logger.info("No cached embedding for target file %s — skipping similarity search", file_id)
+        return []
+
+    # Query only candidates that already have a pre-computed embedding.
+    # Fetch only the columns needed for scoring to minimise memory use.
+    # yield_per streams rows in chunks so we never materialise all 100k+
+    # records at once.
+    candidates = (
+        db.query(
+            FileRecord.id,
+            FileRecord.original_filename,
+            FileRecord.document_title,
+            FileRecord.mime_type,
+            FileRecord.created_at,
+            FileRecord.embedding,
+        )
+        .filter(
+            FileRecord.id != file_id,
+            FileRecord.embedding.isnot(None),
+            FileRecord.embedding != "",
+        )
+        .yield_per(500)
+    )
+
+    results: list[dict[str, Any]] = []
+    for row in candidates:
+        try:
+            candidate_embedding: list[float] = json.loads(row.embedding)
+        except (json.JSONDecodeError, TypeError):
+            continue
+
+        score = cosine_similarity(target_embedding, candidate_embedding)
+        if score >= threshold:
+            results.append(
+                {
+                    "file_id": row.id,
+                    "original_filename": row.original_filename,
+                    "document_title": row.document_title,
+                    "similarity_score": round(score, 4),
+                    "mime_type": row.mime_type,
+                    "created_at": row.created_at.isoformat() if row.created_at else None,
+                }
+            )
+
+    # Sort by similarity score descending
+    results.sort(key=lambda x: x["similarity_score"], reverse=True)
+    return results[:limit]

app/utils/step_manager.py

@@ -23,6 +23,7 @@ BASE_MAIN_PROCESSING_STEPS = [
     "embed_metadata_into_pdf",
     "finalize_document_storage",
     "send_to_all_destinations",
+    "compute_embedding",
 ]
 
 OPTIONAL_PROCESSING_STEPS = {
@@ -217,6 +218,7 @@ def get_file_overall_status(db: Session, file_id: int) -> Dict:
         "embed_metadata_into_pdf",
         "finalize_document_storage",
         "send_to_all_destinations",
+        "compute_embedding",
     }
 
     # Add check_for_duplicates if deduplication is enabled
@@ -325,6 +327,7 @@ def get_step_summary(db: Session, file_id: int) -> Dict:
         "embed_metadata_into_pdf",
         "finalize_document_storage",
         "send_to_all_destinations",
+        "compute_embedding",
     }
 
     # Add check_for_duplicates if deduplication is enabled

app/utils/subscription.py

@@ -0,0 +1,780 @@
+"""
+Subscription tier definitions and enforcement utilities for DocuElevate SaaS.
+
+All plans are priced per user per month (or per year with ~20 % discount).
+Four tiers (prices ex-VAT; German customers +19 % MwSt):
+  - free         $0/mo    — 50 lifetime docs, 150 lifetime OCR pages, 1 dest
+  - starter      $2.99/mo — 50/mo, 300 OCR pp/mo, 2 dests, 1 mailbox
+  - professional $5.99/mo — 150/mo, 750 OCR pp/mo, 5 dests, 3 mailboxes
+  - power        $7.99/mo — 300/mo, 1500 OCR pp/mo, 10 dests, unlimited mailboxes
+
+Limits use 0 to represent "unlimited".
+All paid tiers include a 30-day free trial (trial_days field).
+
+--- Cost analysis at maximum usage (Hetzner Option-A infra, Azure Read + GPT-4o mini) ---
+Infrastructure: CX32 (app+Redis €7.59) + CX22 (worker €3.79) + BX21 (storage €7.22) ≈ $24/mo
+At 100 users infra share ≈ $0.24/user/mo.
+
+  Starter     : OCR $0.45 + AI $0.012 + infra $0.24 + Stripe $0.34 = $1.04 → 65 % gross margin
+  Professional: OCR $1.13 + AI $0.035 + infra $0.24 + Stripe $0.42 = $1.82 → 70 % gross margin
+  Power       : OCR $2.25 + AI $0.069 + infra $0.24 + Stripe $0.48 = $3.04 → 62 % gross margin
+
+After ~30 % German corporate tax: Starter 45 %, Professional 49 %, Power 43 %.
+At average usage (~40 % of quota) margins improve to 55-65 % after tax.
+
+⚠ If GPT-4o (not mini) is configured, Power AI cost at max rises to ~$1.92/user,
+  reducing after-tax margin to ~33 %. Recommend GPT-4o mini as default in production.
+"""
+
+from __future__ import annotations
+
+import logging
+from datetime import date, datetime, timezone
+from typing import Any
+
+from sqlalchemy import func
+from sqlalchemy.orm import Session
+
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Tier catalogue
+# ---------------------------------------------------------------------------
+
+TIER_DEFAULTS: dict[str, dict[str, Any]] = {
+    "free": {
+        "id": "free",
+        "name": "Free",
+        "tagline": "Try DocuElevate free — no credit card needed",
+        "price_monthly": 0,
+        "price_yearly": 0,
+        "trial_days": 0,
+        "highlight": False,
+        # Hard caps — 0 = unlimited
+        "lifetime_file_limit": 50,  # total docs ever processed (enforced at upload)
+        "daily_upload_limit": 0,  # no per-day cap (lifetime cap applies instead)
+        "monthly_upload_limit": 0,  # no per-month cap (lifetime cap applies instead)
+        "max_storage_destinations": 1,
+        "max_ocr_pages_monthly": 150,  # informational; enforced when OCR quota tracking lands
+        "max_file_size_mb": 5,
+        "max_mailboxes": 0,  # no email ingestion on free tier
+        "api_access": False,
+        # Marketing feature list (shown on pricing page)
+        "features": [
+            "50 documents — lifetime total",
+            "150 OCR pages — lifetime total",
+            "1 storage destination",
+            "5 MB max file size",
+            "Basic AI metadata extraction",
+            "Community support",
+        ],
+        "cta": "Get started free",
+        "badge": None,
+    },
+    "starter": {
+        "id": "starter",
+        "name": "Starter",
+        # Use case: freelancer sending ~50 invoices, contracts, or scanned receipts a month
+        "tagline": "Perfect for freelancers and side-project owners",
+        "price_monthly": 2.99,
+        "price_yearly": 28.99,  # ≈ 80 % of monthly × 12 — save ~19 % (≈ 2½ months free)
+        "trial_days": 30,
+        "highlight": False,
+        "lifetime_file_limit": 0,
+        "daily_upload_limit": 0,  # no daily cap
+        "monthly_upload_limit": 50,
+        "max_storage_destinations": 2,
+        "max_ocr_pages_monthly": 300,
+        "max_file_size_mb": 25,
+        "max_mailboxes": 1,
+        "api_access": True,
+        "features": [
+            "50 documents / month — invoices, contracts, receipts",
+            "2 storage destinations",
+            "300 OCR pages / month",
+            "25 MB max file size",
+            "Full AI metadata extraction",
+            "1 email ingestion mailbox",
+            "API access",
+            "Email support",
+        ],
+        "cta": "Start free trial",
+        "badge": None,
+    },
+    "professional": {
+        "id": "professional",
+        "name": "Professional",
+        # Use case: consultant or knowledge worker handling ~150 docs/month across multiple platforms
+        "tagline": "For knowledge workers managing documents daily",
+        "price_monthly": 5.99,
+        "price_yearly": 57.99,  # ≈ 80 % of monthly × 12 — save ~19 %
+        "trial_days": 30,
+        "highlight": True,  # shown as "Most popular"
+        "lifetime_file_limit": 0,
+        "daily_upload_limit": 0,  # no daily cap
+        "monthly_upload_limit": 150,
+        "max_storage_destinations": 5,
+        "max_ocr_pages_monthly": 750,
+        "max_file_size_mb": 100,
+        "max_mailboxes": 3,
+        "api_access": True,
+        "features": [
+            "150 documents / month — reports, contracts, invoices",
+            "5 storage destinations",
+            "750 OCR pages / month",
+            "100 MB max file size",
+            "Advanced AI workflows",
+            "3 email ingestion mailboxes",
+            "Email & URL ingestion",
+            "Webhooks",
+            "Priority email support",
+        ],
+        "cta": "Start free trial",
+        "badge": "Most Popular",
+    },
+    "business": {
+        "id": "business",
+        "name": "Power",
+        # Use case: power user — real estate agent, bookkeeper, or researcher processing ~10 docs/day
+        "tagline": "For power users with high-volume document workflows",
+        "price_monthly": 7.99,
+        "price_yearly": 76.99,  # ≈ 80 % of monthly × 12 — save ~20 %
+        "trial_days": 30,
+        "highlight": False,
+        "lifetime_file_limit": 0,
+        "daily_upload_limit": 0,  # no daily cap
+        "monthly_upload_limit": 300,
+        "max_storage_destinations": 10,
+        "max_ocr_pages_monthly": 1500,
+        "max_file_size_mb": 0,  # unlimited file size
+        "max_mailboxes": 0,  # unlimited mailboxes
+        "api_access": True,
+        "features": [
+            "300 documents / month — ~10 documents per day",
+            "10 storage destinations",
+            "1,500 OCR pages / month",
+            "Unlimited file size",
+            "All AI processing steps",
+            "Unlimited email ingestion mailboxes",
+            "All ingestion methods",
+            "Webhooks & full API access",
+            "Priority support",
+        ],
+        "cta": "Start free trial",
+        "badge": "Best Value",
+    },
+}
+
+# Backward-compatible alias
+TIERS = TIER_DEFAULTS
+
+# Display order for the pricing page
+TIER_ORDER = ["free", "starter", "professional", "business"]
+
+# Default tier assigned to new users
+DEFAULT_TIER = "free"
+
+
+# ---------------------------------------------------------------------------
+# DB → dict conversion
+# ---------------------------------------------------------------------------
+
+
+def _plan_to_dict(plan: Any) -> dict[str, Any]:
+    """Convert a SubscriptionPlan ORM object to the same dict shape as TIER_DEFAULTS entries."""
+    import json
+
+    features: list[str] = []
+    if plan.features:
+        try:
+            features = json.loads(plan.features)
+        except (json.JSONDecodeError, TypeError):
+            features = []
+    return {
+        "id": plan.plan_id,
+        "name": plan.name,
+        "tagline": plan.tagline or "",
+        "price_monthly": plan.price_monthly,
+        "price_yearly": plan.price_yearly,
+        "trial_days": plan.trial_days,
+        "highlight": plan.is_highlighted,
+        "lifetime_file_limit": plan.lifetime_file_limit,
+        "daily_upload_limit": plan.daily_upload_limit,
+        "monthly_upload_limit": plan.monthly_upload_limit,
+        "max_storage_destinations": plan.max_storage_destinations,
+        "max_ocr_pages_monthly": plan.max_ocr_pages_monthly,
+        "max_file_size_mb": plan.max_file_size_mb,
+        "max_mailboxes": plan.max_mailboxes,
+        "api_access": plan.api_access,
+        "features": features,
+        "cta": plan.cta_text or "Get started",
+        "badge": plan.badge_text,
+        "overage_percent": plan.overage_percent,
+        "allow_overage_billing": plan.allow_overage_billing,
+    }
+
+
+# ---------------------------------------------------------------------------
+# Getters
+# ---------------------------------------------------------------------------
+
+
+def get_tier(tier_id: str, db: Session | None = None) -> dict[str, Any]:
+    """Return plan config dict; DB-first when db is provided, falls back to TIER_DEFAULTS."""
+    if db is not None:
+        from app.models import SubscriptionPlan
+
+        plan = (
+            db.query(SubscriptionPlan)
+            .filter(
+                SubscriptionPlan.plan_id == tier_id,
+                SubscriptionPlan.is_active.is_(True),
+            )
+            .first()
+        )
+        if plan is not None:
+            return _plan_to_dict(plan)
+    return TIER_DEFAULTS.get(tier_id, TIER_DEFAULTS["free"])
+
+
+def get_all_tiers(db: Session | None = None) -> list[dict[str, Any]]:
+    """Return plans in display order; DB-first when db is provided."""
+    if db is not None:
+        from app.models import SubscriptionPlan
+
+        plans = (
+            db.query(SubscriptionPlan)
+            .filter(SubscriptionPlan.is_active.is_(True))
+            .order_by(SubscriptionPlan.sort_order)
+            .all()
+        )
+        if plans:
+            return [_plan_to_dict(p) for p in plans]
+    return [TIER_DEFAULTS[tid] for tid in TIER_ORDER]
+
+
+def seed_default_plans(db: Session) -> int:
+    """Seed subscription_plans table from TIER_DEFAULTS if the table is empty.
+
+    Called at application startup. Returns the number of plans inserted (0 if already seeded).
+    """
+    import json
+
+    from app.models import SubscriptionPlan
+
+    try:
+        if db.query(SubscriptionPlan).count() > 0:
+            return 0
+    except Exception:
+        return 0  # table may not exist yet during first migration
+
+    inserted = 0
+    for sort_order, (_, tier) in enumerate(TIER_DEFAULTS.items()):
+        plan = SubscriptionPlan(
+            plan_id=tier["id"],
+            name=tier["name"],
+            tagline=tier.get("tagline", ""),
+            price_monthly=tier["price_monthly"],
+            price_yearly=tier["price_yearly"],
+            trial_days=tier.get("trial_days", 0),
+            is_highlighted=tier.get("highlight", False),
+            badge_text=tier.get("badge"),
+            cta_text=tier.get("cta", "Get started"),
+            lifetime_file_limit=tier["lifetime_file_limit"],
+            daily_upload_limit=tier["daily_upload_limit"],
+            monthly_upload_limit=tier["monthly_upload_limit"],
+            max_storage_destinations=tier["max_storage_destinations"],
+            max_ocr_pages_monthly=tier["max_ocr_pages_monthly"],
+            max_file_size_mb=tier["max_file_size_mb"],
+            max_mailboxes=tier.get("max_mailboxes", 0),
+            api_access=tier.get("api_access", False),
+            features=json.dumps(tier.get("features", [])),
+            overage_percent=20,
+            allow_overage_billing=False,
+            sort_order=sort_order,
+            is_active=True,
+        )
+        db.add(plan)
+        inserted += 1
+    try:
+        db.commit()
+        logger.info("Seeded %d default subscription plans", inserted)
+    except Exception as exc:
+        db.rollback()
+        logger.error("Failed to seed subscription plans: %s", exc)
+        inserted = 0
+    return inserted
+
+
+# ---------------------------------------------------------------------------
+# Usage queries
+# ---------------------------------------------------------------------------
+
+
+def _today_utc() -> date:
+    return datetime.now(timezone.utc).date()
+
+
+def _scalar_count(query: Any) -> int:
+    """Execute a count query and return an int, defaulting to 0 for NULL."""
+    return query.scalar() or 0
+
+
+def get_lifetime_file_count(db: Session, owner_id: str) -> int:
+    """Total files ever processed by this user (not counting duplicates)."""
+    from app.models import FileRecord
+
+    return _scalar_count(
+        db.query(func.count(FileRecord.id)).filter(FileRecord.owner_id == owner_id, FileRecord.is_duplicate.is_(False))
+    )
+
+
+def get_today_file_count(db: Session, owner_id: str) -> int:
+    """Files processed by this user today (UTC, not counting duplicates)."""
+    from app.models import FileRecord
+
+    today = _today_utc()
+    return _scalar_count(
+        db.query(func.count(FileRecord.id)).filter(
+            FileRecord.owner_id == owner_id,
+            FileRecord.is_duplicate.is_(False),
+            func.date(FileRecord.created_at) == today,
+        )
+    )
+
+
+def get_month_file_count(db: Session, owner_id: str) -> int:
+    """Files processed by this user this calendar month (UTC, not counting duplicates)."""
+    from app.models import FileRecord
+
+    today = _today_utc()
+    return _scalar_count(
+        db.query(func.count(FileRecord.id)).filter(
+            FileRecord.owner_id == owner_id,
+            FileRecord.is_duplicate.is_(False),
+            func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"),
+        )
+    )
+
+
+def get_year_file_count(db: Session, owner_id: str, period_start: datetime) -> int:
+    """Files processed since the start of the current annual subscription period."""
+    from app.models import FileRecord
+
+    return _scalar_count(
+        db.query(func.count(FileRecord.id)).filter(
+            FileRecord.owner_id == owner_id,
+            FileRecord.is_duplicate.is_(False),
+            FileRecord.created_at >= period_start,
+        )
+    )
+
+
+def _months_elapsed(period_start: datetime, now: datetime) -> int:
+    """Calendar months elapsed since *period_start*, clamped to [1, 12]."""
+    elapsed = (now.year - period_start.year) * 12 + (now.month - period_start.month) + 1
+    return max(1, min(elapsed, 12))
+
+
+# ---------------------------------------------------------------------------
+# Limit enforcement
+# ---------------------------------------------------------------------------
+
+
+class QuotaExceeded(Exception):
+    """Raised when a user has hit a subscription limit."""
+
+    def __init__(self, message: str, limit_type: str, limit_value: int, current_value: int) -> None:
+        super().__init__(message)
+        self.limit_type = limit_type
+        self.limit_value = limit_value
+        self.current_value = current_value
+
+
+def check_upload_allowed(db: Session, owner_id: str | None, tier_id: str | None) -> None:
+    """Raise :class:`QuotaExceeded` if this user is not allowed to upload another file.
+
+    Skipped entirely when *owner_id* or *tier_id* is ``None`` (single-user mode).
+
+    Enforcement model
+    -----------------
+    * **Announced limit** — the quota shown on the pricing page
+      (``monthly_upload_limit`` in the plan).
+    * **Overage buffer** — each plan stores ``overage_percent`` (default 20).
+      Enforcement = announced × (1 + overage_percent / 100). A 150-doc/month
+      plan with 20 % buffer is enforced at 180 docs.
+    * **Overage flag** — if ``UserProfile.allow_overage`` is ``True``, quota
+      checks are bypassed entirely so usage can be billed retroactively.
+      (Not yet exposed in the admin UI — baked in for future billing.)
+    * **Yearly carry-over** — yearly subscribers have cumulative quota:
+      effective limit = monthly_limit × months_elapsed × overage_factor.
+      Unused quota from earlier months rolls forward automatically.
+    * **No daily cap** — ``daily_upload_limit`` is kept for display purposes
+      only; it is never enforced.
+    """
+    if owner_id is None or tier_id is None:
+        return
+
+    tier = get_tier(tier_id, db)
+
+    # Per-plan overage_percent overrides global config default
+    overage_percent: int = tier.get("overage_percent", settings.subscription_overage_percent)
+    overage_factor: float = 1.0 + overage_percent / 100.0
+
+    from app.models import UserProfile
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    allow_overage: bool = bool(profile.allow_overage) if profile else False
+    billing_cycle: str = (profile.subscription_billing_cycle if profile else None) or "monthly"
+    period_start: datetime | None = profile.subscription_period_start if profile else None
+
+    # 1. Lifetime file cap (free tier) — always enforced regardless of overage flag
+    lifetime_limit: int = tier["lifetime_file_limit"]
+    if lifetime_limit > 0:
+        enforcement_limit = int(lifetime_limit * overage_factor)
+        count = get_lifetime_file_count(db, owner_id)
+        if count >= enforcement_limit:
+            raise QuotaExceeded(
+                f"Lifetime file limit of {lifetime_limit} reached for the {tier['name']} plan. "
+                "Please upgrade to continue processing documents.",
+                limit_type="lifetime",
+                limit_value=lifetime_limit,
+                current_value=count,
+            )
+
+    # 2. Monthly cap — bypassed when allow_overage is True (future billing)
+    if allow_overage:
+        return
+
+    monthly_limit: int = tier["monthly_upload_limit"]
+    if monthly_limit > 0:
+        if billing_cycle == "yearly" and period_start is not None:
+            now = datetime.now(timezone.utc)
+            months = _months_elapsed(period_start, now)
+            cumulative_budget = int(monthly_limit * months * overage_factor)
+            cumulative_used = get_year_file_count(db, owner_id, period_start)
+            if cumulative_used >= cumulative_budget:
+                raise QuotaExceeded(
+                    f"Annual document quota for the {tier['name']} plan has been reached. "
+                    "Unused monthly quota carries forward — your limit resets on your annual "
+                    "renewal date, or you can upgrade your plan.",
+                    limit_type="monthly",
+                    limit_value=monthly_limit,
+                    current_value=cumulative_used,
+                )
+        else:
+            count = get_month_file_count(db, owner_id)
+            enforcement_limit = int(monthly_limit * overage_factor)
+            if count >= enforcement_limit:
+                raise QuotaExceeded(
+                    f"Monthly file limit of {monthly_limit} reached for the {tier['name']} plan. "
+                    "Please upgrade your plan for more documents this month.",
+                    limit_type="monthly",
+                    limit_value=monthly_limit,
+                    current_value=count,
+                )
+
+
+def get_user_tier_id(db: Session, owner_id: str | None) -> str:
+    """Return the subscription tier id for *owner_id*, defaulting to 'free'."""
+    if owner_id is None:
+        return DEFAULT_TIER
+    from app.models import UserProfile
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if profile and profile.subscription_tier:
+        return profile.subscription_tier
+    return DEFAULT_TIER
+
+
+def get_user_usage(db: Session, owner_id: str) -> dict[str, int]:
+    """Return file counts for *owner_id*, including carry-over data for yearly plans."""
+    from app.models import UserProfile
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    result: dict[str, int] = {
+        "lifetime": get_lifetime_file_count(db, owner_id),
+        "today": get_today_file_count(db, owner_id),
+        "month": get_month_file_count(db, owner_id),
+    }
+    if profile and (profile.subscription_billing_cycle or "monthly") == "yearly" and profile.subscription_period_start:
+        result["year_to_date"] = get_year_file_count(db, owner_id, profile.subscription_period_start)
+    return result
+
+
+# ---------------------------------------------------------------------------
+# Subscription change management
+# ---------------------------------------------------------------------------
+
+
+class SubscriptionChangeError(Exception):
+    """Raised when a requested subscription change is not permitted."""
+
+
+def _tier_rank(tier_id: str) -> int:
+    """Return the numeric rank of *tier_id* (0 = free … 3 = business).
+
+    Unknown tier IDs are treated as rank 0 (free).
+    """
+    try:
+        return TIER_ORDER.index(tier_id)
+    except ValueError:
+        return 0
+
+
+def apply_pending_subscription_changes(db: Session, owner_id: str) -> bool:
+    """Apply any pending subscription change that is now due.
+
+    Checks whether the scheduled change date has arrived and, if so, applies
+    the new tier immediately.
+
+    Args:
+        db: Database session.
+        owner_id: Stable user identifier.
+
+    Returns:
+        ``True`` if a pending change was applied, ``False`` otherwise.
+    """
+    from app.models import UserProfile
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if not profile:
+        return False
+
+    pending_tier = profile.subscription_change_pending_tier
+    pending_date = profile.subscription_change_pending_date
+    if not pending_tier or not pending_date:
+        return False
+
+    now = datetime.now(timezone.utc)
+    # Normalise pending_date to UTC-aware for comparison
+    if pending_date.tzinfo is None:
+        pending_date = pending_date.replace(tzinfo=timezone.utc)
+
+    if now < pending_date:
+        return False  # Not yet due
+
+    old_tier = profile.subscription_tier or DEFAULT_TIER
+    profile.subscription_tier = pending_tier
+    profile.subscription_period_start = pending_date  # New period started at change date
+    profile.subscription_change_pending_tier = None
+    profile.subscription_change_pending_date = None
+    try:
+        db.commit()
+        logger.info(
+            "Applied pending subscription change for %s: %s → %s",
+            owner_id,
+            old_tier,
+            pending_tier,
+        )
+    except Exception as exc:
+        db.rollback()
+        logger.error("Failed to apply pending subscription change for %s: %s", owner_id, exc)
+        return False
+    return True
+
+
+def request_subscription_change(
+    db: Session,
+    owner_id: str,
+    new_tier_id: str,
+    billing_cycle: str = "monthly",
+) -> dict[str, Any]:
+    """Process a user-initiated subscription change request.
+
+    Upgrade rules
+    -------------
+    Upgrades (moving to a higher-ranked tier) take effect **immediately**:
+    the tier is switched and the period start is reset to *now*.  Any
+    previously scheduled downgrade is cancelled.
+
+    Downgrade rules
+    ---------------
+    Downgrades (moving to a lower-ranked tier) are **always scheduled** for
+    the end of the current billing period:
+
+    * If ``subscription_period_start`` is set and the period end is in the
+      future, the change is queued for that date.
+    * If there is no period start (e.g. admin-assigned tier), the period start
+      is treated as *now* and the change is scheduled one month out.
+    * If the period has already elapsed the change is applied immediately.
+
+    Cancelling a pending downgrade
+    --------------------------------
+    Requesting the *current* tier when there is a pending change cancels that
+    pending change.
+
+    Args:
+        db: Database session.
+        owner_id: Stable user identifier.
+        new_tier_id: Target plan ID (e.g. ``"starter"``).
+        billing_cycle: ``"monthly"`` or ``"yearly"`` — stored on upgrade.
+
+    Returns:
+        A dict with keys ``immediate`` (bool), ``effective_date`` (ISO-8601 str
+        or ``None``), ``old_tier``, ``new_tier``, ``message``.
+
+    Raises:
+        SubscriptionChangeError: If the requested change is not allowed.
+    """
+    from app.models import UserProfile
+
+    now = datetime.now(timezone.utc)
+
+    # Validate target tier
+    valid_ids = [t["id"] for t in get_all_tiers(db)]
+    if new_tier_id not in valid_ids:
+        raise SubscriptionChangeError(f"Unknown subscription plan: {new_tier_id!r}")
+
+    # Ensure profile row exists
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if not profile:
+        profile = UserProfile(user_id=owner_id)
+        db.add(profile)
+        db.flush()
+
+    old_tier_id = profile.subscription_tier or DEFAULT_TIER
+
+    # Cancel pending change when user re-selects their current active tier
+    if new_tier_id == old_tier_id:
+        if profile.subscription_change_pending_tier:
+            profile.subscription_change_pending_tier = None
+            profile.subscription_change_pending_date = None
+            db.commit()
+            return {
+                "immediate": True,
+                "effective_date": None,
+                "old_tier": old_tier_id,
+                "new_tier": old_tier_id,
+                "message": "Pending subscription change cancelled.",
+            }
+        raise SubscriptionChangeError("You are already on this plan.")
+
+    old_rank = _tier_rank(old_tier_id)
+    new_rank = _tier_rank(new_tier_id)
+    is_upgrade = new_rank > old_rank
+
+    if is_upgrade:
+        # Apply immediately — reset period start
+        profile.subscription_tier = new_tier_id
+        profile.subscription_billing_cycle = billing_cycle
+        profile.subscription_period_start = now
+        # Cancel any previously scheduled downgrade
+        profile.subscription_change_pending_tier = None
+        profile.subscription_change_pending_date = None
+        try:
+            db.commit()
+        except Exception as exc:
+            db.rollback()
+            raise SubscriptionChangeError("Failed to apply subscription upgrade.") from exc
+        logger.info("Immediate upgrade for %s: %s → %s", owner_id, old_tier_id, new_tier_id)
+        return {
+            "immediate": True,
+            "effective_date": None,
+            "old_tier": old_tier_id,
+            "new_tier": new_tier_id,
+            "message": f"You have been upgraded to {get_tier(new_tier_id, db)['name']}. "
+            "Your new limits are active immediately.",
+        }
+
+    # --- Downgrade path ---
+    # Determine end of the *first* billing period for the current plan.
+    # Rule: a downgrade is immediate if the user has completed at least one
+    # full month on the current plan; otherwise it is scheduled for the
+    # end of that first month.  This prevents gaming: a user who just
+    # upgraded cannot immediately downgrade to avoid paying the first month.
+    import calendar
+
+    period_start: datetime | None = profile.subscription_period_start
+    if period_start is None:
+        # No recorded start → treat today as start; schedule for one month out
+        period_start = now
+        profile.subscription_period_start = period_start
+
+    if period_start.tzinfo is None:
+        period_start = period_start.replace(tzinfo=timezone.utc)
+
+    # End of the first billing month (same day next month, clamped to valid day)
+    next_month_num = period_start.month % 12 + 1
+    next_year = period_start.year + (1 if period_start.month == 12 else 0)
+    max_day = calendar.monthrange(next_year, next_month_num)[1]
+    next_day = min(period_start.day, max_day)
+    change_date = period_start.replace(year=next_year, month=next_month_num, day=next_day)
+
+    if change_date <= now:
+        profile.subscription_tier = new_tier_id
+        profile.subscription_billing_cycle = billing_cycle
+        profile.subscription_period_start = now
+        profile.subscription_change_pending_tier = None
+        profile.subscription_change_pending_date = None
+        try:
+            db.commit()
+        except Exception as exc:
+            db.rollback()
+            raise SubscriptionChangeError("Failed to apply subscription downgrade.") from exc
+        logger.info("Immediate downgrade for %s: %s → %s (period elapsed)", owner_id, old_tier_id, new_tier_id)
+        return {
+            "immediate": True,
+            "effective_date": None,
+            "old_tier": old_tier_id,
+            "new_tier": new_tier_id,
+            "message": f"Your subscription has been changed to {get_tier(new_tier_id, db)['name']}.",
+        }
+
+    # Schedule the downgrade
+    profile.subscription_change_pending_tier = new_tier_id
+    profile.subscription_change_pending_date = change_date
+    try:
+        db.commit()
+    except Exception as exc:
+        db.rollback()
+        raise SubscriptionChangeError("Failed to schedule subscription downgrade.") from exc
+
+    logger.info(
+        "Scheduled downgrade for %s: %s → %s on %s",
+        owner_id,
+        old_tier_id,
+        new_tier_id,
+        change_date.isoformat(),
+    )
+    return {
+        "immediate": False,
+        "effective_date": change_date.isoformat(),
+        "old_tier": old_tier_id,
+        "new_tier": new_tier_id,
+        "message": (
+            f"Your downgrade to {get_tier(new_tier_id, db)['name']} has been scheduled for "
+            f"{change_date.strftime('%B')} {change_date.day}, {change_date.year}. "
+            "You will continue to have access to your current plan until then."
+        ),
+    }
+
+
+def cancel_pending_subscription_change(db: Session, owner_id: str) -> bool:
+    """Cancel a pending subscription change for *owner_id*.
+
+    Args:
+        db: Database session.
+        owner_id: Stable user identifier.
+
+    Returns:
+        ``True`` if a pending change was cancelled, ``False`` if there was nothing to cancel.
+    """
+    from app.models import UserProfile
+
+    profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+    if not profile or not profile.subscription_change_pending_tier:
+        return False
+
+    profile.subscription_change_pending_tier = None
+    profile.subscription_change_pending_date = None
+    try:
+        db.commit()
+        logger.info("Cancelled pending subscription change for %s", owner_id)
+    except Exception as exc:
+        db.rollback()
+        logger.error("Failed to cancel pending subscription change for %s: %s", owner_id, exc)
+        return False
+    return True

app/utils/suggestion_providers.py

@@ -0,0 +1,451 @@
+"""
+Dynamic suggestion providers for autocomplete-enabled settings.
+
+Each provider function returns a list of strings that are valid values
+for a particular setting.  Providers try to resolve values dynamically
+(e.g. by querying cloud SDKs or scanning installed software) and fall
+back to curated static lists when the runtime environment lacks the
+required libraries, credentials, or connectivity.
+
+**Fallback guarantee**: Every provider wraps its dynamic resolution in a
+``try/except Exception`` so that it *always* returns a usable list.
+Missing libraries (``ImportError``), missing credentials, network
+failures, or unexpected SDK errors all trigger a graceful fallback to
+the bundled static list.
+"""
+
+import logging
+import subprocess  # noqa: S404 — only used with fixed args, no user input
+from typing import Callable
+
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# AWS regions — fetched from boto3 if available
+# ---------------------------------------------------------------------------
+
+_AWS_REGIONS_STATIC: list[str] = [
+    "af-south-1",
+    "ap-east-1",
+    "ap-northeast-1",
+    "ap-northeast-2",
+    "ap-northeast-3",
+    "ap-south-1",
+    "ap-south-2",
+    "ap-southeast-1",
+    "ap-southeast-2",
+    "ap-southeast-3",
+    "ap-southeast-4",
+    "ca-central-1",
+    "ca-west-1",
+    "eu-central-1",
+    "eu-central-2",
+    "eu-north-1",
+    "eu-south-1",
+    "eu-south-2",
+    "eu-west-1",
+    "eu-west-2",
+    "eu-west-3",
+    "il-central-1",
+    "me-central-1",
+    "me-south-1",
+    "sa-east-1",
+    "us-east-1",
+    "us-east-2",
+    "us-west-1",
+    "us-west-2",
+]
+
+
+def get_aws_regions() -> list[str]:
+    """Return available AWS S3 regions via boto3, falling back to a static list."""
+    try:
+        import boto3  # noqa: PLC0415
+
+        session = boto3.session.Session()
+        regions = sorted(session.get_available_regions("s3"))
+        if regions:
+            return regions
+    except Exception:
+        logger.debug("boto3 not available or failed; using static AWS region list")
+    return _AWS_REGIONS_STATIC
+
+
+# ---------------------------------------------------------------------------
+# Azure regions — resolved from known Cognitive Services locations
+# ---------------------------------------------------------------------------
+
+_AZURE_REGIONS_STATIC: list[str] = [
+    "australiacentral",
+    "australiaeast",
+    "australiasoutheast",
+    "brazilsouth",
+    "canadacentral",
+    "canadaeast",
+    "centralindia",
+    "centralus",
+    "eastasia",
+    "eastus",
+    "eastus2",
+    "francecentral",
+    "germanywestcentral",
+    "japaneast",
+    "japanwest",
+    "koreacentral",
+    "koreasouth",
+    "northcentralus",
+    "northeurope",
+    "norwayeast",
+    "polandcentral",
+    "qatarcentral",
+    "southafricanorth",
+    "southcentralus",
+    "southeastasia",
+    "swedencentral",
+    "switzerlandnorth",
+    "uaenorth",
+    "uksouth",
+    "ukwest",
+    "westcentralus",
+    "westeurope",
+    "westus",
+    "westus2",
+    "westus3",
+]
+
+
+def get_azure_regions() -> list[str]:
+    """Return Azure Cognitive Services regions.
+
+    Falls back to a curated static list because there is no
+    unauthenticated public endpoint to enumerate regions.
+    """
+    return _AZURE_REGIONS_STATIC
+
+
+# ---------------------------------------------------------------------------
+# Tesseract languages — probed from `tesseract --list-langs`
+# ---------------------------------------------------------------------------
+
+_TESSERACT_LANGS_STATIC: list[str] = [
+    "afr",
+    "amh",
+    "ara",
+    "asm",
+    "aze",
+    "bel",
+    "ben",
+    "bod",
+    "bos",
+    "bre",
+    "bul",
+    "cat",
+    "ceb",
+    "ces",
+    "chi_sim",
+    "chi_tra",
+    "chr",
+    "cos",
+    "cym",
+    "dan",
+    "deu",
+    "div",
+    "ell",
+    "eng",
+    "enm",
+    "epo",
+    "est",
+    "eus",
+    "fao",
+    "fas",
+    "fil",
+    "fin",
+    "fra",
+    "frk",
+    "frm",
+    "fry",
+    "gla",
+    "gle",
+    "glg",
+    "grc",
+    "guj",
+    "hat",
+    "heb",
+    "hin",
+    "hrv",
+    "hun",
+    "hye",
+    "iku",
+    "ind",
+    "isl",
+    "ita",
+    "jav",
+    "jpn",
+    "kan",
+    "kat",
+    "kaz",
+    "khm",
+    "kir",
+    "kor",
+    "lao",
+    "lat",
+    "lav",
+    "lit",
+    "ltz",
+    "mal",
+    "mar",
+    "mkd",
+    "mlt",
+    "mon",
+    "mri",
+    "msa",
+    "mya",
+    "nep",
+    "nld",
+    "nor",
+    "oci",
+    "ori",
+    "pan",
+    "pol",
+    "por",
+    "pus",
+    "que",
+    "ron",
+    "rus",
+    "san",
+    "sin",
+    "slk",
+    "slv",
+    "snd",
+    "spa",
+    "sqi",
+    "srp",
+    "sun",
+    "swa",
+    "swe",
+    "syr",
+    "tam",
+    "tat",
+    "tel",
+    "tgk",
+    "tha",
+    "tir",
+    "ton",
+    "tur",
+    "uig",
+    "ukr",
+    "urd",
+    "uzb",
+    "vie",
+    "yid",
+    "yor",
+]
+
+
+def get_tesseract_languages() -> list[str]:
+    """Return installed Tesseract language codes, falling back to a static list."""
+    try:
+        result = subprocess.run(  # noqa: S603, S607
+            ["tesseract", "--list-langs"],  # noqa: S607
+            capture_output=True,
+            text=True,
+            timeout=5,
+            check=False,
+        )
+        if result.returncode == 0:
+            lines = result.stdout.strip().splitlines()
+            # First line is the header ("List of available languages ...")
+            langs = sorted(line.strip() for line in lines[1:] if line.strip())
+            if langs:
+                return langs
+    except Exception:
+        logger.debug("tesseract not available; using static language list")
+    return _TESSERACT_LANGS_STATIC
+
+
+# ---------------------------------------------------------------------------
+# EasyOCR languages — probed from the easyocr module
+# ---------------------------------------------------------------------------
+
+_EASYOCR_LANGS_STATIC: list[str] = [
+    "abq",
+    "ady",
+    "af",
+    "ang",
+    "ar",
+    "as",
+    "ava",
+    "az",
+    "be",
+    "bg",
+    "bh",
+    "bn",
+    "bs",
+    "ch_sim",
+    "ch_tra",
+    "che",
+    "cs",
+    "cy",
+    "da",
+    "dar",
+    "de",
+    "en",
+    "es",
+    "et",
+    "fa",
+    "fi",
+    "fr",
+    "ga",
+    "gom",
+    "hi",
+    "hr",
+    "hu",
+    "id",
+    "inh",
+    "is",
+    "it",
+    "ja",
+    "ka",
+    "kk",
+    "km",
+    "kn",
+    "ko",
+    "ku",
+    "la",
+    "lbe",
+    "lez",
+    "lt",
+    "lv",
+    "mah",
+    "mai",
+    "mi",
+    "mn",
+    "mr",
+    "ms",
+    "mt",
+    "ne",
+    "new",
+    "nl",
+    "no",
+    "oc",
+    "pi",
+    "pl",
+    "pt",
+    "ro",
+    "ru",
+    "rs_cyrillic",
+    "rs_latin",
+    "sa",
+    "sck",
+    "sk",
+    "sl",
+    "sq",
+    "sv",
+    "sw",
+    "ta",
+    "tab",
+    "te",
+    "th",
+    "tjk",
+    "tl",
+    "tr",
+    "ug",
+    "uk",
+    "ur",
+    "uz",
+    "vi",
+]
+
+
+def get_easyocr_languages() -> list[str]:
+    """Return supported EasyOCR language codes, falling back to a static list."""
+    try:
+        import easyocr  # noqa: PLC0415
+
+        # easyocr stores the language list internally
+        if hasattr(easyocr, "config") and hasattr(easyocr.config, "all_lang_list"):
+            return sorted(easyocr.config.all_lang_list)
+    except Exception:
+        logger.debug("easyocr not available; using static language list")
+    return _EASYOCR_LANGS_STATIC
+
+
+# ---------------------------------------------------------------------------
+# Embedding models — static list (no standard discovery API)
+# ---------------------------------------------------------------------------
+
+_EMBEDDING_MODELS: list[str] = [
+    "text-embedding-3-small",
+    "text-embedding-3-large",
+    "text-embedding-ada-002",
+    "nomic-embed-text",
+    "nomic-embed-text-v1.5",
+    "mxbai-embed-large",
+    "mxbai-embed-large-v1",
+    "all-MiniLM-L6-v2",
+    "all-MiniLM-L12-v2",
+    "bge-small-en-v1.5",
+    "bge-base-en-v1.5",
+    "bge-large-en-v1.5",
+    "e5-small-v2",
+    "e5-base-v2",
+    "e5-large-v2",
+    "gte-small",
+    "gte-base",
+    "gte-large",
+    "voyage-3",
+    "voyage-3-lite",
+    "voyage-code-3",
+]
+
+
+def get_embedding_models() -> list[str]:
+    """Return known embedding model names."""
+    return _EMBEDDING_MODELS
+
+
+# ---------------------------------------------------------------------------
+# Registry — maps setting keys to their provider functions
+# ---------------------------------------------------------------------------
+
+SUGGESTION_PROVIDERS: dict[str, Callable[[], list[str]]] = {
+    "aws_region": get_aws_regions,
+    "azure_region": get_azure_regions,
+    "tesseract_language": get_tesseract_languages,
+    "easyocr_languages": get_easyocr_languages,
+    "embedding_model": get_embedding_models,
+}
+
+
+def get_suggestions(key: str, query: str = "", limit: int = 10) -> list[str]:
+    """
+    Return autocomplete suggestions for the given setting key.
+
+    Fetches the full list from the registered provider, filters by
+    case-insensitive substring match on *query*, and returns at most
+    *limit* results.
+
+    Args:
+        key: The setting key (must be registered in SUGGESTION_PROVIDERS).
+        query: Substring to filter by (case-insensitive).
+        limit: Maximum number of results to return.
+
+    Returns:
+        Filtered list of suggestion strings.
+
+    Raises:
+        KeyError: If no provider is registered for *key*.
+    """
+    provider = SUGGESTION_PROVIDERS.get(key)
+    if provider is None:
+        raise KeyError(f"No suggestion provider registered for setting '{key}'")
+
+    all_values = provider()
+    q = query.strip().lower()
+
+    if q:
+        filtered = [v for v in all_values if q in v.lower()]
+    else:
+        filtered = list(all_values)
+
+    return filtered[:limit]

app/utils/user_scope.py

@@ -0,0 +1,84 @@
+"""
+User-scoping utilities for multi-user document isolation.
+
+When ``multi_user_enabled`` is ``True`` in settings, every document query
+is filtered by the authenticated user's identifier so that each user sees
+only their own documents.  When the flag is ``False`` (default), all
+documents are visible to all users (single-user / shared mode).
+"""
+
+import logging
+
+from fastapi import Request
+from sqlalchemy import or_
+from sqlalchemy.orm import Query
+from sqlalchemy.sql import false
+
+from app.config import settings
+from app.models import FileRecord
+
+logger = logging.getLogger(__name__)
+
+
+def get_current_owner_id(request: Request) -> str | None:
+    """Extract the owner identifier for the current authenticated user.
+
+    The owner ID is derived from the user's session data.  It uses the
+    ``sub`` claim (OAuth subject) when available, falling back to
+    ``preferred_username`` or ``email``.  Returns ``None`` when no user
+    is authenticated.
+
+    Args:
+        request: The current FastAPI request with session data.
+
+    Returns:
+        A stable string identifier for the user, or ``None``.
+    """
+    user = request.session.get("user")
+    if not user or not isinstance(user, dict):
+        return None
+    # Prefer 'sub' (OAuth subject), then 'preferred_username', then 'email', then 'id'
+    return user.get("sub") or user.get("preferred_username") or user.get("email") or user.get("id")
+
+
+def apply_owner_filter(query: Query, request: Request) -> Query:
+    """Conditionally filter a ``FileRecord`` query by the current user.
+
+    When multi-user mode is enabled, only files whose ``owner_id``
+    matches the authenticated user are returned.  Admin users bypass
+    the filter and see all documents.
+
+    When ``unowned_docs_visible_to_all`` is ``True`` (default), documents
+    with ``owner_id IS NULL`` (unclaimed) are also included for every
+    authenticated user so they can be discovered and claimed.
+
+    When multi-user mode is disabled the query is returned unchanged.
+
+    Args:
+        query: A SQLAlchemy query selecting ``FileRecord`` rows.
+        request: The current FastAPI request (for session inspection).
+
+    Returns:
+        The (possibly filtered) query.
+    """
+    if not settings.multi_user_enabled:
+        return query
+
+    user = request.session.get("user")
+    if isinstance(user, dict) and user.get("is_admin"):
+        # Admins see all documents in multi-user mode
+        return query
+
+    owner_id = get_current_owner_id(request)
+    if owner_id is None:
+        # No authenticated user — return empty result set
+        return query.filter(false())
+
+    # Build filter: user's own documents
+    conditions = [FileRecord.owner_id == owner_id]
+
+    # Optionally include unclaimed (owner_id IS NULL) documents
+    if settings.unowned_docs_visible_to_all:
+        conditions.append(FileRecord.owner_id.is_(None))
+
+    return query.filter(or_(*conditions))

app/utils/webhook.py

@@ -0,0 +1,171 @@
+"""Webhook delivery utility for notifying external systems of document events.
+
+Provides functions to dispatch webhook payloads with HMAC-SHA256 signatures
+and to query active webhook configurations from the database.
+
+Supported events:
+- ``document.uploaded``  – a new document has been ingested
+- ``document.processed`` – a document finished processing successfully
+- ``document.failed``    – document processing failed
+- ``user.signup``        – a new user account was created
+- ``user.plan_changed``  – a user's subscription plan changed
+- ``user.payment_issue`` – a payment issue was reported for a user
+"""
+
+import hashlib
+import hmac
+import json
+import logging
+import time
+from typing import Any
+
+import requests
+
+from app.database import SessionLocal
+from app.models import WebhookConfig
+
+logger = logging.getLogger(__name__)
+
+#: Events recognised by the webhook subsystem.
+VALID_EVENTS: frozenset[str] = frozenset(
+    {
+        "document.uploaded",
+        "document.processed",
+        "document.failed",
+        "user.signup",
+        "user.plan_changed",
+        "user.payment_issue",
+    }
+)
+
+#: Timeout (seconds) for outgoing webhook HTTP requests.
+WEBHOOK_TIMEOUT = 10
+
+
+def compute_signature(payload_bytes: bytes, secret: str) -> str:
+    """Compute an HMAC-SHA256 hex-digest for *payload_bytes* using *secret*.
+
+    Args:
+        payload_bytes: The raw JSON body to sign.
+        secret: The shared secret string.
+
+    Returns:
+        ``sha256=<hex-digest>`` signature string.
+    """
+    mac = hmac.new(secret.encode("utf-8"), payload_bytes, hashlib.sha256)
+    return f"sha256={mac.hexdigest()}"
+
+
+def deliver_webhook(url: str, payload: dict[str, Any], secret: str | None = None) -> bool:
+    """Send a single webhook POST request.
+
+    Args:
+        url: Target URL.
+        payload: JSON-serialisable dictionary.
+        secret: If provided, an ``X-Webhook-Signature`` header is included.
+
+    Returns:
+        ``True`` when the remote server responds with a 2xx status.
+    """
+    body = json.dumps(payload, default=str, sort_keys=True)
+    body_bytes = body.encode("utf-8")
+
+    headers: dict[str, str] = {"Content-Type": "application/json"}
+    if secret:
+        headers["X-Webhook-Signature"] = compute_signature(body_bytes, secret)
+
+    try:
+        resp = requests.post(url, data=body_bytes, headers=headers, timeout=WEBHOOK_TIMEOUT)
+        if resp.ok:
+            logger.info("Webhook delivered to %s (status %d)", url, resp.status_code)
+            return True
+        logger.warning("Webhook to %s returned status %d", url, resp.status_code)
+        return False
+    except requests.RequestException as exc:
+        logger.error("Webhook delivery to %s failed: %s", url, exc)
+        return False
+
+
+def build_payload(event: str, data: dict[str, Any]) -> dict[str, Any]:
+    """Build a standardised webhook payload envelope.
+
+    Args:
+        event: The event name (e.g. ``document.uploaded``).
+        data: Event-specific data.
+
+    Returns:
+        Dictionary with ``event``, ``timestamp``, and ``data`` keys.
+    """
+    return {
+        "event": event,
+        "timestamp": time.time(),
+        "data": data,
+    }
+
+
+def get_active_webhooks_for_event(event: str) -> list[dict[str, Any]]:
+    """Return all active webhook configs subscribed to *event*.
+
+    Queries the database directly so this helper can be called from both the
+    API layer and Celery tasks.
+
+    Args:
+        event: The event name to filter on.
+
+    Returns:
+        A list of dicts with ``id``, ``url``, ``secret``, and ``events`` keys.
+    """
+    db = SessionLocal()
+    try:
+        configs = db.query(WebhookConfig).filter(WebhookConfig.is_active.is_(True)).all()
+        result: list[dict[str, Any]] = []
+        for cfg in configs:
+            try:
+                subscribed = json.loads(cfg.events)
+            except (json.JSONDecodeError, TypeError):
+                subscribed = []
+            if event in subscribed:
+                result.append(
+                    {
+                        "id": cfg.id,
+                        "url": cfg.url,
+                        "secret": cfg.secret,
+                        "events": subscribed,
+                    }
+                )
+        return result
+    finally:
+        db.close()
+
+
+def dispatch_webhook_event(event: str, data: dict[str, Any]) -> None:
+    """Fan-out a webhook event to all matching active configurations.
+
+    This is the main entry-point used by application code to trigger webhooks.
+    It delegates to :func:`deliver_webhook_task` (Celery) for each matching
+    webhook so delivery happens asynchronously with automatic retries.
+
+    Args:
+        event: Event name (must be in :data:`VALID_EVENTS`).
+        data: Event-specific payload data.
+    """
+    if event not in VALID_EVENTS:
+        logger.warning("Ignoring unknown webhook event: %s", event)
+        return
+
+    webhooks = get_active_webhooks_for_event(event)
+    if not webhooks:
+        logger.debug("No active webhooks for event %s", event)
+        return
+
+    payload = build_payload(event, data)
+
+    # Import here to avoid circular dependency with celery_app
+    from app.tasks.webhook_tasks import deliver_webhook_task
+
+    for wh in webhooks:
+        try:
+            deliver_webhook_task.delay(wh["url"], payload, wh["secret"])
+            logger.debug("Queued webhook delivery to %s for event %s", wh["url"], event)
+        except Exception as exc:
+            logger.error("Failed to queue webhook to %s: %s", wh["url"], exc)

app/views/__init__.py

@@ -4,23 +4,34 @@ Aggregated view routers for the application.
 
 from fastapi import APIRouter
 
+from app.views.admin_users import router as admin_users_router
+from app.views.backup import router as backup_router
+from app.views.db_wizard import router as db_wizard_router
 from app.views.dropbox import router as dropbox_router
 from app.views.filemanager import router as filemanager_router
 
 # Import all the view routers
 from app.views.general import router as general_router
 from app.views.google_drive import router as google_drive_router
+from app.views.help import router as help_router  # Built-in help / How-To docs
 from app.views.license_routes import router as license_router  # Add the license router
+from app.views.onboarding import router as onboarding_router
 from app.views.onedrive import router as onedrive_router
+from app.views.pipelines import router as pipelines_router  # Processing pipelines
+from app.views.plans import router as plans_router  # Admin Plan Designer
 from app.views.queue import router as queue_router
 from app.views.search import router as search_router
 from app.views.settings import router as settings_router
 from app.views.status import router as status_router
+from app.views.subscriptions import router as subscriptions_router  # Pricing + subscription pages
 from app.views.wizard import router as wizard_router
 
 # Create a main router that includes all the view routers
 router = APIRouter()
 router.include_router(wizard_router)  # Wizard first (for /setup)
+router.include_router(db_wizard_router)  # Database wizard
+router.include_router(admin_users_router)  # Admin user management
+router.include_router(backup_router)  # Backup dashboard
 router.include_router(general_router)
 router.include_router(status_router)
 router.include_router(onedrive_router)
@@ -31,3 +42,8 @@ router.include_router(settings_router)
 router.include_router(filemanager_router)
 router.include_router(search_router)
 router.include_router(queue_router)
+router.include_router(subscriptions_router)  # Pricing + subscription pages
+router.include_router(plans_router)  # Admin Plan Designer
+router.include_router(onboarding_router)  # User onboarding wizard
+router.include_router(pipelines_router)  # Processing pipelines
+router.include_router(help_router)  # Built-in help / How-To docs

app/views/admin_users.py

@@ -0,0 +1,44 @@
+"""Admin view: user management dashboard."""
+
+import logging
+
+from fastapi import HTTPException, Request, status
+from fastapi.responses import RedirectResponse
+
+from app.views.base import APIRouter, get_db, require_login, settings, templates  # noqa: F401
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+def _require_admin(request: Request):
+    """Return the session user if they are an admin, else redirect."""
+    user = request.session.get("user")
+    if not user or not user.get("is_admin"):
+        logger.warning("Non-admin user attempted to access /admin/users")
+        return None
+    return user
+
+
+@router.get("/admin/users")
+@require_login
+async def admin_users_page(request: Request):
+    """Admin user management dashboard — lists all known users."""
+    user = _require_admin(request)
+    if user is None:
+        return RedirectResponse(url="/", status_code=status.HTTP_302_FOUND)
+
+    try:
+        return templates.TemplateResponse(
+            "admin_users.html",
+            {
+                "request": request,
+                "app_version": settings.version,
+            },
+        )
+    except Exception as e:
+        logger.error(f"Error loading admin users page: {e}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to load admin users page",
+        )

app/views/backup.py

@@ -0,0 +1,57 @@
+"""
+Backup management dashboard view – admin only.
+"""
+
+import logging
+import os
+
+from fastapi import Depends, HTTPException, Request, status
+from sqlalchemy.orm import Session
+
+from app.config import settings
+from app.models import BackupRecord
+from app.views.base import APIRouter, get_db, require_login, templates
+from app.views.settings import require_admin_access
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.get("/admin/backup")
+@require_login
+@require_admin_access
+async def backup_dashboard(request: Request, db: Session = Depends(get_db)):
+    """Backup management dashboard – admin only."""
+    try:
+        records = db.query(BackupRecord).order_by(BackupRecord.created_at.desc()).limit(500).all()
+
+        # Summarise counts per tier
+        counts: dict[str, int] = {"hourly": 0, "daily": 0, "weekly": 0}
+        for r in records:
+            if r.backup_type in counts:
+                counts[r.backup_type] += 1
+
+        # Compute total local size
+        total_size = sum(r.size_bytes for r in records if r.local_path and os.path.exists(r.local_path))
+
+        return templates.TemplateResponse(
+            "backup.html",
+            {
+                "request": request,
+                "records": records,
+                "counts": counts,
+                "total_size": total_size,
+                "backup_enabled": getattr(settings, "backup_enabled", True),
+                "backup_remote_destination": getattr(settings, "backup_remote_destination", None),
+                "backup_retain_hourly": getattr(settings, "backup_retain_hourly", 96),
+                "backup_retain_daily": getattr(settings, "backup_retain_daily", 21),
+                "backup_retain_weekly": getattr(settings, "backup_retain_weekly", 13),
+                "app_version": settings.version,
+            },
+        )
+    except Exception as e:
+        logger.error(f"Error loading backup dashboard: {e}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to load backup dashboard",
+        )

app/views/base.py

@@ -25,24 +25,40 @@ templates.env.globals["max"] = max
 original_template_response = templates.TemplateResponse
 
 
+def _inject_global_context(ctx: dict) -> None:
+    """Inject shared global variables into every template context dict."""
+    ctx.setdefault("version", settings.version)
+    ctx.setdefault("release_name", getattr(settings, "release_name", None))
+    ctx.setdefault("ui_default_color_scheme", getattr(settings, "ui_default_color_scheme", "system"))
+    ctx.setdefault("multi_user_enabled", getattr(settings, "multi_user_enabled", False))
+    ctx.setdefault("auth_enabled", getattr(settings, "auth_enabled", True))
+    ctx.setdefault(
+        "allow_signup",
+        getattr(settings, "multi_user_enabled", False) and getattr(settings, "allow_local_signup", False),
+    )
+
+    req = ctx.get("request")
+    if req is not None:
+        # CSRF token
+        if hasattr(req, "state") and hasattr(req.state, "csrf_token"):
+            ctx.setdefault("csrf_token", req.state.csrf_token)
+        # Determine whether the current visitor is authenticated
+        session_user = None
+        if hasattr(req, "session"):
+            session_user = req.session.get("user")
+        # When auth is disabled every visitor is effectively "logged in"
+        ctx.setdefault("is_logged_in", not getattr(settings, "auth_enabled", True) or session_user is not None)
+    else:
+        ctx.setdefault("is_logged_in", not getattr(settings, "auth_enabled", True))
+
+
 def template_response_with_version(*args, **kwargs):
     """Wrapper for TemplateResponse to include version and CSRF token in all templates"""
     # If context dict is provided, add version to it
     if len(args) >= 2 and isinstance(args[1], dict):
-        args[1].setdefault("version", settings.version)
-        # Inject CSRF token from request state when available
-        req = args[1].get("request")
-        if req is not None and hasattr(req.state, "csrf_token"):
-            args[1].setdefault("csrf_token", req.state.csrf_token)
-        # Inject default color scheme for dark-mode initialisation
-        args[1].setdefault("ui_default_color_scheme", getattr(settings, "ui_default_color_scheme", "system"))
+        _inject_global_context(args[1])
     elif "context" in kwargs and isinstance(kwargs["context"], dict):
-        kwargs["context"].setdefault("version", settings.version)
-        req = kwargs["context"].get("request")
-        if req is not None and hasattr(req.state, "csrf_token"):
-            kwargs["context"].setdefault("csrf_token", req.state.csrf_token)
-        # Inject default color scheme for dark-mode initialisation
-        kwargs["context"].setdefault("ui_default_color_scheme", getattr(settings, "ui_default_color_scheme", "system"))
+        _inject_global_context(kwargs["context"])
     return original_template_response(*args, **kwargs)
 
 

app/views/db_wizard.py

@@ -0,0 +1,29 @@
+"""
+Database configuration wizard view.
+
+Serves the guided UI for configuring a database connection string
+and migrating data from one database to another.
+"""
+
+import logging
+
+from fastapi import Request
+from fastapi.responses import Response
+
+from app.config import settings
+from app.views.base import APIRouter, templates
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.get("/database-wizard")
+async def database_wizard(request: Request) -> Response:
+    """Render the database configuration wizard page."""
+    return templates.TemplateResponse(
+        "db_wizard.html",
+        {
+            "request": request,
+            "current_database_url": settings.database_url,
+        },
+    )

app/views/files.py

@@ -25,7 +25,7 @@ def files_page(
     request: Request,
     db: Session = Depends(get_db),
     page: int = Query(1, ge=1),
-    per_page: int = Query(50, ge=1, le=200),
+    per_page: int = Query(25, ge=1, le=200),
     sort_by: str = Query("created_at"),
     sort_order: str = Query("desc"),
     search: Optional[str] = Query(None),
@@ -168,8 +168,8 @@ def files_page(
                 "pagination": {
                     "page": page,
                     "per_page": per_page,
-                    "total_items": total_items,
-                    "total_pages": total_pages,
+                    "total": total_items,
+                    "pages": total_pages,
                 },
                 "sort_by": sort_by,
                 "sort_order": sort_order,
@@ -196,7 +196,7 @@ def files_page(
             {
                 "request": request,
                 "files": [],
-                "pagination": {"page": 1, "per_page": per_page, "total_items": 0, "total_pages": 0},
+                "pagination": {"page": 1, "per_page": per_page, "total": 0, "pages": 0},
                 "error": str(e),
                 "upload_concurrency": settings.upload_concurrency,
                 "upload_queue_delay_ms": settings.upload_queue_delay_ms,
@@ -270,6 +270,9 @@ def file_view_page(request: Request, file_id: int, db: Session = Depends(get_db)
         except Exception:
             step_summary = None
 
+        # Resolve the pipeline assigned to this file (explicit or system default)
+        pipeline_info = _resolve_pipeline(db, file_record)
+
         return templates.TemplateResponse(
             "file_view.html",
             {
@@ -279,6 +282,7 @@ def file_view_page(request: Request, file_id: int, db: Session = Depends(get_db)
                 "original_file_exists": original_file_exists,
                 "processed_file_exists": processed_file_exists,
                 "step_summary": step_summary,
+                "pipeline_info": pipeline_info,
             },
         )
     except Exception as e:
@@ -337,8 +341,11 @@ def file_detail_page(request: Request, file_id: int, db: Session = Depends(get_d
                 except Exception as e:
                     logger.warning(f"Failed to load metadata from {metadata_path}: {e}")
 
-        # Compute processing flow for visualization
-        flow_data = _compute_processing_flow(logs)
+        # Resolve the pipeline assigned to this file (explicit or system default)
+        pipeline_info = _resolve_pipeline(db, file_record)
+
+        # Compute processing flow for visualization — filter to pipeline steps when available
+        flow_data = _compute_processing_flow(logs, pipeline_steps=pipeline_info["steps"] if pipeline_info else None)
 
         # Compute step-aligned summary from status table (preferred) or fallback to logs
         try:
@@ -360,6 +367,7 @@ def file_detail_page(request: Request, file_id: int, db: Session = Depends(get_d
                 "gpt_metadata": gpt_metadata,
                 "flow_data": flow_data,
                 "step_summary": step_summary,
+                "pipeline_info": pipeline_info,
             },
         )
     except Exception as e:
@@ -367,15 +375,99 @@ def file_detail_page(request: Request, file_id: int, db: Session = Depends(get_d
         return templates.TemplateResponse("file_detail.html", {"request": request, "file": None, "error": str(e)})
 
 
-def _compute_processing_flow(logs):
+# ---------------------------------------------------------------------------
+# Pipeline ↔ Celery-log stage mapping
+# ---------------------------------------------------------------------------
+
+# Maps each pipeline step_type to the set of Celery task log stage keys that
+# implement it.  Used to filter the flow visualization when a pipeline is
+# assigned to a file.
+#
+# ⚠️  MAINTENANCE NOTE: When a new step type is added to PIPELINE_STEP_TYPES
+# in app/api/pipelines.py it MUST also be added here, otherwise the flow
+# visualization will silently skip its Celery-task stages for files using that
+# step type.  The test ``TestPipelineInfoInViews::test_step_type_mapping_is_complete``
+# enforces this invariant automatically.
+_STEP_TYPE_TO_STAGES: dict[str, list[str]] = {
+    "convert_to_pdf": ["convert_to_pdf"],
+    "check_duplicates": ["check_for_duplicates"],
+    "ocr": ["check_text", "extract_text", "process_with_ocr"],
+    "extract_metadata": ["extract_metadata_with_gpt"],
+    "embed_metadata": ["embed_metadata_into_pdf"],
+    "compute_embedding": ["compute_embedding"],
+    "send_to_destinations": ["finalize_document_storage", "send_to_all_destinations"],
+    # "classify" is defined in PIPELINE_STEP_TYPES but has no Celery log stages yet.
+    # When a classify task is implemented, add its stage key(s) here.
+    "classify": [],
+}
+
+# These internal bookkeeping stages are always shown in the flow regardless of
+# which pipeline steps are defined.
+_ALWAYS_SHOW_STAGES: frozenset[str] = frozenset({"create_file_record"})
+
+
+def _resolve_pipeline(db: Session, file_record) -> dict | None:
+    """Resolve the pipeline information for a file.
+
+    If the file has an explicit ``pipeline_id``, load that pipeline.
+    Otherwise fall back to the active system-default pipeline
+    (``owner_id IS NULL``, ``is_default=True``).
+
+    Returns a dict with keys:
+        id, name, description, is_default, is_system, is_explicit, steps
+    or ``None`` when no pipeline exists in the database.
+    """
+    from app.models import Pipeline, PipelineStep
+
+    pipeline = None
+    if file_record.pipeline_id:
+        pipeline = db.query(Pipeline).filter(Pipeline.id == file_record.pipeline_id).first()
+
+    if pipeline is None:
+        pipeline = (
+            db.query(Pipeline)
+            .filter(
+                Pipeline.owner_id.is_(None),
+                Pipeline.is_default.is_(True),
+                Pipeline.is_active.is_(True),
+            )
+            .first()
+        )
+
+    if pipeline is None:
+        return None
+
+    steps = db.query(PipelineStep).filter(PipelineStep.pipeline_id == pipeline.id).order_by(PipelineStep.position).all()
+
+    return {
+        "id": pipeline.id,
+        "name": pipeline.name,
+        "description": pipeline.description,
+        "is_default": pipeline.is_default,
+        "is_system": pipeline.owner_id is None,
+        # True when the file has a pipeline explicitly assigned (not inferred default)
+        "is_explicit": bool(file_record.pipeline_id),
+        "steps": steps,
+    }
+
+
+def _compute_processing_flow(logs, pipeline_steps=None):
     """
     Compute the processing flow structure from logs for visualization.
 
     Returns a structured representation of the processing pipeline with branches.
     Detects upload sub-tasks and organizes them as branches under the parent upload stage.
+
+    Args:
+        logs: list of ProcessingLog objects (ordered by timestamp asc)
+        pipeline_steps: optional list of PipelineStep objects for the assigned pipeline.
+            When provided, the set of stages shown is filtered to only those that
+            correspond to the pipeline's enabled steps (plus bookkeeping stages like
+            ``create_file_record`` and any stage that actually ran in the logs).
     """
-    # Define the main processing stages
+    # Define the full catalogue of main processing stages
     stages = {
+        "convert_to_pdf": {"label": "Convert to PDF", "next": ["check_for_duplicates", "create_file_record"]},
         "check_for_duplicates": {"label": "Check for Duplicates", "next": ["create_file_record"]},
         "create_file_record": {"label": "Create File Record", "next": ["check_text"]},
         "check_text": {
@@ -389,8 +481,12 @@ def _compute_processing_flow(logs):
         },
         "extract_metadata_with_gpt": {"label": "Extract Metadata (GPT)", "next": ["embed_metadata_into_pdf"]},
         "embed_metadata_into_pdf": {"label": "Embed Metadata into PDF", "next": ["finalize_document_storage"]},
-        "finalize_document_storage": {"label": "Finalize & Queue Distribution", "next": ["send_to_all_destinations"]},
+        "finalize_document_storage": {
+            "label": "Finalize & Queue Distribution",
+            "next": ["send_to_all_destinations", "compute_embedding"],
+        },
         "send_to_all_destinations": {"label": "Upload to Destinations", "next": [], "has_branches": True},
+        "compute_embedding": {"label": "Compute Embedding", "next": []},
     }
 
     # Filter out deduplication step if not enabled or if not showing it
@@ -402,6 +498,22 @@ def _compute_processing_flow(logs):
         if "create_file_record" in stages:
             stages["create_file_record"]["next"] = ["check_text"]
 
+    # When a pipeline is assigned, filter stages to only those relevant to the
+    # pipeline's enabled steps plus always-show bookkeeping stages and any stage
+    # that actually produced log entries (so nothing already-run is hidden).
+    if pipeline_steps is not None:
+        # Collect Celery stage keys that the pipeline's enabled steps map to
+        allowed: set[str] = set(_ALWAYS_SHOW_STAGES)
+        for ps in pipeline_steps:
+            if ps.enabled:
+                allowed.update(_STEP_TYPE_TO_STAGES.get(ps.step_type, []))
+        # Pre-scan logs so we can also keep any stage that already ran
+        ran_stages: set[str] = set()
+        for log in logs:
+            ran_stages.add(log.step_name)
+        allowed.update(ran_stages)
+        stages = {k: v for k, v in stages.items() if k in allowed}
+
     # Define upload sub-tasks (branches)
     upload_tasks = {
         "upload_to_dropbox": "Dropbox",
@@ -718,3 +830,151 @@ def get_processed_text(request: Request, file_id: int, db: Session = Depends(get
         raise HTTPException(
             status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail=f"Failed to extract text: {str(e)}"
         )
+
+
+@router.get("/duplicates")
+@require_login
+def duplicates_page(
+    request: Request,
+    db: Session = Depends(get_db),
+    page: int = Query(1, ge=1),
+    per_page: int = Query(25, ge=1, le=200),
+):
+    """Render the duplicate-document management page.
+
+    Passes exact-duplicate group data (server-side) plus the configured
+    near-duplicate threshold so the JS finder can pre-populate the form.
+    """
+    from app.config import settings
+    from app.models import FileRecord
+
+    try:
+        # Find hashes that have at least one is_duplicate=True record
+        dup_hashes_query = db.query(FileRecord.filehash).filter(FileRecord.is_duplicate.is_(True)).distinct()
+        total_groups = dup_hashes_query.count()
+
+        offset = (page - 1) * per_page
+        dup_hashes = [row.filehash for row in dup_hashes_query.offset(offset).limit(per_page).all()]
+
+        groups = []
+        total_duplicate_files = 0
+
+        for filehash in dup_hashes:
+            original = (
+                db.query(FileRecord)
+                .filter(FileRecord.filehash == filehash, FileRecord.is_duplicate.is_(False))
+                .order_by(FileRecord.id.asc())
+                .first()
+            )
+            duplicates = (
+                db.query(FileRecord)
+                .filter(FileRecord.filehash == filehash, FileRecord.is_duplicate.is_(True))
+                .order_by(FileRecord.id.asc())
+                .all()
+            )
+            total_duplicate_files += len(duplicates)
+
+            def _to_dict(f: FileRecord) -> dict:
+                return {
+                    "id": f.id,
+                    "original_filename": f.original_filename,
+                    "filehash": f.filehash,
+                    "file_size": f.file_size,
+                    "mime_type": f.mime_type,
+                    "is_duplicate": f.is_duplicate,
+                    "duplicate_of_id": f.duplicate_of_id,
+                    "created_at": f.created_at.isoformat() if f.created_at else None,
+                }
+
+            groups.append(
+                {
+                    "filehash": filehash,
+                    "original": _to_dict(original) if original else None,
+                    "duplicates": [_to_dict(d) for d in duplicates],
+                    "duplicate_count": len(duplicates),
+                }
+            )
+
+        total_pages = max(1, (total_groups + per_page - 1) // per_page)
+
+        return templates.TemplateResponse(
+            "duplicates.html",
+            {
+                "request": request,
+                "groups": groups,
+                "total_groups": total_groups,
+                "total_duplicate_files": total_duplicate_files,
+                "pagination": {
+                    "page": page,
+                    "per_page": per_page,
+                    "total": total_groups,
+                    "pages": total_pages,
+                },
+                "near_duplicate_threshold": settings.near_duplicate_threshold,
+            },
+        )
+    except Exception as e:
+        logger.error(f"Error rendering duplicates page: {e}")
+        return templates.TemplateResponse(
+            "duplicates.html",
+            {
+                "request": request,
+                "groups": [],
+                "total_groups": 0,
+                "total_duplicate_files": 0,
+                "pagination": {"page": 1, "per_page": per_page, "total": 0, "pages": 1},
+                "near_duplicate_threshold": 0.85,
+                "error": str(e),
+            },
+        )
+
+
+@router.get("/similarity")
+@require_login
+def similarity_dashboard_page(
+    request: Request,
+    db: Session = Depends(get_db),
+):
+    """Render the corpus-wide similarity dashboard.
+
+    Passes the configured threshold and embedding coverage stats so the
+    template can display them immediately while the JS fetches the actual
+    pairs from the API asynchronously.
+    """
+    from app.config import settings
+    from app.models import FileRecord
+
+    try:
+        total_files = db.query(FileRecord).count()
+        files_with_embedding = (
+            db.query(FileRecord).filter(FileRecord.embedding.isnot(None), FileRecord.embedding != "").count()
+        )
+        files_with_ocr = db.query(FileRecord).filter(FileRecord.ocr_text.isnot(None), FileRecord.ocr_text != "").count()
+
+        return templates.TemplateResponse(
+            "similarity_dashboard.html",
+            {
+                "request": request,
+                "default_threshold": settings.near_duplicate_threshold,
+                "embedding_model": settings.embedding_model,
+                "total_files": total_files,
+                "files_with_embedding": files_with_embedding,
+                "files_with_ocr": files_with_ocr,
+                "files_missing_embedding": files_with_ocr - files_with_embedding,
+            },
+        )
+    except Exception as e:
+        logger.error(f"Error rendering similarity dashboard: {e}")
+        return templates.TemplateResponse(
+            "similarity_dashboard.html",
+            {
+                "request": request,
+                "default_threshold": 0.85,
+                "embedding_model": "text-embedding-3-small",
+                "total_files": 0,
+                "files_with_embedding": 0,
+                "files_with_ocr": 0,
+                "files_missing_embedding": 0,
+                "error": str(e),
+            },
+        )

app/views/general.py

@@ -2,11 +2,12 @@
 General routes for the application homepage and basic pages.
 """
 
-from datetime import date
+from datetime import date, datetime, timezone
 from pathlib import Path
 
 from fastapi import Depends, HTTPException, Request
 from fastapi.responses import FileResponse, RedirectResponse
+from sqlalchemy import func
 from sqlalchemy.orm import Session
 
 from app.utils.config_validator import get_provider_status, validate_storage_configs
@@ -54,25 +55,77 @@ async def serve_index(request: Request, db: Session = Depends(get_db)):
         and provider in ["dropbox", "nextcloud", "sftp", "s3", "ftp", "webdav", "google_drive", "onedrive"]
     )
 
-    # Query the actual file count from the database
-    processed_files = 0
+    from app.models import FileRecord
+
+    today = datetime.now(timezone.utc).date()
+
+    # Global file counts (or per-user in multi-user mode)
+    from app.config import settings
+
+    user = request.session.get("user") or {}
+    is_admin = user.get("is_admin", False)
+
     try:
-        # Import the model here to avoid circular imports
-        from app.models import FileRecord
+        total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0
 
-        processed_files = db.query(FileRecord.id).count()
+        files_today: int = (
+            db.query(func.count(FileRecord.id)).filter(func.date(FileRecord.created_at) == today).scalar() or 0
+        )
+
+        files_month: int = (
+            db.query(func.count(FileRecord.id))
+            .filter(func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"))
+            .scalar()
+            or 0
+        )
+
+        files_with_ocr: int = db.query(func.count(FileRecord.id)).filter(FileRecord.ocr_text.isnot(None)).scalar() or 0
+
+        unique_users: int = (
+            db.query(func.count(func.distinct(FileRecord.owner_id))).filter(FileRecord.owner_id.isnot(None)).scalar()
+            or 0
+        )
     except Exception as e:
-        # Log error but continue (don't break the page if DB query fails)
-        logger.error(f"Error counting files: {str(e)}")
+        logger.error(f"Error computing dashboard stats: {e}")
+        total_files = files_today = files_month = files_with_ocr = unique_users = 0
+
+    # Per-user usage for the subscription widget (multi-user only)
+    user_usage = None
+    user_tier = None
+    if settings.multi_user_enabled:
+        owner_id: str = user.get("username") or user.get("email") or user.get("sub") or ""
+        if owner_id:
+            try:
+                from app.utils.subscription import get_tier, get_user_tier_id, get_user_usage
+
+                tier_id = get_user_tier_id(db, owner_id)
+                user_tier = get_tier(tier_id)
+                user_usage = get_user_usage(db, owner_id)
+            except Exception as e:
+                logger.error(f"Error fetching subscription info: {e}")
 
-    # Create stats object to pass to the template
     stats = {
-        "processed_files": processed_files,
+        "processed_files": total_files,
+        "files_today": files_today,
+        "files_month": files_month,
+        "files_with_ocr": files_with_ocr,
+        "unique_users": unique_users,
         "active_integrations": configured_providers,
         "storage_targets": configured_storage_targets,
     }
 
-    return templates.TemplateResponse("index.html", {"request": request, "stats": stats})
+    return templates.TemplateResponse(
+        "index.html",
+        {
+            "request": request,
+            "stats": stats,
+            "user_usage": user_usage,
+            "user_tier": user_tier,
+            "multi_user_enabled": settings.multi_user_enabled,
+            "is_admin": is_admin,
+            "allow_signup": settings.multi_user_enabled and settings.allow_local_signup,
+        },
+    )
 
 
 @router.get("/about", include_in_schema=False)

app/views/help.py

@@ -0,0 +1,27 @@
+"""
+Help documentation view routes.
+
+Serves the built MkDocs documentation site at /help.
+The static site is built during the Docker image build and placed at docs_build/.
+"""
+
+import logging
+import pathlib
+
+from fastapi import Request
+from fastapi.responses import RedirectResponse
+
+from app.views.base import APIRouter
+
+logger = logging.getLogger(__name__)
+
+router = APIRouter()
+
+# Path to the built MkDocs documentation
+_DOCS_BUILD_DIR = pathlib.Path(__file__).parents[2] / "docs_build"
+
+
+@router.get("/help", include_in_schema=False)
+async def help_redirect(request: Request) -> RedirectResponse:
+    """Redirect /help to /help/ so the MkDocs index is served correctly."""
+    return RedirectResponse(url="/help/", status_code=301)

app/views/onboarding.py

@@ -0,0 +1,91 @@
+"""View route for the user onboarding wizard."""
+
+import logging
+
+from fastapi import Depends, Request
+from sqlalchemy.orm import Session
+
+from app.config import Settings
+from app.config import settings as _settings
+from app.models import UserProfile
+from app.utils.subscription import get_all_tiers
+from app.views.base import APIRouter, get_db, require_login, templates
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+# ---------------------------------------------------------------------------
+# Destination helper
+# ---------------------------------------------------------------------------
+
+_DESTINATION_META: list[dict] = [
+    {"id": "dropbox", "name": "Dropbox", "icon": "fab fa-dropbox"},
+    {"id": "gdrive", "name": "Google Drive", "icon": "fab fa-google-drive"},
+    {"id": "onedrive", "name": "OneDrive", "icon": "fab fa-microsoft"},
+    {"id": "s3", "name": "Amazon S3", "icon": "fab fa-aws"},
+    {"id": "nextcloud", "name": "Nextcloud", "icon": "fas fa-cloud"},
+    {"id": "webdav", "name": "WebDAV", "icon": "fas fa-server"},
+    {"id": "sftp", "name": "SFTP", "icon": "fas fa-terminal"},
+    {"id": "ftp", "name": "FTP", "icon": "fas fa-server"},
+]
+
+
+def _get_configured_destinations(cfg: Settings) -> list[dict]:
+    """Return which storage providers are fully configured in the current settings.
+
+    Each entry is a dict with ``id``, ``name``, and ``icon`` keys.
+
+    Args:
+        cfg: The application settings object (``app.config.settings``).
+
+    Returns:
+        A list of destination dicts for providers that have the required
+        credentials set.
+    """
+    checks: dict[str, bool] = {
+        "dropbox": bool(cfg.dropbox_refresh_token and cfg.dropbox_app_key),
+        "gdrive": bool(cfg.google_drive_credentials_json or cfg.google_drive_refresh_token),
+        "onedrive": bool(cfg.onedrive_refresh_token and cfg.onedrive_client_id),
+        "s3": bool(cfg.aws_access_key_id and cfg.s3_bucket_name),
+        "nextcloud": bool(cfg.nextcloud_upload_url and cfg.nextcloud_username),
+        "webdav": bool(cfg.webdav_url and cfg.webdav_username),
+        "sftp": bool(cfg.sftp_host and cfg.sftp_username),
+        "ftp": bool(cfg.ftp_host and cfg.ftp_username),
+    }
+    return [meta for meta in _DESTINATION_META if checks.get(meta["id"], False)]
+
+
+# ---------------------------------------------------------------------------
+# Route
+# ---------------------------------------------------------------------------
+
+
+@router.get("/onboarding", include_in_schema=False)
+@require_login
+async def onboarding_page(request: Request, db: Session = Depends(get_db)):
+    """Render the multi-step onboarding wizard.
+
+    Redirects to ``/upload`` when the user has already completed onboarding.
+    """
+    user = request.session.get("user") or {}
+    user_id = user.get("sub") or user.get("preferred_username") or user.get("email") or user.get("id")
+
+    if user_id:
+        profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
+        if profile and profile.onboarding_completed:
+            from starlette.responses import RedirectResponse
+
+            return RedirectResponse(url="/upload", status_code=302)
+
+    configured_destinations = _get_configured_destinations(_settings)
+    tiers = get_all_tiers(db)
+
+    return templates.TemplateResponse(
+        "onboarding.html",
+        {
+            "request": request,
+            "user": user,
+            "configured_destinations": configured_destinations,
+            "tiers": tiers,
+        },
+    )

app/views/pipelines.py

@@ -0,0 +1,34 @@
+"""Pipelines view: management UI for processing pipelines."""
+
+import logging
+
+from fastapi import HTTPException, Request, status
+
+from app.views.base import APIRouter, require_login, settings, templates
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.get("/pipelines")
+@require_login
+async def pipelines_page(request: Request):
+    """Processing pipeline management page for the current user.
+
+    Regular users manage their own pipelines.  Admins additionally have access
+    to system-level pipelines through the same UI.
+    """
+    try:
+        return templates.TemplateResponse(
+            "pipelines.html",
+            {
+                "request": request,
+                "app_version": settings.version,
+            },
+        )
+    except Exception as exc:
+        logger.error(f"Error loading pipelines page: {exc}")
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
+            detail="Failed to load pipelines page",
+        )

app/views/plans.py

@@ -0,0 +1,18 @@
+"""View route for the admin Plan Designer page."""
+
+from fastapi import Request
+from fastapi.responses import HTMLResponse
+from fastapi.routing import APIRouter
+from fastapi.templating import Jinja2Templates
+
+from app.auth import require_login
+
+router = APIRouter()
+templates = Jinja2Templates(directory="frontend/templates")
+
+
+@router.get("/admin/plans", response_class=HTMLResponse)
+@require_login
+async def plan_designer(request: Request) -> HTMLResponse:
+    """Admin Plan Designer page."""
+    return templates.TemplateResponse("admin_plans.html", {"request": request})

app/views/status.py

@@ -73,12 +73,16 @@ async def status_dashboard(request: Request):
     # Get notification URLs for the notification box
     notification_urls = getattr(settings, "notification_urls", [])
 
+    # Get release codename if available
+    release_name = getattr(settings, "release_name", None)
+
     return templates.TemplateResponse(
         "status_dashboard.html",
         {
             "request": request,
             "providers": providers,
             "app_version": settings.version,
+            "release_name": release_name,
             "build_date": build_date,
             "debug_enabled": getattr(settings, "debug", False),
             "last_check": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),

app/views/subscriptions.py

@@ -0,0 +1,88 @@
+"""View routes for subscription-related pages.
+
+Routes:
+  GET /pricing          — public marketing pricing page
+  GET /subscription     — authenticated user's current plan & usage
+"""
+
+import logging
+
+from fastapi import Depends, Request
+from sqlalchemy.orm import Session
+
+from app.utils.subscription import (
+    TIER_ORDER,
+    apply_pending_subscription_changes,
+    get_all_tiers,
+    get_tier,
+    get_user_tier_id,
+    get_user_usage,
+)
+from app.views.base import APIRouter, get_db, require_login, templates
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.get("/pricing", include_in_schema=False)
+async def pricing_page(request: Request, db: Session = Depends(get_db)):
+    """Public-facing pricing and plans page."""
+    tiers = get_all_tiers(db)
+    return templates.TemplateResponse(
+        "pricing.html",
+        {
+            "request": request,
+            "tiers": tiers,
+            "tier_order": TIER_ORDER,
+        },
+    )
+
+
+@router.get("/subscription", include_in_schema=False)
+@require_login
+async def my_subscription_page(request: Request, db: Session = Depends(get_db)):
+    """Authenticated user's subscription status and usage page."""
+    from app.config import settings
+    from app.models import UserProfile
+
+    user = request.session.get("user") or {}
+    owner_id: str = user.get("username") or user.get("email") or user.get("sub") or ""
+
+    if settings.multi_user_enabled and owner_id:
+        # Apply any pending changes that have become due before rendering
+        apply_pending_subscription_changes(db, owner_id)
+
+        tier_id = get_user_tier_id(db, owner_id)
+        usage = get_user_usage(db, owner_id)
+        profile = db.query(UserProfile).filter(UserProfile.user_id == owner_id).first()
+        pending_tier_id = profile.subscription_change_pending_tier if profile else None
+        pending_date = profile.subscription_change_pending_date if profile else None
+        period_start = profile.subscription_period_start if profile else None
+    else:
+        tier_id = "business"
+        usage = None
+        pending_tier_id = None
+        pending_date = None
+        period_start = None
+
+    tier = get_tier(tier_id, db)
+    all_tiers = get_all_tiers(db)
+    pending_tier = get_tier(pending_tier_id, db) if pending_tier_id else None
+
+    return templates.TemplateResponse(
+        "subscription.html",
+        {
+            "request": request,
+            "tier": tier,
+            "tier_id": tier_id,
+            "usage": usage,
+            "all_tiers": all_tiers,
+            "multi_user_enabled": settings.multi_user_enabled,
+            "owner_id": owner_id,
+            "tier_order": TIER_ORDER,
+            "pending_tier_id": pending_tier_id,
+            "pending_tier": pending_tier,
+            "pending_date": pending_date,
+            "period_start": period_start,
+        },
+    )

docker-compose.yaml

@@ -72,3 +72,22 @@ services:
     image: redis:alpine
     container_name: document_redis
     restart: always
+
+  # Optional: standalone docs server using MkDocs Material (live-reload during development).
+  # Start with: docker compose --profile docs up docs
+  # Access at: http://localhost:8001
+  # Note: The /help route in the main app serves pre-built docs from the Docker image.
+  # This service is for content authors who want live preview while editing documentation.
+  docs:
+    image: squidfunk/mkdocs-material:latest
+    container_name: docuelevate_docs
+    restart: "no"
+    profiles:
+      - docs
+    volumes:
+      - ./docs:/docs/docs
+      - ./mkdocs.yml:/docs/mkdocs.yml
+    working_dir: /docs
+    command: ["serve", "--dev-addr=0.0.0.0:8000", "--config-file=/docs/mkdocs.yml"]
+    ports:
+      - "8001:8000"

docs/BillingSetup.md

@@ -0,0 +1,182 @@
+# Billing Setup Guide
+
+This guide covers how to configure Stripe billing and local user sign-up in DocuElevate.
+
+## Table of Contents
+
+- [Local User Sign-up](#local-user-sign-up)
+- [Stripe Billing Integration](#stripe-billing-integration)
+  - [Prerequisites](#prerequisites)
+  - [Configuration](#configuration)
+  - [Setting Up Plans](#setting-up-plans)
+  - [Webhook Configuration](#webhook-configuration)
+  - [Billing Flows](#billing-flows)
+- [Compliance Notes](#compliance-notes)
+
+---
+
+## Local User Sign-up
+
+By default, user accounts are created by an administrator. To allow users to self-register with an email address and password, set `ALLOW_LOCAL_SIGNUP=true`.
+
+> **Note:** SMTP is **optional** for local sign-up. When SMTP is configured, new accounts require email verification before they can log in. Without SMTP, accounts are activated immediately upon registration — useful for self-hosted deployments without email infrastructure.
+
+### Configuration
+
+```bash
+ALLOW_LOCAL_SIGNUP=true
+
+# SMTP (optional — enables email verification and password reset)
+EMAIL_HOST=smtp.example.com
+EMAIL_PORT=587
+EMAIL_USERNAME=noreply@example.com
+EMAIL_PASSWORD=yourpassword
+EMAIL_USE_TLS=true
+EMAIL_SENDER=DocuElevate <noreply@example.com>
+```
+
+### Sign-up Flow
+
+**With SMTP configured (recommended):**
+1. User visits `/signup` and fills out the registration form.
+2. DocuElevate sends a verification email with a 24-hour token link.
+3. User clicks the link — their account is activated and they are signed in.
+4. First-time users are redirected to the onboarding wizard.
+
+**Without SMTP:**
+1. User visits `/signup` and fills out the registration form.
+2. Account is activated immediately — no email verification required.
+3. User is redirected to the login page to sign in straight away.
+
+### Admin-Created Accounts
+
+Administrators can create local user accounts directly from the **Admin → User Management** page without requiring self-registration. Admin-created accounts are immediately active regardless of SMTP configuration.
+
+### Password Reset Flow
+
+1. User clicks "Forgot password?" on the login page.
+2. User enters their email address.
+3. DocuElevate sends a password reset email with a 24-hour token link.
+4. User clicks the link, enters a new password, and is redirected to sign in.
+
+### Security
+
+- Passwords are hashed with bcrypt (12 rounds).
+- Verification and reset tokens are 256-bit URL-safe random strings.
+- All tokens expire after 24 hours.
+- Sign-up and login endpoints return generic error messages to prevent user enumeration.
+
+---
+
+## Stripe Billing Integration
+
+DocuElevate integrates with [Stripe](https://stripe.com) to handle subscription payments. Stripe acts as a data processor under a Data Processing Agreement (DPA) and is SOC 2 Type II certified.
+
+### Prerequisites
+
+- A Stripe account (sign up at [stripe.com](https://stripe.com))
+- Products and prices created in the Stripe Dashboard for each paid plan
+- A publicly reachable webhook endpoint (or use [Stripe CLI](https://stripe.com/docs/stripe-cli) for local testing)
+
+### Configuration
+
+```bash
+STRIPE_SECRET_KEY=sk_live_...         # Your Stripe secret key
+STRIPE_PUBLISHABLE_KEY=pk_live_...    # Your Stripe publishable key (for frontend)
+STRIPE_WEBHOOK_SECRET=whsec_...       # Webhook signing secret
+STRIPE_SUCCESS_URL=https://app.example.com/api/billing/success  # Optional override
+STRIPE_CANCEL_URL=https://app.example.com/pricing               # Optional override
+```
+
+> **Security:** Never commit your Stripe secret key. Store it in your environment or secrets manager.
+
+### Setting Up Plans
+
+After starting DocuElevate, go to **Admin → Plans** to configure each plan:
+
+1. Open the **Plan Designer** for a paid tier (e.g. Starter, Professional).
+2. Enter the **Stripe Price ID (monthly)** from your Stripe Dashboard (e.g. `price_1OtAbc...`).
+3. Optionally enter the **Stripe Price ID (yearly)** for annual billing.
+4. Save the plan.
+
+Stripe Price IDs look like `price_1OtAbcDefGhIjKlMnOpQrSt`. Find them in **Products** in your Stripe Dashboard.
+
+### Webhook Configuration
+
+Stripe webhooks allow DocuElevate to sync subscription status in real time.
+
+#### Stripe Dashboard setup
+
+1. Go to **Developers → Webhooks** in the Stripe Dashboard.
+2. Click **Add endpoint**.
+3. Set the endpoint URL to: `https://your-app-domain.com/api/billing/webhook`
+4. Select the following events:
+   - `checkout.session.completed`
+   - `customer.subscription.updated`
+   - `customer.subscription.deleted`
+   - `invoice.payment_failed`
+5. Copy the **Signing secret** and set `STRIPE_WEBHOOK_SECRET` in your environment.
+
+#### Local testing with Stripe CLI
+
+```bash
+# Install Stripe CLI and log in
+stripe login
+
+# Forward webhooks to your local server
+stripe listen --forward-to http://localhost:8000/api/billing/webhook
+
+# Trigger a test event
+stripe trigger checkout.session.completed
+```
+
+### Billing Flows
+
+#### Subscribe to a plan
+
+1. User visits `/pricing`.
+2. User clicks the **CTA button** on a paid plan.
+3. DocuElevate calls `POST /api/billing/create-checkout-session`.
+4. User is redirected to Stripe Checkout.
+5. After payment, Stripe fires `checkout.session.completed`.
+6. DocuElevate webhook handler activates the subscription tier.
+7. User is redirected to `/api/billing/success`.
+
+#### Manage or cancel subscription
+
+1. User visits their account settings.
+2. DocuElevate calls `POST /api/billing/create-portal-session`.
+3. User is redirected to the Stripe Customer Portal.
+4. User can update payment method, upgrade, downgrade, or cancel.
+5. Stripe fires `customer.subscription.updated` or `customer.subscription.deleted`.
+6. DocuElevate webhook handler syncs the change.
+
+#### Cancellation
+
+When a subscription is cancelled, Stripe fires `customer.subscription.deleted` and DocuElevate automatically downgrades the user to the free tier.
+
+---
+
+## Compliance Notes
+
+| Topic | Details |
+|-------|---------|
+| **GDPR** | Stripe acts as a data processor. A Data Processing Agreement (DPA) is available in the Stripe Dashboard. Stripe supports EU data residency. |
+| **SOC 2** | Stripe is SOC 2 Type II certified. |
+| **EU VAT** | Configure [Stripe Tax](https://stripe.com/tax) in the Stripe Dashboard for automatic VAT collection. |
+| **PCI DSS** | Card data is handled entirely by Stripe. DocuElevate never sees or stores card details. |
+
+---
+
+## Environment Variable Reference
+
+| Variable | Type | Default | Description |
+|----------|------|---------|-------------|
+| `ALLOW_LOCAL_SIGNUP` | bool | `false` | Allow users to self-register with email/password |
+| `STRIPE_SECRET_KEY` | string | — | Stripe API secret key |
+| `STRIPE_PUBLISHABLE_KEY` | string | — | Stripe API publishable key |
+| `STRIPE_WEBHOOK_SECRET` | string | — | Webhook signing secret from Stripe Dashboard |
+| `STRIPE_SUCCESS_URL` | string | — | Override redirect URL after successful checkout |
+| `STRIPE_CANCEL_URL` | string | — | Override redirect URL when checkout is cancelled |
+
+See [ConfigurationGuide.md](./ConfigurationGuide.md) for the full environment variable reference.

docs/CIWorkflow.md

@@ -13,8 +13,7 @@ The CI workflow (`.github/workflows/tests.yaml`) runs automatically on every pus
 | `lint` | Ruff | Fast Python linter (replaces Flake8, Black, isort, Bandit) | ✅ |
 | `html-lint` | djLint | HTML template accessibility linter | ✅ |
 | `dependency-scan` | pip-audit | Dependency vulnerability scanning against OSV/PyPA advisories | ✅ |
-| `test-quick` | pytest | Unit + basic integration tests with coverage (~2 min) | ✅ |
-| `test-integration` | pytest | Docker container and external service tests (~5 min) | ✅ |
+| `run-tests` | pytest | All unit + integration tests with coverage (excludes e2e) | ✅ |
 | `mypy` | mypy | Static type checking | ✅ |
 
 ### Pipeline Flow
@@ -22,34 +21,24 @@ The CI workflow (`.github/workflows/tests.yaml`) runs automatically on every pus
 ```
 Stage 1 (parallel):  lint, html-lint, dependency-scan
                           │
-Stage 2 (parallel):  test-quick + mypy
+Stage 2 (parallel):  run-tests + mypy
                           │
-Stage 3:             test-integration  (only after test-quick passes)
-                          │
-Stage 4:             build  (only after all above pass)
+Stage 3:             build  (only after all above pass)
 ```
 
-The pipeline follows a **fail-early** strategy: fast linters and quick tests run first to catch regressions early. Heavier integration tests only run after the quick tests pass, saving CI time when basic issues are present.
+The pipeline follows a **fail-early** strategy: fast linters run first to catch regressions early, then the full test suite and type checks run in parallel.
 
 > **Note:** DocuElevate uses Ruff, a modern all-in-one Python linter that consolidates the functionality of Flake8, Black, isort, and Bandit. This streamlined approach reduces CI complexity while maintaining code quality and security standards.
 
-### Quick Tests (`test-quick`)
+### All Tests (`run-tests`)
 
-- **Timeout:** 15 minutes (job), 120 seconds (per test via `pytest-timeout`)
-- Runs the majority of tests (~2,790 unit + basic integration tests)
-- Excludes tests marked `e2e`, `requires_docker`, `requires_external`, or `slow`
-- Uses a Redis service container for tests that need it
-- Collects coverage and uploads to Codecov
-- Uploads `junit.xml` and `coverage.xml` as workflow artifacts
-
-### Integration Tests (`test-integration`)
-
-- **Timeout:** 20 minutes (job), 300 seconds (per test via `pytest-timeout`)
-- Runs tests marked `requires_docker`, `requires_external`, or `slow` (excluding `e2e`)
+- **Timeout:** 300 seconds (per test via `pytest-timeout`)
+- Runs all tests (unit, integration, and Docker-based) in a single step
+- Excludes tests marked `e2e`
 - Uses Redis and RabbitMQ service containers
 - Docker daemon available for testcontainers (WebDAV, OAuth mock server, etc.)
-- Only runs after quick tests pass (fail-early gate)
-- Uploads `junit-integration.xml` as a workflow artifact
+- Collects coverage and uploads to Codecov
+- Uploads `junit.xml` and `coverage.xml` as workflow artifacts
 
 ### Ruff Lint & Format
 
@@ -73,8 +62,7 @@ The following artifacts are uploaded after every run:
 
 | Artifact | Contents | Condition |
 |-------------------------------|--------------------------------------|----------------------|
-| `test-results-quick` | `junit.xml`, `coverage.xml` | Always (unless cancelled) |
-| `test-results-integration` | `junit-integration.xml` | Always (unless cancelled) |
+| `test-results` | `junit.xml`, `coverage.xml` | Always (unless cancelled) |
 
 ## Running Linters Locally
 
@@ -93,14 +81,14 @@ ruff format --check app/ tests/
 # Run type checking
 mypy app/
 
-# Run quick tests (same as CI quick stage)
-pytest tests/ -v --timeout=120 --cov=app --cov-report=term -m "not e2e and not requires_docker and not requires_external and not slow"
+# Run all tests except e2e (same as CI)
+pytest tests/ -v --timeout=300 --cov=app --cov-report=term -m "not e2e"
 
-# Run integration tests (requires Docker)
+# Run only fast tests locally (skip Docker/external/slow)
+pytest tests/ -v --timeout=120 -m "not e2e and not requires_docker and not requires_external and not slow"
+
+# Run only Docker/external/slow tests locally (requires Docker)
 pytest tests/ -v --timeout=300 -m "(requires_docker or requires_external or slow) and not e2e"
-
-# Run all tests except e2e
-pytest tests/ -v --timeout=120 -m "not e2e"
 ```
 
 Or use pre-commit hooks to run checks automatically on each commit (recommended):

docs/ConfigurationGuide.md

@@ -10,7 +10,7 @@ Configuration is primarily done through environment variables specified in a `.e
 
 | **Variable**           | **Description**                                          | **Example**                    |
 |------------------------|----------------------------------------------------------|--------------------------------|
-| `DATABASE_URL`         | Path/URL to the SQLite database (or other SQL backend). | `sqlite:///./app/database.db`  |
+| `DATABASE_URL`         | Path/URL to the SQLite database (or other SQL backend). Use the [Database Wizard](/database-wizard) for guided setup. See [Database Configuration](DatabaseConfiguration.md). | `sqlite:///./app/database.db`  |
 | `REDIS_URL`            | URL for Redis, used by Celery for broker & result store. | `redis://redis:6379/0`         |
 | `WORKDIR`              | Working directory for the application.                  | `/workdir`                     |
 | `GOTENBERG_URL`        | Gotenberg PDF processing URL.                           | `http://gotenberg:3000`        |
@@ -31,6 +31,40 @@ Control how the `/processall` endpoint handles large batches of files to prevent
 - Total queue time: (25-1) × 3 = 72 seconds
 - Prevents API rate limit issues and ensures smooth processing
 
+### Task Retry Settings
+
+Failed Celery tasks are automatically retried with exponential backoff and optional jitter. Different task types use different default delays (OCR tasks wait longer than upload tasks to account for API rate limits).
+
+| **Variable**             | **Description**                                                                                                                                   | **Default**     |
+|--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|
+| `TASK_RETRY_MAX_RETRIES` | Maximum number of retry attempts for any failed task.                                                                                             | `3`             |
+| `TASK_RETRY_DELAYS`      | Comma-separated list of countdown values in seconds for each retry attempt. Values beyond the list double the last entry for subsequent retries. | `60,300,900`    |
+| `TASK_RETRY_JITTER`      | Apply ±20 % random jitter to countdowns to prevent thundering-herd problems when many tasks fail at the same time.                               | `true`          |
+
+**Per-task-type policies** (not overridable via environment variables; set in code):
+
+| Task type               | Default delays (s)     | Notes                                               |
+|-------------------------|------------------------|-----------------------------------------------------|
+| General tasks           | 60, 300, 900           | Controlled by `TASK_RETRY_DELAYS`                   |
+| OCR / AI tasks          | 120, 600, 1800         | Longer waits for API rate-limit windows to clear    |
+| Cloud-storage uploads   | 60, 300, 900           | Controlled by `TASK_RETRY_DELAYS`                   |
+
+**Example – aggressive retries for a high-availability setup:**
+
+```dotenv
+TASK_RETRY_MAX_RETRIES=5
+TASK_RETRY_DELAYS=30,120,600,1800,3600
+TASK_RETRY_JITTER=true
+```
+
+**Example – conservative retries with longer back-off:**
+
+```dotenv
+TASK_RETRY_MAX_RETRIES=3
+TASK_RETRY_DELAYS=300,900,3600
+TASK_RETRY_JITTER=true
+```
+
 ### Client-Side Upload Throttling
 
 Control how the web UI queues and paces file uploads to avoid overwhelming the backend, especially when dragging large directories (potentially thousands of files) onto the upload area.
@@ -87,9 +121,157 @@ MAX_SINGLE_FILE_SIZE=524288000
 - **With splitting**: Recommended for servers with limited memory or when processing very large scanned documents
 - **Higher limits**: For environments specifically designed to handle large architectural plans, books, or scanned archives
 
-### IMAP Configuration
+### Watch Folder Ingestion
 
-DocuElevate can monitor multiple IMAP mailboxes for document attachments. Each mailbox uses a numbered prefix (e.g., `IMAP1_`, `IMAP2_`).
+DocuElevate can automatically monitor directories for new files and ingest them without any manual action.
+This works for:
+- **Local filesystem paths** — including SMB/CIFS shares, NFS mounts, or any path accessible to the Docker container
+- **FTP server directories** — using the configured FTP connection credentials
+- **SFTP server directories** — using the configured SFTP connection credentials
+
+#### Local Watch Folders
+
+Mount the share or directory into the Docker container and configure one or more paths to watch.
+
+| **Variable**                        | **Description**                                                                              | **Default** |
+|-------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `WATCH_FOLDERS`                     | Comma-separated list of **absolute** local filesystem paths to poll for new files.          | *(empty)*   |
+| `WATCH_FOLDER_POLL_INTERVAL`        | How often to scan the folders, in minutes.                                                  | `1`         |
+| `WATCH_FOLDER_DELETE_AFTER_PROCESS` | Delete source files from the watch folder after they are successfully enqueued. When `false`, processed files are tracked in a cache file to prevent re-ingestion. | `false` |
+
+**Example (docker-compose.yaml):**
+
+```yaml
+services:
+  worker:
+    volumes:
+      - /mnt/smb/scanner:/watchfolders/scanner  # SMB/CIFS share mounted on the host
+      - /mnt/nfs/inbox:/watchfolders/inbox       # NFS mount
+    environment:
+      WATCH_FOLDERS: /watchfolders/scanner,/watchfolders/inbox
+      WATCH_FOLDER_POLL_INTERVAL: 1
+      WATCH_FOLDER_DELETE_AFTER_PROCESS: false
+```
+
+> **Tip for HP Scanners and MFPs**: Configure your scanner's "Scan to Network Folder" to point at an SMB share that is also mounted into the DocuElevate worker container. DocuElevate will pick up the scan files automatically every minute. No email forwarding is required.
+
+#### FTP Ingest (Watch Folder)
+
+DocuElevate can poll an FTP server directory for new files. It reuses the FTP connection settings already configured for uploads.
+
+| **Variable**                      | **Description**                                                                                | **Default** |
+|-----------------------------------|------------------------------------------------------------------------------------------------|-------------|
+| `FTP_INGEST_ENABLED`              | Enable FTP folder watching (`true`/`false`).                                                  | `false`     |
+| `FTP_INGEST_FOLDER`               | Path on the FTP server to poll (e.g. `/incoming`). Uses the existing FTP connection settings. | *(empty)*   |
+| `FTP_INGEST_DELETE_AFTER_PROCESS` | Delete files from the FTP server after they are downloaded and enqueued.                      | `false`     |
+
+**Example:**
+
+```dotenv
+# Existing FTP upload settings (also used for ingest)
+FTP_HOST=ftp.example.com
+FTP_USERNAME=docuelevate
+FTP_PASSWORD=secret
+
+# FTP ingest configuration
+FTP_INGEST_ENABLED=true
+FTP_INGEST_FOLDER=/incoming
+FTP_INGEST_DELETE_AFTER_PROCESS=false
+```
+
+#### SFTP Ingest (Watch Folder)
+
+DocuElevate can poll an SFTP server directory for new files. It reuses the SFTP connection settings already configured for uploads.
+
+| **Variable**                       | **Description**                                                                                 | **Default** |
+|------------------------------------|-------------------------------------------------------------------------------------------------|-------------|
+| `SFTP_INGEST_ENABLED`              | Enable SFTP folder watching (`true`/`false`).                                                  | `false`     |
+| `SFTP_INGEST_FOLDER`               | Path on the SFTP server to poll (e.g. `/uploads/inbox`). Uses the existing SFTP connection settings. | *(empty)* |
+| `SFTP_INGEST_DELETE_AFTER_PROCESS` | Delete files from the SFTP server after they are downloaded and enqueued.                      | `false`     |
+
+**Example:**
+
+```dotenv
+# Existing SFTP upload settings (also used for ingest)
+SFTP_HOST=sftp.example.com
+SFTP_USERNAME=docuelevate
+SFTP_PRIVATE_KEY=/run/secrets/sftp_key
+
+# SFTP ingest configuration
+SFTP_INGEST_ENABLED=true
+SFTP_INGEST_FOLDER=/uploads/inbox
+SFTP_INGEST_DELETE_AFTER_PROCESS=false
+```
+
+#### Supported File Types for Watch Folders
+
+Watch folder ingestion accepts the same file types as the web upload interface: PDF, Word, Excel, PowerPoint, images (JPEG, PNG, TIFF, BMP, GIF), plain text, CSV, RTF, and more. Unsupported files (executables, archives, etc.) are silently skipped.
+
+#### Dropbox Ingest (Watch Folder)
+
+DocuElevate can poll a Dropbox folder for new files. It reuses the Dropbox OAuth credentials already configured for uploads.
+
+| **Variable**                          | **Description**                                                                              | **Default** |
+|---------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `DROPBOX_INGEST_ENABLED`              | Enable Dropbox folder watching (`true`/`false`).                                            | `false`     |
+| `DROPBOX_INGEST_FOLDER`               | Dropbox folder path to poll (e.g. `/Inbox/Scanner`). Uses the existing Dropbox OAuth credentials. | *(empty)* |
+| `DROPBOX_INGEST_DELETE_AFTER_PROCESS` | Delete files from Dropbox after they are downloaded and enqueued.                          | `false`     |
+
+#### Google Drive Ingest (Watch Folder)
+
+DocuElevate can poll a Google Drive folder for new files. It reuses the existing Google Drive service-account or OAuth credentials.
+
+| **Variable**                                 | **Description**                                                                              | **Default** |
+|----------------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `GOOGLE_DRIVE_INGEST_ENABLED`                | Enable Google Drive folder watching (`true`/`false`).                                       | `false`     |
+| `GOOGLE_DRIVE_INGEST_FOLDER_ID`              | Google Drive **folder ID** to poll (copy from the URL of the target folder in Drive). Uses the existing Google Drive credentials. | *(empty)* |
+| `GOOGLE_DRIVE_INGEST_DELETE_AFTER_PROCESS`   | Delete files from Google Drive after they are downloaded and enqueued.                      | `false`     |
+
+#### OneDrive Ingest (Watch Folder)
+
+DocuElevate can poll a OneDrive folder for new files. It reuses the existing OneDrive MSAL (client ID/secret/refresh token) credentials.
+
+| **Variable**                               | **Description**                                                                              | **Default** |
+|--------------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `ONEDRIVE_INGEST_ENABLED`                  | Enable OneDrive folder watching (`true`/`false`).                                           | `false`     |
+| `ONEDRIVE_INGEST_FOLDER_PATH`              | OneDrive folder path to poll (e.g. `/Inbox/Scanner`). Uses the existing OneDrive credentials. | *(empty)* |
+| `ONEDRIVE_INGEST_DELETE_AFTER_PROCESS`     | Delete files from OneDrive after they are downloaded and enqueued.                          | `false`     |
+
+#### Nextcloud Ingest (Watch Folder)
+
+DocuElevate can poll a Nextcloud folder via WebDAV for new files. It reuses the existing Nextcloud upload URL and credentials.
+
+| **Variable**                               | **Description**                                                                              | **Default** |
+|--------------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `NEXTCLOUD_INGEST_ENABLED`                 | Enable Nextcloud folder watching (`true`/`false`).                                          | `false`     |
+| `NEXTCLOUD_INGEST_FOLDER`                  | Nextcloud folder path to poll (e.g. `/Scans/Inbox`). Uses the existing Nextcloud upload URL and credentials. | *(empty)* |
+| `NEXTCLOUD_INGEST_DELETE_AFTER_PROCESS`    | Delete files from Nextcloud after they are downloaded and enqueued.                         | `false`     |
+
+#### Amazon S3 Ingest (Watch Folder)
+
+DocuElevate can poll an S3 bucket prefix for new objects. It reuses the existing S3/AWS credentials and bucket name.
+
+| **Variable**                          | **Description**                                                                              | **Default** |
+|---------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `S3_INGEST_ENABLED`                   | Enable S3 prefix watching (`true`/`false`).                                                 | `false`     |
+| `S3_INGEST_PREFIX`                    | S3 key prefix to poll (e.g. `inbox/scanner/`). Uses the existing S3 bucket and AWS credentials. | *(empty)* |
+| `S3_INGEST_DELETE_AFTER_PROCESS`      | Delete objects from S3 after they are downloaded and enqueued.                              | `false`     |
+
+#### WebDAV Ingest (Watch Folder)
+
+DocuElevate can poll a WebDAV folder for new files. It reuses the existing WebDAV URL and credentials.
+
+| **Variable**                          | **Description**                                                                              | **Default** |
+|---------------------------------------|----------------------------------------------------------------------------------------------|-------------|
+| `WEBDAV_INGEST_ENABLED`               | Enable WebDAV folder watching (`true`/`false`).                                             | `false`     |
+| `WEBDAV_INGEST_FOLDER`                | WebDAV folder path to poll. Uses the existing WebDAV URL and credentials.                   | *(empty)* |
+| `WEBDAV_INGEST_DELETE_AFTER_PROCESS`  | Delete files from WebDAV after they are downloaded and enqueued.                            | `false`     |
+
+### IMAP Email Ingestion
+
+DocuElevate can automatically pull document attachments from IMAP mailboxes — no need to forward emails manually. Configure one or two mailboxes and DocuElevate polls them on the schedule you set.
+
+> **For HP Scanners (Scan to Email)**: If your scanner is set up to email scanned documents to a dedicated mailbox, configure that mailbox in DocuElevate using the settings below. DocuElevate will automatically retrieve the scanned PDFs from the inbox and process them. You do **not** need to configure DocuElevate as an email server — it acts as an email *client* that reads from your existing mailbox.
 
 | **Variable**                  | **Description**                                              | **Example**       |
 |-------------------------------|--------------------------------------------------------------|-------------------|
@@ -115,6 +297,62 @@ DocuElevate can monitor multiple IMAP mailboxes for document attachments. Each m
 | `AUTHENTIK_CONFIG_URL`  | Configuration URL for Authentik OpenID Connect.             |
 | `OAUTH_PROVIDER_NAME`   | Display name for the OAuth provider button.                  |
 
+### Multi-User Mode
+
+When multi-user mode is enabled, each authenticated user gets their own isolated document space.
+Uploads, search results, and file management are scoped to the individual user. Shared settings
+(AI configuration, OCR providers, storage destinations) remain global.
+
+Admin users (determined by `ADMIN_GROUP_NAME`) bypass the user filter and can see all documents.
+
+Requires `AUTH_ENABLED=true`.
+
+| **Variable**                | **Description**                                                                 | **Default** |
+|-----------------------------|---------------------------------------------------------------------------------|-------------|
+| `MULTI_USER_ENABLED`        | Enable multi-user mode with individual document spaces per user.               | `false`     |
+| `DEFAULT_DAILY_UPLOAD_LIMIT`| Maximum document uploads allowed per user per day. `0` = unlimited.            | `0`         |
+| `UNOWNED_DOCS_VISIBLE_TO_ALL` | Show unclaimed documents (no owner) to all users. When `false`, only admins see them. | `true` |
+| `DEFAULT_OWNER_ID`          | Automatically assign this owner to newly ingested documents without a session (e.g. IMAP, API). Leave empty to keep unowned. | *(empty)* |
+
+#### Unclaimed Documents
+
+Documents ingested without a user session (e.g. via IMAP polling, API calls without authentication,
+or legacy imports) have `owner_id = NULL`. These are called **unclaimed** documents.
+
+- When `UNOWNED_DOCS_VISIBLE_TO_ALL=true` (default), every authenticated user sees unclaimed
+  documents alongside their own files. This allows users to discover and claim them.
+- When `UNOWNED_DOCS_VISIBLE_TO_ALL=false`, only admins can see unclaimed documents.
+
+#### Claiming Documents
+
+Users can claim unclaimed documents via the API:
+
+- **`POST /api/files/{file_id}/claim`** — Claim a single unclaimed document.
+- **`POST /api/files/bulk-claim`** — Claim multiple unclaimed documents at once.
+
+Only documents with `owner_id = NULL` can be claimed. Already-owned documents cannot be claimed
+by another user.
+
+#### Admin Owner Assignment
+
+Admins can assign ownership of documents to any user:
+
+- **`POST /api/files/assign-owner?owner_id=<user_id>`** — Assign all unclaimed documents to
+  the specified user, or pass a `file_ids` JSON body to assign specific files.
+
+The `DEFAULT_OWNER_ID` setting can also be configured via the Settings page, which provides an
+autocomplete field that searches existing users by substring.
+
+### Subscriptions & Upload Quotas
+
+DocuElevate supports configurable subscription plans with per-user upload quotas enforced at upload time.
+Plans are managed via the **Plan Designer** at `/admin/plans`. The following global setting controls the
+default overage buffer applied across all plans.
+
+| **Variable**                     | **Description**                                                                                                                                                                                                     | **Default** |
+|----------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
+| `SUBSCRIPTION_OVERAGE_PERCENT`   | Soft-limit overage buffer in percent (0–200). The announced monthly quota is multiplied by `(1 + percent/100)` for actual enforcement. E.g. `20` means a 150-doc/month plan enforces at 180 docs (150 × 1.20). Set `0` to enforce exactly at the announced limit. Per-plan `overage_percent` configured in the Plan Designer overrides this global default. | `20` |
+
 ### Security Headers
 
 DocuElevate supports HTTP security headers to improve browser-side security. **These headers are disabled by default** since most deployments use a reverse proxy (Traefik, Nginx, etc.) that already adds them. Enable only if deploying directly without a reverse proxy. See [Deployment Guide - Security Headers](DeploymentGuide.md#security-headers) for detailed configuration examples.
@@ -716,7 +954,11 @@ For detailed setup instructions, see the [Google Drive Setup Guide](GoogleDriveS
 | `SFTP_PRIVATE_KEY`           | Path to private key file for authentication (optional). |
 | `SFTP_PRIVATE_KEY_PASSPHRASE`| Passphrase for private key if required (optional).     |
 
-### Email
+### Email (shared SMTP – password reset & verification)
+
+> **Note:** These settings configure the shared SMTP connection used for system emails such as
+> password resets and account verification. They do **not** enable the email delivery destination.
+> To send processed documents via email, configure the dedicated `DEST_EMAIL_*` variables below.
 
 | **Variable**                | **Description**                                           |
 |----------------------------|----------------------------------------------------------|
@@ -726,7 +968,22 @@ For detailed setup instructions, see the [Google Drive Setup Guide](GoogleDriveS
 | `EMAIL_PASSWORD`           | SMTP authentication password.                             |
 | `EMAIL_USE_TLS`            | Whether to use TLS (default: `True`).                     |
 | `EMAIL_SENDER`             | From address (e.g., `"DocuElevate <docuelevate@example.com>"`). |
-| `EMAIL_DEFAULT_RECIPIENT`  | Default recipient email if none specified in the task.    |
+
+### Email Destination (document delivery)
+
+> **Note:** These settings are intentionally separate from the shared `EMAIL_*` settings above.
+> Configuring `EMAIL_HOST` for password resets does **not** automatically activate the email
+> delivery destination. You must set `DEST_EMAIL_HOST` to enable it.
+
+| **Variable**                     | **Description**                                                     |
+|----------------------------------|---------------------------------------------------------------------|
+| `DEST_EMAIL_HOST`               | SMTP server hostname for document delivery.                          |
+| `DEST_EMAIL_PORT`               | SMTP port for document delivery (default: `587`).                    |
+| `DEST_EMAIL_USERNAME`           | SMTP authentication username for document delivery.                  |
+| `DEST_EMAIL_PASSWORD`           | SMTP authentication password for document delivery.                  |
+| `DEST_EMAIL_USE_TLS`            | Whether to use TLS for document delivery (default: `True`).          |
+| `DEST_EMAIL_SENDER`             | From address for delivered documents (e.g., `"DocuElevate Delivery <docuelevate@example.com>"`). |
+| `DEST_EMAIL_DEFAULT_RECIPIENT`  | Fallback recipient email when none is specified for a delivery task.  |
 
 ### OneDrive / Microsoft Graph
 
@@ -763,9 +1020,71 @@ For detailed setup instructions, see the [Amazon S3 Setup Guide](AmazonS3Setup.m
 | `NOTIFY_ON_CREDENTIAL_FAILURE` | Send notifications on credential failures (`True`/`False`) |
 | `NOTIFY_ON_STARTUP`        | Send notification when system starts (`True`/`False`)    |
 | `NOTIFY_ON_SHUTDOWN`       | Send notification when system shuts down (`True`/`False`)|
+| `NOTIFY_ON_FILE_PROCESSED` | Send notification when a file is successfully processed (`True`/`False`) |
+| `NOTIFY_ON_USER_SIGNUP`    | Send admin notification when a new user signs up (`True`/`False`, default `True`) |
+| `NOTIFY_ON_PLAN_CHANGE`    | Send admin notification when a user changes their subscription plan (`True`/`False`, default `True`) |
+| `NOTIFY_ON_PAYMENT_ISSUE`  | Send admin notification when a payment issue is reported for a user (`True`/`False`, default `True`) |
+
+#### User-Event Notifications
+
+DocuElevate sends admin push notifications (via Apprise) and fires outbound webhooks for three
+user-lifecycle events:
+
+| Event | Trigger | Notification type |
+|-------|---------|-------------------|
+| **New signup** | A first-time user logs in and a UserProfile is created | `NOTIFY_ON_USER_SIGNUP` |
+| **Plan change** | A user selects a new subscription tier during onboarding, or an admin changes their tier | `NOTIFY_ON_PLAN_CHANGE` |
+| **Payment issue** | An admin POSTs to `/api/admin/users/{user_id}/payment-issue` | `NOTIFY_ON_PAYMENT_ISSUE` |
+
+In addition to the Apprise push notification, each event also fires the matching webhook event
+(`user.signup`, `user.plan_changed`, `user.payment_issue`) to all active webhook configurations
+subscribed to that event, enabling integration with CRM, helpdesk (Jira, Zendesk, etc.), or
+payment processors.
 
 For detailed setup instructions, see the [Notifications Setup Guide](NotificationsSetup.md).
 
+### Webhooks
+
+Webhooks notify external systems via HTTP POST when document events occur.
+Configurations are stored in the database and managed through the API (see [API docs](API.md#webhooks)).
+
+| **Variable**        | **Description**                                                  | **Default** |
+|---------------------|------------------------------------------------------------------|-------------|
+| `WEBHOOK_ENABLED`   | Enable or disable webhook delivery globally (`True`/`False`)    | `True`      |
+
+Webhook URLs, secrets, and subscribed events are configured per-webhook via the `/api/webhooks/` endpoints (admin access required). Each delivery includes an optional HMAC-SHA256 signature for verification and is retried with exponential backoff on failure.
+
+### Backup & Restore
+
+DocuElevate automatically backs up the database on a scheduled basis.
+Backups are managed from the **Admin → Backup & Restore** dashboard.
+
+Supported database backends: **SQLite** (`.db.gz`), **PostgreSQL** (`.pgsql.gz`), **MySQL / MariaDB** (`.mysql.gz`).
+For PostgreSQL and MySQL backups the respective CLI client (`pg_dump` / `psql` or `mysqldump` / `mysql`) must be installed on the Celery worker host.
+See the [Database Configuration Guide](DatabaseConfiguration.md#backup-procedures) for setup details.
+
+| **Variable**                   | **Description**                                                                               | **Default**         |
+|--------------------------------|-----------------------------------------------------------------------------------------------|---------------------|
+| `BACKUP_ENABLED`               | Enable or disable automatic scheduled backups (`True`/`False`).                              | `True`              |
+| `BACKUP_DIR`                   | Filesystem path where local backup archives are stored. Defaults to `<WORKDIR>/backups`.     | *(workdir/backups)* |
+| `BACKUP_REMOTE_DESTINATION`    | Storage provider to copy backups to. Options: `s3`, `dropbox`, `google_drive`, `onedrive`, `nextcloud`, `webdav`, `ftp`, `sftp`, `email`. Leave empty for local-only storage. | *(empty)*           |
+| `BACKUP_REMOTE_FOLDER`         | Sub-folder / key prefix used when uploading to the remote destination.                       | `backups`           |
+| `BACKUP_RETAIN_HOURLY`         | Number of hourly snapshots to keep (1 per hour = 96 covers 4 days).                         | `96`                |
+| `BACKUP_RETAIN_DAILY`          | Number of daily snapshots to keep (21 = 3 weeks).                                           | `21`                |
+| `BACKUP_RETAIN_WEEKLY`         | Number of weekly snapshots to keep (13 ≈ 3 months).                                         | `13`                |
+
+**Retention schedule:**
+
+| Tier    | Frequency        | Default retention | Coverage     |
+|---------|------------------|-------------------|--------------|
+| Hourly  | Every hour       | 96 snapshots      | ~4 days      |
+| Daily   | Daily at 02:00   | 21 snapshots      | ~3 weeks     |
+| Weekly  | Sundays at 03:00 | 13 snapshots      | ~3 months    |
+
+Archives beyond the retention window are automatically pruned after each new backup. The **Clean Up** button on the dashboard applies retention immediately. When a remote destination is configured, remote copies follow the same retention policy.
+
+> **Note:** Backup and restore is currently supported only for SQLite databases.
+
 ### Uptime Kuma
 
 | **Variable**                | **Description**                                                |
@@ -799,6 +1118,139 @@ Administrators can set the **site-wide default** colour scheme that is applied w
 UI_DEFAULT_COLOR_SCHEME=dark
 ```
 
+## Duplicate Document Detection
+
+DocuElevate detects and flags documents that share the same content, even if they arrive as separate uploads.
+
+### Exact Duplicate Detection (SHA-256)
+
+When `ENABLE_DEDUPLICATION=True` (the default), each new document is hashed with SHA-256 before processing begins. If the hash matches an existing file record the new document is stored as a duplicate (`is_duplicate=True`, `duplicate_of_id=<original_id>`) and no further processing is performed.
+
+| Variable | Description | Default |
+|---|---|---|
+| `ENABLE_DEDUPLICATION` | Hash-based exact duplicate detection on ingest. | `True` |
+| `SHOW_DEDUPLICATION_STEP` | Show the "Check for Duplicates" step in the processing timeline UI. | `True` |
+
+An immediate duplicate warning is also included in the `/api/ui-upload` JSON response so the frontend can alert the user before the pipeline completes.
+
+### Near-Duplicate Detection (Content Similarity)
+
+Near-duplicate detection catches documents that contain the **same content but carry different SHA-256 hashes** — for example, the same letter scanned twice on different days.
+
+After OCR processes a document, its extracted text is converted to a vector embedding using the configured AI provider. The cosine similarity between two documents' embeddings reflects how semantically similar their content is.
+
+| Variable | Description | Default |
+|---|---|---|
+| `NEAR_DUPLICATE_THRESHOLD` | Minimum cosine similarity (0–1) for two documents to be considered near-duplicates. `0.85` means ≥ 85 % semantic overlap. | `0.85` |
+| `EMBEDDING_MODEL` | Model name for generating text embeddings via the OpenAI-compatible API.  Must be supported by the endpoint configured with `OPENAI_BASE_URL`. | `text-embedding-3-small` |
+| `EMBEDDING_MAX_TOKENS` | Maximum tokens to send to the embedding model.  Text is truncated to approximately this many tokens before calling the API.  Set below the model's context window (e.g. 8 000 for an 8 192-token model). | `8000` |
+
+Near-duplicate detection:
+- Embeddings are computed **automatically during document ingestion** as a processing step ("Compute Embedding").
+- A periodic **backfill task** (every 5 minutes) picks up any files that were processed before the embedding pipeline was enabled.
+- The **Similarity dashboard** (`/similarity`) shows all pairs of documents above the threshold, ranked by score.
+- The **Duplicates** management page (`/duplicates` → "Near-Duplicate Finder" tab) allows per-file lookup.
+- Debug endpoints are available to inspect embedding status and trigger recomputation (see API docs).
+- Documents without OCR text cannot be compared and are excluded from results.
+
+A score of **≥ 0.90** reliably identifies the same document scanned twice. A score of **0.70–0.90** suggests partial content overlap. Adjust `NEAR_DUPLICATE_THRESHOLD` to tune sensitivity.
+
+## PDF/A Archival Conversion
+
+DocuElevate can optionally generate **PDF/A** archival copies of both the
+original ingested file and the processed file.  PDF/A copies are saved as
+parallel variants alongside the standard files—they do **not** replace the
+originals.  This provides better legal coverage by producing time-stamped,
+self-contained archival documents suitable for long-term storage and
+compliance.
+
+The conversion uses **ocrmypdf** (backed by Ghostscript), which is already
+bundled in the Docker images.
+
+> **Note:** PDF/A conversion may alter font rendering, especially for OCR text
+> overlays produced by Microsoft Azure Document Intelligence.  This is expected
+> and is why PDF/A copies are kept as parallel variants rather than
+> replacements.
+
+| Variable                      | Description                                                                                                       | Default                    |
+|-------------------------------|-------------------------------------------------------------------------------------------------------------------|----------------------------|
+| `ENABLE_PDFA_CONVERSION`     | Enable PDF/A archival variant generation for both original and processed files.                                     | `false`                    |
+| `PDFA_FORMAT`                | PDF/A format variant: `1` (PDF/A-1b), `2` (PDF/A-2b), `3` (PDF/A-3b).                                             | `2`                        |
+| `PDFA_UPLOAD_ORIGINAL`       | Upload the original-file PDF/A variant to all configured storage providers.                                        | `false`                    |
+| `PDFA_UPLOAD_PROCESSED`      | Upload the processed-file PDF/A variant to all configured storage providers.                                       | `false`                    |
+| `PDFA_UPLOAD_FOLDER`         | Subfolder name appended to each provider's folder for PDF/A uploads.                                               | `pdfa`                     |
+| `GOOGLE_DRIVE_PDFA_FOLDER_ID`| Google Drive folder ID for PDF/A uploads (uses folder IDs, not paths). Empty = use default folder.                 | *(empty)*                  |
+| `PDFA_TIMESTAMP_ENABLED`     | Enable RFC 3161 timestamping of PDF/A files (creates `.tsr` proof-of-existence files).                             | `false`                    |
+| `PDFA_TIMESTAMP_URL`         | URL of the RFC 3161 Timestamp Authority.                                                                           | `https://freetsa.org/tsr`  |
+
+### Storage Layout
+
+When enabled, PDF/A copies are stored under `workdir/pdfa/`:
+
+```
+workdir/
+├── original/          # Immutable copy of ingested file
+├── processed/         # Processed file with embedded metadata
+├── pdfa/
+│   ├── original/      # PDF/A copy of the ingested file
+│   │   └── *.pdf.tsr  # RFC 3161 timestamps (when timestamping enabled)
+│   └── processed/     # PDF/A copy of the processed file (with -PDFA suffix)
+│       └── *.pdf.tsr  # RFC 3161 timestamps (when timestamping enabled)
+└── tmp/               # Temporary processing area
+```
+
+### Per-Provider Folder Overrides
+
+When uploading PDF/A files to storage providers, DocuElevate appends the
+`PDFA_UPLOAD_FOLDER` value as a subfolder to each provider's configured folder.
+For example:
+
+| Provider     | Regular Folder              | PDF/A Upload Folder              |
+|--------------|-----------------------------|----------------------------------|
+| Dropbox      | `/Documents`                | `/Documents/pdfa`                |
+| S3           | `docs/uploads/`             | `docs/uploads/pdfa/`             |
+| Nextcloud    | `/Files`                    | `/Files/pdfa`                    |
+| OneDrive     | `Documents/Uploads`         | `Documents/Uploads/pdfa`         |
+| Google Drive | *(folder ID)*               | `GOOGLE_DRIVE_PDFA_FOLDER_ID`    |
+
+Set `PDFA_UPLOAD_FOLDER` to an empty string to upload PDF/A files into the
+same folder as regular uploads.
+
+### RFC 3161 Timestamping
+
+When `PDFA_TIMESTAMP_ENABLED=true`, each PDF/A file is timestamped using
+the configured TSA (default: [FreeTSA](https://freetsa.org)).  This creates
+a `.tsr` file alongside each PDF/A file, providing cryptographic proof that
+the document existed at a specific point in time.
+
+Requires `openssl` on the PATH (included in Docker images).
+
+**Other TSA options:**
+- **GlobalSign** – enterprise, eIDAS qualified
+- **DigiStamp** – high assurance, legal
+- **IdenTrust** – legal, free with certificate purchase
+
+### Configuration Example
+
+```bash
+# Enable PDF/A archival copies
+ENABLE_PDFA_CONVERSION=true
+
+# Use PDF/A-2b format (default, recommended for most use cases)
+PDFA_FORMAT=2
+
+# Upload both original and processed PDF/A to providers
+PDFA_UPLOAD_ORIGINAL=true
+PDFA_UPLOAD_PROCESSED=true
+
+# PDF/A files go into a 'pdfa' subfolder on each provider
+PDFA_UPLOAD_FOLDER=pdfa
+
+# Enable RFC 3161 timestamping via FreeTSA
+PDFA_TIMESTAMP_ENABLED=true
+PDFA_TIMESTAMP_URL=https://freetsa.org/tsr
+```
+
 ## Performance & Caching
 
 DocuElevate automatically optimizes database access and uses Redis as a
@@ -881,6 +1333,10 @@ AUTHENTIK_CLIENT_SECRET=...
 AUTHENTIK_CONFIG_URL=https://auth.example.com/.well-known/openid-configuration
 OAUTH_PROVIDER_NAME=Authentik SSO
 
+# Multi-user mode (requires AUTH_ENABLED=true)
+MULTI_USER_ENABLED=false
+DEFAULT_DAILY_UPLOAD_LIMIT=0
+
 # Storage services
 PAPERLESS_NGX_API_TOKEN=...
 PAPERLESS_HOST=https://paperless.example.com
@@ -929,14 +1385,22 @@ SFTP_FOLDER=/Documents/Uploads
 # SFTP_PRIVATE_KEY=/path/to/key.pem
 # SFTP_PRIVATE_KEY_PASSPHRASE=passphrase
 
-# Email
+# Email (shared SMTP – password reset & verification)
 EMAIL_HOST=smtp.example.com
 EMAIL_PORT=587
 EMAIL_USERNAME=docuelevate@example.com
 EMAIL_PASSWORD=password
 EMAIL_USE_TLS=True
 EMAIL_SENDER=DocuElevate System <docuelevate@example.com>
-EMAIL_DEFAULT_RECIPIENT=recipient@example.com
+
+# Email Destination (document delivery – separate from shared email above)
+DEST_EMAIL_HOST=smtp.example.com
+DEST_EMAIL_PORT=587
+DEST_EMAIL_USERNAME=docuelevate@example.com
+DEST_EMAIL_PASSWORD=password
+DEST_EMAIL_USE_TLS=True
+DEST_EMAIL_SENDER=DocuElevate Delivery <docuelevate@example.com>
+DEST_EMAIL_DEFAULT_RECIPIENT=recipient@example.com
 
 # Notification Settings
 # Configure notification services using Apprise URL format
@@ -965,6 +1429,15 @@ S3_ACL=private
 # Uptime Kuma
 UPTIME_KUMA_URL=https://kuma.example.com/api/push/abcde12345?status=up
 UPTIME_KUMA_PING_INTERVAL=5
+
+# Backup & Restore
+BACKUP_ENABLED=True
+BACKUP_DIR=/data/backups
+BACKUP_REMOTE_DESTINATION=s3         # or dropbox, google_drive, onedrive, nextcloud, webdav, ftp, sftp, email
+BACKUP_REMOTE_FOLDER=backups
+BACKUP_RETAIN_HOURLY=96
+BACKUP_RETAIN_DAILY=21
+BACKUP_RETAIN_WEEKLY=13
 ```
 
 ## Selective Service Configuration

docs/DatabaseConfiguration.md

@@ -4,6 +4,8 @@ DocuElevate uses [SQLAlchemy](https://www.sqlalchemy.org/) as its ORM and [Alemb
 
 ## Table of Contents
 
+- [Database Configuration Wizard](#database-configuration-wizard)
+- [Database Migration Tool](#database-migration-tool)
 - [Supported Databases](#supported-databases)
 - [Configuration](#configuration)
 - [SQLite (Development)](#sqlite-development)
@@ -17,6 +19,74 @@ DocuElevate uses [SQLAlchemy](https://www.sqlalchemy.org/) as its ORM and [Alemb
 
 ---
 
+## Database Configuration Wizard
+
+DocuElevate includes a guided **Database Configuration Wizard** accessible at `/database-wizard`.  The wizard walks you through building a connection string step by step — no need to remember the exact URL format.
+
+### How to Access
+
+Navigate to **`/database-wizard`** in your browser, or find the link under **Admin → Settings**.  On the Settings page, click the **DB Wizard** button in the toolbar, or look for the **Open Database Wizard** link next to the `database_url` setting.
+
+### Wizard Steps
+
+1. **Choose Database Type** — select SQLite, PostgreSQL, or MySQL/MariaDB.
+2. **Connection Details** — enter host, port, database name, credentials, and SSL mode (auto-populated with sensible defaults).
+3. **Test & Apply** — test the connection before committing, then copy the generated `DATABASE_URL` into your `.env` file.
+
+The wizard generates the full SQLAlchemy connection string and lets you test connectivity directly from the UI.  After testing, copy the `DATABASE_URL=…` line into your `.env` file (or Docker Compose environment) and restart DocuElevate.
+
+### REST API
+
+The wizard is backed by a REST API under `/api/database/`:
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/database/backends` | GET | List supported database backends |
+| `/api/database/build-url` | POST | Build a connection string from components |
+| `/api/database/parse-url` | POST | Parse a connection string into components |
+| `/api/database/validate-url` | POST | Validate URL format without connecting |
+| `/api/database/test-connection` | POST | Test connectivity to a database |
+
+> **Note:** All write endpoints require admin authentication.
+
+---
+
+## Database Migration Tool
+
+The **Migrate Data** tab (on the same `/database-wizard` page) lets you copy all your data from one database to another — for example, migrating from the built-in SQLite database to an external PostgreSQL or MySQL instance.
+
+### When to Use
+
+- Moving from a development SQLite database to a production PostgreSQL instance.
+- Migrating to a managed cloud database (AWS RDS, Google Cloud SQL, Azure Database for PostgreSQL, Supabase, etc.).
+- Consolidating data from one database engine to another.
+
+### How It Works
+
+1. Enter (or auto-fill) the **Source Database URL** — this is your current database.
+2. Enter the **Target Database URL** — the new, empty database to copy data into.
+3. Click **Test Source** and **Test Target** to verify both connections.
+4. Click **Preview Migration** to see a table-by-table row count.
+5. Confirm and click **Start Migration** to copy all data.
+
+The migration tool:
+- Creates the full schema in the target database from the application models.
+- Copies all rows table by table in dependency order (parent tables first).
+- Stamps the Alembic migration version to `head` in the target.
+
+After migration, update your `DATABASE_URL` environment variable to point at the new database and restart DocuElevate.
+
+### REST API
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/api/database/preview-migration` | POST | Preview tables and row counts |
+| `/api/database/migrate` | POST | Execute the full data migration |
+
+> **Warning:** Always run the migration against an **empty** target database.  The tool does not delete existing data in the target before copying.
+
+---
+
 ## Supported Databases
 
 | Database | Recommended Use | Notes |
@@ -271,29 +341,91 @@ Disable `prepared_statements` when using PgBouncer in transaction mode.
 
 ## Backup Procedures
 
-### PostgreSQL
+DocuElevate's built-in **Backup & Restore** feature (Admin → Backup & Restore) supports all three
+database backends natively, using the native dump tools of each database.
 
-**Manual backup:**
+| Backend        | Backup tool  | Archive extension | Restore tool |
+|----------------|--------------|-------------------|--------------|
+| SQLite         | `sqlite3` (built-in Python) | `.db.gz`    | `sqlite3` (built-in Python) |
+| PostgreSQL     | `pg_dump`    | `.pgsql.gz`       | `psql`       |
+| MySQL/MariaDB  | `mysqldump`  | `.mysql.gz`       | `mysql`      |
+
+Passwords are passed via the `PGPASSWORD` (PostgreSQL) and `MYSQL_PWD` (MySQL) environment
+variables so they are never exposed on the process command line.
+
+### Prerequisites
+
+For PostgreSQL and MySQL backups the corresponding CLI client must be installed on the
+worker host (the container / server that runs Celery workers):
+
+```bash
+# PostgreSQL clients (Debian/Ubuntu)
+apt-get install -y postgresql-client
+
+# MySQL clients (Debian/Ubuntu)
+apt-get install -y default-mysql-client
+```
+
+The binaries required are:
+
+- **PostgreSQL**: `pg_dump` (backup) and `psql` (restore)
+- **MySQL / MariaDB**: `mysqldump` (backup) and `mysql` (restore)
+
+### Using the Admin Dashboard
+
+Navigate to **Admin → Backup & Restore** to:
+
+- Trigger manual backups (hourly / daily / weekly)
+- Download backup archives
+- Upload and restore a backup archive
+- Configure retention and remote storage destinations
+
+### PostgreSQL – manual backup/restore
+
+**Manual backup using DocuElevate's archive format (for use with the UI restore):**
+
+```bash
+pg_dump --format=plain --no-password \
+  -h localhost -U docuelevate docuelevate \
+  | gzip > docuelevate_$(date +%Y%m%d_%H%M).pgsql.gz
+```
+
+**Restore via the DocuElevate UI:**  upload the `.pgsql.gz` file on the Backup & Restore page.
+
+**Manual restore using native tools (custom format):**
 
 ```bash
 pg_dump -h localhost -U docuelevate -F c docuelevate > docuelevate_$(date +%Y%m%d_%H%M).dump
-```
-
-**Restore:**
-
-```bash
 pg_restore -h localhost -U docuelevate -d docuelevate docuelevate_20240101_1200.dump
 ```
 
 **Automated daily backup (cron example):**
 
 ```cron
-0 2 * * * pg_dump -h localhost -U docuelevate -F c docuelevate | gzip > /backups/docuelevate_$(date +\%Y\%m\%d).dump.gz
+0 2 * * * pg_dump --format=plain -h localhost -U docuelevate docuelevate | gzip > /backups/docuelevate_$(date +\%Y\%m\%d).pgsql.gz
 ```
 
 Use your cloud provider's automated backup feature when available (e.g., RDS automated snapshots, Cloud SQL backups).
 
-### SQLite
+### MySQL / MariaDB – manual backup/restore
+
+**Manual backup using DocuElevate's archive format (for use with the UI restore):**
+
+```bash
+MYSQL_PWD=yourpassword mysqldump --single-transaction --routines --triggers \
+  -h localhost -u docuelevate docuelevate \
+  | gzip > docuelevate_$(date +%Y%m%d_%H%M).mysql.gz
+```
+
+**Restore via the DocuElevate UI:**  upload the `.mysql.gz` file on the Backup & Restore page.
+
+**Manual restore using native tools:**
+
+```bash
+gunzip -c docuelevate_20240101_1200.mysql.gz | mysql -h localhost -u docuelevate -p docuelevate
+```
+
+### SQLite – manual backup/restore
 
 ```bash
 # Stop the application first, or use SQLite's online backup API

docs/HowToGuides.md

@@ -0,0 +1,80 @@
+# How-To Guides
+
+Welcome to the DocuElevate How-To Guides. These step-by-step articles help you connect common scanners, printers, and mobile devices to DocuElevate for automatic document ingestion and processing.
+
+---
+
+## Document Ingestion Guides
+
+Choose the method that matches your hardware or workflow:
+
+### 🖨️ Scanners & Printers
+
+| Guide | Description |
+|-------|-------------|
+| [HP Enterprise Printer Setup](howto/HPPrinterSetup.md) | Configure HP MFPs for automatic scan-to-DocuElevate via email, network folder, or WebDAV |
+| [Fujitsu ScanSnap Setup](howto/SnapScanSetup.md) | Set up ScanSnap iX1600/iX1500/iX500 to send scans directly to DocuElevate |
+| [Watched Folder Setup](howto/WatchedFolderSetup.md) | Monitor a local or network folder and automatically ingest any files dropped into it |
+
+### 📧 Email & Cloud
+
+| Guide | Description |
+|-------|-------------|
+| [Email Ingestion](howto/EmailIngestion.md) | Route documents sent as email attachments into DocuElevate automatically |
+| [Mobile Scanning](howto/MobileScanning.md) | Use iOS/Android apps to capture and upload documents from your phone |
+
+---
+
+## Quick Comparison: Which Method Is Right for You?
+
+| Method | Best For | Setup Complexity |
+|--------|----------|-----------------|
+| **Web Upload** | Occasional one-off uploads | ⭐ Very Easy |
+| **Watched Folder** | Any scanner that saves to a folder | ⭐⭐ Easy |
+| **Email Ingestion** | Scanners with Scan-to-Email, fax services | ⭐⭐ Easy |
+| **HP Printer (SMB)** | HP Enterprise MFPs on a corporate network | ⭐⭐⭐ Medium |
+| **ScanSnap + Cloud** | Home/office ScanSnap via Dropbox/Drive | ⭐⭐ Easy |
+| **Mobile App** | On-the-go document capture | ⭐ Very Easy |
+| **API Integration** | Custom workflows, developer integrations | ⭐⭐⭐⭐ Advanced |
+
+---
+
+## General Configuration Tips
+
+### Enable OCR for Searchable PDFs
+
+DocuElevate applies OCR (Optical Character Recognition) to scanned documents, making them fully searchable. Configure the OCR engine in your `.env`:
+
+```env
+OCR_ENABLED=true
+OCR_ENGINE=tesseract          # or: azure, google
+OCR_LANGUAGE=eng              # ISO 639-2 language code
+```
+
+### Set Up Automatic Cloud Backup
+
+After processing, DocuElevate can store documents in your preferred cloud storage. Configure in Settings or `.env`:
+
+```env
+DEFAULT_STORAGE_TARGET=dropbox   # or: gdrive, onedrive, s3, nextcloud
+```
+
+### Configure Notifications
+
+Get notified when documents are processed:
+
+```env
+NOTIFICATION_ENABLED=true
+NOTIFICATION_EMAIL=you@example.com
+```
+
+See [Notifications Setup](NotificationsSetup.md) for webhook, Slack, and other integrations.
+
+---
+
+## Need Help?
+
+- Browse the full [Documentation](UserGuide.md)
+- Check [Troubleshooting](Troubleshooting.md) for common issues
+- View [Configuration Reference](ConfigurationGuide.md) for all settings
+- Explore the [API Reference](API.md) for programmatic access

docs/ReleaseNaming.md

@@ -0,0 +1,127 @@
+# Release Naming Guide
+
+DocuElevate uses **automated semantic versioning** via [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) combined with **named release anchors** (codenames) for milestone releases. This guide explains how the two systems work together.
+
+## How It Works
+
+### Automated Versioning (Patch & Minor Releases)
+
+Every merge to `main` is analyzed by `python-semantic-release`:
+
+- **`feat:` commits** → minor version bump (e.g., 0.5.0 → 0.6.0)
+- **`fix:` / `perf:` commits** → patch version bump (e.g., 0.5.0 → 0.5.1)
+- **`docs:` / `chore:` / etc.** → no version bump
+
+This happens automatically — no manual intervention needed.
+
+### Named Release Anchors (Codenames)
+
+Major milestone releases carry a **codename** that anchors the release in project history. Codenames:
+
+- Are defined in [`release_names.json`](../release_names.json) at the project root
+- Map to **minor version ranges** (e.g., all `0.5.x` releases share the codename "Foundation")
+- Appear in the status dashboard, build metadata, and footer
+- Do **not** interfere with automatic version numbering
+
+### Current Release Names
+
+| Version Range | Codename       | Description                                              |
+|---------------|----------------|----------------------------------------------------------|
+| 0.5.x         | **Foundation** | Core platform with multi-provider storage, AI, and UI    |
+| 0.6.x         | **Clarity**    | Enhanced search, filtering, and improved UI/UX           |
+| 0.7.x         | **Conductor**  | Workflow automation, custom pipelines, rule-based logic   |
+| 1.0.x         | **Summit**     | Enterprise-ready: multi-tenancy, RBAC, horizontal scaling|
+| 1.1.x         | **Bridge**     | Collaboration features, document sharing, analytics      |
+| 2.0.x         | **Horizon**    | On-premise AI, advanced management, platform expansion   |
+
+## Adding a New Release Name
+
+1. **Edit `release_names.json`** at the project root:
+
+```json
+{
+  "releases": {
+    "0.8": {
+      "codename": "YourCodename",
+      "description": "Short description of what this release series focuses on",
+      "milestone": "v0.8.0 - Your Milestone Name"
+    }
+  }
+}
+```
+
+2. **Update `ROADMAP.md`** to include the codename in the appropriate milestone section.
+
+3. **Update this documentation** to add the new entry to the table above.
+
+The codename will automatically appear in:
+- The application footer (all pages)
+- The status dashboard (`/status`)
+- Build metadata (`RUNTIME_INFO` file)
+
+## How the Lookup Works
+
+The application resolves codenames using a cascading lookup against the current version:
+
+1. **Exact match**: Checks if the full version (e.g., `0.5.3`) has an entry
+2. **Minor prefix**: Checks the minor version prefix (e.g., `0.5`)
+3. **Major prefix**: Checks the major version prefix (e.g., `0`)
+
+This means all patch releases within a minor version series inherit the same codename.
+
+## Codename Naming Conventions
+
+When choosing codenames, follow these guidelines:
+
+- **Use single, evocative words** that relate to the release's theme
+- **Keep names professional** — they appear in user-facing UI
+- **Pick names that hint at the milestone's focus** (e.g., "Foundation" for core platform, "Conductor" for workflow automation)
+- **Avoid names that could become dated** or reference external products
+- **Ensure uniqueness** — no two releases should share a codename
+
+## Integration with Milestones
+
+Each codename maps to a GitHub milestone. The `milestone` field in `release_names.json` matches the milestone title used for issue tracking:
+
+```
+v0.7.0 - Workflow Automation  →  codename: "Conductor"
+v1.0.0 - Enterprise           →  codename: "Summit"
+```
+
+This creates a clear link between planning (milestones), delivery (releases), and communication (codenames).
+
+## Best Practices: Blending Automated and Named Releases
+
+### Do
+
+- ✅ Let semantic-release handle all version numbering automatically
+- ✅ Use codenames for **milestone releases** (minor/major versions), not every patch
+- ✅ Reference codenames in release notes and changelogs for major versions
+- ✅ Keep `release_names.json` in sync with `ROADMAP.md`
+- ✅ Announce codenames in GitHub Release descriptions for milestone versions
+
+### Don't
+
+- ❌ Manually edit the `VERSION` file — it's managed by semantic-release
+- ❌ Create codenames for every patch release (0.5.1, 0.5.2, etc.)
+- ❌ Use codenames that conflict with version numbers
+- ❌ Skip updating `release_names.json` when adding a new milestone to the roadmap
+
+## Where Codenames Appear
+
+| Location                | Format                                  |
+|-------------------------|-----------------------------------------|
+| Status dashboard        | `App Version: 0.5.3 "Foundation"`       |
+| Page footer             | `Version 0.5.3 "Foundation"`            |
+| RUNTIME_INFO metadata   | `Release Name: Foundation`              |
+| ROADMAP.md              | Section headers include codenames       |
+
+## File Reference
+
+| File                      | Purpose                                    |
+|---------------------------|--------------------------------------------|
+| `release_names.json`      | Source of truth for version-to-codename map|
+| `app/config.py`           | `release_name` property reads the JSON     |
+| `scripts/generate_build_metadata.sh` | Includes codename in RUNTIME_INFO |
+| `app/views/base.py`       | Injects `release_name` into all templates  |
+| `ROADMAP.md`              | Displays codenames alongside milestones    |

docs/SettingsManagement.md

@@ -24,23 +24,33 @@ Each setting in the UI shows a badge indicating its current source:
 
 Settings are organized into logical categories for easy navigation:
 
-- **Core**: Database, Redis, working directory, external hostname, debug mode
-- **Authentication**: Login settings, session secrets, OAuth configuration
-- **AI Services**: AI provider selection and credentials (OpenAI, Azure, Anthropic, Gemini, Ollama, OpenRouter, Portkey, LiteLLM)
+- **Core**: Database, Redis, working directory, external hostname, debug mode, Meilisearch, file size limits
+- **Authentication**: Login settings, session secrets, OAuth configuration, admin group
+- **AI Services**: AI provider selection, model configuration, embeddings, and credentials (OpenAI, Azure, Anthropic, Gemini, Ollama, OpenRouter, Portkey, LiteLLM)
+- **OCR Engines**: OCR provider selection and configuration (Tesseract, EasyOCR, Mistral, Google DocAI, AWS Textract)
 - **Storage Providers**: Dropbox, Google Drive, OneDrive, S3, FTP, SFTP, WebDAV, Nextcloud, Paperless
 - **Email**: SMTP configuration for sending emails
-- **IMAP**: Email ingestion configuration (supports multiple accounts)
+- **IMAP**: Email ingestion configuration (supports two mailbox accounts)
 - **Monitoring**: Uptime Kuma integration
 - **Notifications**: Apprise notification URLs and settings
-- **Processing**: Batch processing and HTTP timeout settings
-- **Feature Flags**: Enable/disable specific features
+- **Processing**: Batch processing, upload throttling, deduplication, text quality, task retry, timeouts
+- **Feature Flags**: Enable/disable specific features (file deletion, search, webhooks)
+- **PDF/A Archival**: PDF/A conversion, upload, and RFC 3161 timestamping settings
+- **Security**: Security headers, audit logging, rate limiting, CORS configuration
+- **UI**: Color scheme preferences
 
 ### Setting Types
 
 - **String**: Text values (API keys, URLs, paths)
 - **Boolean**: True/false toggles (enable/disable features)
 - **Integer**: Numeric values (ports, timeouts, thresholds)
-- **List**: Comma-separated values (notification URLs)
+- **Slider**: Numeric values with visual range controls (e.g., similarity threshold 0–1, quality threshold 0–100)
+- **Dropdown**: Predefined option lists (e.g., PDF/A format, S3 storage class, S3 ACL)
+- **Multi-select**: Comma-separated selections from a list (e.g., OCR providers)
+- **Model Picker**: Free-text with suggested model names (e.g., AI model selection)
+- **User Autocomplete**: Typeahead search for existing user identifiers (e.g., default owner assignment), fetches from `GET /api/users/search`
+- **Autocomplete**: Typeahead search with dynamic suggestions fetched from `GET /api/settings/{key}/suggestions`. Used for AWS/Azure regions, OCR language codes, and embedding models. Providers attempt dynamic resolution (e.g., boto3 for AWS regions, `tesseract --list-langs` for Tesseract) and fall back to curated static lists if the SDK or tool is unavailable.
+- **List**: Comma-separated values (notification URLs, CORS origins)
 
 ### Sensitive Data
 

docs/SubscriptionTiers.md

@@ -0,0 +1,157 @@
+# Subscription Tiers
+
+DocuElevate uses database-backed subscription plans that are fully configurable by admins via the **Plan Designer** at `/admin/plans`. Four default tiers are seeded automatically on first startup.
+
+All plans are priced **per user, per month** (or per year with ~20 % discount). There are no team, business, or enterprise tiers — every plan is a single-user subscription.
+
+## Default Plans
+
+| Plan | Monthly | Yearly | Docs/Month | Lifetime Docs | OCR Pages/Mo | Max File | Mailboxes | Destinations |
+|------|---------|--------|-----------|---------------|--------------|----------|-----------|--------------|
+| **Free** | $0 | $0 | — | 50 total | 150 total | 5 MB | 0 | 1 |
+| **Starter** | $2.99 | $28.99 | 50 | — | 300 | 25 MB | 1 | 2 |
+| **Professional** | $5.99 | $57.99 | 150 | — | 750 | 100 MB | 3 | 5 |
+| **Power** | $7.99 | $76.99 | 300 | — | 1,500 | Unlimited | Unlimited | 10 |
+
+> Prices ex-VAT. German customers add 19% MwSt.
+
+All paid plans include a **30-day free trial**.
+
+### Intended Use Cases
+
+- **Free** — Try DocuElevate with no commitment. Good for one-off experiments or evaluating the service.
+- **Starter** — Freelancers and side-project owners sending ~50 invoices, contracts, or scanned receipts a month.
+- **Professional** — Knowledge workers (consultants, paralegals, accountants) handling ~150 multi-page documents a month across several cloud destinations.
+- **Power** — Power users with heavy daily workloads: real estate agents, bookkeepers, or researchers processing ~10 documents a day (≈ 300/month) with no file-size restrictions.
+
+> The **plan_id** in the database remains `"business"` for the Power tier to preserve backwards compatibility. The display name shown to users is "Power".
+
+## How Plans Are Stored
+
+Plans are stored in the `subscription_plans` database table. On application startup, `seed_default_plans()` is called automatically — if the table is empty, the four built-in defaults are inserted. If plans already exist, the seed is a no-op.
+
+Users are assigned a plan via `UserProfile.subscription_tier` (stores the `plan_id` string). The subscription utility functions (`get_tier`, `get_all_tiers`) query the database first and fall back to the hard-coded `TIER_DEFAULTS` dict if the database is unavailable or the plan doesn't exist.
+
+## Overage Buffer
+
+### Announced vs. Enforced Limit
+
+DocuElevate uses a **soft-limit overage buffer** that is invisible to users:
+
+- The **announced limit** is what appears on the pricing page (e.g., "150 docs/month").
+- The **enforced limit** = announced × (1 + overage_percent / 100).
+  - With the default 20% buffer: a 150-doc plan enforces at **180 docs**.
+  - This prevents hard cutoffs at the exact announced limit, giving users a graceful landing.
+
+### Per-Plan vs. Global Buffer
+
+Each plan has its own `overage_percent` field (set in the Plan Designer). There is also a global fallback: `settings.subscription_overage_percent` (default: 20, range: 0–200), which applies when a plan does not have an explicit value.
+
+Set `subscription_overage_percent=0` in your `.env` to enforce exactly at the announced limit with no buffer.
+
+## Yearly Billing & Carry-Over
+
+When a user's `subscription_billing_cycle` is set to `yearly`:
+
+- Unused quota from earlier months **rolls forward automatically**.
+- Enforcement = `monthly_limit × months_elapsed × overage_factor` (cumulative budget from the subscription start date).
+- Example: A 50-doc/month Starter plan in month 3 of its annual period has a cumulative budget of 150 docs (plus overage buffer). If the user only used 20 docs in months 1–2, they can use 130 docs in month 3.
+- The `subscription_period_start` field on `UserProfile` tracks the start of the annual period.
+
+## No Daily Cap
+
+`daily_upload_limit` is kept for display and future reference only — it is **never enforced**. All enforcement is lifetime (free tier) or monthly/cumulative-yearly (paid tiers).
+
+## allow_overage Flag
+
+Setting `UserProfile.allow_overage = True` bypasses monthly quota checks entirely for that user. Usage is still tracked so future billing integrations can charge retroactively. This field is not yet exposed in the admin UI.
+
+## is_complimentary Flag (Complimentary Plans)
+
+Setting `UserProfile.is_complimentary = True` marks a user as being on a **complimentary (uncharged) plan**. The user retains all quota benefits of their assigned subscription tier but is **never billed via Stripe**. This is useful for:
+
+- **Admin accounts** — automatically set on every admin user profile at login time.
+- **Gifted access** — granting full plan benefits to partners, testers, or sponsored users.
+
+### Admin Auto-Provisioning
+
+When an admin user logs in for the first time (via OAuth, local account, or the built-in admin credentials), DocuElevate automatically:
+
+1. Creates a `UserProfile` row if one does not already exist.
+2. Assigns the **highest available subscription tier** (currently `business`).
+3. Sets `is_complimentary = True` so the account is never billed.
+4. Sets `onboarding_completed = True` so admins skip the first-time setup wizard.
+
+On subsequent logins for existing admin profiles:
+- `is_complimentary` is ensured to be `True`.
+- If the profile was still on the `free` tier it is upgraded to the highest tier.
+- All other admin-managed settings (custom limits, notes, etc.) are preserved.
+
+### Managing via Admin UI
+
+The **User Management** page (`/admin/users`) shows a green gift icon (🎁) next to the plan badge for any user with `is_complimentary = True`. The toggle is available in the user edit modal under **Billing**.
+
+### API Field
+
+`is_complimentary` is exposed in the `PUT /api/admin/users/{user_id}` body and in all user detail responses.
+
+## Plan Designer
+
+Navigate to `/admin/plans` (admin only) to:
+
+1. **View** all plans (active and inactive) with key stats.
+2. **Create** a new plan with a custom `plan_id` slug.
+3. **Edit** any plan's pricing, limits, overage buffer, features, and display settings.
+4. **Reorder** plans using the up/down arrows (order reflects pricing page display order).
+5. **Delete** a plan (does not affect existing users assigned to it).
+6. **Restore Defaults** — seeds the four built-in plans (no-op if plans already exist).
+
+### Overage Designer
+
+The Plan Designer includes an overage slider (0–100%). The live preview shows:
+
+> "Announce **X** docs, enforce at **Y** docs"
+
+Overage billing and per-doc overage pricing are planned future features (currently disabled in the UI).
+
+## API Endpoints
+
+All plan endpoints are under `/api/plans/`.
+
+| Method | Path | Auth | Description |
+|--------|------|------|-------------|
+| `GET` | `/api/plans/` | Public | List active plans in sort order |
+| `GET` | `/api/plans/admin` | Admin | List all plans including inactive |
+| `POST` | `/api/plans/?plan_id=<id>` | Admin | Create a new plan |
+| `GET` | `/api/plans/{plan_id}` | Public | Get a single active plan |
+| `PUT` | `/api/plans/{plan_id}` | Admin | Update an existing plan |
+| `DELETE` | `/api/plans/{plan_id}` | Admin | Delete a plan |
+| `POST` | `/api/plans/seed` | Admin | Seed default plans (no-op if non-empty) |
+| `POST` | `/api/plans/reorder` | Admin | Update sort order; body: `{"order": ["free", "starter", ...]}` |
+
+### Example: List Active Plans
+
+```bash
+curl http://localhost:8000/api/plans/
+```
+
+### Example: Update a Plan's Monthly Limit
+
+```bash
+curl -X PUT http://localhost:8000/api/plans/starter \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "Starter",
+    "monthly_upload_limit": 75,
+    "overage_percent": 15,
+    ...
+  }'
+```
+
+## Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `SUBSCRIPTION_OVERAGE_PERCENT` | `20` | Global overage buffer (0–200). Per-plan setting overrides this. |
+
+See `docs/ConfigurationGuide.md` for all available settings.

docs/UserGuide.md

@@ -28,8 +28,29 @@ If OpenID Connect authentication is configured:
 3. Log in with your existing credentials on that platform
 4. You'll be redirected back to DocuElevate after successful authentication
 
-#### User Sessions
-- Once authenticated, your session will remain active until you log out or it expires
+#### Local User Accounts
+If your administrator has created a local (email/password) account for you:
+
+1. You'll see a "Sign in with username" form on the login page
+2. Enter your **username or email address** — both are accepted
+3. Enter your password and click **Sign in**
+
+##### Forgot your password?
+If you can't remember your password:
+1. Click **Forgot password?** below the sign-in form
+2. Enter your email address and click **Send reset link**
+3. Check your inbox for a password reset email (valid for 24 hours)
+4. Click the link in the email and enter your new password
+
+##### Forgot your username?
+If you can't remember your username:
+1. Click **Forgot username?** below the sign-in form
+2. Enter your email address and click **Send username reminder**
+3. You'll receive an email with your username
+
+> **Tip:** You can always sign in with your email address directly — you don't need to look up your username.
+
+
 - Click the "Logout" button in the top navigation bar to end your session
 - For security, sessions automatically expire after a period of inactivity
 
@@ -74,14 +95,37 @@ For even more convenience, you can upload files directly from the **Files** page
 
 This feature allows you to quickly add new files without navigating away from your document management view.
 
-### Email Attachments
+### Email Attachments (IMAP Ingestion)
 
-If configured, DocuElevate can automatically fetch documents from email attachments:
+DocuElevate acts as an email *client* that automatically retrieves document attachments from one or more IMAP mailboxes. You do not need to set up DocuElevate as an email server — it simply polls an existing mailbox that you designate for document delivery.
 
-1. Send an email with attachments to the configured email account
-2. DocuElevate will poll the mailbox at the configured interval
-3. Attachments will be automatically downloaded and processed
-4. No further action is required
+**How it works:**
+1. A document is sent as an email attachment to the configured mailbox (e.g. from a scanner, a colleague, or any email client)
+2. DocuElevate polls the mailbox at the configured interval (typically every 1–5 minutes)
+3. Email attachments in supported formats are automatically downloaded and enqueued for processing
+4. Processed emails are marked with a label or star (Gmail) or tracked locally, so they are not re-processed
+
+> **HP Scanners and MFPs (Scan to Email)**: Configure your scanner's "Scan to Email" feature to send scanned documents to a dedicated email account. Point DocuElevate at that mailbox using the IMAP settings. DocuElevate will retrieve the scanned PDFs automatically — no manual forwarding required.
+
+### Watch Folders (Automatic Folder Ingestion)
+
+Watch folders allow DocuElevate to automatically monitor directories for new files and ingest them without any manual action.
+
+#### Local Watch Folders (including SMB/CIFS and NFS)
+
+Mount a network share or local directory into the DocuElevate worker container and configure the path in `WATCH_FOLDERS`. DocuElevate scans the folder every minute (configurable via `WATCH_FOLDER_POLL_INTERVAL`) and enqueues any new documents it finds.
+
+This is the recommended approach for:
+- **HP Scanners / MFPs** using "Scan to Network Folder" — point the scanner at a shared folder that DocuElevate also has access to
+- **SMB/CIFS shares** — mount the Windows/Samba share and add the path to `WATCH_FOLDERS`
+- **NFS mounts** — works identically, just configure the mount path
+- **Any local directory** on the server running DocuElevate
+
+#### FTP / SFTP Watch Folders
+
+DocuElevate can poll an FTP or SFTP directory for new files. Enable this with `FTP_INGEST_ENABLED` or `SFTP_INGEST_ENABLED` and set the corresponding ingest folder. DocuElevate downloads new files, enqueues them for processing, and optionally deletes them from the remote server.
+
+See [Configuration Guide — Watch Folder Ingestion](ConfigurationGuide.md#watch-folder-ingestion) for full setup instructions.
 
 ## Managing Documents
 
@@ -334,6 +378,55 @@ PAPERLESS_CUSTOM_FIELDS_MAPPING='{"absender": "Sender", "empfaenger": "Recipient
 3. After successful upload, custom fields are automatically populated
 4. You can view the populated fields in your Paperless-ngx document details
 
+## Processing Pipelines
+
+Processing pipelines let you define exactly what happens to your documents when they are uploaded. Each pipeline is an ordered sequence of **steps** — for example: convert to PDF → OCR → extract metadata → send to storage.
+
+### Key concepts
+
+| Term | Meaning |
+|------|---------|
+| **Pipeline** | A named, ordered list of processing steps |
+| **Step** | A single processing action (e.g., OCR, metadata extraction) |
+| **System pipeline** | Created by an admin; visible to all users as a shared default |
+| **User pipeline** | Created by a regular user; private to that user |
+| **Default pipeline** | Marked `is_default=true`; used automatically for new uploads |
+
+### Managing your pipelines
+
+1. Navigate to **Pipelines** in the top navigation bar.
+2. Click **New Pipeline** to create a pipeline, give it a name and optional description.
+3. Expand the pipeline card and click **Add Step** to build the workflow.
+4. Use the ↑ / ↓ arrows to reorder steps, or click the edit icon to change step settings.
+5. Mark a pipeline as **Default** so new documents are automatically processed by it.
+
+### Available step types
+
+| Step Type | Description |
+|-----------|-------------|
+| `convert_to_pdf` | Convert non-PDF files to PDF using Gotenberg |
+| `check_duplicates` | Detect duplicate files by content hash |
+| `ocr` | Extract text with Azure Document Intelligence or local Tesseract |
+| `extract_metadata` | Extract structured metadata (type, sender, tags) with AI |
+| `embed_metadata` | Write extracted metadata into the PDF document properties |
+| `compute_embedding` | Compute semantic embeddings for similarity search |
+| `send_to_destinations` | Upload the processed document to all configured storage destinations |
+| `classify` | Classify the document type with AI |
+
+### Assigning a pipeline to a file
+
+You can assign (or change) the pipeline for an individual document via the file detail page or the API:
+
+```bash
+POST /api/files/{file_id}/assign-pipeline?pipeline_id=3
+```
+
+Pass no `pipeline_id` to clear the assignment and fall back to the system default.
+
+### Admin: system-wide pipelines
+
+Admins can create **system pipelines** that appear in every user's pipeline list. These can be set as the global default so all users benefit from a consistent processing baseline. Navigate to **Pipelines** and check the **System pipeline** box when creating a new one (admin only).
+
 ## API Access
 
 For programmatic access, DocuElevate provides a comprehensive REST API:

docs/howto/EmailIngestion.md

@@ -0,0 +1,156 @@
+# How to Set Up Automatic Document Ingestion via Email
+
+Many devices (scanners, printers, fax services, and apps) can send documents as email attachments. This guide explains how to automatically route those attachments into DocuElevate.
+
+---
+
+## Overview
+
+The Email Ingestion workflow works like this:
+
+```
+Scanner/Device → Email (SMTP) → Monitored Mailbox → DocuElevate API → Processing & Storage
+```
+
+DocuElevate periodically checks a designated email inbox, downloads PDF/image attachments, and processes them through the standard document pipeline.
+
+---
+
+## Prerequisites
+
+- An email account dedicated to document ingestion (e.g., `scan@yourdomain.com`)
+- IMAP access enabled for that account
+- DocuElevate running with the Celery worker active
+
+---
+
+## Configuration
+
+Add the following to your DocuElevate `.env` file:
+
+```env
+# Email ingestion settings
+EMAIL_INGESTION_ENABLED=true
+EMAIL_INGESTION_IMAP_HOST=mail.yourdomain.com
+EMAIL_INGESTION_IMAP_PORT=993
+EMAIL_INGESTION_IMAP_SSL=true
+EMAIL_INGESTION_USERNAME=scan@yourdomain.com
+# Use an app-specific password (Gmail, Outlook) – NOT your main account password.
+# See the Security Considerations section below for details.
+EMAIL_INGESTION_PASSWORD=your-app-specific-password
+EMAIL_INGESTION_FOLDER=INBOX
+EMAIL_INGESTION_INTERVAL=60           # Check every 60 seconds
+EMAIL_INGESTION_MARK_SEEN=true        # Mark emails as read after processing
+EMAIL_INGESTION_ALLOWED_SENDERS=      # Comma-separated allowlist (empty = allow all)
+```
+
+Restart DocuElevate after saving the configuration:
+
+```bash
+docker compose restart api worker
+```
+
+---
+
+## Supported File Types
+
+DocuElevate will process the following attachment types from emails:
+
+| Type | Extension | Notes |
+|------|-----------|-------|
+| PDF | `.pdf` | Native support; OCR applied if not searchable |
+| JPEG/PNG | `.jpg`, `.jpeg`, `.png` | Converted to PDF before processing |
+| TIFF | `.tif`, `.tiff` | Common format from older scanners/fax |
+| Multi-page TIFF | `.tif` | Full multi-page support |
+
+---
+
+## Setting Up Your Scanner/Device
+
+### HP Printers – Scan to Email
+
+See the detailed guide: [HP Enterprise Printer Setup](./HPPrinterSetup.md#option-a-scan-to-email--docuelevate-api-upload)
+
+### Fujitsu ScanSnap – Send by Email
+
+See the detailed guide: [ScanSnap Setup](./SnapScanSetup.md#method-3-scan-to-email--docuelevate)
+
+### iOS/Android Scanning Apps
+
+Most mobile scanning apps (Adobe Scan, Microsoft Lens, SwiftScan) can email scans:
+
+1. Scan your document.
+2. Use the app's **Share** or **Send** function.
+3. Select **Email** and enter `scan@yourdomain.com`.
+4. DocuElevate will pick up the attachment within the configured interval.
+
+### Fax-to-Email Services
+
+Services like eFax, RingCentral Fax, or Twilio Fax can forward incoming faxes as email attachments. Configure them to send to `scan@yourdomain.com`.
+
+---
+
+## Security Considerations
+
+> **Important:** Only process emails from trusted sources to avoid ingesting malicious documents.
+
+Use the `EMAIL_INGESTION_ALLOWED_SENDERS` setting to restrict which email addresses can submit documents:
+
+```env
+EMAIL_INGESTION_ALLOWED_SENDERS=scanner@office.com,printer@office.com,fax@office.com
+```
+
+DocuElevate will silently skip emails from addresses not in the allowlist.
+
+Additionally:
+
+- Use a **dedicated email account** solely for document ingestion
+- Enable **app-specific passwords** (Gmail, Outlook) instead of your main account password
+- Store credentials in environment variables, never in config files committed to version control
+
+---
+
+## Monitoring
+
+Check the DocuElevate worker logs to verify email ingestion is running:
+
+```bash
+docker logs document_worker --follow
+```
+
+You should see log entries like:
+
+```
+INFO  Email ingestion: checking inbox scan@yourdomain.com
+INFO  Email ingestion: found 3 new messages
+INFO  Email ingestion: processing attachment invoice-2024.pdf from printer@office.com
+INFO  Email ingestion: queued document ID 142 for processing
+```
+
+---
+
+## Troubleshooting
+
+**No emails are being processed?**
+→ Verify IMAP credentials and that IMAP is enabled on your mail server.  
+→ Check firewall rules: port 993 (SSL) or 143 (plain) must be open from DocuElevate to the mail server.
+
+**Gmail not working?**
+→ Enable "App Passwords" in your Google Account security settings.  
+→ Use the App Password (not your main Google password) for `EMAIL_INGESTION_PASSWORD`.
+
+**Attachments processed but files are empty?**
+→ Some email clients send inline images instead of attachments. Check the raw email source.
+
+**Emails keep getting re-processed?**
+→ Set `EMAIL_INGESTION_MARK_SEEN=true` to mark emails as read after processing.  
+→ Alternatively, configure a separate ingestion folder and move/delete emails after processing.
+
+---
+
+## Related Documentation
+
+- [HP Enterprise Printer Setup](./HPPrinterSetup.md)
+- [ScanSnap Setup](./SnapScanSetup.md)
+- [Configuration Guide](../ConfigurationGuide.md)
+- [Notifications Setup](../NotificationsSetup.md)

docs/howto/HPPrinterSetup.md

@@ -0,0 +1,217 @@
+# How to Set Up Automatic Document Ingestion with an HP Enterprise Printer
+
+This guide explains how to configure an HP Enterprise printer/MFP (Multi-Function Printer) to automatically send scanned documents to DocuElevate for processing and storage.
+
+---
+
+## Prerequisites
+
+- An HP Enterprise printer or MFP with **Scan to Email** or **Scan to Network Folder** capability
+- DocuElevate running and accessible on your network
+- Admin access to the HP printer's Embedded Web Server (EWS)
+- (Optional) An SMTP server or a configured email address for Scan to Email
+
+---
+
+## Option A: Scan to Email → DocuElevate API Upload
+
+HP Enterprise printers can send scanned documents as email attachments. You can set up a dedicated inbox that forwards documents to DocuElevate via the REST API.
+
+### Step 1: Configure Scan to Email on the Printer
+
+1. Open a browser and navigate to the printer's IP address (e.g., `http://192.168.1.100`) to access the **Embedded Web Server (EWS)**.
+2. Go to **Scan** → **Scan to E-mail**.
+3. Enable **Scan to E-mail** and configure your SMTP server settings.
+4. Create a **Quick Set** (shortcut) for the destination:
+   - **From:** `scanner@yourdomain.com`
+   - **To:** `docuelevate-inbox@yourdomain.com` (the receiving address you'll configure)
+   - **File Type:** PDF
+   - **Resolution:** 200–300 DPI (recommended)
+   - **Color Mode:** Grayscale or Black & White for text documents
+
+### Step 2: Set Up an Email-to-DocuElevate Bridge
+
+Use a lightweight tool like [imapfilter](https://github.com/lefcha/imapfilter) or a simple Python script (see below) to poll the inbox and upload attachments to DocuElevate via its REST API.
+
+**Example Python script (`email_to_docuelevate.py`):**
+
+```python
+import imaplib
+import email
+import os
+import sys
+
+import requests
+
+IMAP_HOST = "mail.yourdomain.com"
+IMAP_USER = "docuelevate-inbox@yourdomain.com"
+# Use an app-specific password (Gmail/Outlook), NOT your main account password.
+# Store credentials as environment variables – never hardcode them.
+IMAP_PASS = os.environ.get("IMAP_PASS")
+DOCUELEVATE_URL = "http://your-docuelevate-host:8000"
+API_KEY = os.environ.get("DOCUELEVATE_API_KEY")
+
+if not IMAP_PASS:
+    sys.exit("Error: IMAP_PASS environment variable is not set.")
+if not API_KEY:
+    sys.exit("Error: DOCUELEVATE_API_KEY environment variable is not set.")
+
+
+def fetch_and_upload():
+    mail = imaplib.IMAP4_SSL(IMAP_HOST)
+    mail.login(IMAP_USER, IMAP_PASS)
+    mail.select("INBOX")
+
+    _, msg_ids = mail.search(None, "UNSEEN")
+    for msg_id in msg_ids[0].split():
+        _, msg_data = mail.fetch(msg_id, "(RFC822)")
+        msg = email.message_from_bytes(msg_data[0][1])
+
+        for part in msg.walk():
+            if part.get_content_maintype() == "multipart":
+                continue
+            if part.get("Content-Disposition") is None:
+                continue
+            filename = part.get_filename()
+            if filename and filename.lower().endswith(".pdf"):
+                payload = part.get_payload(decode=True)
+                files = {"file": (filename, payload, "application/pdf")}
+                headers = {"Authorization": f"Bearer {API_KEY}"}
+                resp = requests.post(
+                    f"{DOCUELEVATE_URL}/api/upload",
+                    files=files,
+                    headers=headers
+                )
+                print(f"Uploaded {filename}: {resp.status_code}")
+        mail.store(msg_id, "+FLAGS", "\\Seen")
+
+    mail.logout()
+
+if __name__ == "__main__":
+    fetch_and_upload()
+```
+
+Run this script via a cron job every few minutes:
+
+```bash
+*/5 * * * * /usr/bin/python3 /opt/email_to_docuelevate.py >> /var/log/docuelevate_import.log 2>&1
+```
+
+---
+
+## Option B: Scan to Network Folder (SMB/CIFS)
+
+HP Enterprise printers can scan directly to a network folder. You can configure a watched folder that DocuElevate monitors for new files.
+
+### Step 1: Set Up a Shared Network Folder
+
+On your DocuElevate server (or any reachable server), create a shared folder:
+
+```bash
+# Create the shared folder
+mkdir -p /srv/scanner-inbox
+
+# Create a dedicated Samba user for the printer
+sudo useradd -M -s /sbin/nologin scanner
+sudo smbpasswd -a scanner   # set a password for the printer to authenticate with
+
+# Install Samba
+sudo apt-get install samba
+
+# Add to /etc/samba/smb.conf:
+[scanner-inbox]
+  path = /srv/scanner-inbox
+  writable = yes
+  guest ok = no
+  valid users = scanner
+  create mask = 0660
+  directory mask = 0770
+```
+
+> **Security note:** Use a dedicated user (`scanner`) with a strong password instead of `guest ok = yes`. This prevents unauthorised devices on your network from depositing files.
+
+Restart Samba: `sudo systemctl restart smbd`
+
+### Step 2: Configure the HP Printer for Scan to Network Folder
+
+1. Open the printer's **Embedded Web Server (EWS)**.
+2. Go to **Scan** → **Scan to Network Folder**.
+3. Click **Add** to create a new Quick Set:
+   - **UNC Path:** `\\192.168.1.200\scanner-inbox` (replace with your server's IP)
+   - **Username:** `scanner` (the Samba user created above)
+   - **Password:** the password set with `smbpasswd`
+   - **File Type:** PDF (Searchable PDF if available)
+   - **Resolution:** 200–300 DPI
+4. Test the connection from the EWS interface.
+
+### Step 3: Configure DocuElevate to Watch the Folder
+
+In your DocuElevate `.env` configuration:
+
+```env
+# Enable folder watching
+WATCH_FOLDER_ENABLED=true
+WATCH_FOLDER_PATH=/srv/scanner-inbox
+WATCH_FOLDER_INTERVAL=30  # seconds between checks
+```
+
+DocuElevate's Celery worker will automatically detect and process new files placed in the watched folder.
+
+---
+
+## Option C: Scan to FTP/WebDAV
+
+DocuElevate supports FTP and WebDAV as upload targets. HP printers can send scanned documents directly.
+
+### WebDAV Configuration
+
+1. In the EWS, go to **Scan** → **Save to SharePoint** or **Save to Network Folder**.
+2. Some HP models support WebDAV directly — configure the WebDAV URL to point to DocuElevate's WebDAV endpoint (if enabled):
+   - **URL:** `http://your-docuelevate-host:8000/webdav/inbox/`
+   - **Username/Password:** Your DocuElevate credentials
+
+### FTP Configuration
+
+1. Ensure an FTP server is running alongside DocuElevate (or configure one in docker-compose).
+2. In the EWS, configure **Scan to FTP**:
+   - **FTP Server:** `192.168.1.200`
+   - **Port:** `21`
+   - **Remote Path:** `/scanner-inbox/`
+3. DocuElevate's watch folder will pick up the FTP-delivered files.
+
+---
+
+## Recommended Scanner Quick Set Settings
+
+| Setting | Recommended Value |
+|---------|------------------|
+| File Type | PDF (Searchable PDF / PDF/A if available) |
+| Resolution | 200–300 DPI |
+| Color Mode | Auto Detect or Grayscale |
+| Sides | Auto Detect (2-sided) |
+| Original Size | Auto Detect |
+| Orientation | Auto Detect |
+
+---
+
+## Troubleshooting
+
+**Printer can't connect to the network folder?**  
+→ Verify the IP address and that the Samba/SMB service is running. Check firewall rules (port 445/TCP).
+
+**Scanned PDFs aren't being processed?**  
+→ Check DocuElevate's Celery worker logs: `docker logs document_worker --follow`
+
+**Email attachments not arriving?**  
+→ Verify SMTP settings on the printer. Check spam filters on the receiving mailbox.
+
+**Poor OCR quality?**  
+→ Increase scan resolution to 300 DPI and use Grayscale mode for text documents.
+
+---
+
+## Related Documentation
+
+- [DocuElevate Configuration Guide](../ConfigurationGuide.md)
+- [Storage Architecture](../StorageArchitecture.md)
+- [Troubleshooting](../Troubleshooting.md)