fix: resolve merge conflicts with main branch

Merge origin/main into feature branch, resolving 18 conflicts: - app/api/__init__.py: add classification_rules_router alongside system_reset_router - app/config.py: keep system reset settings from main - app/models.py: keep ClassificationRuleModel alongside main's models - app/utils/settings_service.py: keep factory reset settings from main - app/views/__init__.py: keep system_reset_router from main - tests/conftest.py: add ClassificationRuleModel import - migrations/env.py: add ClassificationRuleModel import - .env.demo, docs/*, frontend/*, mobile/*: keep additions from main - BUILD_DATE, GIT_SHA, RUNTIME_INFO, VERSION, CHANGELOG.md: accept main's version Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-17 11:09:39 +00:00
parent bdc26846bd
commit c7d67031d4
33 changed files with 2275 additions and 11 deletions
@@ -2562,3 +2562,64 @@ query GetDocument($id: Int!) {
 }
 ```
 Variables: `{ "id": 42 }`
+
+## System Reset
+
+Admin-only endpoints for resetting the system to a clean state.  Requires `ENABLE_FACTORY_RESET=True`.
+
+### GET /api/admin/system-reset/status
+
+Check whether the system reset feature is enabled.
+
+**Response (200):**
+```json
+{
+  "enabled": true,
+  "factory_reset_on_startup": false
+}
+```
+
+### POST /api/admin/system-reset/full
+
+Wipe all user data (database + work-files).
+
+**Request:**
+```json
+{
+  "confirmation": "DELETE"
+}
+```
+
+**Response (200):**
+```json
+{
+  "status": "ok",
+  "result": {
+    "database": { "files": 42, "processing_logs": 100 },
+    "filesystem": { "deleted_dirs": 5, "deleted_files": 12 }
+  }
+}
+```
+
+### POST /api/admin/system-reset/reimport
+
+Move original files to a reimport folder, wipe everything, and configure the reimport folder as a watch folder for re-ingestion.
+
+**Request:**
+```json
+{
+  "confirmation": "REIMPORT"
+}
+```
+
+**Response (200):**
+```json
+{
+  "status": "ok",
+  "result": {
+    "database": { "files": 42 },
+    "filesystem": { "deleted_dirs": 5, "deleted_files": 12 },
+    "reimport": { "files_moved": 42, "reimport_folder": "/workdir/reimport" }
+  }
+}
+```
@@ -17,6 +17,8 @@ Configuration is primarily done through environment variables specified in a `.e
 | `EXTERNAL_HOSTNAME`    | The external hostname for the application.             | `docuelevate.example.com`      |
 | `ALLOW_FILE_DELETE`    | Enable file deletion in the web interface (`true`/`false`). | `true`                      |
 | `COMPLIANCE_ENABLED`   | Enable the compliance templates dashboard (GDPR, HIPAA, SOC 2). | `true`                      |
+| `FACTORY_RESET_ON_STARTUP` | Wipe all user data on every startup (demo/testing). | `false` |
+| `ENABLE_FACTORY_RESET` | Show the System Reset page in the admin UI.         | `false` |

 ### Batch Processing Settings

@@ -1856,6 +1858,52 @@ BACKUP_RETAIN_WEEKLY=13

 You can choose which document storage services to use by only including the relevant environment variables. For example, if you only want to use Dropbox, include only the Dropbox variables and omit the Paperless NGX and Nextcloud variables.

+## System Reset / Factory Reset
+
+DocuElevate provides two mechanisms for resetting the system to a clean state.  Both are **disabled by default** and must be explicitly enabled.
+
+### Automatic Reset on Startup
+
+Set `FACTORY_RESET_ON_STARTUP=true` to wipe all user data (database rows and work-files) every time the application starts.  This is useful for demo, testing, or ephemeral environments where you always want a fresh instance.
+
+```dotenv
+FACTORY_RESET_ON_STARTUP=true
+```
+
+> **Warning:** This destroys all documents, processing history, audit logs, and backups on every restart.  Application settings and configuration are preserved.
+
+### Admin UI Reset Page
+
+Set `ENABLE_FACTORY_RESET=true` to display the **System Reset** page in the admin navigation menu.  From this page, administrators can:
+
+| Action | Confirmation | Description |
+|--------|-------------|-------------|
+| **Full Reset** | Type `DELETE` | Wipes all database rows and work-files.  The system returns to its initial state. |
+| **Reset & Re-import** | Type `REIMPORT` | Copies original files to a `reimport/` folder inside the workdir, wipes everything, then configures the reimport folder as a watch folder so files are automatically re-ingested with the same processing pipeline, rate limits, and backoff strategy as regular uploads. |
+
+```dotenv
+ENABLE_FACTORY_RESET=true
+```
+
+### API Endpoints
+
+When `ENABLE_FACTORY_RESET=true`, two admin-only API endpoints are available:
+
+- `POST /api/admin/system-reset/full` — body: `{"confirmation": "DELETE"}`
+- `POST /api/admin/system-reset/reimport` — body: `{"confirmation": "REIMPORT"}`
+- `GET  /api/admin/system-reset/status` — returns current feature-flag state
+
+### What Gets Deleted
+
+| Deleted | Preserved |
+|---------|-----------|
+| All document records (`files` table) | Application settings (`application_settings` table) |
+| Processing logs and steps | User accounts and profiles |
+| Audit logs | Subscription plans |
+| Backup records | Pipelines and scheduled jobs |
+| Original, processed, and temporary files | The workdir directory itself |
+| Watch-folder caches and ingestion state | OAuth and integration configuration |
+
 ## Configuration File Location

 The `.env` file should be placed at the root of the project directory. When using Docker Compose, you can reference it with the `env_file` directive in your `docker-compose.yml`.
@@ -287,6 +287,27 @@ alembic revision --autogenerate -m "describe your change"

 Review the generated file in `migrations/versions/` before applying it.

+> **Tip:** For detailed guidance on naming conventions, idempotent patterns, parallel-branch workflows, and resolving merge conflicts, see the [Migration Workflow Guide](MigrationWorkflow.md).
+
+### Validating the Migration Chain
+
+A CI check and pre-commit hook validate that the migration chain has no broken
+references, duplicate revisions, or diverged heads.  Run the check locally:
+
+```bash
+python scripts/check_alembic_migrations.py
+python scripts/check_alembic_migrations.py --verbose   # extra detail
+```
+
+If you see **"Multiple migration heads detected"**, two branches added
+migrations from the same parent.  Create a merge migration:
+
+```bash
+alembic merge heads -m "merge_parallel_branches"
+```
+
+For a complete walk-through, see the [Migration Workflow Guide](MigrationWorkflow.md).
+
 ### Automating Migrations in Docker Compose

 Add a short-lived `migrate` service that runs before the API and Worker:
@@ -0,0 +1,360 @@
+# Migration Workflow
+
+This guide explains how to create, test, and merge Alembic database migrations in DocuElevate — especially when **multiple feature branches** add migrations in parallel.
+
+## Table of Contents
+
+- [Quick Reference](#quick-reference)
+- [Creating a New Migration](#creating-a-new-migration)
+- [Migration Naming Convention](#migration-naming-convention)
+- [Idempotent Migration Patterns](#idempotent-migration-patterns)
+- [Parallel Branch Development](#parallel-branch-development)
+- [Resolving Migration Conflicts](#resolving-migration-conflicts)
+- [CI Validation](#ci-validation)
+- [Pre-commit Hook](#pre-commit-hook)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Quick Reference
+
+```bash
+# Create a new migration after editing app/models.py
+alembic revision --autogenerate -m "add_foobar_column"
+
+# Apply all pending migrations
+alembic upgrade head
+
+# Check current database version
+alembic current
+
+# View migration history
+alembic history --verbose
+
+# Detect multiple heads (diverged branches)
+alembic heads
+
+# Create a merge migration to resolve multiple heads
+alembic merge heads -m "merge_parallel_branches"
+
+# Validate migration chain integrity (CI script)
+python scripts/check_alembic_migrations.py
+python scripts/check_alembic_migrations.py --verbose
+```
+
+---
+
+## Creating a New Migration
+
+1. **Edit `app/models.py`** — add or modify SQLAlchemy model classes.
+
+2. **Generate the migration** from the repo root.  Use `--rev-id` to set the
+   revision identifier directly (avoids renaming afterwards):
+
+   ```bash
+   alembic revision --autogenerate --rev-id 037_add_my_new_table -m "add my new table"
+   ```
+
+   This creates `migrations/versions/037_add_my_new_table_add_my_new_table.py`
+   with `revision = "037_add_my_new_table"`.  Rename the file to match:
+
+   ```bash
+   mv migrations/versions/037_add_my_new_table_add_my_new_table.py \
+      migrations/versions/037_add_my_new_table.py
+   ```
+
+   Alternatively, generate with the default hash and then rename:
+
+   ```bash
+   alembic revision --autogenerate -m "add_my_new_table"
+   # Rename: mv migrations/versions/<hash>_add_my_new_table.py migrations/versions/037_add_my_new_table.py
+   # Update revision inside the file to match the filename stem.
+   ```
+
+   Alembic uses the `migrations/script.py.mako` template to generate the file.  The template includes inline comments about idempotent patterns — read them.
+
+3. **Review the generated code** — autogenerate is helpful but not perfect.  Check:
+   - Are new tables and columns detected correctly?
+   - Does the `downgrade()` reverse all changes?
+   - Are SQLite-incompatible operations wrapped in `batch_alter_table()`?
+
+4. **Test the migration** against a fresh database:
+
+   ```bash
+   # Apply
+   alembic upgrade head
+
+   # Rollback
+   alembic downgrade -1
+
+   # Re-apply
+   alembic upgrade head
+   ```
+
+5. **Run the chain validation**:
+
+   ```bash
+   python scripts/check_alembic_migrations.py
+   ```
+
+---
+
+## Migration Naming Convention
+
+All migration files follow a **sequential numeric prefix** scheme:
+
+```
+NNN_short_description.py
+```
+
+| Component | Rule |
+|-----------|------|
+| `NNN` | Three-digit zero-padded number, incrementing from the previous migration |
+| `short_description` | Lowercase snake_case summary of the change |
+
+The **`revision`** variable inside the file **must match the filename stem** exactly:
+
+```python
+# File: migrations/versions/037_add_classification_rules.py
+revision: str = "037_add_classification_rules"
+down_revision: Union[str, None] = "036_add_document_translation_fields"
+```
+
+The CI check (`scripts/check_alembic_migrations.py`) enforces this consistency.
+
+---
+
+## Idempotent Migration Patterns
+
+Migrations should be **idempotent** — safe to run even if the change already exists.  This is critical for SQLite compatibility and for recovering from partial failures.
+
+### Add a Column (only if missing)
+
+```python
+def upgrade() -> None:
+    conn = op.get_bind()
+    inspector = sa.inspect(conn)
+
+    if "my_table" in inspector.get_table_names():
+        existing = {c["name"] for c in inspector.get_columns("my_table")}
+        if "new_col" not in existing:
+            with op.batch_alter_table("my_table") as batch_op:
+                batch_op.add_column(sa.Column("new_col", sa.String(128), nullable=True))
+```
+
+### Create a Table (only if missing)
+
+```python
+def upgrade() -> None:
+    conn = op.get_bind()
+    inspector = sa.inspect(conn)
+
+    if "new_table" not in inspector.get_table_names():
+        op.create_table(
+            "new_table",
+            sa.Column("id", sa.Integer(), primary_key=True),
+            sa.Column("name", sa.String(255), nullable=False),
+        )
+```
+
+### Drop a Column (only if present)
+
+```python
+def downgrade() -> None:
+    conn = op.get_bind()
+    inspector = sa.inspect(conn)
+
+    if "my_table" in inspector.get_table_names():
+        existing = {c["name"] for c in inspector.get_columns("my_table")}
+        if "new_col" in existing:
+            with op.batch_alter_table("my_table") as batch_op:
+                batch_op.drop_column("new_col")
+```
+
+### Use `batch_alter_table` for SQLite
+
+SQLite does not support `ALTER TABLE DROP COLUMN` or `ALTER TABLE RENAME COLUMN` natively.  Alembic's `batch_alter_table` context manager works around this by recreating the table:
+
+```python
+with op.batch_alter_table("users") as batch_op:
+    batch_op.add_column(sa.Column("phone", sa.String(20), nullable=True))
+    batch_op.drop_column("fax")
+```
+
+---
+
+## Parallel Branch Development
+
+When two feature branches both add migrations from the same parent, the migration chain **diverges** into multiple heads.  This is normal and expected — Alembic supports it — but the heads must be merged before the code reaches `main`.
+
+### Example
+
+```
+main:   001 → 002 → 003
+                         ↘ Branch A: 004_add_widgets
+                         ↘ Branch B: 004_add_gadgets   ← two heads!
+```
+
+### How to Avoid Conflicts
+
+1. **Coordinate** — if two developers are both adding migrations, assign different sequence numbers (e.g., `037_` and `038_`).  Even if both depend on `036_`, different numbers prevent filename collisions.
+
+2. **Rebase early** — before opening a PR, rebase your branch onto the latest `main`:
+
+   ```bash
+   git fetch origin main
+   git rebase origin/main
+   ```
+
+   If `main` now has a new migration `037_*`, renumber yours to `038_*` and update `down_revision` to point at `037_*`.
+
+3. **Check for multiple heads** locally:
+
+   ```bash
+   python scripts/check_alembic_migrations.py
+   # or
+   alembic heads
+   ```
+
+---
+
+## Resolving Migration Conflicts
+
+If your PR's CI check reports **"Multiple migration heads detected"**, follow these steps:
+
+### Step 1 — Update Your Branch
+
+```bash
+git fetch origin main
+git merge origin/main
+# or
+git rebase origin/main
+```
+
+### Step 2 — Check Heads
+
+```bash
+python scripts/check_alembic_migrations.py --verbose
+```
+
+The output lists the conflicting heads.
+
+### Step 3 — Create a Merge Migration
+
+```bash
+alembic merge heads -m "merge_parallel_branches"
+```
+
+This generates a new migration with **two parents** (a merge point):
+
+```python
+down_revision = ("037_add_widgets", "037_add_gadgets")
+```
+
+### Step 4 — Rename and Validate
+
+Rename the merge migration to the next sequence number:
+
+```bash
+mv migrations/versions/<hash>_merge_parallel_branches.py \
+   migrations/versions/038_merge_parallel_branches.py
+```
+
+Update the `revision` inside to match, then validate:
+
+```bash
+python scripts/check_alembic_migrations.py
+```
+
+### Step 5 — Test
+
+```bash
+alembic upgrade head
+alembic downgrade -1
+alembic upgrade head
+```
+
+---
+
+## CI Validation
+
+The CI pipeline (`.github/workflows/ci.yml`) includes a **migration-chain** job that runs:
+
+```bash
+python scripts/check_alembic_migrations.py
+```
+
+This script checks for:
+
+| Check | Description |
+|-------|-------------|
+| Multiple heads | Diverged migration chains that need a merge migration |
+| Broken references | A `down_revision` that points to a non-existent revision |
+| Duplicate revisions | Two files declaring the same `revision` identifier |
+| Filename mismatches | The `revision` variable doesn't match the filename stem |
+
+The job runs in Stage 1 (fast-fail gates) alongside lint checks.  If it fails, the build is blocked until the migration chain is fixed.
+
+---
+
+## Pre-commit Hook
+
+A local pre-commit hook is configured in `.pre-commit-config.yaml` that runs the same check whenever you commit a change to `migrations/versions/`:
+
+```yaml
+- repo: local
+  hooks:
+    - id: check-alembic-migrations
+      name: Check Alembic migration chain
+      entry: python scripts/check_alembic_migrations.py
+      language: python
+      pass_filenames: false
+      files: ^migrations/versions/.*\.py$
+```
+
+Install the hook:
+
+```bash
+pip install pre-commit
+pre-commit install
+```
+
+---
+
+## Troubleshooting
+
+### "Multiple migration heads detected"
+
+See [Resolving Migration Conflicts](#resolving-migration-conflicts) above.
+
+### "Broken chain: revision X references down_revision Y which does not exist"
+
+You removed or renamed a migration that another migration depends on.  Either restore the missing file or update the dependent migration's `down_revision`.
+
+### "Filename mismatch: file declares revision=X but filename stem is Y"
+
+The `revision` string inside the Python file must match the filename (without `.py`).  Rename the file or update the variable.
+
+### "relation already exists" when running `alembic upgrade head`
+
+The database has a table that a pending migration tries to create.  Stamp the current state:
+
+```bash
+alembic stamp head
+```
+
+### Autogenerate doesn't detect my changes
+
+Ensure all models are imported in `migrations/env.py`.  The `from app.models import ...` block at the top must include your new model class.
+
+### SQLite "no such column" after downgrade
+
+SQLite has limited `ALTER TABLE` support.  Always use `op.batch_alter_table()` for column operations on existing tables.
+
+---
+
+## Further Reading
+
+- [Alembic Tutorial](https://alembic.sqlalchemy.org/en/latest/tutorial.html)
+- [Alembic Branch / Merge](https://alembic.sqlalchemy.org/en/latest/branches.html)
+- [Database Configuration Guide](DatabaseConfiguration.md)
@@ -279,7 +279,19 @@ If you wish to use **direct FCM/APNs** without Expo's relay, replace the `send_e

 ```
 mobile/
-├── App.tsx                      # Root component
+├── App.tsx                      # Root component (legacy, not used at runtime)
+├── app/                         # Expo Router file-based routes
+│   ├── _layout.tsx              # Root layout (AuthGuard + providers)
+│   ├── index.tsx                # Root redirect → /(auth)/
+│   ├── (auth)/                  # Unauthenticated route group
+│   │   ├── _layout.tsx          # Stack navigator (headerless)
+│   │   ├── index.tsx            # Welcome screen
+│   │   └── login.tsx            # Login screen
+│   └── (tabs)/                  # Authenticated route group
+│       ├── _layout.tsx          # Tab navigator
+│       ├── index.tsx            # Upload screen (default tab)
+│       ├── files.tsx            # Files screen
+│       └── profile.tsx          # Profile screen
 ├── app.json                     # Expo/EAS configuration
 ├── eas.json                     # EAS Build profiles
 ├── package.json
@@ -301,6 +313,26 @@ mobile/

 ## Troubleshooting

+### App shows "Hello World" / default Expo page after update
+
+If the iOS or Android app shows a generic "Hello World – This is the first page of your app" screen instead of the DocuElevate UI, it means a stale default `index.tsx` file (generated by Expo CLI scaffolding) is being picked up in the `mobile/app/` directory.
+
+**To fix:**
+
+1. Delete any leftover default `mobile/app/index.tsx` that is **not** the repository version (the repo version contains a `<Redirect>` to `/(auth)/`).
+2. Clear the Metro bundler cache and rebuild:
+   ```bash
+   cd mobile
+   npx expo start --clear
+   ```
+3. For production builds, run a clean EAS build:
+   ```bash
+   eas build --platform ios --clear-cache
+   ```
+
+The repository includes a root `app/index.tsx` that immediately redirects to the authentication flow, so this issue should not recur once the correct file is present.
+
+
 ### "Session expired Local session" during iOS build

 EAS stores an Apple ID session locally (in `~/.expo/`) to manage code-signing certificates and provisioning profiles.  This session expires after a few weeks.