fix(migrations): merge main and rechain compliance migration as 031 after 030_add_mobile_devices

Merge main branch into compliance templates feature branch.

Main had advanced with migrations 027-030 (ensure_shared_links,
audit_logs, user_language_preference, mobile_devices) since this
branch forked. Our compliance migration was 027 with down_revision
026, which conflicted with main's 027_ensure_shared_links_table.

Changes:
- Merge main (including i18n, audit logs, mobile, GraphQL features)
- Resolve conflicts in app/api/__init__.py, app/models.py, tests/conftest.py
- Rename 027_add_compliance_templates → 031_add_compliance_templates
- Rechain: down_revision 026_add_scheduled_jobs → 030_add_mobile_devices
- Add ComplianceTemplate to migrations/env.py imports
- Alembic now has single head: 031_add_compliance_templates
This commit is contained in:
copilot-swe-agent[bot]
2026-03-12 00:00:29 +00:00
94 changed files with 23120 additions and 98 deletions
+188
View File
@@ -2063,3 +2063,191 @@ print(response.json())
## Further Assistance
For additional help with the API, please contact our support team or refer to the [Development Guide](../CONTRIBUTING.md).
## Mobile App API
The mobile API provides endpoints used by the native iOS and Android app. All endpoints require authentication (Bearer token or active session cookie).
For full mobile app documentation see [MobileApp.md](./MobileApp.md).
### POST /api/mobile/generate-token
Exchange an active web session for a long-lived API token scoped to the mobile app.
**Request:**
```json
{ "device_name": "John's iPhone" }
```
**Response (201 Created):**
```json
{
"token": "de_AbCdEfGhIjKl...",
"token_id": 42,
"name": "Mobile App John's iPhone",
"created_at": "2026-03-10T09:30:00Z"
}
```
> The `token` is shown **once only**.
### POST /api/mobile/register-device
Register an Expo push token to receive push notifications.
**Request:**
```json
{
"push_token": "ExponentPushToken[xxxxxx]",
"device_name": "John's iPhone",
"platform": "ios"
}
```
**Response (201 Created):** Device record with `id`, `platform`, `is_active`, `created_at`.
### GET /api/mobile/devices
List all registered push-notification devices for the current user.
**Response (200 OK):** Array of device records.
### DELETE /api/mobile/devices/{device_id}
Deactivate a push-notification device. The device will no longer receive push notifications.
**Response (204 No Content)**
### GET /api/mobile/whoami
Return basic profile information for the authenticated user.
**Response (200 OK):**
```json
{
"owner_id": "john@example.com",
"display_name": "John Doe",
"email": "john@example.com",
"avatar_url": "https://www.gravatar.com/avatar/...",
"is_admin": false
}
```
---
## GraphQL API
DocuElevate exposes a GraphQL API at `/graphql` alongside the REST API. It
supports flexible queries with field selection, making it ideal for dashboards
and integrations that only need a subset of the available data.
### Endpoint
| Method | URL | Description |
|--------|-----|-------------|
| `POST` | `/graphql` | Execute a GraphQL query or mutation |
| `GET` | `/graphql` | Open the GraphiQL interactive playground |
### Authentication
The GraphQL endpoint honours the same authentication rules as the REST API:
- **`AUTH_ENABLED=False`** (default, single-user mode): all queries are
allowed without credentials.
- **`AUTH_ENABLED=True`** (multi-user mode): a valid session cookie **or**
an `Authorization: Bearer <token>` API token is required. Admin-only
queries (settings, users) additionally require the `is_admin` flag.
### Available Queries
| Field | Returns | Notes |
|-------|---------|-------|
| `documents(ownerId, limit, offset)` | `[DocumentType]` | Paginated list of documents |
| `document(id)` | `DocumentType` | Single document by primary key |
| `pipelines(ownerId, limit, offset)` | `[PipelineType]` | Paginated list of pipelines with steps |
| `pipeline(id)` | `PipelineType` | Single pipeline by primary key |
| `settings(limit, offset)` | `[SettingType]` | Non-sensitive app settings (**admin only**) |
| `users(limit, offset)` | `[UserType]` | User profiles (**admin only**) |
| `user(userId)` | `UserType` | Single user profile (**admin only**) |
> **Note:** Sensitive configuration keys (API secrets, passwords, tokens) are
> automatically excluded from the `settings` query regardless of the caller's
> privilege level.
### GraphiQL Playground
Navigate to `http://<your-instance>/graphql` in a browser to open the
interactive GraphiQL IDE, which provides schema documentation, auto-complete,
and the ability to run queries directly.
### Example Queries
**List recent documents:**
```graphql
{
documents(limit: 5) {
id
originalFilename
mimeType
fileSize
documentTitle
createdAt
}
}
```
**Fetch a pipeline with its steps:**
```graphql
{
pipeline(id: 1) {
id
name
description
isDefault
isActive
steps {
position
stepType
label
enabled
}
}
}
```
**List application settings (admin only):**
```graphql
{
settings {
key
value
updatedAt
}
}
```
**List user profiles (admin only):**
```graphql
{
users(limit: 10) {
userId
displayName
subscriptionTier
isBlocked
}
}
```
**Using variables:**
```graphql
query GetDocument($id: Int!) {
document(id: $id) {
id
originalFilename
documentTitle
isDuplicate
ocrQualityScore
}
}
```
Variables: `{ "id": 42 }`
+55
View File
@@ -399,6 +399,61 @@ default overage buffer applied across all plans.
DocuElevate supports HTTP security headers to improve browser-side security. **These headers are disabled by default** since most deployments use a reverse proxy (Traefik, Nginx, etc.) that already adds them. Enable only if deploying directly without a reverse proxy. See [Deployment Guide - Security Headers](DeploymentGuide.md#security-headers) for detailed configuration examples.
### Audit Logging
DocuElevate provides comprehensive audit logging that records significant actions (logins, document CRUD, settings changes) to an append-only database table. Every entry captures the timestamp, user, action, resource, client IP, and optional JSON details.
| **Variable** | **Description** | **Default** |
|--------------------------------|---------------------------------------------------------------------------------------------------|-------------|
| `AUDIT_LOGGING_ENABLED` | Enable the HTTP request audit-logging middleware. | `true` |
| `AUDIT_LOG_INCLUDE_CLIENT_IP` | Include the client IP address in audit log entries. Disable for GDPR-sensitive deployments. | `true` |
#### SIEM Integration
Audit events can be forwarded in real time to external SIEM systems for centralised monitoring, alerting, and long-term retention. Two transports are supported:
* **Syslog** RFC 5424 structured-data messages over UDP or TCP. Works with rsyslog, syslog-ng, Graylog, Datadog, etc.
* **HTTP** JSON POST payloads compatible with Splunk HEC, Logstash HTTP input, Grafana Loki push API, and any generic webhook.
| **Variable** | **Description** | **Default** |
|-------------------------------------|---------------------------------------------------------------------------------------------------|---------------|
| `AUDIT_SIEM_ENABLED` | Enable forwarding of audit events to an external SIEM system. | `false` |
| `AUDIT_SIEM_TRANSPORT` | Transport: `syslog` or `http`. | `syslog` |
| `AUDIT_SIEM_SYSLOG_HOST` | Hostname or IP of the syslog receiver. | `localhost` |
| `AUDIT_SIEM_SYSLOG_PORT` | Port of the syslog receiver. | `514` |
| `AUDIT_SIEM_SYSLOG_PROTOCOL` | Protocol for syslog: `udp` or `tcp`. | `udp` |
| `AUDIT_SIEM_HTTP_URL` | HTTP endpoint URL for SIEM delivery (e.g. Splunk HEC, Logstash, Loki). | *(empty)* |
| `AUDIT_SIEM_HTTP_TOKEN` | Bearer / HEC token for the SIEM HTTP endpoint. | *(empty)* |
| `AUDIT_SIEM_HTTP_CUSTOM_HEADERS` | Comma-separated `Key:Value` extra headers for SIEM HTTP requests. | *(empty)* |
**Example Syslog to rsyslog:**
```bash
AUDIT_SIEM_ENABLED=true
AUDIT_SIEM_TRANSPORT=syslog
AUDIT_SIEM_SYSLOG_HOST=syslog.internal.example.com
AUDIT_SIEM_SYSLOG_PORT=514
AUDIT_SIEM_SYSLOG_PROTOCOL=udp
```
**Example Splunk HEC:**
```bash
AUDIT_SIEM_ENABLED=true
AUDIT_SIEM_TRANSPORT=http
AUDIT_SIEM_HTTP_URL=https://splunk.example.com:8088/services/collector/event
AUDIT_SIEM_HTTP_TOKEN=your-hec-token
```
**Example Logstash HTTP input:**
```bash
AUDIT_SIEM_ENABLED=true
AUDIT_SIEM_TRANSPORT=http
AUDIT_SIEM_HTTP_URL=https://logstash.example.com:8080
AUDIT_SIEM_HTTP_TOKEN=
```
### Rate Limiting
DocuElevate implements rate limiting to protect against DoS attacks and API abuse. **Rate limiting is enabled by default** and uses Redis for distributed rate limiting across multiple workers.
+231
View File
@@ -0,0 +1,231 @@
# Internationalization (i18n) & Localization (l10n) Guide
DocuElevate supports **10 languages** for its web UI, with automatic browser
language detection, user-preference persistence, and an AI-powered fallback
translator for strings that haven't been manually translated yet.
## Supported Languages
| Code | Language | Native Name | Priority |
|------|------------|-------------|----------|
| `en` | English | English | Tier 1 |
| `de` | German | Deutsch | Tier 1 |
| `fr` | French | Français | Tier 1 |
| `es` | Spanish | Español | Tier 1 |
| `it` | Italian | Italiano | Tier 1 |
| `pt` | Portuguese | Português | Tier 1 |
| `nl` | Dutch | Nederlands | Tier 2 |
| `pl` | Polish | Polski | Tier 2 |
| `zh` | Chinese | 中文 | Tier 2 |
| `ru` | Russian | Русский | Tier 2 |
> **Tier 1** languages (European priority) have complete, manually-reviewed
> translations. **Tier 2** languages have complete translations but may
> receive less frequent updates.
## How Language Is Detected
DocuElevate resolves the display language in the following priority order:
1. **User profile preference** — stored in the database (`UserProfile.preferred_language`)
and loaded into the session on login
2. **Cookie**`docuelevate_lang` cookie (30-day expiry, set when user selects a language)
3. **Browser `Accept-Language` header** — the highest-priority match among supported languages
4. **Default** — English (`en`)
## Selecting Your Language
### Via the Navigation Bar
Click the 🌐 **globe icon** in the top navigation bar. A dropdown menu shows all
available languages with their native names and flag emoji. The current language
is highlighted with a blue checkmark.
### Via the API
```bash
# Set language to German
curl -X POST http://localhost:8000/api/i18n/language \
-H "Content-Type: application/json" \
-d '{"language": "de"}'
# List all available languages
curl http://localhost:8000/api/i18n/languages
```
### Via Cookie (Programmatic)
Set the `docuelevate_lang` cookie to any supported language code:
```javascript
document.cookie = "docuelevate_lang=fr; max-age=2592000; path=/";
location.reload();
```
## For Developers
### Translation File Structure
Translations are stored as flat JSON files in `frontend/translations/`:
```
frontend/translations/
├── en.json # English (base / reference)
├── de.json # German
├── fr.json # French
├── es.json # Spanish
├── it.json # Italian
├── pt.json # Portuguese
├── nl.json # Dutch
├── pl.json # Polish
├── zh.json # Chinese
└── ru.json # Russian
```
Each file is a flat key-value dictionary with dot-notation namespacing:
```json
{
"nav.dashboard": "Dashboard",
"nav.upload": "Upload",
"upload.max_size": "Maximum file size: {size}",
"footer.copyright": "DocuElevate {year}"
}
```
### Using Translations in Templates
The `_()` function is available globally in all Jinja2 templates:
```jinja2
{# Simple translation #}
<h1>{{ _("dashboard.title") }}</h1>
{# Translation with placeholders #}
<p>{{ _("upload.max_size", size="50 MB") }}</p>
{# Translation in attributes #}
<button aria-label="{{ _('common.save') }}">{{ _("common.save") }}</button>
```
### Using Translations in Python
```python
from app.utils.i18n import translate
# Basic translation
text = translate("nav.dashboard", "de") # → "Übersicht"
# With placeholders
text = translate("footer.copyright", "fr", year="2025") # → "DocuElevate 2025"
```
### Localization Helpers
Format dates, times, and numbers according to locale conventions:
```jinja2
{# In templates — locale is automatically detected #}
<span>{{ format_date_l10n(document.created_at) }}</span>
<span>{{ format_number_l10n(file_count) }}</span>
```
```python
# In Python
from app.utils.i18n import format_date, format_number
format_date(date(2025, 3, 15), "de") # → "15. March 2025"
format_date(date(2025, 3, 15), "de", short=True) # → "15.03.2025"
format_number(1234567, "de") # → "1.234.567"
format_number(1234.56, "en") # → "1,234.56"
```
### Adding a New Translation Key
1. Add the key and English text to `frontend/translations/en.json`
2. Add translations for all other languages in their respective files
3. Use `{{ _("your.new.key") }}` in templates
### AI Fallback Translation
When a translation key exists in English but not in the target language,
DocuElevate can use the configured AI provider (OpenAI, Anthropic, etc.)
to translate the string on-the-fly:
```python
from app.utils.i18n import translate_with_ai_fallback
# Falls back to AI if no manual translation exists
translated = translate_with_ai_fallback("Welcome to our platform", "de")
```
The AI fallback:
- Uses the `AI_MODEL` or `OPENAI_MODEL` setting
- Caches results in memory for the process lifetime
- Returns the original English text if the AI call fails
- Is designed for graceful degradation — the UI never breaks
### Adding a New Language
1. Create a new JSON file in `frontend/translations/` (e.g., `ja.json`)
2. Copy the structure from `en.json` and translate all values
3. Add the language to `SUPPORTED_LANGUAGES` in `app/utils/i18n.py`:
```python
{"code": "ja", "name": "Japanese", "native": "日本語", "flag": "🇯🇵"},
```
4. Add locale formatting rules to `_LOCALE_FORMATS` in the same file
5. Create a database migration if needed (the `preferred_language` column
already accepts any string up to 10 characters)
### Database Migration
Migration `027_add_user_language_preference` adds a `preferred_language`
column to the `user_profiles` table. This column stores the user's chosen
UI language as an ISO 639-1 code (e.g., `"de"`, `"fr"`). A `NULL` value
means "auto-detect from browser settings."
### API Reference
#### `GET /api/i18n/languages`
Returns all supported languages and the current active language.
**Response:**
```json
{
"languages": [
{"code": "en", "name": "English", "native": "English", "flag": "🇬🇧"},
{"code": "de", "name": "German", "native": "Deutsch", "flag": "🇩🇪"}
],
"current": "en",
"default": "en"
}
```
#### `POST /api/i18n/language`
Set the preferred UI language. Persists in session, cookie, and database.
**Request:**
```json
{"language": "de"}
```
**Response:**
```json
{
"language": "de",
"message": "Language changed to Deutsch"
}
```
## Configuration
No additional configuration is required. The i18n system works out of the box
with the default English language and automatically detects browser preferences.
| Setting | Default | Description |
|---------|---------|-------------|
| Browser `Accept-Language` | Auto-detected | Used when no explicit preference is set |
| `docuelevate_lang` cookie | Not set | Set when user selects a language via the UI |
| `UserProfile.preferred_language` | `NULL` | Stored in DB for authenticated users |
+252
View File
@@ -0,0 +1,252 @@
# Mobile App
DocuElevate includes a native mobile application for iOS and Android built with **React Native** and **Expo**. The app allows users to capture documents with the device camera, pick files from the device storage, and receive push notifications when documents finish processing.
## Features
| Feature | iOS | Android |
|---------|-----|---------|
| SSO login (OAuth2) | ✅ | ✅ |
| Local / basic auth login | ✅ | ✅ |
| Auto-generated API token | ✅ | ✅ |
| Camera capture → upload | ✅ | ✅ |
| File picker upload | ✅ | ✅ |
| Share Sheet / Share Intent | ✅ | ✅ |
| Push notifications | ✅ | ✅ |
| Document list | ✅ | ✅ |
| Dark mode | ✅ | ✅ |
## Getting Started (Development)
### Prerequisites
- Node.js 18 or later
- [Expo CLI](https://docs.expo.dev/get-started/installation/): `npm install -g @expo/cli`
- [Expo Go](https://expo.dev/client) app on your iOS or Android device (for development)
- A running DocuElevate server reachable from your device
### Run in development mode
```bash
cd mobile
npm install
npx expo start
```
Scan the QR code with **Expo Go** on your device. On iOS you can also use the Camera app.
## Building for Production
DocuElevate uses **Expo Application Services (EAS)** to produce App Store / Play Store binaries.
```bash
# Install EAS CLI globally
npm install -g eas-cli
# Authenticate with Expo
eas login
# Build for iOS (requires Apple Developer account)
eas build --platform ios
# Build for Android
eas build --platform android
```
See the [EAS Build documentation](https://docs.expo.dev/build/introduction/) for full setup instructions.
## Authentication
### SSO Login Flow
The mobile app uses the server's existing OAuth2/SSO setup:
1. User enters the DocuElevate server URL on the login screen.
2. The app opens `<server>/login?mobile=1&redirect_uri=docuelevate://callback` in the **system browser** (Safari / Chrome).
3. The user authenticates via SSO or local credentials.
4. The server redirects back to `docuelevate://callback`.
5. The app calls `POST /api/mobile/generate-token` to exchange the session for a **long-lived API token**.
6. The token is stored securely in the device's keychain (`expo-secure-store`).
### Auto-generated Mobile Token
When the mobile app completes login it automatically creates a named API token (`"Mobile App <device name>"`) via `POST /api/mobile/generate-token`. This token:
- Works identically to tokens created manually in the web UI.
- Is shown in the **API Tokens** page (`/api-tokens`) and can be revoked there.
- Is stored in the device's secure keychain, never in plain storage.
## Push Notifications
Push notifications are delivered via the **Expo Push Notification** service, which routes through Apple Push Notification service (APNs) for iOS and Firebase Cloud Messaging (FCM) for Android.
**No server-side APNs/FCM credentials are required** Expo's servers handle the provider integration.
### How it works
1. After login, the app requests notification permission from the operating system.
2. If granted, the app obtains an **Expo Push Token** (`ExponentPushToken[…]`).
3. The token is registered with the backend via `POST /api/mobile/register-device`.
4. When a document finishes processing, the server sends a push notification to all registered devices for that user.
### Managing registered devices
Users can see and remove their registered devices from the **Profile** tab in the app, or via the API:
```bash
# List registered devices
curl -H "Authorization: Bearer <token>" https://your-server/api/mobile/devices
# Remove a device
curl -X DELETE -H "Authorization: Bearer <token>" https://your-server/api/mobile/devices/<id>
```
## Uploading Documents
### Camera Capture
1. Open the **Upload** tab.
2. Tap **Camera**.
3. Point the camera at the document and take a photo.
4. The image is immediately uploaded and queued for processing.
### File Picker
1. Open the **Upload** tab.
2. Tap **File Picker**.
3. Browse to and select one or more files (PDF, DOCX, images, etc.).
4. Files are uploaded and queued for processing.
### Share Sheet (iOS) / Share Intent (Android)
The app registers itself as a share target so any file can be sent directly to DocuElevate from another app:
1. Open a file in Files, Mail, Safari, or any other app.
2. Tap the **Share** button (iOS) or **Share** (Android).
3. Find and tap **DocuElevate** in the share sheet.
4. The file is immediately uploaded.
> **Note:** The app must be installed on the device for it to appear in the share sheet.
## Mobile API Endpoints
The backend exposes a dedicated `/api/mobile/` namespace:
| Method | Endpoint | Auth | Description |
|--------|----------|------|-------------|
| `POST` | `/api/mobile/generate-token` | Session | Exchange SSO session for API token |
| `POST` | `/api/mobile/register-device` | Bearer | Register Expo push token |
| `GET` | `/api/mobile/devices` | Bearer | List registered devices |
| `DELETE` | `/api/mobile/devices/{id}` | Bearer | Deactivate a device |
| `GET` | `/api/mobile/whoami` | Bearer | Get current user profile |
All other API endpoints (file upload, file listing, etc.) work with Bearer token authentication.
### POST /api/mobile/generate-token
Exchanges an active web session (cookie) for a permanent API token suitable for use in the mobile app.
**Request:**
```json
{ "device_name": "John's iPhone" }
```
**Response (201):**
```json
{
"token": "de_AbCdEfGhIjKl...",
"token_id": 42,
"name": "Mobile App John's iPhone",
"created_at": "2026-03-10T09:30:00Z"
}
```
> ⚠️ The `token` value is returned **once only**. Store it in the device's secure keychain immediately.
### POST /api/mobile/register-device
Registers an Expo push token for the authenticated user.
**Request:**
```json
{
"push_token": "ExponentPushToken[xxxxxxxxxxxxxxxxxxxxxx]",
"device_name": "John's iPhone",
"platform": "ios"
}
```
Supported platforms: `ios`, `android`, `web`.
Re-registering the same token is safe (idempotent).
### GET /api/mobile/whoami
Returns the current user's profile.
**Response (200):**
```json
{
"owner_id": "john@example.com",
"display_name": "John Doe",
"email": "john@example.com",
"avatar_url": "https://www.gravatar.com/avatar/...",
"is_admin": false
}
```
## Configuration
No server-side configuration is required to enable the mobile app. The Expo push notification routing does not need FCM or APNs credentials on the server.
If you wish to use **direct FCM/APNs** without Expo's relay, replace the `send_expo_push_notification` function in `app/utils/push_notification.py` with your own implementation.
## Project Structure (mobile/)
```
mobile/
├── App.tsx # Root component
├── app.json # Expo/EAS configuration
├── eas.json # EAS Build profiles
├── package.json
├── tsconfig.json
└── src/
├── context/
│ └── AuthContext.tsx # Auth state + SSO login flow
├── hooks/
│ └── usePushNotifications.ts # Push token registration
├── screens/
│ ├── LoginScreen.tsx # Server URL + SSO button
│ ├── UploadScreen.tsx # Camera capture + file picker
│ ├── FilesScreen.tsx # Processed document list
│ └── ProfileScreen.tsx # User profile + sign out
└── services/
└── api.ts # DocuElevate REST API client
```
## Troubleshooting
### "Authentication was cancelled or failed"
- Ensure the server URL is correct (including `https://`).
- Verify the server is reachable from your device's network.
- Confirm that `AUTH_ENABLED=True` on the server.
### Push notifications not arriving
1. Check that the app has notification permission (Settings → DocuElevate → Notifications).
2. Verify the device is registered: `GET /api/mobile/devices`.
3. Ensure the server can reach `https://exp.host` (outbound HTTPS on port 443).
4. On Android, add `google-services.json` to the `mobile/` directory if you are building your own binary.
### "Connection refused" or timeout
- Verify that the DocuElevate server is running and accessible.
- Ensure the server's `EXTERNAL_HOSTNAME` or reverse proxy is configured correctly.
- Check that the server accepts CORS requests from `docuelevate://`.
## Related Documentation
- [API Documentation](./API.md)
- [Configuration Guide](./ConfigurationGuide.md)
- [Deployment Guide](./DeploymentGuide.md)