fix: merge main (v0.161.0) into classification feature branch

Resolve all 40 merge conflicts from merging origin/main into the
classification feature branch. Key resolutions:

- Auto-generated files (BUILD_DATE, VERSION, etc.): use main's version
- API tokens: take main's version (token expiry, reactivation, hard-delete)
- Auth: take main's Dropbox credential sharing + token expiry checking
- Config: take main's social_auth_dropbox_use_global_credentials option
- Files API: take main's improved duplicate handling + rate limiting
- Models: keep ClassificationRuleModel alongside main's new models
- Mobile: take main's mature implementation
- Templates/translations: take main's versions (device deletion, reactivation keys)
- Migration: renumber 038_add_classification_rules → 039_add_classification_rules
  to chain after main's 038_add_api_token_expires_at
- Requirements: take main's version (adds segno QR library)
- Tests: take main's more complete token tests, keep classification imports
This commit is contained in:
copilot-swe-agent[bot]
2026-03-20 13:08:41 +00:00
88 changed files with 5893 additions and 889 deletions
+101 -17
View File
@@ -27,9 +27,11 @@ DocuElevate implements rate limiting to protect against abuse and DoS attacks. R
### Default Limits
- **Default endpoints**: 100 requests per minute
- **File upload**: 600 requests per minute
- **File upload**: 600 requests per minute (global) + 20 per user per 60 s (per-user, health-aware)
- **Authentication**: 10 requests per minute
**Per-user upload rate limiting**: Upload endpoints (`/api/ui-upload`, `/api/process-url`) enforce a per-user sliding-window limit that adapts to system load. Under heavy queue depth or high CPU usage, the effective limit is reduced automatically. See the [Configuration Guide](ConfigurationGuide.md#per-user-upload-rate-limiting) for details.
**Note**: Document processing endpoints (OCR, metadata extraction) use built-in queue throttling to control processing rates and prevent upstream API overloads. No additional API-level rate limit is applied to processing endpoints.
### Rate Limit Headers
@@ -53,6 +55,10 @@ RATE_LIMITING_ENABLED=true
RATE_LIMIT_DEFAULT=100/minute
RATE_LIMIT_UPLOAD=600/minute
RATE_LIMIT_AUTH=10/minute
# Per-user upload rate limiting (health-aware)
UPLOAD_RATE_LIMIT_PER_USER=20 # Max uploads per user per window
UPLOAD_RATE_LIMIT_WINDOW=60 # Sliding window in seconds
```
See [Configuration Guide](ConfigurationGuide.md) for more details.
@@ -115,7 +121,8 @@ curl -X GET "http://<your-docuelevate-instance>/api/files" \
|--------|----------|-------------|
| `POST` | `/api/api-tokens/` | Create a new token |
| `GET` | `/api/api-tokens/` | List all your tokens |
| `DELETE` | `/api/api-tokens/{id}` | Revoke a token |
| `DELETE` | `/api/api-tokens/{id}` | Revoke (active) or permanently delete (revoked) a token |
| `POST` | `/api/api-tokens/{id}/reactivate` | Reactivate a revoked token |
### Session Authentication
@@ -235,17 +242,33 @@ The DocuElevate browser extension uses this endpoint to send files directly from
**POST** `/api/ui-upload`
Upload one or more files from your computer for processing.
Upload a file from your computer for processing.
**Request**:
- Multipart form data with file(s)
- Multipart form data with a single `file` field
**Response**:
**Response** (new file):
```json
{
"success": true,
"file_ids": [123, 124],
"message": "Files uploaded and queued for processing"
"task_id": "abc-123",
"status": "queued",
"original_filename": "invoice.pdf",
"stored_filename": "a1b2c3d4.pdf"
}
```
**Response** (exact duplicate, when `ENABLE_DEDUPLICATION=True`):
```json
{
"status": "duplicate",
"original_filename": "invoice.pdf",
"stored_filename": "e5f6a7b8.pdf",
"duplicate_of": {
"duplicate_type": "exact",
"original_file_id": 42,
"original_filename": "invoice.pdf",
"message": "This file is an exact duplicate of an already-processed document. It has not been queued for processing again."
}
}
```
@@ -1356,7 +1379,7 @@ Test an integration connection without saving. Useful for "Test connection" UI b
{"success": true, "message": "IMAP connection successful"}
```
Supported connection tests: `IMAP`, `S3`, `WEBDAV`, `NEXTCLOUD`. Other types return a message that testing is not yet supported.
Supported connection tests: `DROPBOX`, `IMAP`, `S3`, `WEBDAV`, `NEXTCLOUD`. Other types return a message that testing is not yet supported.
### GET /api/integrations/quota/
@@ -1571,6 +1594,47 @@ Lightweight endpoint returning the total number of queued + in-progress items. D
## Diagnostic
### GET /api/diagnostic/healthz/live
Lightweight liveness probe for Kubernetes. Returns **200 OK** as long as the process is running. This endpoint does **not** check external dependencies and is intentionally cheap.
**Authentication:** None (designed for kubelet probes)
**Response (200 OK):**
```json
{
"status": "ok"
}
```
### GET /api/diagnostic/healthz/ready
Readiness probe for Kubernetes. Verifies that the application can serve traffic by checking database and Redis connectivity.
**Authentication:** None (designed for kubelet probes)
**Response (200 OK) ready to serve traffic:**
```json
{
"status": "ready",
"checks": {
"database": {"status": "ok"},
"redis": {"status": "ok"}
}
}
```
**Response (503 Service Unavailable) database unreachable:**
```json
{
"status": "not_ready",
"checks": {
"database": {"status": "error", "detail": "..."},
"redis": {"status": "ok"}
}
}
```
### GET /api/diagnostic/health
System health endpoint designed for monitoring tools such as Grafana, Uptime Kuma, Prometheus blackbox exporter, or any HTTP-based health checker.
@@ -2127,12 +2191,14 @@ Usage tracking records when each token was last used and from which IP address.
### POST /api/api-tokens/
Create a new API token.
Create a new API token. Optionally specify a lifetime in days via
`expires_in_days` (13650). If omitted the token never expires.
**Request:**
```json
{
"name": "CI Pipeline"
"name": "CI Pipeline",
"expires_in_days": 90
}
```
@@ -2147,7 +2213,8 @@ Create a new API token.
"last_used_at": null,
"last_used_ip": null,
"created_at": "2026-03-08T12:00:00Z",
"revoked_at": null
"revoked_at": null,
"expires_at": "2026-06-06T12:00:00Z"
}
```
@@ -2169,15 +2236,20 @@ List all tokens for the authenticated user. The full token value is never includ
"last_used_at": "2026-03-08T15:30:00Z",
"last_used_ip": "203.0.113.42",
"created_at": "2026-03-08T12:00:00Z",
"revoked_at": null
"revoked_at": null,
"expires_at": "2026-06-06T12:00:00Z"
}
]
```
### DELETE /api/api-tokens/{token_id}
Revoke a token. The token is soft-deleted (kept for audit purposes) and can no
longer be used for authentication.
Revoke or permanently delete a token:
* **Active token** soft-revoked (kept for audit purposes, marked inactive).
Response: `{"detail": "Token revoked"}`
* **Already-revoked token** permanently deleted from the database.
Response: `{"detail": "Token deleted"}`
**Response (200):**
```json
@@ -2186,6 +2258,13 @@ longer be used for authentication.
}
```
### POST /api/api-tokens/{token_id}/reactivate
Reactivate a previously revoked token. Clears `revoked_at` and sets
`is_active` back to `true`.
**Response (200):** The updated `TokenResponse` object.
### Using API Tokens
Include the token in the `Authorization` header of any API request:
@@ -2425,9 +2504,14 @@ List all registered push-notification devices for the current user.
### DELETE /api/mobile/devices/{device_id}
Deactivate a push-notification device. The device will no longer receive push notifications.
Deactivate or permanently delete a push-notification device:
**Response (204 No Content)**
* **Active device** soft-deactivated (record kept, will no longer receive push notifications).
Response: `{"detail": "Device deactivated"}`
* **Already-inactive device** permanently deleted from the database.
Response: `{"detail": "Device deleted"}`
**Response (200)**
### GET /api/mobile/whoami
+285
View File
@@ -0,0 +1,285 @@
# Apple App Store Compliance Audit Report
This document details the findings from a comprehensive audit of the DocuElevate mobile app against Apple's App Store Review Guidelines, Human Interface Guidelines (HIG), and privacy requirements. It covers all areas of compliance, risks for rejection, and recommendations.
> **Last Audited:** March 2026
> **App Version:** 1.0.0
> **Expo SDK:** 54.0.0
> **Bundle ID:** `org.docuelevate.mobile`
---
## Executive Summary
The DocuElevate mobile app is broadly compliant with Apple's App Store requirements. The following issues were identified and resolved as part of this audit:
| Issue | Severity | Status |
|-------|----------|--------|
| Unused `fetch` background mode declared | High | ✅ Fixed |
| Missing privacy manifest for required reason APIs | High | ✅ Fixed |
| No account deletion option (Guideline 5.1.1(v)) | Critical | ✅ Fixed |
| No Privacy Policy / Terms of Service links in-app | High | ✅ Fixed |
| Emoji used as UI icons instead of platform-native icons | Medium | ✅ Fixed |
| Missing app version display | Low | ✅ Fixed |
| Unused `Switch` import in ProfileScreen | Low | ✅ Fixed |
---
## 1. Human Interface Guidelines (HIG)
### 1.1 Navigation & Tab Bar ✅
- The app uses a standard bottom tab bar with three tabs: Upload, Files, and Profile.
- Tab icons use **Ionicons** (an icon set that closely maps to Apple's SF Symbols).
- Active/inactive tab colors follow iOS conventions (`#1e40af` active, `#9ca3af` inactive).
- Header styling uses a solid color background with white text, consistent with iOS navigation bar patterns.
### 1.2 Icons & Visual Assets ✅
- **App icon:** Custom `icon.png` provided at root level; Expo handles generating all required sizes.
- **Splash screen:** Uses branded splash with `contain` resize mode and matching background color.
- **Adaptive icon (Android):** Properly configured with foreground image and background color.
- **Action buttons:** Previously used emoji characters (📷, 🖼️, 📄) which render inconsistently across iOS versions. **Fixed:** Now using Ionicons (`camera-outline`, `images-outline`, `document-outline`).
- **Status indicators:** Previously used emoji (✅, ❌, ⏳, ⚙️). **Fixed:** Now using Ionicons with semantic colors.
### 1.3 Typography & Colors ✅
- Uses system fonts (default React Native text rendering uses San Francisco on iOS).
- Color palette (`#1e40af` primary blue, semantic reds/greens/grays) provides sufficient contrast ratios.
- Text sizes follow iOS recommended minimums (body text ≥ 13pt).
### 1.4 Touch Targets ✅
- All interactive elements have `minHeight: 44` or `minHeight: 48` (meets Apple's 44×44pt minimum).
- Back links, cancel buttons, and retry buttons all meet minimum touch target requirements.
### 1.5 Safe Areas ✅
- The app uses `react-native-safe-area-context` (`SafeAreaProvider`) to respect device notches, Dynamic Island, and home indicator.
### 1.6 Dark Mode ✅
- `userInterfaceStyle: "automatic"` is set in `app.json`, enabling automatic dark mode support.
---
## 2. Privacy & Data Usage
### 2.1 Permission Descriptions ✅
All iOS permission strings (Info.plist keys) are present and provide clear, specific descriptions of why each permission is needed:
| Permission | Key | Description |
|-----------|-----|-------------|
| Camera | `NSCameraUsageDescription` | "DocuElevate uses the camera to scan QR codes for login and to capture documents for upload." |
| Photo Library (Read) | `NSPhotoLibraryUsageDescription` | "DocuElevate accesses your photo library to select documents for upload." |
| Photo Library (Write) | `NSPhotoLibraryAddUsageDescription` | "DocuElevate saves scanned documents to your photo library." |
**Assessment:** All descriptions clearly explain the purpose, which is a requirement for App Review approval.
### 2.2 Push Notifications ✅
- Push notification permission is requested at runtime (not at launch) when the user enters the authenticated area.
- The app works gracefully without push notifications if permission is denied.
- Device tokens are registered via a dedicated backend endpoint.
### 2.3 Background Modes ✅ (Fixed)
- **Previous state:** `UIBackgroundModes` included `["fetch", "remote-notification"]`.
- **Issue:** The app does not implement background fetch (`application:performFetchWithCompletionHandler:`). Apple may reject apps that declare background modes they don't actively use (Guideline 2.5.4).
- **Fix:** Removed `fetch` from `UIBackgroundModes`. Only `remote-notification` remains, which is required for push notification delivery.
### 2.4 Privacy Manifest ✅ (Fixed)
Starting in Spring 2024, Apple requires a privacy manifest (`PrivacyInfo.xcprivacy`) for apps using specific APIs. The following required reason APIs are used by the app's dependencies:
| API Category | Reason Code | Justification |
|-------------|-------------|---------------|
| `NSPrivacyAccessedAPICategoryUserDefaults` | `CA92.1` | Used by `@react-native-async-storage/async-storage` for user preferences |
| `NSPrivacyAccessedAPICategoryFileTimestamp` | `C617.1` | Used by `expo-file-system` to read file metadata |
| `NSPrivacyAccessedAPICategoryDiskSpace` | `E174.1` | Used by Expo runtime for storage space checks |
| `NSPrivacyAccessedAPICategorySystemBootTime` | `35F9.1` | Used by React Native's timing APIs |
The privacy manifest is configured via `expo-build-properties` plugin in `app.json`, which ensures it is included in the generated Xcode project during EAS Build.
### 2.5 Tracking & Analytics ✅
- `NSPrivacyTracking: false` — the app does **not** track users.
- `NSPrivacyCollectedDataTypes: []` — no data types are collected for tracking.
- No analytics SDKs (Firebase Analytics, Amplitude, Mixpanel, etc.) are included.
- No App Tracking Transparency (ATT) prompt is needed.
### 2.6 Encryption Declaration ✅
- `ITSAppUsesNonExemptEncryption: false` — the app uses only standard HTTPS/TLS for network communication, which is exempt from export compliance requirements.
### 2.7 Data Storage Security ✅
- API tokens are stored in the device keychain via `expo-secure-store` (uses iOS Keychain Services).
- No sensitive data is stored in `AsyncStorage` or `UserDefaults`.
- Server URL is stored in secure storage, not in plain text files.
---
## 3. App Store Review Guidelines Compliance
### 3.1 Functionality (Guideline 2.x) ✅
- **2.1 App Completeness:** The app provides a complete, functional experience. All advertised features (camera capture, file upload, document list, push notifications) work as described.
- **2.3 Accurate Metadata:** App name ("DocuElevate"), description, and screenshots should accurately reflect the app's functionality.
- **2.5.4 Background Modes:** Only `remote-notification` is declared, which is actively used. ✅ Fixed.
### 3.2 Content & Intellectual Property (Guideline 3.x) ✅
- No third-party trademarked content is used.
- The app does not display user-generated content publicly (documents are private to each user).
- No copyrighted content is bundled with the app.
### 3.3 Business (Guideline 3.1.x) ✅
- The app does not include in-app purchases, subscriptions, or payment processing.
- No physical goods or services are sold through the app.
- Authentication is handled via self-hosted or enterprise SSO — no Apple Sign-In requirement applies (Apple Sign-In is required only when third-party social login options like Google/Facebook are offered as the primary login method; enterprise SSO to a self-hosted server is exempt).
### 3.4 Safety & Privacy (Guideline 5.x) ✅
- **5.1.1 Data Collection and Storage:** The app collects only what is necessary for its functionality (server URL, auth token, push token).
- **5.1.1(v) Account Deletion:** ✅ Fixed. Users can now initiate account deletion from the Profile screen, which opens the server's account deletion page in the browser.
- **5.1.2 Data Use and Sharing:** No data is shared with third parties or used for advertising.
### 3.5 Privacy Policy ✅ (Fixed)
- **Requirement:** Apple requires all apps to have an accessible privacy policy.
- **Fix:** Privacy Policy and Terms of Service links are now accessible from the Profile screen, opening the server's hosted policy pages.
- **App Store Connect:** The privacy policy URL must also be provided in App Store Connect during submission.
### 3.6 Login & Authentication ✅
- Two login methods are available: SSO (browser-based OAuth) and QR code scanning.
- Both methods provide clear error messages on failure.
- The app correctly handles authentication cancellation.
- Session restoration on app launch is implemented.
- **Demo Account:** For App Review, a demo account may need to be provided in App Store Connect's review notes. Ensure the review team can access a test server.
---
## 4. Technical Compliance
### 4.1 API Usage ✅
- No private APIs are used (all functionality comes from Expo SDK and React Native public APIs).
- No deprecated APIs are used that would trigger rejection.
### 4.2 Network Security ✅
- The app validates server URLs require `http://` or `https://` scheme.
- All API calls use Bearer token authentication over HTTPS.
- App Transport Security (ATS) is not explicitly disabled — default iOS ATS rules apply.
### 4.3 Deep Linking ✅
- Custom URL scheme `docuelevate://` is properly registered.
- Deep link handling for QR login (`docuelevate://qr-login`) and file sharing is implemented correctly.
- `WebBrowser.openAuthSessionAsync` is used for OAuth, which properly handles the authentication session lifecycle.
### 4.4 Document Handling ✅
- `CFBundleDocumentTypes` properly declares supported file types.
- `LSSupportsOpeningDocumentsInPlace: false` ensures iOS copies shared files to the app's accessible Inbox directory, avoiding security-scoped URL issues.
- The `+not-found.tsx` handler correctly intercepts iOS "Open In…" file paths.
- `UploadScreen` uses `expo-file-system` to copy external files to cache before uploading for reliable file access.
### 4.5 Crash Resistance ✅
- All network calls are wrapped in try/catch blocks.
- Error states are displayed to users with actionable recovery options (retry buttons).
- Permission denials are handled gracefully with explanatory messages.
---
## 5. Onboarding & First-Run Experience
### 5.1 Welcome Screen ✅
- Clean, informative welcome screen with app branding and feature highlights.
- Clear "Get Started" call-to-action leading to the login screen.
- No misleading claims or functionality promises.
### 5.2 Login Flow ✅
- Server URL entry with input validation.
- Two clear authentication options (SSO and QR code).
- Error handling with user-friendly alert dialogs.
- Back navigation available from all auth screens.
### 5.3 First-Run Permissions ✅
- Camera permission is requested at the point of use (when tapping Camera button), not at launch.
- Photo library permission is requested at the point of use.
- Push notification permission is requested after authentication, not before.
- All permission requests include clear usage descriptions.
---
## 6. Remaining Recommendations
### 6.1 App Store Connect Preparation
Before submission, ensure the following are configured in App Store Connect:
- [ ] **Privacy Policy URL** — must point to the server's `/privacy` endpoint
- [ ] **App Store description** — accurate description of features
- [ ] **Screenshots** — for iPhone and iPad (since `supportsTablet: true`)
- [ ] **App category** — "Business" or "Productivity"
- [ ] **Age rating** — complete the questionnaire (likely 4+)
- [ ] **Review notes** — provide demo server URL and test credentials for the Apple review team
- [ ] **Privacy Nutrition Labels** — declare data types collected (device ID for push notifications, authentication tokens)
### 6.2 Accessibility Enhancements (Recommended)
While the app includes `accessibilityRole` and `accessibilityLabel` on interactive elements, consider:
- Adding `accessibilityHint` to buttons where the action isn't immediately obvious.
- Testing with VoiceOver to ensure all screens are fully navigable.
- Ensuring all status changes are announced to screen readers.
### 6.3 iPad Support
The app declares `supportsTablet: true`. Ensure:
- UI scales appropriately on iPad screen sizes.
- Split View and Slide Over multitasking work correctly.
- Touch targets remain accessible on larger screens.
### 6.4 Localization (Future Enhancement)
- The app currently uses English-only strings.
- For broader App Store reach, consider localizing the app name, description, and in-app strings.
---
## 7. Compliance Checklist Summary
| Area | Status | Notes |
|------|--------|-------|
| Human Interface Guidelines | ✅ Pass | Ionicons used for platform-consistent iconography |
| App Icons & Visual Assets | ✅ Pass | All required assets provided |
| Device Data Usage | ✅ Pass | Camera, photos, notifications properly handled |
| Privacy Disclosures | ✅ Pass | Info.plist keys and privacy manifest configured |
| Background Modes | ✅ Pass | Only `remote-notification` declared |
| Restricted APIs | ✅ Pass | No private or deprecated APIs used |
| Content Standards | ✅ Pass | No misleading or inappropriate content |
| Functionality | ✅ Pass | Complete, functional app experience |
| Business Model | ✅ Pass | No IAP conflicts |
| Safety & Privacy | ✅ Pass | Account deletion available, privacy policy linked |
| Onboarding | ✅ Pass | Clear, permission-respectful first-run experience |
| Privacy Manifest | ✅ Pass | Required reason APIs declared |
---
## References
- [Apple App Store Review Guidelines](https://developer.apple.com/app-store/review/guidelines/)
- [Apple Human Interface Guidelines](https://developer.apple.com/design/human-interface-guidelines/)
- [Apple Privacy Manifest Requirements](https://developer.apple.com/documentation/bundleresources/privacy_manifest_files)
- [App Store Connect Help](https://developer.apple.com/help/app-store-connect/)
+45 -2
View File
@@ -11,10 +11,15 @@ Configuration is primarily done through environment variables specified in a `.e
| **Variable** | **Description** | **Example** |
|------------------------|----------------------------------------------------------|--------------------------------|
| `DATABASE_URL` | Path/URL to the SQLite database (or other SQL backend). Use the [Database Wizard](/database-wizard) for guided setup. See [Database Configuration](DatabaseConfiguration.md). | `sqlite:///./app/database.db` |
| `DB_POOL_SIZE` | Number of persistent connections in the pool per worker (PostgreSQL/MySQL only; ignored for SQLite). | `10` |
| `DB_MAX_OVERFLOW` | Additional connections beyond `DB_POOL_SIZE` under burst load (PostgreSQL/MySQL only). | `20` |
| `DB_POOL_TIMEOUT` | Seconds to wait for a pool connection before raising `TimeoutError` (PostgreSQL/MySQL only). | `30` |
| `DB_POOL_RECYCLE` | Recycle connections after this many seconds to avoid stale connections (PostgreSQL/MySQL only). | `1800` |
| `REDIS_URL` | URL for Redis, used by Celery for broker & result store. | `redis://redis:6379/0` |
| `WORKDIR` | Working directory for the application. | `/workdir` |
| `GOTENBERG_URL` | Gotenberg PDF processing URL. | `http://gotenberg:3000` |
| `EXTERNAL_HOSTNAME` | The external hostname for the application. | `docuelevate.example.com` |
| `PUBLIC_BASE_URL` | Full public base URL including scheme (e.g., `https://docuelevate.example.com`). When set, overrides auto-detected URLs used for OAuth redirect URIs. **Required when your reverse proxy does not forward `X-Forwarded-Proto` headers.** | *(not set)* |
| `ALLOW_FILE_DELETE` | Enable file deletion in the web interface (`true`/`false`). | `true` |
| `COMPLIANCE_ENABLED` | Enable the compliance templates dashboard (GDPR, HIPAA, SOC 2). | `true` |
| `FACTORY_RESET_ON_STARTUP` | Wipe all user data on every startup (demo/testing). | `false` |
@@ -81,6 +86,28 @@ Control how the web UI queues and paces file uploads to avoid overwhelming the b
**Example**: With `UPLOAD_CONCURRENCY=3` and `UPLOAD_QUEUE_DELAY_MS=500`, a directory of 5,000 files is uploaded ≈ 3 at a time with 500 ms pacing the backend processes files at its own rate while the queue drains in the background without triggering API rate limits.
### Per-User Upload Rate Limiting
Server-side rate limiting that prevents any single user from overwhelming the system with bulk uploads. The limiter uses a Redis-backed sliding window and dynamically adjusts limits based on system health.
| **Variable** | **Description** | **Default** |
|--------------------------------|------------------------------------------------------------------------------------------------------------------------------|-------------|
| `UPLOAD_RATE_LIMIT_PER_USER` | Maximum uploads allowed per user within the sliding window. Effective limit may be reduced under load. | `20` |
| `UPLOAD_RATE_LIMIT_WINDOW` | Sliding window size in seconds. | `60` |
**Health-aware dynamic limiting**: The effective per-user limit is automatically reduced when the system is under heavy load:
| **System condition** | **Effective limit** | **Trigger** |
|--------------------------------|---------------------|--------------------------------|
| Normal | 100 % of base | Queue < 50, CPU load normal |
| Moderate load | 50 % of base | Queue 50100 or CPU > 1.5× |
| High load | 25 % of base | Queue 100200 or CPU > 2× |
| Critical load | 10 % of base | Queue > 200 or CPU > 3× |
When a user exceeds the limit, the server returns **HTTP 429 Too Many Requests** with a `Retry-After` header. The browser client (see *Client-Side Upload Throttling* above) automatically pauses and retries.
> **Note**: The limiter fails open — if Redis is unavailable, all uploads are allowed through so that a monitoring outage never blocks document processing.
### File Upload Size Limits
**Security Feature**: Control file upload sizes to prevent resource exhaustion attacks. See [SECURITY_AUDIT.md](../SECURITY_AUDIT.md#5-file-upload-size-limits) for security details.
@@ -1555,14 +1582,30 @@ DocuElevate detects and flags documents that share the same content, even if the
### Exact Duplicate Detection (SHA-256)
When `ENABLE_DEDUPLICATION=True` (the default), each new document is hashed with SHA-256 before processing begins. If the hash matches an existing file record the new document is stored as a duplicate (`is_duplicate=True`, `duplicate_of_id=<original_id>`) and no further processing is performed.
When `ENABLE_DEDUPLICATION=True` (the default), each new document is hashed with SHA-256 before processing begins. If the hash matches an existing file record the upload is rejected immediately — no processing task is created, and the temporary file is removed from disk. The `/api/ui-upload` response returns `"status": "duplicate"` together with a `duplicate_of` object that identifies the original file.
If the same file somehow reaches the Celery worker (e.g. via a watch-folder ingest) it is still caught there and stored as a duplicate (`is_duplicate=True`, `duplicate_of_id=<original_id>`) with no further processing.
| Variable | Description | Default |
|---|---|---|
| `ENABLE_DEDUPLICATION` | Hash-based exact duplicate detection on ingest. | `True` |
| `SHOW_DEDUPLICATION_STEP` | Show the "Check for Duplicates" step in the processing timeline UI. | `True` |
An immediate duplicate warning is also included in the `/api/ui-upload` JSON response so the frontend can alert the user before the pipeline completes.
When the upload is an exact duplicate the `/api/ui-upload` response looks like:
```json
{
"status": "duplicate",
"original_filename": "invoice.pdf",
"stored_filename": "abc-123.pdf",
"duplicate_of": {
"duplicate_type": "exact",
"original_file_id": 42,
"original_filename": "invoice.pdf",
"message": "This file is an exact duplicate of an already-processed document. It has not been queued for processing again."
}
}
```
### Near-Duplicate Detection (Content Similarity)
+25 -8
View File
@@ -337,18 +337,26 @@ The Helm chart includes a pre-install and pre-upgrade Job hook that runs `alembi
## Connection Pooling
SQLAlchemy manages a connection pool automatically. The defaults are suitable for most deployments. For high-concurrency or Kubernetes deployments you may want to tune:
SQLAlchemy manages a connection pool automatically. DocuElevate selects the pool
strategy based on the database backend:
- **SQLite** — uses `NullPool` (a fresh connection per request, closed immediately).
This avoids the `QueuePool limit reached` `TimeoutError` that can occur under
concurrent load because SQLite does not benefit from persistent connection pooling.
- **PostgreSQL / MySQL** — uses a bounded `QueuePool` whose size is configurable
via environment variables.
```bash
# Optional — these are set via environment variables if you extend app/database.py
# Typical production values:
DB_POOL_SIZE=10 # Number of persistent connections per worker
DB_MAX_OVERFLOW=20 # Additional connections allowed beyond pool_size
DB_POOL_TIMEOUT=30 # Seconds to wait for a connection from the pool
DB_POOL_RECYCLE=1800 # Recycle connections after 30 minutes (avoids stale connections)
# Tune these for PostgreSQL / MySQL (ignored when using SQLite):
DB_POOL_SIZE=10 # Number of persistent connections per worker (default: 10)
DB_MAX_OVERFLOW=20 # Additional connections allowed beyond pool_size (default: 20)
DB_POOL_TIMEOUT=30 # Seconds to wait for a connection from the pool (default: 30)
DB_POOL_RECYCLE=1800 # Recycle connections after 30 minutes (default: 1800)
```
> **Note:** These environment variables are not exposed in the default `app/config.py`. If you need to tune them, extend the database engine creation in `app/database.py`.
All backends also enable `pool_pre_ping`, which sends a lightweight health-check
before each connection is handed out. This detects stale or dropped connections
and transparently reconnects.
For **PgBouncer** (external connection pooling), point `DATABASE_URL` at your PgBouncer instance and use transaction-mode pooling:
@@ -512,4 +520,13 @@ Then retry `alembic upgrade head`.
Either increase `max_connections` in `postgresql.conf` or add PgBouncer in front of PostgreSQL. The default PostgreSQL `max_connections` is `100`; reduce `DB_POOL_SIZE` per worker to stay within this limit.
### "QueuePool limit reached" TimeoutError (SQLite)
If you see `TimeoutError: QueuePool limit of size 5 overflow 10 reached`, your
deployment is still running an older version of DocuElevate that used a bounded
connection pool for SQLite. Upgrade to the latest release — SQLite now uses
`NullPool`, which eliminates this error entirely. If you are already on the
latest version and are still seeing pool exhaustion, ensure you are not
overriding the engine creation manually.
For more help, see the [Troubleshooting Guide](Troubleshooting.md).
+10 -6
View File
@@ -349,16 +349,18 @@ workdir:
## Scaling
DocuElevate is designed for horizontal scaling. Both API and worker pods are stateless and can be scaled independently.
### Docker Compose
Add more worker containers:
Scale workers (task processing) and API pods (request handling) independently:
```yaml
worker:
deploy:
replicas: 3
```bash
docker compose up -d --scale worker=3 --scale api=2
```
> **Note:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. Do not scale it. It publishes periodic tasks to the Redis broker; workers pick them up.
### Kubernetes / Helm
Enable HPA:
@@ -377,13 +379,15 @@ worker:
maxReplicas: 10
```
The Helm chart deploys a separate **beat** pod (always 1 replica, `Recreate` strategy) so that scheduled tasks are never duplicated when workers scale.
---
## Monitoring
- **Docker Compose**: `docker-compose logs -f`, `docker stats`
- **Kubernetes**: `kubectl logs -l app.kubernetes.io/component=api -f`
- **Prometheus / Grafana**: Scrape the `/api/health` endpoint for readiness; add custom metrics as needed.
- **Prometheus / Grafana**: Scrape the `/api/diagnostic/healthz/ready` endpoint for readiness; add custom metrics as needed.
- **Uptime Kuma**: Set `UPTIME_KUMA_URL` to your push URL for heartbeat monitoring.
---
+25 -1
View File
@@ -128,7 +128,31 @@ If you encounter issues with Dropbox integration:
1. **Authentication Errors**: Make sure your App Key and App Secret are correct
2. **Token Expired**: Click "Refresh Token" button on the setup page to obtain a new token
3. **Folder Permissions**: Ensure your app has the correct permissions enabled for file operations
4. **Invalid Redirect URI**: Verify that the redirect URI in your app settings matches the one used in the authentication flow
4. **Invalid Redirect URI**: See section below for the most common cause and fix.
5. **Rate Limiting**: Dropbox API has rate limits; if exceeded, wait and try again
### Fixing "Invalid redirect_uri" Error
This error appears on the Dropbox authorization page when the redirect URI in the OAuth request does not match any URI registered in your Dropbox app console.
**Most common cause**: The application is deployed behind a reverse proxy (Traefik, Nginx, Caddy) that does **not** forward the `X-Forwarded-Proto: https` header to DocuElevate. Without this header, the server cannot determine that it is being accessed over HTTPS and may construct an `http://` redirect URI, while the registered URI in Dropbox is `https://`.
**Fix**:
Option 1 Configure your proxy to forward `X-Forwarded-Proto`:
```nginx
proxy_set_header X-Forwarded-Proto $scheme;
```
Option 2 Set `PUBLIC_BASE_URL` in your environment (recommended for most deployments):
```bash
PUBLIC_BASE_URL=https://docuelevate.example.com
```
When `PUBLIC_BASE_URL` is set, DocuElevate uses it directly for all OAuth redirect URIs instead of trying to infer the scheme from request headers. This is the most reliable option.
After setting `PUBLIC_BASE_URL`, ensure the Dropbox app console redirect URI matches exactly (e.g., `https://docuelevate.example.com/dropbox-callback`). The setup wizard at `/dropbox-setup` will show you the exact URI to register.
For more general configuration issues, see the [Configuration Troubleshooting Guide](ConfigurationTroubleshooting.md).
+13 -5
View File
@@ -373,6 +373,8 @@ worker:
replicaCount: 4
```
> **Beat scheduler:** The Helm chart deploys a dedicated `beat` pod (always exactly 1 replica with `Recreate` strategy) that publishes periodic tasks to the Redis broker. Workers consume these tasks — scaling workers does **not** duplicate scheduled jobs.
### Horizontal Pod Autoscaler
```yaml
@@ -433,24 +435,30 @@ externalRedis:
### Kubernetes Probes
The Helm chart configures liveness and readiness probes on the API pods via `/api/health`. Default settings:
The Helm chart configures **unauthenticated** liveness and readiness probes on the API pods so kubelet can reach them without credentials. Default settings:
```yaml
api:
livenessProbe:
httpGet:
path: /api/health
path: /api/diagnostic/healthz/live
port: 8000
initialDelaySeconds: 30
periodSeconds: 30
periodSeconds: 20
readinessProbe:
httpGet:
path: /api/health
path: /api/diagnostic/healthz/ready
port: 8000
initialDelaySeconds: 10
initialDelaySeconds: 15
periodSeconds: 10
```
| Endpoint | Auth | Purpose |
|----------|------|---------|
| `/api/diagnostic/healthz/live` | None | Lightweight liveness check — returns 200 if the process is running |
| `/api/diagnostic/healthz/ready` | None | Readiness check — verifies database and Redis connectivity (503 when DB is down) |
| `/api/diagnostic/health` | Required | Full health status for monitoring dashboards (Grafana, Uptime Kuma) |
### Prometheus Scraping
Add annotations to expose metrics (if using a Prometheus-compatible exporter):
+127 -8
View File
@@ -12,9 +12,14 @@ DocuElevate includes a native mobile application for iOS and Android built with
| Auto-generated API token | ✅ | ✅ |
| Camera capture → upload | ✅ | ✅ |
| File picker upload | ✅ | ✅ |
| Multi-image selection from library | ✅ | ✅ |
| Share Sheet / Share Intent | ✅ | ✅ |
| Push notifications | ✅ | ✅ |
| Document list | ✅ | ✅ |
| Document list with search | ✅ | ✅ |
| File detail view with processing logs | ✅ | ✅ |
| Pre-login legal pages (GDPR) | ✅ | ✅ |
| Localization (EN, DE, ES, FR, IT) | ✅ | ✅ |
| Language selection | ✅ | ✅ |
| Dark mode | ✅ | ✅ |
## Getting Started (Development)
@@ -171,8 +176,8 @@ curl -X DELETE -H "Authorization: Bearer <token>" https://your-server/api/mobile
1. Open the **Upload** tab.
2. Tap **Photos**.
3. Select an existing photo from the device's photo library.
4. The image is uploaded and queued for processing.
3. Select one or more photos from the device's photo library (multi-selection is supported).
4. All selected images are uploaded and queued for processing.
### File Picker
@@ -198,6 +203,32 @@ The app registers itself as a share target so any file can be sent directly to D
The URL may arrive as a standard `file://` path **or** under the app's custom `docuelevate://` scheme (e.g. `docuelevate://private/var/mobile/Library/…/file.pdf`). The root layout detects the custom-scheme form and rewrites it to a `file://` URL before forwarding it to the Upload screen through `ShareContext`.
##### Handling "unmatched route" errors from "Open In…"
iOS sometimes delivers the file path under the `docuelevate://` scheme, e.g.:
```
docuelevate://private/var/mobile/Library/Mobile Documents/…/Invoice.pdf
```
expo-router strips the scheme and tries to match `/private/var/mobile/…` as an in-app route. Because no such route exists, it previously threw an **"unmatched route docuelevate://"** error and the upload never completed.
The fix is a catch-all `+not-found.tsx` route (see `mobile/app/+not-found.tsx`). When expo-router cannot match the path, it renders this screen instead. The screen detects that the path is a filesystem path rather than a real in-app route, adds the file directly to `ShareContext`, and redirects to the Upload tab. `UploadScreen` picks up the pending file and begins uploading automatically. The `Linking` listener in the root layout may also fire for the same URL; `ShareContext.addPendingFile` deduplicates by URI so the file is only uploaded once.
##### File accessibility and local caching
Shared files may reference paths outside the app's sandbox or use security-scoped URLs that React Native's `fetch` cannot read directly. To guarantee reliable uploads:
- **`LSSupportsOpeningDocumentsInPlace`** is set to `false` in `app.json`, which tells iOS to copy shared files into the app's `Documents/Inbox` directory before handing them to the app.
- **`UploadScreen`** uses `expo-file-system` (`FileSystem.copyAsync`) to copy any `file://` URI that is outside the app's cache/documents directory to a local cache path before uploading. This ensures the file is readable regardless of its origin.
- **MIME type inference**: Both `+not-found.tsx` and the `Linking` handler in `_layout.tsx` infer the MIME type from the file extension (e.g. `.pdf``application/pdf`) so the server receives a correct `Content-Type` instead of `application/octet-stream`.
##### iOS Action / Share Extension (future enhancement)
Apps like DeepL ("Translate in DeepL") and Microsoft Word ("Convert to Word") appear as **Action Extensions** in the iOS share sheet — a system-level feature that requires a separate Xcode target built with Swift or Objective-C. A proper Action Extension runs in its own process and must share authentication credentials with the main app via an iOS **App Group** (shared keychain / shared container).
This level of iOS-native integration is a planned future enhancement. Until it is available, the recommended workflow is the current one: tap **Share → DocuElevate** (the app appears in the "Open With" row of the share sheet via `CFBundleDocumentTypes`).
#### Android implementation
`app.json` declares `ACTION_SEND` and `ACTION_SEND_MULTIPLE` intent filters for `mimeType: "*/*"` in the `android.intentFilters` section. Incoming content URIs are received the same way as on iOS.
@@ -215,6 +246,75 @@ If a file upload fails (e.g. due to network issues or a server error), the faile
The retry re-uses the original file URI so no re-selection is needed.
## Document Search
The **Files** tab includes a search bar at the top that lets users search through their processed documents by filename. Searches are debounced (400ms) to avoid excessive API calls. Clear the search with the ✕ button to return to the full list.
## File Detail View
Tapping any document in the **Files** tab opens a detail view showing:
- **File metadata**: filename, file size, MIME type, upload date, and file hash
- **Processing status**: current status with a colour-coded icon
- **Processing log**: chronological list of processing steps with individual status indicators and timestamps
Pull-to-refresh updates the detail view. This replicates the web interface at `/files/{id}` and `/files/{id}/detail` in a mobile-friendly layout.
## Legal & Compliance
### GDPR & Apple App Store Compliance
Privacy Policy, Terms of Service, and Imprint links are accessible **before login** from both the **Welcome Screen** and the **Login Screen**. This ensures compliance with:
- **GDPR** (General Data Protection Regulation) users must be able to review the privacy policy before providing personal data
- **Apple App Store Review Guidelines** apps must provide accessible privacy information before account creation
Post-login, the same links are available in the **Profile** tab under the "Legal" section.
## Localization (i18n)
The mobile app supports five languages with automatic device-locale detection:
| Language | Code | Status |
|----------|------|--------|
| English | `en` | ✅ Complete |
| German (Deutsch) | `de` | ✅ Complete |
| Spanish (Español) | `es` | ✅ Complete |
| French (Français) | `fr` | ✅ Complete |
| Italian (Italiano) | `it` | ✅ Complete |
### How it works
Language priority (highest to lowest):
1. **Server preference**`preferred_language` returned by `GET /api/mobile/whoami` on login or app resume. Allows a language set on the desktop web interface to propagate to mobile automatically.
2. **AsyncStorage** — the last language explicitly selected on the device, used as an offline fallback when the server is unreachable.
3. **Device locale** — detected via `expo-localization` on first launch.
4. **English** — final fallback when none of the above match a supported locale.
When a user selects a language on mobile the choice is:
- Applied immediately to all screens (via `LocaleContext`)
- Persisted locally to AsyncStorage
- Synced to the server via `POST /api/i18n/language` (fire-and-forget), so the next desktop login reflects the same preference.
> **Note**: If the server's preferred language is not supported by the mobile app (e.g. a locale added to the web frontend but not yet translated for mobile), the mobile app falls back to the next priority in the list above.
### Adding a new language
1. Create a new translation file in `mobile/src/i18n/` (e.g. `pt.json` for Portuguese)
2. Copy the structure from `en.json` and translate all values
3. Import the new file in `mobile/src/i18n/index.ts`
4. Add it to the `translations` object and `getSupportedLanguages()` array
## User Settings
The **Profile** tab includes a **Settings** section where users can:
- **Change language**: Select from the supported languages (English, German, Spanish, French, Italian)
- View server connection details
- Access legal documents (Privacy Policy, Terms of Service, Imprint)
- Sign out or delete their account
## Mobile API Endpoints
The backend exposes a dedicated `/api/mobile/` namespace:
@@ -225,7 +325,8 @@ The backend exposes a dedicated `/api/mobile/` namespace:
| `POST` | `/api/mobile/register-device` | Bearer | Register Expo push token |
| `GET` | `/api/mobile/devices` | Bearer | List registered devices |
| `DELETE` | `/api/mobile/devices/{id}` | Bearer | Deactivate a device |
| `GET` | `/api/mobile/whoami` | Bearer | Get current user profile |
| `GET` | `/api/mobile/whoami` | Bearer | Get current user profile (includes `preferred_language`) |
| `POST` | `/api/i18n/language` | Bearer | Sync language preference to server |
All other API endpoints (file upload, file listing, etc.) work with Bearer token authentication.
@@ -269,7 +370,7 @@ Re-registering the same token is safe (idempotent).
### GET /api/mobile/whoami
Returns the current user's profile.
Returns the current user's profile, including the server-stored language preference.
**Response (200):**
```json
@@ -278,10 +379,15 @@ Returns the current user's profile.
"display_name": "John Doe",
"email": "john@example.com",
"avatar_url": "https://www.gravatar.com/avatar/...",
"is_admin": false
"is_admin": false,
"preferred_language": "de"
}
```
`preferred_language` is `null` when no preference has been saved. The mobile
app applies this value on login / app resume, falling back to AsyncStorage and
then the device locale when it is `null` or unsupported.
## Configuration
No server-side configuration is required to enable the mobile app. The Expo push notification routing does not need FCM or APNs credentials on the server.
@@ -320,8 +426,20 @@ mobile/
│ ├── LoginScreen.tsx # Server URL + SSO button + QR code scanner
│ ├── QRScannerScreen.tsx # Camera-based QR code scanner for login
│ ├── UploadScreen.tsx # Camera capture + photo library + file picker
│ ├── FilesScreen.tsx # Processed document list
── ProfileScreen.tsx # User profile + sign out
│ ├── FilesScreen.tsx # Processed document list with search
── FileDetailScreen.tsx # File detail view with processing logs
│ ├── ProfileScreen.tsx # User profile + settings + sign out
│ └── WelcomeScreen.tsx # Pre-login welcome with legal links
├── i18n/ # Localization (i18n)
│ ├── index.ts # i18n module (locale detection, t() function)
│ ├── en.json # English translations
│ ├── de.json # German translations
│ ├── es.json # Spanish translations
│ ├── fr.json # French translations
│ └── it.json # Italian translations
├── utils/
│ ├── mimeTypes.ts # MIME type mapping for file extensions
│ └── normalizeUri.ts # URI normalization for deduplication
└── services/
└── api.ts # DocuElevate REST API client
```
@@ -408,3 +526,4 @@ eas build --platform ios
- [API Documentation](./API.md)
- [Configuration Guide](./ConfigurationGuide.md)
- [Deployment Guide](./DeploymentGuide.md)
- [Apple App Store Compliance Audit](./AppleAppStoreCompliance.md)
+38 -15
View File
@@ -34,7 +34,7 @@ Use this checklist to track readiness before going live.
- [ ] **Redis** — Running and accessible only from internal network
- [ ] **Meilisearch** — Running and accessible only from internal network
- [ ] **Worker replicas** — At least 2 workers configured for redundancy
- [ ] **Monitoring**`/api/health` polled by uptime checker
- [ ] **Monitoring**`/api/diagnostic/health` polled by uptime checker
- [ ] **Backups** — Automated backup of database, workdir, and Meilisearch data
- [ ] **Log retention** — Logs shipped to a persistent store or aggregator
- [ ] **Secrets management** — API keys not committed to source control
@@ -285,22 +285,24 @@ For SSO/OIDC (Authentik, Keycloak, Auth0, etc.) see the [Authentication Setup Gu
### Docker Compose
Use the `deploy.replicas` setting (requires Docker Swarm mode) or simply run multiple workers:
```yaml
worker:
deploy:
replicas: 3
```
Or scale after deployment:
Scale workers independently:
```bash
docker-compose up -d --scale worker=3
docker compose up -d --scale worker=3
```
Each worker processes tasks from the Celery queue independently. Ensure the shared `workdir` volume is accessible from all worker containers.
> **Important:** The `beat` service (Celery Beat scheduler) must always run as exactly **one** instance. It is defined as a dedicated service in `docker-compose.yaml` with a fixed `container_name`. Do not scale it.
### Scaling the API
API pods are fully stateless (sessions use encrypted cookies, not server-side state) and can be scaled behind a load balancer:
```bash
docker compose up -d --scale api=3
```
### Kubernetes (Helm)
```yaml
@@ -339,11 +341,32 @@ celery -A app.celery_worker worker -Q default,celery --concurrency=2
### Health Check Endpoint
DocuElevate exposes `/api/health` for readiness probing. Configure your uptime monitor to poll this endpoint:
DocuElevate exposes three health-related endpoints:
| Endpoint | Auth | Purpose |
|----------|------|---------|
| `GET /api/diagnostic/healthz/live` | None | Lightweight liveness probe — returns 200 if the process is running |
| `GET /api/diagnostic/healthz/ready` | None | Readiness probe — checks database and Redis (503 when DB is down) |
| `GET /api/diagnostic/health` | Required | Full status for monitoring dashboards (Grafana, Uptime Kuma) |
For **Kubernetes probes**, use the unauthenticated endpoints:
```yaml
livenessProbe:
httpGet:
path: /api/diagnostic/healthz/live
port: 8000
readinessProbe:
httpGet:
path: /api/diagnostic/healthz/ready
port: 8000
```
For **uptime monitors** (Uptime Kuma, Grafana, etc.), use the authenticated endpoint:
```bash
curl http://docuelevate.example.com/api/health
# Expected: {"status": "ok", ...}
curl http://docuelevate.example.com/api/diagnostic/health
# Expected: {"status": "healthy", ...}
```
Set `UPTIME_KUMA_URL` to your Uptime Kuma push URL for heartbeat monitoring:
@@ -502,4 +525,4 @@ For a dedicated Kubernetes deployment guide, including architecture diagrams, PV
- **Image Pull Policy**: Use `IfNotPresent` in production with pinned image tags (not `latest`) for reproducible deployments.
- **Liveness & Readiness Probes**: Already configured in the Helm chart via `/api/health`. Verify they are tuned to your startup time.
- **Liveness & Readiness Probes**: Already configured in the Helm chart via unauthenticated endpoints (`/api/diagnostic/healthz/live` and `/api/diagnostic/healthz/ready`). Verify they are tuned to your startup time.
+1 -1
View File
@@ -87,7 +87,7 @@ DocuElevate provides multiple convenient ways to upload documents to the system.
#### Supported File Types
- **Documents**: PDF, Word (.doc, .docx), Excel (.xls, .xlsx), PowerPoint (.ppt, .pptx)
- **Images**: JPEG, PNG, GIF, BMP, TIFF, WebP, SVG
- **Images**: JPEG, PNG, GIF, BMP, TIFF, WebP, SVG, HEIC, HEIF
- **Text**: Plain text (.txt), CSV, RTF, HTML, XML, Markdown
- **Maximum file size**: 500MB per file