Compare commits

..

1 Commits

Author SHA1 Message Date
google-labs-jules[bot] 16de54f66b 🧹 [code health] Remove unused TokenRevokeRequest model
🎯 What: The code health issue addressed (removing unused TokenRevokeRequest model) was already resolved in the codebase.
💡 Why: How this improves maintainability (removing dead code).
 Verification: I confirmed by searching the repository and inspecting the files that the code does not exist.
 Result: The codebase is already in the desired state.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-23 14:27:10 +00:00
119 changed files with 1084 additions and 3471 deletions
+31 -3
View File
@@ -3,10 +3,21 @@ WORKDIR=/workdir
DATABASE_URL=sqlite:///./app/database.db DATABASE_URL=sqlite:///./app/database.db
REDIS_URL=redis://redis:6379/0 REDIS_URL=redis://redis:6379/0
EXTERNAL_HOSTNAME=docuelevate.example.com EXTERNAL_HOSTNAME=docuelevate.example.com
# PUBLIC_BASE_URL=https://docuelevate.example.com # Full URL with scheme; required when X-Forwarded-Proto is not forwarded by your proxy
GOTENBERG_URL=http://gotenberg:3000 GOTENBERG_URL=http://gotenberg:3000
ALLOW_FILE_DELETE=true # Allow deletion of file records ALLOW_FILE_DELETE=true # Allow deletion of file records
COMPLIANCE_ENABLED=true # Enable compliance templates dashboard (GDPR, HIPAA, SOC 2) COMPLIANCE_ENABLED=true # Enable compliance templates dashboard (GDPR, HIPAA, SOC 2)
# **Database Connection Pool** (PostgreSQL / MySQL only; ignored for SQLite)
# DB_POOL_SIZE=10 # Persistent connections per worker (default: 10)
# DB_MAX_OVERFLOW=20 # Extra connections under burst (default: 20)
# DB_POOL_TIMEOUT=30 # Seconds to wait for a pool connection (default: 30)
# DB_POOL_RECYCLE=1800 # Recycle connections after N seconds (default: 1800)
# **Per-User Upload Rate Limiting** (health-aware, Redis-backed)
# UPLOAD_RATE_LIMIT_PER_USER=20 # Max uploads per user per window (default: 20)
# UPLOAD_RATE_LIMIT_WINDOW=60 # Sliding window in seconds (default: 60)
# **System Reset / Factory Reset** # **System Reset / Factory Reset**
# FACTORY_RESET_ON_STARTUP=false # Wipe all user data on every startup (demo/testing only) # FACTORY_RESET_ON_STARTUP=false # Wipe all user data on every startup (demo/testing only)
# ENABLE_FACTORY_RESET=false # Show the System Reset page in admin UI # ENABLE_FACTORY_RESET=false # Show the System Reset page in admin UI
@@ -625,9 +636,26 @@ EMBEDDING_MAX_TOKENS=8000
# Profiling is only active when SENTRY_TRACES_SAMPLE_RATE > 0. Default: 0.0 (disabled). # Profiling is only active when SENTRY_TRACES_SAMPLE_RATE > 0. Default: 0.0 (disabled).
# SENTRY_PROFILES_SAMPLE_RATE=0.0 # SENTRY_PROFILES_SAMPLE_RATE=0.0
# #
# Attach PII (IP addresses, user agents) to Sentry events. # Attach PII (IP addresses, user agents) to Sentry events.
# Disable (default) to stay GDPR/CCPA compliant. # Disable (default) to stay GDPR/CCPA compliant.
# SENTRY_SEND_DEFAULT_PII=false # SENTRY_SEND_DEFAULT_PII=false
#
# --- Browser (JavaScript) SDK ---
# The same DSN is reused for the Sentry Browser SDK which is injected into
# every rendered page. The DSN is a *public* key and is intentionally
# embedded in client-side code.
#
# Fraction of browser navigations captured for client-side performance tracing.
# 0.0 (default) disables browser tracing; 1.0 captures every navigation.
# SENTRY_JS_TRACES_SAMPLE_RATE=0.0
#
# Fraction of browser sessions recorded by Sentry Session Replay.
# 0.0 (default) disables session recording; 1.0 records every session.
# SENTRY_JS_REPLAY_SESSION_SAMPLE_RATE=0.0
#
# Fraction of error sessions recorded by Sentry Session Replay.
# Defaults to 0.1 (10 %) so errors are captured with replay context.
# SENTRY_JS_REPLAY_ON_ERROR_SAMPLE_RATE=0.1
# **Mobile App Push Notifications** # **Mobile App Push Notifications**
# Push notifications are delivered via Expo's push notification service # Push notifications are delivered via Expo's push notification service
+1 -1
View File
@@ -100,7 +100,7 @@ jobs:
python-version: "3.11" python-version: "3.11"
cache: 'pip' cache: 'pip'
- run: pip install pip-audit>=2.7.0 - run: pip install pip-audit>=2.7.0
- run: pip-audit -r requirements.txt --desc on --ignore-vuln CVE-2026-4539 - run: pip-audit -r requirements.txt --desc on
run-tests: run-tests:
name: Execute All Tests (Quick + Integration) name: Execute All Tests (Quick + Integration)
+3 -2
View File
@@ -200,5 +200,6 @@ cython_debug/
# Build metadata files - generated at build time # Build metadata files - generated at build time
GIT_SHA GIT_SHA
RUNTIME_INFO RUNTIME_INFO
node_modules
frontend/node_modules # Frontend build tooling
frontend/node_modules/
+3
View File
@@ -0,0 +1,3 @@
[submodule "vendor/embed-pdf-viewer"]
path = vendor/embed-pdf-viewer
url = https://github.com/embedpdf/embed-pdf-viewer.git
+8 -4
View File
@@ -1,4 +1,8 @@
## 2026-06-01 - [Fix XSS in status_dashboard.html] ## 2024-05-24 - SSRF in WebDAV connection test
**Vulnerability:** A Cross-Site Scripting (XSS) vulnerability existed in `frontend/templates/status_dashboard.html` where untrusted configuration settings (`value`), external service messages (`data.message`), and token expirations (`data.token_info.expires_in_human`) were injected directly into the DOM via `.innerHTML` without sanitization. **Vulnerability:** The `_test_webdav_connection` function had a custom SSRF check that failed to resolve DNS names, allowing attackers to bypass the check by providing a domain that resolves to an internal IP (e.g., `127.0.0.1`).
**Learning:** Even internal or admin-focused dashboards can be vulnerable if they display external or user-configurable data without escaping. Constructing HTML strings dynamically from unvalidated sources is a common vector for DOM-based XSS. **Learning:** DNS resolution is required for robust SSRF protection when validating URLs provided by users.
**Prevention:** Always use a sanitization function like `escapeHtml` to escape dangerous characters (`<`, `>`, `&`, `"`, `'`) before assigning dynamic content to `.innerHTML`, or prefer `.textContent` when only plaintext is intended. **Prevention:** Use a centralized `is_private_ip` function (now in `app/utils/network.py`) that resolves the hostname to its IPs and checks if any are private.
## 2026-03-22 - B310: urllib.request.urlopen replaced with httpx
**Vulnerability:** The `_test_webdav_connection` function used `urllib.request.urlopen`, which natively supports dangerous schemes like `file://` or `ftp://` and follows redirects by default, potentially allowing SSRF bypasses or Local File Inclusion.
**Learning:** `urllib.request` should be avoided for user-supplied URLs. Even when URL schemes are manually validated, `urllib`'s default redirect following behavior can bypass SSRF protections (e.g. redirecting to `127.0.0.1`).
**Prevention:** Use a modern, safer HTTP client like `httpx` with `follow_redirects=False` when testing user-provided URLs.
+1 -1
View File
@@ -1 +1 @@
2026-06-01T03:41:15Z 2026-03-22T18:47:07Z
-593
View File
@@ -10,599 +10,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
<!-- version list --> <!-- version list -->
## v0.173.4 (2026-06-01)
### Bug Fixes
- Address status dashboard xss review nits
([`425805a`](https://github.com/christianlouis/DocuElevate/commit/425805ab23882944aee0cb02b5e497bc536549c0))
### Build System
- **deps**: Update redis requirement from >=4.5.0 to >=8.0.0
([#904](https://github.com/christianlouis/DocuElevate/pull/904),
[`20bde93`](https://github.com/christianlouis/DocuElevate/commit/20bde939eb96ec8e6700b206e046388b6b891e38))
- **deps-dev**: Update pytest-asyncio requirement
([#903](https://github.com/christianlouis/DocuElevate/pull/903),
[`4a12559`](https://github.com/christianlouis/DocuElevate/commit/4a125590b9b6cb2f07bae187a9bff708dd92a5c3))
### Documentation
- **changelog**: Update changelog [skip ci]
([`98252e0`](https://github.com/christianlouis/DocuElevate/commit/98252e06c30bba78520a8460d55f508dbf2bdd47))
- **changelog**: Update changelog [skip ci]
([`8be7965`](https://github.com/christianlouis/DocuElevate/commit/8be7965ed1892978092c3f5e02e6252918c237fd))
## Unreleased
### Build System
- **deps**: Update redis requirement from >=4.5.0 to >=8.0.0
([#904](https://github.com/christianlouis/DocuElevate/pull/904),
[`20bde93`](https://github.com/christianlouis/DocuElevate/commit/20bde939eb96ec8e6700b206e046388b6b891e38))
- **deps-dev**: Update pytest-asyncio requirement
([#903](https://github.com/christianlouis/DocuElevate/pull/903),
[`4a12559`](https://github.com/christianlouis/DocuElevate/commit/4a125590b9b6cb2f07bae187a9bff708dd92a5c3))
### Documentation
- **changelog**: Update changelog [skip ci]
([`8be7965`](https://github.com/christianlouis/DocuElevate/commit/8be7965ed1892978092c3f5e02e6252918c237fd))
## Unreleased
### Build System
- **deps-dev**: Update pytest-asyncio requirement
([#903](https://github.com/christianlouis/DocuElevate/pull/903),
[`4a12559`](https://github.com/christianlouis/DocuElevate/commit/4a125590b9b6cb2f07bae187a9bff708dd92a5c3))
## v0.173.3 (2026-05-31)
### Bug Fixes
- Preserve falsy values in escapeHtml
([`37c27c0`](https://github.com/christianlouis/DocuElevate/commit/37c27c02139ae4a462e1705bda9360b23eb835b3))
### Documentation
- **changelog**: Update changelog [skip ci]
([`cc56127`](https://github.com/christianlouis/DocuElevate/commit/cc561277c9d7d0177b4ac63921637659abb66fba))
- **changelog**: Update changelog [skip ci]
([`948d118`](https://github.com/christianlouis/DocuElevate/commit/948d118926be042cc3c2a68f58241cc2fcfa23ef))
- **changelog**: Update changelog [skip ci]
([`65bd6d7`](https://github.com/christianlouis/DocuElevate/commit/65bd6d71d00ff7078a05ce12b00f228405e2ec1e))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`948d118`](https://github.com/christianlouis/DocuElevate/commit/948d118926be042cc3c2a68f58241cc2fcfa23ef))
- **changelog**: Update changelog [skip ci]
([`65bd6d7`](https://github.com/christianlouis/DocuElevate/commit/65bd6d71d00ff7078a05ce12b00f228405e2ec1e))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`65bd6d7`](https://github.com/christianlouis/DocuElevate/commit/65bd6d71d00ff7078a05ce12b00f228405e2ec1e))
## Unreleased
## v0.173.2 (2026-05-23)
### Bug Fixes
- Resolve dependabot npm alerts
([`6fc00b8`](https://github.com/christianlouis/DocuElevate/commit/6fc00b8de10b50b4b2f92f6fadbcf7ebbee7136f))
### Documentation
- Extend product roadmap and milestones
([`e46f9b9`](https://github.com/christianlouis/DocuElevate/commit/e46f9b9a21e5838f883eca4370445e0f1b57e6c9))
- **changelog**: Update changelog [skip ci]
([`427db10`](https://github.com/christianlouis/DocuElevate/commit/427db102d85fa676dcf01118fd3757fb300b979d))
## Unreleased
### Documentation
- Extend product roadmap and milestones
([`e46f9b9`](https://github.com/christianlouis/DocuElevate/commit/e46f9b9a21e5838f883eca4370445e0f1b57e6c9))
## v0.173.1 (2026-05-22)
### Bug Fixes
- Register Evernote task settings
([`1ca7f56`](https://github.com/christianlouis/DocuElevate/commit/1ca7f562ef284d2fcceca84b38661abe9038912a))
## v0.173.0 (2026-05-22)
### Code Style
- Apply ruff auto-fix ([#862](https://github.com/christianlouis/DocuElevate/pull/862),
[`4b46c4b`](https://github.com/christianlouis/DocuElevate/commit/4b46c4baf890f3179ac060569ce2e076cf0551b0))
### Features
- **storage**: Add Evernote destination
([#862](https://github.com/christianlouis/DocuElevate/pull/862),
[`4b46c4b`](https://github.com/christianlouis/DocuElevate/commit/4b46c4baf890f3179ac060569ce2e076cf0551b0))
## v0.172.12 (2026-05-17)
### Bug Fixes
- Validate webhook targets before delivery
([#846](https://github.com/christianlouis/DocuElevate/pull/846),
[`e2fa963`](https://github.com/christianlouis/DocuElevate/commit/e2fa96318f5bd45607baa0fe08a0bf14e1ca83d4))
### Testing
- Cover webhook SSRF validation ([#846](https://github.com/christianlouis/DocuElevate/pull/846),
[`e2fa963`](https://github.com/christianlouis/DocuElevate/commit/e2fa96318f5bd45607baa0fe08a0bf14e1ca83d4))
## v0.172.11 (2026-05-17)
### Bug Fixes
- Escape search result template values
([#853](https://github.com/christianlouis/DocuElevate/pull/853),
[`1a02187`](https://github.com/christianlouis/DocuElevate/commit/1a0218799b9a1eb4154e2f4fbb2572cb3922106a))
### Documentation
- **changelog**: Update changelog [skip ci]
([`048f28a`](https://github.com/christianlouis/DocuElevate/commit/048f28a6717fa7f5cf4b235f9142e625e80e5d59))
## Unreleased
## v0.172.10 (2026-05-17)
### Bug Fixes
- **url-upload**: Handle unsafe redirects as client errors
([`871f788`](https://github.com/christianlouis/DocuElevate/commit/871f788f0bd782ba8ad3a7d70e5cd4ccd24f749b))
### Documentation
- **changelog**: Update changelog [skip ci]
([`58b14ae`](https://github.com/christianlouis/DocuElevate/commit/58b14ae769b85e25290126256de936743609af06))
## Unreleased
## v0.172.9 (2026-04-07)
### Bug Fixes
- **api**: Resolve merge conflicts, add type safety for endpoint_url in S3 connection test
([`57db4c7`](https://github.com/christianlouis/DocuElevate/commit/57db4c7c82f4a8df2e7e5e5505e1d5c01768fc16))
### Chores
- **ci**: Ignore CVE-2026-4539 in pip-audit until pygments releases a fix
([`6927e76`](https://github.com/christianlouis/DocuElevate/commit/6927e7643f9cbe1664f4a1a093511df5b079ed0a))
### Code Style
- Apply ruff auto-fix
([`8295279`](https://github.com/christianlouis/DocuElevate/commit/8295279ec93570da4eb0445ede8084d1eb2aba99))
- Sort imports in test_url_upload.py
([`bdfa3ba`](https://github.com/christianlouis/DocuElevate/commit/bdfa3ba1e0a5702414e3b449fbde6a6d3149557a))
### Documentation
- **changelog**: Update changelog [skip ci]
([`c6e0b80`](https://github.com/christianlouis/DocuElevate/commit/c6e0b80becab81a75aea4ee78f5aaf8b6ac54854))
- **changelog**: Update changelog [skip ci]
([`9b9882c`](https://github.com/christianlouis/DocuElevate/commit/9b9882c4d62691d0ddd20444e3b77bfe6eecc8c3))
- **changelog**: Update changelog [skip ci]
([`69053bf`](https://github.com/christianlouis/DocuElevate/commit/69053bfb08d3e2f12a86878044667ac500888837))
- **changelog**: Update changelog [skip ci]
([`76f202f`](https://github.com/christianlouis/DocuElevate/commit/76f202f7f1b94e39a4e79cd984770310599405bf))
### Testing
- Add coverage for url_upload redirect SSRF bypass prevention hook
([`152ee15`](https://github.com/christianlouis/DocuElevate/commit/152ee15b06ebf7beb6216423b4c8d93ec2243165))
- Add tests for SSRF validation in integrations
([`470f08d`](https://github.com/christianlouis/DocuElevate/commit/470f08d89322f2904b78a8b0f820973611486c26))
## Unreleased
### Chores
- **ci**: Ignore CVE-2026-4539 in pip-audit until pygments releases a fix
([`6927e76`](https://github.com/christianlouis/DocuElevate/commit/6927e7643f9cbe1664f4a1a093511df5b079ed0a))
### Documentation
- **changelog**: Update changelog [skip ci]
([`9b9882c`](https://github.com/christianlouis/DocuElevate/commit/9b9882c4d62691d0ddd20444e3b77bfe6eecc8c3))
- **changelog**: Update changelog [skip ci]
([`69053bf`](https://github.com/christianlouis/DocuElevate/commit/69053bfb08d3e2f12a86878044667ac500888837))
- **changelog**: Update changelog [skip ci]
([`76f202f`](https://github.com/christianlouis/DocuElevate/commit/76f202f7f1b94e39a4e79cd984770310599405bf))
### Testing
- Add tests for SSRF validation in integrations
([`470f08d`](https://github.com/christianlouis/DocuElevate/commit/470f08d89322f2904b78a8b0f820973611486c26))
## Unreleased
### Chores
- **ci**: Ignore CVE-2026-4539 in pip-audit until pygments releases a fix
([`6927e76`](https://github.com/christianlouis/DocuElevate/commit/6927e7643f9cbe1664f4a1a093511df5b079ed0a))
### Documentation
- **changelog**: Update changelog [skip ci]
([`69053bf`](https://github.com/christianlouis/DocuElevate/commit/69053bfb08d3e2f12a86878044667ac500888837))
- **changelog**: Update changelog [skip ci]
([`76f202f`](https://github.com/christianlouis/DocuElevate/commit/76f202f7f1b94e39a4e79cd984770310599405bf))
### Testing
- Add tests for SSRF validation in integrations
([`470f08d`](https://github.com/christianlouis/DocuElevate/commit/470f08d89322f2904b78a8b0f820973611486c26))
## Unreleased
### Chores
- **ci**: Ignore CVE-2026-4539 in pip-audit until pygments releases a fix
([`6927e76`](https://github.com/christianlouis/DocuElevate/commit/6927e7643f9cbe1664f4a1a093511df5b079ed0a))
### Documentation
- **changelog**: Update changelog [skip ci]
([`76f202f`](https://github.com/christianlouis/DocuElevate/commit/76f202f7f1b94e39a4e79cd984770310599405bf))
## Unreleased
### Chores
- **ci**: Ignore CVE-2026-4539 in pip-audit until pygments releases a fix
([`6927e76`](https://github.com/christianlouis/DocuElevate/commit/6927e7643f9cbe1664f4a1a093511df5b079ed0a))
## v0.172.8 (2026-03-25)
### Bug Fixes
- **api**: Track env_file_written accurately in save_google_drive_settings
([`4136033`](https://github.com/christianlouis/DocuElevate/commit/4136033bf0e580cbabe811084ab47ea0d6af8f9a))
- **tests**: Add admin override fixture to TestSaveDropboxSettings
([`cafc0e4`](https://github.com/christianlouis/DocuElevate/commit/cafc0e45230ffea096664c1947ac20753b63f8e9))
- **tests**: Restore correct route URLs and fix auth/exception handling broken by d221753
([`48331f6`](https://github.com/christianlouis/DocuElevate/commit/48331f6e91e6c0dae31ab3be31f9da1eccd0a549))
### Documentation
- **changelog**: Update changelog [skip ci]
([`c3124b0`](https://github.com/christianlouis/DocuElevate/commit/c3124b08bd48faa32e76d21d744c9902214048a7))
## Unreleased
## v0.172.7 (2026-03-24)
### Bug Fixes
- **api**: Remove duplicate Depends from AdminUser parameters in dropbox, onedrive, google_drive
([`7f20c90`](https://github.com/christianlouis/DocuElevate/commit/7f20c903ef23e138518e715c7c7a0297d3e46ff8))
- **dockerfile**: Add frontend-builder stage to compile Tailwind CSS
([`3fd8b32`](https://github.com/christianlouis/DocuElevate/commit/3fd8b32724e3d390ff723e5b090a5603c3b1fc93))
- **main**: Replace silent except-pass with exception logging to fix S110
([`8fcc223`](https://github.com/christianlouis/DocuElevate/commit/8fcc223ef19cf609d8413fb6091eabfa0b34d4a6))
## v0.172.6 (2026-03-24)
### Bug Fixes
- Resolve multiple test failures in imap_tasks, main lifespan, and API settings endpoints
([`f041f28`](https://github.com/christianlouis/DocuElevate/commit/f041f28d9f64011df52506ead4bbc87d0797c20e))
- Restore all code deleted/truncated by d2217531 Jules SSRF commit
([`c7d3ec5`](https://github.com/christianlouis/DocuElevate/commit/c7d3ec57c3aca4faeaa0ad2fdbde3a1f770b86a5))
- **migrations**: Restore accidentally deleted migration files 038-042
([`11a49eb`](https://github.com/christianlouis/DocuElevate/commit/11a49eb7fd2218062922a9b8bf01b9a91572bea7))
- **tasks**: Add -- end-of-options separator to ocrmypdf command in convert_to_pdfa
([`7dec570`](https://github.com/christianlouis/DocuElevate/commit/7dec570ce6ae40b934ad075bc06f4cf1dfd9ff2e))
### Code Style
- Apply ruff auto-fix
([`627a857`](https://github.com/christianlouis/DocuElevate/commit/627a8579def3a6a9d4f78da6469cfde889154402))
## v0.172.5 (2026-03-23)
### Bug Fixes
- **main**: Suppress S110 ruff warnings with noqa comments for intentional try-except-pass
([`0b8f967`](https://github.com/christianlouis/DocuElevate/commit/0b8f967eb5e304155752b4492584d4a7509a454c))
- **settings**: Move os.path.exists inside try block in update_env_file so exceptions are non-fatal
([`c9bb2b6`](https://github.com/christianlouis/DocuElevate/commit/c9bb2b6807b371d04edff12d16f838f411b60514))
### Testing
- **google_drive**: Fix exception handling test to expect non-fatal 200 like OneDrive equivalent
([`2f5e2a0`](https://github.com/christianlouis/DocuElevate/commit/2f5e2a0fcdd9f9532fc55c6d7ce1675b8be3d3e8))
- **main,imap**: Fix failing IMAP tests and add coverage for shutdown exception paths
([`c03ce8c`](https://github.com/christianlouis/DocuElevate/commit/c03ce8cdb2e7849361ea50db888b7e3080eaafcd))
## v0.172.4 (2026-03-23)
### Bug Fixes
- Resolve failing tests in main
([`3be93be`](https://github.com/christianlouis/DocuElevate/commit/3be93be35a1564cb6009c5b7d2229820c2b8fafd))
- **api/dropbox**: _require_admin bypasses auth when AUTH_ENABLED=False,
([`3be93be`](https://github.com/christianlouis/DocuElevate/commit/3be93be35a1564cb6009c5b7d2229820c2b8fafd))
### Chores
- Simplify and fix naming for save settings endpoints
([`341839f`](https://github.com/christianlouis/DocuElevate/commit/341839fe5edafa3451f89e2bb57092882d8fd6f0))
- Simplify and fix naming for save settings endpoints
([`57795ee`](https://github.com/christianlouis/DocuElevate/commit/57795ee4871bb0bb0727037a889542bf46a8bb9e))
### Code Style
- Apply ruff auto-fix
([`2014a93`](https://github.com/christianlouis/DocuElevate/commit/2014a93c1ba4f41b8cfb589584be8d39baeaffe1))
### Documentation
- **changelog**: Update changelog [skip ci]
([`bcdbf9d`](https://github.com/christianlouis/DocuElevate/commit/bcdbf9d17885ab3f8750d8426c0ee9f181ced736))
## Unreleased
### Chores
- Simplify and fix naming for save settings endpoints
([`341839f`](https://github.com/christianlouis/DocuElevate/commit/341839fe5edafa3451f89e2bb57092882d8fd6f0))
- Simplify and fix naming for save settings endpoints
([`57795ee`](https://github.com/christianlouis/DocuElevate/commit/57795ee4871bb0bb0727037a889542bf46a8bb9e))
## v0.172.3 (2026-03-23)
### Bug Fixes
- Improve join_url - use walrus op, remove posixpath.normpath
([`15dd1a8`](https://github.com/christianlouis/DocuElevate/commit/15dd1a847133aa02aedf65e7fc75d857151cc26e))
### Code Style
- Apply ruff auto-fix
([`b50a534`](https://github.com/christianlouis/DocuElevate/commit/b50a534454f0432e2ada8140e0090535b7c97051))
- Apply ruff auto-fix
([`326adb1`](https://github.com/christianlouis/DocuElevate/commit/326adb185853e17ac02d30b1bcce33b3a1cf4c5c))
### Documentation
- **changelog**: Update changelog [skip ci]
([`248619d`](https://github.com/christianlouis/DocuElevate/commit/248619d91e91aa9c5660267813367e4cd6f5040f))
- **changelog**: Update changelog [skip ci]
([`26963a8`](https://github.com/christianlouis/DocuElevate/commit/26963a84643c8c5caeb8536ed4dc55302a517adf))
- **changelog**: Update changelog [skip ci]
([`78bd5b5`](https://github.com/christianlouis/DocuElevate/commit/78bd5b5904d41d77d8df2a0e3978f630be080f0f))
- **changelog**: Update changelog [skip ci]
([`cc5e879`](https://github.com/christianlouis/DocuElevate/commit/cc5e879ea98507ec5656cce7162a69d385ee00f2))
- **changelog**: Update changelog [skip ci]
([`0497fbb`](https://github.com/christianlouis/DocuElevate/commit/0497fbbbad71fd728e528498508bbfc7802dab70))
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`26963a8`](https://github.com/christianlouis/DocuElevate/commit/26963a84643c8c5caeb8536ed4dc55302a517adf))
- **changelog**: Update changelog [skip ci]
([`78bd5b5`](https://github.com/christianlouis/DocuElevate/commit/78bd5b5904d41d77d8df2a0e3978f630be080f0f))
- **changelog**: Update changelog [skip ci]
([`cc5e879`](https://github.com/christianlouis/DocuElevate/commit/cc5e879ea98507ec5656cce7162a69d385ee00f2))
- **changelog**: Update changelog [skip ci]
([`0497fbb`](https://github.com/christianlouis/DocuElevate/commit/0497fbbbad71fd728e528498508bbfc7802dab70))
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`78bd5b5`](https://github.com/christianlouis/DocuElevate/commit/78bd5b5904d41d77d8df2a0e3978f630be080f0f))
- **changelog**: Update changelog [skip ci]
([`cc5e879`](https://github.com/christianlouis/DocuElevate/commit/cc5e879ea98507ec5656cce7162a69d385ee00f2))
- **changelog**: Update changelog [skip ci]
([`0497fbb`](https://github.com/christianlouis/DocuElevate/commit/0497fbbbad71fd728e528498508bbfc7802dab70))
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`cc5e879`](https://github.com/christianlouis/DocuElevate/commit/cc5e879ea98507ec5656cce7162a69d385ee00f2))
- **changelog**: Update changelog [skip ci]
([`0497fbb`](https://github.com/christianlouis/DocuElevate/commit/0497fbbbad71fd728e528498508bbfc7802dab70))
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`0497fbb`](https://github.com/christianlouis/DocuElevate/commit/0497fbbbad71fd728e528498508bbfc7802dab70))
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
### Documentation
- **changelog**: Update changelog [skip ci]
([`45d3ac8`](https://github.com/christianlouis/DocuElevate/commit/45d3ac8cf07d39d49930dd6866f76e6015067b08))
### Testing
- Add assertions for task enqueuing parameters
([`eeae47d`](https://github.com/christianlouis/DocuElevate/commit/eeae47ddec01339421e503ba484157e798750b8a))
## Unreleased
## v0.172.2 (2026-03-23)
### Bug Fixes
- Adapt TemplateResponse calls to Starlette 1.0 new-style API
([`c4e10be`](https://github.com/christianlouis/DocuElevate/commit/c4e10bee5e096e71a5bc4fac4928f69e5c04f2fb))
- Update test assertions and lint fixes for Starlette 1.0 TemplateResponse API
([`93629ff`](https://github.com/christianlouis/DocuElevate/commit/93629ff44083d43f79fdd49431457023e53d13e4))
- **build**: Remove --omit=dev from npm ci in Dockerfile frontend-builder stage
([`b4e0067`](https://github.com/christianlouis/DocuElevate/commit/b4e0067a27e2fb161349bd38c6d3b3f3bcb86972))
### Documentation
- **changelog**: Update changelog [skip ci]
([`0841713`](https://github.com/christianlouis/DocuElevate/commit/084171395d1076c716aa500a516118db49468ff5))
## Unreleased
## v0.172.2 (2026-03-23)
### Bug Fixes
- Adapt TemplateResponse calls to Starlette 1.0 new-style API
([`c4e10be`](https://github.com/christianlouis/DocuElevate/commit/c4e10bee5e096e71a5bc4fac4928f69e5c04f2fb))
- Update test assertions and lint fixes for Starlette 1.0 TemplateResponse API
([`93629ff`](https://github.com/christianlouis/DocuElevate/commit/93629ff44083d43f79fdd49431457023e53d13e4))
- **build**: Remove --omit=dev from npm ci in Dockerfile frontend-builder stage
([`b4e0067`](https://github.com/christianlouis/DocuElevate/commit/b4e0067a27e2fb161349bd38c6d3b3f3bcb86972))
### Documentation
- **changelog**: Update changelog [skip ci]
([`0841713`](https://github.com/christianlouis/DocuElevate/commit/084171395d1076c716aa500a516118db49468ff5))
## Unreleased ## Unreleased
+9 -11
View File
@@ -27,21 +27,20 @@ RUN pip install --no-cache-dir -r requirements.txt \
&& find /opt/venv -type f -name "*.pyc" -delete \ && find /opt/venv -type f -name "*.pyc" -delete \
&& find /opt/venv -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true && find /opt/venv -type d -name "__pycache__" -exec rm -rf {} + 2>/dev/null || true
# ── Stage 2: Frontend asset builder ───────────────────────────────────────── # ── Stage 2: Frontend asset builder (Tailwind CSS) ──────────────────────────
# Compiles Tailwind CSS (a devDependency) into the minified styles.css. FROM node:20-alpine AS frontend-builder
# npm ci installs ALL deps (including devDependencies) so the tailwindcss CLI
# is available; using --omit=dev would cause 'tailwindcss: not found'.
FROM node:20-slim AS frontend-builder
WORKDIR /frontend WORKDIR /frontend
# Install dependencies first (layer-cached unless package.json/lockfile changes)
COPY frontend/package.json frontend/package-lock.json ./ COPY frontend/package.json frontend/package-lock.json ./
RUN npm ci RUN npm ci --omit=dev
# Copy source files and compile Tailwind CSS
COPY frontend/ ./ COPY frontend/ ./
RUN npm run build RUN npm run build
# ── Stage 4: Documentation builder ────────────────────────────────────────── # ── Stage 3: Documentation builder ──────────────────────────────────────────
FROM python:3.14.3-slim AS docs-builder FROM python:3.14.3-slim AS docs-builder
WORKDIR /docs WORKDIR /docs
@@ -57,7 +56,7 @@ COPY mkdocs.yml /docs/mkdocs.yml
# Build the static documentation site # Build the static documentation site
RUN mkdocs build --config-file /docs/mkdocs.yml --site-dir /docs/docs_build RUN mkdocs build --config-file /docs/mkdocs.yml --site-dir /docs/docs_build
# ── Stage 5: Runtime image ─────────────────────────────────────────────────── # ── Stage 4: Runtime image ───────────────────────────────────────────────────
FROM python:3.14.3-slim FROM python:3.14.3-slim
WORKDIR /app WORKDIR /app
@@ -82,6 +81,8 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
# Copy application code # Copy application code
COPY ./app /app/app COPY ./app /app/app
COPY ./frontend /app/frontend COPY ./frontend /app/frontend
# Overlay compiled Tailwind CSS from the frontend build stage
COPY --from=frontend-builder /frontend/static/styles.css /app/frontend/static/styles.css
COPY ./migrations /app/migrations COPY ./migrations /app/migrations
COPY ./alembic.ini /app/alembic.ini COPY ./alembic.ini /app/alembic.ini
COPY ./LICENSE /app/LICENSE COPY ./LICENSE /app/LICENSE
@@ -95,9 +96,6 @@ COPY ./RUNTIME_INFO /app/RUNTIME_INFO
# Copy the pre-built MkDocs documentation site (served at /help) # Copy the pre-built MkDocs documentation site (served at /help)
COPY --from=docs-builder /docs/docs_build /app/docs_build COPY --from=docs-builder /docs/docs_build /app/docs_build
# Copy the compiled Tailwind CSS (built in the frontend-builder stage)
COPY --from=frontend-builder /frontend/static/styles.css /app/frontend/static/styles.css
# Create necessary runtime directories in a single layer # Create necessary runtime directories in a single layer
RUN mkdir -p /app/runtime_info /workdir RUN mkdir -p /app/runtime_info /workdir
+1 -1
View File
@@ -1 +1 @@
425805a 76c0e91
+69 -115
View File
@@ -1,6 +1,6 @@
# DocuElevate Milestones # DocuElevate Milestones
**Last Updated:** 2026-05-23 **Last Updated:** 2026-02-08
This document outlines the release milestones, versioning strategy, and detailed feature breakdown for DocuElevate. This document outlines the release milestones, versioning strategy, and detailed feature breakdown for DocuElevate.
@@ -19,15 +19,17 @@ DocuElevate follows [Semantic Versioning 2.0.0](https://semver.org/):
--- ---
## Current State (Continuous Releases) ## Current Release: v0.5.0 (February 2026)
DocuElevate ships continuously via automated semantic versioning. Use **GitHub Releases** for the latest build artifacts and **GitHub Milestones** (below) for roadmap tracking. ### Status: Stable
- Production-ready document processing
### Last Shipped Milestone: v0.5.0 (Released February 8, 2026) - Multi-provider storage support
- Database-backed settings management with encryption - **Database-backed settings management with encryption**
- Setup wizard for first-time configuration - **Setup wizard for first-time configuration**
- Admin UI for runtime configuration - **Admin UI for runtime configuration**
- Release automation via semantic-release - **Automated semantic versioning and releases**
- OAuth2 authentication with admin group support
- Basic web UI and REST API
### Important Note on Versioning ### Important Note on Versioning
As of February 2026, DocuElevate uses **automated semantic versioning**: As of February 2026, DocuElevate uses **automated semantic versioning**:
@@ -128,78 +130,87 @@ As of February 2026, DocuElevate uses **automated semantic versioning**:
## Upcoming Milestones ## Upcoming Milestones
### v0.6.0 - Clarity: Enhanced Search & UI (Target: July 31, 2026) ### v0.6.0 - Enhanced Search & UI Improvements (April 2026)
**Target Date:** July 31, 2026 **Target Date:** April 1, 2026
**Status:** 📋 Planned **Status:** 📋 Planned
**Theme:** Search, Discovery, Modern UX **Theme:** User Experience, Search, Performance
**Epic:** #863
#### Goals #### Goals
- Hybrid discovery: keyword + semantic search, fast filtering, saved searches - Implement full-text search across documents
- Preview-first UX (open, skim, and act quickly) - Responsive mobile interface
- Modern UX polish (accessibility, responsiveness, performance) - Dark mode support
- Document preview in browser
- Performance optimizations
- Improved error handling and user feedback
#### Deliverables #### Deliverables
- Semantic search foundation (vectorization + ranking signals) - Full-text search API and UI
- Saved searches / smart views - Advanced filtering capabilities
- In-browser preview + “quick actions” (tag, route, export) - Responsive CSS framework integration
- Bulk operations and pagination improvements - Dark mode toggle
- UX polish (dark mode/accessibility where applicable) - In-browser document viewer
- Loading states and progress indicators
- Performance benchmarks
- Mobile-optimized interface
#### Breaking Changes #### Breaking Changes
- Potential pagination/search response changes (must be versioned and documented) - API response format changes for search endpoints (documented)
#### Migration Path #### Migration Path
- Version endpoints where needed and keep previous versions working for at least 2 minor milestones - Search endpoint changes will be versioned (/api/v1/search → /api/v2/search)
- Old endpoints deprecated but functional for 2 releases
--- ---
### v0.7.0 - Conductor: Workflow Automation & Integrations (Target: September 30, 2026) ### v0.4.5 - Workflow Automation (June 2026)
**Target Date:** September 30, 2026 **Target Date:** June 1, 2026
**Status:** 📋 Planned **Status:** 📋 Planned
**Theme:** Automation, Integration, Webhooks **Theme:** Automation, Integration, Webhooks
**Epic:** #864
#### Goals #### Goals
- First-class workflow model (steps, state, retries) that matches what the system actually executes - Custom processing pipelines
- Workflow-aware UI status, retries, and observability - Conditional routing based on document type
- Webhooks + event-driven automation foundations - Webhook support for external integrations
- Rule-based classification
- Scheduled batch processing
#### Deliverables #### Deliverables
- Workflow object model and storage - Pipeline configuration UI
- Workflow-aware file detail view + status dashboard - Webhook management interface
- Scheduling primitives (recurring jobs / delayed runs) - Rule engine for document routing
- Webhook system (outbound events + inbound triggers) - Batch processing scheduler
- Integration templates and documentation - Integration examples and templates
- Webhook payload documentation
--- ---
### v0.8.0 - Signal: AI Quality, RAG, and Multi-language (Target: November 30, 2026) ### v0.7.0 - Advanced AI & Multi-language (August 2026)
**Target Date:** November 30, 2026 **Target Date:** August 1, 2026
**Status:** 📋 Planned **Status:** 📋 Planned
**Theme:** AI Quality, Retrieval, Internationalization **Theme:** AI Enhancement, Internationalization
**Epic:** #865
#### Goals #### Goals
- “Chat with Library” foundations (retrieval + UI) - Custom AI model support
- Local AI options for privacy-sensitive setups - Multi-language OCR
- Measurable AI quality (confidence + human review loop) - Document similarity detection
- Expand multilingual capability across OCR + UI - Duplicate detection
- UI internationalization (i18n)
- API localization
#### Deliverables #### Deliverables
- Vector DB integration and embeddings pipeline - Custom model integration API
- Chat UI foundations and retrieval API - Multi-language OCR configuration
- Confidence scoring + human review/edit loop for extracted fields - Similarity algorithm implementation
- Multi-language OCR configuration improvements - Duplicate detection service
- Expanded i18n coverage + localized docs - Translation framework (10+ languages)
- Localized documentation
--- ---
### v1.0.0 - Summit: Enterprise Edition (Target: March 31, 2027) ### v1.0.0 - Enterprise Edition (November 2026)
**Target Date:** March 31, 2027 **Target Date:** November 1, 2026
**Status:** 📋 Planned **Status:** 📋 Planned
**Theme:** Enterprise Features, Scalability, Multi-tenancy **Theme:** Enterprise Features, Scalability, Multi-tenancy
**Epic:** #866
This is our first major release, marking production-ready enterprise capabilities. This is our first major release, marking production-ready enterprise capabilities.
@@ -233,60 +244,6 @@ This is our first major release, marking production-ready enterprise capabilitie
--- ---
### v2.0.0 - Horizon: Platform Expansion (Target: September 30, 2027)
**Target Date:** September 30, 2027
**Status:** 📋 Planned
**Theme:** Ecosystem, Platform, Distribution
**Epic:** #867
#### Goals
- Make DocuElevate extensible by design (plugins + templates)
- Expand integrations and developer experience
- Harden multi-surface experiences (web, mobile, extension, CLI) as a cohesive product
#### Deliverables
- Plugin system foundations and public extension points
- Template library for pipelines/workflows + “starter kits”
- Integration hub patterns (webhooks, events, connectors)
- SDK + documentation for extensions
---
### v2.1.0 - Sentinel: Governance & Policy (Target: March 31, 2028)
**Target Date:** March 31, 2028
**Status:** 📋 Planned
**Theme:** Governance, Compliance, Policy-driven Automation
**Epic:** #868
#### Goals
- Make governance first-class (retention, legal hold, PII workflows)
- Provide tamper-evident auditing and admin controls
- Introduce policy-driven approvals for sensitive automation
#### Deliverables
- Retention policies + legal hold primitives
- PII detection + redaction workflows
- Tamper-evident audit trails + admin activity feed
- Policy-as-code concepts for workflows (with approval gates)
---
### v3.0.0 - Constellation: Integration Hub & Agent Platform (Target: September 30, 2028)
**Target Date:** September 30, 2028
**Status:** 📋 Planned
**Theme:** Ecosystem, Agents, Interoperability
**Epic:** #869
#### Goals
- Make DocuElevate the “system of record” for document intelligence in an organization
- Support external automation ecosystems (Zapier/Make/n8n) and agent runtimes
- Provide a clean interoperability layer for modern AI tools
#### Deliverables
- DocuElevate MCP server (search, retrieve, summarize, route) and documentation
- Connector marketplace concepts (curated + community)
- Event stream + webhooks at scale (delivery guarantees, retries, signing)
## Release Process ## Release Process
### Automated Semantic Versioning (v0.6.0+) ### Automated Semantic Versioning (v0.6.0+)
@@ -298,15 +255,15 @@ Starting with v0.6.0, releases are fully automated using `python-semantic-releas
4. **Automatic Updates**: 4. **Automatic Updates**:
- Updates `VERSION` file - Updates `VERSION` file
- Generates/updates `CHANGELOG.md` - Generates/updates `CHANGELOG.md`
- Creates Git tag (e.g., `v0.173.1`) - Creates Git tag (e.g., `v0.6.0`)
- Creates GitHub Release with notes - Creates GitHub Release with notes
- Triggers Docker image builds - Triggers Docker image builds
5. **No Manual Steps**: VERSION and CHANGELOG are never edited manually 5. **No Manual Steps**: VERSION and CHANGELOG are never edited manually
### Version Bump Rules ### Version Bump Rules
- `feat:` commits → Minor version (e.g., 0.173.1 → 0.174.0) - `feat:` commits → Minor version (0.5.0 → 0.6.0)
- `fix:`, `perf:` → Patch version (e.g., 0.173.1 → 0.173.2) - `fix:`, `perf:` → Patch version (0.5.0 → 0.5.1)
- `feat!:`, `BREAKING CHANGE:` → Major version (e.g., 0.173.1 → 1.0.0) - `feat!:`, `BREAKING CHANGE:` → Major version (0.5.0 → 1.0.0)
- Other types (docs, chore, etc.) → No version bump - Other types (docs, chore, etc.) → No version bump
### Pre-release Checklist (Automated) ### Pre-release Checklist (Automated)
@@ -340,13 +297,10 @@ Starting with v0.6.0, releases are fully automated using `python-semantic-releas
| v0.3.2 | 2026-02-06 | Security Updates | Released | | v0.3.2 | 2026-02-06 | Security Updates | Released |
| v0.3.3 | 2026-02-08 | Drag-and-Drop Upload | Released | | v0.3.3 | 2026-02-08 | Drag-and-Drop Upload | Released |
| v0.5.0 | 2026-02-08 | **Settings & Encryption** | **Released** | | v0.5.0 | 2026-02-08 | **Settings & Encryption** | **Released** |
| v0.6.0 | 2026-07-31 | **Clarity:** Search & UX | Planned | | v0.6.0 | 2026-04 | Search & UX | Planned |
| v0.7.0 | 2026-09-30 | **Conductor:** Workflows & Integrations | Planned | | v0.7.0 | 2026-08 | Advanced AI | Planned |
| v0.8.0 | 2026-11-30 | **Signal:** AI Quality, RAG, Multi-language | Planned | | v1.0.0 | 2026-11 | Enterprise | Planned |
| v1.0.0 | 2027-03-31 | **Summit:** Enterprise | Planned | | v2.0.0 | 2027-Q3 | Platform Expansion | Future |
| v2.0.0 | 2027-09-30 | **Horizon:** Platform Expansion | Future |
| v2.1.0 | 2028-03-31 | **Sentinel:** Governance & Policy | Future |
| v3.0.0 | 2028-09-30 | **Constellation:** Integration Hub & Agents | Future |
--- ---
+1 -4
View File
@@ -35,7 +35,7 @@ DocuElevate is an intelligent document processing system that automates the inge
- **AI-Powered Metadata Extraction** — pluggable AI providers including OpenAI, Anthropic Claude, Google Gemini, Ollama (local), OpenRouter, Portkey, and Azure OpenAI via LiteLLM - **AI-Powered Metadata Extraction** — pluggable AI providers including OpenAI, Anthropic Claude, Google Gemini, Ollama (local), OpenRouter, Portkey, and Azure OpenAI via LiteLLM
- **Multi-Engine OCR** — Azure Document Intelligence, Tesseract, EasyOCR, Mistral OCR, Google Cloud Document AI, and AWS Textract with configurable merge strategies - **Multi-Engine OCR** — Azure Document Intelligence, Tesseract, EasyOCR, Mistral OCR, Google Cloud Document AI, and AWS Textract with configurable merge strategies
- **13 Storage Destinations** — Dropbox, Google Drive, OneDrive, Amazon S3, Nextcloud, WebDAV, FTP, SFTP, iCloud Drive, Email (SMTP), Paperless-ngx, Evernote, and Rclone - **12 Storage Destinations** — Dropbox, Google Drive, OneDrive, Amazon S3, Nextcloud, WebDAV, FTP, SFTP, iCloud Drive, Email (SMTP), Paperless-ngx, and Rclone
- **Multi-Channel Ingestion** — web upload, browser extension, mobile app, CLI, REST API, IMAP email, and watched folders (local, cloud, FTP/SFTP) - **Multi-Channel Ingestion** — web upload, browser extension, mobile app, CLI, REST API, IMAP email, and watched folders (local, cloud, FTP/SFTP)
- **Processing Pipelines** — customizable multi-step workflows with conditional routing rules - **Processing Pipelines** — customizable multi-step workflows with conditional routing rules
- **Full-Text Search** — powered by Meilisearch for instant document discovery - **Full-Text Search** — powered by Meilisearch for instant document discovery
@@ -106,7 +106,6 @@ Processed documents are distributed to any combination of configured destination
| **iCloud Drive** | Apple cloud | | **iCloud Drive** | Apple cloud |
| **Email (SMTP)** | Send as attachment | | **Email (SMTP)** | Send as attachment |
| **Paperless-ngx** | Document management system | | **Paperless-ngx** | Document management system |
| **Evernote** | Notes with PDF attachments |
| **Rclone** | 70+ cloud providers via Rclone | | **Rclone** | 70+ cloud providers via Rclone |
## Features ## Features
@@ -246,7 +245,6 @@ See the [Kubernetes Deployment Guide](docs/KubernetesDeployment.md) for full det
| [Google Drive](docs/GoogleDriveSetup.md) | Google Drive service account / OAuth | | [Google Drive](docs/GoogleDriveSetup.md) | Google Drive service account / OAuth |
| [OneDrive](docs/OneDriveSetup.md) | Microsoft OneDrive setup | | [OneDrive](docs/OneDriveSetup.md) | Microsoft OneDrive setup |
| [Amazon S3](docs/AmazonS3Setup.md) | S3 bucket configuration | | [Amazon S3](docs/AmazonS3Setup.md) | S3 bucket configuration |
| [Evernote](docs/EvernoteSetup.md) | Evernote note creation |
| [Authentication](docs/AuthenticationSetup.md) | OAuth2, OIDC, and social login | | [Authentication](docs/AuthenticationSetup.md) | OAuth2, OIDC, and social login |
| [Notifications](docs/NotificationsSetup.md) | Notification backend setup | | [Notifications](docs/NotificationsSetup.md) | Notification backend setup |
@@ -329,7 +327,6 @@ The following is a summary of the licenses used by our direct dependencies:
| pypdf | BSD | | pypdf | BSD |
| Requests | Apache 2.0 | | Requests | Apache 2.0 |
| Dropbox SDK | MIT | | Dropbox SDK | MIT |
| Evernote SDK | BSD |
| Azure AI Document Intelligence | MIT | | Azure AI Document Intelligence | MIT |
| Authlib | BSD | | Authlib | BSD |
| Starlette | BSD | | Starlette | BSD |
+133 -103
View File
@@ -1,139 +1,169 @@
# DocuElevate Roadmap # DocuElevate Roadmap
**Last Updated:** 2026-05-23 **Last Updated:** 2026-02-08
**Version:** 2.0 **Version:** 1.0
## Vision ## Vision
DocuElevate aims to be the premier open-source intelligent document processing platform, providing seamless integration with cloud storage providers, advanced AI-powered metadata extraction, and enterprise-grade security and scalability. DocuElevate aims to be the premier open-source intelligent document processing platform, providing seamless integration with cloud storage providers, advanced AI-powered metadata extraction, and enterprise-grade security and scalability.
## How to Read This Roadmap
DocuElevate ships frequently (automated semantic versioning), so this roadmap is organized around **milestone outcomes** and **themes**, not exact build numbers.
- **P0** = required for the milestone to feel “done”
- **P1** = strongly desired; may slip if needed
- **P2** = nice-to-have / opportunistic
For the detailed milestone breakdown and target dates, see [MILESTONES.md](MILESTONES.md).
## Release Naming ## Release Naming
Each major milestone release carries a codename to anchor key project moments. These names appear in the status dashboard, build metadata, and changelog. For details, see [docs/ReleaseNaming.md](docs/ReleaseNaming.md). Each major milestone release carries a codename to anchor key project moments. These names appear in the status dashboard, build metadata, and changelog. For details, see [docs/ReleaseNaming.md](docs/ReleaseNaming.md).
| Milestone | Codename | Theme | | Version Range | Codename | Theme |
|----------|------------------|-------| |---------------|---------------|--------------------------------------------------|
| v0.6.0 | **Clarity** | Search, discovery, and modern UX | | 0.5.x | **Foundation** | Core platform, multi-provider storage, AI, UI |
| v0.7.0 | **Conductor** | Workflows, orchestration, and integrations | | 0.6.x | **Clarity** | Enhanced search, filtering, UI/UX improvements |
| v0.8.0 | **Signal** | AI quality, multilingual, and “Chat with Library” foundations | | 0.7.x | **Conductor** | Workflow automation, pipelines, rule-based logic |
| v1.0.0 | **Summit** | Enterprise readiness (multi-tenancy, RBAC, scaling) | | 1.0.x | **Summit** | Enterprise features, multi-tenancy, RBAC |
| v2.0.0 | **Horizon** | Platform expansion and ecosystem maturity | | 1.1.x | **Bridge** | Collaboration, sharing, analytics |
| v2.1.0+ | **Sentinel** | Governance, compliance, and policy-driven automation | | 2.0.x | **Horizon** | On-premise AI, platform expansion |
| v3.0.0 | **Constellation**| Integration hub, agents, and interoperability |
## Current Product Capabilities (Today) ## Current Status (v0.5.0 "Foundation")
### Core Features ✅ ### Core Features ✅
- Multi-channel ingestion (web upload, IMAP email, watched folders, mobile, CLI, API) - Multi-provider document storage (Dropbox, Google Drive, OneDrive, Nextcloud, S3, etc.)
- Multi-engine OCR + AI extraction with configurable providers - IMAP email integration for document ingestion
- Customizable processing pipelines and routing rules - OCR processing via Azure Document Intelligence
- Full-text search and document discovery - AI-powered metadata extraction via OpenAI
- Multi-destination distribution (cloud providers, DMS, protocols, email) - PDF conversion via Gotenberg
- Admin UI for configuration (database-backed settings, encryption, setup wizard) - Web UI for document upload and management
- Production hardening building blocks (CI/CD, security docs, deployment guides) - **Database-backed settings management with admin UI**
- **Fernet encryption for sensitive configuration**
- **Setup wizard for first-time installation**
- REST API with OpenAPI documentation
- Celery-based async task processing
- OAuth2 authentication via Authentik with admin group support
## Feature Landscape (Themes) ## Short-term Goals (Q1-Q2 2026) - v0.4.x to v0.5.x "Foundation"
### 1) Search & Discovery ### Quality & Stability 🎯
- **P0:** hybrid search (keyword + semantic), fast faceted filtering, saved searches - **Test Coverage** (High Priority)
- **P1:** “explain results” (why a document matched), query suggestions, pinned results - [ ] Achieve 80% code coverage for core modules
- **P2:** entity search (people/companies/amounts/dates) and graph-style exploration - [ ] Add integration tests for all storage providers
- [ ] Add end-to-end workflow tests
- [ ] Performance benchmarks and load testing
### 2) AI Quality & Trust - **Code Quality** (High Priority)
- **P0:** confidence scoring, human review/edit loop, extraction evaluation harness - [ ] Enable strict linting in CI/CD
- **P1:** per-document-type schemas/templates, active learning (feedback improves extraction) - [ ] Refactor large modules for better maintainability
- **P2:** multi-model routing (choose model by cost/latency/accuracy per step) - [ ] Add comprehensive type hints
- [ ] Improve error handling and user feedback
### 3) Workflow Automation & Orchestration - **Security** (Critical Priority)
- **P0:** first-class workflow model (steps, state, retries), workflow-aware UI status - [x] Fix known vulnerabilities in dependencies
- **P1:** visual workflow builder, scheduling, webhooks, and event-driven triggers - [ ] Implement rate limiting on API endpoints
- **P2:** agentic workflows (“autopilot” suggestions with approval gates) - [ ] Add CSRF protection
- [ ] Security audit by external party
- [ ] Implement API key rotation
- [ ] Add audit logging for sensitive operations
### 4) Integrations & Ecosystem (Including MCP) - **Release Automation** (Completed ✅)
- **P0:** stable webhooks + outbound actions (Slack/Teams, email, DMS), bi-directional sync where supported - [x] Implement semantic-release for automated versioning
- **P1:** “Integration Hub” (Zapier/Make/n8n style), connector templates, secrets handling patterns - [x] Add conventional commit validation
- **P2:** **MCP**: ship a DocuElevate MCP server (search, retrieve, summarize, route) + allow MCP tools as pipeline steps - [x] Automate CHANGELOG generation
- [x] Integrate Docker builds with releases
### 5) Governance, Compliance, and Security ### Features - v0.4.0
- **P0:** audit trails, tamper-evident logs, API key lifecycle/rotation, admin activity feed - **Enhanced Search & Filtering** → _preparing for v0.6.0 "Clarity"_
- **P1:** retention policies, legal hold, PII detection + redaction, data residency controls - [ ] Full-text search across documents
- **P2:** compliance packs (SOC2/GDPR/HIPAA), BYOK/KMS integration paths - [ ] Advanced filtering by metadata, tags, date ranges
- [ ] Saved search queries
- [ ] Bulk operations on search results
### 6) Enterprise & Scale - **Improved UI/UX**
- **P0:** multi-tenancy, RBAC, horizontal scaling reference architecture - [ ] Responsive mobile interface
- **P1:** SCIM provisioning, SAML/Okta/Azure AD hardening, quotas/billing at org level - [ ] Dark mode support
- **P2:** multi-region deployment patterns and disaster recovery playbooks - [ ] Document preview in browser
- [ ] Drag-and-drop file upload
- [ ] Progress indicators for long-running tasks
- [ ] Real-time notifications via WebSocket
## Release Plan (Extended) ### Features - v0.5.0 "Foundation"
- **Workflow Automation** → _evolving into v0.7.0 "Conductor"_
- [ ] Custom processing pipelines
- [ ] Conditional routing based on document type
- [ ] Scheduled batch processing
- [ ] Webhook support for external integrations
- [ ] Rule-based document classification
This plan extends the existing milestones with a clearer thematic arc and a forward-looking “beyond v2.0” horizon. Each milestone links to an epic issue that owns scope and sub-issues. - **Advanced AI Features**
- [ ] Custom AI models for specialized document types
- [ ] Multi-language OCR support
- [ ] Document similarity detection
- [ ] Automatic duplicate detection
- [ ] Intelligent document splitting
### v0.6.0 — Clarity (Search & UX) ## Medium-term Goals (Q3-Q4 2026) - v1.0.x "Summit"
- **Outcome:** users can reliably find, preview, and act on documents in seconds
- **P0:** semantic search + hybrid ranking, saved searches, fast filters, preview-first UX
- **P1:** bulk operations, query suggestions, accessibility/dark mode polish
- **Tracking:** GitHub milestone `v0.6.0 - Enhanced Search & UI` (epic #863)
### v0.7.0 — Conductor (Workflows & Integrations) ### Enterprise Features - v1.0.0 "Summit"
- **Outcome:** workflows are explicit, inspectable, and automatable end-to-end - **Multi-tenancy**
- **P0:** workflow object model + workflow-aware UI status, retries, pipeline definitions - [ ] Organization/team management
- **P1:** workflow builder, scheduling, inbound/outbound webhooks - [ ] Role-based access control (RBAC)
- **P2:** integration templates + “connector marketplace” concepts - [ ] Per-tenant configuration
- **Tracking:** GitHub milestone `v0.7.0 - Workflow Automation` (epic #864) - [ ] Resource quotas and limits
- [ ] Audit logs per organization
### v0.8.0 — Signal (AI Quality + “Chat with Library” Foundations) - **Scalability**
- **Outcome:** AI features are measurable, reviewable, and safe to trust - [ ] Horizontal scaling support
- **P0:** vector DB + embeddings pipeline, chat UI foundations, local AI options - [ ] Distributed task processing
- **P1:** confidence scoring and review loop, extraction evaluation harness - [ ] Caching layer (Redis/Memcached)
- **P2:** multilingual UX + localization expansion - [ ] Database connection pooling
- **Tracking:** GitHub milestone `v0.8.0 - Advanced AI & Multi-language` (epic #865) - [ ] Message queue optimization
### v1.0.0 — Summit (Enterprise Readiness) - **Advanced Integrations**
- **Outcome:** teams can run DocuElevate with strong isolation, access control, and scale - [ ] Microsoft SharePoint integration
- **P0:** multi-tenancy, RBAC, audit logging, scaling guidance - [ ] Slack/Teams bot integration
- **P1:** SSO hardening (SAML/LDAP), org-level quotas and billing hooks - [ ] Zapier/Make.com integration
- **P2:** enterprise admin experience (policies, approvals, reporting) - [ ] Custom webhook receivers
- **Tracking:** GitHub milestone `v1.0.0 - Enterprise Edition` (epic #866) - [ ] GraphQL API
### v2.0.0 — Horizon (Platform Expansion) ### Features - v1.1.0 "Bridge"
- **Outcome:** DocuElevate becomes an extensible platform with a thriving ecosystem - **Collaboration**
- **P0:** plugin system foundations, SDK + templates, deeper integrations - [ ] Document sharing with expiring links
- **P1:** marketplace patterns, app distribution, mobile/extension maturity - [ ] Comments and annotations
- **P2:** multi-workspace experiences (personal + org) - [ ] Version history and rollback
- **Tracking:** GitHub milestone `v2.0.0 - Platform Expansion` (epic #867) - [ ] Real-time collaborative editing metadata
- [ ] Activity feed
### v2.1.0+ — Sentinel (Governance & Policy) - **Reporting & Analytics**
- **Outcome:** governance becomes a first-class layer (policy-driven automation) - [ ] Processing statistics dashboard
- **P0:** retention + legal hold, PII detection/redaction, tamper-evident audit trails - [ ] Storage usage analytics
- **P1:** BYOK/KMS integration patterns, advanced access policies, compliance reporting - [ ] AI confidence scores and accuracy tracking
- **P2:** “policy as code” for workflows + approvals (change management) - [ ] Cost analysis per provider
- **Tracking:** GitHub milestone `v2.1.0 - Governance & Policy (Sentinel)` (epic #868) - [ ] Export reports (PDF, CSV, Excel)
### v3.0.0 — Constellation (Integration Hub & Agent Platform) ## Long-term Goals (2027+) - v2.0+ "Horizon"
- **Outcome:** DocuElevate plugs into modern automation and AI ecosystems as a first-class system of record
- **P0:** MCP server, durable event stream + production-grade webhooks
- **P1:** connector templates + curated catalog, agent-friendly permissioning and auditing
- **P2:** bring-your-own-agent patterns (sandboxing, scoped credentials)
- **Tracking:** GitHub milestone `v3.0.0 - Integration Hub & Agent Platform (Constellation)` (epic #869)
## Research Bets (Optional / Experimental) ### Strategic Initiatives
- **On-Premise AI Models**
- [ ] Self-hosted OCR (Tesseract, EasyOCR)
- [ ] Local LLM integration (Ollama, LLaMA)
- [ ] GPU acceleration support
- [ ] Model fine-tuning interface
- [ ] Hybrid cloud/on-premise processing
These are longer-horizon bets that should only be productized if they prove real user value. - **Advanced Document Management**
- [ ] Document lifecycle management
- [ ] Retention policies and auto-deletion
- [ ] Compliance templates (GDPR, HIPAA, SOC2)
- [ ] Digital signature support
- [ ] Encryption at rest and in transit
- Knowledge graph over extracted entities (contracts ↔ vendors ↔ invoices) - **Platform Expansion**
- Auto-generated “case files” (collections) from intent (“tax 2025”, “project alpha”) - [ ] Desktop applications (Electron)
- Privacy-preserving learning (federated patterns) to improve extraction quality - [ ] Mobile apps (iOS/Android)
- Document provenance (signing, attestations) and tamper detection - [ ] Browser extensions
- [ ] Command-line interface (CLI)
- [ ] VS Code extension for developers
### Research & Innovation
- [ ] Machine learning for custom document types
- [ ] Blockchain for document provenance
- [ ] Federated learning for privacy-preserving AI
- [ ] Edge computing support
- [ ] Quantum-resistant encryption
## Community & Ecosystem ## Community & Ecosystem
+6 -6
View File
@@ -1,10 +1,10 @@
DocuElevate Build Information DocuElevate Build Information
============================== ==============================
Version: 0.173.4 Version: 0.172.1
Build Date: 2026-06-01T03:41:15Z Build Date: 2026-03-22T18:47:07Z
Git Commit: 425805ab23882944aee0cb02b5e497bc536549c0 Git Commit: 76c0e91500963fac4e8d4a43123340a7cc64731f
Git Short SHA: 425805a Git Short SHA: 76c0e91
Git Branch: main Git Branch: main
Commit Date: 2026-06-01T05:40:53+02:00 Commit Date: 2026-03-22T19:46:48+01:00
Build Timestamp: 2026-06-01T03:41:16Z Build Timestamp: 2026-03-22T18:47:07Z
============================== ==============================
+1 -1
View File
@@ -1 +1 @@
0.173.4 0.172.1
+1 -1
View File
@@ -260,7 +260,7 @@ async def stripe_webhook(request: Request, db: Session = Depends(get_db)) -> dic
@require_login @require_login
async def billing_success(request: Request) -> Any: async def billing_success(request: Request) -> Any:
"""Show a success page after a completed Stripe Checkout.""" """Show a success page after a completed Stripe Checkout."""
return _templates.TemplateResponse(request, "billing_success.html") return _templates.TemplateResponse("billing_success.html", {"request": request})
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+2 -13
View File
@@ -25,17 +25,6 @@ logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
def _require_admin(request: Request) -> dict:
"""Ensure the caller is an admin. Raises 403 otherwise."""
user = request.session.get("user")
if not user or not user.get("is_admin"):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
return user
AdminUser = Annotated[dict, Depends(_require_admin)]
def _build_dropbox_redirect_uri(request: Request) -> str: def _build_dropbox_redirect_uri(request: Request) -> str:
"""Build the Dropbox OAuth callback redirect URI. """Build the Dropbox OAuth callback redirect URI.
@@ -396,14 +385,14 @@ async def list_dropbox_folders(
@router.post("/dropbox/save-settings") @router.post("/dropbox/save-settings")
@require_login
async def save_dropbox_settings( async def save_dropbox_settings(
request: Request, request: Request,
refresh_token: Annotated[str, Form(...)], refresh_token: Annotated[str, Form(...)],
_admin: AdminUser,
db: Session = Depends(get_db),
app_key: Annotated[Optional[str], Form()] = None, app_key: Annotated[Optional[str], Form()] = None,
app_secret: Annotated[Optional[str], Form()] = None, app_secret: Annotated[Optional[str], Form()] = None,
folder_path: Annotated[Optional[str], Form()] = None, folder_path: Annotated[Optional[str], Form()] = None,
db: Session = Depends(get_db),
): ):
""" """
Save Dropbox settings to database (primary) and .env file (best-effort). Save Dropbox settings to database (primary) and .env file (best-effort).
+12 -25
View File
@@ -23,17 +23,6 @@ logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
def _require_admin(request: Request) -> dict:
"""Ensure the caller is an admin. Raises 403 otherwise."""
user = request.session.get("user")
if not user or not user.get("is_admin"):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
return user
AdminUser = Annotated[dict, Depends(_require_admin)]
@router.post("/google-drive/exchange-token") @router.post("/google-drive/exchange-token")
@require_login @require_login
async def exchange_google_drive_token( async def exchange_google_drive_token(
@@ -373,15 +362,15 @@ def format_time_remaining(time_delta):
@router.post("/google-drive/save-settings") @router.post("/google-drive/save-settings")
@require_login
async def save_google_drive_settings( async def save_google_drive_settings(
request: Request, request: Request,
refresh_token: Annotated[str, Form(...)], refresh_token: Annotated[str, Form(...)],
_admin: AdminUser,
db: Session = Depends(get_db),
client_id: Annotated[Optional[str], Form()] = None, client_id: Annotated[Optional[str], Form()] = None,
client_secret: Annotated[Optional[str], Form()] = None, client_secret: Annotated[Optional[str], Form()] = None,
folder_id: Annotated[Optional[str], Form()] = None, folder_id: Annotated[Optional[str], Form()] = None,
use_oauth: Annotated[str, Form()] = "true", use_oauth: Annotated[str, Form()] = "true",
db: Session = Depends(get_db),
): ):
""" """
Save Google Drive settings to the .env file (best-effort) and persist to database. Save Google Drive settings to the .env file (best-effort) and persist to database.
@@ -414,10 +403,9 @@ async def save_google_drive_settings(
if folder_id: if folder_id:
drive_settings["GOOGLE_DRIVE_FOLDER_ID"] = folder_id drive_settings["GOOGLE_DRIVE_FOLDER_ID"] = folder_id
# Best-effort .env file write — failures here are non-fatal # Try to update the .env file, but don't fail if it doesn't exist (for Docker containers)
env_file_written = False if os.path.exists(env_path):
try: try:
if os.path.exists(env_path):
logger.info(f"Updating Google Drive settings in {env_path}") logger.info(f"Updating Google Drive settings in {env_path}")
# Read the current .env file # Read the current .env file
@@ -450,13 +438,12 @@ async def save_google_drive_settings(
f.write("\n".join(new_env_lines) + "\n") f.write("\n".join(new_env_lines) + "\n")
logger.info("Successfully updated Google Drive settings in .env file") logger.info("Successfully updated Google Drive settings in .env file")
env_file_written = True except Exception as e:
else: logger.warning(f"Failed to update .env file: {str(e)}, but will continue with in-memory update")
logger.warning( else:
f".env file not found at {env_path}, skipping file update but continuing with in-memory update" logger.warning(
) f".env file not found at {env_path}, skipping file update but continuing with in-memory update"
except Exception as env_err: )
logger.warning(f"Failed to write .env file (non-fatal): {env_err}")
# Update the settings in memory (this always happens) # Update the settings in memory (this always happens)
if refresh_token: if refresh_token:
@@ -494,7 +481,7 @@ async def save_google_drive_settings(
return { return {
"status": "success", "status": "success",
"message": "Google Drive settings have been saved", "message": "Google Drive settings have been saved",
"in_memory_only": not env_file_written, "in_memory_only": not os.path.exists(env_path),
} }
except Exception as e: except Exception as e:
-6
View File
@@ -17,7 +17,6 @@ from sqlalchemy.orm import Session
from app.database import get_db from app.database import get_db
from app.models import UserImapAccount from app.models import UserImapAccount
from app.utils.encryption import decrypt_value, encrypt_value from app.utils.encryption import decrypt_value, encrypt_value
from app.utils.network import is_private_ip
from app.utils.subscription import get_tier, get_user_tier_id from app.utils.subscription import get_tier, get_user_tier_id
from app.utils.user_scope import get_current_owner_id from app.utils.user_scope import get_current_owner_id
@@ -188,11 +187,6 @@ def _test_imap_connection(host: str, port: int, username: str, password: str, us
Returns a dict with ``{"success": bool, "message": str}``. Returns a dict with ``{"success": bool, "message": str}``.
""" """
# Security: Prevent SSRF by blocking connections to internal IPs
if is_private_ip(host):
logger.warning("SSRF blocked: Attempt to connect to private IP %s", host)
return {"success": False, "message": "Connection error: Invalid hostname or IP address"}
try: try:
if use_ssl: if use_ssl:
mail = imaplib.IMAP4_SSL(host, port) mail = imaplib.IMAP4_SSL(host, port)
+1 -18
View File
@@ -515,12 +515,6 @@ def _test_imap_connection(config: dict[str, Any] | None, credentials: dict[str,
if not host or not username or not password: if not host or not username or not password:
return {"success": False, "message": "Missing required fields: host, username, and password"} return {"success": False, "message": "Missing required fields: host, username, and password"}
from app.utils.network import is_private_ip
if is_private_ip(host):
logger.warning("SSRF blocked: Attempt to connect to private IP %s", host)
return {"success": False, "message": "Connection error: Invalid hostname or IP address"}
try: try:
if use_ssl: if use_ssl:
mail = imaplib.IMAP4_SSL(host, port) mail = imaplib.IMAP4_SSL(host, port)
@@ -549,28 +543,17 @@ def _test_s3_connection(config: dict[str, Any] | None, credentials: dict[str, An
creds = credentials or {} creds = credentials or {}
bucket = cfg.get("bucket", "") bucket = cfg.get("bucket", "")
region = cfg.get("region", "us-east-1") region = cfg.get("region", "us-east-1")
endpoint_url = cfg.get("endpoint_url")
if not bucket: if not bucket:
return {"success": False, "message": "Missing required field: bucket"} return {"success": False, "message": "Missing required field: bucket"}
if endpoint_url:
from urllib.parse import urlparse
from app.utils.network import is_private_ip
parsed_url = urlparse(endpoint_url)
if parsed_url.hostname and is_private_ip(parsed_url.hostname):
logger.warning("SSRF blocked: Attempt to connect to private IP via S3 endpoint %s", endpoint_url)
return {"success": False, "message": "Connection error: Invalid endpoint URL or private IP"}
try: try:
client = boto3.client( client = boto3.client(
"s3", "s3",
region_name=region, region_name=region,
aws_access_key_id=creds.get("access_key_id", ""), aws_access_key_id=creds.get("access_key_id", ""),
aws_secret_access_key=creds.get("secret_access_key", ""), aws_secret_access_key=creds.get("secret_access_key", ""),
endpoint_url=endpoint_url, endpoint_url=cfg.get("endpoint_url"),
) )
client.head_bucket(Bucket=bucket) client.head_bucket(Bucket=bucket)
return {"success": True, "message": f"S3 bucket '{bucket}' is accessible"} return {"success": True, "message": f"S3 bucket '{bucket}' is accessible"}
+9 -9
View File
@@ -101,9 +101,9 @@ async def signup_page(request: Request) -> Any:
if not settings.allow_local_signup: if not settings.allow_local_signup:
return RedirectResponse(url="/login?error=Registration+is+not+enabled", status_code=302) return RedirectResponse(url="/login?error=Registration+is+not+enabled", status_code=302)
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"signup.html", "signup.html",
context={ {
"request": request,
"csrf_token": getattr(request.state, "csrf_token", ""), "csrf_token": getattr(request.state, "csrf_token", ""),
"app_version": settings.version, "app_version": settings.version,
}, },
@@ -113,16 +113,16 @@ async def signup_page(request: Request) -> Any:
@router.get("/verify-email-sent", include_in_schema=False) @router.get("/verify-email-sent", include_in_schema=False)
async def verify_email_sent_page(request: Request) -> Any: async def verify_email_sent_page(request: Request) -> Any:
"""Render the verify-email-sent confirmation page.""" """Render the verify-email-sent confirmation page."""
return templates.TemplateResponse(request, "verify_email_sent.html") return templates.TemplateResponse("verify_email_sent.html", {"request": request})
@router.get("/forgot-username", include_in_schema=False) @router.get("/forgot-username", include_in_schema=False)
async def forgot_username_page(request: Request) -> Any: async def forgot_username_page(request: Request) -> Any:
"""Render the forgot-username page where users can request a username reminder email.""" """Render the forgot-username page where users can request a username reminder email."""
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"forgot_username.html", "forgot_username.html",
context={ {
"request": request,
"csrf_token": getattr(request.state, "csrf_token", ""), "csrf_token": getattr(request.state, "csrf_token", ""),
"app_version": settings.version, "app_version": settings.version,
}, },
@@ -133,9 +133,9 @@ async def forgot_username_page(request: Request) -> Any:
async def forgot_password_page(request: Request) -> Any: async def forgot_password_page(request: Request) -> Any:
"""Render the forgot-password page where users can request a reset email.""" """Render the forgot-password page where users can request a reset email."""
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"forgot_password.html", "forgot_password.html",
context={ {
"request": request,
"csrf_token": getattr(request.state, "csrf_token", ""), "csrf_token": getattr(request.state, "csrf_token", ""),
"app_version": settings.version, "app_version": settings.version,
}, },
@@ -147,9 +147,9 @@ async def reset_password_page(request: Request) -> Any:
"""Render the password reset form page.""" """Render the password reset form page."""
token = request.query_params.get("token", "") token = request.query_params.get("token", "")
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"password_reset_form.html", "password_reset_form.html",
context={ {
"request": request,
"token": token, "token": token,
"csrf_token": getattr(request.state, "csrf_token", ""), "csrf_token": getattr(request.state, "csrf_token", ""),
"app_version": settings.version, "app_version": settings.version,
+2 -13
View File
@@ -25,17 +25,6 @@ logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
def _require_admin(request: Request) -> dict:
"""Ensure the caller is an admin. Raises 403 otherwise."""
user = request.session.get("user")
if not user or not user.get("is_admin"):
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Admin access required")
return user
AdminUser = Annotated[dict, Depends(_require_admin)]
@router.post("/onedrive/exchange-token") @router.post("/onedrive/exchange-token")
@require_login @require_login
async def exchange_onedrive_token( async def exchange_onedrive_token(
@@ -313,15 +302,15 @@ def format_time_remaining(time_delta):
@router.post("/onedrive/save-settings") @router.post("/onedrive/save-settings")
@require_login
async def save_onedrive_settings( async def save_onedrive_settings(
request: Request, request: Request,
refresh_token: Annotated[str, Form(...)], refresh_token: Annotated[str, Form(...)],
_admin: AdminUser,
db: Session = Depends(get_db),
client_id: Annotated[Optional[str], Form()] = None, client_id: Annotated[Optional[str], Form()] = None,
client_secret: Annotated[Optional[str], Form()] = None, client_secret: Annotated[Optional[str], Form()] = None,
tenant_id: Annotated[str, Form()] = "common", tenant_id: Annotated[str, Form()] = "common",
folder_path: Annotated[Optional[str], Form()] = None, folder_path: Annotated[Optional[str], Form()] = None,
db: Session = Depends(get_db),
): ):
""" """
Saves to database (primary) and .env file (best-effort). Saves to database (primary) and .env file (best-effort).
+3 -13
View File
@@ -9,7 +9,7 @@ Public endpoints:
""" """
import logging import logging
from datetime import datetime, time, timedelta, timezone from datetime import datetime, timezone
from typing import Annotated, Any from typing import Annotated, Any
from fastapi import APIRouter, Depends, HTTPException, Request, status from fastapi import APIRouter, Depends, HTTPException, Request, status
@@ -207,29 +207,19 @@ def platform_stats(request: Request, db: DbSession, _admin: AdminUser) -> dict[s
from app.models import FileRecord, UserProfile from app.models import FileRecord, UserProfile
today = datetime.now(timezone.utc).date() today = datetime.now(timezone.utc).date()
day_start = datetime.combine(today, time.min, tzinfo=timezone.utc)
day_end = day_start + timedelta(days=1)
month_start = day_start.replace(day=1)
if month_start.month == 12:
month_end = month_start.replace(year=month_start.year + 1, month=1)
else:
month_end = month_start.replace(month=month_start.month + 1)
# Total files # Total files
total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0 total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0
# Files today # Files today
files_today: int = ( files_today: int = (
db.query(func.count(FileRecord.id)) db.query(func.count(FileRecord.id)).filter(func.date(FileRecord.created_at) == today).scalar() or 0
.filter(FileRecord.created_at >= day_start, FileRecord.created_at < day_end)
.scalar()
or 0
) )
# Files this month # Files this month
files_this_month: int = ( files_this_month: int = (
db.query(func.count(FileRecord.id)) db.query(func.count(FileRecord.id))
.filter(FileRecord.created_at >= month_start, FileRecord.created_at < month_end) .filter(func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"))
.scalar() .scalar()
or 0 or 0
) )
-29
View File
@@ -28,10 +28,6 @@ logger = logging.getLogger(__name__)
router = APIRouter() router = APIRouter()
class UnsafeRedirectError(httpx.RequestError):
"""Raised when a redirect target fails URL safety checks."""
class URLUploadRequest(BaseModel): class URLUploadRequest(BaseModel):
"""Request model for URL-based file upload""" """Request model for URL-based file upload"""
@@ -110,26 +106,6 @@ def validate_file_type(content_type: str, filename: str) -> bool:
return False return False
async def verify_redirect(response: httpx.Response) -> None:
"""
Event hook to intercept redirects and validate the new destination URL.
Prevents SSRF bypasses via redirects to internal networks or metadata endpoints.
"""
if response.status_code in (301, 302, 303, 307, 308):
location = response.headers.get("Location")
if location:
# Resolve relative redirects
new_url = str(response.url.join(location))
# Validate the new URL
try:
validate_url_safety(new_url)
except HTTPException as e:
raise UnsafeRedirectError(
f"Redirect to unsafe URL blocked: {e.detail}",
request=response.request,
) from e
@router.post("/process-url") @router.post("/process-url")
@require_login @require_login
async def process_url( async def process_url(
@@ -186,7 +162,6 @@ async def process_url(
async with httpx.AsyncClient( async with httpx.AsyncClient(
timeout=settings.http_request_timeout, timeout=settings.http_request_timeout,
follow_redirects=True, follow_redirects=True,
event_hooks={"response": [verify_redirect]},
headers={ headers={
"User-Agent": "DocuElevate/1.0", # Identify ourselves "User-Agent": "DocuElevate/1.0", # Identify ourselves
}, },
@@ -276,10 +251,6 @@ async def process_url(
logger.error(f"HTTP error while downloading file from URL: {url} - {str(e)}") logger.error(f"HTTP error while downloading file from URL: {url} - {str(e)}")
raise HTTPException(status_code=e.response.status_code, detail=f"HTTP error: {str(e)}") raise HTTPException(status_code=e.response.status_code, detail=f"HTTP error: {str(e)}")
except UnsafeRedirectError as e:
logger.warning(f"Unsafe redirect blocked while downloading file from URL: {url} - {str(e)}")
raise HTTPException(status_code=400, detail=str(e))
except httpx.RequestError as e: except httpx.RequestError as e:
logger.error(f"Error downloading file from URL: {url} - {str(e)}") logger.error(f"Error downloading file from URL: {url} - {str(e)}")
raise HTTPException(status_code=500, detail=f"Failed to download file: {str(e)}") raise HTTPException(status_code=500, detail=f"Failed to download file: {str(e)}")
+2 -2
View File
@@ -536,9 +536,9 @@ async def login(request: Request):
return RedirectResponse(url="/oauth-login", status_code=status.HTTP_302_FOUND) return RedirectResponse(url="/oauth-login", status_code=status.HTTP_302_FOUND)
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"login.html", "login.html",
context={ {
"request": request,
"error": error, "error": error,
"message": message, "message": message,
"show_oauth": show_oauth, "show_oauth": show_oauth,
-1
View File
@@ -46,7 +46,6 @@ from app.tasks.translate_to_default_language import translate_to_default_languag
# Import new send tasks # Import new send tasks
from app.tasks.upload_to_dropbox import upload_to_dropbox # noqa: F401 from app.tasks.upload_to_dropbox import upload_to_dropbox # noqa: F401
from app.tasks.upload_to_email import upload_to_email # noqa: F401 from app.tasks.upload_to_email import upload_to_email # noqa: F401
from app.tasks.upload_to_evernote import upload_to_evernote # noqa: F401
from app.tasks.upload_to_ftp import upload_to_ftp # noqa: F401 from app.tasks.upload_to_ftp import upload_to_ftp # noqa: F401
from app.tasks.upload_to_google_drive import upload_to_google_drive # noqa: F401 from app.tasks.upload_to_google_drive import upload_to_google_drive # noqa: F401
from app.tasks.upload_to_icloud import upload_to_icloud # noqa: F401 from app.tasks.upload_to_icloud import upload_to_icloud # noqa: F401
-11
View File
@@ -154,17 +154,6 @@ class Settings(BaseSettings):
# "language": "Language", "correspondent": "Correspondent"} # "language": "Language", "correspondent": "Correspondent"}
paperless_custom_fields_mapping: Optional[str] = None paperless_custom_fields_mapping: Optional[str] = None
# Evernote destination settings
evernote_enabled: bool = Field(
default=True,
description="Enable Evernote as an upload destination. Set to False to disable uploads even when credentials are configured.",
)
evernote_auth_token: Optional[str] = None
evernote_sandbox: bool = False
evernote_notebook_guid: Optional[str] = None
evernote_default_tags: Optional[str] = None
evernote_include_metadata: bool = True
azure_ai_key: str azure_ai_key: str
azure_region: str azure_region: str
azure_endpoint: str azure_endpoint: str
+7 -12
View File
@@ -294,16 +294,10 @@ async def lifespan(app: FastAPI):
yield yield
# Shutdown: Cleanup tasks # Shutdown: Cleanup tasks
try: logging.info("Application shutting down")
logging.info("Application shutting down")
except Exception:
_startup_logger.exception("Error during shutdown logging")
# Send shutdown notification # Send shutdown notification
try: notify_shutdown()
notify_shutdown()
except Exception:
_startup_logger.exception("Error sending shutdown notification")
app = FastAPI( app = FastAPI(
@@ -430,13 +424,15 @@ async def http_exception_handler(request: Request, exc: HTTPException):
# For frontend routes, return appropriate HTML templates # For frontend routes, return appropriate HTML templates
# Handle 404 errors with a custom template # Handle 404 errors with a custom template
if exc.status_code == 404: if exc.status_code == 404:
return _error_templates.TemplateResponse(request, "404.html", status_code=status.HTTP_404_NOT_FOUND) return _error_templates.TemplateResponse(
"404.html", {"request": request}, status_code=status.HTTP_404_NOT_FOUND
)
# For other HTTP errors, we could create specific templates or use a generic one # For other HTTP errors, we could create specific templates or use a generic one
# For now, return a simple error page # For now, return a simple error page
return _error_templates.TemplateResponse( return _error_templates.TemplateResponse(
request,
"404.html", # Reuse 404 template for other errors, or create a generic error template "404.html", # Reuse 404 template for other errors, or create a generic error template
{"request": request},
status_code=exc.status_code, status_code=exc.status_code,
) )
@@ -456,9 +452,8 @@ async def custom_500_handler(request: Request, exc: Exception):
# Serve the 500 template for non-API routes # Serve the 500 template for non-API routes
return _error_templates.TemplateResponse( return _error_templates.TemplateResponse(
request,
"500.html", "500.html",
context={"exc": exc}, {"request": request, "exc": exc},
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
) )
+1 -1
View File
@@ -205,7 +205,7 @@ def convert_to_pdf(
".pdf", # PDF (already in PDF format but can be processed) ".pdf", # PDF (already in PDF format but can be processed)
} }
IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".tif", ".webp", ".svg"} IMAGE_EXTENSIONS = {".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".tif", ".webp", ".svg", ".heic", ".heif"}
HTML_EXTENSIONS = {".html", ".htm"} HTML_EXTENSIONS = {".html", ".htm"}
+1 -1
View File
@@ -78,7 +78,7 @@ def _convert_pdf_to_pdfa(input_path: str, output_path: str, pdfa_format: str = "
output_type, output_type,
"--quiet", "--quiet",
"--invalidate-digital-signatures", "--invalidate-digital-signatures",
"--", # end-of-options separator: prevents file paths from being interpreted as options "--",
input_path, input_path,
output_path, output_path,
] ]
+1 -1
View File
@@ -76,7 +76,7 @@ def extract_metadata_with_gpt(self, filename: str, cleaned_text: str, file_id: i
"Your task is to analyze the given text and return a well-structured JSON object.\n\n" "Your task is to analyze the given text and return a well-structured JSON object.\n\n"
"Extract and return the following fields:\n" "Extract and return the following fields:\n"
"1. **filename**: Machine-readable filename " "1. **filename**: Machine-readable filename "
"(YYYY-MM-DD_DescriptiveTitle, use only letters, numbers, periods, and underscores).\n" "(YYYY-MM-DD_DescriptiveTitle, use only letters, numbers, spaces, dashes, periods, and underscores).\n"
'2. **empfaenger**: The recipient, or "Unknown" if not found.\n' '2. **empfaenger**: The recipient, or "Unknown" if not found.\n'
'3. **absender**: The sender, or "Unknown" if not found.\n' '3. **absender**: The sender, or "Unknown" if not found.\n'
"4. **correspondent**: The entity or company that issued the document " "4. **correspondent**: The entity or company that issued the document "
-6
View File
@@ -18,7 +18,6 @@ from app.utils.allowed_types import (
DEFAULT_CATEGORIES, DEFAULT_CATEGORIES,
get_allowed_types_for_categories, get_allowed_types_for_categories,
) )
from app.utils.network import is_private_ip
# Database session for per-user IMAP accounts (imported lazily to avoid circular imports) # Database session for per-user IMAP accounts (imported lazily to avoid circular imports)
_db_session_factory = None _db_session_factory = None
@@ -406,11 +405,6 @@ def pull_inbox(
) )
processed_emails = load_processed_emails() processed_emails = load_processed_emails()
# Security: Prevent SSRF by blocking connections to internal IPs
if is_private_ip(host):
logger.warning("SSRF blocked: Attempt to pull mailbox from private IP %s", host)
return
try: try:
mail = imaplib.IMAP4_SSL(host, port) if use_ssl else imaplib.IMAP4(host, port) mail = imaplib.IMAP4_SSL(host, port) if use_ssl else imaplib.IMAP4(host, port)
mail.login(username, password) mail.login(username, password)
-12
View File
@@ -10,7 +10,6 @@ from app.models import FileRecord, IntegrationDirection, UserIntegration
from app.tasks.retry_config import BaseTaskWithRetry from app.tasks.retry_config import BaseTaskWithRetry
from app.tasks.upload_to_dropbox import upload_to_dropbox from app.tasks.upload_to_dropbox import upload_to_dropbox
from app.tasks.upload_to_email import upload_to_email from app.tasks.upload_to_email import upload_to_email
from app.tasks.upload_to_evernote import upload_to_evernote
from app.tasks.upload_to_ftp import upload_to_ftp from app.tasks.upload_to_ftp import upload_to_ftp
from app.tasks.upload_to_google_drive import upload_to_google_drive from app.tasks.upload_to_google_drive import upload_to_google_drive
from app.tasks.upload_to_icloud import upload_to_icloud from app.tasks.upload_to_icloud import upload_to_icloud
@@ -101,11 +100,6 @@ def _should_upload_to_email():
) )
def _should_upload_to_evernote():
token = getattr(settings, "evernote_auth_token", None)
return bool(getattr(settings, "evernote_enabled", True) and isinstance(token, str) and token.strip())
def _should_upload_to_onedrive(): def _should_upload_to_onedrive():
return bool( return bool(
getattr(settings, "onedrive_enabled", True) getattr(settings, "onedrive_enabled", True)
@@ -157,7 +151,6 @@ def get_configured_services_from_validator():
"FTP Storage": "ftp", "FTP Storage": "ftp",
"SFTP Storage": "sftp", "SFTP Storage": "sftp",
"Email": "email", "Email": "email",
"Evernote": "evernote",
"OneDrive": "onedrive", "OneDrive": "onedrive",
"S3 Storage": "s3", "S3 Storage": "s3",
"SharePoint": "sharepoint", "SharePoint": "sharepoint",
@@ -261,11 +254,6 @@ def send_to_all_destinations(self, file_path: str, use_validator=True, file_id:
"should_upload": _should_upload_to_email, "should_upload": _should_upload_to_email,
"upload_func": upload_to_email, "upload_func": upload_to_email,
}, },
{
"name": "evernote",
"should_upload": _should_upload_to_evernote,
"upload_func": upload_to_evernote,
},
{ {
"name": "onedrive", "name": "onedrive",
"should_upload": _should_upload_to_onedrive, "should_upload": _should_upload_to_onedrive,
-242
View File
@@ -1,242 +0,0 @@
#!/usr/bin/env python3
import hashlib
import json
import logging
import mimetypes
import os
from html import escape
from typing import Any
from app.celery_app import celery
from app.config import settings
from app.tasks.retry_config import UploadTaskWithRetry
from app.utils import log_task_progress
logger = logging.getLogger(__name__)
_UNKNOWN_PLACEHOLDERS = {"", "Unknown", "unknown", "N/A", "n/a", "None", "none", "null"}
_MAX_EVERNOTE_TITLE_LENGTH = 255
def _get_evernote_sdk():
"""Import the Evernote SDK lazily so the missing dependency error is actionable."""
try:
from evernote.edam.notestore import NoteStore
from evernote.edam.type import ttypes as Types
from evernote.edam.userstore import UserStore
from thrift.protocol import TBinaryProtocol
from thrift.transport import THttpClient
except ImportError as exc:
raise RuntimeError("Evernote upload requires the evernote3 package. Install requirements.txt again.") from exc
return NoteStore, Types, UserStore, TBinaryProtocol, THttpClient
def _build_thrift_client(client_cls, url: str, binary_protocol, http_transport):
transport = http_transport.THttpClient(url)
protocol = binary_protocol.TBinaryProtocol(transport)
return client_cls(protocol)
def _get_note_store(auth_token: str):
NoteStore, Types, UserStore, TBinaryProtocol, THttpClient = _get_evernote_sdk()
base_url = (
"https://sandbox.evernote.com" if getattr(settings, "evernote_sandbox", False) else "https://www.evernote.com"
)
user_store = _build_thrift_client(UserStore.Client, f"{base_url}/edam/user", TBinaryProtocol, THttpClient)
user = user_store.getUser(auth_token)
shard_id = getattr(user, "shardId", None)
if not shard_id:
raise RuntimeError("Evernote user response did not include a shard ID")
note_store_url = f"{base_url}/shard/{shard_id}/notestore"
note_store = _build_thrift_client(NoteStore.Client, note_store_url, TBinaryProtocol, THttpClient)
return note_store, Types
def _load_metadata(file_path: str) -> dict[str, Any]:
"""Load extracted DocuElevate metadata from the companion JSON file, when present."""
json_path = os.path.splitext(file_path)[0] + ".json"
if not os.path.exists(json_path):
return {}
try:
with open(json_path, "r", encoding="utf-8") as metadata_file:
data = json.load(metadata_file)
except Exception as exc: # noqa: BLE001
logger.warning("Failed to load Evernote metadata from %s: %s", json_path, exc)
return {}
return data if isinstance(data, dict) else {}
def _normalize_metadata_value(value: Any) -> str:
if value is None:
return ""
if isinstance(value, (list, tuple, set)):
normalized = ", ".join(str(item) for item in value if item is not None)
elif isinstance(value, dict):
normalized = json.dumps(value, ensure_ascii=False, sort_keys=True)
else:
normalized = str(value)
normalized = normalized.strip()
return "" if normalized in _UNKNOWN_PLACEHOLDERS else normalized
def _metadata_rows(metadata: dict[str, Any]) -> list[tuple[str, str]]:
rows = []
for key in sorted(metadata):
value = _normalize_metadata_value(metadata[key])
if value:
rows.append((key, value))
return rows
def _extract_tags(metadata: dict[str, Any]) -> list[str]:
tags: list[str] = []
def add_tag(value: Any) -> None:
normalized = _normalize_metadata_value(value)
if normalized and normalized not in tags:
tags.append(normalized)
default_tags = getattr(settings, "evernote_default_tags", None)
if default_tags:
for tag in str(default_tags).split(","):
add_tag(tag)
metadata_tags = metadata.get("tags")
if isinstance(metadata_tags, str):
for tag in metadata_tags.split(","):
add_tag(tag)
elif isinstance(metadata_tags, (list, tuple, set)):
for tag in metadata_tags:
add_tag(tag)
return tags
def _note_title(file_path: str, metadata: dict[str, Any]) -> str:
title = (
_normalize_metadata_value(metadata.get("title"))
or _normalize_metadata_value(metadata.get("filename"))
or os.path.basename(file_path)
)
return title[:_MAX_EVERNOTE_TITLE_LENGTH]
def _build_enml(metadata: dict[str, Any], resource_hash: str, resource_mime: str, include_metadata: bool) -> str:
body_parts = ['<?xml version="1.0" encoding="UTF-8"?>']
body_parts.append('<!DOCTYPE en-note SYSTEM "http://xml.evernote.com/pub/enml2.dtd">')
body_parts.append("<en-note>")
if include_metadata:
rows = _metadata_rows(metadata)
if rows:
body_parts.append("<div><b>DocuElevate metadata</b></div>")
for key, value in rows:
body_parts.append(f"<div><b>{escape(key)}:</b> {escape(value)}</div>")
body_parts.append("<br/>")
body_parts.append(f'<en-media type="{escape(resource_mime)}" hash="{resource_hash}"/>')
body_parts.append("</en-note>")
return "".join(body_parts)
def _create_evernote_note(file_path: str, metadata: dict[str, Any], task_id: str):
auth_token = getattr(settings, "evernote_auth_token", None)
if not auth_token:
raise ValueError("Evernote auth token is not configured (EVERNOTE_AUTH_TOKEN)")
note_store, Types = _get_note_store(auth_token)
filename = os.path.basename(file_path)
with open(file_path, "rb") as pdf_file:
resource_body = pdf_file.read()
body_hash = hashlib.md5(resource_body).digest() # noqa: S324 - Evernote API requires MD5 resource hashes.
body_hash_hex = hashlib.md5(resource_body).hexdigest() # noqa: S324 - Evernote ENML references MD5 hashes.
resource_mime = mimetypes.guess_type(filename)[0] or "application/pdf"
data = Types.Data()
data.size = len(resource_body)
data.bodyHash = body_hash
data.body = resource_body
resource = Types.Resource()
resource.mime = resource_mime
resource.data = data
resource.attributes = Types.ResourceAttributes(fileName=filename)
note = Types.Note()
note.title = _note_title(file_path, metadata)
note.content = _build_enml(
metadata,
body_hash_hex,
resource_mime,
include_metadata=getattr(settings, "evernote_include_metadata", True),
)
note.resources = [resource]
notebook_guid = getattr(settings, "evernote_notebook_guid", None)
if notebook_guid:
note.notebookGuid = notebook_guid
tag_names = _extract_tags(metadata)
if tag_names:
note.tagNames = tag_names
created_note = note_store.createNote(auth_token, note)
logger.info("[%s] Created Evernote note %s for %s", task_id, getattr(created_note, "guid", None), file_path)
return created_note
@celery.task(base=UploadTaskWithRetry, bind=True)
def upload_to_evernote(self, file_path: str, file_id: int = None):
"""
Upload a document to Evernote by creating a note with metadata and a PDF attachment.
Args:
file_path: Path to the PDF file to upload
file_id: Optional file ID to associate with logs
"""
task_id = self.request.id
filename = os.path.basename(file_path)
logger.info("[%s] Starting Evernote upload: %s", task_id, file_path)
log_task_progress(
task_id, "upload_to_evernote", "in_progress", f"Uploading to Evernote: {filename}", file_id=file_id
)
if not os.path.exists(file_path):
error_msg = f"File not found: {file_path}"
logger.error("[%s] %s", task_id, error_msg)
log_task_progress(task_id, "upload_to_evernote", "failure", error_msg, file_id=file_id)
raise FileNotFoundError(error_msg)
if not getattr(settings, "evernote_auth_token", None):
error_msg = "Evernote auth token is not configured (EVERNOTE_AUTH_TOKEN)"
logger.error("[%s] %s", task_id, error_msg)
log_task_progress(task_id, "upload_to_evernote", "failure", error_msg, file_id=file_id)
raise ValueError(error_msg)
try:
metadata = _load_metadata(file_path)
created_note = _create_evernote_note(file_path, metadata, task_id)
except Exception as exc:
error_msg = f"Failed to upload to Evernote: {exc}"
logger.error("[%s] %s", task_id, error_msg)
log_task_progress(task_id, "upload_to_evernote", "failure", error_msg, file_id=file_id)
raise
note_guid = getattr(created_note, "guid", None)
log_task_progress(task_id, "upload_to_evernote", "success", f"Uploaded to Evernote: {note_guid}", file_id=file_id)
return {
"status": "Completed",
"file_path": file_path,
"evernote_note_guid": note_guid,
"evernote_title": getattr(created_note, "title", None),
"evernote_notebook_guid": getattr(created_note, "notebookGuid", None),
}
+2 -1
View File
@@ -555,8 +555,9 @@ def _upload_rclone(file_path: str, cfg: dict[str, Any], creds: dict[str, Any], t
dest = dest.replace("//", "/") dest = dest.replace("//", "/")
try: try:
# SECURITY: Separate options from positional arguments using -- to prevent command injection
result = subprocess.run( # nosec B603 # noqa: S603 S607 result = subprocess.run( # nosec B603 # noqa: S603 S607
["rclone", "copyto", f"--config={conf_path}", file_path, dest], # noqa: S603 S607 ["rclone", "copyto", f"--config={conf_path}", "--", file_path, dest], # noqa: S603 S607
capture_output=True, capture_output=True,
text=True, text=True,
timeout=300, timeout=300,
-15
View File
@@ -174,21 +174,6 @@ def get_provider_status() -> dict[str, dict[str, object]]:
}, },
} }
providers["Evernote"] = {
"name": "Evernote",
"icon": "fa-brands fa-evernote",
"configured": bool(getattr(settings, "evernote_auth_token", None)),
"enabled": getattr(settings, "evernote_enabled", True),
"description": "Create Evernote notes with document metadata and PDF attachments",
"details": {
"auth_token": mask_sensitive_value(getattr(settings, "evernote_auth_token", None)),
"sandbox": getattr(settings, "evernote_sandbox", False),
"notebook_guid": getattr(settings, "evernote_notebook_guid", "Not set"),
"default_tags": getattr(settings, "evernote_default_tags", "Not set"),
"include_metadata": getattr(settings, "evernote_include_metadata", True),
},
}
# Add FTP configuration to providers # Add FTP configuration to providers
providers["FTP Storage"] = { providers["FTP Storage"] = {
"name": "FTP Storage", "name": "FTP Storage",
-6
View File
@@ -145,12 +145,6 @@ def validate_storage_configs() -> dict[str, list[str]]:
email_issues.append("DEST_EMAIL_DEFAULT_RECIPIENT is not configured") email_issues.append("DEST_EMAIL_DEFAULT_RECIPIENT is not configured")
issues["email"] = email_issues issues["email"] = email_issues
# Validate Evernote
evernote_issues = []
if not getattr(settings, "evernote_auth_token", None):
evernote_issues.append("EVERNOTE_AUTH_TOKEN is not configured")
issues["evernote"] = evernote_issues
# Validate S3 # Validate S3
s3_issues = [] s3_issues = []
if not getattr(settings, "s3_bucket_name", None): if not getattr(settings, "s3_bucket_name", None):
+3 -4
View File
@@ -15,7 +15,7 @@ import logging
import re import re
from typing import Any from typing import Any
from sqlalchemy import MetaData, create_engine, func, inspect, select, table from sqlalchemy import MetaData, create_engine, inspect, text
from sqlalchemy.engine import Engine from sqlalchemy.engine import Engine
from sqlalchemy.engine.url import make_url from sqlalchemy.engine.url import make_url
from sqlalchemy.orm import sessionmaker from sqlalchemy.orm import sessionmaker
@@ -89,9 +89,8 @@ def preview_migration(source_url: str) -> dict[str, Any]:
logger.warning(f"Skipping table with invalid name format: {table_name}") logger.warning(f"Skipping table with invalid name format: {table_name}")
continue continue
# table_name is safe — sourced from inspect().get_table_names(), not user input # table_name is safe — sourced from inspect().get_table_names(), not user input
t = table(table_name) quoted_table = conn.dialect.identifier_preparer.quote(table_name)
query = select(func.count()).select_from(t) row = conn.execute(text(f"SELECT COUNT(*) FROM {quoted_table}")).fetchone() # noqa: S608
row = conn.execute(query).fetchone()
count = row[0] if row else 0 count = row[0] if row else 0
result.append({"name": table_name, "row_count": count}) result.append({"name": table_name, "row_count": count})
total += count total += count
+2 -1
View File
@@ -1,10 +1,11 @@
import logging import logging
import os import os
from typing import Dict
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def update_env_file(settings_to_update: dict[str, str]) -> bool: def update_env_file(settings_to_update: Dict[str, str]) -> bool:
""" """
Updates the .env file with the given settings (best-effort). Updates the .env file with the given settings (best-effort).
Creates or modifies existing keys. Creates or modifies existing keys.
+1 -12
View File
@@ -7,19 +7,8 @@ def hash_file(filepath: str | Path, chunk_size: int = 65536) -> str:
Returns the SHA-256 hash of the file at 'filepath'. Returns the SHA-256 hash of the file at 'filepath'.
Reads the file in chunks to handle large files efficiently. Reads the file in chunks to handle large files efficiently.
""" """
from app.config import settings
filepath_obj = Path(filepath).resolve()
workdir_obj = Path(settings.workdir).resolve()
# Security check: Ensure the resolved path is strictly within the allowed workdir
try:
filepath_obj.relative_to(workdir_obj)
except ValueError:
raise FileNotFoundError(f"Access denied: path traversal attempt or file outside workdir '{filepath}'")
sha256 = hashlib.sha256() sha256 = hashlib.sha256()
with open(filepath_obj, "rb") as f: with open(filepath, "rb") as f:
while True: while True:
data = f.read(chunk_size) data = f.read(chunk_size)
if not data: if not data:
+5 -31
View File
@@ -1,7 +1,6 @@
import ipaddress import ipaddress
import logging import logging
import socket import socket
from urllib.parse import urlsplit, urlunsplit
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -28,33 +27,8 @@ def is_private_ip(hostname: str) -> bool:
return True return True
return False return False
except (socket.gaierror, socket.error): except (socket.gaierror, socket.error):
# Cannot resolve. # Cannot resolve - allow for testing/development
# Fail securely: block unresolved domains to prevent DNS rebinding # In production, DNS should work properly
# and SSRF bypasses via unresolvable addresses. # Log this for debugging
logger.warning(f"Could not resolve hostname (blocking securely): {hostname}") logger.warning(f"Could not resolve hostname: {hostname}")
return True return False # Changed from True to False to allow external domains in tests
def join_url(base: str, *parts: str) -> str:
"""
Safely join a base URL with one or more path parts.
Uses urllib.parse to correctly handle scheme/netloc/query/fragment so that
only the path component is modified. Leading and trailing slashes are
stripped from each part before joining, preventing double-slash sequences
at segment boundaries without touching the scheme separator or query string.
Examples:
join_url("https://example.com/dav/", "/remote/", "file.pdf")
-> "https://example.com/dav/remote/file.pdf"
"""
parsed = urlsplit(base)
# Strip each part once and filter out empty segments; use walrus operator
# to avoid calling strip twice per iteration.
stripped_parts = [s for p in parts if (s := p.strip("/"))]
base_path = parsed.path.rstrip("/")
new_path = base_path + "/" + "/".join(stripped_parts) if stripped_parts else base_path
# Ensure path is non-empty so the reconstructed URL is valid.
if not new_path:
new_path = "/"
return urlunsplit((parsed.scheme, parsed.netloc, new_path, parsed.query, parsed.fragment))
-49
View File
@@ -1089,55 +1089,6 @@ SETTING_METADATA = {
"required": False, "required": False,
"restart_required": False, "restart_required": False,
}, },
# Storage Providers - Evernote
"evernote_enabled": {
"category": "Storage Providers",
"description": "Enable Evernote as an upload destination. When disabled, no documents will be sent to Evernote even if credentials are configured.",
"type": "boolean",
"sensitive": False,
"required": False,
"restart_required": False,
},
"evernote_auth_token": {
"category": "Storage Providers",
"description": "Evernote developer token for note creation",
"type": "string",
"sensitive": True,
"required": False,
"restart_required": False,
},
"evernote_sandbox": {
"category": "Storage Providers",
"description": "Use the Evernote sandbox environment instead of production",
"type": "boolean",
"sensitive": False,
"required": False,
"restart_required": False,
},
"evernote_notebook_guid": {
"category": "Storage Providers",
"description": "Optional Evernote notebook GUID for uploaded notes",
"type": "string",
"sensitive": False,
"required": False,
"restart_required": False,
},
"evernote_default_tags": {
"category": "Storage Providers",
"description": "Comma-separated Evernote tags to apply to uploaded notes",
"type": "string",
"sensitive": False,
"required": False,
"restart_required": False,
},
"evernote_include_metadata": {
"category": "Storage Providers",
"description": "Include extracted document metadata in Evernote note content",
"type": "boolean",
"sensitive": False,
"required": False,
"restart_required": False,
},
# Storage Providers - Google Drive # Storage Providers - Google Drive
"google_drive_enabled": { "google_drive_enabled": {
"category": "Storage Providers", "category": "Storage Providers",
+5 -21
View File
@@ -29,7 +29,7 @@ At average usage (~40 % of quota) margins improve to 55-65 % after tax.
from __future__ import annotations from __future__ import annotations
import logging import logging
from datetime import date, datetime, time, timedelta, timezone from datetime import date, datetime, timezone
from typing import Any from typing import Any
from sqlalchemy import func from sqlalchemy import func
@@ -317,20 +317,6 @@ def _today_utc() -> date:
return datetime.now(timezone.utc).date() return datetime.now(timezone.utc).date()
def _day_bounds_utc(day: date) -> tuple[datetime, datetime]:
start = datetime.combine(day, time.min, tzinfo=timezone.utc)
return start, start + timedelta(days=1)
def _month_bounds_utc(day: date) -> tuple[datetime, datetime]:
start = datetime.combine(day.replace(day=1), time.min, tzinfo=timezone.utc)
if start.month == 12:
end = start.replace(year=start.year + 1, month=1)
else:
end = start.replace(month=start.month + 1)
return start, end
def _scalar_count(query: Any) -> int: def _scalar_count(query: Any) -> int:
"""Execute a count query and return an int, defaulting to 0 for NULL.""" """Execute a count query and return an int, defaulting to 0 for NULL."""
return query.scalar() or 0 return query.scalar() or 0
@@ -349,13 +335,12 @@ def get_today_file_count(db: Session, owner_id: str) -> int:
"""Files processed by this user today (UTC, not counting duplicates).""" """Files processed by this user today (UTC, not counting duplicates)."""
from app.models import FileRecord from app.models import FileRecord
day_start, day_end = _day_bounds_utc(_today_utc()) today = _today_utc()
return _scalar_count( return _scalar_count(
db.query(func.count(FileRecord.id)).filter( db.query(func.count(FileRecord.id)).filter(
FileRecord.owner_id == owner_id, FileRecord.owner_id == owner_id,
FileRecord.is_duplicate.is_(False), FileRecord.is_duplicate.is_(False),
FileRecord.created_at >= day_start, func.date(FileRecord.created_at) == today,
FileRecord.created_at < day_end,
) )
) )
@@ -364,13 +349,12 @@ def get_month_file_count(db: Session, owner_id: str) -> int:
"""Files processed by this user this calendar month (UTC, not counting duplicates).""" """Files processed by this user this calendar month (UTC, not counting duplicates)."""
from app.models import FileRecord from app.models import FileRecord
month_start, month_end = _month_bounds_utc(_today_utc()) today = _today_utc()
return _scalar_count( return _scalar_count(
db.query(func.count(FileRecord.id)).filter( db.query(func.count(FileRecord.id)).filter(
FileRecord.owner_id == owner_id, FileRecord.owner_id == owner_id,
FileRecord.is_duplicate.is_(False), FileRecord.is_duplicate.is_(False),
FileRecord.created_at >= month_start, func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"),
FileRecord.created_at < month_end,
) )
) )
-21
View File
@@ -12,13 +12,11 @@ import smtplib
from email.mime.multipart import MIMEMultipart from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText from email.mime.text import MIMEText
from typing import Any from typing import Any
from urllib.parse import urlparse
import httpx import httpx
from app.database import SessionLocal from app.database import SessionLocal
from app.models import InAppNotification, UserNotificationPreference, UserNotificationTarget from app.models import InAppNotification, UserNotificationPreference, UserNotificationTarget
from app.utils.network import is_private_ip
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -30,11 +28,6 @@ USER_EVENT_LABELS: dict[str, str] = {
EVENT_DOCUMENT_PROCESSED: "Document Processed", EVENT_DOCUMENT_PROCESSED: "Document Processed",
EVENT_DOCUMENT_FAILED: "Document Processing Failed", EVENT_DOCUMENT_FAILED: "Document Processing Failed",
} }
METADATA_ENDPOINTS = {
"169.254.169.254",
"169.254.169.253",
"metadata.google.internal",
}
def create_in_app_notification( def create_in_app_notification(
@@ -135,20 +128,6 @@ def _send_webhook_notification(target_config: dict[str, Any], event_type: str, t
logger.warning("Webhook notification target missing url") logger.warning("Webhook notification target missing url")
return False return False
parsed_url = urlparse(url)
if parsed_url.scheme not in {"http", "https"}:
logger.warning("Webhook notification to %s blocked: invalid scheme %s", url, parsed_url.scheme)
return False
hostname = parsed_url.hostname
if not hostname:
logger.warning("Webhook notification to %s blocked: missing hostname", url)
return False
if hostname in METADATA_ENDPOINTS or is_private_ip(hostname):
logger.warning("Webhook notification to %s blocked: private or metadata endpoint", url)
return False
payload = { payload = {
"event": event_type, "event": event_type,
"title": title, "title": title,
-21
View File
@@ -18,13 +18,11 @@ import json
import logging import logging
import time import time
from typing import Any from typing import Any
from urllib.parse import urlparse
import requests import requests
from app.database import SessionLocal from app.database import SessionLocal
from app.models import WebhookConfig from app.models import WebhookConfig
from app.utils.network import is_private_ip
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -42,11 +40,6 @@ VALID_EVENTS: frozenset[str] = frozenset(
#: Timeout (seconds) for outgoing webhook HTTP requests. #: Timeout (seconds) for outgoing webhook HTTP requests.
WEBHOOK_TIMEOUT = 10 WEBHOOK_TIMEOUT = 10
METADATA_ENDPOINTS = {
"169.254.169.254",
"169.254.169.253",
"metadata.google.internal",
}
def compute_signature(payload_bytes: bytes, secret: str) -> str: def compute_signature(payload_bytes: bytes, secret: str) -> str:
@@ -74,20 +67,6 @@ def deliver_webhook(url: str, payload: dict[str, Any], secret: str | None = None
Returns: Returns:
``True`` when the remote server responds with a 2xx status. ``True`` when the remote server responds with a 2xx status.
""" """
parsed_url = urlparse(url)
if parsed_url.scheme not in {"http", "https"}:
logger.warning("Webhook to %s blocked: invalid scheme %s", url, parsed_url.scheme)
return False
hostname = parsed_url.hostname
if not hostname:
logger.warning("Webhook to %s blocked: missing hostname", url)
return False
if hostname in METADATA_ENDPOINTS or is_private_ip(hostname):
logger.warning("Webhook to %s blocked: private or metadata endpoint", url)
return False
body = json.dumps(payload, default=str, sort_keys=True) body = json.dumps(payload, default=str, sort_keys=True)
body_bytes = body.encode("utf-8") body_bytes = body.encode("utf-8")
+5 -29
View File
@@ -162,36 +162,12 @@ def _inject_global_context(ctx: dict) -> None:
def template_response_with_version(*args, **kwargs): def template_response_with_version(*args, **kwargs):
"""Wrapper for TemplateResponse to include version and CSRF token in all templates. """Wrapper for TemplateResponse to include version and CSRF token in all templates"""
# If context dict is provided, add version to it
Handles both old-style and new-style Starlette TemplateResponse calls: if len(args) >= 2 and isinstance(args[1], dict):
- Old-style (Starlette <1.0): TemplateResponse(name, {"request": req, ...}, ...) _inject_global_context(args[1])
- New-style (Starlette 1.0+): TemplateResponse(request, name, context={...}, ...) elif "context" in kwargs and isinstance(kwargs["context"], dict):
"""
if len(args) >= 1 and isinstance(args[0], str):
# Old-style call: first positional arg is the template name (string).
# Convert to new-style: (request, name, context=..., ...)
name = args[0]
if len(args) >= 2 and isinstance(args[1], dict):
context = args[1]
# Old-style may have status_code as 3rd positional arg
if len(args) >= 3 and "status_code" not in kwargs:
kwargs["status_code"] = args[2]
else:
context = kwargs.pop("context", {})
request_obj = context.pop("request", None)
if request_obj is not None:
context["request"] = request_obj
_inject_global_context(context)
if request_obj is not None:
return original_template_response(request_obj, name, context=context, **kwargs)
return original_template_response(name, context=context, **kwargs)
# New-style call: (request, name, context=..., ...)
if "context" in kwargs and isinstance(kwargs["context"], dict):
_inject_global_context(kwargs["context"]) _inject_global_context(kwargs["context"])
elif len(args) >= 3 and isinstance(args[2], dict):
_inject_global_context(args[2])
return original_template_response(*args, **kwargs) return original_template_response(*args, **kwargs)
-2
View File
@@ -717,7 +717,6 @@ def _compute_processing_flow(logs, pipeline_steps=None):
"upload_to_ftp": "FTP Storage", "upload_to_ftp": "FTP Storage",
"upload_to_sftp": "SFTP Storage", "upload_to_sftp": "SFTP Storage",
"upload_to_email": "Email", "upload_to_email": "Email",
"upload_to_evernote": "Evernote",
"queue_dropbox": "Dropbox", "queue_dropbox": "Dropbox",
"queue_nextcloud": "Nextcloud", "queue_nextcloud": "Nextcloud",
"queue_paperless": "Paperless-ngx", "queue_paperless": "Paperless-ngx",
@@ -728,7 +727,6 @@ def _compute_processing_flow(logs, pipeline_steps=None):
"queue_ftp": "FTP Storage", "queue_ftp": "FTP Storage",
"queue_sftp": "SFTP Storage", "queue_sftp": "SFTP Storage",
"queue_email": "Email", "queue_email": "Email",
"queue_evernote": "Evernote",
} }
# Create a map of step names to their log entries # Create a map of step names to their log entries
+3 -13
View File
@@ -2,7 +2,7 @@
General routes for the application homepage and basic pages. General routes for the application homepage and basic pages.
""" """
from datetime import date, datetime, time, timedelta, timezone from datetime import date, datetime, timezone
from pathlib import Path from pathlib import Path
from fastapi import Depends, HTTPException, Request from fastapi import Depends, HTTPException, Request
@@ -64,27 +64,17 @@ async def serve_index(request: Request, db: Session = Depends(get_db)):
user = request.session.get("user") or {} user = request.session.get("user") or {}
is_admin = user.get("is_admin", False) is_admin = user.get("is_admin", False)
day_start = datetime.combine(today, time.min, tzinfo=timezone.utc)
day_end = day_start + timedelta(days=1)
month_start = day_start.replace(day=1)
if month_start.month == 12:
month_end = month_start.replace(year=month_start.year + 1, month=1)
else:
month_end = month_start.replace(month=month_start.month + 1)
try: try:
total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0 total_files: int = db.query(func.count(FileRecord.id)).scalar() or 0
files_today: int = ( files_today: int = (
db.query(func.count(FileRecord.id)) db.query(func.count(FileRecord.id)).filter(func.date(FileRecord.created_at) == today).scalar() or 0
.filter(FileRecord.created_at >= day_start, FileRecord.created_at < day_end)
.scalar()
or 0
) )
files_month: int = ( files_month: int = (
db.query(func.count(FileRecord.id)) db.query(func.count(FileRecord.id))
.filter(FileRecord.created_at >= month_start, FileRecord.created_at < month_end) .filter(func.strftime("%Y-%m", FileRecord.created_at) == today.strftime("%Y-%m"))
.scalar() .scalar()
or 0 or 0
) )
+1 -2
View File
@@ -23,7 +23,6 @@ templates = Jinja2Templates(directory=str(_templates_dir))
async def shared_link_view(request: Request, token: str): async def shared_link_view(request: Request, token: str):
"""Render the public share landing page for a given token.""" """Render the public share landing page for a given token."""
return templates.TemplateResponse( return templates.TemplateResponse(
request,
"shared_link_view.html", "shared_link_view.html",
context={"token": token}, {"request": request, "token": token},
) )
+25 -4
View File
@@ -3,7 +3,7 @@ services:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
container_name: document_api # No container_name — allows `docker compose up --scale api=N`
restart: always restart: always
# We'll keep the code in /app, but set working_dir to the shared data directory # We'll keep the code in /app, but set working_dir to the shared data directory
@@ -24,7 +24,7 @@ services:
depends_on: depends_on:
- redis - redis
- worker - beat
# Mount the shared working directory for data # Mount the shared working directory for data
volumes: volumes:
@@ -34,13 +34,14 @@ services:
build: build:
context: . context: .
dockerfile: Dockerfile dockerfile: Dockerfile
container_name: document_worker # No container_name — allows `docker compose up --scale worker=N`
restart: always restart: always
# same shared working directory # same shared working directory
working_dir: /workdir working_dir: /workdir
command: ["celery", "-A", "app.celery_worker", "worker", "-B", "--loglevel=info", "-Q", "document_processor,default,celery"] # Workers process tasks only — no -B flag (Beat runs in the dedicated beat service)
command: ["celery", "-A", "app.celery_worker", "worker", "--loglevel=info", "-Q", "document_processor,default,celery"]
env_file: env_file:
- .env - .env
environment: environment:
@@ -54,6 +55,26 @@ services:
volumes: volumes:
- /var/docparse/workdir:/workdir - /var/docparse/workdir:/workdir
# Dedicated Celery Beat scheduler — exactly one instance must run at all times.
# Beat publishes periodic tasks to the Redis broker; workers pick them up.
# Do NOT scale this service (replicas must stay at 1).
beat:
build:
context: .
dockerfile: Dockerfile
container_name: document_beat
restart: always
working_dir: /workdir
command: ["celery", "-A", "app.celery_worker", "beat", "--loglevel=info"]
env_file:
- .env
environment:
- PYTHONPATH=/app
depends_on:
- redis
volumes:
- /var/docparse/workdir:/workdir
gotenberg: gotenberg:
image: gotenberg/gotenberg:latest image: gotenberg/gotenberg:latest
container_name: gotenberg container_name: gotenberg
-13
View File
@@ -1330,19 +1330,6 @@ For detailed setup instructions, see the [Google Drive Setup Guide](GoogleDriveS
| `DEST_EMAIL_SENDER` | From address for delivered documents (e.g., `"DocuElevate Delivery <docuelevate@example.com>"`). | | `DEST_EMAIL_SENDER` | From address for delivered documents (e.g., `"DocuElevate Delivery <docuelevate@example.com>"`). |
| `DEST_EMAIL_DEFAULT_RECIPIENT` | Fallback recipient email when none is specified for a delivery task. | | `DEST_EMAIL_DEFAULT_RECIPIENT` | Fallback recipient email when none is specified for a delivery task. |
### Evernote
| **Variable** | **Description** |
|-------------------------------|---------------------------------------------------------------------|
| `EVERNOTE_ENABLED` | Set to `false` to disable Evernote uploads without removing credentials. Default: `true` |
| `EVERNOTE_AUTH_TOKEN` | Evernote developer token or OAuth access token used to create notes. |
| `EVERNOTE_SANDBOX` | Use Evernote sandbox API endpoints. Default: `false` |
| `EVERNOTE_NOTEBOOK_GUID` | Optional target notebook GUID. If omitted, Evernote uses the default notebook. |
| `EVERNOTE_DEFAULT_TAGS` | Optional comma-separated tags applied to every created note. |
| `EVERNOTE_INCLUDE_METADATA` | Include extracted metadata in the Evernote note body. Default: `true` |
For detailed setup instructions, see the [Evernote Setup Guide](EvernoteSetup.md).
### OneDrive / Microsoft Graph ### OneDrive / Microsoft Graph
| **Variable** | **Description** | | **Variable** | **Description** |
-1
View File
@@ -13,7 +13,6 @@ DocuElevate is designed to be highly configurable through environment variables,
- [Dropbox Setup](DropboxSetup.md) - How to set up Dropbox integration - [Dropbox Setup](DropboxSetup.md) - How to set up Dropbox integration
- [OneDrive Setup](OneDriveSetup.md) - How to set up Microsoft OneDrive/Graph integration - [OneDrive Setup](OneDriveSetup.md) - How to set up Microsoft OneDrive/Graph integration
- [Amazon S3 Setup](AmazonS3Setup.md) - How to set up Amazon S3 integration - [Amazon S3 Setup](AmazonS3Setup.md) - How to set up Amazon S3 integration
- [Evernote Setup](EvernoteSetup.md) - How to set up Evernote note creation
- [Authentication Setup](AuthenticationSetup.md) - How to set up user authentication - [Authentication Setup](AuthenticationSetup.md) - How to set up user authentication
- [Notifications Setup](NotificationsSetup.md) - How to set up system notifications - [Notifications Setup](NotificationsSetup.md) - How to set up system notifications
-51
View File
@@ -1,51 +0,0 @@
# Setting up Evernote Integration
This guide explains how to configure DocuElevate to create Evernote notes for processed documents.
## Overview
The Evernote destination creates one note per processed document. The note contains:
- A visible metadata section populated from DocuElevate's extracted metadata JSON
- The processed PDF attached as an Evernote resource
- Optional tags from `EVERNOTE_DEFAULT_TAGS` plus extracted document tags
## Required Configuration
| Variable | Description |
|----------|-------------|
| `EVERNOTE_ENABLED` | Set to `false` to disable Evernote uploads without removing credentials. Default: `true` |
| `EVERNOTE_AUTH_TOKEN` | Evernote developer token or OAuth access token with note creation permissions |
## Optional Configuration
| Variable | Description |
|----------|-------------|
| `EVERNOTE_SANDBOX` | Use Evernote sandbox API endpoints. Default: `false` |
| `EVERNOTE_NOTEBOOK_GUID` | Target notebook GUID. If omitted, Evernote uses the account default notebook |
| `EVERNOTE_DEFAULT_TAGS` | Comma-separated tags to apply to every created note, for example `docuelevate,archive` |
| `EVERNOTE_INCLUDE_METADATA` | Include extracted metadata in the note body. Default: `true` |
## Example
```dotenv
EVERNOTE_ENABLED=true
EVERNOTE_AUTH_TOKEN=your-evernote-token
EVERNOTE_NOTEBOOK_GUID=optional-notebook-guid
EVERNOTE_DEFAULT_TAGS=docuelevate,processed
EVERNOTE_INCLUDE_METADATA=true
```
## Metadata and Attachments
DocuElevate reads the companion metadata file next to the processed PDF, for example `invoice.pdf` and `invoice.json`. Non-empty metadata fields are rendered into the Evernote note body. Values such as `Unknown`, empty strings, and null values are skipped.
If the metadata contains a `tags` field, those tags are applied to the note together with any tags configured in `EVERNOTE_DEFAULT_TAGS`.
The processed PDF is attached directly to the note using Evernote's resource model, so it appears as a normal Evernote attachment.
## Notes
- Evernote tokens can expire or be revoked. If uploads start failing with authentication errors, generate or refresh the token and update `EVERNOTE_AUTH_TOKEN`.
- If `EVERNOTE_NOTEBOOK_GUID` points to a missing or inaccessible notebook, Evernote will reject the note creation request.
- Evernote enforces account upload quotas and per-note size limits. Large PDFs may fail if they exceed those limits.
-1
View File
@@ -24,7 +24,6 @@ Welcome to the DocuElevate documentation. This directory contains comprehensive
- [OneDrive Setup](OneDriveSetup.md) - How to set up Microsoft OneDrive/Graph integration - [OneDrive Setup](OneDriveSetup.md) - How to set up Microsoft OneDrive/Graph integration
- [SharePoint Setup](SharePointSetup.md) - How to set up Microsoft SharePoint Online integration - [SharePoint Setup](SharePointSetup.md) - How to set up Microsoft SharePoint Online integration
- [Amazon S3 Setup](AmazonS3Setup.md) - How to set up Amazon S3 integration - [Amazon S3 Setup](AmazonS3Setup.md) - How to set up Amazon S3 integration
- [Evernote Setup](EvernoteSetup.md) - How to set up Evernote note creation
- [Authentication Setup](AuthenticationSetup.md) - How to set up user authentication - [Authentication Setup](AuthenticationSetup.md) - How to set up user authentication
- [Notifications Setup](NotificationsSetup.md) - How to set up system notifications - [Notifications Setup](NotificationsSetup.md) - How to set up system notifications
+1 -1
View File
@@ -28,7 +28,7 @@ Settings are organized into logical categories for easy navigation:
- **Authentication**: Login settings, session secrets, OAuth configuration, admin group - **Authentication**: Login settings, session secrets, OAuth configuration, admin group
- **AI Services**: AI provider selection, model configuration, embeddings, and credentials (OpenAI, Azure, Anthropic, Gemini, Ollama, OpenRouter, Portkey, LiteLLM) - **AI Services**: AI provider selection, model configuration, embeddings, and credentials (OpenAI, Azure, Anthropic, Gemini, Ollama, OpenRouter, Portkey, LiteLLM)
- **OCR Engines**: OCR provider selection and configuration (Tesseract, EasyOCR, Mistral, Google DocAI, AWS Textract) - **OCR Engines**: OCR provider selection and configuration (Tesseract, EasyOCR, Mistral, Google DocAI, AWS Textract)
- **Storage Providers**: Dropbox, Google Drive, OneDrive, SharePoint, S3, FTP, SFTP, WebDAV, Nextcloud, Paperless, Evernote - **Storage Providers**: Dropbox, Google Drive, OneDrive, SharePoint, S3, FTP, SFTP, WebDAV, Nextcloud, Paperless
- **Email**: SMTP configuration for sending emails - **Email**: SMTP configuration for sending emails
- **IMAP**: Email ingestion configuration (supports two mailbox accounts) - **IMAP**: Email ingestion configuration (supports two mailbox accounts)
- **Monitoring**: Uptime Kuma integration - **Monitoring**: Uptime Kuma integration
-1
View File
@@ -125,7 +125,6 @@ Several cloud storage integrations include their own guided configuration pages
| Google Drive | `/google-drive-setup` | [GoogleDriveSetup.md](GoogleDriveSetup.md) | | Google Drive | `/google-drive-setup` | [GoogleDriveSetup.md](GoogleDriveSetup.md) |
| OneDrive / SharePoint | `/onedrive-setup` | [OneDriveSetup.md](OneDriveSetup.md) | | OneDrive / SharePoint | `/onedrive-setup` | [OneDriveSetup.md](OneDriveSetup.md) |
| Amazon S3 | Configured via settings | [AmazonS3Setup.md](AmazonS3Setup.md) | | Amazon S3 | Configured via settings | [AmazonS3Setup.md](AmazonS3Setup.md) |
| Evernote | Configured via settings | [EvernoteSetup.md](EvernoteSetup.md) |
These pages are accessed **after** the main Setup Wizard is complete and are independent wizard flows specific to each integration. These pages are accessed **after** the main Setup Wizard is complete and are independent wizard flows specific to each integration.
+13 -13
View File
@@ -473,9 +473,9 @@
} }
}, },
"node_modules/nanoid": { "node_modules/nanoid": {
"version": "3.3.12", "version": "3.3.11",
"resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.12.tgz", "resolved": "https://registry.npmjs.org/nanoid/-/nanoid-3.3.11.tgz",
"integrity": "sha512-ZB9RH/39qpq5Vu6Y+NmUaFhQR6pp+M2Xt76XBnEwDaGcVAqhlvxrl3B2bKS5D3NH3QR76v3aSrKaF/Kiy7lEtQ==", "integrity": "sha512-N8SpfPUnUp1bK+PMYW8qSWdl9U+wwNWI4QKxOYDy9JAro3WMX7p2OeVRF9v+347pnakNevPmiHhNmZ2HbFA76w==",
"dev": true, "dev": true,
"funding": [ "funding": [
{ {
@@ -536,9 +536,9 @@
"license": "ISC" "license": "ISC"
}, },
"node_modules/picomatch": { "node_modules/picomatch": {
"version": "2.3.2", "version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz", "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==", "integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"dev": true, "dev": true,
"license": "MIT", "license": "MIT",
"engines": { "engines": {
@@ -569,9 +569,9 @@
} }
}, },
"node_modules/postcss": { "node_modules/postcss": {
"version": "8.5.15", "version": "8.5.8",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.15.tgz", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.8.tgz",
"integrity": "sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==", "integrity": "sha512-OW/rX8O/jXnm82Ey1k44pObPtdblfiuWnrd8X7GJ7emImCOstunGbXUpp7HdBrFQX6rJzn3sPT397Wp5aCwCHg==",
"dev": true, "dev": true,
"funding": [ "funding": [
{ {
@@ -589,7 +589,7 @@
], ],
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"nanoid": "^3.3.12", "nanoid": "^3.3.11",
"picocolors": "^1.1.1", "picocolors": "^1.1.1",
"source-map-js": "^1.2.1" "source-map-js": "^1.2.1"
}, },
@@ -974,9 +974,9 @@
} }
}, },
"node_modules/tinyglobby/node_modules/picomatch": { "node_modules/tinyglobby/node_modules/picomatch": {
"version": "4.0.4", "version": "4.0.3",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz",
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
"dev": true, "dev": true,
"license": "MIT", "license": "MIT",
"engines": { "engines": {
+2 -12
View File
@@ -293,24 +293,13 @@ function processFiles(files, progressContainer, statusMessage) {
updateStatus(); updateStatus();
} }
function _escapeHtml(str) {
if (!str) return '';
return String(str)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#039;');
}
// Pre-create one progress row per file. // Pre-create one progress row per file.
const queueItems = fileArray.map((file) => { const queueItems = fileArray.map((file) => {
const row = document.createElement('div'); const row = document.createElement('div');
row.className = 'flex flex-col mb-2'; row.className = 'flex flex-col mb-2';
const safeFileName = _escapeHtml(file.name);
row.innerHTML = ` row.innerHTML = `
<div class="flex justify-between"> <div class="flex justify-between">
<span class="text-sm truncate" title="${safeFileName}">${safeFileName}</span> <span class="text-sm truncate" title="${file.name}">${file.name}</span>
<span class="text-xs text-gray-500">${formatFileSize(file.size)}</span> <span class="text-xs text-gray-500">${formatFileSize(file.size)}</span>
</div> </div>
<div class="w-full bg-gray-200 h-2 rounded-full mt-1"> <div class="w-full bg-gray-200 h-2 rounded-full mt-1">
@@ -557,3 +546,4 @@ function initDragAndDrop(element, progressContainer, statusMessage, options = {}
}); });
} }
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{ _("billing.success_page_title") }}</title> <title>{{ _("billing.success_page_title") }}</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
+6 -5
View File
@@ -1056,7 +1056,7 @@
html += ` html += `
<div role="listitem"> <div role="listitem">
<a href="/files/${doc.file_id}/detail" aria-label="${title} — ${scorePercent}% similarity (${scoreLabel})" style="text-decoration: none; color: inherit; display: block;"> <a href="/files/${doc.file_id}" aria-label="${title} — ${scorePercent}% similarity (${scoreLabel})" style="text-decoration: none; color: inherit; display: block;">
<div style="display: flex; align-items: center; gap: 1rem; padding: 0.75rem 1rem; background-color: #f7fafc; border-radius: 0.5rem; border: 1px solid #e2e8f0; transition: border-color 0.2s; cursor: pointer;" onmouseover="this.style.borderColor='#4299e1'" onmouseout="this.style.borderColor='#e2e8f0'"> <div style="display: flex; align-items: center; gap: 1rem; padding: 0.75rem 1rem; background-color: #f7fafc; border-radius: 0.5rem; border: 1px solid #e2e8f0; transition: border-color 0.2s; cursor: pointer;" onmouseover="this.style.borderColor='#4299e1'" onmouseout="this.style.borderColor='#e2e8f0'">
<div style="flex-shrink: 0; width: 48px; height: 48px; border-radius: 50%; display: flex; align-items: center; justify-content: center; font-weight: 700; font-size: 0.875rem; color: white; background-color: ${scorePercent >= 80 ? '#48bb78' : scorePercent >= 50 ? '#ecc94b' : '#718096'};" aria-hidden="true"> <div style="flex-shrink: 0; width: 48px; height: 48px; border-radius: 50%; display: flex; align-items: center; justify-content: center; font-weight: 700; font-size: 0.875rem; color: white; background-color: ${scorePercent >= 80 ? '#48bb78' : scorePercent >= 50 ? '#ecc94b' : '#718096'};" aria-hidden="true">
${scorePercent}% ${scorePercent}%
@@ -1091,19 +1091,20 @@
} }
</script> </script>
{% endif %} {% endif %}
{% endblock %} {% endblock %}
{% block content %} {% block content %}
<div class="detail-container"> <div class="detail-container">
<div style="display:flex;align-items:center;gap:1rem;margin-bottom:1.5rem;flex-wrap:wrap;"> <div style="display:flex;align-items:center;gap:1rem;margin-bottom:1.5rem;flex-wrap:wrap;">
<a href="/files" class="back-button" style="margin-bottom:0;" aria-label="Back to File List"> <a href="/files/{{ file.id }}" class="back-button" style="margin-bottom:0;" aria-label="Back to File Summary">
<i class="fas fa-arrow-left" aria-hidden="true"></i> <i class="fas fa-arrow-left" aria-hidden="true"></i>
Back to File List Back to File Summary
</a> </a>
{% if file %} {% if file %}
<a href="/files/{{ file.id }}" class="back-button" style="margin-bottom:0;" aria-label="View document for {{ file.original_filename }}"> <a href="/files/{{ file.id }}/detail" class="back-button" style="margin-bottom:0;" aria-label="View document detail for {{ file.original_filename }}">
<i class="fas fa-eye" aria-hidden="true"></i> <i class="fas fa-eye" aria-hidden="true"></i>
View Document Document Detail
</a> </a>
{% endif %} {% endif %}
</div> </div>
+11 -39
View File
@@ -1538,28 +1538,6 @@
}); });
} }
function escapeHtml(str) {
if (!str) return '';
return String(str)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#039;');
}
function sanitizeHighlight(html) {
if (!html) return '';
let safe = String(html)
.replace(/<mark>/gi, '\x00MARK_OPEN\x00')
.replace(/<\/mark>/gi, '\x00MARK_CLOSE\x00');
safe = escapeHtml(safe);
safe = safe
.replace(/\x00MARK_OPEN\x00/g, '<mark>')
.replace(/\x00MARK_CLOSE\x00/g, '</mark>');
return safe;
}
function renderSearchResults(data, q) { function renderSearchResults(data, q) {
const panel = document.getElementById('search-results-panel'); const panel = document.getElementById('search-results-panel');
const list = document.getElementById('search-results-list'); const list = document.getElementById('search-results-list');
@@ -1577,31 +1555,25 @@
list.innerHTML = results.map(hit => { list.innerHTML = results.map(hit => {
const fmt = hit._formatted || {}; const fmt = hit._formatted || {};
const titleRaw = fmt.document_title || hit.document_title || hit.original_filename || __i18n.untitled; const title = fmt.document_title || hit.document_title || hit.original_filename || __i18n.untitled;
const filenameRaw = fmt.original_filename || hit.original_filename || ''; const filename = fmt.original_filename || hit.original_filename || '';
const snippetRaw = fmt.ocr_text || ''; const snippet = fmt.ocr_text || '';
const tagsRaw = Array.isArray(hit.tags) ? hit.tags.join(', ') : (hit.tags || ''); const tags = Array.isArray(hit.tags) ? hit.tags.join(', ') : (hit.tags || '');
const docTypeRaw = hit.document_type || ''; const docType = hit.document_type || '';
const safeTitle = fmt.document_title ? sanitizeHighlight(titleRaw) : escapeHtml(titleRaw);
const safeFilename = escapeHtml(filenameRaw);
const safeSnippet = sanitizeHighlight(snippetRaw);
const safeTags = escapeHtml(tagsRaw);
const safeDocType = escapeHtml(docTypeRaw);
return `<div style="padding: 0.75rem 1rem; border-bottom: 1px solid #f3f4f6; display: flex; gap: 0.75rem; align-items: flex-start;"> return `<div style="padding: 0.75rem 1rem; border-bottom: 1px solid #f3f4f6; display: flex; gap: 0.75rem; align-items: flex-start;">
<div style="flex-shrink: 0; color: #3b82f6; font-size: 1.25rem; padding-top: 0.1rem;"> <div style="flex-shrink: 0; color: #3b82f6; font-size: 1.25rem; padding-top: 0.1rem;">
<i class="fas fa-file-pdf"></i> <i class="fas fa-file-pdf"></i>
</div> </div>
<div style="flex: 1; min-width: 0;"> <div style="flex: 1; min-width: 0;">
<div style="font-weight: 600; font-size: 0.9rem; color: #111827;">${safeTitle}</div> <div style="font-weight: 600; font-size: 0.9rem; color: #111827;">${title}</div>
${safeFilename ? `<div style="font-size: 0.8rem; color: #6b7280; margin-top: 0.15rem;">${safeFilename}</div>` : ''} ${filename ? `<div style="font-size: 0.8rem; color: #6b7280; margin-top: 0.15rem;">${filename}</div>` : ''}
${safeDocType ? `<span style="display: inline-block; margin-top: 0.25rem; padding: 0.1rem 0.5rem; background: #eff6ff; color: #1d4ed8; border-radius: 9999px; font-size: 0.75rem;">${safeDocType}</span>` : ''} ${docType ? `<span style="display: inline-block; margin-top: 0.25rem; padding: 0.1rem 0.5rem; background: #eff6ff; color: #1d4ed8; border-radius: 9999px; font-size: 0.75rem;">${docType}</span>` : ''}
${safeTags ? `<span style="display: inline-block; margin-top: 0.25rem; margin-left: 0.25rem; padding: 0.1rem 0.5rem; background: #f0fdf4; color: #15803d; border-radius: 9999px; font-size: 0.75rem;">${safeTags}</span>` : ''} ${tags ? `<span style="display: inline-block; margin-top: 0.25rem; margin-left: 0.25rem; padding: 0.1rem 0.5rem; background: #f0fdf4; color: #15803d; border-radius: 9999px; font-size: 0.75rem;">${tags}</span>` : ''}
${safeSnippet ? `<div style="margin-top: 0.4rem; font-size: 0.8rem; color: #374151; white-space: pre-wrap; word-break: break-word;">…${safeSnippet}…</div>` : ''} ${snippet ? `<div style="margin-top: 0.4rem; font-size: 0.8rem; color: #374151; white-space: pre-wrap; word-break: break-word;">…${snippet}…</div>` : ''}
</div> </div>
<div style="flex-shrink: 0;"> <div style="flex-shrink: 0;">
<a href="/files/${escapeHtml(hit.file_id)}" style="padding: 0.25rem 0.6rem; background: #f3f4f6; color: #374151; border-radius: 0.25rem; font-size: 0.8rem; text-decoration: none; white-space: nowrap;" title="${__i18n.viewFile}"> <a href="/files/${hit.file_id}" style="padding: 0.25rem 0.6rem; background: #f3f4f6; color: #374151; border-radius: 0.25rem; font-size: 0.8rem; text-decoration: none; white-space: nowrap;" title="${__i18n.viewFile}">
<i class="fas fa-external-link-alt"></i> <i class="fas fa-external-link-alt"></i>
</a> </a>
</div> </div>
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DocuElevate - Forgot Password</title> <title>DocuElevate - Forgot Password</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DocuElevate - Forgot Username</title> <title>DocuElevate - Forgot Username</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
@@ -1392,7 +1392,7 @@ function integrationsDashboard() {
body: JSON.stringify({ body: JSON.stringify({
integration_type: intg.integration_type, integration_type: intg.integration_type,
config: intg.config, config: intg.config,
credentials: creds, credentials: creds.credentials,
}), }),
}); });
const data = await resp.json(); const data = await resp.json();
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{ _("app.name") }} - {{ _("auth.login_title") }}</title> <title>{{ _("app.name") }} - {{ _("auth.login_title") }}</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>DocuElevate - Reset Password</title> <title>DocuElevate - Reset Password</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
+3 -7
View File
@@ -298,13 +298,9 @@
} }
function escapeHtml(str) { function escapeHtml(str) {
if (str === null || str === undefined) return ''; const d = document.createElement('div');
return String(str) d.textContent = str;
.replace(/&/g, '&amp;') return d.innerHTML;
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#039;');
} }
/** /**
+2 -2
View File
@@ -4,8 +4,8 @@
<meta charset="UTF-8" /> <meta charset="UTF-8" />
<title>Shared Document DocuElevate</title> <title>Shared Document DocuElevate</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0" /> <meta name="viewport" content="width=device-width, initial-scale=1.0" />
<!-- Tailwind CSS --> <!-- Tailwind CSS v3 (compiled) -->
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet" /> <link rel="stylesheet" href="/static/styles.css" />
<!-- Font Awesome --> <!-- Font Awesome -->
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
+6 -15
View File
@@ -393,16 +393,6 @@ const i18nStrings = {
configureNow: {{ _("status.configure_now") | tojson }}, configureNow: {{ _("status.configure_now") | tojson }},
}; };
function escapeHtml(str) {
if (!str) return '';
return String(str)
.replace(/&/g, '&amp;')
.replace(/</g, '&lt;')
.replace(/>/g, '&gt;')
.replace(/"/g, '&quot;')
.replace(/'/g, '&#039;');
}
document.addEventListener('DOMContentLoaded', function() { document.addEventListener('DOMContentLoaded', function() {
// Modal elements // Modal elements
const resultModal = document.getElementById('resultModal'); const resultModal = document.getElementById('resultModal');
@@ -479,8 +469,9 @@ document.addEventListener('DOMContentLoaded', function() {
const isSensitive = sensitiveKeys.some(sKey => key.toLowerCase().includes(sKey)); const isSensitive = sensitiveKeys.some(sKey => key.toLowerCase().includes(sKey));
if (isSensitive && value !== 'Not set' && value !== '') { if (isSensitive && value !== 'Not set' && value !== '') {
valueCell.textContent = value.slice(4) + '********' + value.slice(-4);
// For better readability, we can also use HTML to mask the middle part of the string // For better readability, we can also use HTML to mask the middle part of the string
valueCell.innerHTML = escapeHtml(value.slice(0, 4)) + '<span class="text-gray-400">********</span>' + escapeHtml(value.slice(-4)); valueCell.innerHTML = value.slice(0, 4) + '<span class="text-gray-400">********</span>' + value.slice(-4);
} else { } else {
valueCell.textContent = value; valueCell.textContent = value;
} }
@@ -595,9 +586,9 @@ document.addEventListener('DOMContentLoaded', function() {
if (data.status === 'success') { if (data.status === 'success') {
// If there's token info, we need to handle it specially // If there's token info, we need to handle it specially
if (data.token_info && data.token_info.expires_in_human) { if (data.token_info && data.token_info.expires_in_human) {
let message = escapeHtml(data.message || 'Connection successful'); let message = data.message || 'Connection successful';
message += `<br><br><div class="bg-blue-50 p-3 rounded mt-2"> message += `<br><br><div class="bg-blue-50 p-3 rounded mt-2">
<span class="font-medium">${i18nStrings.tokenValidFor}</span> ${escapeHtml(data.token_info.expires_in_human)} <span class="font-medium">${i18nStrings.tokenValidFor}</span> ${data.token_info.expires_in_human}
</div>`; </div>`;
modalTitle.textContent = i18nStrings.testSuccessful; modalTitle.textContent = i18nStrings.testSuccessful;
@@ -656,12 +647,12 @@ document.addEventListener('DOMContentLoaded', function() {
.then(data => { .then(data => {
if (data.status === 'success') { if (data.status === 'success') {
// Create successful message // Create successful message
let message = escapeHtml(data.message || 'Connection successful'); let message = data.message || 'Connection successful';
// Add token expiration info if available (especially for Google Drive) // Add token expiration info if available (especially for Google Drive)
if (data.token_info && data.token_info.expires_in_human) { if (data.token_info && data.token_info.expires_in_human) {
message += `<br><br><div class="bg-blue-50 p-3 rounded mt-2"> message += `<br><br><div class="bg-blue-50 p-3 rounded mt-2">
<span class="font-medium">${i18nStrings.tokenValidFor}</span> ${escapeHtml(data.token_info.expires_in_human)} <span class="font-medium">${i18nStrings.tokenValidFor}</span> ${data.token_info.expires_in_human}
</div>`; </div>`;
// Show the message with HTML // Show the message with HTML
+1 -1
View File
@@ -4,7 +4,7 @@
<meta charset="UTF-8"> <meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>{{ _("auth.verify_email_page_title") }}</title> <title>{{ _("auth.verify_email_page_title") }}</title>
<link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet"> <link rel="stylesheet" href="/static/styles.css" />
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css" <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.5.1/css/all.min.css"
integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA==" integrity="sha512-DTOQO9RWCH3ppGqcWaEA1BIZOC6xxalwEsw9c2QQeAIftl+Vegovlnee1c9QX4TctnWMn13TZye+giMm8e2LwA=="
crossorigin="anonymous" referrerpolicy="no-referrer" /> crossorigin="anonymous" referrerpolicy="no-referrer" />
@@ -0,0 +1,83 @@
{{- /*
Celery Beat scheduler — publishes periodic tasks to the broker.
Exactly ONE replica must run; never scale this deployment.
*/ -}}
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "docuelevate.fullname" . }}-beat
namespace: {{ .Release.Namespace }}
labels:
{{- include "docuelevate.labels" . | nindent 4 }}
app.kubernetes.io/component: beat
spec:
replicas: 1
strategy:
type: Recreate # Prevent two Beat instances from running simultaneously
selector:
matchLabels:
{{- include "docuelevate.selectorLabels" . | nindent 6 }}
app.kubernetes.io/component: beat
template:
metadata:
labels:
{{- include "docuelevate.selectorLabels" . | nindent 8 }}
app.kubernetes.io/component: beat
{{- with .Values.beat.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
spec:
serviceAccountName: {{ include "docuelevate.serviceAccountName" . }}
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.beat.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: beat
image: {{ include "docuelevate.image" . }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command:
- celery
- -A
- app.celery_worker
- beat
- --loglevel=info
envFrom:
- configMapRef:
name: {{ include "docuelevate.fullname" . }}-config
- secretRef:
name: {{ include "docuelevate.fullname" . }}-secret
{{- with .Values.beat.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
resources:
{{- toYaml .Values.beat.resources | nindent 12 }}
volumeMounts:
- name: workdir
mountPath: /workdir
volumes:
- name: workdir
{{- if .Values.workdir.persistence.enabled }}
persistentVolumeClaim:
claimName: {{ .Values.workdir.persistence.existingClaim | default (printf "%s-workdir" (include "docuelevate.fullname" .)) }}
{{- else }}
emptyDir: {}
{{- end }}
{{- with .Values.beat.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.beat.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.beat.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
@@ -42,7 +42,6 @@ spec:
- -A - -A
- app.celery_worker - app.celery_worker
- worker - worker
- -B
- --loglevel=info - --loglevel=info
- -Q - -Q
- document_processor,default,celery - document_processor,default,celery
+33 -4
View File
@@ -121,10 +121,10 @@ api:
type: ClusterIP type: ClusterIP
port: 8000 port: 8000
# Liveness / readiness probes # Liveness / readiness probes (unauthenticated endpoints for kubelet)
livenessProbe: livenessProbe:
httpGet: httpGet:
path: /api/health path: /api/diagnostic/healthz/live
port: 8000 port: 8000
initialDelaySeconds: 30 initialDelaySeconds: 30
periodSeconds: 20 periodSeconds: 20
@@ -132,7 +132,7 @@ api:
readinessProbe: readinessProbe:
httpGet: httpGet:
path: /api/health path: /api/diagnostic/healthz/ready
port: 8000 port: 8000
initialDelaySeconds: 15 initialDelaySeconds: 15
periodSeconds: 10 periodSeconds: 10
@@ -191,7 +191,36 @@ worker:
drop: ["ALL"] drop: ["ALL"]
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Shared workdir volume (api + worker mount the same PVC) # Celery Beat scheduler (singleton — always exactly 1 replica)
# Beat publishes periodic tasks; workers consume them from the broker.
# ---------------------------------------------------------------------------
beat:
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
cpu: 500m
memory: 512Mi
podAnnotations: {}
nodeSelector: {}
tolerations: []
affinity: {}
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: false
capabilities:
drop: ["ALL"]
# ---------------------------------------------------------------------------
# Shared workdir volume (api + worker + beat mount the same PVC)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
workdir: workdir:
persistence: persistence:
-201
View File
@@ -1,201 +0,0 @@
"""
Base setup for views, containing shared functionality and imports.
"""
import logging
from pathlib import Path
from fastapi import APIRouter, Depends, HTTPException, Request # noqa: F401
from fastapi.templating import Jinja2Templates
from sqlalchemy.orm import Session # noqa: F401
from app.auth import require_login # noqa: F401
from app.config import settings
from app.database import SessionLocal, get_db # noqa: F401
from app.models import UserProfile
from app.utils.i18n import (
SUPPORTED_LANGUAGES,
detect_language,
format_date,
format_datetime,
format_number,
get_suggested_languages,
translate,
)
# Set up Jinja2 templates
templates_dir = Path(__file__).parent.parent.parent / "frontend" / "templates"
templates = Jinja2Templates(directory=str(templates_dir))
# Add Python built-in functions to Jinja2 template globals
templates.env.globals["min"] = min
templates.env.globals["max"] = max
# ---------------------------------------------------------------------------
# i18n Jinja2 integration
# ---------------------------------------------------------------------------
# The _() function is available in every template to translate UI strings.
# Usage: {{ _("nav.dashboard") }} or {{ _("upload.max_size", size="10 MB") }}
# The locale is automatically resolved from the request context.
# A default English implementation is registered as a global so error handlers
# that don't go through _inject_global_context still have the function available.
# ---------------------------------------------------------------------------
templates.env.globals["supported_languages"] = SUPPORTED_LANGUAGES
templates.env.globals["_"] = lambda key, **kwargs: translate(key, "en", **kwargs)
# Customize Jinja2Templates to include app_version in all templates
original_template_response = templates.TemplateResponse
def _hydrate_language_from_db(request: Request, session_user: object) -> None:
"""Load the user's preferred language from the DB into the session.
Called once per session when ``preferred_language`` is not yet in the
session. A lightweight DB query fetches the stored preference so that
:func:`detect_language` picks it up from the session on all subsequent
requests without further DB access.
"""
from app.utils.i18n import SUPPORTED_LANGUAGE_CODES
user_id: str | None = None
if isinstance(session_user, dict):
user_id = (
session_user.get("sub")
or session_user.get("preferred_username")
or session_user.get("email")
or session_user.get("id")
)
elif isinstance(session_user, str):
user_id = session_user
if not user_id:
return
db = SessionLocal()
try:
profile = db.query(UserProfile).filter(UserProfile.user_id == user_id).first()
if profile and profile.preferred_language and profile.preferred_language in SUPPORTED_LANGUAGE_CODES:
request.session["preferred_language"] = profile.preferred_language
except Exception: # noqa: BLE001 — intentionally broad; DB may be temporarily unavailable
logger.debug("Could not hydrate language preference for user_id=%s", user_id)
finally:
db.close()
def _inject_global_context(ctx: dict) -> None:
"""Inject shared global variables into every template context dict."""
ctx.setdefault("version", settings.version)
ctx.setdefault("release_name", getattr(settings, "release_name", None))
ctx.setdefault("ui_default_color_scheme", getattr(settings, "ui_default_color_scheme", "system"))
ctx.setdefault("multi_user_enabled", getattr(settings, "multi_user_enabled", False))
ctx.setdefault("auth_enabled", getattr(settings, "auth_enabled", True))
ctx.setdefault(
"allow_signup",
getattr(settings, "multi_user_enabled", False) and getattr(settings, "allow_local_signup", False),
)
ctx.setdefault("enable_factory_reset", getattr(settings, "enable_factory_reset", False))
# Sentry Browser SDK config (injected into every page so the JS SDK can initialise)
# Normalize empty-string DSN to None so the {% if sentry_dsn %} template guard works correctly.
_raw_dsn = getattr(settings, "sentry_dsn", None)
ctx.setdefault("sentry_dsn", _raw_dsn if _raw_dsn else None)
ctx.setdefault("sentry_environment", getattr(settings, "sentry_environment", "production"))
ctx.setdefault("sentry_js_traces_sample_rate", getattr(settings, "sentry_js_traces_sample_rate", 0.0))
ctx.setdefault(
"sentry_js_replay_session_sample_rate",
getattr(settings, "sentry_js_replay_session_sample_rate", 0.0),
)
ctx.setdefault(
"sentry_js_replay_on_error_sample_rate",
getattr(settings, "sentry_js_replay_on_error_sample_rate", 0.1),
)
req = ctx.get("request")
if req is not None:
# CSRF token
if hasattr(req, "state") and hasattr(req.state, "csrf_token"):
ctx.setdefault("csrf_token", req.state.csrf_token)
# Determine whether the current visitor is authenticated
session_user = None
if hasattr(req, "session"):
session_user = req.session.get("user")
# When auth is disabled every visitor is effectively "logged in"
ctx.setdefault("is_logged_in", not getattr(settings, "auth_enabled", True) or session_user is not None)
# --- Hydrate session language from DB (once per session) ---
# If the session doesn't have a preferred_language yet but the user
# is logged in, load the stored preference from the database so that
# detect_language() picks it up from the session on this and all
# subsequent requests.
if hasattr(req, "session") and "preferred_language" not in req.session and session_user is not None:
_hydrate_language_from_db(req, session_user)
# --- i18n: detect language and register template helpers ---
current_locale = detect_language(req)
ctx.setdefault("current_locale", current_locale)
# Smart language suggestions for the compact nav-bar dropdown (5-7 languages)
accept_header = req.headers.get("accept-language", "") if hasattr(req, "headers") else ""
ctx.setdefault("suggested_languages", get_suggested_languages(current_locale, accept_header))
def _translate(key: str, **kwargs: object) -> str:
return translate(key, current_locale, **kwargs)
def _format_date(value: object, short: bool = False) -> str:
return format_date(value, current_locale, short=short) # type: ignore[arg-type]
def _format_datetime(value: object) -> str:
return format_datetime(value, current_locale) # type: ignore[arg-type]
def _format_number(value: object) -> str:
return format_number(value, current_locale) # type: ignore[arg-type]
ctx.setdefault("_", _translate)
ctx.setdefault("format_date_l10n", _format_date)
ctx.setdefault("format_datetime_l10n", _format_datetime)
ctx.setdefault("format_number_l10n", _format_number)
else:
ctx.setdefault("is_logged_in", not getattr(settings, "auth_enabled", True))
ctx.setdefault("current_locale", "en")
ctx.setdefault("_", lambda key, **kw: translate(key, "en", **kw))
def template_response_with_version(*args, **kwargs):
"""Wrapper for TemplateResponse to include version and CSRF token in all templates.
Handles both old-style and new-style Starlette TemplateResponse calls:
- Old-style (Starlette <1.0): TemplateResponse(name, {"request": req, ...}, ...)
- New-style (Starlette 1.0+): TemplateResponse(request, name, context={...}, ...)
"""
if len(args) >= 1 and isinstance(args[0], str):
# Old-style call: first positional arg is the template name (string).
# Convert to new-style: (request, name, context=..., ...)
name = args[0]
if len(args) >= 2 and isinstance(args[1], dict):
context = args[1]
# Old-style may have status_code as 3rd positional arg
if len(args) >= 3 and "status_code" not in kwargs:
kwargs["status_code"] = args[2]
else:
context = kwargs.pop("context", {})
request_obj = context.pop("request", None)
if request_obj is not None:
context["request"] = request_obj
_inject_global_context(context)
if request_obj is not None:
return original_template_response(request_obj, name, context=context, **kwargs)
return original_template_response(name, context=context, **kwargs)
# New-style call: (request, name, context=..., ...)
if "context" in kwargs and isinstance(kwargs["context"], dict):
_inject_global_context(kwargs["context"])
elif len(args) >= 3 and isinstance(args[2], dict):
_inject_global_context(args[2])
return original_template_response(*args, **kwargs)
templates.TemplateResponse = template_response_with_version
# Set up logging
logger = logging.getLogger(__name__)
-1
View File
@@ -55,7 +55,6 @@ nav:
- Google Drive: GoogleDriveSetup - Google Drive: GoogleDriveSetup
- OneDrive: OneDriveSetup - OneDrive: OneDriveSetup
- Amazon S3: AmazonS3Setup - Amazon S3: AmazonS3Setup
- Evernote: EvernoteSetup
- Authentication: AuthenticationSetup - Authentication: AuthenticationSetup
- Notifications: NotificationsSetup - Notifications: NotificationsSetup
- Security: - Security:
+335 -346
View File
File diff suppressed because it is too large Load Diff
-7
View File
@@ -58,13 +58,6 @@
"eslint-config-expo": "~10.0.0", "eslint-config-expo": "~10.0.0",
"typescript": "^5.3.0" "typescript": "^5.3.0"
}, },
"overrides": {
"postcss": "8.5.15",
"uuid": "11.1.1",
"brace-expansion@^1.1.7": "1.1.13",
"brace-expansion@^2.0.2": "2.1.0",
"brace-expansion@^5.0.2": "5.0.6"
},
"engines": { "engines": {
"node": ">=20.19.4" "node": ">=20.19.4"
}, },
+1 -1
View File
@@ -1,5 +1,5 @@
[build-system] [build-system]
requires = ["setuptools>=82.0.1", "wheel"] requires = ["setuptools>=45", "wheel"]
build-backend = "setuptools.build_meta" build-backend = "setuptools.build_meta"
[project] [project]
-34
View File
@@ -1,34 +0,0 @@
import sys
from unittest.mock import MagicMock
from fastapi.templating import Jinja2Templates
import os
# We don't really need a real path, but let's mock it
os.makedirs("templates", exist_ok=True)
with open("templates/files.html", "w") as f:
f.write("Hello")
templates = Jinja2Templates(directory="templates")
original_template_response = templates.TemplateResponse
def template_response_with_version(*args, **kwargs):
if len(args) == 2 and isinstance(args[0], str) and isinstance(args[1], dict):
context = args[1]
request = context.get("request")
if request is not None:
# THIS IS MY FIX
print("Running fix logic")
return original_template_response(request=request, name=args[0], context=context, **kwargs)
print("Running original fallback logic")
return original_template_response(*args, **kwargs)
templates.TemplateResponse = template_response_with_version
req = MagicMock()
try:
templates.TemplateResponse("files.html", {"request": req})
print("SUCCESS")
except Exception as e:
import traceback
traceback.print_exc()
-33
View File
@@ -1,33 +0,0 @@
import sys
from unittest.mock import MagicMock
from fastapi.templating import Jinja2Templates
import os
os.makedirs("templates", exist_ok=True)
with open("templates/files.html", "w") as f:
f.write("Hello")
templates = Jinja2Templates(directory="templates")
original_template_response = templates.TemplateResponse
def template_response_with_version(*args, **kwargs):
if len(args) == 2 and isinstance(args[0], str) and isinstance(args[1], dict):
context = args[1]
request = context.get("request")
if request is not None:
# THIS IS MY FIX
print("Running fix logic")
return original_template_response(request=request, name=args[0], context=context, **kwargs)
print("Running original fallback logic", args, kwargs)
return original_template_response(*args, **kwargs)
templates.TemplateResponse = template_response_with_version
req = MagicMock()
try:
templates.TemplateResponse(request=req, name="files.html", context={"request": req})
print("SUCCESS")
except Exception as e:
import traceback
traceback.print_exc()
-33
View File
@@ -1,33 +0,0 @@
import sys
from unittest.mock import MagicMock
from fastapi.templating import Jinja2Templates
import os
os.makedirs("templates", exist_ok=True)
with open("templates/files.html", "w") as f:
f.write("Hello")
templates = Jinja2Templates(directory="templates")
original_template_response = templates.TemplateResponse
def template_response_with_version(*args, **kwargs):
if len(args) == 2 and isinstance(args[0], str) and isinstance(args[1], dict):
context = args[1]
request = context.get("request")
if request is not None:
# THIS IS MY FIX
print("Running fix logic")
return original_template_response(request=request, name=args[0], context=context, **kwargs)
print("Running original fallback logic", args, kwargs)
return original_template_response(*args, **kwargs)
templates.TemplateResponse = template_response_with_version
req = MagicMock()
try:
templates.TemplateResponse("files.html", {"request": req}, status_code=200)
print("SUCCESS")
except Exception as e:
import traceback
traceback.print_exc()
-35
View File
@@ -1,35 +0,0 @@
import sys
from unittest.mock import MagicMock
from fastapi.templating import Jinja2Templates
import os
os.makedirs("templates", exist_ok=True)
with open("templates/files.html", "w") as f:
f.write("Hello")
templates = Jinja2Templates(directory="templates")
original_template_response = templates.TemplateResponse
def template_response_with_version(*args, **kwargs):
print("ARGS:", args)
print("KWARGS:", kwargs)
if len(args) == 2 and isinstance(args[0], str) and isinstance(args[1], dict):
context = args[1]
request = context.get("request")
if request is not None:
# THIS IS MY FIX
print("Running fix logic")
return original_template_response(request=request, name=args[0], context=context, **kwargs)
print("Running original fallback logic", args, kwargs)
return original_template_response(*args, **kwargs)
templates.TemplateResponse = template_response_with_version
req = MagicMock()
try:
templates.TemplateResponse("files.html", context={"request": req})
print("SUCCESS")
except Exception as e:
import traceback
traceback.print_exc()
+3 -3
View File
@@ -4,14 +4,14 @@
# Testing # Testing
pytest>=8.0.0 pytest>=8.0.0
pytest-cov>=4.1.0 pytest-cov>=4.1.0
pytest-asyncio>=1.4.0 pytest-asyncio>=0.23.0
pytest-mock>=3.12.0 pytest-mock>=3.12.0
pytest-timeout>=2.3.0 # Per-test timeout enforcement to prevent CI hangs pytest-timeout>=2.3.0 # Per-test timeout enforcement to prevent CI hangs
httpx>=0.26.0 # For async test client httpx>=0.26.0 # For async test client
testcontainers>=3.7.1 # For integration tests with real containers testcontainers>=3.7.1 # For integration tests with real containers
fpdf2>=2.8.0 # For generating test PDF documents in integration tests fpdf2>=2.8.0 # For generating test PDF documents in integration tests
minio>=7.1.0 # For MinIO/S3 integration tests minio>=7.1.0 # For MinIO/S3 integration tests
redis>=8.0.0 # For Redis integration tests redis>=4.5.0 # For Redis integration tests
boto3>=1.26.0 # For S3 integration tests boto3>=1.26.0 # For S3 integration tests
# Code quality # Code quality
@@ -34,7 +34,7 @@ pip-audit>=2.7.0 # Dependency vulnerability scanning against OSV/PyPA advisory
pre-commit>=3.6.0 pre-commit>=3.6.0
# License compliance # License compliance
pip-licenses==5.5.5 # For license compliance checking pip-licenses==5.5.1 # For license compliance checking
# Release automation # Release automation
python-semantic-release>=9.0.0 python-semantic-release>=9.0.0
+3 -7
View File
@@ -2,9 +2,8 @@ fastapi[all] # Web framework with all extras
uvicorn # ASGI server uvicorn # ASGI server
celery # Task queue celery # Task queue
redis # Message broker for Celery redis # Message broker for Celery
sqlalchemy # Database ORM sqlalchemy # Database ORM
psycopg[binary]>=3.2,<4.0 # PostgreSQL driver for HA database deployments pydantic # Data validation
pydantic # Data validation
cryptography>=41.0.0 # Encryption for sensitive settings in database cryptography>=41.0.0 # Encryption for sensitive settings in database
openai # GPT integration for metadata extraction openai # GPT integration for metadata extraction
pypdf>=3.9.0 # PDF processing for text extraction, metadata editing and rotation (upgraded from PyPDF2 to fix CVE-2023-36464) pypdf>=3.9.0 # PDF processing for text extraction, metadata editing and rotation (upgraded from PyPDF2 to fix CVE-2023-36464)
@@ -38,9 +37,6 @@ paramiko>=3.4.0 # SSH/SFTP implementation for Python (LGPL license)
# iCloud Drive # iCloud Drive
pyicloud>=2.4.0 # Unofficial Apple iCloud API client (MIT license) pyicloud>=2.4.0 # Unofficial Apple iCloud API client (MIT license)
# Evernote
evernote3>=1.25.14 # Evernote Cloud API SDK for Python 3 (BSD license)
# Safe XML parsing (protection against XML bomb / XXE attacks) # Safe XML parsing (protection against XML bomb / XXE attacks)
defusedxml>=0.7.1 defusedxml>=0.7.1
@@ -55,7 +51,7 @@ pytesseract>=0.3.10 # Python wrapper for Tesseract OCR
pdf2image>=1.17.0 # Convert PDF pages to images (used by Tesseract and EasyOCR providers) pdf2image>=1.17.0 # Convert PDF pages to images (used by Tesseract and EasyOCR providers)
ocrmypdf>=16.0.0,<18.0.0 # Post-processing: embeds searchable text layers into PDFs via Tesseract ocrmypdf>=16.0.0,<18.0.0 # Post-processing: embeds searchable text layers into PDFs via Tesseract
meilisearch>=0.31.0 # Full-text search engine client meilisearch>=0.31.0 # Full-text search engine client
stripe>=7.0.0,<16.0.0 # Stripe billing SDK (MIT license) stripe>=7.0.0,<15.0.0 # Stripe billing SDK (MIT license)
# Error and performance monitoring # Error and performance monitoring
sentry-sdk[fastapi,celery,sqlalchemy]>=2.20.0,<3.0.0 sentry-sdk[fastapi,celery,sqlalchemy]>=2.20.0,<3.0.0
-1
View File
@@ -1 +0,0 @@
Hello
+2
View File
@@ -83,6 +83,8 @@ class TestGotenbergCoverageDocuments:
".tif", ".tif",
".webp", ".webp",
".svg", ".svg",
".heic",
".heif",
} }
_html_extensions = {".html", ".htm"} _html_extensions = {".html", ".htm"}
_markdown_extensions = {".md", ".markdown"} _markdown_extensions = {".md", ".markdown"}
-9
View File
@@ -267,15 +267,6 @@ class TestTestDropboxToken:
class TestSaveDropboxSettings: class TestSaveDropboxSettings:
"""Tests for save_dropbox_settings endpoint.""" """Tests for save_dropbox_settings endpoint."""
@pytest.fixture(autouse=True)
def _admin_override(self):
from app.api.dropbox import _require_admin
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
@patch("app.api.dropbox.settings") @patch("app.api.dropbox.settings")
def test_save_settings_env_not_found(self, mock_settings, client): def test_save_settings_env_not_found(self, mock_settings, client):
"""Test that missing .env file is non-fatal — DB write still succeeds.""" """Test that missing .env file is non-fatal — DB write still succeeds."""
+8 -16
View File
@@ -360,15 +360,6 @@ class TestFormatTimeRemaining:
class TestSaveGoogleDriveSettings: class TestSaveGoogleDriveSettings:
"""Tests for POST /google-drive/save-settings endpoint.""" """Tests for POST /google-drive/save-settings endpoint."""
@pytest.fixture(autouse=True)
def _admin_override(self):
from app.api.google_drive import _require_admin
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
@patch("builtins.open", new_callable=mock_open, read_data="# Existing config\n") @patch("builtins.open", new_callable=mock_open, read_data="# Existing config\n")
@patch("os.path.exists") @patch("os.path.exists")
@patch("os.path.dirname") @patch("os.path.dirname")
@@ -490,17 +481,18 @@ class TestSaveGoogleDriveSettings:
assert response.status_code == 200 assert response.status_code == 200
@patch("app.api.google_drive.os") @patch("os.path.exists")
@patch("os.path.dirname")
@patch("app.config.settings") @patch("app.config.settings")
def test_save_settings_exception_handling(self, mock_settings, mock_os, client: TestClient): def test_save_settings_exception_handling(self, mock_settings, mock_dirname, mock_exists, client: TestClient):
"""Test that exceptions in .env write are non-fatal — DB write still succeeds.""" """Test exception handling in save settings."""
mock_os.path.exists.side_effect = Exception("Unexpected error") mock_exists.side_effect = Exception("Unexpected error")
response = client.post("/api/google-drive/save-settings", data={"refresh_token": "token", "use_oauth": "true"}) response = client.post("/api/google-drive/save-settings", data={"refresh_token": "token", "use_oauth": "true"})
# .env write exception is caught; endpoint succeeds via DB write assert response.status_code == 500
assert response.status_code == 200 data = response.json()
assert response.json()["status"] == "success" assert "failed to save" in data["detail"].lower()
@pytest.mark.unit @pytest.mark.unit
-9
View File
@@ -195,15 +195,6 @@ class TestGetGoogleDriveTokenInfo:
class TestSaveGoogleDriveSettings: class TestSaveGoogleDriveSettings:
"""Test save_google_drive_settings endpoint edge cases.""" """Test save_google_drive_settings endpoint edge cases."""
@pytest.fixture(autouse=True)
def _admin_override(self):
from app.api.google_drive import _require_admin
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
@patch("app.api.google_drive.settings") @patch("app.api.google_drive.settings")
@patch("os.path.exists") @patch("os.path.exists")
def test_save_settings_env_file_not_exists(self, mock_exists, mock_settings, client: TestClient): def test_save_settings_env_file_not_exists(self, mock_exists, mock_settings, client: TestClient):
+4 -10
View File
@@ -152,16 +152,11 @@ class TestGetTokenInfoCredentialsBranches:
@pytest.mark.unit @pytest.mark.unit
class TestSaveGoogleDriveSettingsFalsyFields: class TestSaveGoogleDriveSettingsFalsyFields:
"""Cover branches 395->397, 449->451, 468->470 in save_google_drive_settings.""" """Cover branches 395->397, 449->451, 468->470 in save_google_drive_settings.
@pytest.fixture(autouse=True) Note: the Google Drive save endpoint is named save_google_drive_settings in the
def _admin_override(self): source (app/api/google_drive.py).
from app.api.google_drive import _require_admin """
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
@patch("app.api.google_drive.settings") @patch("app.api.google_drive.settings")
@patch("os.path.exists", return_value=False) @patch("os.path.exists", return_value=False)
@@ -182,7 +177,6 @@ class TestSaveGoogleDriveSettingsFalsyFields:
with patch("app.api.google_drive.notify_settings_updated"): with patch("app.api.google_drive.notify_settings_updated"):
result = await save_google_drive_settings( result = await save_google_drive_settings(
request=mock_request, request=mock_request,
_admin={"is_admin": True},
refresh_token="", # falsy → branches 395->397 and 449->451 refresh_token="", # falsy → branches 395->397 and 449->451
client_id="cid", client_id="cid",
client_secret=None, client_secret=None,
+3 -12
View File
@@ -524,10 +524,7 @@ class TestTestImapConnection:
from app.api.imap_accounts import _test_imap_connection from app.api.imap_accounts import _test_imap_connection
mock_mail = MagicMock() mock_mail = MagicMock()
with ( with patch("imaplib.IMAP4_SSL", return_value=mock_mail):
patch("app.api.imap_accounts.is_private_ip", return_value=False),
patch("imaplib.IMAP4_SSL", return_value=mock_mail),
):
result = _test_imap_connection( result = _test_imap_connection(
"imap.example.com", "imap.example.com",
993, 993,
@@ -544,10 +541,7 @@ class TestTestImapConnection:
"""An exception raised by IMAP4_SSL returns success=False.""" """An exception raised by IMAP4_SSL returns success=False."""
from app.api.imap_accounts import _test_imap_connection from app.api.imap_accounts import _test_imap_connection
with ( with patch("imaplib.IMAP4_SSL", side_effect=Exception("auth failed")):
patch("app.api.imap_accounts.is_private_ip", return_value=False),
patch("imaplib.IMAP4_SSL", side_effect=Exception("auth failed")),
):
result = _test_imap_connection( result = _test_imap_connection(
"imap.example.com", "imap.example.com",
993, 993,
@@ -563,10 +557,7 @@ class TestTestImapConnection:
"""An OSError returns success=False with a network error message.""" """An OSError returns success=False with a network error message."""
from app.api.imap_accounts import _test_imap_connection from app.api.imap_accounts import _test_imap_connection
with ( with patch("imaplib.IMAP4", side_effect=OSError("connection refused")):
patch("app.api.imap_accounts.is_private_ip", return_value=False),
patch("imaplib.IMAP4", side_effect=OSError("connection refused")),
):
result = _test_imap_connection( result = _test_imap_connection(
"bad-host", "bad-host",
143, 143,
-30
View File
@@ -998,23 +998,6 @@ class TestConnectionTestEndpoint:
assert data["success"] is False assert data["success"] is False
assert "Missing" in data["message"] assert "Missing" in data["message"]
def test_test_imap_blocks_private_ip(self, int_client):
"""IMAP test with private IP returns failure (SSRF protection)."""
payload = {
"integration_type": "IMAP",
"config": {
"host": "127.0.0.1",
"port": 993,
"username": "user",
},
"credentials": {"password": "pass"},
}
resp = int_client.post("/api/integrations/test", json=payload)
assert resp.status_code == 200
data = resp.json()
assert data["success"] is False
assert "Invalid hostname or IP address" in data["message"]
def test_test_s3_missing_bucket(self, int_client): def test_test_s3_missing_bucket(self, int_client):
"""S3 test with missing bucket returns failure.""" """S3 test with missing bucket returns failure."""
payload = { payload = {
@@ -1028,19 +1011,6 @@ class TestConnectionTestEndpoint:
assert data["success"] is False assert data["success"] is False
assert "bucket" in data["message"].lower() assert "bucket" in data["message"].lower()
def test_test_s3_blocks_private_ip(self, int_client):
"""S3 test with private IP endpoint returns failure (SSRF protection)."""
payload = {
"integration_type": "S3",
"config": {"bucket": "my-bucket", "endpoint_url": "http://127.0.0.1:9000"},
"credentials": {"access_key_id": "AKIA", "secret_access_key": "secret"},
}
resp = int_client.post("/api/integrations/test", json=payload)
assert resp.status_code == 200
data = resp.json()
assert data["success"] is False
assert "Invalid endpoint URL or private IP" in data["message"]
def test_test_webdav_missing_url(self, int_client): def test_test_webdav_missing_url(self, int_client):
"""WebDAV test with missing URL returns failure.""" """WebDAV test with missing URL returns failure."""
payload = { payload = {
-9
View File
@@ -342,15 +342,6 @@ class TestFormatTimeRemaining:
class TestSaveOneDriveSettings: class TestSaveOneDriveSettings:
"""Tests for POST /onedrive/save-settings endpoint.""" """Tests for POST /onedrive/save-settings endpoint."""
@pytest.fixture(autouse=True)
def _admin_override(self):
from app.api.onedrive import _require_admin
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
@patch("builtins.open", new_callable=mock_open, read_data="# Existing config\n") @patch("builtins.open", new_callable=mock_open, read_data="# Existing config\n")
@patch("os.path.exists") @patch("os.path.exists")
@patch("os.path.dirname") @patch("os.path.dirname")
-9
View File
@@ -294,15 +294,6 @@ class TestTokenRotationEnvAppendLine:
class TestSaveSettingsException: class TestSaveSettingsException:
"""Cover lines 324-326: save_onedrive_settings outer exception handler.""" """Cover lines 324-326: save_onedrive_settings outer exception handler."""
@pytest.fixture(autouse=True)
def _admin_override(self):
from app.api.onedrive import _require_admin
from app.main import app as fastapi_app
fastapi_app.dependency_overrides[_require_admin] = lambda: {"is_admin": True}
yield
fastapi_app.dependency_overrides.pop(_require_admin, None)
def test_save_settings_outer_exception(self, client: TestClient): def test_save_settings_outer_exception(self, client: TestClient):
"""Trigger the outer exception handler in save_onedrive_settings.""" """Trigger the outer exception handler in save_onedrive_settings."""
with patch("app.api.onedrive.notify_settings_updated", side_effect=Exception("Unexpected boom")): with patch("app.api.onedrive.notify_settings_updated", side_effect=Exception("Unexpected boom")):
-21
View File
@@ -47,27 +47,6 @@ class TestProcessEndpoints:
data = response.json() data = response.json()
assert data["task_id"] == "test-task-id" assert data["task_id"] == "test-task-id"
assert data["status"] == "queued" assert data["status"] == "queued"
mock_task.delay.assert_called_once_with(str(test_file))
def test_send_to_dropbox_endpoint_file_not_found(self, client):
"""Test POST /api/send_to_dropbox/ directly mapping to endpoint name with non-existent file."""
response = client.post("/api/send_to_dropbox/?file_path=nonexistent_endpoint.pdf")
assert response.status_code == 400
def test_send_to_dropbox_endpoint_success(self, client, tmp_path):
"""Test POST /api/send_to_dropbox/ directly mapping to endpoint name with existing file."""
test_file = tmp_path / "processed" / "test_endpoint.pdf"
test_file.parent.mkdir(parents=True)
test_file.write_text("test content endpoint")
with patch("app.api.process.upload_to_dropbox") as mock_task:
mock_task.delay.return_value = Mock(id="test-task-id-endpoint")
response = client.post(f"/api/send_to_dropbox/?file_path={test_file}")
assert response.status_code == 200
data = response.json()
assert data["task_id"] == "test-task-id-endpoint"
assert data["status"] == "queued"
mock_task.delay.assert_called_once_with(str(test_file))
def test_send_to_paperless_file_not_found(self, client): def test_send_to_paperless_file_not_found(self, client):
"""Test POST /api/send_to_paperless/ with non-existent file.""" """Test POST /api/send_to_paperless/ with non-existent file."""
+160 -337
View File
@@ -1,368 +1,191 @@
"""Tests for the saved searches API (app/api/saved_searches.py)."""
import pytest import pytest
from fastapi.testclient import TestClient from fastapi.testclient import TestClient
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.pool import StaticPool
from app.database import Base, get_db
from app.models import SavedSearch from app.models import SavedSearch
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Saved searches CRUD tests # Test data constants
# ---------------------------------------------------------------------------
_OWNER = "test_user@example.com"
_OTHER_OWNER = "other_user@example.com"
# ---------------------------------------------------------------------------
# Shared fixture helpers
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@pytest.mark.unit @pytest.fixture()
class TestSavedSearchesCRUD: def int_engine():
"""Tests for saved searches CRUD API endpoints.""" """In-memory SQLite engine for integration tests."""
engine = create_engine(
"sqlite:///:memory:",
connect_args={"check_same_thread": False},
poolclass=StaticPool,
)
Base.metadata.create_all(bind=engine)
yield engine
Base.metadata.drop_all(bind=engine)
def test_list_saved_searches_empty(self, client: TestClient):
"""GET /api/saved-searches returns empty list when no searches exist."""
response = client.get("/api/saved-searches")
assert response.status_code == 200
assert response.json() == []
def test_create_saved_search(self, client: TestClient): @pytest.fixture()
"""POST /api/saved-searches creates a new saved search.""" def int_session(int_engine):
payload = { """DB session scoped to one test."""
"name": "My Invoices", Session = sessionmaker(bind=int_engine)
"filters": {"tags": "invoice", "status": "completed"}, session = Session()
} yield session
response = client.post("/api/saved-searches", json=payload) session.close()
assert response.status_code == 201
data = response.json()
def _make_client(int_engine, owner_id: str = _OWNER):
"""Return a TestClient with *owner_id* injected as the authenticated user."""
from unittest.mock import patch
from app.main import app
def override_db():
Session = sessionmaker(bind=int_engine)
session = Session()
try:
yield session
finally:
session.close()
app.dependency_overrides[get_db] = override_db
with patch("app.api.saved_searches._get_user_id", return_value=owner_id):
with TestClient(app, base_url="http://localhost", raise_server_exceptions=False) as client:
yield client
app.dependency_overrides.clear()
@pytest.fixture()
def int_client(int_engine):
"""TestClient authenticated as _OWNER."""
yield from _make_client(int_engine, _OWNER)
# ---------------------------------------------------------------------------
# CRUD tests
# ---------------------------------------------------------------------------
@pytest.mark.integration
class TestSavedSearchesAPI:
"""Tests for Saved Searches endpoints."""
def test_list_saved_searches_empty(self, int_client):
"""No saved searches returns empty list."""
resp = int_client.get("/api/saved-searches")
assert resp.status_code == 200
assert resp.json() == []
def test_create_saved_search(self, int_client):
"""Create a saved search and verify the response."""
payload = {"name": "My Invoices", "filters": {"tags": "invoice", "document_type": "Invoice"}}
resp = int_client.post("/api/saved-searches", json=payload)
assert resp.status_code == 201
data = resp.json()
assert data["name"] == "My Invoices" assert data["name"] == "My Invoices"
assert data["filters"]["tags"] == "invoice" assert data["filters"] == {"tags": "invoice", "document_type": "Invoice"}
assert data["filters"]["status"] == "completed"
assert "id" in data assert "id" in data
def test_create_and_list_saved_search(self, client: TestClient): def test_create_saved_search_invalid_filters(self, int_client):
"""Creating a saved search makes it appear in the list.""" """Creating with invalid filters returns 422."""
payload = { # Missing filters parameter (or empty after sanitization)
"name": "PDF Files", payload = {"name": "My Invoices", "filters": {}}
"filters": {"mime_type": "application/pdf"}, resp = int_client.post("/api/saved-searches", json=payload)
} assert resp.status_code == 422
client.post("/api/saved-searches", json=payload)
response = client.get("/api/saved-searches") # Invalid filters format
assert response.status_code == 200 payload2 = {"name": "My Invoices", "filters": "not_a_dict"}
searches = response.json() resp2 = int_client.post("/api/saved-searches", json=payload2)
assert len(searches) == 1 assert resp2.status_code == 422
assert searches[0]["name"] == "PDF Files"
def test_create_saved_search_missing_name(self, client: TestClient): def test_create_saved_search_duplicate(self, int_client):
"""POST /api/saved-searches without name returns 422.""" """Creating a duplicate named search returns 409."""
payload = {"filters": {"status": "completed"}} payload = {"name": "Duplicate", "filters": {"q": "test"}}
response = client.post("/api/saved-searches", json=payload) int_client.post("/api/saved-searches", json=payload)
assert response.status_code == 422 resp = int_client.post("/api/saved-searches", json=payload)
assert resp.status_code == 409
def test_create_saved_search_empty_filters(self, client: TestClient): def test_create_saved_search_limit(self, int_client, int_session):
"""POST /api/saved-searches with empty filters returns 422.""" """Exceeding MAX_SAVED_SEARCHES_PER_USER returns 409."""
payload = {"name": "Empty", "filters": {}} # Create 50 searches using the API to ensure they are visible
response = client.post("/api/saved-searches", json=payload) for i in range(50):
assert response.status_code == 422 resp = int_client.post("/api/saved-searches", json={"name": f"Search LIMIT {i}", "filters": {"q": "test"}})
assert resp.status_code == 201
def test_create_saved_search_invalid_filter_keys(self, client: TestClient): payload = {"name": "One too many", "filters": {"q": "test"}}
"""POST /api/saved-searches ignores unknown filter keys.""" resp = int_client.post("/api/saved-searches", json=payload)
payload = { assert resp.status_code == 409
"name": "With unknown keys",
"filters": {"invalid_key": "value", "status": "completed"},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 201
data = response.json()
# Only valid filter key should remain
assert "invalid_key" not in data["filters"]
assert data["filters"]["status"] == "completed"
def test_create_saved_search_only_invalid_keys(self, client: TestClient): def test_update_saved_search(self, int_client):
"""POST with only invalid filter keys returns 422.""" """Update an existing saved search."""
payload = { payload = {"name": "Original Name", "filters": {"q": "test"}}
"name": "All invalid", created = int_client.post("/api/saved-searches", json=payload).json()
"filters": {"bad_key": "value"}, search_id = created["id"]
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 422
def test_create_duplicate_name(self, client: TestClient): update_payload = {"name": "Updated Name", "filters": {"tags": "new"}}
"""POST /api/saved-searches with duplicate name returns 409.""" resp = int_client.put(f"/api/saved-searches/{search_id}", json=update_payload)
payload = {"name": "My Search", "filters": {"status": "completed"}} assert resp.status_code == 200
response1 = client.post("/api/saved-searches", json=payload) data = resp.json()
assert response1.status_code == 201 assert data["name"] == "Updated Name"
assert data["filters"] == {"tags": "new"}
response2 = client.post("/api/saved-searches", json=payload) def test_update_saved_search_not_found(self, int_client):
assert response2.status_code == 409 """Updating a non-existent search returns 404."""
update_payload = {"name": "Updated Name"}
resp = int_client.put("/api/saved-searches/999", json=update_payload)
assert resp.status_code == 404
def test_update_saved_search(self, client: TestClient): def test_update_saved_search_duplicate_name(self, int_client):
"""PUT /api/saved-searches/{id} updates the saved search.""" """Updating name to an existing search name returns 409."""
# Create payload1 = {"name": "Search 1", "filters": {"q": "a"}}
create_resp = client.post( payload2 = {"name": "Search 2", "filters": {"q": "b"}}
"/api/saved-searches", int_client.post("/api/saved-searches", json=payload1)
json={"name": "Original", "filters": {"status": "pending"}}, created2 = int_client.post("/api/saved-searches", json=payload2).json()
) search2_id = created2["id"]
search_id = create_resp.json()["id"]
# Update update_payload = {"name": "Search 1"}
update_resp = client.put( resp = int_client.put(f"/api/saved-searches/{search2_id}", json=update_payload)
f"/api/saved-searches/{search_id}", assert resp.status_code == 409
json={"name": "Updated", "filters": {"status": "completed"}},
)
assert update_resp.status_code == 200
data = update_resp.json()
assert data["name"] == "Updated"
assert data["filters"]["status"] == "completed"
def test_update_saved_search_not_found(self, client: TestClient): def test_delete_saved_search(self, int_client, int_session):
"""PUT /api/saved-searches/999 returns 404.""" """Delete an existing search."""
response = client.put( payload = {"name": "To be deleted", "filters": {"q": "test"}}
"/api/saved-searches/999", created = int_client.post("/api/saved-searches", json=payload).json()
json={"name": "Nope", "filters": {"status": "completed"}}, search_id = created["id"]
)
assert response.status_code == 404
def test_delete_saved_search(self, client: TestClient): resp = int_client.delete(f"/api/saved-searches/{search_id}")
"""DELETE /api/saved-searches/{id} removes the saved search.""" assert resp.status_code == 204
# Create
create_resp = client.post(
"/api/saved-searches",
json={"name": "To Delete", "filters": {"status": "failed"}},
)
search_id = create_resp.json()["id"]
# Delete assert int_session.query(SavedSearch).filter(SavedSearch.id == search_id).first() is None
del_resp = client.delete(f"/api/saved-searches/{search_id}")
assert del_resp.status_code == 204
# Verify it's gone def test_delete_saved_search_not_found(self, int_client):
list_resp = client.get("/api/saved-searches") """Deleting a non-existent search returns 404."""
assert len(list_resp.json()) == 0 resp = int_client.delete("/api/saved-searches/999")
assert resp.status_code == 404
def test_delete_saved_search_not_found(self, client: TestClient): def test_other_users_searches_isolated(self, int_engine, int_session):
"""DELETE /api/saved-searches/999 returns 404.""" """Users only see and can only modify their own saved searches."""
response = client.delete("/api/saved-searches/999") int_session.add(SavedSearch(user_id=_OTHER_OWNER, name="Other Search", filters='{"q": "test"}'))
assert response.status_code == 404 int_session.commit()
def test_create_name_too_long(self, client: TestClient): client = next(_make_client(int_engine, _OWNER))
"""POST /api/saved-searches with name > 100 chars returns 422.""" resp = client.get("/api/saved-searches")
payload = { assert resp.status_code == 200
"name": "x" * 101, assert len(resp.json()) == 0
"filters": {"status": "completed"},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 422
def test_saved_search_filters_sanitized(self, client: TestClient): other_search = int_session.query(SavedSearch).first()
"""Saved search filters are sanitized to allowed keys only.""" resp = client.put(f"/api/saved-searches/{other_search.id}", json={"name": "Hacked"})
payload = { assert resp.status_code == 404
"name": "Sanitized",
"filters": {
"search": "invoice",
"mime_type": "application/pdf",
"date_from": "2026-01-01",
"date_to": "2026-12-31",
"storage_provider": "dropbox",
"tags": "invoice,amazon",
"sort_by": "created_at",
"sort_order": "desc",
},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 201
data = response.json()
assert len(data["filters"]) == 8
assert data["filters"]["search"] == "invoice"
assert data["filters"]["tags"] == "invoice,amazon"
def test_saved_search_with_fulltext_query(self, client: TestClient): resp = client.delete(f"/api/saved-searches/{other_search.id}")
"""Saved search can include full-text query (q) for the search view.""" assert resp.status_code == 404
payload = {
"name": "Invoice Search",
"filters": {"q": "invoice total amount", "document_type": "Invoice"},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 201
data = response.json()
assert data["filters"]["q"] == "invoice total amount"
assert data["filters"]["document_type"] == "Invoice"
def test_saved_search_content_finding_filters(self, client: TestClient):
"""Saved search accepts content-finding filter keys (language, sender, text_quality)."""
payload = {
"name": "German Invoices",
"filters": {
"q": "rechnung",
"language": "de",
"sender": "ACME GmbH",
"text_quality": "high",
"tags": "invoice",
},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 201
data = response.json()
assert data["filters"]["q"] == "rechnung"
assert data["filters"]["language"] == "de"
assert data["filters"]["sender"] == "ACME GmbH"
assert data["filters"]["text_quality"] == "high"
assert data["filters"]["tags"] == "invoice"
def test_create_saved_search_max_limit(self, client: TestClient, db_session, mocker):
"""POST /api/saved-searches returns 409 when max limit is reached."""
from app.api.saved_searches import MAX_SAVED_SEARCHES_PER_USER
user_id = "test_user"
mocker.patch("app.api.saved_searches._get_user_id", return_value=user_id)
for i in range(MAX_SAVED_SEARCHES_PER_USER):
search = SavedSearch(user_id=user_id, name=f"Search {i}", filters="""{"tags": "invoice"}""")
db_session.add(search)
db_session.commit()
payload = {
"name": "One More",
"filters": {"tags": "invoice"},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 409
def test_create_saved_search_db_error(self, client: TestClient, mocker):
"""POST /api/saved-searches handles db.commit errors gracefully."""
mocker.patch("sqlalchemy.orm.Session.commit", side_effect=Exception("DB Error"))
payload = {
"name": "Fail Me",
"filters": {"tags": "invoice"},
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 500
def test_update_saved_search_db_error(self, client: TestClient, mocker):
"""PUT /api/saved-searches/{id} handles db.commit errors gracefully."""
# Create a search first
payload = {
"name": "Update Target",
"filters": {"tags": "invoice"},
}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
mocker.patch("sqlalchemy.orm.Session.commit", side_effect=Exception("DB Error"))
update_payload = {"name": "New Name"}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 500
def test_delete_saved_search_db_error(self, client: TestClient, mocker):
"""DELETE /api/saved-searches/{id} handles db.commit errors gracefully."""
# Create a search first
payload = {
"name": "Delete Target",
"filters": {"tags": "invoice"},
}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
mocker.patch("sqlalchemy.orm.Session.commit", side_effect=Exception("DB Error"))
response = client.delete(f"/api/saved-searches/{search_id}")
assert response.status_code == 500
def test_update_saved_search_name_conflict(self, client: TestClient):
"""PUT /api/saved-searches/{id} returns 409 when the new name conflicts with an existing search."""
# Create search 1
payload1 = {"name": "Search One", "filters": {"tags": "invoice"}}
client.post("/api/saved-searches", json=payload1)
# Create search 2
payload2 = {"name": "Search Two", "filters": {"status": "completed"}}
response2 = client.post("/api/saved-searches", json=payload2)
search2_id = response2.json()["id"]
# Try to update search 2 to have name "Search One"
update_payload = {"name": "Search One"}
response = client.put(f"/api/saved-searches/{search2_id}", json=update_payload)
assert response.status_code == 409
def test_update_saved_search_empty_filters(self, client: TestClient):
"""PUT /api/saved-searches/{id} returns 422 if filters are empty or invalid."""
payload = {"name": "Search XYZ", "filters": {"tags": "invoice"}}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
# Empty filters
update_payload = {"filters": {}}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 422
# Invalid keys
update_payload = {"filters": {"invalid_key": "value"}}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 422
def test_update_saved_search_invalid_name(self, client: TestClient):
"""PUT /api/saved-searches/{id} returns 422 if name is invalid or too long."""
payload = {"name": "Search XYZ", "filters": {"tags": "invoice"}}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
# Empty name
update_payload = {"name": ""}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 422
# Too long name
update_payload = {"name": "A" * 101}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 422
def test_update_saved_search_same_name(self, client: TestClient):
"""PUT /api/saved-searches/{id} with the same name does not trigger duplicate check error."""
# Create a search
payload = {"name": "Same Name", "filters": {"tags": "invoice"}}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
# Update with the exact same name
update_payload = {"name": "Same Name"}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 200
assert response.json()["name"] == "Same Name"
def test_get_user_id_branches_real(self, client: TestClient):
from app.api.saved_searches import _get_user_id
# We need a mock request
class MockRequest:
session = {}
state = type("obj", (object,), {"user": None})
req = MockRequest()
assert _get_user_id(req) == "anonymous"
req.session["user"] = {"preferred_username": "pref"}
assert _get_user_id(req) == "pref"
req.session["user"] = {"email": "em@il.com"}
assert _get_user_id(req) == "em@il.com"
req.session["user"] = {"name": "named"}
assert _get_user_id(req) == "named"
req.session["user"] = {}
assert _get_user_id(req) == "anonymous"
def test_validate_filters_not_dict(self, client: TestClient):
"""POST /api/saved-searches with non-dict filters returns 422."""
payload = {
"name": "Invalid Filters",
"filters": "not a dict",
}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 422
def test_create_saved_search_non_dict_filters(self, client: TestClient):
payload = {"name": "Test", "filters": []}
response = client.post("/api/saved-searches", json=payload)
assert response.status_code == 422
def test_update_saved_search_non_dict_filters(self, client: TestClient):
payload = {"name": "Test", "filters": {"tags": "invoice"}}
response = client.post("/api/saved-searches", json=payload)
search_id = response.json()["id"]
update_payload = {"filters": []}
response = client.put(f"/api/saved-searches/{search_id}", json=update_payload)
assert response.status_code == 422
+3 -3
View File
@@ -430,8 +430,8 @@ class TestLoginFunction:
# Verify TemplateResponse was called with correct context # Verify TemplateResponse was called with correct context
mock_templates.TemplateResponse.assert_called_once() mock_templates.TemplateResponse.assert_called_once()
call_args = mock_templates.TemplateResponse.call_args call_args = mock_templates.TemplateResponse.call_args
assert call_args[0][1] == "login.html" assert call_args[0][0] == "login.html"
context = call_args.kwargs["context"] context = call_args[0][1]
assert context["error"] == "Test error" assert context["error"] == "Test error"
assert context["message"] == "Test message" assert context["message"] == "Test message"
@@ -450,7 +450,7 @@ class TestLoginFunction:
mock_templates.TemplateResponse.assert_called_once() mock_templates.TemplateResponse.assert_called_once()
call_args = mock_templates.TemplateResponse.call_args call_args = mock_templates.TemplateResponse.call_args
context = call_args.kwargs["context"] context = call_args[0][1]
assert context["error"] is None assert context["error"] is None
assert context["message"] is None assert context["message"] is None
+1 -1
View File
@@ -281,7 +281,7 @@ class TestLoginEndpoint:
# Verify template was rendered with OAuth enabled # Verify template was rendered with OAuth enabled
mock_templates.TemplateResponse.assert_called_once() mock_templates.TemplateResponse.assert_called_once()
call_args = mock_templates.TemplateResponse.call_args call_args = mock_templates.TemplateResponse.call_args
context = call_args.kwargs["context"] context = call_args[0][1]
assert context["show_oauth"] is True assert context["show_oauth"] is True
assert context["oauth_provider_name"] == "Test SSO" assert context["oauth_provider_name"] == "Test SSO"
-8
View File
@@ -35,7 +35,6 @@ class TestValidateStorageConfigs:
"google_drive", "google_drive",
"onedrive", "onedrive",
"email", "email",
"evernote",
"paperless", "paperless",
"uptime_kuma", "uptime_kuma",
] ]
@@ -84,13 +83,6 @@ class TestValidateStorageConfigs:
assert "DEST_EMAIL_HOST is not configured" in result["email"] assert "DEST_EMAIL_HOST is not configured" in result["email"]
assert "DEST_EMAIL_DEFAULT_RECIPIENT is not configured" in result["email"] assert "DEST_EMAIL_DEFAULT_RECIPIENT is not configured" in result["email"]
def test_evernote_missing_token(self):
"""Test validation when Evernote destination auth token is missing."""
with patch("app.utils.config_validator.validators.settings") as mock_settings:
mock_settings.evernote_auth_token = None
result = validate_storage_configs()
assert "EVERNOTE_AUTH_TOKEN is not configured" in result["evernote"]
@pytest.mark.unit @pytest.mark.unit
class TestValidateEmailConfig: class TestValidateEmailConfig:
+2 -2
View File
@@ -520,14 +520,14 @@ class TestURLUploadAdditionalCoverage:
assert exc_info.value.status_code == 400 assert exc_info.value.status_code == 400
def test_is_private_ip_unresolvable_hostname(self): def test_is_private_ip_unresolvable_hostname(self):
"""Cover DNS resolution failure branch blocking unresolvable domains.""" """Cover DNS resolution failure branch (lines 67-72)."""
import socket as _socket import socket as _socket
from app.utils.network import is_private_ip from app.utils.network import is_private_ip
with patch("socket.getaddrinfo", side_effect=_socket.gaierror("nope")): with patch("socket.getaddrinfo", side_effect=_socket.gaierror("nope")):
result = is_private_ip("nonexistent.invalid.hostname.test") result = is_private_ip("nonexistent.invalid.hostname.test")
assert result is True # Fail securely by returning True assert result is False
def test_is_private_ip_hostname_resolves_to_private(self): def test_is_private_ip_hostname_resolves_to_private(self):
"""Cover branch where hostname resolves to a private IP (line 64-65).""" """Cover branch where hostname resolves to a private IP (line 64-65)."""
+2 -2
View File
@@ -69,8 +69,8 @@ class TestViewsBase:
context = {"request": req} context = {"request": req}
template_response_with_version("template.html", context) template_response_with_version("template.html", context)
_, kwargs = mock_orig.call_args args, _ = mock_orig.call_args
assert kwargs["context"].get("csrf_token") == "my-csrf" assert args[1].get("csrf_token") == "my-csrf"
def test_kwargs_context_no_request(self): def test_kwargs_context_no_request(self):
"""Test kwargs context path when request is not in context.""" """Test kwargs context path when request is not in context."""

Some files were not shown because too many files have changed in this diff Show More