Commit Graph

221 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] 571cc81789 feat(api): add per-user health-aware upload rate limiting
Introduces a Redis-backed sliding-window rate limiter for upload
endpoints (/api/ui-upload, /api/process-url) that:
- Enforces per-user limits (default: 20 uploads / 60 s)
- Dynamically reduces limits under system stress (queue depth, CPU load)
- Returns 429 with Retry-After header when exceeded
- Fails open when Redis is unavailable
- Works with the existing client-side adaptive back-off

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-17 12:12:50 +00:00
Christian Krakau-Louis 8905031d16 Merge branch 'main' into copilot/implement-log-off-everywhere-functionality 2026-03-17 12:12:50 +01:00
copilot-swe-agent[bot] 03712cfb08 feat(ui): add dedicated Devices page for mobile tokens and registered devices
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-17 11:00:36 +00:00
copilot-swe-agent[bot] a4588a57cb feat(auth): add server-side session management and QR code login backend
- Add UserSession and QRLoginChallenge models for session tracking
  and mobile QR authentication
- Add session_manager utility with create/validate/revoke/cleanup
  functions and QR challenge helpers
- Add /api/sessions endpoints for listing, revoking, and
  'log off everywhere' functionality
- Add /api/qr-auth endpoints for challenge creation, polling, and
  claiming with API token issuance
- Add session config fields (lifetime, custom override, QR TTL)
- Update get_current_user to validate server-side sessions
- Create server-side sessions on all login paths (local, OAuth,
  social, admin)
- Revoke server-side session on logout
- Configure SessionMiddleware max_age from session lifetime settings
- Graceful degradation: old sessions without _session_token continue
  to work

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-16 22:29:04 +00:00
copilot-swe-agent[bot] a88d790445 feat(system-reset): add system reset and factory reset feature
- Add FACTORY_RESET_ON_STARTUP and ENABLE_FACTORY_RESET config settings
- Create app/utils/system_reset.py with core reset logic (wipe DB + files, reimport)
- Create app/api/system_reset.py with admin-only API endpoints
- Create app/views/system_reset.py with admin-only UI view
- Create frontend/templates/system_reset.html with confirmation dialogs
- Auto-reset on startup when FACTORY_RESET_ON_STARTUP=true
- Re-import uses watch folder mechanism for re-ingestion
- Register routers in API and views init files
- Add i18n keys and SETTING_METADATA entries
- Add nav links in base.html (desktop + mobile)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 22:26:17 +00:00
copilot-swe-agent[bot] 825f3cc3a8 fix(translation): fix apply_owner_filter called with wrong arguments
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 15:25:29 +00:00
Christian Krakau-Louis ebbbd3d62e Merge pull request #731 from christianlouis/copilot/fix-mobile-app-login
fix: resolve CI failures from stale request mocks, Annotated+Depends pattern, and CodeQL sensitive data logging
2026-03-16 15:29:15 +01:00
Christian Krakau-Louis c81914f78d Merge pull request #729 from christianlouis/copilot/add-default-language-version-support
feat(translation): automatic document translation to configurable default language
2026-03-16 13:55:25 +01:00
copilot-swe-agent[bot] 7bd667e271 fix(security): clarify billing log message wording
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 12:23:07 +00:00
copilot-swe-agent[bot] 075a505085 feat(profile): expose default_document_language in profile API and UI
- Add default_document_language to ProfileResponse and ProfileUpdateRequest
- Handle validation in PATCH /api/profile endpoint
- Add dropdown in profile.html template with Alpine.js binding
- Add translation keys for profile UI labels
- Add comprehensive tests for profile default language feature

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 12:13:54 +00:00
copilot-swe-agent[bot] b7a3b301a3 fix(security): remove clear-text logging of sensitive data and fix Annotated+Depends pattern
- audit_logs.py: Remove _db_dep pattern that fails with latest FastAPI on
  Python 3.11. Use clean DbSession = Annotated[Session, Depends(get_db)]
  without default values.
- billing.py: Remove owner_id from log messages to fix CodeQL clear-text
  logging of sensitive information alerts.
- files.py: Remove owner_id from log messages to fix CodeQL clear-text
  logging of sensitive information alerts.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 12:07:04 +00:00
copilot-swe-agent[bot] 3a221a62cd feat(translation): add model, config, task, API, and UI for default document language translation
- Add detected_language, default_language_text, default_language_code columns to FileRecord
- Add default_document_language column to UserProfile
- Add DEFAULT_DOCUMENT_LANGUAGE config setting (defaults to "en")
- Create translate_to_default_language Celery task
- Integrate translation trigger into embed_metadata_into_pdf pipeline
- Add /api/files/{id}/translate and /api/files/{id}/translation/default API endpoints
- Add /files/{id}/text/default-language view endpoint
- Update file_view.html with translation sections (default language, on-the-fly)
- Add translation keys to en.json
- Create Alembic migration 036
- Update .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 11:54:13 +00:00
Christian Krakau-Louis 3447a408db Merge pull request #714 from christianlouis/perf/async-onedrive-token-refresh-7215258063318287629
 Async OneDrive token validation using httpx
2026-03-16 12:15:39 +01:00
Christian Krakau-Louis 2a8a4b7471 Merge branch 'main' into code-health/simplify-ui-upload-12009112042086709661 2026-03-16 12:14:11 +01:00
Christian Krakau-Louis b8a1ac52b3 Merge branch 'main' into perf/async-url-upload-7099409897484162483 2026-03-16 12:12:21 +01:00
Christian Krakau-Louis e10f0bff42 Merge branch 'main' into refactor/audit-logs-annotated-b008-6104828608171783200 2026-03-16 12:10:32 +01:00
google-labs-jules[bot] 8ad90d7da9 style: resolve conflicts and use Annotated pattern in audit_logs.py
- Resolves merge conflicts with main.
- Implements Annotated pattern for FastAPI dependencies and query parameters.
- Maintains compatibility with decorators by using module-level dependency singletons.
- Fixes Ruff B008 issues.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 10:01:19 +00:00
Christian Krakau-Louis bcd49aa793 Merge pull request #695 from christianlouis/fix/rename-save-dropbox-settings-9203732237520055590
🧹 [code health improvement] Rename save_dropbox_settings to save_google_drive_settings
2026-03-16 10:58:07 +01:00
Christian Krakau-Louis 3be78708cd Merge pull request #716 from christianlouis/code-health/simplify-save-onedrive-settings-12338634662690804221
🧹 [Code Health] Simplify save_onedrive_settings logic and extract env_utils
2026-03-16 10:56:25 +01:00
Christian Krakau-Louis 992adad978 Merge pull request #718 from christianlouis/perf-upload-15584535240168516797
 Optimize file upload endpoint to use asynchronous aiofiles
2026-03-16 10:55:57 +01:00
google-labs-jules[bot] 7b21a69ceb perf: optimize url upload with async i/o
Replaced synchronous `requests.get` and `open().write` in the `process_url` endpoint with `httpx.AsyncClient` and `aiofiles.open`. This prevents the FastAPI event loop from blocking during large file downloads.

Updated test suite in `tests/test_url_upload.py` to use `AsyncMock` to mock `httpx.AsyncClient.stream` contexts and async generators properly, covering all original conditions and HTTP error handling paths.

Added dependencies `aiofiles` and `types-aiofiles` to resolve MyPy typing CI failures, and mitigated CodeQL security alerts regarding user-provided path extensions by deriving the extension from original_filename and filtering out all non-alphanumerics.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:53:47 +00:00
Christian Krakau-Louis 00be8d7b8e Merge pull request #703 from christianlouis/perf/fix-n-plus-1-shared-links-7028382104319809262
 fix N+1 query in list_shared_links
2026-03-16 10:52:34 +01:00
Christian Krakau-Louis 6e2e54aac6 Merge pull request #705 from christianlouis/perf/fix-n-plus-1-duplicates-17979853535459179799
 Fix N+1 Query in Duplicate Grouping List
2026-03-16 10:51:59 +01:00
Christian Krakau-Louis 056292dbe6 Merge pull request #710 from christianlouis/jules/perf-opt-reorder-plans-5035457001465312501
 Optimize reorder_plans by fixing N+1 database query
2026-03-16 10:51:02 +01:00
Christian Krakau-Louis c00a35bbac Merge pull request #711 from christianlouis/performance-optimize-dropbox-token-2748375428782384195
 Optimize Dropbox token check by using async httpx
2026-03-16 10:49:34 +01:00
google-labs-jules[bot] 30718218cc perf: optimize url upload with async i/o
Replaced synchronous `requests.get` and `open().write` in the `process_url` endpoint with `httpx.AsyncClient` and `aiofiles.open`. This prevents the FastAPI event loop from blocking during large file downloads.

Updated test suite in `tests/test_url_upload.py` to use `AsyncMock` to mock `httpx.AsyncClient.stream` contexts and async generators properly, covering all original conditions and HTTP error handling paths.

Added dependencies `aiofiles` and `types-aiofiles` to resolve MyPy typing CI failures, and mitigated CodeQL security alerts regarding user-provided path extensions by leveraging `os.path.basename` around the generated target file paths, and filtering out non-alphanumerics from the file extension.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:49:01 +00:00
google-labs-jules[bot] b8db664c2e perf: optimize url upload with async i/o
Replaced synchronous `requests.get` and `open().write` in the `process_url` endpoint with `httpx.AsyncClient` and `aiofiles.open`. This prevents the FastAPI event loop from blocking during large file downloads.

Updated test suite in `tests/test_url_upload.py` to use `AsyncMock` to mock `httpx.AsyncClient.stream` contexts and async generators properly, covering all original conditions and HTTP error handling paths.

Added dependencies `aiofiles` and `types-aiofiles` to resolve MyPy typing CI failures, and mitigated CodeQL security alerts regarding user-provided path extensions by leveraging `os.path.basename` around the generated target file paths.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:45:06 +00:00
google-labs-jules[bot] fffb7cf357 Fix ruff linting errors resulting from aiofiles addition
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:44:59 +00:00
google-labs-jules[bot] 320a2acedd perf: optimize url upload with async i/o
Replaced synchronous `requests.get` and `open().write` in the `process_url` endpoint with `httpx.AsyncClient` and `aiofiles.open`. This prevents the FastAPI event loop from blocking during large file downloads.

Updated test suite in `tests/test_url_upload.py` to use `AsyncMock` to mock `httpx.AsyncClient.stream` contexts and async generators properly, covering all original conditions and HTTP error handling paths.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:41:17 +00:00
github-actions[bot] 80a0ddcbfc style: apply ruff auto-fix
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-16 09:36:54 +00:00
google-labs-jules[bot] b290cffb98 Performance Optimization: Replace synchronous file upload read with async aiofiles
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:36:33 +00:00
google-labs-jules[bot] c941738644 Refactor save_onedrive_settings and test_onedrive_token to use shared env utility
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:34:10 +00:00
github-actions[bot] ca2d023d81 style: apply ruff auto-fix
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-16 09:25:47 +00:00
google-labs-jules[bot] 7242f3c168 perf(onedrive): use async httpx for token refresh
Replaces the synchronous `requests.post` and `requests.get` calls in
`app/api/onedrive.py:test_onedrive_token` with an asynchronous
`httpx.AsyncClient` implementation. This unblocks the FastAPI event loop
when this endpoint is hit.

Tests were updated to mock `httpx.AsyncClient` and a sync wrapper using `asyncio.run` was added to integration tests to maintain test coverage without massive test refactoring.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:24:58 +00:00
google-labs-jules[bot] fe20e02f78 perf(api): fix n+1 query issue in user notification preferences update
- Added a benchmark script in tests/test_notifications_api.py that proved the N+1 issue issue.
- Replaced iterative DB lookups inside `for item in body.preferences:` with single pre-fetch query and local `prefs_dict` lookups.
- Verified test benchmark time drops from ~0.0964s to ~0.0141s for a batch of 100 items.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:20:41 +00:00
google-labs-jules[bot] 84c6e1c5dd perf: optimize dropbox token refresh by replacing blocking requests with httpx
Replaced the synchronous `requests.post` calls in `app/api/dropbox.py` with asynchronous `httpx.AsyncClient().post` calls. This ensures that the FastAPI event loop is not blocked during network I/O, allowing better concurrent performance.

Also updated the `test_api_dropbox.py` tests to use `httpx.AsyncClient.post` in mocks and properly construct `httpx.RequestError` in exception handling tests.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:19:55 +00:00
google-labs-jules[bot] d8372c6fb8 perf(api): optimize reorder_plans to prevent N+1 queries
Replaced the loop over `body.order` which generated an N+1 issue
with a single bulk query fetching all relevant `SubscriptionPlan`
records via the `.in_()` clause.

Added an in-memory dictionary map of `plan_id` to `SubscriptionPlan`
objects to allow `O(1)` lookups while updating the order.

Benchmark speedup: 14.71x faster on 500 records.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:19:50 +00:00
google-labs-jules[bot] e4e3ac4077 perf(duplicates): fix N+1 query in group listing
Replaced the loop over duplicate hashes that resulted in O(N) database queries
per page with a single efficient `in_` batch query to retrieve both originals
and duplicates. The records are then grouped in memory using dictionaries.

This resolves the N+1 performance bottleneck and reduces response time from
an average of 1.65 seconds to ~0.45 seconds locally for 500 groups.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:16:31 +00:00
google-labs-jules[bot] 040f4dcdd4 fix N+1 query in list_shared_links
Replaced the N+1 query in `list_shared_links` which fetched `FileRecord` for each link. It now uses a single query with an `outerjoin` to fetch `original_filename` alongside the `SharedLink` object.
Measured a significant improvement from ~0.4547s to ~0.0579s per 1000 links.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:14:36 +00:00
google-labs-jules[bot] df4b4ae18c 🧹 [Code Health] Simplify complex endpoint ui_upload
Extracted file chunk saving and duplicate detection logic into separate helper functions (`_save_upload_file_chunks` and `_check_for_exact_duplicate`) to improve readability and maintainability of the `ui_upload` endpoint in `app/api/files.py`.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 09:00:35 +00:00
google-labs-jules[bot] c3d06d1876 Fix naming inconsistency in Google Drive API
The function handling the `/google-drive/save-settings` endpoint was incorrectly named `save_dropbox_settings`, likely due to a copy-paste error. This commits renames it to `save_google_drive_settings` and updates all the tests referencing it.

Tested using standard procedures, although test execution resulted in missing dependency errors due to lack of network access in the environment.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 08:59:13 +00:00
github-actions[bot] b7a195d3bc style: apply ruff auto-fix
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-15 04:10:28 +00:00
google-labs-jules[bot] 8079db7893 🛡️ Sentinel: [HIGH] Fix SSRF in WebDAV connection test
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-15 04:10:11 +00:00
Christian Krakau-Louis d5fb0da335 Merge pull request #639 from christianlouis/refactor-audit-logs-annotated-b008-6104828608171783200
🧹 Use Annotated pattern in audit_logs.py to resolve Ruff B008
2026-03-14 12:44:26 +01:00
copilot-swe-agent[bot] 1416d901d0 fix(i18n): persist language preference server-side, fix dropdown rendering and avatar 404
- Fix language dropdown in base.html by moving tojson data out of HTML
  attribute into a script tag (prevents quote conflicts breaking Alpine.js)
- Fix avatar placeholder 404 by correcting filename reference from
  avatar-placeholder.svg to default-avatar.svg
- Add session hydration from DB in _inject_global_context() so
  detect_language() uses the stored preference on every request
- Sync session and cookie in PATCH /api/profile when language changes
- Reload page after language change in profile to reflect new locale
- Add tests for session/cookie sync and DB hydration

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-14 10:30:50 +00:00
google-labs-jules[bot] f96fcbe6c9 style: use Annotated pattern in audit_logs.py to resolve Ruff B008
Refactor `app/api/audit_logs.py` to use the `Annotated` type hint pattern for FastAPI dependencies (`Depends`) and query parameters (`Query`).

- Resolves B008: Function-call in default argument.
- Improves code maintainability and readability by following modern FastAPI best practices.
- Maintains consistency with other modules in the codebase (e.g., `files.py`, `integrations.py`).
- No changes to API runtime behavior.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-14 09:42:23 +00:00
copilot-swe-agent[bot] f70fa65969 Merge branch 'main' into copilot/add-conditional-routing
Resolve conflicts in app/api/__init__.py and app/models.py.
Renumber migration 027_add_routing_rules → 035_add_routing_rules.
Fix migration chain: down_revision → 034_add_user_profile_settings.
Add PipelineRoutingRule to migrations/env.py.
2026-03-12 22:07:54 +00:00
Christian Krakau-Louis 4345d5128f Merge pull request #607 from christianlouis/copilot/add-user-settings-management
fix(tests): update whoami_handler call sites to pass required db argument
2026-03-12 22:44:13 +01:00
copilot-swe-agent[bot] 9583d6d96f fix(profile): address code review feedback - early size check, CSRF helper, test constants
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:10:46 +00:00
copilot-swe-agent[bot] b0d6f1ab60 feat(profile): add user self-service profile settings page and API
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-12 13:00:19 +00:00