Commit Graph

176 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] 9ad32e40e6 feat(security): add configurable file upload size limits with optional splitting
- Add MAX_UPLOAD_SIZE config (default 1GB) to prevent resource exhaustion
- Add MAX_SINGLE_FILE_SIZE config for optional PDF file splitting
- Implement automatic PDF splitting when files exceed single file limit
- Update upload endpoint to use configured limits instead of hardcoded 500MB
- Add comprehensive tests for upload limits and file splitting
- Document configuration in ConfigurationGuide.md and SECURITY_AUDIT.md
- Reference SECURITY_AUDIT.md in error messages for user guidance

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-10 10:51:30 +00:00
copilot-swe-agent[bot] a3d0af2efc fix(security): address code review feedback on validation logic
- Improve comment documentation for defense-in-depth validation
- Fix test assertion to properly validate basename sanitization
- Note regex pattern duplication for future refactoring

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-10 10:28:39 +00:00
copilot-swe-agent[bot] 489aa67a13 fix(security): remediate path traversal vulnerabilities in file operations
- Fix critical vulnerability: sanitize GPT metadata filename before use
- Fix insecure string-based path validation with pathlib methods
- Add validation for GPT-extracted filenames
- Add comprehensive security test suite (24 tests)
- Document all findings in SECURITY_AUDIT.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-10 10:26:42 +00:00
copilot-swe-agent[bot] 3d2b21b3c8 style: fix linting issues (whitespace and formatting)
- Remove trailing whitespace from blank lines
- Apply black formatting to test file
- All tests still pass

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 21:05:44 +00:00
copilot-swe-agent[bot] 43b512fee8 refactor(security): improve sanitization logic and tests based on code review
- Change consecutive dots regex to simple replace for better precision
- Update tests to verify exact sanitized output
- Fix docstring syntax warning with raw string
- Add detailed comments explaining sanitization behavior
- All 43 tests pass (21 file upload + 22 filename utils)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 21:04:03 +00:00
copilot-swe-agent[bot] 2bcd774d6d fix(security): enhance path traversal protection in file uploads
- Import and use sanitize_filename utility in ui_upload endpoint
- Enhance sanitize_filename to handle Windows-style paths (backslashes)
- Add protection against path traversal patterns (..)
- Replace all path separators with underscores
- Add comprehensive security tests for Windows-style paths and mixed separators
- All existing tests pass with improved security

This addresses the "Uncontrolled data used in path expression" code scanning alert
by ensuring all user-provided filenames are properly sanitized before being used
in any file operations or stored in the database.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 21:00:51 +00:00
copilot-swe-agent[bot] 98cf9e0e0b refactor: consolidate get_db into single module
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 15:37:23 +00:00
copilot-swe-agent[bot] 546db7eff9 refactor: remove duplicate utils.py, dead license_routes, and pytest.ini
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-09 15:11:26 +00:00
copilot-swe-agent[bot] bb155501b4 fix(tasks): apply Black formatting to extract_metadata_with_gpt.py and rotate_pdf_pages.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 18:21:45 +00:00
copilot-swe-agent[bot] d08040ac4a style: fix all flake8 linter errors across app/ directory
- Run Black formatter and isort on all app/ files
- Remove unused imports (F401) across multiple files
- Add # noqa: F401 for intentional re-exports in celery_worker.py,
  tasks/__init__.py, utils.py, frontend.py, views/base.py
- Fix f-strings without placeholders (F541) in azure.py, notification.py,
  check_credentials.py, upload_to_onedrive.py, settings.py
- Fix bare except (E722) in upload_to_sftp.py
- Fix block comment format (E265) in models.py
- Move imports to top of file to fix E402 in celery_app.py, celery_worker.py
- Fix line-too-long (E501) by wrapping strings in multiple files
- Remove unused variable (F841) in upload_to_nextcloud.py

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 17:42:33 +00:00
copilot-swe-agent[bot] 334714423e fix(ui): address code review feedback on inline preview
- Remove invalid type attribute from iframe elements
- Add dedicated download endpoint with attachment disposition
- Update download links to use new endpoint instead of preview

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 16:16:31 +00:00
copilot-swe-agent[bot] ed0bac7345 feat(ui): add auto-refresh after upload and inline file preview
- Auto-refresh files table after successful uploads using custom event
- Add inline preview support for PDFs, images, and text files
- Set Content-Disposition header to inline for preview endpoint
- Add download button as secondary action in file details view

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 16:12:44 +00:00
copilot-swe-agent[bot] b97c80d6bd fix: address code review feedback on security and type hints
- Use Optional[int] type hint for timeout parameter in oauth_helper
- Replace bare Exception with specific ValueError and JSONDecodeError
- Strengthen rclone remote name validation (must start with alphanumeric)
- Fix path traversal validation to check against workdir for absolute paths
- Add comprehensive comments for security validations

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:31:21 +00:00
copilot-swe-agent[bot] d2eb9846d3 style: fix linting issues and remove unused imports
- Remove unused imports from all modified files
- Fix flake8 violations (unused variables, f-strings without placeholders)
- Apply Black formatting consistently
- Shorten long line in google_drive.py

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:29:38 +00:00
copilot-swe-agent[bot] 551b23a80c fix(security): reduce code duplication and fix security issues in OAuth and file handling
- Extract common OAuth token exchange logic to shared utility (oauth_helper.py)
- Remove sensitive data logging (client_secret, authorization codes)
- Add path traversal validation in resolve_file_path()
- Add input validation for rclone destination parameter
- Replace bare Exception catches with specific exception types
- Use RuntimeError instead of generic Exception for better error handling

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:26:10 +00:00
copilot-swe-agent[bot] 78004cd9a4 style: Apply Black formatting to modified files
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:16:54 +00:00
copilot-swe-agent[bot] 0a461343e7 fix: Add service containers and fix test failures
- Add Redis and RabbitMQ services to CI workflow
- Fix Jinja2 template error by passing file=None in error cases
- Fix test expecting dict response format for list_files endpoint
- Fix NOT NULL constraint by providing valid local_filename
- Fix retry-subtask to validate subtask name before checking processed file
- Add mock for process_document in reprocess test

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:14:09 +00:00
Christian Krakau-Louis b006e92b91 Add RedirectResponse import to general routes 2026-02-08 08:40:59 +01:00
Christian Krakau-Louis 29bfb52afb fixed logger error 2026-02-08 08:36:27 +01:00
Christian Krakau-Louis aa953f2f36 Add local development Dockerfile and fallback for build metadata 2026-02-08 08:32:08 +01:00
copilot-swe-agent[bot] b6755f8b36 Fix line length to comply with Black formatting (120 chars)
- Break long line in lifespan function to multiple lines
- Remove test_startup.py (not needed, existing tests validate startup)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 07:18:21 +00:00
copilot-swe-agent[bot] ad31f26beb Migrate from deprecated @app.on_event to lifespan context manager
- Convert sync and async startup handlers to single lifespan function
- Remove deprecated @app.on_event("startup") and @app.on_event("shutdown")
- Add asynccontextmanager import for lifespan pattern
- Consolidate all startup/shutdown logic into one place
- Fixes Python 3.14 compatibility issue causing startup failure

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 07:16:57 +00:00
copilot-swe-agent[bot] a168723db9 Bump version to 0.5.0 for settings management release
Settings management represents a significant new feature warranting a minor version bump:
- Database-backed configuration with 102 settings
- Fernet encryption for sensitive values
- Setup wizard for first-time installations
- Complete admin UI and REST API
- OAuth admin group support

Version updated: 0.3.3 → 0.5.0

Keep 0.3.3 release notes intact (drag-and-drop feature from main branch).
Add 0.5.0 as new current release with settings features.

Updated files:
- VERSION: 0.5.0
- CHANGELOG.md: Added 0.5.0 release, kept 0.3.3 intact
- MILESTONES.md: Added v0.5.0 section, adjusted future versions
- TODO.md: Updated current version
- ROADMAP.md: Updated current status
- app/config.py: Default version 0.5.0-dev
- docs/BuildMetadata.md: Updated reference
- ANALYSIS_SUMMARY.md: Updated version

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:35:02 +00:00
copilot-swe-agent[bot] 3046db0864 Update version to 0.3.3 and release documentation
VERSION:
- Update from 0.1.0-test to 0.3.3

CHANGELOG.md:
- Create comprehensive changelog for all releases
- Document v0.3.3 features: settings management, encryption, setup wizard
- List all new files, changes, fixes, and security improvements

TODO.md:
- Update version reference to v0.3.3
- Mark settings management features as complete
- Add completed items for 2026-02-08
- Update last review date

MILESTONES.md:
- Update last updated date
- Mark v0.3.3 as Released (2026-02-08)
- Add comprehensive release notes with all features
- Update version history table
- Update current release to v0.3.3 with new features listed

ROADMAP.md:
- Update to reflect v0.3.3 current status
- Add settings management features to current status
- Update last updated date

Other files:
- app/config.py: Update default version to 0.3.3-dev
- docs/BuildMetadata.md: Update default version reference
- ANALYSIS_SUMMARY.md: Update current version

All documentation now reflects v0.3.3 release with complete feature list.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:29:55 +00:00
copilot-swe-agent[bot] 1a01811882 Add encryption and setup wizard features
ENCRYPTION:
- Add cryptography library for secure storage
- Implement Fernet encryption for sensitive settings
- Key derived from SESSION_SECRET
- Auto-encrypt/decrypt transparent to app
- "enc:" prefix identifies encrypted values
- Graceful fallback if crypto unavailable

SETUP WIZARD:
- Detect fresh installs needing configuration
- 3-step wizard: Infrastructure, Security, AI Services
- "/" redirects to wizard if setup required
- Auto-generate session_secret option
- Skip option for advanced users
- Beautiful UI with progress indicators

UI IMPROVEMENTS:
- Enhanced sensitive field display
- Lock icon showing encryption status
- Improved show/hide toggle for passwords
- Better visual hierarchy

FILES:
- app/utils/encryption.py - Encryption utilities
- app/utils/setup_wizard.py - Wizard detection logic
- app/views/wizard.py - Wizard routes
- frontend/templates/setup_wizard.html - Wizard UI
- requirements.txt - Added cryptography
- IMPLEMENTATION_CHECKLIST.md - Status tracking

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:18:49 +00:00
copilot-swe-agent[bot] 121b0d811f Add settings source indicators and improve form UX
- Remove HTML 'required' attributes - all fields optional
- Add source detection (DB/ENV/DEFAULT) for each setting
- Display color-coded badges showing setting source
- Update template with precedence order explanation
- Pre-fill form with current values from DB/ENV/defaults
- Update documentation with source badge explanations
- Test and verify form prefilling works correctly

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:13:25 +00:00
copilot-swe-agent[bot] 02fa9ccde4 Address code review feedback
- Clarify test expectations for auth-disabled environment
- Skip timestamp update test (DB backend dependent)
- Document require_admin_access sync/async support
- Restore session_secret and external_hostname as required
- Add comments explaining validation and security requirements

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:09:23 +00:00
copilot-swe-agent[bot] c9e1de4845 Fix /settings redirect issue and add OAuth admin support
- Convert require_admin_access to proper decorator pattern
- Fix redirect loop that was sending all users to /
- Add is_admin flag handling for OAuth users (checks groups)
- Update SETTING_METADATA with all 102 settings from config.py
- Improve API admin check with type hints

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:02:21 +00:00
copilot-swe-agent[bot] 4b4c208035 Security fixes: use tojson filter, fix Tuple import, remove raw value exposure
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:38:58 +00:00
copilot-swe-agent[bot] 54e93dcdf2 Fix settings API route prefix to avoid double /api/ in URL
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:36:32 +00:00
copilot-swe-agent[bot] 58b960db14 Add comprehensive tests for settings management and fix Union import
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:33:49 +00:00
copilot-swe-agent[bot] e1bb9de915 Add settings management infrastructure: models, API, views, and database loading
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:31:06 +00:00
Christian Krakau-Louis db96c0d9e7 Merge pull request #123 from christianlouis/copilot/add-files-detail-view-features
Add branching visualization and per-subtask retry to file detail view
2026-02-07 23:20:32 +01:00
copilot-swe-agent[bot] 910fe86603 Add branching visualization, per-subtask retry, and step summary to file detail view
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:08:09 +00:00
copilot-swe-agent[bot] 90d69979dc Fix S3 upload error: Pass file_id as keyword argument in Celery tasks
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 21:56:38 +00:00
copilot-swe-agent[bot] 76ab698105 Add backend endpoints and enhanced file detail view
- Added /api/files/{file_id}/reprocess endpoint for single file reprocessing
- Added /api/files/{file_id}/preview endpoint for viewing original/processed files
- Enhanced file detail view with process flow computation
- Updated frontend template with retry button, process flow visualization, and PDF previews
- Added JavaScript for async retry functionality

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 20:52:08 +00:00
Christian Krakau-Louis feefae51f1 Merge pull request #113 from christianlouis/copilot/run-security-audit-with-bandit
Security audit: Fix 21 vulnerabilities and integrate Bandit into CI
2026-02-07 21:42:07 +01:00
copilot-swe-agent[bot] 7fc2703d90 Increase HTTP request timeout to 120s for large file processing (up to 1GB PDFs)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 19:39:57 +00:00
copilot-swe-agent[bot] 5f175fabee Add automated build metadata generation system
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 19:37:40 +00:00
copilot-swe-agent[bot] edd1acb3a1 Fix all high and medium severity security issues found by Bandit
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 19:35:40 +00:00
copilot-swe-agent[bot] 74724973d8 Fix API routes to return JSON instead of HTML on errors
- Modified exception handlers in app/main.py to check if request path starts with /api/
- HTTPException handler returns JSON for API routes, HTML for frontend routes
- General exception handler (500) also checks and returns appropriate format
- Enhanced frontend deleteFile() to handle non-JSON responses gracefully
- Added content-type checking before parsing JSON
- Added comprehensive tests for API error handling

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 19:18:42 +00:00
copilot-swe-agent[bot] dabd9a944c Final review fixes: consistent parameter order and improved test documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:59:42 +00:00
copilot-swe-agent[bot] e20c88dbc7 Address code review feedback: improve variable naming and parameter order
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:58:08 +00:00
copilot-swe-agent[bot] 22a14431b3 Standardize upload task signatures to accept file_id parameter
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:55:30 +00:00
copilot-swe-agent[bot] 7d24b7c668 Apply code formatting and fix linting issues
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:37:50 +00:00
copilot-swe-agent[bot] 2db58afedf Add original_filename parameter to preserve user's filename
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:36:36 +00:00
copilot-swe-agent[bot] 3f1006b035 Code review feedback: improve readability of S3 upload call
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:28:04 +00:00
copilot-swe-agent[bot] c0e049ae96 Fix OneDrive and S3 upload signature - add file_id parameter
- Updated upload_to_onedrive to accept file_id parameter with bind=True
- Updated upload_to_s3 to accept file_id parameter with bind=True
- Added proper logging with task_id and file_id tracking
- Added comprehensive unit tests for both functions
- All tests passing (8/8)

Fixes #99 and #100

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:25:23 +00:00
Christian Krakau-Louis c9af116ae9 Merge pull request #90 from christianlouis/copilot/fix-worker-file-processing
Fix DetachedInstanceError in process_document task
2026-02-07 18:09:45 +01:00
copilot-swe-agent[bot] 850afd26bd Fix DetachedInstanceError in process_document task by storing file_id before session closes
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 15:59:39 +00:00