Commit Graph

557 Commits

Author SHA1 Message Date
github-actions[bot] ce40cbcdd8 chore(release): update build metadata files [skip ci] 2026-02-11 21:53:37 +00:00
Christian Krakau-Louis 7222740204 feat(auth): add configurable admin group name for OAuth user access 2026-02-11 22:53:13 +01:00
github-actions[bot] 0a3c8987f8 chore(release): update build metadata files [skip ci] 2026-02-11 21:39:55 +00:00
Christian Krakau-Louis 50cc0035d9 feat(database): add file path columns to files table during migration
fix(general): update processed files count query to use FileRecord.id

chore: add VSCode settings for pytest configuration
2026-02-11 22:39:21 +01:00
github-actions[bot] 90b50d3625 chore(release): update build metadata files [skip ci] 2026-02-11 20:36:24 +00:00
Christian Krakau-Louis 3a23d0b2d7 Merge pull request #249 from christianlouis/copilot/reorganize-document-storage-structure
Implement immutable storage architecture with collision-resistant naming
2026-02-11 21:36:05 +01:00
copilot-swe-agent[bot] a247393e18 fix: update test to use new collision handling function
- Replace unique_filepath with get_unique_filepath_with_counter in tests
- Update test expectations for -0001 suffix format
- All 40 tests passing successfully

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 20:17:03 +00:00
copilot-swe-agent[bot] c7dff41492 docs: add comprehensive documentation for storage reorganization
- Create StorageArchitecture.md with detailed architecture explanation
- Document collision handling with -0001 suffix format
- Explain immutable original and processed file storage
- Document forced Cloud OCR reprocessing feature
- Update API.md with new reprocess-with-cloud-ocr endpoint
- Update UserGuide.md with user-facing feature descriptions
- Include database schema, lifecycle diagrams, and examples

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 20:14:56 +00:00
copilot-swe-agent[bot] 952e27b339 test: add tests for storage reorganization features
- Add tests for get_unique_filepath_with_counter collision handling
- Add tests for immutable original storage
- Add tests for metadata augmentation with file paths
- Add tests for force_cloud_ocr parameter
- Verify existing process_document tests still pass

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 20:13:17 +00:00
copilot-swe-agent[bot] 73a0222e56 feat(storage): implement immutable original/processed file storage and collision handling
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 20:10:19 +00:00
copilot-swe-agent[bot] b5c669f3fe Initial plan 2026-02-11 20:04:57 +00:00
github-actions[bot] 3103ebed2b chore(release): update build metadata files [skip ci] 2026-02-11 19:58:10 +00:00
Christian Krakau-Louis 9771277239 Merge pull request #248 from christianlouis/copilot/fix-dashboard-file-status
Fix dashboard status tracking with explicit FileProcessingStep table
2026-02-11 20:57:48 +01:00
copilot-swe-agent[bot] 9145060d8e docs: Add comprehensive architecture documentation for status tracking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:49:41 +00:00
copilot-swe-agent[bot] 65b6dc6985 feat: Add comprehensive tests and migration utility for status table
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:48:17 +00:00
copilot-swe-agent[bot] ad74bc852d feat: Add explicit FileProcessingStep table for status tracking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:45:09 +00:00
copilot-swe-agent[bot] 52b65868a4 refactor: Address code review feedback - remove redundant imports and use _ for unused variables
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:42:17 +00:00
copilot-swe-agent[bot] 72cea0d8c9 fix: Update existing test to include timestamp in MockLog
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:40:34 +00:00
copilot-swe-agent[bot] d8d4294ff0 fix: Make _compute_step_summary order-independent to fix dashboard status bug
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 19:38:29 +00:00
copilot-swe-agent[bot] 2339a449b8 Initial plan 2026-02-11 19:33:10 +00:00
github-actions[bot] 97f815258d chore(release): update build metadata files [skip ci] 2026-02-11 19:15:15 +00:00
Christian Krakau-Louis 83abb3984a Merge pull request #245 from christianlouis/copilot/fix-dashboard-status-metrics
Fix file status and metrics by tracking latest state per step
2026-02-11 20:14:49 +01:00
copilot-swe-agent[bot] 0fc3fca1b2 style: format test file with black
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:36:42 +00:00
copilot-swe-agent[bot] 2470768599 test: add timestamps to MockLog for consistency and clarity
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:35:29 +00:00
copilot-swe-agent[bot] 4fb696e1eb fix: correct file status and metrics to use latest status per step
- Fix _compute_status_from_logs to track latest status per unique step
- Fix _compute_step_summary to count only latest status per step
- Add comprehensive tests for both fixes
- Resolves issue where completed files showed as "Processing"
- Resolves issue where metrics showed incorrect counts

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:33:41 +00:00
copilot-swe-agent[bot] 1e5cbc4f70 Initial plan 2026-02-11 18:28:35 +00:00
Christian Krakau-Louis ada648630c Merge pull request #243 from christianlouis/copilot/fix-404-error-process-url-endpoint
Add regression tests for API endpoint registration
2026-02-11 19:27:43 +01:00
copilot-swe-agent[bot] 8c5ab00428 refactor: extract test URL constant for better maintainability
Address code review feedback by extracting the hardcoded test URL
into a module-level constant to improve maintainability and ensure
consistency across all test methods.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:24:03 +00:00
copilot-swe-agent[bot] fad63fcf4b test: add endpoint registration verification tests
Add comprehensive tests to verify critical API endpoints are registered,
including the /api/process-url endpoint. These tests will prevent future
regressions where endpoints might not be properly registered in the app.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:22:48 +00:00
copilot-swe-agent[bot] cf88754be6 Initial plan 2026-02-11 18:14:21 +00:00
github-actions[bot] 0a20f3e0b5 chore(release): update build metadata files [skip ci] 2026-02-11 18:10:22 +00:00
Christian Krakau-Louis b8b350440d Merge pull request #241 from christianlouis/copilot/fix-e2e-integration-tests
fix(ci): exclude E2E tests requiring Docker-in-Docker from CI workflow
2026-02-11 19:09:56 +01:00
copilot-swe-agent[bot] 624b1ddb57 fix(ci): exclude E2E tests requiring Docker-in-Docker from CI
- Add missing pytest markers (e2e, requires_docker) to pyproject.toml
- Update CI workflow to skip E2E tests with -m "not e2e"
- Update integration test documentation with CI configuration notes
- E2E tests can still be run locally with: pytest -m e2e

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 18:03:47 +00:00
copilot-swe-agent[bot] bf8f295f3b Initial plan 2026-02-11 18:01:00 +00:00
github-actions[bot] fdd524e6dd chore(release): update build metadata files [skip ci] 2026-02-11 18:00:23 +00:00
Christian Krakau-Louis f254e1b14b fix: force-add build metadata files to override .gitignore 2026-02-11 18:59:59 +01:00
Christian Krakau-Louis bb1370d46e Merge pull request #239 from christianlouis/copilot/update-tests-external-apis
Fix DeepSource issues: lazy logging, static methods, unused vars, reimports
2026-02-11 18:56:48 +01:00
copilot-swe-agent[bot] bf0151340d fix: resolve DeepSource issues in test files
- Use lazy % formatting in logging (PYL-W1203) in test_external_integrations.py
- Add @staticmethod to 3 methods not using self (PYL-R0201)
- Remove unused mock_media and mock_smtp variables (PYL-W0612)
- Remove redundant local reimports of upload_to_webdav (PYL-W0404)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 17:53:30 +00:00
copilot-swe-agent[bot] 5e82f7c03a refactor: fix static method warnings, hard-coded credentials, and Annotated type hints
- Add @staticmethod to 9 test methods in test_external_integrations.py
  that don't use self (PYL-R0201)
- Extract hard-coded password literals to constants in 6 test files
  to resolve S2068 warnings (fixtures_integration, test_imap_tasks,
  test_upload_tasks, test_upload_webdav_comprehensive,
  test_upload_webdav_integration, test_views_coverage)
- Migrate Form() dependency injection to Annotated type hints in
  dropbox.py, google_drive.py, onedrive.py (Sonar fastapi convention)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 17:03:54 +00:00
copilot-swe-agent[bot] 3a40fe59f5 refactor(api): migrate Depends() to Annotated type hint style in app/api/
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 16:54:55 +00:00
copilot-swe-agent[bot] 7c4d6fd025 fix: extract hard-coded test credentials to module-level constants (S2068)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 16:50:37 +00:00
copilot-swe-agent[bot] 7ad2bdfc4b fix(test): fix 5 pre-existing test failures in rate limiting, path traversal, and e2e tests
- test_rate_limiting: remove references to non-existent rate_limit_process setting
- test_path_traversal_security: fix sanitize_filename assertion to match actual
  strip behavior, fix os.path.basename test for Linux (backslash not a separator),
  remove erroneous task_mock arg from embed_metadata_into_pdf direct call
- test_e2e_full_stack: add psycopg2 availability check to skip Postgres test
  when driver is not installed

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 15:31:46 +00:00
copilot-swe-agent[bot] 9747487e2b fix(test): correct spelling of recognizable in test comment
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 15:17:39 +00:00
copilot-swe-agent[bot] e658dec83c feat(test): add comprehensive external API integration tests with PDF generation
- Add fpdf2 dependency for dynamic test PDF generation
- Create tests/test_external_integrations.py with end-to-end pipeline tests:
  - OpenAI: key validation, metadata extraction via chat completion
  - Azure Document Intelligence: admin connectivity, full OCR on generated PDF
  - S3: bucket access, upload/download/delete pipeline
  - Dropbox: token refresh, upload/download/delete pipeline
  - OneDrive: token refresh, upload/download/delete pipeline
  - Authentik: OIDC discovery endpoint, credential consistency
  - Full pipeline: Azure OCR → OpenAI metadata extraction
- Update tests/conftest.py to capture original env vars before test overrides
- Add has_real_env() helper and original_env fixture for credential detection
- All external tests use @pytest.mark.requires_external and skipif guards
- Test files are dynamically generated with unique content per run
- Uploaded test files are cleaned up in finally blocks

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 15:16:53 +00:00
copilot-swe-agent[bot] e70d06e789 Initial plan 2026-02-11 14:53:55 +00:00
Christian Krakau-Louis ef97746a5f Merge pull request #237 from christianlouis/copilot/fix-autorelease-version-sync
fix(build): sync VERSION file with semantic-release tags and add build time
2026-02-11 15:51:59 +01:00
copilot-swe-agent[bot] e8bba38f98 docs: fix numbering in BuildMetadata.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 14:45:22 +00:00
copilot-swe-agent[bot] 037b4aa94e fix(build): sync VERSION file with semantic-release tags and add build time
- Update generate_build_metadata.sh to sync VERSION from latest git tag
- Change BUILD_DATE format from date-only to ISO 8601 with time (YYYY-MM-DDTHH:MM:SSZ)
- Fix VERSION file from 0.5.0 to 0.9.1 (matching latest git tag v0.9.1)
- Change version fallback from hardcoded '0.5.0-dev' to 'unknown' in config.py
- Update release.yml to commit all build metadata files (not just VERSION)
- Update BuildMetadata.md documentation to reflect automated versioning
- Add tests for build_date with time format and version unknown fallback

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 14:42:37 +00:00
copilot-swe-agent[bot] fac4758116 Initial plan 2026-02-11 14:31:03 +00:00
Christian Krakau-Louis 0be7b7185b Merge pull request #236 from christianlouis/copilot/fix-backend-url-processing-error
fix(api): fix process-url endpoint crashing on request.session access
2026-02-11 15:28:30 +01:00