Commit Graph

122 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] 56f7f2351f feat(auth): add admin user management dashboard
- Add UserProfile model (app/models.py) with per-user settings: display_name, daily_upload_limit, notes, is_blocked
- Add Alembic migration 013_add_user_profiles for the new table
- Add REST API at /api/admin/users/ with list, get, upsert (PUT), delete endpoints (admin-only)
- Add HTML template admin_users.html with Alpine.js: filterable user list, paginated table, edit/create modal, delete confirmation modal
- Add view handler at /admin/users (admin-only redirect guard)
- Register routers in app/api/__init__.py and app/views/__init__.py
- Add 'Users' link to admin nav dropdown in base.html (desktop + mobile)
- Add 27 tests covering auth, list, get, upsert, delete, and model constraints
- Register UserProfile in conftest.py model imports
- Document new endpoints in docs/API.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-06 13:45:35 +00:00
Christian Krakau-Louis 26b4d04d76 Merge pull request #482 from christianlouis/copilot/add-database-configuration-wizard
feat(database): integrate wizard into settings page, improve accessibility and test coverage
2026-03-06 11:28:44 +01:00
copilot-swe-agent[bot] 7dae15fd29 docs(database): update configuration guides with wizard cross-references, clean up review feedback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-06 10:13:40 +00:00
copilot-swe-agent[bot] cb3bf809cc docs(database): add wizard and migration tool documentation
Update DatabaseConfiguration.md with sections for the new Database
Configuration Wizard and Database Migration Tool. Also update API.md
with the new /api/database/ endpoint documentation.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-05 22:19:39 +00:00
copilot-swe-agent[bot] c8bc4afc93 fix: address code review — use modern type hints and Callable annotation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-05 22:03:30 +00:00
copilot-swe-agent[bot] 3601e2ca5c feat(ui): add user autocomplete widget for default_owner_id, user search API, and documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-05 21:40:41 +00:00
copilot-swe-agent[bot] d7b7f1478f test(multi-user): add comprehensive tests for multi-user isolation and feature flag
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-05 11:34:41 +00:00
copilot-swe-agent[bot] 3080e37ddf docs(settings): update SettingsManagement.md with new categories and setting types
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 16:45:15 +00:00
Christian Krakau-Louis 8fee5c8d0b Merge branch 'main' into copilot/add-pdfa-export-option 2026-03-02 16:26:31 +01:00
copilot-swe-agent[bot] a03b3af933 feat(pdfa): add FreeTSA timestamping, per-provider folder overrides, individual upload toggles
- Add RFC 3161 timestamping via FreeTSA (PDFA_TIMESTAMP_ENABLED, PDFA_TIMESTAMP_URL)
- Replace PDFA_UPLOAD_TO_PROVIDERS with individual PDFA_UPLOAD_ORIGINAL and PDFA_UPLOAD_PROCESSED
- Add PDFA_UPLOAD_FOLDER setting for per-provider subfolder configuration
- Add GOOGLE_DRIVE_PDFA_FOLDER_ID for Google Drive-specific folder override
- Add folder_override parameter to all 8 folder-using upload tasks
- Add folder_overrides dict parameter to send_to_all_destinations
- Add _compute_pdfa_folder_overrides() and _timestamp_file() helpers
- Expand tests to 26 (timestamping, folder overrides, individual toggles)
- Update docs/ConfigurationGuide.md and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:55:34 +00:00
copilot-swe-agent[bot] eea99eb01d feat(pdfa): add PDF/A archival conversion using ocrmypdf
- Add ENABLE_PDFA_CONVERSION, PDFA_FORMAT, PDFA_UPLOAD_TO_PROVIDERS config settings
- Add original_pdfa_path and processed_pdfa_path columns to FileRecord model
- Create Alembic migration 011_add_pdfa_paths
- Create app/tasks/convert_to_pdfa.py Celery task using ocrmypdf + Ghostscript
- Integrate PDF/A conversion into finalize_document_storage pipeline
- Add comprehensive unit tests (15 tests)
- Update .env.demo and docs/ConfigurationGuide.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:35:13 +00:00
copilot-swe-agent[bot] 8e955f3c81 fix(similarity): truncate text to fit embedding model context window, fix step tracking
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:27:34 +00:00
copilot-swe-agent[bot] ebea83a750 feat(duplicates): add duplicate document detection and management
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 10:21:57 +00:00
Christian Krakau-Louis 1e1ba1d8d7 Merge branch 'main' into copilot/add-document-similarity-detection 2026-03-01 22:06:55 +01:00
copilot-swe-agent[bot] a85992ee79 docs(webhooks): fix spelling - Behaviour to Behavior
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 20:47:51 +00:00
copilot-swe-agent[bot] b7c78177e9 docs(similarity): add API documentation and fix template accessibility
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 20:46:13 +00:00
copilot-swe-agent[bot] a953b726fc docs(webhooks): add webhook API and configuration documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 20:45:20 +00:00
Christian Krakau-Louis ecca0ecdfa Merge pull request #461 from christianlouis/copilot/enhance-release-versioning 2026-03-01 20:31:17 +01:00
Christian Krakau-Louis 68ea6aabc7 Merge pull request #448 from christianlouis/copilot/add-retry-logic-backoff 2026-03-01 20:30:45 +01:00
copilot-swe-agent[bot] 85da309740 feat(api): add GET /api/diagnostic/health endpoint for monitoring
- Add health check endpoint at GET /api/diagnostic/health
- Auth-protected via @require_login (no-op when AUTH_ENABLED=False)
- Checks database (SELECT 1) and Redis (ping) with 2s timeouts
- Returns healthy/degraded/unhealthy with per-check detail
- Returns HTTP 503 when database is down, 200 otherwise
- 7 new unit tests covering all status scenarios
- Update docs/API.md with Grafana/monitoring integration notes
- Fixes test_cors_headers_absent_when_disabled CI timeout

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 18:36:38 +00:00
copilot-swe-agent[bot] d18c10996a feat(release): add named release anchors with codenames and roadmap integration
- Add release_names.json mapping version ranges to codenames
- Add release_name property to Settings in app/config.py
- Update build metadata script to include codename in RUNTIME_INFO
- Display release codename in status dashboard and page footer
- Inject release_name globally via template response wrapper
- Update ROADMAP.md with codenames for all milestone releases
- Add docs/ReleaseNaming.md with naming guide and best practices
- Add comprehensive tests for release name resolution

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 18:32:46 +00:00
copilot-swe-agent[bot] 5ff7b72a80 feat(tasks): add retry logic with exponential backoff and jitter
- Rewrite app/tasks/retry_config.py with compute_countdown() function
  implementing per-retry delays with ±20% jitter (default: 60s, 300s, 900s)
- Add BaseTaskWithRetry.retry() override to inject proper countdown
- Add OcrTaskWithRetry (120s, 600s, 1800s) for OCR/AI tasks
- Add UploadTaskWithRetry for cloud-storage upload tasks
- Add config settings: TASK_RETRY_MAX_RETRIES, TASK_RETRY_DELAYS, TASK_RETRY_JITTER
- Update process_with_ocr and process_with_azure tasks to use OcrTaskWithRetry
- Update all 11 upload tasks to use UploadTaskWithRetry
- Add 38 unit tests in tests/test_retry_config.py
- Update docs/ConfigurationGuide.md and .env.demo

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 17:37:41 +00:00
copilot-swe-agent[bot] af34ce88df feat(api): implement proper pagination for file list API
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 17:31:56 +00:00
Christian Krakau-Louis 0134ed37d5 Merge pull request #446 from christianlouis/copilot/optimize-test-timeouts
fix(ci): split test pipeline into quick/integration stages, fix hanging test
2026-03-01 18:23:06 +01:00
Christian Krakau-Louis 8ac4179b8b Merge pull request #445 from christianlouis/copilot/enforce-database-upgrades-migration
refactor(database): enforce Alembic-only schema migrations, deprecate manual migration code
2026-03-01 18:22:48 +01:00
copilot-swe-agent[bot] 87d1b9d935 refactor(database): enforce Alembic-only database migrations, deprecate manual schema migrations
- Create alembic.ini and migrations/env.py for full Alembic CLI + programmatic support
- Add Alembic migration 006: detail column on processing_logs
- Add Alembic migration 007: ocr_quality_score column + drop unique filehash index
- Add Alembic migration 008: performance indexes
- Replace _run_schema_migrations() call in init_db() with _run_alembic_upgrade()
- Deprecate _run_schema_migrations() with DeprecationWarning
- Update tests for new Alembic-based approach and deprecation
- Update DatabaseConfiguration.md documentation

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:47:41 +00:00
copilot-swe-agent[bot] 6a21adcdd4 fix(ci): increase quick test job timeout to 15 minutes for CI runner headroom
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:46:57 +00:00
copilot-swe-agent[bot] c100250b23 docs: update CI workflow and integration test documentation for two-stage pipeline
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:45:44 +00:00
Christian Krakau-Louis 91bd4ad837 Merge pull request #444 from christianlouis/copilot/optimize-database-queries
fix(db): check column existence before creating indexes; fix mypy error in cache.py
2026-03-01 17:27:03 +01:00
copilot-swe-agent[bot] ce87b53331 style: fix spelling (British to American English)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:41:29 +00:00
copilot-swe-agent[bot] d5a95e9270 docs: add Performance & Caching section to ConfigurationGuide
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:40:19 +00:00
copilot-swe-agent[bot] f8c5dd539d feat(imap): add IMAP_READONLY_MODE feature flag to safeguard shared mailboxes
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:28:51 +00:00
Christian Krakau-Louis 96b14ec74f Merge pull request #440 from christianlouis/copilot/update-search-functionality
feat(search): add content-finding filters, saved searches, and text quality to Search view
2026-03-01 15:19:57 +01:00
copilot-swe-agent[bot] de09dba285 docs: update API and User Guide with search filters and saved searches
- Document new /api/search endpoint with all filter parameters
- Update saved searches docs with new allowed filter keys
- Update User Guide with search view filters and saved searches usage

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:55:55 +00:00
copilot-swe-agent[bot] 83c3405c98 feat: persist ocr_quality_score and use it for numeric filtering
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:46:43 +00:00
copilot-swe-agent[bot] 705b970522 feat: add bulk download, cloud OCR, and basic OCR quality filter
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:38:42 +00:00
Christian Krakau-Louis 05c35a6775 Merge pull request #437 from christianlouis/copilot/add-file-download-endpoint
fix(api): change download endpoint default version from `original` to `processed`
2026-03-01 13:30:30 +01:00
copilot-swe-agent[bot] cc9fd1a7fe docs(api): document advanced filtering and saved searches endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:29:47 +00:00
copilot-swe-agent[bot] 7b5494bafb feat(api): default download endpoint to processed file version
- Change default `version` param from 'original' to 'processed' so
  GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:15:14 +00:00
copilot-swe-agent[bot] 7a8ccba9be test(preview): add 23 tests for document preview features
Also updates UserGuide.md with documentation for pdf.js viewer,
image zoom/pan, text preview, and the file-list preview modal.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 07:42:48 +00:00
copilot-swe-agent[bot] 1f48503217 docs(a11y): add comprehensive AccessibilityGuide.md
Add docs/AccessibilityGuide.md with WCAG 2.1 Level AA standards, ARIA
patterns, keyboard navigation requirements, color contrast guidelines,
form/table/modal accessibility patterns, testing instructions, and a
developer checklist for UI changes.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-28 16:18:35 +00:00
copilot-swe-agent[bot] 4dd018f210 refactor(queue): address code review feedback — extract constants and sync refresh interval
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-27 00:47:58 +00:00
copilot-swe-agent[bot] a05690bd5f feat(upload): adaptive 429 backoff, full Gotenberg file types, directory traversal
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 23:24:26 +00:00
Christian Krakau-Louis 6ecf07abf9 Merge pull request #422 from christianlouis/copilot/fix-search-field-bug
fix(ui): fix search input clearing on keystroke; add dedicated /search page
2026-02-26 11:45:44 +01:00
copilot-swe-agent[bot] e9dd3ef1c2 docs(search): update UserGuide with search page documentation and fix import order
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:32:08 +00:00
copilot-swe-agent[bot] d83dfb12c6 fix: regenerate logo PNGs from SVG to fix DocuNova → DocuElevate branding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:05:38 +00:00
copilot-swe-agent[bot] 0821e2f989 feat(privacy): implement GDPR and global privacy compliance
- Add dismissable cookie notice banner to base template (essential
  cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
  UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
  Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
  Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
  decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
  rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
  covering cookie strategy, data transfer mechanisms, data subject
  rights handling matrix with response timelines, and market-specific
  notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
  key compliance content areas

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 15:15:24 +00:00
copilot-swe-agent[bot] 6e78e9e7cf docs: add missing docs to mkdocs nav and README index
- Add BrowserExtension to Getting Started section in mkdocs.yml nav
- Add SettingsManagement to Configuration section in mkdocs.yml nav
- Add new Security section with CredentialRotationGuide in mkdocs.yml nav
- Mirror all nav additions in docs/README.md index

Closes the documentation gaps identified in the health check audit:
all user-relevant guides (browser extension, settings management,
credential rotation) are now discoverable via ReadTheDocs/mkdocs.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 14:37:50 +00:00
Christian Krakau-Louis 9584c0263c Merge pull request #408 from christianlouis/copilot/audit-documentation-gaps
fix(mypy): add missing type annotations in meilisearch_client.py
2026-02-25 15:18:51 +01:00
copilot-swe-agent[bot] c428a7ec00 feat(ui): add dark mode support with system preference detection and localStorage persistence
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:59:49 +00:00