Commit Graph

200 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] af34ce88df feat(api): implement proper pagination for file list API
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 17:31:56 +00:00
Christian Krakau-Louis 0134ed37d5 Merge pull request #446 from christianlouis/copilot/optimize-test-timeouts
fix(ci): split test pipeline into quick/integration stages, fix hanging test
2026-03-01 18:23:06 +01:00
Christian Krakau-Louis 8ac4179b8b Merge pull request #445 from christianlouis/copilot/enforce-database-upgrades-migration
refactor(database): enforce Alembic-only schema migrations, deprecate manual migration code
2026-03-01 18:22:48 +01:00
copilot-swe-agent[bot] 87d1b9d935 refactor(database): enforce Alembic-only database migrations, deprecate manual schema migrations
- Create alembic.ini and migrations/env.py for full Alembic CLI + programmatic support
- Add Alembic migration 006: detail column on processing_logs
- Add Alembic migration 007: ocr_quality_score column + drop unique filehash index
- Add Alembic migration 008: performance indexes
- Replace _run_schema_migrations() call in init_db() with _run_alembic_upgrade()
- Deprecate _run_schema_migrations() with DeprecationWarning
- Update tests for new Alembic-based approach and deprecation
- Update DatabaseConfiguration.md documentation

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:47:41 +00:00
copilot-swe-agent[bot] 6a21adcdd4 fix(ci): increase quick test job timeout to 15 minutes for CI runner headroom
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:46:57 +00:00
copilot-swe-agent[bot] c100250b23 docs: update CI workflow and integration test documentation for two-stage pipeline
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:45:44 +00:00
Christian Krakau-Louis 91bd4ad837 Merge pull request #444 from christianlouis/copilot/optimize-database-queries
fix(db): check column existence before creating indexes; fix mypy error in cache.py
2026-03-01 17:27:03 +01:00
copilot-swe-agent[bot] ce87b53331 style: fix spelling (British to American English)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:41:29 +00:00
copilot-swe-agent[bot] d5a95e9270 docs: add Performance & Caching section to ConfigurationGuide
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:40:19 +00:00
copilot-swe-agent[bot] f8c5dd539d feat(imap): add IMAP_READONLY_MODE feature flag to safeguard shared mailboxes
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:28:51 +00:00
Christian Krakau-Louis 96b14ec74f Merge pull request #440 from christianlouis/copilot/update-search-functionality
feat(search): add content-finding filters, saved searches, and text quality to Search view
2026-03-01 15:19:57 +01:00
copilot-swe-agent[bot] de09dba285 docs: update API and User Guide with search filters and saved searches
- Document new /api/search endpoint with all filter parameters
- Update saved searches docs with new allowed filter keys
- Update User Guide with search view filters and saved searches usage

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:55:55 +00:00
copilot-swe-agent[bot] 83c3405c98 feat: persist ocr_quality_score and use it for numeric filtering
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:46:43 +00:00
copilot-swe-agent[bot] 705b970522 feat: add bulk download, cloud OCR, and basic OCR quality filter
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:38:42 +00:00
Christian Krakau-Louis 05c35a6775 Merge pull request #437 from christianlouis/copilot/add-file-download-endpoint
fix(api): change download endpoint default version from `original` to `processed`
2026-03-01 13:30:30 +01:00
copilot-swe-agent[bot] cc9fd1a7fe docs(api): document advanced filtering and saved searches endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:29:47 +00:00
copilot-swe-agent[bot] 7b5494bafb feat(api): default download endpoint to processed file version
- Change default `version` param from 'original' to 'processed' so
  GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:15:14 +00:00
copilot-swe-agent[bot] 7a8ccba9be test(preview): add 23 tests for document preview features
Also updates UserGuide.md with documentation for pdf.js viewer,
image zoom/pan, text preview, and the file-list preview modal.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 07:42:48 +00:00
copilot-swe-agent[bot] 1f48503217 docs(a11y): add comprehensive AccessibilityGuide.md
Add docs/AccessibilityGuide.md with WCAG 2.1 Level AA standards, ARIA
patterns, keyboard navigation requirements, color contrast guidelines,
form/table/modal accessibility patterns, testing instructions, and a
developer checklist for UI changes.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-28 16:18:35 +00:00
copilot-swe-agent[bot] 4dd018f210 refactor(queue): address code review feedback — extract constants and sync refresh interval
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-27 00:47:58 +00:00
copilot-swe-agent[bot] a05690bd5f feat(upload): adaptive 429 backoff, full Gotenberg file types, directory traversal
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 23:24:26 +00:00
Christian Krakau-Louis 6ecf07abf9 Merge pull request #422 from christianlouis/copilot/fix-search-field-bug
fix(ui): fix search input clearing on keystroke; add dedicated /search page
2026-02-26 11:45:44 +01:00
copilot-swe-agent[bot] e9dd3ef1c2 docs(search): update UserGuide with search page documentation and fix import order
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:32:08 +00:00
copilot-swe-agent[bot] d83dfb12c6 fix: regenerate logo PNGs from SVG to fix DocuNova → DocuElevate branding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:05:38 +00:00
copilot-swe-agent[bot] 0821e2f989 feat(privacy): implement GDPR and global privacy compliance
- Add dismissable cookie notice banner to base template (essential
  cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
  UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
  Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
  Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
  decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
  rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
  covering cookie strategy, data transfer mechanisms, data subject
  rights handling matrix with response timelines, and market-specific
  notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
  key compliance content areas

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 15:15:24 +00:00
copilot-swe-agent[bot] 6e78e9e7cf docs: add missing docs to mkdocs nav and README index
- Add BrowserExtension to Getting Started section in mkdocs.yml nav
- Add SettingsManagement to Configuration section in mkdocs.yml nav
- Add new Security section with CredentialRotationGuide in mkdocs.yml nav
- Mirror all nav additions in docs/README.md index

Closes the documentation gaps identified in the health check audit:
all user-relevant guides (browser extension, settings management,
credential rotation) are now discoverable via ReadTheDocs/mkdocs.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 14:37:50 +00:00
Christian Krakau-Louis 9584c0263c Merge pull request #408 from christianlouis/copilot/audit-documentation-gaps
fix(mypy): add missing type annotations in meilisearch_client.py
2026-02-25 15:18:51 +01:00
copilot-swe-agent[bot] c428a7ec00 feat(ui): add dark mode support with system preference detection and localStorage persistence
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:59:49 +00:00
copilot-swe-agent[bot] 139e23c9e4 docs: add Setup Wizard, Production Readiness, Database, K8s, and Licensing guides
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:52:22 +00:00
copilot-swe-agent[bot] a08bfb9f1a feat(helm): add Helm chart for Kubernetes deployment and update DeploymentGuide
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:31:54 +00:00
copilot-swe-agent[bot] 89d5df71c8 feat(ocr): fine-tune OCR quality criteria with stricter threshold and head-to-head comparison
- Raise quality acceptance threshold from 65→85 (configurable via TEXT_QUALITY_THRESHOLD)
- Reject text with significant issues (excessive_typos, garbage_characters,
  incoherent_text, fragmented_sentences) even when score is above threshold
  (configurable via TEXT_QUALITY_SIGNIFICANT_ISSUES)
- Add compare_text_quality() for AI-powered head-to-head comparison of
  original embedded text vs fresh OCR output
- Update process_document to pass original text to OCR task for comparison
- Update process_with_ocr to run comparison and keep the higher-quality text
- Add new settings to settings_service.py metadata
- Update docs/ConfigurationGuide.md with new settings
- Add comprehensive tests for new threshold and comparison logic

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:03:21 +00:00
copilot-swe-agent[bot] b03bfb5e02 feat(ocr): add AI-based embedded text quality check with automatic OCR fallback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-24 18:41:27 +00:00
copilot-swe-agent[bot] 8a2a4dc2b3 feat(ocr): auto-install Tesseract/EasyOCR languages from settings
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
  missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
  pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
  running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
  settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
  endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-24 16:28:14 +00:00
copilot-swe-agent[bot] fec032643b feat(ocr): embed searchable text layer for providers without native PDF output
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-24 13:19:52 +00:00
copilot-swe-agent[bot] f9d849f7eb feat(api): remove redundant /env and /api/diagnostic/settings endpoints
- Remove `/api/diagnostic/settings` endpoint (superseded by `/api/settings/`)
- Remove `/env` view route (superseded by `/settings` admin page)
- Delete `env_debug.html` template
- Remove `/env` nav links from base.html (desktop + mobile)
- Update status_dashboard.html to link to /settings instead of /env
- Remove corresponding tests for deleted endpoints
- Update RateLimitingStrategy.md docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 23:23:28 +00:00
copilot-swe-agent[bot] 044da0ff47 feat: update all OpenAI/ChatGPT refs + settings dropdown for fixed-value fields
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 20:06:57 +00:00
copilot-swe-agent[bot] dcdef44303 feat: add Portkey provider support and null-content guard to AI abstraction layer
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 19:42:03 +00:00
copilot-swe-agent[bot] a46cd6634b feat(api): implement API key rotation mechanisms (#168)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-22 19:24:38 +00:00
copilot-swe-agent[bot] ca27a0b687 feat(security): add request size limits to API endpoints
- Add RequestSizeLimitMiddleware that checks Content-Length header
  before request body is read: non-multipart requests capped at
  MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
  MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
  in 64 KB chunks (bounded memory usage), removing the post-write
  os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
  the now-unused os.path.getsize mock

Closes #173

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-22 14:41:04 +00:00
copilot-swe-agent[bot] ac51f7206a feat(ci): add pip-audit dependency vulnerability scanning to CI/CD
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-21 21:17:15 +00:00
copilot-swe-agent[bot] 10ff772fbe docs: add GitHub Copilot configuration documentation
Add comprehensive documentation for .github/copilot.yml network allowlist configuration.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-16 20:37:13 +00:00
copilot-swe-agent[bot] 5380c6de6c docs: update browser extension documentation for web clipping
- Updated browser-extension/README.md with web clipping features
- Updated docs/BrowserExtension.md with dual-mode architecture
- Added documentation for clip mode data flow and API endpoints
- Updated permissions explanation for new clipping capabilities
- Added troubleshooting for clip-specific issues
- Documented version 1.1.0 features and changes

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 23:24:10 +00:00
copilot-swe-agent[bot] 16ed181cdd docs: add CI de-duplication summary document
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 11:11:39 +00:00
copilot-swe-agent[bot] 6c6bc6d90a feat: remove DeepSource and update CI documentation
- Remove .deepsource.toml (redundant with Ruff + CodeQL)
- Update docs/CIWorkflow.md to reflect current tool stack (Ruff instead of legacy tools)
- Create comprehensive docs/CIToolsGuide.md documenting streamlined CI setup
- Update README.md documentation index
- Update TODO.md to reference Ruff instead of legacy tools
- Document tool overlap analysis and de-duplication rationale

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 11:10:01 +00:00
copilot-swe-agent[bot] b03ebab747 chore: apply ruff formatting and fix whitespace issues
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 09:12:11 +00:00
copilot-swe-agent[bot] 5e8ee5440f docs: add comprehensive OAuth testing CI/CD guide
- Document GitHub Actions configuration for mock and real OAuth
- Add security best practices for OAuth secrets
- Include troubleshooting guide
- Provide complete workflow examples

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 11:11:01 +00:00
Christian Krakau-Louis c85a6710b6 Merge pull request #256 from christianlouis/copilot/add-browser-plugin-for-docuelevate
feat(browser): add browser extension for URL-based file submission
2026-02-12 04:17:46 +01:00
copilot-swe-agent[bot] 0a3fe11f66 docs(browser): add permissions guide and improve documentation
- Create comprehensive PERMISSIONS.md explaining all permissions
- Document empty host_permissions array and privacy benefits
- Add session cookie security best practices to BrowserExtension.md
- Clarify future use case for content script message handler
- Improve error message clarity for JSON parsing failures

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 03:11:33 +00:00
copilot-swe-agent[bot] d025923ef5 ci: split linters into independent parallel jobs and add CI documentation
Refactors .github/workflows/tests.yaml so that flake8, black, mypy,
pylint, and bandit each run as their own job in parallel with the test
job. This ensures a failure in one tool never blocks the others, and
contributors see full feedback from every tool on every CI run.

- Upgrades actions/checkout to v4 and actions/setup-python to v5
- All linter jobs are enforced (no continue-on-error)
- Test artifacts (junit.xml, coverage.xml) always uploaded
- Bandit JSON report always uploaded as artifact
- Adds docs/CIWorkflow.md with maintainer documentation
- Updates CONTRIBUTING.md with CI workflow table and link

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 03:07:53 +00:00
copilot-swe-agent[bot] 817dc31e12 feat(browser): implement browser extension for sending files to DocuElevate
- Add complete browser extension with popup UI and background workers
- Support Chrome, Firefox, Edge, and other Chromium-based browsers
- Include context menu integration for quick file sending
- Add comprehensive documentation for users and administrators
- Update main README and API docs to include browser extension
- Implement secure configuration storage in browser extension storage
- Add SSRF-protected URL upload endpoint integration

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 03:04:36 +00:00