Commit Graph

126 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] a2592629db feat(settings): add all missing config settings to settings page with enhanced UX
Add 64 previously missing settings from config.py to SETTING_METADATA,
making them all configurable via the settings page UI.

New categories: PDF/A Archival, Security
Enhanced UX:
- Slider inputs for threshold values (text_quality_threshold, near_duplicate_threshold)
- Dropdown selects for s3_storage_class, s3_acl, pdfa_format, security_header_x_frame_options_value
- All 8 PDF/A settings now configurable via the UI
- Security headers, audit logging, rate limiting, CORS settings added
- IMAP 1 & 2, S3, Meilisearch, Deduplication, Embedding settings added
- Task retry, step timeout, file size limits settings added

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 16:37:28 +00:00
copilot-swe-agent[bot] 55543be3b0 fix(similarity): address code review - column-only queries, configurable batch size, WCAG touch targets
- Use column-only query in embeddings overview to reduce memory for 100K+ files
- Add embedding_backfill_batch_size config setting (default 50)
- Fix WCAG touch target on backfill button (min-height/min-width 44px)
- Add inline comment explaining 3 chars/token truncation estimate
- Import settings in compute_embedding task for configurable batch size

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:39:10 +00:00
copilot-swe-agent[bot] c724b8d83a feat(similarity): add similarity pairs dashboard, step tracking, and fix tests for pre-computed embeddings
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:12:59 +00:00
copilot-swe-agent[bot] 8d7c8e7c4e feat(similarity): add embedding pipeline, debug endpoints, backfill task, and scalable similarity search
- Add embedding_model config setting (replaces hardcoded text-embedding-3-small)
- Add compute_document_embedding Celery task for ingestion-time embedding
- Chain embedding task into finalize_document_storage pipeline
- Add backfill_missing_embeddings periodic task (every 5 min) for legacy files
- Add debug API endpoints: embedding-status, compute-embedding, diagnostic/embeddings, diagnostic/compute-all-embeddings
- Refactor find_similar_documents to only use pre-computed embeddings (no lazy API calls)
- Use yield_per(500) and column-only queries for 100K+ scale
- Add embedding status indicator and recompute button in file detail UI

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 13:01:57 +00:00
copilot-swe-agent[bot] ebea83a750 feat(duplicates): add duplicate document detection and management
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 10:21:57 +00:00
copilot-swe-agent[bot] b7c78177e9 docs(similarity): add API documentation and fix template accessibility
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 20:46:13 +00:00
copilot-swe-agent[bot] 9748103782 feat(similarity): add document similarity detection with embeddings and cosine similarity
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 20:38:52 +00:00
copilot-swe-agent[bot] d18c10996a feat(release): add named release anchors with codenames and roadmap integration
- Add release_names.json mapping version ranges to codenames
- Add release_name property to Settings in app/config.py
- Update build metadata script to include codename in RUNTIME_INFO
- Display release codename in status dashboard and page footer
- Inject release_name globally via template response wrapper
- Update ROADMAP.md with codenames for all milestone releases
- Add docs/ReleaseNaming.md with naming guide and best practices
- Add comprehensive tests for release name resolution

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 18:32:46 +00:00
copilot-swe-agent[bot] af34ce88df feat(api): implement proper pagination for file list API
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 17:31:56 +00:00
Christian Krakau-Louis 96b14ec74f Merge pull request #440 from christianlouis/copilot/update-search-functionality
feat(search): add content-finding filters, saved searches, and text quality to Search view
2026-03-01 15:19:57 +01:00
copilot-swe-agent[bot] 68857c1785 style(search): address code review - use const/let, fix date timezone, deduplicate filter keys
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:59:23 +00:00
Christian Krakau-Louis b65d937a36 Merge pull request #439 from christianlouis/copilot/add-bulk-file-operations
feat: bulk operations (delete, reprocess, Cloud OCR, ZIP download) + persisted OCR quality score filter
2026-03-01 14:58:01 +01:00
copilot-swe-agent[bot] 83c3405c98 feat: persist ocr_quality_score and use it for numeric filtering
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:46:43 +00:00
copilot-swe-agent[bot] 5bf0a4c0b9 feat(search): add content-finding filters, saved searches, and text quality to Search view
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:46:15 +00:00
copilot-swe-agent[bot] 705b970522 feat: add bulk download, cloud OCR, and basic OCR quality filter
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:38:42 +00:00
copilot-swe-agent[bot] 81c08ed9d0 fix: append new drops to upload queue instead of overwriting it
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:26:49 +00:00
copilot-swe-agent[bot] d5884f6d2c fix(api): address code review feedback - logging, wildcard escaping, UX
- Add exception logging in saved search error handlers
- Escape SQL LIKE wildcards (%, _) in tags filter to prevent unintended matching
- Improve UI alert message for empty filter save attempt

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:32:34 +00:00
copilot-swe-agent[bot] e5a4c6c64a feat(api): add advanced filtering and saved searches
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:22:07 +00:00
copilot-swe-agent[bot] d095e30467 fix: address code review feedback on tests and placeholder messages
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 07:51:07 +00:00
copilot-swe-agent[bot] c814af1145 fix(ui): preview modal falls back to original version when processed unavailable
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 07:48:44 +00:00
copilot-swe-agent[bot] 372e8cdff1 feat(ui): add pdf.js viewer, image zoom/pan, text preview, and preview modal
- Replace iframe in file_view.html with pdf.js for PDF files
- Add image viewer with zoom/pan controls to file_view.html
- Add text file viewer with line numbers to file_view.html
- Add preview side-panel modal to files.html (file list)
- Fix file_detail.html bottom preview section variable names
- Add aria-labels and WCAG compliance to all new elements

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 07:32:30 +00:00
copilot-swe-agent[bot] 0969436abd feat(ui): add ARIA attributes for accessibility improvements
Add aria-hidden="true" to decorative Font Awesome icons across templates,
aria-label to icon-only buttons, aria-live to dynamic content regions,
aria-labelledby to modals, aria-expanded to toggle buttons, scope="col"
to table headers, and aria-label to tables. Also improve alt text on
500.html error image and add aria-label to password toggle buttons.

Templates updated: settings.html, credentials.html, status_dashboard.html,
file_view.html, file_detail.html, index.html, queue_dashboard.html,
audit_log.html, 500.html

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-02-28 16:04:33 +00:00
copilot-swe-agent[bot] bf6bbe3512 feat(ui): fix accessibility in upload, search, files, and login templates
- upload.html: Add role/tabindex/aria-label to drop zone, aria-live to status messages, aria-label to file inputs
- search.html: Add role="search", sr-only label, aria-live for results, input type="search"
- files.html: Add table aria-label, scope="col" to headers, aria-sort on sortable columns, aria-labels to action buttons, dialog roles to modals, aria-live to status areas, pagination nav with aria-labels
- login.html: Wrap form in main landmark, aria-hidden on decorative icons

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-28 15:59:12 +00:00
copilot-swe-agent[bot] 364ff9375a feat(ui): add accessibility foundations to base.html and styles.css
- Add skip-to-content link for keyboard navigation (WCAG 2.4.1)
- Add ARIA landmarks: nav aria-label, footer role=contentinfo
- Add aria-current="page" on active nav links
- Add aria-label to admin dropdown button and mobile menu toggle
- Add role="menu" and role="menuitem" to admin dropdown
- Add aria-hidden="true" to all decorative Font Awesome icons
- Add footer nav element with aria-label for footer links
- Add focus-visible outline styles for keyboard navigation (WCAG 2.4.7)
- Add sr-only utility class
- Add dark mode support for skip-link and focus indicators

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-28 15:55:27 +00:00
copilot-swe-agent[bot] dc9aec2bab fix(upload): fix multi-file drag-and-drop only uploading one file
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-27 16:19:06 +00:00
copilot-swe-agent[bot] 4dd018f210 refactor(queue): address code review feedback — extract constants and sync refresh interval
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-27 00:47:58 +00:00
copilot-swe-agent[bot] c7c5718f78 feat(queue): add queue monitoring dashboard and pending banner on files page
- Add /api/queue/stats endpoint with Redis queue lengths, Celery worker
  inspection, and DB processing summaries
- Add /api/queue/pending-count lightweight endpoint for the files page banner
- Add /admin/queue admin-only view with auto-refreshing queue dashboard
- Add queue pending banner on /files page showing queued/processing count
- Add Queue Monitor link to admin dropdown in navigation (desktop + mobile)
- Add comprehensive tests for all new endpoints and views

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-27 00:42:55 +00:00
Christian Krakau-Louis 585b30a93e Merge pull request #426 from christianlouis/copilot/support-directory-upload
feat(upload): directory drag-and-drop, adaptive 429 throttling, full Gotenberg file type support
2026-02-27 00:56:17 +01:00
copilot-swe-agent[bot] a05690bd5f feat(upload): adaptive 429 backoff, full Gotenberg file types, directory traversal
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 23:24:26 +00:00
copilot-swe-agent[bot] 35d2ed05e9 feat(upload): add directory traversal, queue throttling, and upload config settings
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 23:11:44 +00:00
copilot-swe-agent[bot] c540dd4f52 feat(views): add document-centric view at /files/{id}
- Revert search.html and files.html links back to /files/{id}
- Add GET /files/{file_id} route (file_view_page) with workdir path-
  containment guards (os.path.commonpath) to prevent traversal
- Create file_view.html: document-centric page showing AI metadata,
  inline PDF preview, extracted OCR text, download actions, file info,
  status pill, and link to /files/{id}/detail for process pipeline view
- Existing /files/{id}/detail route is unchanged

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 23:02:25 +00:00
copilot-swe-agent[bot] da865b5f7a fix(search): correct file detail page URL in search results
Both the /search page and the inline search in /files were
generating links to /files/{id} which returns 404. The correct
route is /files/{id}/detail.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 22:46:41 +00:00
Christian Krakau-Louis 6ecf07abf9 Merge pull request #422 from christianlouis/copilot/fix-search-field-bug
fix(ui): fix search input clearing on keystroke; add dedicated /search page
2026-02-26 11:45:44 +01:00
copilot-swe-agent[bot] 483a5b71a1 fix(ui): sanitize Meilisearch HTML output, use event delegation, improve error messages
Address code review feedback:
- Sanitize highlighted HTML from Meilisearch to prevent XSS (only allow
  <mark> tags, escape everything else)
- Replace inline onclick handlers with event delegation for pagination
- Improve error message to be more user-friendly with technical detail
  in smaller text

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:34:31 +00:00
copilot-swe-agent[bot] 09a6337c6f fix(ui): prevent search input from clearing on each keystroke and add dedicated /search page
The debounceSearch() function in files.html called clearFullTextSearch()
when the query was shorter than 2 characters. Since clearFullTextSearch()
sets input.value = '', every single keystroke was immediately erased —
users could paste text but not type.

Fix: debounceSearch now only hides the results panel for short queries
without touching the input value.

Also adds a dedicated /search page with Google-style results showing
content previews (document title, filename, type badges, tag badges,
sender, and OCR text snippets with highlighted matches).

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:18:28 +00:00
copilot-swe-agent[bot] d83dfb12c6 fix: regenerate logo PNGs from SVG to fix DocuNova → DocuElevate branding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-26 09:05:38 +00:00
copilot-swe-agent[bot] 0821e2f989 feat(privacy): implement GDPR and global privacy compliance
- Add dismissable cookie notice banner to base template (essential
  cookies only, ePrivacy Directive compliant, localStorage persistence)
- Expand Privacy Notice to cover all target markets: EU/GDPR,
  UK GDPR, Switzerland nFADP, Ukraine, US CCPA/CPRA, Canada PIPEDA/
  Law 25, Brazil LGPD/Latin America, and Asia-Pacific & Japan (APPI,
  Australia Privacy Act, South Korea PIPA, Singapore PDPA, India DPDP)
- Add International Data Transfers section (SCCs, IDTAs, adequacy
  decisions) and Data Minimization & Purpose Limitation section
- Update Cookie Policy with precise cookie table, ePrivacy exemption
  rationale, and localStorage notice dismissal documentation
- Create docs/PrivacyCompliance.md: full multi-market compliance guide
  covering cookie strategy, data transfer mechanisms, data subject
  rights handling matrix with response timelines, and market-specific
  notes for all supported regions
- Add docs/PrivacyCompliance.md to mkdocs.yml Compliance nav section
- Add 10 new targeted tests to test_views_general.py validating all
  key compliance content areas

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 15:15:24 +00:00
copilot-swe-agent[bot] c428a7ec00 feat(ui): add dark mode support with system preference detection and localStorage persistence
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:59:49 +00:00
copilot-swe-agent[bot] ff96340154 fix(config): default meilisearch_url to http://meilisearch:7700 for Docker/K8s service discovery
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:27:09 +00:00
copilot-swe-agent[bot] 66a45703cf fix(ui): resolve merge conflict markers in settings.html and add CI guard
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-24 17:13:44 +00:00
copilot-swe-agent[bot] f9d849f7eb feat(api): remove redundant /env and /api/diagnostic/settings endpoints
- Remove `/api/diagnostic/settings` endpoint (superseded by `/api/settings/`)
- Remove `/env` view route (superseded by `/settings` admin page)
- Delete `env_debug.html` template
- Remove `/env` nav links from base.html (desktop + mobile)
- Update status_dashboard.html to link to /settings instead of /env
- Remove corresponding tests for deleted endpoints
- Update RateLimitingStrategy.md docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 23:23:28 +00:00
Christian Krakau-Louis 4cc9233524 Merge branch 'main' into copilot/add-self-hosted-ocr-support 2026-02-23 23:51:43 +01:00
copilot-swe-agent[bot] a94b52ee14 feat(ai): handle temperature incompatibility for gpt-5 and o-series models, add model picker UI
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 22:12:08 +00:00
copilot-swe-agent[bot] 219e03ed3d feat(ui): add copy button to text modals in file detail view
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 21:34:57 +00:00
copilot-swe-agent[bot] ce73f23a89 feat(api): add POST /api/ai/test-extraction endpoint and Test Extraction UI button
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 21:09:27 +00:00
copilot-swe-agent[bot] 1f843fe460 feat(ocr): add self-hosted OCR engine support (Tesseract, EasyOCR) with multi-provider cross-checking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 20:59:43 +00:00
copilot-swe-agent[bot] 044da0ff47 feat: update all OpenAI/ChatGPT refs + settings dropdown for fixed-value fields
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 20:06:57 +00:00
copilot-swe-agent[bot] d08e175352 feat(ui): implement responsive mobile interface
- Add `relative` to nav for correct mobile dropdown positioning
- Increase hamburger button and mobile nav links to ≥44px touch targets
- Add mobile card view on files page (table hidden on small screens)
- Make bulk actions bar and pagination responsive/wrapping
- Add file type `accept` attribute and camera capture button on upload page
- Increase all interactive button/input sizes to ≥44px
- Add responsive CSS for pagination wrap and filter stacking at 480px"

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 19:09:49 +00:00
copilot-swe-agent[bot] 7d9d1ac698 fix(ui): fix Remove from DB button layout on settings page
- Change button container from flex-col to flex-row to eliminate overlap
- Move Remove from DB button to the left of the Save button
- Change button color from orange to red (bg-red-600)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 18:30:48 +00:00
copilot-swe-agent[bot] cda0c87eb1 fix(ui): remove @tailwindcss/browser@4 to fix button visibility on settings page
The Remove from DB button (and other buttons) was invisible due to CSS
conflicts from loading both @tailwindcss/browser@4 and tailwindcss@2.2.19
simultaneously. The v4 browser script injected a Preflight reset that set
button { background-color: transparent; color: inherit; } and caused state
variants (hover:, focus:, disabled:) to generate empty CSS rules.

Removing the v4 CDN script leaves Tailwind v2 as the single source of
truth for utility classes, restoring correct button styling site-wide.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 12:59:49 +00:00