Commit Graph

78 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] c428a7ec00 feat(ui): add dark mode support with system preference detection and localStorage persistence
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:59:49 +00:00
copilot-swe-agent[bot] ff96340154 fix(config): default meilisearch_url to http://meilisearch:7700 for Docker/K8s service discovery
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 13:27:09 +00:00
copilot-swe-agent[bot] 66a45703cf fix(ui): resolve merge conflict markers in settings.html and add CI guard
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-24 17:13:44 +00:00
copilot-swe-agent[bot] f9d849f7eb feat(api): remove redundant /env and /api/diagnostic/settings endpoints
- Remove `/api/diagnostic/settings` endpoint (superseded by `/api/settings/`)
- Remove `/env` view route (superseded by `/settings` admin page)
- Delete `env_debug.html` template
- Remove `/env` nav links from base.html (desktop + mobile)
- Update status_dashboard.html to link to /settings instead of /env
- Remove corresponding tests for deleted endpoints
- Update RateLimitingStrategy.md docs

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 23:23:28 +00:00
Christian Krakau-Louis 4cc9233524 Merge branch 'main' into copilot/add-self-hosted-ocr-support 2026-02-23 23:51:43 +01:00
copilot-swe-agent[bot] a94b52ee14 feat(ai): handle temperature incompatibility for gpt-5 and o-series models, add model picker UI
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 22:12:08 +00:00
copilot-swe-agent[bot] 219e03ed3d feat(ui): add copy button to text modals in file detail view
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 21:34:57 +00:00
copilot-swe-agent[bot] ce73f23a89 feat(api): add POST /api/ai/test-extraction endpoint and Test Extraction UI button
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 21:09:27 +00:00
copilot-swe-agent[bot] 1f843fe460 feat(ocr): add self-hosted OCR engine support (Tesseract, EasyOCR) with multi-provider cross-checking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 20:59:43 +00:00
copilot-swe-agent[bot] 044da0ff47 feat: update all OpenAI/ChatGPT refs + settings dropdown for fixed-value fields
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 20:06:57 +00:00
copilot-swe-agent[bot] d08e175352 feat(ui): implement responsive mobile interface
- Add `relative` to nav for correct mobile dropdown positioning
- Increase hamburger button and mobile nav links to ≥44px touch targets
- Add mobile card view on files page (table hidden on small screens)
- Make bulk actions bar and pagination responsive/wrapping
- Add file type `accept` attribute and camera capture button on upload page
- Increase all interactive button/input sizes to ≥44px
- Add responsive CSS for pagination wrap and filter stacking at 480px"

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 19:09:49 +00:00
copilot-swe-agent[bot] 7d9d1ac698 fix(ui): fix Remove from DB button layout on settings page
- Change button container from flex-col to flex-row to eliminate overlap
- Move Remove from DB button to the left of the Save button
- Change button color from orange to red (bg-red-600)

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 18:30:48 +00:00
copilot-swe-agent[bot] cda0c87eb1 fix(ui): remove @tailwindcss/browser@4 to fix button visibility on settings page
The Remove from DB button (and other buttons) was invisible due to CSS
conflicts from loading both @tailwindcss/browser@4 and tailwindcss@2.2.19
simultaneously. The v4 browser script injected a Preflight reset that set
button { background-color: transparent; color: inherit; } and caused state
variants (hover:, focus:, disabled:) to generate empty CSS rules.

Removing the v4 CDN script leaves Tailwind v2 as the single source of
truth for utility classes, restoring correct button styling site-wide.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 12:59:49 +00:00
copilot-swe-agent[bot] 1cf6adbc7f feat(ui): always show Remove from DB button for DB-sourced settings with confirmation dialog
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 12:26:45 +00:00
copilot-swe-agent[bot] 862ba99802 fix(settings): rollback restores old_value instead of new_value and supports deletion fallback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 12:15:44 +00:00
copilot-swe-agent[bot] 96fd0b8dd5 feat(settings): patch API in-memory settings on save and add Back to ENV button
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 11:42:37 +00:00
copilot-swe-agent[bot] ecc8100e24 feat(settings): persist storage provider settings to DB, add export endpoint, enrich setup wizard
- dropbox/google_drive/onedrive save-settings: switch to DB as primary,
  .env write as best-effort (no longer fails when .env is absent)
- onedrive/google_drive update-settings: persist changes to DB alongside
  in-memory update; call notify_settings_updated()
- onedrive test_onedrive_token: persist rotated refresh token to DB
- settings_service: add get_settings_for_export() (db / effective modes)
- settings API: add GET /api/settings/export-env (admin-only, downloads .env)
- wizard: enrich settings with current values (DB > ENV > default) and
  value_source badges; pass setup_skipped to template; call
  notify_settings_updated() on save; add /setup/undo-skip route
- setup_wizard.html: pre-populate inputs with current_value; show
  DB/ENV/DEFAULT source badges; skip/undo-skip messaging
- settings.html: replace single Audit Log button with Setup Wizard link,
  Export .env dropdown, and Audit Log button group

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-02-23 02:41:04 +00:00
copilot-swe-agent[bot] 90e5e0037c feat(settings): per-option save, live worker sync, audit log, and rollback
A) Per-option Save Button
- Add per-setting Save button in settings.html (visible only when value changed)
- Button calls POST /api/settings/{key} directly; existing bulk Save retained
- Add Audit Log link in settings page header

B) Immediate Worker Sync
- New app/utils/settings_sync.py with notify_settings_updated() (Redis version key)
  and register_settings_reload_signal() (Celery task_prerun handler)
- Register signal in celery_worker.py at startup
- All API write paths call notify_settings_updated() after successful saves

C) Audit Log
- Add SettingsAuditLog model (key, old_value, new_value, changed_by, changed_at, action)
- save_setting_to_db / delete_setting_from_db accept changed_by and write audit entries
- New get_audit_log() service function (masks sensitive values)
- New GET /api/settings/audit-log endpoint (admin-only)
- New GET /admin/settings/audit-log view + audit_log.html template
- Visible to all admins (per clarified requirement)

D) Config Rollback / History
- New get_setting_history() and rollback_setting() service functions
- New GET /api/settings/{key}/history endpoint
- New POST /api/settings/{key}/rollback/{history_id} endpoint
- Rollback buttons in audit_log.html with confirmation dialog
- Tests: 25 new tests covering audit log, rollback, worker sync helpers, and API endpoints

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-23 02:28:08 +00:00
copilot-swe-agent[bot] a9dcdb7a0d fix(ui): add X-CSRF-Token header to retry fetch calls in file_detail.html
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-22 21:37:22 +00:00
copilot-swe-agent[bot] 3fa24c0686 feat(ui): add Credential Audit page under Admin menu
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-22 19:39:01 +00:00
copilot-swe-agent[bot] c54cbcbdd6 feat(auth): implement CSRF token protection for state-changing operations
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-22 16:27:37 +00:00
copilot-swe-agent[bot] 54736ea35a feat(admin): enhance file manager with DB reconciliation view (filesystem/database/reconcile tabs)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-21 09:20:59 +00:00
copilot-swe-agent[bot] 16e7b6478e feat(admin): add admin-only file manager, admin menu, de-emphasize status, improve dashboard
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-21 09:15:27 +00:00
copilot-swe-agent[bot] 7524819c77 fix(security): replace innerHTML with DOM API to prevent XSS in common.js
- Replace innerHTML assignments with createElement/textContent to prevent XSS
- Properly escape user data (display name, picture URL) from OAuth responses
- Remove stale 'files copy.html' template file

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 11:45:29 +00:00
copilot-swe-agent[bot] b03ebab747 chore: apply ruff formatting and fix whitespace issues
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 09:12:11 +00:00
copilot-swe-agent[bot] 00abb33bbe docs: update all references from PyPDF2 to pypdf in documentation
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 02:54:48 +00:00
Christian Krakau-Louis 9a2c2d20b1 feat(duplication): implement duplicate file handling and detection across processing steps 2026-02-12 02:38:00 +01:00
Christian Krakau-Louis 83c65f3c40 feat: enhance processing status tracking and UI display for file details 2026-02-12 01:07:22 +01:00
Christian Krakau-Louis 02e1445e01 Enhance file detail page with original and processed file previews, GPT metadata display, and text extraction functionality
- Added endpoints for previewing original and processed PDF files.
- Implemented on-demand text extraction from original and processed PDFs.
- Updated file detail page to show original and processed file paths with existence status.
- Introduced GPT metadata display with a collapsible JSON view.
- Enhanced front-end with PDF.js for in-browser PDF rendering and improved user experience.
- Added integration tests for new features including metadata display and file previews.
2026-02-11 23:32:29 +01:00
copilot-swe-agent[bot] 47a1de953d refactor: address code review comments
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 13:39:35 +00:00
copilot-swe-agent[bot] 245e298991 feat: add URL-based file upload with SSRF protection
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 13:36:23 +00:00
copilot-swe-agent[bot] b18f5ee13c feat(ui): add verbose worker log detail to processing history on file detail page
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 09:06:30 +00:00
copilot-swe-agent[bot] 11c7d15a90 fix(tasks): skip duplicate check when reprocessing and enable retry from failed pipeline step
- Add file_id parameter to process_document to skip duplicate hash check on reprocess
- Pass file_id from reprocess_single_file and bulk_reprocess_files endpoints
- Extend retry-subtask endpoint to support pipeline steps (process_document,
  process_with_azure_document_intelligence, extract_metadata_with_gpt,
  embed_metadata_into_pdf) in addition to upload tasks
- Add retry button for failed main pipeline steps in file detail UI
- Add comprehensive tests for reprocessing and pipeline step retry

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-10 21:09:53 +00:00
copilot-swe-agent[bot] 334714423e fix(ui): address code review feedback on inline preview
- Remove invalid type attribute from iframe elements
- Add dedicated download endpoint with attachment disposition
- Update download links to use new endpoint instead of preview

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 16:16:31 +00:00
copilot-swe-agent[bot] ed0bac7345 feat(ui): add auto-refresh after upload and inline file preview
- Auto-refresh files table after successful uploads using custom event
- Add inline preview support for PDFs, images, and text files
- Set Content-Disposition header to inline for preview endpoint
- Add download button as secondary action in file details view

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 16:12:44 +00:00
copilot-swe-agent[bot] 0a461343e7 fix: Add service containers and fix test failures
- Add Redis and RabbitMQ services to CI workflow
- Fix Jinja2 template error by passing file=None in error cases
- Fix test expecting dict response format for list_files endpoint
- Fix NOT NULL constraint by providing valid local_filename
- Fix retry-subtask to validate subtask name before checking processed file
- Add mock for process_document in reprocess test

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 08:14:09 +00:00
Christian Krakau-Louis 9cc98f39d5 Merge branch 'main' into copilot/implement-database-backed-settings-page 2026-02-08 08:00:33 +01:00
copilot-swe-agent[bot] 1a01811882 Add encryption and setup wizard features
ENCRYPTION:
- Add cryptography library for secure storage
- Implement Fernet encryption for sensitive settings
- Key derived from SESSION_SECRET
- Auto-encrypt/decrypt transparent to app
- "enc:" prefix identifies encrypted values
- Graceful fallback if crypto unavailable

SETUP WIZARD:
- Detect fresh installs needing configuration
- 3-step wizard: Infrastructure, Security, AI Services
- "/" redirects to wizard if setup required
- Auto-generate session_secret option
- Skip option for advanced users
- Beautiful UI with progress indicators

UI IMPROVEMENTS:
- Enhanced sensitive field display
- Lock icon showing encryption status
- Improved show/hide toggle for passwords
- Better visual hierarchy

FILES:
- app/utils/encryption.py - Encryption utilities
- app/utils/setup_wizard.py - Wizard detection logic
- app/views/wizard.py - Wizard routes
- frontend/templates/setup_wizard.html - Wizard UI
- requirements.txt - Added cryptography
- IMPLEMENTATION_CHECKLIST.md - Status tracking

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:18:49 +00:00
copilot-swe-agent[bot] 121b0d811f Add settings source indicators and improve form UX
- Remove HTML 'required' attributes - all fields optional
- Add source detection (DB/ENV/DEFAULT) for each setting
- Display color-coded badges showing setting source
- Update template with precedence order explanation
- Pre-fill form with current values from DB/ENV/defaults
- Update documentation with source badge explanations
- Test and verify form prefilling works correctly

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:13:25 +00:00
copilot-swe-agent[bot] a29af88dd4 Update documentation and version for drag-and-drop feature
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:10:04 +00:00
copilot-swe-agent[bot] a7128cecd2 Add drag-and-drop upload functionality to Files view
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 06:05:35 +00:00
copilot-swe-agent[bot] 4b4c208035 Security fixes: use tojson filter, fix Tuple import, remove raw value exposure
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:38:58 +00:00
copilot-swe-agent[bot] e1bb9de915 Add settings management infrastructure: models, API, views, and database loading
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:31:06 +00:00
copilot-swe-agent[bot] 910fe86603 Add branching visualization, per-subtask retry, and step summary to file detail view
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 22:08:09 +00:00
copilot-swe-agent[bot] b4e066bf2d Security fix: Use JSON serialization for template variable in JavaScript
- Fixed XSS vulnerability in file detail template
- Use tojson filter to safely encode file.id into JavaScript
- CodeQL scan confirms no security issues

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 20:58:23 +00:00
copilot-swe-agent[bot] 76ab698105 Add backend endpoints and enhanced file detail view
- Added /api/files/{file_id}/reprocess endpoint for single file reprocessing
- Added /api/files/{file_id}/preview endpoint for viewing original/processed files
- Enhanced file detail view with process flow computation
- Updated frontend template with retry button, process flow visualization, and PDF previews
- Added JavaScript for async retry functionality

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 20:52:08 +00:00
copilot-swe-agent[bot] 74724973d8 Fix API routes to return JSON instead of HTML on errors
- Modified exception handlers in app/main.py to check if request path starts with /api/
- HTTPException handler returns JSON for API routes, HTML for frontend routes
- General exception handler (500) also checks and returns appropriate format
- Enhanced frontend deleteFile() to handle non-JSON responses gracefully
- Added content-type checking before parsing JSON
- Added comprehensive tests for API error handling

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 19:18:42 +00:00
copilot-swe-agent[bot] 77678f6368 Fix single file deletion frontend to parse JSON response properly
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 17:53:12 +00:00
copilot-swe-agent[bot] 34ba8a4548 Improve error messages for bulk operations
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 15:55:44 +00:00
copilot-swe-agent[bot] 08775459aa Fix /files view issues and add bulk operations
- Fixed status filter in /files view endpoint
- Added bulk delete and reprocess API endpoints
- Added bulk selection UI with checkboxes
- Added bulk actions bar with reprocess and delete buttons
- Updated JavaScript to handle bulk operations

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-07 15:52:38 +00:00