Add imports for convert_to_pdfa, finalize_document_storage,
process_with_ocr, upload_with_rclone, and webhook_tasks which
were missing from celery_worker.py, causing "unregistered task"
errors at runtime.
Add dynamic test that discovers all task modules in app/tasks/
and verifies each is imported in celery_worker.py.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When init_db() called Base.metadata.create_all() before Alembic migrations,
the ORM model created the webhook_configs table. Alembic migration 009 then
failed with OperationalError: table webhook_configs already exists.
Fix: check for alembic_version table before calling create_all(). Tracked
databases skip create_all and let Alembic handle all schema changes instead.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add health check endpoint at GET /api/diagnostic/health
- Auth-protected via @require_login (no-op when AUTH_ENABLED=False)
- Checks database (SELECT 1) and Redis (ping) with 2s timeouts
- Returns healthy/degraded/unhealthy with per-check detail
- Returns HTTP 503 when database is down, 200 otherwise
- 7 new unit tests covering all status scenarios
- Update docs/API.md with Grafana/monitoring integration notes
- Fixes test_cors_headers_absent_when_disabled CI timeout
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add release_names.json mapping version ranges to codenames
- Add release_name property to Settings in app/config.py
- Update build metadata script to include codename in RUNTIME_INFO
- Display release codename in status dashboard and page footer
- Inject release_name globally via template response wrapper
- Update ROADMAP.md with codenames for all milestone releases
- Add docs/ReleaseNaming.md with naming guide and best practices
- Add comprehensive tests for release name resolution
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
redis.Redis.scan() returns a tuple at runtime but mypy infers
Awaitable[Any] from the generic ResponseT return type, causing
a "not iterable" error on tuple unpacking.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
_ensure_indexes() now verifies the target column exists in the table
before executing CREATE INDEX IF NOT EXISTS. This prevents failures
when migrating legacy database schemas that don't yet have all columns
(e.g. files table without created_at or mime_type).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The Alembic migration alone doesn't run automatically. Add the
saved_searches table creation to _run_schema_migrations() in
database.py so existing databases are upgraded at startup.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default `version` param from 'original' to 'processed' so
GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The status calculation required ALL steps to be explicitly marked as
success/skipped before a file could be "completed". This failed for
dynamic pipelines where:
1. check_for_duplicates was logged before the file record existed (no
file_id), so its FileProcessingStep was never updated from "pending"
2. extract_text was not marked as "skipped" for non-PDF files that go
through PDF conversion first
Fix:
- Move check_for_duplicates success log to after initialize_file_steps()
with the correct file_id so the step actually gets updated
- Mark extract_text as "skipped" for non-PDF files
- Add terminal-step fallback: if send_to_all_destinations is "success",
the file is "completed" even if intermediate steps remain "pending"
(handles any other dynamic pipeline edge cases)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.
- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
completed filter
- process_document: call initialize_file_steps after creating a new
file record so all mandatory steps are pre-created as pending
Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.
Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Both get_file_preview and download_file were ignoring the DB-stored
original_file_path and processed_file_path fields, instead relying on
local_filename (a temp path that may be gone) and guessing patterns for
the processed file. This caused "Processed file not found" and
potentially "Original file not found" in the /files/{id} view even when
the files existed at their stored paths.
- version=original: check original_file_path first, fall back to local_filename
- version=processed: check processed_file_path first, fall back to
hash/filename guessing patterns
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]