Replace the fragile global DOM query approach in `updateOverallStatus`
with per-batch closure-based counters inside `processFiles`.
- Add `total`, `done`, `updateStatus()`, and `markDone()` as closure
variables/functions within each `processFiles` invocation
- Change `_uploadSingleFile` to accept an `onTerminal` callback instead
of `statusMessage`, called when a file reaches a terminal state
- Pass `markDone` as the `onTerminal` callback from `scheduleNext`
- Remove the now-unused global `updateOverallStatus` function
The previous implementation queried `document.querySelectorAll('.file-status')`
globally and relied on text `startsWith` checks to count completed files.
This was fragile and could produce a stale done=0 count in practice.
The new approach uses deterministic closure counters, so the displayed
"Uploading files (X/N)" count correctly increments as files complete.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When init_db() called Base.metadata.create_all() before Alembic migrations,
the ORM model created the webhook_configs table. Alembic migration 009 then
failed with OperationalError: table webhook_configs already exists.
Fix: check for alembic_version table before calling create_all(). Tracked
databases skip create_all and let Alembic handle all schema changes instead.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Refactor CI workflow to simplify configuration and improve readability. Consolidate steps, update job dependencies, and enhance linting and testing stages.
- Move mypy to Stage 1 (runs in parallel with lint & html-lint, no needs)
- Decouple dependency-scan from test-quick; tests now start as soon as
static analysis passes (needs: [lint, html-lint, mypy])
- dependency-scan runs as a parallel background track and still gates
build/deploy to prevent shipping with known CVEs
- Integration tests remain sequentially after quick tests pass (Stage 4)
- Build & deploy remain gated on ALL stages including dependency-scan (Stage 5)
- Reorder job definitions to match logical stage flow for readability
- Update section comments to reflect the new 5-stage architecture
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]