gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(similarity): truncate text to fit embedding model context window, fix step tracking

Browse Source 
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 13:27:34 +00:00
parent c724b8d83a
commit 8e955f3c81
7 changed files with 134 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ConfigurationGuide.md
+7 -2
View File
@@ -868,10 +868,15 @@ After OCR processes a document, its extracted text is converted to a vector embe
| Variable | Description | Default |
|---|---|---|
| `NEAR_DUPLICATE_THRESHOLD` | Minimum cosine similarity (0–1) for two documents to be considered near-duplicates. `0.85` means ≥ 85 % semantic overlap. | `0.85` |
| `EMBEDDING_MODEL` | Model name for generating text embeddings via the OpenAI-compatible API. Must be supported by the endpoint configured with `OPENAI_BASE_URL`. | `text-embedding-3-small` |
| `EMBEDDING_MAX_TOKENS` | Maximum tokens to send to the embedding model. Text is truncated to approximately this many tokens before calling the API. Set below the model's context window (e.g. 8 000 for an 8 192-token model). | `8000` |
Near-duplicate detection:
- Is performed **on demand** via `GET /api/files/{id}/duplicates` — not automatically during ingest (OCR text is required).
- Is exposed in the **Duplicates** management page (`/duplicates` → "Near-Duplicate Finder" tab).
- Embeddings are computed **automatically during document ingestion** as a processing step ("Compute Embedding").
- A periodic **backfill task** (every 5 minutes) picks up any files that were processed before the embedding pipeline was enabled.
- The **Similarity dashboard** (`/similarity`) shows all pairs of documents above the threshold, ranked by score.
- The **Duplicates** management page (`/duplicates` → "Near-Duplicate Finder" tab) allows per-file lookup.
- Debug endpoints are available to inspect embedding status and trigger recomputation (see API docs).
- Documents without OCR text cannot be compared and are excluded from results.
A score of **≥ 0.90** reliably identifies the same document scanned twice. A score of **0.70–0.90** suggests partial content overlap. Adjust `NEAR_DUPLICATE_THRESHOLD` to tune sensitivity.