gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(similarity): truncate text to fit embedding model context window, fix step tracking

Browse Source 
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 13:27:34 +00:00
parent c724b8d83a
commit 8e955f3c81
7 changed files with 134 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.demo
+7 -1
View File
@@ -346,4 +346,10 @@ ENABLE_DEDUPLICATION=True
SHOW_DEDUPLICATION_STEP=True
# Minimum cosine similarity score (0–1) for two documents to be flagged as
# near-duplicates. 0.85 means 85 % semantic overlap. Lower = more matches.
NEAR_DUPLICATE_THRESHOLD=0.85
NEAR_DUPLICATE_THRESHOLD=0.85
# Model used to generate text embeddings for document similarity.
# Must be supported by your OpenAI-compatible API endpoint.
EMBEDDING_MODEL=text-embedding-3-small
# Maximum tokens to send to the embedding model. Set below the model's
# context window (e.g. 8000 for an 8192-token model).
EMBEDDING_MAX_TOKENS=8000