feat(similarity): add embedding pipeline, debug endpoints, backfill task, and scalable similarity search

- Add embedding_model config setting (replaces hardcoded text-embedding-3-small)
- Add compute_document_embedding Celery task for ingestion-time embedding
- Chain embedding task into finalize_document_storage pipeline
- Add backfill_missing_embeddings periodic task (every 5 min) for legacy files
- Add debug API endpoints: embedding-status, compute-embedding, diagnostic/embeddings, diagnostic/compute-all-embeddings
- Refactor find_similar_documents to only use pre-computed embeddings (no lazy API calls)
- Use yield_per(500) and column-only queries for 100K+ scale
- Add embedding status indicator and recompute button in file detail UI

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 13:01:57 +00:00
parent b435957a9b
commit 8d7c8e7c4e
8 changed files with 964 additions and 33 deletions
+7
View File
@@ -323,6 +323,13 @@ class Settings(BaseSettings):
"them near-duplicates. Higher values require closer content matches. Default: 0.85."
),
)
embedding_model: str = Field(
default="text-embedding-3-small",
description=(
"Model name used for generating text embeddings via the OpenAI-compatible API. "
"Embeddings drive the document similarity feature. Default: text-embedding-3-small."
),
)
# Text quality check - AI-based assessment of embedded PDF text
enable_text_quality_check: bool = Field(