feat(similarity): add embedding pipeline, debug endpoints, backfill task, and scalable similarity search
- Add embedding_model config setting (replaces hardcoded text-embedding-3-small) - Add compute_document_embedding Celery task for ingestion-time embedding - Chain embedding task into finalize_document_storage pipeline - Add backfill_missing_embeddings periodic task (every 5 min) for legacy files - Add debug API endpoints: embedding-status, compute-embedding, diagnostic/embeddings, diagnostic/compute-all-embeddings - Refactor find_similar_documents to only use pre-computed embeddings (no lazy API calls) - Use yield_per(500) and column-only queries for 100K+ scale - Add embedding status indicator and recompute button in file detail UI Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -323,6 +323,13 @@ class Settings(BaseSettings):
|
||||
"them near-duplicates. Higher values require closer content matches. Default: 0.85."
|
||||
),
|
||||
)
|
||||
embedding_model: str = Field(
|
||||
default="text-embedding-3-small",
|
||||
description=(
|
||||
"Model name used for generating text embeddings via the OpenAI-compatible API. "
|
||||
"Embeddings drive the document similarity feature. Default: text-embedding-3-small."
|
||||
),
|
||||
)
|
||||
|
||||
# Text quality check - AI-based assessment of embedded PDF text
|
||||
enable_text_quality_check: bool = Field(
|
||||
|
||||
Reference in New Issue
Block a user