feat(duplicates): add duplicate document detection and management

- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 10:21:57 +00:00
parent 3361042186
commit ebea83a750
10 changed files with 1293 additions and 3 deletions
+7
View File
@@ -316,6 +316,13 @@ class Settings(BaseSettings):
" If False, the check is still performed but not displayed. Default: True."
),
)
near_duplicate_threshold: float = Field(
default=0.85,
description=(
"Minimum cosine similarity score (01) between two documents' text embeddings to consider "
"them near-duplicates. Higher values require closer content matches. Default: 0.85."
),
)
# Text quality check - AI-based assessment of embedded PDF text
enable_text_quality_check: bool = Field(