feat(duplicates): add duplicate document detection and management

- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 10:21:57 +00:00
parent 3361042186
commit ebea83a750
10 changed files with 1293 additions and 3 deletions
+11 -1
View File
@@ -336,4 +336,14 @@ MEILISEARCH_URL=http://meilisearch:7700
# Optional master/API key for secured Meilisearch instances
# MEILISEARCH_API_KEY=your_master_key_here
MEILISEARCH_INDEX_NAME=documents
ENABLE_SEARCH=True
ENABLE_SEARCH=True
# **Duplicate Detection**
# Exact duplicate detection (SHA-256) is always on during document processing.
# The settings below control near-duplicate detection (same scanned content,
# different hash) and the visibility of deduplication steps.
ENABLE_DEDUPLICATION=True
SHOW_DEDUPLICATION_STEP=True
# Minimum cosine similarity score (01) for two documents to be flagged as
# near-duplicates. 0.85 means 85 % semantic overlap. Lower = more matches.
NEAR_DUPLICATE_THRESHOLD=0.85