feat(duplicates): add duplicate document detection and management

- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-02 10:21:57 +00:00
parent 3361042186
commit ebea83a750
10 changed files with 1293 additions and 3 deletions
+6
View File
@@ -101,6 +101,9 @@
<a href="/admin/files" role="menuitem" class="flex items-center px-4 py-2 text-sm text-gray-700 hover:bg-gray-100">
<i class="fas fa-folder-open w-4 mr-2 text-gray-500" aria-hidden="true"></i> File Manager
</a>
<a href="/duplicates" role="menuitem" class="flex items-center px-4 py-2 text-sm text-gray-700 hover:bg-gray-100">
<i class="fas fa-clone w-4 mr-2 text-orange-500" aria-hidden="true"></i> Duplicates
</a>
<a href="/admin/queue" role="menuitem" class="flex items-center px-4 py-2 text-sm text-gray-700 hover:bg-gray-100">
<i class="fas fa-stream w-4 mr-2 text-blue-500" aria-hidden="true"></i> Queue Monitor
</a>
@@ -178,6 +181,9 @@
<a href="/admin/files" class="block px-3 py-3 rounded-md text-base font-medium text-gray-700 hover:text-gray-900 hover:bg-gray-50">
<i class="fas fa-folder-open mr-2 text-gray-400" aria-hidden="true"></i> File Manager
</a>
<a href="/duplicates" class="block px-3 py-3 rounded-md text-base font-medium text-gray-700 hover:text-gray-900 hover:bg-gray-50">
<i class="fas fa-clone mr-2 text-orange-400" aria-hidden="true"></i> Duplicates
</a>
<a href="/admin/queue" class="block px-3 py-3 rounded-md text-base font-medium text-gray-700 hover:text-gray-900 hover:bg-gray-50">
<i class="fas fa-stream mr-2 text-blue-400" aria-hidden="true"></i> Queue Monitor
</a>