e4fbd45636
Initial plan
2026-03-02 16:26:35 +00:00
24e0053def
chore(release): update build metadata files [skip ci]
2026-03-02 15:29:38 +00:00
9f1a86d554
0.75.0
...
Automatically generated by python-semantic-release
2026-03-02 15:29:36 +00:00
08099f94e4
Merge pull request #470 from christianlouis/copilot/add-pdfa-export-option
...
feat(pdfa): PDF/A archival conversion with FreeTSA timestamping and per-provider upload control
2026-03-02 16:29:18 +01:00
fea8f5c0de
style: apply ruff auto-fix
...
- Auto-formatted code with ruff format
- Applied ruff linting fixes with --fix
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2026-03-02 15:26:48 +00:00
8fee5c8d0b
Merge branch 'main' into copilot/add-pdfa-export-option
2026-03-02 16:26:31 +01:00
8e0bd50e88
chore(release): update build metadata files [skip ci]
2026-03-02 15:23:30 +00:00
a79f7ba50f
0.74.0
...
Automatically generated by python-semantic-release
2026-03-02 15:23:28 +00:00
d43ff23079
Merge pull request #469 from christianlouis/copilot/debug-file-similarity-analysis
...
feat(similarity): proactive embedding pipeline, context window fix, corpus-wide similarity dashboard
2026-03-02 16:23:09 +01:00
144a90fa73
fix(pdfa): address code review - validate pdfa_format, add S3 comment, add format test
...
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 14:01:26 +00:00
a03b3af933
feat(pdfa): add FreeTSA timestamping, per-provider folder overrides, individual upload toggles
...
- Add RFC 3161 timestamping via FreeTSA (PDFA_TIMESTAMP_ENABLED, PDFA_TIMESTAMP_URL)
- Replace PDFA_UPLOAD_TO_PROVIDERS with individual PDFA_UPLOAD_ORIGINAL and PDFA_UPLOAD_PROCESSED
- Add PDFA_UPLOAD_FOLDER setting for per-provider subfolder configuration
- Add GOOGLE_DRIVE_PDFA_FOLDER_ID for Google Drive-specific folder override
- Add folder_override parameter to all 8 folder-using upload tasks
- Add folder_overrides dict parameter to send_to_all_destinations
- Add _compute_pdfa_folder_overrides() and _timestamp_file() helpers
- Expand tests to 26 (timestamping, folder overrides, individual toggles)
- Update docs/ConfigurationGuide.md and .env.demo
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:55:34 +00:00
55543be3b0
fix(similarity): address code review - column-only queries, configurable batch size, WCAG touch targets
...
- Use column-only query in embeddings overview to reduce memory for 100K+ files
- Add embedding_backfill_batch_size config setting (default 50)
- Fix WCAG touch target on backfill button (min-height/min-width 44px)
- Add inline comment explaining 3 chars/token truncation estimate
- Import settings in compute_embedding task for configurable batch size
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:39:10 +00:00
eea99eb01d
feat(pdfa): add PDF/A archival conversion using ocrmypdf
...
- Add ENABLE_PDFA_CONVERSION, PDFA_FORMAT, PDFA_UPLOAD_TO_PROVIDERS config settings
- Add original_pdfa_path and processed_pdfa_path columns to FileRecord model
- Create Alembic migration 011_add_pdfa_paths
- Create app/tasks/convert_to_pdfa.py Celery task using ocrmypdf + Ghostscript
- Integrate PDF/A conversion into finalize_document_storage pipeline
- Add comprehensive unit tests (15 tests)
- Update .env.demo and docs/ConfigurationGuide.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:35:13 +00:00
8e955f3c81
fix(similarity): truncate text to fit embedding model context window, fix step tracking
...
- Add EMBEDDING_MAX_TOKENS config (default 8000) for safe text truncation
- Use conservative 3 chars/token estimate (was 4) to prevent ContextWindowExceededError
- Add compute_embedding to REAL_MAIN_STEPS in both get_file_overall_status and get_step_summary
- Fix test_near_duplicates_returned to use pre-computed embeddings
- Update .env.demo and docs with EMBEDDING_MAX_TOKENS setting
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:27:34 +00:00
34b9d529e7
Initial plan
2026-03-02 13:21:15 +00:00
c724b8d83a
feat(similarity): add similarity pairs dashboard, step tracking, and fix tests for pre-computed embeddings
...
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:12:59 +00:00
8d7c8e7c4e
feat(similarity): add embedding pipeline, debug endpoints, backfill task, and scalable similarity search
...
- Add embedding_model config setting (replaces hardcoded text-embedding-3-small)
- Add compute_document_embedding Celery task for ingestion-time embedding
- Chain embedding task into finalize_document_storage pipeline
- Add backfill_missing_embeddings periodic task (every 5 min) for legacy files
- Add debug API endpoints: embedding-status, compute-embedding, diagnostic/embeddings, diagnostic/compute-all-embeddings
- Refactor find_similar_documents to only use pre-computed embeddings (no lazy API calls)
- Use yield_per(500) and column-only queries for 100K+ scale
- Add embedding status indicator and recompute button in file detail UI
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 13:01:57 +00:00
b435957a9b
Initial plan
2026-03-02 12:46:39 +00:00
26d5f99425
chore(release): update build metadata files [skip ci]
2026-03-02 10:54:06 +00:00
d7d1175ba5
0.73.0
...
Automatically generated by python-semantic-release
2026-03-02 10:54:04 +00:00
054b3491e7
Merge pull request #466 from christianlouis/copilot/add-duplicate-document-detection
...
feat(duplicates): add duplicate document detection and management
2026-03-02 11:53:44 +01:00
e98c2ea63e
chore(release): update build metadata files [skip ci]
2026-03-02 10:35:06 +00:00
bb21486536
0.72.1
...
Automatically generated by python-semantic-release
2026-03-02 10:35:04 +00:00
7bb7e00927
Merge pull request #468 from christianlouis/copilot/fix-worker-api-startup-error
...
fix(database): skip create_all for Alembic-tracked databases to prevent OperationalError on webhook_configs
2026-03-02 11:34:48 +01:00
b87dd6083c
fix(database): skip create_all for Alembic-tracked databases to prevent OperationalError on webhook_configs
...
When init_db() called Base.metadata.create_all() before Alembic migrations,
the ORM model created the webhook_configs table. Alembic migration 009 then
failed with OperationalError: table webhook_configs already exists.
Fix: check for alembic_version table before calling create_all(). Tracked
databases skip create_all and let Alembic handle all schema changes instead.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 10:25:34 +00:00
ebea83a750
feat(duplicates): add duplicate document detection and management
...
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 10:21:57 +00:00
c0a3753e55
Initial plan
2026-03-02 10:14:21 +00:00
3361042186
Initial plan
2026-03-02 10:04:21 +00:00
716efcf73d
chore(release): update build metadata files [skip ci]
2026-03-02 10:03:51 +00:00
ffc795e5b1
0.72.0
...
Automatically generated by python-semantic-release
2026-03-02 10:03:48 +00:00
c938a7c219
Merge pull request #463 from christianlouis/copilot/add-document-similarity-detection
...
fix(migrations): resolve multiple Alembic heads causing CI test crash
2026-03-02 11:03:31 +01:00
744a433cc9
docs(changelog): update changelog [skip ci]
2026-03-02 10:03:30 +00:00
e8e91fd29a
Refactor CI workflow stages and steps
...
Updated CI workflow for improved clarity and efficiency.
2026-03-02 11:03:09 +01:00
fe9f84ac05
docs(changelog): update changelog [skip ci]
2026-03-02 09:54:22 +00:00
7cb5407bcf
Refactor CI workflow for clarity and efficiency
...
Refactor CI workflow to simplify configuration and improve readability. Consolidate steps, update job dependencies, and enhance linting and testing stages.
2026-03-02 10:54:05 +01:00
f7cf7e2a4c
fix(migrations): resolve multiple Alembic heads causing CI test crash
...
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 09:50:59 +00:00
3d38813da5
docs(changelog): update changelog [skip ci]
2026-03-02 09:47:27 +00:00
fc72532aea
Merge pull request #465 from christianlouis/copilot/optimize-ci-pipeline
...
ci: restructure pipeline into 5-stage fail-fast architecture
2026-03-02 10:47:08 +01:00
ea0f7fb54f
ci: optimize pipeline for fail-fast feedback loop
...
- Move mypy to Stage 1 (runs in parallel with lint & html-lint, no needs)
- Decouple dependency-scan from test-quick; tests now start as soon as
static analysis passes (needs: [lint, html-lint, mypy])
- dependency-scan runs as a parallel background track and still gates
build/deploy to prevent shipping with known CVEs
- Integration tests remain sequentially after quick tests pass (Stage 4)
- Build & deploy remain gated on ALL stages including dependency-scan (Stage 5)
- Reorder job definitions to match logical stage flow for readability
- Update section comments to reflect the new 5-stage architecture
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-02 09:45:59 +00:00
0fc8148c47
Initial plan
2026-03-02 09:41:10 +00:00
ed8579757c
docs(changelog): update changelog [skip ci]
2026-03-02 09:37:44 +00:00
c8d456e6eb
Merge pull request #464 from christianlouis/dependabot/pip/ocrmypdf-gte-16.0.0-and-lt-18.0.0
...
Update ocrmypdf requirement from <17.0.0,>=16.0.0 to >=16.0.0,<18.0.0
2026-03-02 10:37:27 +01:00
25532aad87
Update ocrmypdf requirement from <17.0.0,>=16.0.0 to >=16.0.0,<18.0.0
...
Updates the requirements on [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF ) to permit the latest version.
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases )
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v16.0.0...v17.3.0 )
---
updated-dependencies:
- dependency-name: ocrmypdf
dependency-version: 17.3.0
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-03-01 23:09:06 +00:00
1e1ba1d8d7
Merge branch 'main' into copilot/add-document-similarity-detection
2026-03-01 22:06:55 +01:00
698c6a222e
chore(release): update build metadata files [skip ci]
2026-03-01 21:06:18 +00:00
c7c9040321
0.71.0
...
Automatically generated by python-semantic-release
2026-03-01 21:06:16 +00:00
1b3f35d77d
Merge pull request #462 from christianlouis/copilot/add-webhook-support
...
feat(webhooks): implement webhook support for external integrations
2026-03-01 22:05:57 +01:00
a85992ee79
docs(webhooks): fix spelling - Behaviour to Behavior
...
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-01 20:47:51 +00:00
b7c78177e9
docs(similarity): add API documentation and fix template accessibility
...
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-01 20:46:13 +00:00
a953b726fc
docs(webhooks): add webhook API and configuration documentation
...
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com >
2026-03-01 20:45:20 +00:00
copilot-swe-agent[bot]