- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add release_names.json mapping version ranges to codenames
- Add release_name property to Settings in app/config.py
- Update build metadata script to include codename in RUNTIME_INFO
- Display release codename in status dashboard and page footer
- Inject release_name globally via template response wrapper
- Update ROADMAP.md with codenames for all milestone releases
- Add docs/ReleaseNaming.md with naming guide and best practices
- Add comprehensive tests for release name resolution
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
When enabled, IMAP processing will fetch and process attachments but
will NOT modify the mailbox state (no starring, labeling, deleting,
or flag changes). This allows preprod instances to safely share a
Gmail inbox with production without interfering with production
email processing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Raise quality acceptance threshold from 65→85 (configurable via TEXT_QUALITY_THRESHOLD)
- Reject text with significant issues (excessive_typos, garbage_characters,
incoherent_text, fragmented_sentences) even when score is above threshold
(configurable via TEXT_QUALITY_SIGNIFICANT_ISSUES)
- Add compare_text_quality() for AI-powered head-to-head comparison of
original embedded text vs fresh OCR output
- Update process_document to pass original text to OCR task for comparison
- Update process_with_ocr to run comparison and keep the higher-quality text
- Add new settings to settings_service.py metadata
- Update docs/ConfigurationGuide.md with new settings
- Add comprehensive tests for new threshold and comparison logic
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add CORSMiddleware (disabled by default, enabled via CORS_ENABLED=true)
- Add cors_enabled, cors_allowed_origins, cors_allow_credentials,
cors_allowed_methods, cors_allowed_headers settings to config.py
- Add parse_comma_separated_list validator for CORS list env vars
- Insert CORS middleware between SessionMiddleware and ProxyHeaders
so preflight runs before CSRF/auth but after proxy-header processing
- Document CORS env vars in .env.demo with rationale for proxy-first approach
- Mark CORS TODO as completed in SECURITY_AUDIT.md
- Add tests/test_cors.py with 12 unit and integration tests
Closes#175
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add RequestSizeLimitMiddleware that checks Content-Length header
before request body is read: non-multipart requests capped at
MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
in 64 KB chunks (bounded memory usage), removing the post-write
os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
the now-unused os.path.getsize mock
Closes#173
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add [[tool.mypy.overrides]] section for app/utils/** with disallow_untyped_defs=true
- Add type annotations to all functions in app/utils/ (12 files)
- Fix type annotations in app/config.py and app/database.py (imported by utils)
- All 85 source files now pass mypy type checking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Implement step timeout detection to prevent files from getting stuck in 'pending' state
- Add monitor_stalled_steps periodic task running every minute (Celery Beat)
- Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes)
- Add step_timeout configuration setting (default: 600 seconds)
- Recover stalled steps with error message indicating when timeout was triggered
- Fix duplicate check to exclude self-comparison (file not duplicate of itself)
When processing crashes or hangs:
1. Worker detects stalled steps (in_progress for >10 minutes)
2. Marks them as failed with timeout error message
3. Updates UI to show failure status
4. Allows file to be retried or handled by user
This prevents files from being indefinitely stuck in processing state and provides
visibility into what went wrong.
- Add enable_deduplication and show_deduplication_step config options
- Rename hash_file step to check_for_duplicates
- Make deduplication step conditional based on configuration
- Add is_duplicate and duplicate_of_id fields to FileRecord model
- Create database migration for new deduplication fields
- Update process_document task to log deduplication results
- Update step visualization to show/hide step based on config
- Update status calculations to include deduplication step conditionally
- Default: deduplication enabled, step displayed
- Can be configured to hide from UI while still processing
- Update generate_build_metadata.sh to sync VERSION from latest git tag
- Change BUILD_DATE format from date-only to ISO 8601 with time (YYYY-MM-DDTHH:MM:SSZ)
- Fix VERSION file from 0.5.0 to 0.9.1 (matching latest git tag v0.9.1)
- Change version fallback from hardcoded '0.5.0-dev' to 'unknown' in config.py
- Update release.yml to commit all build metadata files (not just VERSION)
- Update BuildMetadata.md documentation to reflect automated versioning
- Add tests for build_date with time format and version unknown fallback
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Security headers are now disabled by default since most deployments use a reverse proxy (Traefik, Nginx) that already adds these headers. Enable with SECURITY_HEADERS_ENABLED=true for direct deployments.
Changes:
- Set security_headers_enabled default to False in app/config.py
- Update all documentation to reflect new default
- Comment out examples in .env.demo (now showing disabled state)
- Update SECURITY_AUDIT.md to reflect reverse proxy as default deployment
- Tests still pass (3 passed, 8 skipped as expected with headers disabled)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add MAX_UPLOAD_SIZE config (default 1GB) to prevent resource exhaustion
- Add MAX_SINGLE_FILE_SIZE config for optional PDF file splitting
- Implement automatic PDF splitting when files exceed single file limit
- Update upload endpoint to use configured limits instead of hardcoded 500MB
- Add comprehensive tests for upload limits and file splitting
- Document configuration in ConfigurationGuide.md and SECURITY_AUDIT.md
- Reference SECURITY_AUDIT.md in error messages for user guidance
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Run Black formatter and isort on all app/ files
- Remove unused imports (F401) across multiple files
- Add # noqa: F401 for intentional re-exports in celery_worker.py,
tasks/__init__.py, utils.py, frontend.py, views/base.py
- Fix f-strings without placeholders (F541) in azure.py, notification.py,
check_credentials.py, upload_to_onedrive.py, settings.py
- Fix bare except (E722) in upload_to_sftp.py
- Fix block comment format (E265) in models.py
- Move imports to top of file to fix E402 in celery_app.py, celery_worker.py
- Fix line-too-long (E501) by wrapping strings in multiple files
- Remove unused variable (F841) in upload_to_nextcloud.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Settings management represents a significant new feature warranting a minor version bump:
- Database-backed configuration with 102 settings
- Fernet encryption for sensitive values
- Setup wizard for first-time installations
- Complete admin UI and REST API
- OAuth admin group support
Version updated: 0.3.3 → 0.5.0
Keep 0.3.3 release notes intact (drag-and-drop feature from main branch).
Add 0.5.0 as new current release with settings features.
Updated files:
- VERSION: 0.5.0
- CHANGELOG.md: Added 0.5.0 release, kept 0.3.3 intact
- MILESTONES.md: Added v0.5.0 section, adjusted future versions
- TODO.md: Updated current version
- ROADMAP.md: Updated current status
- app/config.py: Default version 0.5.0-dev
- docs/BuildMetadata.md: Updated reference
- ANALYSIS_SUMMARY.md: Updated version
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
VERSION:
- Update from 0.1.0-test to 0.3.3
CHANGELOG.md:
- Create comprehensive changelog for all releases
- Document v0.3.3 features: settings management, encryption, setup wizard
- List all new files, changes, fixes, and security improvements
TODO.md:
- Update version reference to v0.3.3
- Mark settings management features as complete
- Add completed items for 2026-02-08
- Update last review date
MILESTONES.md:
- Update last updated date
- Mark v0.3.3 as Released (2026-02-08)
- Add comprehensive release notes with all features
- Update version history table
- Update current release to v0.3.3 with new features listed
ROADMAP.md:
- Update to reflect v0.3.3 current status
- Add settings management features to current status
- Update last updated date
Other files:
- app/config.py: Update default version to 0.3.3-dev
- docs/BuildMetadata.md: Update default version reference
- ANALYSIS_SUMMARY.md: Update current version
All documentation now reflects v0.3.3 release with complete feature list.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]