@@ -5,36 +5,154 @@ All notable changes to DocuElevate will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## Important Note
**As of v0.6.0**: This CHANGELOG is automatically generated and maintained by [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release). Do not edit manually.
**Prior to v0.6.0**: This CHANGELOG was manually maintained. The transition to automated releases includes:
- Standardized tag format with `v` prefix (e.g., `v0.6.0`)
- Automated version bumping based on conventional commits
- Auto-generated release notes from commit messages
> **This CHANGELOG is automatically generated and maintained by [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release). Do not edit it manually.**
> New entries are prepended automatically on every merge to `main` that triggers a version bump.
---
## [Unreleased]
### Added
-Automated semantic versioning with python-semantic-release
-Conventional commit validation via commitlint
- Automated CHANGELOG generation
- Automated GitHub Release creation with notes
- Documentation archive for one-off documents
### Changed
- Docker image name from `christianlouis/document-processor` to `christianlouis/docuelevate`
- GHCR image references updated to `docuelevate`
- Comprehensive documentation updates for versioning and release process
### Fixed
-Fixed CHANGELOG.md not being updated by semantic-release (`autoescape = true` Jinja2 bug)
-Added explicit `changelog_file`, `mode = "update"`, and `output_format = "md"` to semantic-release config
- Added fallback `semantic-release changelog` step in release workflow for safety
### Documentation
- Added conventional commits guide to CONTRIBUTING.md
-Updated AGENTIC_CODING.md with versioning/release process
-Updated .github/copilot-instructions.md with commit format rules
-Archived historical one-off documentation to docs/archive/
- Added Documentation-First Development section to CONTRIBUTING.md and AGENTIC_CODING.md
-Added note to README.md screenshots section indicating they may lag the current UI
-Updated TODO.md to reflect current version (v0.40.0) and completed tasks
- Retroactively populated CHANGELOG.md with all post-v0.5.0 changes
---
## [0.40.0] - 2026-02-23
> **Retroactive summary.** Releases v0.6.0 through v0.40.0 were cut automatically by
> `python-semantic-release` from conventional commits, but the CHANGELOG was not updated
> at the time due to a configuration bug (`autoescape = true`). This section documents all
> known changes made after v0.5.0.
### Added
#### Security Middleware Stack
- **CSRF Protection** (`app/middleware/csrf.py`): Per-session cryptographic tokens validated on all state-changing requests (POST/PUT/DELETE/PATCH). Token delivered via `X-CSRF-Token` header or `csrf_token` form field. No-op when `AUTH_ENABLED=False`.
- **Rate Limiting** (`app/middleware/rate_limit.py`): SlowAPI + Redis-backed rate limiting. Configurable defaults: 100 req/min (API), 600 req/min (uploads), 10 req/min (auth). Falls back to in-memory for development.
- **Security Headers** (`app/middleware/security_headers.py`): Configurable HSTS, CSP, `X-Frame-Options`, and `X-Content-Type-Options` headers. Each header individually togglable for reverse-proxy deployments.
- **Audit Logging** (`app/middleware/audit_log.py`): Per-request structured log entries with sensitive-value masking. Elevated `[SECURITY]` log level for 401/403/login/5xx events.
- **Request Size Limiting** (`app/middleware/request_size_limit.py`): Separate limits for JSON/form bodies (`MAX_REQUEST_BODY_SIZE`, default 1 MB) and file upload multipart bodies (`MAX_UPLOAD_SIZE`, default 1 GB). Returns HTTP 413 immediately without reading the full body.
- **CORS** (`main.py`): Configurable CORS policy via `CORS_ENABLED`, `CORS_ALLOWED_ORIGINS`, `CORS_ALLOW_CREDENTIALS`, `CORS_ALLOWED_METHODS`, and `CORS_ALLOWED_HEADERS`.
#### New Storage Providers
- **Amazon S3** (`app/tasks/upload_to_s3.py`): Upload to S3-compatible buckets. Configured via `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, and `S3_BUCKET_NAME`.
- **SFTP** (`app/tasks/upload_to_sftp.py`): Secure file transfer with password or private-key authentication. Supports `SFTP_DISABLE_HOST_KEY_VERIFICATION` flag.
- **FTP/FTPS** (`app/tasks/upload_to_ftp.py`): FTP with automatic FTPS upgrade attempt; plaintext fallback configurable via `FTP_ALLOW_PLAINTEXT`.
- **Email/SMTP** (`app/tasks/upload_to_email.py`): Send processed documents as email attachments. Supports TLS, configurable sender, and default recipient.
- **rclone** (`app/tasks/upload_with_rclone.py`): Delegate uploads to a locally configured `rclone` binary, enabling support for any of rclone's 40+ cloud providers.
#### File Processing Improvements
- **PDF Page Rotation** (`app/tasks/rotate_pdf_pages.py`): Detects and corrects skewed pages using Azure Document Intelligence angle metadata.
- **Metadata Embedding** (`app/tasks/embed_metadata_into_pdf.py`): Writes GPT-extracted metadata as PDF document properties using pypdf.
- **PDF Splitting** (`app/utils/file_splitting.py`): Splits oversized PDFs at page boundaries into chunks ≤ `MAX_SINGLE_FILE_SIZE` bytes. Each chunk is a valid, readable PDF.
- **Document Deduplication**: SHA-256 hash-based duplicate detection. Controlled by `ENABLE_DEDUPLICATION` and `SHOW_DEDUPLICATION_STEP` settings.
- **Forced Cloud OCR**: `force_cloud_ocr` flag on `process_document` task to bypass local text extraction and always use Azure Document Intelligence. Useful for reprocessing.
#### Processing Status Tracking (Dual-Table Architecture)
- **`FileProcessingStep` model**: New DB table; single row per (file, step) pair tracking current state (`pending`, `in_progress`, `success`, `failure`, `skipped`). Replaces log-scanning for status queries.
- **Step Timeout Detection** (`app/utils/step_timeout.py`): Marks steps stuck in `in_progress` as `failure` after configurable `STEP_TIMEOUT` seconds (default 600).
- **Stalled Step Monitor** (`app/tasks/monitor_stalled_steps.py`): Periodic Celery task (runs every minute) that calls the timeout detection logic.
- **Log Migration Utility** (`app/utils/migrate_logs_to_steps.py`): Back-fills `FileProcessingStep` from existing `ProcessingLog` entries for files processed before the new table existed.
#### New API Endpoints
- **`POST /api/upload-from-url`** (`app/api/url_upload.py`): Upload a document from a remote URL. Includes SSRF protection (blocks private IPs and loopback addresses).
- **`GET /api/logs`** (`app/api/logs.py`): Paginated, filterable list of `ProcessingLog` entries; filterable by `file_id` and `task_name`.
- **`GET /api/diagnostic/settings`** (`app/api/diagnostic.py`): Admin-only endpoint that dumps non-sensitive configuration to logs and returns summary info.
- **`GET /api/whoami`** (`app/api/user.py`): Returns session user info including a Gravatar URL derived from the authenticated user's email.
#### Notification System
- **Apprise integration** (`app/utils/notification.py`): Multi-channel notifications via the Apprise library (70+ services: Slack, email, Telegram, PushOver, etc.). Configured via `NOTIFICATION_URLS`.
- **Uptime Kuma integration** (`app/tasks/uptime_kuma_tasks.py`): Periodic heartbeat ping to a configured Uptime Kuma push URL (`UPTIME_KUMA_URL`, `UPTIME_KUMA_PING_INTERVAL`).
#### Admin & Operations
- **Admin File Manager** (`app/views/filemanager.py`): Three-pane admin view at `/filemanager`:
- *Filesystem view*: Browse `workdir` tree with DB cross-reference per file.
- *Database view*: List all `FileRecord` rows with on-disk existence flag.
- *Reconcile view*: Delta view showing orphan disk files and ghost DB records.
- **Credential Checker Task** (`app/tasks/check_credentials.py`): Periodic Celery task that validates all configured provider credentials (OpenAI, Azure, Dropbox, Google Drive, OneDrive) and sends a notification on failure.
- **Settings Audit Log** (`ApplicationSettings` + `SettingsAuditLog` models): Every settings change is recorded with timestamp, user, and before/after values.
- **ProcessAll Throttling**: Configurable `PROCESSALL_THROTTLE_THRESHOLD` and `PROCESSALL_THROTTLE_DELAY` to prevent flooding the task queue during bulk reprocessing.
- **Worker Settings Sync** (`app/utils/settings_sync.py`): Publishes a version token to Redis whenever settings change; Celery workers reload settings from DB before each task, ensuring config changes propagate without a restart.
#### Browser Extension (v1.1.0)
- Send files and web pages from the browser directly to DocuElevate with one click.
- Context menu integration on links, images, and pages.
- Manifest v3 compatible; works with Chrome, Firefox, Edge, and Chromium-based browsers.
- In-browser notifications for upload status.
#### OpenAI Customization
-`OPENAI_BASE_URL` setting (default `https://api.openai.com/v1`): Enables use of OpenAI-compatible endpoints (Azure OpenAI, local models, etc.).
-`OPENAI_MODEL` setting (default `gpt-4o-mini`): Model selection without code changes.
#### Configuration & Developer Experience
- **Config Loader** (`app/utils/config_loader.py`): Hot-reload of settings from DB without service restart.
- **Config Validator** (`app/utils/config_validator/`): Modular validation with provider status, masked display, and per-provider readiness checks.
- **Input Validation** (`app/utils/input_validation.py`): Centralised validators for sort fields, sort order, search query length, task ID format (UUID v4), and settings key format.
- **Filename Utilities** (`app/utils/filename_utils.py`): `sanitize_filename`, `get_unique_filename`, and `extract_remote_path` helpers shared across upload tasks.
- **OAuth Helper** (`app/utils/oauth_helper.py`): Shared token-exchange logic reused by Dropbox, Google Drive, and OneDrive OAuth flows.
- **Retry Configuration** (`app/tasks/retry_config.py`): `BaseTaskWithRetry` base class with auto-retry (3 attempts, 10 s initial delay, exponential backoff) shared by all upload tasks.
- **HTTP Request Timeout**: Configurable `HTTP_REQUEST_TIMEOUT` (default 120 s) to handle large file operations gracefully.
- **File Deletion Toggle**: `ALLOW_FILE_DELETE` setting to prevent accidental deletions in production.
### Changed
- Docker image renamed from `christianlouis/document-processor` to `christianlouis/docuelevate`
-`app/routes/` deprecated; all endpoints migrated to `app/api/` and `app/views/`
-`FileRecord` status now derived from `FileProcessingStep` rows instead of scanning `ProcessingLog`
- Settings changes now propagated to Celery workers via Redis version key (no restart required)
- Dependency scanner switched from `safety` to `pip-audit` in CI pipeline
- CI pipeline streamlined: removed redundant DeepSource integration (40–50% faster CI runs)
-`app/utils/logging.py` introduced as canonical import point for `log_task_progress`
### Fixed
- **Critical: Path Traversal via GPT Metadata Filename** — GPT-extracted `filename` metadata was used directly in file path construction. Fixed by running all GPT-suggested filenames through `sanitize_filename` before use.
- **Medium: Path Traversal in File API** — `file_path` query parameters sanitised to block `../` sequences.
- **Medium: Unvalidated Sort Parameters** — sort field and order inputs in file list endpoint now validated against an allowlist.
- OAuth admin group detection now correctly handles groups list from Authentik userinfo response.
- Session secret validation raises a clear error at startup instead of silently using an insecure default.
- Redirect loop for logged-in non-admin users on `/settings` route resolved.
### Security
- CSRF protection added to all state-changing endpoints
- Rate limiting prevents brute-force and DoS attacks on auth and upload endpoints
- CI de-duplication summary archived in `docs/CI_DEDUPLICATION_SUMMARY.md`
- OAuth testing summary archived in `OAUTH_IMPLEMENTATION_SUMMARY.md`
- WebDAV testing summary archived in `WEBDAV_TESTING_SUMMARY.md`
---
@@ -200,9 +318,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## Historical Release Links
**Note**: Tags v0.3.1, v0.3.2, v0.3.3, and v0.5.0 do not exist as GitHub Releases. The features described in those versions were included in the codebase but not formally released. The latest actual release is 0.4.3. Going forward (v0.6.0+), all releases will have corresponding GitHub Releases and tags created automatically by semantic-release.
**Note**: Tags v0.3.1, v0.3.2, v0.3.3, v0.5.0, and the retroactive v0.40.0 summary do not correspond
one-to-one with formal GitHub Releases from that period. Going forward all releases have corresponding
GitHub Releases and tags created automatically by `python-semantic-release`.
- Keep `docs/` files in sync with code changes in the same PR
- Update `TODO.md` when completing or adding tasks
-`CHANGELOG.md` is generated automatically—**do not add regular release entries manually**. Retroactive corrections to historical entries are the only acceptable exception.
- Screenshots in README and docs should reflect current UI; update them when the UI changes significantly
- Use present tense and second person ("you") in user-facing docs
### Automated Changelog
`CHANGELOG.md` is generated automatically by [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) on every merge to `main`. **Do not edit it manually.** Your commit messages (following Conventional Commits) drive the changelog content.
---
## Pull Request Checklist
Before submitting a pull request:
@@ -155,7 +185,7 @@ Before submitting a pull request:
- [ ] Commit messages follow conventional commit format
- [ ] Pre-commit hooks installed and passing (see below)
- [ ] Tests added/updated for new functionality
- [ ] Documentation updated if user-facing changes
- [ ]**Documentation updated** for any user-facing, API, or configuration changes
- [ ] No manual edits to `VERSION` or `CHANGELOG.md`
**Current Version:** v0.40.0 (see `VERSION` file; managed by semantic-release)
This document tracks actionable tasks for the current development cycle. For long-term planning, see [ROADMAP.md](ROADMAP.md) and [MILESTONES.md](MILESTONES.md).
@@ -57,6 +57,7 @@ As of this update, DocuElevate uses **automated semantic versioning** via `pytho
- [x] Add CodeQL scanning
- [x] Implement semantic-release for automated versioning
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Christian Krakau-Louis