A) Per-option Save Button
- Add per-setting Save button in settings.html (visible only when value changed)
- Button calls POST /api/settings/{key} directly; existing bulk Save retained
- Add Audit Log link in settings page header
B) Immediate Worker Sync
- New app/utils/settings_sync.py with notify_settings_updated() (Redis version key)
and register_settings_reload_signal() (Celery task_prerun handler)
- Register signal in celery_worker.py at startup
- All API write paths call notify_settings_updated() after successful saves
C) Audit Log
- Add SettingsAuditLog model (key, old_value, new_value, changed_by, changed_at, action)
- save_setting_to_db / delete_setting_from_db accept changed_by and write audit entries
- New get_audit_log() service function (masks sensitive values)
- New GET /api/settings/audit-log endpoint (admin-only)
- New GET /admin/settings/audit-log view + audit_log.html template
- Visible to all admins (per clarified requirement)
D) Config Rollback / History
- New get_setting_history() and rollback_setting() service functions
- New GET /api/settings/{key}/history endpoint
- New POST /api/settings/{key}/rollback/{history_id} endpoint
- Rollback buttons in audit_log.html with confirmation dialog
- Tests: 25 new tests covering audit log, rollback, worker sync helpers, and API endpoints
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
`GET /api/settings/{key}` was calling `validate_setting_key()` which raises
HTTP 404 for keys not in SETTING_METADATA. The test expects 200 with value=None
for unknown keys.
Added `validate_setting_key_format()` to `input_validation.py` that validates
only the key format without the SETTING_METADATA existence check. Updated
`get_setting` to use the format-only validator; POST/DELETE endpoints continue
using the full `validate_setting_key()` for write-side security.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add CORSMiddleware (disabled by default, enabled via CORS_ENABLED=true)
- Add cors_enabled, cors_allowed_origins, cors_allow_credentials,
cors_allowed_methods, cors_allowed_headers settings to config.py
- Add parse_comma_separated_list validator for CORS list env vars
- Insert CORS middleware between SessionMiddleware and ProxyHeaders
so preflight runs before CSRF/auth but after proxy-header processing
- Document CORS env vars in .env.demo with rationale for proxy-first approach
- Mark CORS TODO as completed in SECURITY_AUDIT.md
- Add tests/test_cors.py with 12 unit and integration tests
Closes#175
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add RequestSizeLimitMiddleware that checks Content-Length header
before request body is read: non-multipart requests capped at
MAX_REQUEST_BODY_SIZE (default 1 MB), multipart uploads capped at
MAX_UPLOAD_SIZE (default 1 GB). Returns HTTP 413 on violation.
- Register middleware in app/main.py
- Add max_request_body_size setting to app/config.py
- Fix ui_upload in files.py to check Content-Length early and read
in 64 KB chunks (bounded memory usage), removing the post-write
os.path.getsize check
- Document MAX_REQUEST_BODY_SIZE in .env.demo and ConfigurationGuide.md
- Mark SECURITY_AUDIT.md item #4 as resolved
- Add 9 tests in test_request_size_limit.py
- Update test_upload_file_too_large to use patch.object instead of
the now-unused os.path.getsize mock
Closes#173
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix ruff format: add blank line before nested function in app/views/filemanager.py
- Fix ruff format: use double-quote escaping in tests/test_config.py
- Remove deploy job (Portainer webhook) from ci.yml
- Add update-k8s-manifest job: after main-branch build, updates
apps/docuelevate/preprod/docuelevate-stack.yaml in christianlouis/k8s-cluster-state
with the new GHCR image tag (ghcr.io/christianlouis/docuelevate:main-<short-sha>)
using mikefarah/yq@v4.44.6 and GH_PAT secret for cross-repo write access
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add [[tool.mypy.overrides]] section for app/utils/** with disallow_untyped_defs=true
- Add type annotations to all functions in app/utils/ (12 files)
- Fix type annotations in app/config.py and app/database.py (imported by utils)
- All 85 source files now pass mypy type checking
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive tests for imap_tasks.py covering:
- Lock acquire/release mechanisms
- pull_all_inboxes task with various scenarios
- pull_inbox for Gmail and non-Gmail with edge cases
- find_all_mail_xlist functionality
- Extended fetch_attachments tests for all MIME types
- Edge cases: invalid JSON, missing Message-ID, already processed, etc.
- Achieve 98.26% coverage for imap_tasks.py (up from 48.78%)
- Fix config_validator.py to include validate_auth_config export
- Update tests to verify all exports including validate_auth_config
- Note: config_validator.py file is shadowed by config_validator/ directory
in Python's module resolution, so it cannot be directly imported or tested.
The package's __init__.py (which has 100% coverage) is what's actually used.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add comprehensive logging to all pipeline retry branches (process_document, process_with_azure_document_intelligence, extract_metadata_with_gpt, embed_metadata_into_pdf)
- Add original_file_path as 3rd fallback for embed_metadata_into_pdf (checks: local_filename, processed_file_path, original_file_path, workdir/tmp fallback)
- Include all checked paths with existence status in 400 error responses for easier debugging
- Add enhanced logging to upload task retry path showing which processed file paths were checked
- Log successful file path when found
- Add comprehensive test suite covering new logging and fallback behavior
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _retry_pipeline_step to check for file in tmp, processed, and fallback locations
- Pass full path to extract_metadata_with_gpt instead of just basename
- Update extract_metadata_with_gpt to handle both basename and full path parameters
- Add test case for retrying when file is in processed directory
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Move get_all_settings_from_db import to module level in app/views/settings.py
- Move get_provider_status and get_settings_for_display imports to module level in app/views/status.py
- Fix CI workflow: replace deprecated 'file' parameter with 'files' in codecov-action
- Fix CI workflow: update test results upload to use codecov-action@v5 with report_type instead of deprecated test-results-action@v1
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Replace isinstance(ftp, ftplib.FTP_TLS) with a boolean flag to avoid issues when FTP_TLS is mocked in tests. Also fix Google Drive test parameter passing.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Fix PLW2901: Use different variable name for stripped lines in loop
- Fix E721: Use 'is' instead of '==' for type comparisons
- Add noqa comments for intentional security warnings (S321, S507, S110, S603)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Implement step timeout detection to prevent files from getting stuck in 'pending' state
- Add monitor_stalled_steps periodic task running every minute (Celery Beat)
- Automatically mark in-progress steps as failed if they exceed timeout (default: 10 minutes)
- Add step_timeout configuration setting (default: 600 seconds)
- Recover stalled steps with error message indicating when timeout was triggered
- Fix duplicate check to exclude self-comparison (file not duplicate of itself)
When processing crashes or hangs:
1. Worker detects stalled steps (in_progress for >10 minutes)
2. Marks them as failed with timeout error message
3. Updates UI to show failure status
4. Allows file to be retried or handled by user
This prevents files from being indefinitely stuck in processing state and provides
visibility into what went wrong.
copilot-swe-agent[bot]