Commit Graph

23 Commits

Author SHA1 Message Date
copilot-swe-agent[bot] e518bce922 fix: merge main branch and renumber migration 037→040
Resolve all merge conflicts between our automation feature branch and
current main (v0.163.0, 920 commits ahead).

Conflicts resolved:
- app/api/__init__.py: add automation_router alongside main's new routers
  (classification_rules, qr_auth, sessions, system_reset)
- app/config.py: add main's new settings (dropbox_use_global_credentials,
  factory_reset_on_startup, enable_factory_reset)
- app/models.py: add main's new models (ClassificationRuleModel, UserSession,
  QRLoginChallenge, SharePoint integration type)
- app/utils/settings_service.py: merge automation_hooks_enabled with main's
  new metadata entries
- docs/API.md: merge automation API docs with main's classification rules docs
- docs/ConfigurationGuide.md: add factory reset settings
- tests/conftest.py: import both AutomationHook and new main models

Migration renumbered:
- 037_add_automation_hooks → 040_add_automation_hooks
- down_revision: 039_add_classification_rules (was 036_add_document_translation_fields)
- Chain: 036 → 037 → 038 → 039 → 040 (automation hooks)

For all non-automation files with conflicts, main's version was taken since
our branch did not modify those files (conflicts were from a stale prior merge).

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Agent-Logs-Url: https://github.com/christianlouis/DocuElevate/sessions/cb62f012-3b69-4415-835e-3857ce3e9f45
2026-03-20 23:54:04 +00:00
copilot-swe-agent[bot] 204000aabc fix: merge main branch and renumber migration 027→037
Resolve 3 merge conflicts and renumber the automation_hooks migration
to follow main's migration chain (036_add_document_translation_fields).

Conflicts resolved:
- app/api/__init__.py: add automation_router alongside main's new routers
- app/utils/settings_service.py: add automation_hooks_enabled alongside compliance_enabled
- tests/conftest.py: add AutomationHook alongside AuditLog/ComplianceTemplate imports

Migration renumbered:
- 027_add_automation_hooks → 037_add_automation_hooks
- down_revision: 026_add_scheduled_jobs → 036_add_document_translation_fields

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-16 22:40:15 +00:00
copilot-swe-agent[bot] b87dd6083c fix(database): skip create_all for Alembic-tracked databases to prevent OperationalError on webhook_configs
When init_db() called Base.metadata.create_all() before Alembic migrations,
the ORM model created the webhook_configs table. Alembic migration 009 then
failed with OperationalError: table webhook_configs already exists.

Fix: check for alembic_version table before calling create_all(). Tracked
databases skip create_all and let Alembic handle all schema changes instead.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-02 10:25:34 +00:00
Christian Krakau-Louis 8ac4179b8b Merge pull request #445 from christianlouis/copilot/enforce-database-upgrades-migration
refactor(database): enforce Alembic-only schema migrations, deprecate manual migration code
2026-03-01 18:22:48 +01:00
copilot-swe-agent[bot] 87d1b9d935 refactor(database): enforce Alembic-only database migrations, deprecate manual schema migrations
- Create alembic.ini and migrations/env.py for full Alembic CLI + programmatic support
- Add Alembic migration 006: detail column on processing_logs
- Add Alembic migration 007: ocr_quality_score column + drop unique filehash index
- Add Alembic migration 008: performance indexes
- Replace _run_schema_migrations() call in init_db() with _run_alembic_upgrade()
- Deprecate _run_schema_migrations() with DeprecationWarning
- Update tests for new Alembic-based approach and deprecation
- Update DatabaseConfiguration.md documentation

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 16:47:41 +00:00
copilot-swe-agent[bot] febdf41469 fix(db): check column existence before creating performance indexes
_ensure_indexes() now verifies the target column exists in the table
before executing CREATE INDEX IF NOT EXISTS. This prevents failures
when migrating legacy database schemas that don't yet have all columns
(e.g. files table without created_at or mime_type).

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 15:09:32 +00:00
copilot-swe-agent[bot] b79ba81e05 perf(db): add indexes on query-hot columns and Redis caching layer
- Add database indexes on FileRecord.created_at, FileRecord.mime_type,
  ProcessingLog.file_id, ProcessingLog.timestamp, and
  FileProcessingStep.status for faster filtering, sorting, and joins.
- Add _ensure_indexes() migration for existing databases.
- Create app/utils/cache.py with fail-open Redis GET/SET/DELETE helpers.
- Cache MIME types dropdown query in files view (120s TTL).
- Optimize batch status query to load only needed columns.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 14:38:28 +00:00
copilot-swe-agent[bot] 83c3405c98 feat: persist ocr_quality_score and use it for numeric filtering
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 13:46:43 +00:00
copilot-swe-agent[bot] 82f7113d39 fix(db): add saved_searches table to runtime schema migrations
The Alembic migration alone doesn't run automatically. Add the
saved_searches table creation to _run_schema_migrations() in
database.py so existing databases are upgraded at startup.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-03-01 11:36:42 +00:00
copilot-swe-agent[bot] db581ba355 fix(database): add runtime migrations for ocr_text, ai_metadata, document_title columns
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-25 14:37:56 +00:00
copilot-swe-agent[bot] d3b3575f8a style: fix import ordering in database.py
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-14 00:32:02 +00:00
copilot-swe-agent[bot] 53a9efa56d feat: add strict mypy type-checking for app/utils/ module
- Add [[tool.mypy.overrides]] section for app/utils/** with disallow_untyped_defs=true
- Add type annotations to all functions in app/utils/ (12 files)
- Fix type annotations in app/config.py and app/database.py (imported by utils)
- All 85 source files now pass mypy type checking

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-14 00:27:52 +00:00
copilot-swe-agent[bot] 43bc58770d refactor: consolidate linting tools into Ruff
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-13 09:10:52 +00:00
copilot-swe-agent[bot] dff32e08e1 test: add comprehensive tests for critical untested files and fix deprecated warnings
- Add tests for migrate_logs_to_steps.py, upload_to_paperless.py, dropbox API,
  upload_to_dropbox.py, upload_to_nextcloud.py, upload_to_onedrive.py,
  upload_with_rclone.py, and config_validator.py
- Fix PydanticDeprecatedSince20: @validator → @field_validator in config.py, url_upload.py
- Fix PydanticDeprecatedSince20: class Config → model_config = SettingsConfigDict
- Fix PydanticDeprecatedSince211: filter Pydantic internals in settings_display.py
- Fix MovedIn20Warning: use sqlalchemy.orm.declarative_base instead of ext.declarative

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-12 17:32:17 +00:00
Christian Krakau-Louis 9a2c2d20b1 feat(duplication): implement duplicate file handling and detection across processing steps 2026-02-12 02:38:00 +01:00
Christian Krakau-Louis 974dd64660 fix(database): add runtime schema migrations for deduplication columns
- Implement lightweight migration system for is_duplicate and duplicate_of_id columns
- Migrations run automatically on application startup
- Idempotent migrations safe to run multiple times
- Fixes SQLite OperationalError for missing columns
- Resolves issue where database schema didn't match model definitions
2026-02-12 02:38:00 +01:00
Christian Krakau-Louis 50cc0035d9 feat(database): add file path columns to files table during migration
fix(general): update processed files count query to use FileRecord.id

chore: add VSCode settings for pytest configuration
2026-02-11 22:39:21 +01:00
copilot-swe-agent[bot] b18f5ee13c feat(ui): add verbose worker log detail to processing history on file detail page
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-11 09:06:30 +00:00
copilot-swe-agent[bot] d08040ac4a style: fix all flake8 linter errors across app/ directory
- Run Black formatter and isort on all app/ files
- Remove unused imports (F401) across multiple files
- Add # noqa: F401 for intentional re-exports in celery_worker.py,
  tasks/__init__.py, utils.py, frontend.py, views/base.py
- Fix f-strings without placeholders (F541) in azure.py, notification.py,
  check_credentials.py, upload_to_onedrive.py, settings.py
- Fix bare except (E722) in upload_to_sftp.py
- Fix block comment format (E265) in models.py
- Move imports to top of file to fix E402 in celery_app.py, celery_worker.py
- Fix line-too-long (E501) by wrapping strings in multiple files
- Remove unused variable (F841) in upload_to_nextcloud.py

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-08 17:42:33 +00:00
Christian Krakau-Louis ae674e8fc4 added robust code to create the sqlite file 2025-03-25 14:33:46 +01:00
Christian Krakau-Louis 5eb8fa586b added duplicate check and database 2025-03-25 13:55:15 +01:00
Christian Krakau-Louis 19c763e595 added a favicon, updated the README.md file 2025-03-25 10:48:55 +01:00
Christian Krakau-Louis a924fe9f9e Create database.py 2025-02-09 16:20:56 +01:00