perf(db): add indexes on query-hot columns and Redis caching layer

- Add database indexes on FileRecord.created_at, FileRecord.mime_type,
  ProcessingLog.file_id, ProcessingLog.timestamp, and
  FileProcessingStep.status for faster filtering, sorting, and joins.
- Add _ensure_indexes() migration for existing databases.
- Create app/utils/cache.py with fail-open Redis GET/SET/DELETE helpers.
- Cache MIME types dropdown query in files view (120s TTL).
- Optimize batch status query to load only needed columns.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-01 14:38:28 +00:00
parent a4fafe5c56
commit b79ba81e05
7 changed files with 467 additions and 10 deletions
+8 -2
View File
@@ -106,9 +106,15 @@ def get_files_processing_status(db: Session, file_ids: List[int]) -> Dict[int, D
if settings.enable_deduplication:
REAL_STEPS.add("check_for_duplicates")
# Get all REAL steps for these files in one query
# Get all REAL steps for these files in one query (load only needed columns)
steps = (
db.query(FileProcessingStep)
db.query(
FileProcessingStep.file_id,
FileProcessingStep.step_name,
FileProcessingStep.status,
FileProcessingStep.updated_at,
FileProcessingStep.created_at,
)
.filter(FileProcessingStep.file_id.in_(file_ids), FileProcessingStep.step_name.in_(REAL_STEPS))
.all()
)