fix(status): prevent false Completed status when mandatory pipeline steps have not run

Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.

- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
  total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
  completed filter
- process_document: call initialize_file_steps after creating a new
  file record so all mandatory steps are pre-created as pending

Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.

Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-02-27 00:51:14 +00:00
parent 515f5b97e9
commit e6dd39c27d
7 changed files with 165 additions and 12 deletions
+11 -5
View File
@@ -76,7 +76,7 @@ def sample_files(db_session):
)
db_session.add(step3)
# File 4: completed (has success step, no failures)
# File 4: completed (has success step including terminal step)
file4 = FileRecord(
filehash="hash4",
original_filename="completed.pdf",
@@ -87,10 +87,11 @@ def sample_files(db_session):
db_session.add(file4)
db_session.flush()
step4 = FileProcessingStep(file_id=file4.id, step_name="extract_text", status="success")
db_session.add(step4)
step4a = FileProcessingStep(file_id=file4.id, step_name="extract_text", status="success")
step4b = FileProcessingStep(file_id=file4.id, step_name="send_to_all_destinations", status="success")
db_session.add_all([step4a, step4b])
# File 5: completed with multiple success steps
# File 5: completed with multiple success steps including terminal step
file5 = FileRecord(
filehash="hash5",
original_filename="completed2.pdf",
@@ -111,7 +112,12 @@ def sample_files(db_session):
step_name="extract_metadata_with_gpt",
status="success",
)
db_session.add_all([step5a, step5b])
step5c = FileProcessingStep(
file_id=file5.id,
step_name="send_to_all_destinations",
status="success",
)
db_session.add_all([step5a, step5b, step5c])
# File 6: has success but also failure (should be filtered out from completed)
file6 = FileRecord(