fix(status): prevent false Completed status when mandatory pipeline steps have not run
Add a terminal-step guard (send_to_all_destinations) to all status calculation paths so that files are only marked Completed once the entire processing pipeline has been recorded. - get_file_overall_status: require TERMINAL_STEP to be present - get_files_processing_status: same guard for bulk status - get_step_summary: count missing terminal step as queued so total_main_steps > main_completed when pipeline is incomplete - apply_status_filter: SQL sub-query requires terminal step for completed filter - process_document: call initialize_file_steps after creating a new file record so all mandatory steps are pre-created as pending Define TERMINAL_STEP constant in step_manager.py and reference it in file_status.py and file_queries.py to avoid magic strings. Tests updated: add send_to_all_destinations to completed-file fixtures; add test verifying initialize_file_steps is called for new files. Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -76,7 +76,7 @@ def sample_files(db_session):
|
||||
)
|
||||
db_session.add(step3)
|
||||
|
||||
# File 4: completed (has success step, no failures)
|
||||
# File 4: completed (has success step including terminal step)
|
||||
file4 = FileRecord(
|
||||
filehash="hash4",
|
||||
original_filename="completed.pdf",
|
||||
@@ -87,10 +87,11 @@ def sample_files(db_session):
|
||||
db_session.add(file4)
|
||||
db_session.flush()
|
||||
|
||||
step4 = FileProcessingStep(file_id=file4.id, step_name="extract_text", status="success")
|
||||
db_session.add(step4)
|
||||
step4a = FileProcessingStep(file_id=file4.id, step_name="extract_text", status="success")
|
||||
step4b = FileProcessingStep(file_id=file4.id, step_name="send_to_all_destinations", status="success")
|
||||
db_session.add_all([step4a, step4b])
|
||||
|
||||
# File 5: completed with multiple success steps
|
||||
# File 5: completed with multiple success steps including terminal step
|
||||
file5 = FileRecord(
|
||||
filehash="hash5",
|
||||
original_filename="completed2.pdf",
|
||||
@@ -111,7 +112,12 @@ def sample_files(db_session):
|
||||
step_name="extract_metadata_with_gpt",
|
||||
status="success",
|
||||
)
|
||||
db_session.add_all([step5a, step5b])
|
||||
step5c = FileProcessingStep(
|
||||
file_id=file5.id,
|
||||
step_name="send_to_all_destinations",
|
||||
status="success",
|
||||
)
|
||||
db_session.add_all([step5a, step5b, step5c])
|
||||
|
||||
# File 6: has success but also failure (should be filtered out from completed)
|
||||
file6 = FileRecord(
|
||||
|
||||
Reference in New Issue
Block a user