fix(status): prevent false Completed status when mandatory pipeline steps have not run

Add a terminal-step guard (send_to_all_destinations) to all status
calculation paths so that files are only marked Completed once the
entire processing pipeline has been recorded.

- get_file_overall_status: require TERMINAL_STEP to be present
- get_files_processing_status: same guard for bulk status
- get_step_summary: count missing terminal step as queued so
  total_main_steps > main_completed when pipeline is incomplete
- apply_status_filter: SQL sub-query requires terminal step for
  completed filter
- process_document: call initialize_file_steps after creating a new
  file record so all mandatory steps are pre-created as pending

Define TERMINAL_STEP constant in step_manager.py and reference it in
file_status.py and file_queries.py to avoid magic strings.

Tests updated: add send_to_all_destinations to completed-file
fixtures; add test verifying initialize_file_steps is called for
new files.

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-02-27 00:51:14 +00:00
parent 515f5b97e9
commit e6dd39c27d
7 changed files with 165 additions and 12 deletions
+4
View File
@@ -17,6 +17,7 @@ from app.tasks.extract_metadata_with_gpt import extract_metadata_with_gpt
from app.tasks.process_with_ocr import process_with_ocr
from app.tasks.retry_config import BaseTaskWithRetry
from app.utils import get_unique_filepath_with_counter, hash_file, log_task_progress
from app.utils.step_manager import initialize_file_steps
from app.utils.text_quality import check_text_quality, detect_pdf_text_source
logger = logging.getLogger(__name__)
@@ -202,6 +203,9 @@ def process_document(
db.commit()
db.refresh(new_record)
logger.info(f"[{task_id}] File record created with ID: {new_record.id}")
# Pre-initialize all expected processing steps as "pending" so that
# status tracking reflects the complete pipeline from the start.
initialize_file_steps(db, new_record.id)
log_task_progress(
task_id,
"create_file_record",