fix(status): prevent false Completed status when mandatory pipeline steps have not run
Add a terminal-step guard (send_to_all_destinations) to all status calculation paths so that files are only marked Completed once the entire processing pipeline has been recorded. - get_file_overall_status: require TERMINAL_STEP to be present - get_files_processing_status: same guard for bulk status - get_step_summary: count missing terminal step as queued so total_main_steps > main_completed when pipeline is incomplete - apply_status_filter: SQL sub-query requires terminal step for completed filter - process_document: call initialize_file_steps after creating a new file record so all mandatory steps are pre-created as pending Define TERMINAL_STEP constant in step_manager.py and reference it in file_status.py and file_queries.py to avoid magic strings. Tests updated: add send_to_all_destinations to completed-file fixtures; add test verifying initialize_file_steps is called for new files. Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -17,6 +17,7 @@ from app.tasks.extract_metadata_with_gpt import extract_metadata_with_gpt
|
||||
from app.tasks.process_with_ocr import process_with_ocr
|
||||
from app.tasks.retry_config import BaseTaskWithRetry
|
||||
from app.utils import get_unique_filepath_with_counter, hash_file, log_task_progress
|
||||
from app.utils.step_manager import initialize_file_steps
|
||||
from app.utils.text_quality import check_text_quality, detect_pdf_text_source
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -202,6 +203,9 @@ def process_document(
|
||||
db.commit()
|
||||
db.refresh(new_record)
|
||||
logger.info(f"[{task_id}] File record created with ID: {new_record.id}")
|
||||
# Pre-initialize all expected processing steps as "pending" so that
|
||||
# status tracking reflects the complete pipeline from the start.
|
||||
initialize_file_steps(db, new_record.id)
|
||||
log_task_progress(
|
||||
task_id,
|
||||
"create_file_record",
|
||||
|
||||
Reference in New Issue
Block a user