feat(storage): implement immutable original/processed file storage and collision handling
- Add original_file_path and processed_file_path columns to FileRecord model
- Create database migration for new fields
- Implement get_unique_filepath_with_counter() with -0001 suffix format
- Update process_document to save immutable copy to /workdir/original
- Add force_cloud_ocr parameter to process_document for forced OCR
- Update embed_metadata to use new collision handling
- Update metadata JSON to include file path references
- Add /files/{file_id}/reprocess-with-cloud-ocr API endpoint
- Update processed_file_path in database during embedding
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
@@ -30,6 +30,14 @@ class FileRecord(Base):
|
||||
# The name/path we store on disk (e.g. /workdir/tmp/<uuid>.pdf)
|
||||
local_filename = Column(String, nullable=False)
|
||||
|
||||
# Immutable original copy path (e.g. /workdir/original/<uuid>.pdf)
|
||||
# This is the first copy made when the file is ingested
|
||||
original_file_path = Column(String)
|
||||
|
||||
# Processed copy path (e.g. /workdir/processed/2024-01-01_Invoice.pdf)
|
||||
# This is the final file with embedded metadata before upload
|
||||
processed_file_path = Column(String)
|
||||
|
||||
# Size of the file in bytes
|
||||
file_size = Column(Integer, nullable=False)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user