diff --git a/NOTICE b/NOTICE index 2088f264..8ce08359 100644 --- a/NOTICE +++ b/NOTICE @@ -52,9 +52,9 @@ OpenAI (MIT License) Copyright (c) 2023 OpenAI https://github.com/openai/openai-python -PyPDF2 (BSD License) -Copyright (c) 2006-2008, Mathieu Fenniak -https://github.com/py-pdf/PyPDF2 +pypdf (BSD License) +Copyright (c) 2006-2024, pypdf contributors +https://github.com/py-pdf/pypdf Requests (Apache 2.0 License) Copyright 2019 Kenneth Reitz diff --git a/README.md b/README.md index 8e0361c8..db733ffa 100644 --- a/README.md +++ b/README.md @@ -205,7 +205,7 @@ The following is a summary of the licenses used by our direct dependencies: | SQLAlchemy | MIT | | Pydantic | MIT | | OpenAI | MIT | -| PyPDF2 | BSD | +| pypdf | BSD | | Requests | Apache 2.0 | | puremagic | MIT | | filetype | MIT | diff --git a/ROADMAP.md b/ROADMAP.md index 389693fe..5b1de5e7 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -180,7 +180,7 @@ DocuElevate aims to be the premier open-source intelligent document processing p ## Technology Debt ### Refactoring Needed -- [ ] Migrate from PyPDF2 to pypdf (modern fork) +- [x] Migrate from PyPDF2 to pypdf (modern fork) - ✅ Completed 2026-02-12 - [ ] Standardize error handling across modules - [ ] Consolidate configuration management - [ ] Optimize database queries diff --git a/SECURITY_AUDIT.md b/SECURITY_AUDIT.md index 7b9eac99..36b9d271 100644 --- a/SECURITY_AUDIT.md +++ b/SECURITY_AUDIT.md @@ -1,12 +1,51 @@ # Security Audit Report -**Date:** 2026-02-07 +**Date:** 2026-02-12 **Status:** Bandit Security Scan Completed - All Critical/High/Medium Issues Resolved ## Executive Summary This document tracks security vulnerabilities found in DocuElevate and their remediation status. A comprehensive security audit using Bandit has been completed, with all critical, high, and medium severity issues addressed. +## Recent Security Fixes + +### CVE-2023-36464: PyPDF2/pypdf Infinite Loop Vulnerability ✅ FIXED (2026-02-12) + +**Severity:** Moderate (CVSS: 5.5) +**CVE:** [CVE-2023-36464](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-36464) +**Advisory:** [GHSA-4vvm-4w3v-6mr8](https://github.com/advisories/GHSA-4vvm-4w3v-6mr8) + +**Issue:** Certain versions of PyPDF2 (>=2.2.0, <=3.0.1) and pypdf (prior to 3.9.0) contain a vulnerability where specially crafted PDF files can trigger an infinite loop in `__parse_content_stream`, causing 100% CPU usage and potential denial of service. + +**Impact:** +- **Availability:** High (can block process and consume 100% CPU) +- **Confidentiality:** None +- **Integrity:** None +- **Attack Vector:** Local +- **Privileges Required:** None + +**Remediation:** +- Upgraded from `PyPDF2>=3.0.0` (vulnerable) to `pypdf>=3.9.0` (fixed) +- Updated all imports from `PyPDF2` to `pypdf` across the codebase +- Verified pypdf 6.7.0 installed successfully +- **Files Updated:** + - `requirements.txt` - Updated dependency specification + - `app/tasks/process_document.py` + - `app/tasks/rotate_pdf_pages.py` + - `app/utils/file_splitting.py` + - `app/tasks/embed_metadata_into_pdf.py` + - `app/tasks/process_with_azure_document_intelligence.py` + - `app/views/files.py` + - `app/api/files.py` + - `tests/test_external_integrations.py` + - `tests/test_file_splitting.py` + +**Testing:** All affected modules verified for syntax correctness and basic import functionality. + +**References:** +- [py-pdf/pypdf#1828](https://github.com/py-pdf/pypdf/pull/1828) - Fix implementation +- [py-pdf/pypdf#969](https://github.com/py-pdf/pypdf/pull/969) - Issue introduction + ## Bandit Security Scan Results (2026-02-07) **Scan Summary:** @@ -151,7 +190,7 @@ This document tracks security vulnerabilities found in DocuElevate and their rem - `MAX_UPLOAD_SIZE`: Maximum file upload size in bytes (default: 1GB) - `MAX_SINGLE_FILE_SIZE`: Optional maximum size for a single file chunk - **Automatic page-based PDF splitting** for large PDFs when max_single_file_size is configured - - Splits PDFs at **page boundaries** using PyPDF2, NOT by byte position + - Splits PDFs at **page boundaries** using pypdf, NOT by byte position - Each output file is a structurally valid, complete PDF - No risk of corrupted or broken PDF files - Split files are processed sequentially to prevent overwhelming the system diff --git a/TODO.md b/TODO.md index 4042e3cf..cbea172b 100644 --- a/TODO.md +++ b/TODO.md @@ -188,7 +188,7 @@ As of this update, DocuElevate uses **automated semantic versioning** via `pytho ## 🔧 Technical Debt ### Refactoring Needed -- [ ] Replace PyPDF2 with pypdf (modern maintained fork) +- [x] Replace PyPDF2 with pypdf (modern maintained fork) - ✅ Completed 2026-02-12 - [ ] Migrate from string-based task names to explicit imports in Celery - [ ] Standardize logging format across all modules - [ ] Remove duplicated configuration loading code diff --git a/app/api/files.py b/app/api/files.py index 7da4f6ac..cad47904 100644 --- a/app/api/files.py +++ b/app/api/files.py @@ -460,7 +460,7 @@ def reprocess_with_cloud_ocr(request: Request, file_id: int, db: DbSession): def _extract_text_from_pdf(file_path: str) -> str: """ - Extract text from a PDF file using PyPDF2. + Extract text from a PDF file using pypdf. Args: file_path: Path to the PDF file @@ -468,11 +468,11 @@ def _extract_text_from_pdf(file_path: str) -> str: Returns: Extracted text from all pages """ - import PyPDF2 + import pypdf # Upgraded from PyPDF2 to fix CVE-2023-36464 extracted_text = "" with open(file_path, "rb") as f: - pdf_reader = PyPDF2.PdfReader(f) + pdf_reader = pypdf.PdfReader(f) for page in pdf_reader.pages: extracted_text += page.extract_text() + "\n" return extracted_text diff --git a/app/tasks/embed_metadata_into_pdf.py b/app/tasks/embed_metadata_into_pdf.py index fd8ec487..84dbbbff 100644 --- a/app/tasks/embed_metadata_into_pdf.py +++ b/app/tasks/embed_metadata_into_pdf.py @@ -7,7 +7,7 @@ import shutil import tempfile from pathlib import Path -import PyPDF2 # Replace fitz with PyPDF2 +import pypdf # Upgraded from PyPDF2 to fix CVE-2023-36464 # Import the shared Celery instance from app.celery_app import celery @@ -128,8 +128,8 @@ def embed_metadata_into_pdf(self, local_file_path: str, extracted_text: str, met # Open the PDF and modify metadata with open(processed_file, "rb") as file: - pdf_reader = PyPDF2.PdfReader(file) - pdf_writer = PyPDF2.PdfWriter() + pdf_reader = pypdf.PdfReader(file) + pdf_writer = pypdf.PdfWriter() # Copy all pages from the reader to the writer for page in pdf_reader.pages: diff --git a/app/tasks/process_document.py b/app/tasks/process_document.py index 44c0c179..aadc8e33 100644 --- a/app/tasks/process_document.py +++ b/app/tasks/process_document.py @@ -6,8 +6,8 @@ import os import shutil import uuid -import PyPDF2 # Replace fitz with PyPDF2 -from PyPDF2.errors import PdfReadError +import pypdf # Upgraded from PyPDF2 to fix CVE-2023-36464 +from pypdf.errors import PdfReadError from app.celery_app import celery from app.config import settings @@ -343,7 +343,7 @@ def process_document(self, original_local_file: str, original_filename: str = No ) try: with open(new_local_path, "rb") as file: - pdf_reader = PyPDF2.PdfReader(file) + pdf_reader = pypdf.PdfReader(file) has_text = False for page in pdf_reader.pages: if page.extract_text().strip(): @@ -391,7 +391,7 @@ def process_document(self, original_local_file: str, original_filename: str = No ) extracted_text = "" with open(new_local_path, "rb") as file: - pdf_reader = PyPDF2.PdfReader(file) + pdf_reader = pypdf.PdfReader(file) for page in pdf_reader.pages: extracted_text += page.extract_text() + "\n" diff --git a/app/tasks/process_with_azure_document_intelligence.py b/app/tasks/process_with_azure_document_intelligence.py index 9eab8d39..63aa0282 100644 --- a/app/tasks/process_with_azure_document_intelligence.py +++ b/app/tasks/process_with_azure_document_intelligence.py @@ -2,7 +2,7 @@ import logging import os import azure.core.exceptions -import PyPDF2 +import pypdf # Upgraded from PyPDF2 to fix CVE-2023-36464 from azure.ai.documentintelligence import DocumentIntelligenceClient from azure.ai.documentintelligence.models import AnalyzeOutputOption, AnalyzeResult from azure.core.credentials import AzureKeyCredential @@ -39,7 +39,7 @@ def get_pdf_page_count(file_path): """Get the number of pages in a PDF file.""" try: with open(file_path, "rb") as file: - pdf_reader = PyPDF2.PdfReader(file) + pdf_reader = pypdf.PdfReader(file) return len(pdf_reader.pages) except Exception as e: logger.error(f"Error getting PDF page count: {e}") diff --git a/app/tasks/rotate_pdf_pages.py b/app/tasks/rotate_pdf_pages.py index 4de2f55e..9b21435d 100644 --- a/app/tasks/rotate_pdf_pages.py +++ b/app/tasks/rotate_pdf_pages.py @@ -2,7 +2,7 @@ import json import logging import os -import PyPDF2 +import pypdf # Upgraded from PyPDF2 to fix CVE-2023-36464 from app.celery_app import celery from app.config import settings @@ -21,7 +21,7 @@ def determine_rotation_angle(detected_angle): detected_angle: The angle detected by Azure Document Intelligence Returns: - int: The angle to rotate the page in PyPDF2 (must be multiple of 90 degrees) + int: The angle to rotate the page in pypdf (must be multiple of 90 degrees) """ # Normalize angle to be between 0 and 360 normalized_angle = detected_angle % 360 @@ -35,15 +35,15 @@ def determine_rotation_angle(detected_angle): # For angles close to 90, 180, or 270 degrees (±5°), round to nearest 90° increment for target in [90, 180, 270]: if abs(normalized_angle - target) < 5: - # PyPDF2 uses clockwise rotation, so we need to use the complementary angle + # pypdf uses clockwise rotation, so we need to use the complementary angle rotation_value = (360 - target) % 360 logger.info(f"Detected angle {detected_angle}° is close to {target}°, will rotate by {rotation_value}°") return rotation_value # For other significant angles, round to nearest 90° increment - # (PyPDF2 only supports rotations in 90-degree increments) + # (pypdf only supports rotations in 90-degree increments) closest_90_multiple = round(normalized_angle / 90) * 90 - # Convert to PyPDF2 rotation value (clockwise) + # Convert to pypdf rotation value (clockwise) rotation_value = (360 - closest_90_multiple) % 360 logger.info(f"Detected angle {detected_angle}° rounded to {closest_90_multiple}°, will rotate by {rotation_value}°") return rotation_value @@ -110,8 +110,8 @@ def rotate_pdf_pages(self, filename: str, extracted_text: str, rotation_data=Non # Load the PDF with open(pdf_path, "rb") as file: - pdf_reader = PyPDF2.PdfReader(file) - pdf_writer = PyPDF2.PdfWriter() + pdf_reader = pypdf.PdfReader(file) + pdf_writer = pypdf.PdfWriter() # Process each page for page_idx in range(len(pdf_reader.pages)): @@ -123,7 +123,7 @@ def rotate_pdf_pages(self, filename: str, extracted_text: str, rotation_data=Non rotation_angle = determine_rotation_angle(detected_angle) if rotation_angle > 0: - # PyPDF2 uses clockwise rotation in 90-degree increments + # pypdf uses clockwise rotation in 90-degree increments page.rotate(rotation_angle) logger.info( f"[{task_id}] Page {page_idx+1} rotated by {rotation_angle}° " diff --git a/app/utils/file_splitting.py b/app/utils/file_splitting.py index ea501769..f4d08d37 100644 --- a/app/utils/file_splitting.py +++ b/app/utils/file_splitting.py @@ -10,7 +10,7 @@ import logging import os from typing import List, Optional -from PyPDF2 import PdfReader, PdfWriter +from pypdf import PdfReader, PdfWriter # Upgraded from PyPDF2 to fix CVE-2023-36464 logger = logging.getLogger(__name__) diff --git a/app/views/files.py b/app/views/files.py index 862bb2e3..9ec97380 100644 --- a/app/views/files.py +++ b/app/views/files.py @@ -492,8 +492,8 @@ def get_original_text(request: Request, file_id: int, db: Session = Depends(get_ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Original file not found on disk") try: - # Extract text from PDF using PyPDF2 - from PyPDF2 import PdfReader + # Extract text from PDF using pypdf + from pypdf import PdfReader # Upgraded from PyPDF2 to fix CVE-2023-36464 reader = PdfReader(file_record.original_file_path) text = "" @@ -532,8 +532,8 @@ def get_processed_text(request: Request, file_id: int, db: Session = Depends(get raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Processed file not found on disk") try: - # Extract text from PDF using PyPDF2 - from PyPDF2 import PdfReader + # Extract text from PDF using pypdf + from pypdf import PdfReader # Upgraded from PyPDF2 to fix CVE-2023-36464 reader = PdfReader(file_record.processed_file_path) text = "" diff --git a/docs/ConfigurationGuide.md b/docs/ConfigurationGuide.md index 4aaf381d..cdec2742 100644 --- a/docs/ConfigurationGuide.md +++ b/docs/ConfigurationGuide.md @@ -58,7 +58,7 @@ MAX_SINGLE_FILE_SIZE=524288000 **File Splitting Behavior:** - When `MAX_SINGLE_FILE_SIZE` is configured and a PDF exceeds this size, it is automatically split into smaller chunks - **IMPORTANT:** Splitting is done at **PAGE BOUNDARIES**, not by byte position - - Uses PyPDF2 to properly parse PDF structure + - Uses pypdf to properly parse PDF structure - Each output file is a complete, valid PDF containing whole pages - No risk of corrupted or broken PDF files - Pages are distributed across output files to stay under size limit diff --git a/docs/FileDetailPageLayout.md b/docs/FileDetailPageLayout.md index c7b932db..6ebefb89 100644 --- a/docs/FileDetailPageLayout.md +++ b/docs/FileDetailPageLayout.md @@ -15,7 +15,7 @@ POST /api/files/{id}/retry-subtask → Retry specific task ## Text Extraction Text extraction is performed **on-demand** when the user clicks "View Extracted Text": -- Uses PyPDF2 to extract text from PDF files in real-time +- Uses pypdf to extract text from PDF files in real-time - Returns JSON: `{"text": "...", "page_count": 3}` - Client-side caching prevents re-extraction on subsequent views - Loading indicator shown during extraction diff --git a/docs/UserGuide.md b/docs/UserGuide.md index a0de2635..64d2c241 100644 --- a/docs/UserGuide.md +++ b/docs/UserGuide.md @@ -182,7 +182,7 @@ Both previews support: - Full text extraction viewing via modal overlays **View Extracted Text**: Each preview includes a button to view the complete extracted text in a fullscreen modal. When you click this button: -- The system extracts text from the PDF file on-demand using PyPDF2 +- The system extracts text from the PDF file on-demand using pypdf - A loading indicator shows while extraction is in progress - The extracted text is displayed in a scrollable, copy-friendly format - The text is cached so subsequent views load instantly diff --git a/docs/archive/ANALYSIS_SUMMARY.md b/docs/archive/ANALYSIS_SUMMARY.md index e4512bae..89b55baa 100644 --- a/docs/archive/ANALYSIS_SUMMARY.md +++ b/docs/archive/ANALYSIS_SUMMARY.md @@ -230,7 +230,7 @@ httpx>=0.26.0 ### Medium Priority (Next Month) - [ ] Fix Pydantic V1 → V2 migration warnings -- [ ] Migrate from PyPDF2 to pypdf (modern fork) +- [x] Migrate from PyPDF2 to pypdf (modern fork) - ✅ Completed 2026-02-12 - [ ] Consolidate storage provider code - [ ] Add API pagination - [ ] Implement retry logic for Celery tasks diff --git a/frontend/templates/attribution.html b/frontend/templates/attribution.html index 8d7e9103..47016dcb 100644 --- a/frontend/templates/attribution.html +++ b/frontend/templates/attribution.html @@ -67,9 +67,9 @@ https://github.com/openai/openai-python