Add comprehensive planning and agentic coding documentation

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
2026-02-06 21:59:48 +00:00
parent f0f48b39a9
commit ccf4cd8c8f
5 changed files with 1530 additions and 0 deletions
@@ -0,0 +1,661 @@
+# Agentic Coding Guide for DocuElevate
+
+**Version:** 1.0  
+**Last Updated:** 2026-02-06
+
+This guide helps AI coding agents work effectively with the DocuElevate codebase. It provides context, conventions, and best practices for autonomous code contributions.
+
+---
+
+## 🎯 Project Overview
+
+### What is DocuElevate?
+DocuElevate is an intelligent document processing system that:
+- Ingests documents from multiple sources (email, web upload, API)
+- Processes documents (OCR, PDF conversion, metadata extraction)
+- Stores documents in various cloud storage providers
+- Uses AI (OpenAI, Azure) for intelligent document classification and metadata extraction
+
+### Tech Stack
+```
+Backend:  FastAPI, SQLAlchemy, Celery, Redis
+Frontend: Jinja2 templates, Tailwind CSS
+AI/ML:    OpenAI API, Azure Document Intelligence
+Storage:  Dropbox, Google Drive, OneDrive, S3, Nextcloud, Paperless-NGX
+Auth:     Authentik (OAuth2), Basic Auth
+Infra:    Docker, Docker Compose, Alembic (migrations)
+```
+
+### Key Directories
+```
+DocuElevate/
+├── app/
+│   ├── api/          # REST API endpoints
+│   ├── tasks/        # Celery background tasks
+│   ├── routes/       # Deprecated - being migrated to api/
+│   ├── views/        # UI routes and templates
+│   ├── utils/        # Utility functions
+│   ├── config.py     # Configuration (Pydantic Settings)
+│   ├── database.py   # SQLAlchemy setup
+│   ├── models.py     # Database models
+│   ├── main.py       # FastAPI app initialization
+│   └── auth.py       # Authentication logic
+├── frontend/
+│   ├── static/       # CSS, JS, images
+│   └── templates/    # Jinja2 HTML templates
+├── tests/           # Pytest test suite
+├── docs/            # User documentation
+├── migrations/      # Alembic database migrations
+└── docker/          # Docker configuration
+```
+
+---
+
+## 🤖 Agent Guidelines
+
+### Before Making Changes
+
+1. **Understand the Context**
+   - Read relevant documentation in `docs/`
+   - Check `TODO.md` for current priorities
+   - Review `SECURITY_AUDIT.md` for security considerations
+   - Check `ROADMAP.md` for feature direction
+
+2. **Check Existing Patterns**
+   - Look at similar existing code first
+   - Follow the established patterns in the codebase
+   - Don't introduce new patterns without good reason
+
+3. **Identify Dependencies**
+   - Check if your change affects multiple modules
+   - Ensure you understand the Celery task flow
+   - Consider impact on database schema
+
+### Code Conventions
+
+#### Python Style
+```python
+# Use Black formatting (line length: 120)
+# Use type hints
+def process_document(file_path: str, metadata: Dict[str, Any]) -> DocumentMetadata:
+    """
+    Process a document and extract metadata.
+    
+    Args:
+        file_path: Absolute path to the document file
+        metadata: Additional metadata to include
+        
+    Returns:
+        DocumentMetadata object with extracted information
+        
+    Raises:
+        FileNotFoundError: If file doesn't exist
+        ProcessingError: If processing fails
+    """
+    pass
+
+# Use descriptive variable names
+user_document_path = Path("/workdir/documents/invoice.pdf")
+ocr_result = extract_text_from_pdf(user_document_path)
+
+# Prefer explicit over implicit
+if storage_provider == "dropbox":
+    upload_to_dropbox(file_path, metadata)
+elif storage_provider == "google_drive":
+    upload_to_google_drive(file_path, metadata)
+else:
+    raise ValueError(f"Unknown storage provider: {storage_provider}")
+```
+
+#### Configuration
+```python
+# Always use settings from config.py
+from app.config import settings
+
+# Good
+api_key = settings.openai_api_key
+
+# Bad - never hardcode
+api_key = "sk-abc123..."
+
+# Check if optional services are configured
+if settings.dropbox_app_key:
+    # Dropbox is configured
+    upload_to_dropbox()
+```
+
+#### Error Handling
+```python
+# Use appropriate exception types
+from fastapi import HTTPException, status
+
+# API endpoints should return HTTP errors
+@router.get("/files/{file_id}")
+async def get_file(file_id: int):
+    file = get_file_from_db(file_id)
+    if not file:
+        raise HTTPException(
+            status_code=status.HTTP_404_NOT_FOUND,
+            detail=f"File with ID {file_id} not found"
+        )
+    return file
+
+# Tasks should log and handle errors gracefully
+@celery_app.task(bind=True, max_retries=3)
+def process_document_task(self, file_path: str):
+    try:
+        result = process_document(file_path)
+        return result
+    except TemporaryError as e:
+        logger.warning(f"Temporary error processing {file_path}: {e}")
+        raise self.retry(exc=e, countdown=60)
+    except PermanentError as e:
+        logger.error(f"Permanent error processing {file_path}: {e}")
+        # Don't retry permanent errors
+        return {"error": str(e)}
+```
+
+#### Testing
+```python
+# Mark tests appropriately
+@pytest.mark.unit
+def test_hash_file():
+    """Unit test for file hashing utility."""
+    pass
+
+@pytest.mark.integration
+def test_upload_api_endpoint(client):
+    """Integration test for upload API."""
+    pass
+
+@pytest.mark.requires_external
+@pytest.mark.skip(reason="Requires OpenAI API key")
+def test_openai_metadata_extraction():
+    """Test actual OpenAI integration."""
+    pass
+
+# Use fixtures for common setup
+def test_document_processing(sample_pdf_path, db_session):
+    """Test uses fixtures from conftest.py"""
+    pass
+```
+
+---
+
+## 📝 Common Tasks
+
+### Adding a New API Endpoint
+
+1. Create endpoint in `app/api/`:
+```python
+# app/api/my_feature.py
+from fastapi import APIRouter, HTTPException
+from app.database import get_db
+from app.models import MyModel
+
+router = APIRouter(prefix="/api/my-feature", tags=["my-feature"])
+
+@router.get("/")
+async def list_items(db=Depends(get_db)):
+    """List all items."""
+    items = db.query(MyModel).all()
+    return items
+```
+
+2. Register router in `app/api/__init__.py`:
+```python
+from app.api import my_feature
+
+router.include_router(my_feature.router)
+```
+
+3. Add tests in `tests/test_api_my_feature.py`
+
+### Adding a New Celery Task
+
+1. Create task in `app/tasks/`:
+```python
+# app/tasks/my_task.py
+from app.celery_app import celery_app
+import logging
+
+logger = logging.getLogger(__name__)
+
+@celery_app.task(bind=True, max_retries=3)
+def my_background_task(self, param: str):
+    """
+    Description of what this task does.
+    
+    Args:
+        param: Description of parameter
+    """
+    try:
+        logger.info(f"Processing task with param: {param}")
+        # Task logic here
+        return {"status": "success"}
+    except Exception as e:
+        logger.error(f"Task failed: {e}")
+        raise self.retry(exc=e, countdown=60)
+```
+
+2. Import in `app/tasks/__init__.py`
+3. Add tests in `tests/test_tasks.py`
+
+### Adding a Database Model
+
+1. Define model in `app/models.py`:
+```python
+class MyModel(Base):
+    __tablename__ = "my_table"
+    
+    id = Column(Integer, primary_key=True, index=True)
+    name = Column(String, nullable=False)
+    created_at = Column(DateTime, default=datetime.utcnow)
+```
+
+2. Create migration:
+```bash
+cd /path/to/DocuElevate
+alembic revision --autogenerate -m "Add MyModel table"
+alembic upgrade head
+```
+
+3. Add model to tests fixtures
+
+### Adding a Storage Provider
+
+1. Create provider module in `app/tasks/storage/`:
+```python
+# app/tasks/storage/my_provider.py
+from app.config import settings
+import logging
+
+logger = logging.getLogger(__name__)
+
+def upload_to_my_provider(file_path: str, metadata: dict) -> str:
+    """
+    Upload file to My Provider.
+    
+    Args:
+        file_path: Local path to file
+        metadata: Document metadata
+        
+    Returns:
+        URL or ID of uploaded file
+        
+    Raises:
+        ProviderError: If upload fails
+    """
+    if not settings.my_provider_api_key:
+        raise ValueError("MY_PROVIDER_API_KEY not configured")
+    
+    # Implementation
+    pass
+```
+
+2. Add configuration to `app/config.py`:
+```python
+class Settings(BaseSettings):
+    # ... existing settings ...
+    my_provider_api_key: Optional[str] = None
+    my_provider_endpoint: Optional[str] = None
+```
+
+3. Add to `.env.demo`:
+```bash
+# My Provider
+MY_PROVIDER_API_KEY=your_api_key_here
+MY_PROVIDER_ENDPOINT=https://api.myprovider.com
+```
+
+4. Add validator in `app/utils/config_validator/`
+5. Add tests with mocked API calls
+
+---
+
+## 🔒 Security Best Practices
+
+### What to NEVER Do
+- ❌ Hardcode API keys, passwords, or secrets
+- ❌ Log sensitive data (passwords, tokens, API keys)
+- ❌ Accept unsanitized user input for file paths
+- ❌ Disable security features without documentation
+- ❌ Commit `.env` files or credentials
+
+### What to ALWAYS Do
+- ✅ Use `settings` from `app/config.py` for all configuration
+- ✅ Validate and sanitize all user inputs
+- ✅ Use parameterized database queries (SQLAlchemy handles this)
+- ✅ Check file paths for directory traversal (`Path.resolve()`)
+- ✅ Use appropriate HTTP status codes (401, 403, 404, etc.)
+- ✅ Log security-relevant events
+- ✅ Add rate limiting for sensitive endpoints
+- ✅ Use HTTPS in production (documented in deployment guide)
+
+### Input Validation Example
+```python
+from pathlib import Path
+from fastapi import HTTPException, status
+
+def validate_file_path(file_path: str, base_dir: str = "/workdir") -> Path:
+    """Validate file path is within allowed directory."""
+    try:
+        path = Path(file_path).resolve()
+        base = Path(base_dir).resolve()
+        
+        # Ensure path is within base directory
+        if not path.is_relative_to(base):
+            raise ValueError("Path outside allowed directory")
+        
+        return path
+    except Exception as e:
+        raise HTTPException(
+            status_code=status.HTTP_400_BAD_REQUEST,
+            detail=f"Invalid file path: {e}"
+        )
+```
+
+---
+
+## 🧪 Testing Strategy
+
+### Test Coverage Goals
+- **Target:** 80% overall coverage
+- **Critical modules:** 90%+ (auth, config, database)
+- **Tasks:** 70%+ (complex to test with external services)
+- **API endpoints:** 85%+
+
+### Test Types
+```python
+# Unit tests - fast, isolated, no external dependencies
+@pytest.mark.unit
+def test_hash_file_empty(tmp_path):
+    """Test hashing an empty file."""
+    file = tmp_path / "empty.txt"
+    file.write_text("")
+    assert hash_file(str(file)) == "expected_hash"
+
+# Integration tests - test multiple components together
+@pytest.mark.integration
+def test_upload_and_process(client, sample_pdf):
+    """Test full upload and processing flow."""
+    response = client.post("/api/upload", files={"file": sample_pdf})
+    assert response.status_code == 200
+
+# External service tests - skipped by default
+@pytest.mark.requires_external
+@pytest.mark.skipif(not os.getenv("OPENAI_API_KEY"), reason="No API key")
+def test_real_openai_extraction():
+    """Test actual OpenAI API (skipped in CI)."""
+    pass
+```
+
+### Running Tests
+```bash
+# All tests
+pytest
+
+# Specific category
+pytest -m unit
+pytest -m integration
+
+# With coverage
+pytest --cov=app --cov-report=html
+
+# Specific file
+pytest tests/test_api.py -v
+
+# Skip external services
+pytest -m "not requires_external"
+```
+
+---
+
+## 🚀 Performance Considerations
+
+### Async/Await
+- FastAPI endpoints are async by default
+- Use `async def` for I/O-bound operations
+- Use regular `def` for CPU-bound operations
+
+```python
+# Good - async for I/O
+@router.get("/files")
+async def list_files(db: Session = Depends(get_db)):
+    files = db.query(FileRecord).all()
+    return files
+
+# Also good - sync for CPU-heavy
+@router.post("/hash")
+def hash_large_file(file: UploadFile):
+    return compute_hash(file.file.read())
+```
+
+### Database Queries
+```python
+# Good - single query with join
+files = db.query(FileRecord).options(
+    joinedload(FileRecord.metadata)
+).filter(FileRecord.user_id == user_id).all()
+
+# Bad - N+1 queries
+files = db.query(FileRecord).filter(FileRecord.user_id == user_id).all()
+for file in files:
+    metadata = file.metadata  # Triggers separate query each time
+```
+
+### Celery Tasks
+```python
+# Long-running tasks should update progress
+@celery_app.task(bind=True)
+def process_large_batch(self, file_ids: List[int]):
+    total = len(file_ids)
+    for i, file_id in enumerate(file_ids):
+        process_file(file_id)
+        self.update_state(
+            state='PROGRESS',
+            meta={'current': i + 1, 'total': total}
+        )
+```
+
+---
+
+## 📚 Documentation Requirements
+
+### Code Documentation
+```python
+def complex_function(param1: str, param2: int = 10) -> Dict[str, Any]:
+    """
+    One-line summary of what the function does.
+    
+    More detailed explanation if needed. Can span multiple
+    lines and include examples.
+    
+    Args:
+        param1: Description of param1
+        param2: Description of param2, defaults to 10
+        
+    Returns:
+        Dictionary containing:
+            - key1: Description
+            - key2: Description
+            
+    Raises:
+        ValueError: If param1 is empty
+        FileNotFoundError: If file doesn't exist
+        
+    Examples:
+        >>> result = complex_function("test", 5)
+        >>> print(result['key1'])
+        'value'
+    """
+    pass
+```
+
+### API Documentation
+- Use FastAPI's automatic OpenAPI generation
+- Add descriptions to endpoints
+- Document request/response models
+- Include example requests/responses
+
+```python
+@router.post(
+    "/upload",
+    response_model=UploadResponse,
+    status_code=status.HTTP_201_CREATED,
+    summary="Upload a document",
+    description="Upload a document for processing. Supports PDF, images, and Office documents.",
+    responses={
+        201: {"description": "Document uploaded successfully"},
+        400: {"description": "Invalid file format"},
+        413: {"description": "File too large"},
+    }
+)
+async def upload_document(
+    file: UploadFile = File(..., description="Document file to upload"),
+    tags: List[str] = Query([], description="Optional tags for the document"),
+):
+    """Upload endpoint implementation."""
+    pass
+```
+
+---
+
+## 🐛 Debugging
+
+### Logging
+```python
+import logging
+
+logger = logging.getLogger(__name__)
+
+# Use appropriate log levels
+logger.debug("Detailed information for debugging")
+logger.info("General information about operation")
+logger.warning("Warning about potential issue")
+logger.error("Error that needs attention")
+logger.critical("Critical error that needs immediate attention")
+
+# Include context in logs
+logger.info(f"Processing document: {file_id}, user: {user_id}")
+
+# Don't log sensitive data
+logger.info(f"User authenticated")  # Good
+logger.info(f"Password: {password}")  # BAD!
+```
+
+### Common Issues
+
+1. **Import Errors**
+   - Check if module is in `__init__.py`
+   - Verify Python path includes project root
+   - Look for circular imports
+
+2. **Database Issues**
+   - Check if migrations are up to date: `alembic upgrade head`
+   - Verify DATABASE_URL is set correctly
+   - Check if tables exist: `sqlite3 app/database.db .schema`
+
+3. **Celery Issues**
+   - Verify Redis is running: `redis-cli ping`
+   - Check Celery worker logs
+   - Ensure tasks are imported in `celery_worker.py`
+
+4. **Test Failures**
+   - Check if test database is clean (use fixtures)
+   - Verify environment variables are set in `conftest.py`
+   - Run single test to isolate issue: `pytest tests/test_file.py::test_name -v`
+
+---
+
+## 🔄 Git Workflow
+
+### Branch Names
+- `feature/description` - New features
+- `bugfix/description` - Bug fixes
+- `hotfix/description` - Urgent production fixes
+- `refactor/description` - Code refactoring
+- `docs/description` - Documentation updates
+
+### Commit Messages
+```
+type(scope): Short description (max 72 chars)
+
+Longer description if needed. Explain:
+- What changed
+- Why it changed
+- Any breaking changes
+
+Fixes #123
+```
+
+Types: `feat`, `fix`, `docs`, `style`, `refactor`, `test`, `chore`
+
+### Pull Requests
+1. Create PR with descriptive title
+2. Fill out PR template
+3. Link related issues
+4. Ensure CI passes
+5. Request reviews
+6. Address feedback
+7. Squash merge when approved
+
+---
+
+## ✅ Pre-commit Checklist
+
+Before submitting code:
+
+- [ ] Code follows style guide (Black formatted)
+- [ ] All tests pass (`pytest`)
+- [ ] New code has tests
+- [ ] Coverage doesn't decrease
+- [ ] Documentation updated if needed
+- [ ] No secrets or credentials in code
+- [ ] Linting passes (`flake8`, `pylint`)
+- [ ] Type hints added (`mypy` clean)
+- [ ] CHANGELOG.md updated (if user-facing)
+- [ ] Security scan passed (`bandit`)
+
+Run full check:
+```bash
+pytest --cov=app
+black app/ tests/
+flake8 app/ --max-line-length=120
+mypy app/
+bandit -r app/
+```
+
+---
+
+## 🤝 Agent Collaboration
+
+### When to Ask for Help
+- Breaking changes needed
+- Unsure about architecture decision
+- Security implications unclear
+- Performance impact unknown
+- Tests consistently failing
+
+### How to Document Changes
+1. Update relevant documentation
+2. Add comments for complex logic
+3. Update TODO.md if introducing tech debt
+4. Note breaking changes in commit message
+5. Update API documentation if endpoints changed
+
+---
+
+## 📞 Resources
+
+- **Main README:** [README.md](README.md)
+- **API Docs:** http://localhost:8000/docs (when running)
+- **User Guide:** [docs/UserGuide.md](docs/UserGuide.md)
+- **Deployment:** [docs/DeploymentGuide.md](docs/DeploymentGuide.md)
+- **Troubleshooting:** [docs/Troubleshooting.md](docs/Troubleshooting.md)
+- **GitHub Issues:** Track bugs and features
+- **GitHub Discussions:** Questions and community
+
+---
+
+*This guide is a living document. Improvements welcome via PR!*