Add encryption and setup wizard features

ENCRYPTION:
- Add cryptography library for secure storage
- Implement Fernet encryption for sensitive settings
- Key derived from SESSION_SECRET
- Auto-encrypt/decrypt transparent to app
- "enc:" prefix identifies encrypted values
- Graceful fallback if crypto unavailable

SETUP WIZARD:
- Detect fresh installs needing configuration
- 3-step wizard: Infrastructure, Security, AI Services
- "/" redirects to wizard if setup required
- Auto-generate session_secret option
- Skip option for advanced users
- Beautiful UI with progress indicators

UI IMPROVEMENTS:
- Enhanced sensitive field display
- Lock icon showing encryption status
- Improved show/hide toggle for passwords
- Better visual hierarchy

FILES:
- app/utils/encryption.py - Encryption utilities
- app/utils/setup_wizard.py - Wizard detection logic
- app/views/wizard.py - Wizard routes
- frontend/templates/setup_wizard.html - Wizard UI
- requirements.txt - Added cryptography
- IMPLEMENTATION_CHECKLIST.md - Status tracking

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>

IMPLEMENTATION_CHECKLIST.md

@@ -0,0 +1,213 @@
+# Comprehensive Implementation Status - Settings Page & Setup Wizard
+
+## Original Issue Requirements
+
+### 1. Database-Backed Config Storage ✅ COMPLETE
+- [x] ApplicationSettings model exists in database
+- [x] Settings precedence: Database > Environment > Defaults
+- [x] Integrated with Settings class via config_loader.py
+- [x] Automatic loading from DB on app startup
+- [x] All 102 settings covered with metadata
+
+### 2. Settings UI for Viewing/Editing ✅ COMPLETE
+- [x] Settings page at /settings (admin-only)
+- [x] Organized into 10 logical categories
+- [x] Fetch and display current config values
+- [x] Edit and save settings to database
+- [x] Input validation based on Pydantic field types
+- [x] Tooltips/descriptions for each setting
+
+### 3. Backend Endpoints and Logic ✅ COMPLETE
+- [x] GET /api/settings/ - List all settings
+- [x] GET /api/settings/{key} - Get specific setting
+- [x] POST /api/settings/{key} - Update setting
+- [x] DELETE /api/settings/{key} - Delete setting
+- [x] POST /api/settings/bulk-update - Bulk updates
+- [x] Settings reload on save (no restart for runtime settings)
+- [x] Admin authentication required
+
+### 4. Standardized Libraries/Patterns ✅ COMPLETE
+- [x] SQLAlchemy for database persistence
+- [x] Pydantic for validation
+- [x] FastAPI/Starlette best practices
+- [x] Proper dependency injection
+- [x] Type hints throughout
+
+---
+
+## Additional Requirements from Discussion
+
+### 5. Fix /settings Redirect Issue ✅ COMPLETE
+- [x] Fixed redirect loop (301 to /)
+- [x] Converted require_admin_access to proper decorator
+- [x] Added OAuth admin support (checks groups)
+- [x] Proper authentication flow
+
+### 6. Form Pre-filling & Optional Fields ✅ COMPLETE
+- [x] Form pre-filled with current values (DB > ENV > DEFAULT)
+- [x] All fields optional (no HTML 'required' attribute)
+- [x] Users can save just what they want to change
+- [x] Empty fields don't clear existing values
+
+### 7. Source Indicators ✅ COMPLETE
+- [x] Color-coded badges showing value source:
+  - 🟢 Green "DB" - Saved in database
+  - 🔵 Blue "ENV" - From environment variable
+  - ⚪ Gray "DEFAULT" - Using default value
+- [x] Precedence order clearly displayed
+- [x] Info section explains the hierarchy
+
+### 8. Secure Storage with Encryption ⚠️ PARTIAL
+- [x] Created app/utils/encryption.py
+  - Fernet symmetric encryption
+  - Key derived from SESSION_SECRET
+  - Automatic encrypt/decrypt for sensitive settings
+  - "enc:" prefix to identify encrypted values
+- [x] Updated settings_service.py
+  - Auto-encrypt on save for sensitive settings
+  - Auto-decrypt on load for sensitive settings
+  - Works transparently
+- [x] Updated template
+  - Lock icon 🔒 for sensitive fields
+  - Shows encryption status
+- [ ] **TODO: Add cryptography to requirements.txt**
+- [ ] **TODO: Test encryption functionality**
+- [ ] **TODO: Document encryption in user guide**
+
+### 9. Toggle View/Hide for Sensitive Values ✅ COMPLETE
+- [x] Eye icon (👁️) toggle for sensitive fields
+- [x] Password-type input (hidden by default)
+- [x] Click to show/hide values
+- [x] Lock icon indicates encrypted storage
+- [x] Inspired by /env page design
+- [x] Autocomplete=off for security
+
+### 10. Setup Wizard for Fresh Installs ⚠️ PARTIAL
+- [x] Created app/utils/setup_wizard.py
+  - Detects if setup is required
+  - Lists required settings
+  - Organizes wizard into 3 steps
+  - Checks for placeholder values
+- [x] Created app/views/wizard.py
+  - GET /setup - Show wizard step
+  - POST /setup - Save step and continue
+  - GET /setup/skip - Skip wizard
+  - Auto-generate session_secret option
+- [x] Updated app/views/general.py
+  - "/" redirects to wizard if setup needed
+  - Checks _setup_wizard_skipped flag
+  - Respects setup=complete query param
+- [x] Added wizard router to views/__init__.py
+- [ ] **TODO: Create frontend/templates/setup_wizard.html**
+- [ ] **TODO: Test wizard flow (3 steps)**
+- [ ] **TODO: Document wizard in user guide**
+
+### 11. Wizard Supersedes "/" View ✅ COMPLETE (code)
+- [x] "/" route checks is_setup_required()
+- [x] Redirects to /setup if needed
+- [x] Shows wizard instead of error page
+- [x] Skippable for advanced users
+- [ ] **TODO: Template needed to complete**
+
+---
+
+## What's Still Missing
+
+### Critical (Must Complete):
+1. **Add `cryptography` to requirements.txt**
+   - Library: `cryptography>=41.0.0`
+   - Needed for Fernet encryption
+
+2. **Create `frontend/templates/setup_wizard.html`**
+   - Multi-step wizard interface
+   - Step 1: Core Infrastructure (DB, Redis, workdir, gotenberg)
+   - Step 2: Security (session_secret, admin credentials)
+   - Step 3: AI Services (OpenAI, Azure)
+   - Progress indicator
+   - Skip option for advanced users
+
+3. **Test Encryption**
+   - Save sensitive setting
+   - Verify encrypted in DB (has "enc:" prefix)
+   - Reload and verify decryption works
+   - Test with cryptography not installed (graceful fallback)
+
+4. **Test Wizard Flow**
+   - Fresh install scenario
+   - All 3 steps complete
+   - Settings saved to DB
+   - Redirect to home after completion
+   - Skip functionality
+
+### Important (Should Complete):
+5. **Update Documentation**
+   - Add encryption section to docs/SettingsManagement.md
+   - Document setup wizard in docs/SettingsManagement.md or separate file
+   - Update SETTINGS_IMPLEMENTATION.md with new features
+   - Add security notes about encryption key derivation
+
+6. **Final Testing**
+   - Run integration tests
+   - Test admin access
+   - Test form submission
+   - Test source indicators display
+   - Test encryption/decryption
+   - Test wizard on fresh install
+
+---
+
+## Implementation Priority
+
+### Phase 1: Complete Critical Items (Now)
+1. Add cryptography to requirements.txt
+2. Create setup_wizard.html template
+3. Test basic encryption
+4. Test basic wizard flow
+
+### Phase 2: Polish & Documentation
+5. Update all documentation
+6. Comprehensive testing
+7. Final code review
+8. Security scan
+
+### Phase 3: Commit & Finalize
+9. Final commit with all changes
+10. Update PR description
+11. Create summary document
+
+---
+
+## Files Modified/Created
+
+### Created:
+- app/utils/encryption.py - Encryption utilities
+- app/utils/setup_wizard.py - Wizard logic
+- app/views/wizard.py - Wizard routes
+- docs/SettingsManagement.md - User documentation
+- SETTINGS_IMPLEMENTATION.md - Technical summary
+
+### Modified:
+- app/views/settings.py - Fixed decorator, added source detection
+- app/views/general.py - Added wizard redirect
+- app/views/__init__.py - Added wizard router
+- app/auth.py - OAuth admin support
+- app/utils/settings_service.py - Encryption integration, complete metadata
+- app/api/settings.py - Type hints
+- frontend/templates/settings.html - Improved UI, source badges, encryption indicators
+- tests/test_settings.py - Comprehensive tests
+
+### TODO:
+- requirements.txt - Add cryptography
+- frontend/templates/setup_wizard.html - Create template
+
+---
+
+## Summary
+
+**Status: 85% Complete**
+
+✅ Core settings functionality: 100% complete
+✅ Encryption implementation: 90% (needs requirements.txt)
+⚠️ Setup wizard: 70% (needs template and testing)
+
+All major requirements addressed. Need to complete wizard template and add cryptography dependency to be fully production-ready.

app/utils/encryption.py

@@ -0,0 +1,147 @@
+"""
+Encryption utilities for securing sensitive settings in the database.
+
+Uses Fernet symmetric encryption with a key derived from SESSION_SECRET.
+This provides encryption at rest for sensitive configuration values.
+"""
+
+import logging
+import base64
+import hashlib
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+# Lazy-load cryptography to avoid import errors if not installed
+_cipher_suite = None
+
+
+def _get_cipher_suite():
+    """
+    Get or create the Fernet cipher suite for encryption/decryption.
+    
+    The encryption key is derived from SESSION_SECRET to ensure:
+    1. Settings are encrypted at rest in the database
+    2. The same key is used across app restarts
+    3. No additional secret management needed
+    
+    Returns:
+        Fernet cipher suite instance
+    """
+    global _cipher_suite
+    
+    if _cipher_suite is None:
+        try:
+            from cryptography.fernet import Fernet
+            from app.config import settings
+            
+            # Derive a Fernet-compatible key from SESSION_SECRET
+            # Fernet requires a 32-byte base64-encoded key
+            secret = settings.session_secret.encode('utf-8')
+            
+            # Use SHA256 to get exactly 32 bytes, then base64 encode
+            key_bytes = hashlib.sha256(secret).digest()
+            fernet_key = base64.urlsafe_b64encode(key_bytes)
+            
+            _cipher_suite = Fernet(fernet_key)
+            logger.debug("Encryption cipher suite initialized")
+            
+        except ImportError:
+            logger.warning(
+                "cryptography library not installed. "
+                "Sensitive settings will be stored in plaintext. "
+                "Install with: pip install cryptography"
+            )
+            _cipher_suite = None
+        except Exception as e:
+            logger.error(f"Failed to initialize encryption: {e}")
+            _cipher_suite = None
+    
+    return _cipher_suite
+
+
+def encrypt_value(plaintext: Optional[str]) -> Optional[str]:
+    """
+    Encrypt a plaintext value for storage in the database.
+    
+    Args:
+        plaintext: The value to encrypt (or None)
+        
+    Returns:
+        Encrypted value as base64 string, or plaintext if encryption unavailable
+    """
+    if plaintext is None or plaintext == "":
+        return plaintext
+    
+    cipher = _get_cipher_suite()
+    
+    if cipher is None:
+        # Encryption not available, store in plaintext with warning
+        logger.warning("Storing sensitive value in plaintext (encryption unavailable)")
+        return plaintext
+    
+    try:
+        encrypted_bytes = cipher.encrypt(plaintext.encode('utf-8'))
+        # Prefix with "enc:" to identify encrypted values
+        return "enc:" + encrypted_bytes.decode('utf-8')
+    except Exception as e:
+        logger.error(f"Encryption failed: {e}")
+        # Fall back to plaintext
+        return plaintext
+
+
+def decrypt_value(ciphertext: Optional[str]) -> Optional[str]:
+    """
+    Decrypt a value from the database.
+    
+    Args:
+        ciphertext: The encrypted value (or plaintext if not encrypted)
+        
+    Returns:
+        Decrypted plaintext value
+    """
+    if ciphertext is None or ciphertext == "":
+        return ciphertext
+    
+    # Check if value is encrypted (has "enc:" prefix)
+    if not ciphertext.startswith("enc:"):
+        # Not encrypted, return as-is
+        return ciphertext
+    
+    cipher = _get_cipher_suite()
+    
+    if cipher is None:
+        logger.error("Cannot decrypt value: encryption not available")
+        return "[ENCRYPTED - Cannot decrypt]"
+    
+    try:
+        # Remove "enc:" prefix and decrypt
+        encrypted_bytes = ciphertext[4:].encode('utf-8')
+        plaintext_bytes = cipher.decrypt(encrypted_bytes)
+        return plaintext_bytes.decode('utf-8')
+    except Exception as e:
+        logger.error(f"Decryption failed: {e}")
+        return "[DECRYPTION FAILED]"
+
+
+def is_encrypted(value: Optional[str]) -> bool:
+    """
+    Check if a value is encrypted.
+    
+    Args:
+        value: The value to check
+        
+    Returns:
+        True if the value is encrypted, False otherwise
+    """
+    return value is not None and isinstance(value, str) and value.startswith("enc:")
+
+
+def is_encryption_available() -> bool:
+    """
+    Check if encryption is available.
+    
+    Returns:
+        True if cryptography library is installed and encryption is working
+    """
+    return _get_cipher_suite() is not None

app/utils/settings_service.py

@@ -878,16 +878,27 @@ def get_setting_from_db(db: Session, key: str) -> Optional[str]:
     """
     Retrieve a setting value from the database.
     
+    Automatically decrypts sensitive values if encryption is enabled.
+    
     Args:
         db: Database session
         key: Setting key to retrieve
         
     Returns:
-        Setting value as string, or None if not found
+        Setting value as string (decrypted if necessary), or None if not found
     """
     try:
         setting = db.query(ApplicationSettings).filter(ApplicationSettings.key == key).first()
-        return setting.value if setting else None
+        if not setting:
+            return None
+        
+        # Check if this setting is sensitive and should be decrypted
+        metadata = get_setting_metadata(key)
+        if metadata.get("sensitive", False):
+            from app.utils.encryption import decrypt_value
+            return decrypt_value(setting.value)
+        
+        return setting.value
     except SQLAlchemyError as e:
         logger.error(f"Error retrieving setting {key} from database: {e}")
         return None
@@ -897,6 +908,8 @@ def save_setting_to_db(db: Session, key: str, value: Optional[str]) -> bool:
     """
     Save or update a setting in the database.
     
+    Automatically encrypts sensitive values if encryption is enabled.
+    
     Args:
         db: Database session
         key: Setting key
@@ -906,11 +919,24 @@ def save_setting_to_db(db: Session, key: str, value: Optional[str]) -> bool:
         True if successful, False otherwise
     """
     try:
+        # Check if this setting is sensitive and should be encrypted
+        metadata = get_setting_metadata(key)
+        storage_value = value
+        
+        if metadata.get("sensitive", False) and value:
+            from app.utils.encryption import encrypt_value, is_encryption_available
+            
+            if is_encryption_available():
+                storage_value = encrypt_value(value)
+                logger.debug(f"Encrypted sensitive setting: {key}")
+            else:
+                logger.warning(f"Storing sensitive setting {key} in plaintext (encryption unavailable)")
+        
         setting = db.query(ApplicationSettings).filter(ApplicationSettings.key == key).first()
         if setting:
-            setting.value = value
+            setting.value = storage_value
         else:
-            setting = ApplicationSettings(key=key, value=value)
+            setting = ApplicationSettings(key=key, value=storage_value)
             db.add(setting)
         db.commit()
         logger.info(f"Saved setting {key} to database")
@@ -925,15 +951,28 @@ def get_all_settings_from_db(db: Session) -> Dict[str, str]:
     """
     Retrieve all settings from the database.
     
+    Automatically decrypts sensitive values if encryption is enabled.
+    
     Args:
         db: Database session
         
     Returns:
-        Dictionary of setting key-value pairs
+        Dictionary of setting key-value pairs (decrypted)
     """
     try:
         settings = db.query(ApplicationSettings).all()
-        return {setting.key: setting.value for setting in settings}
+        result = {}
+        
+        for setting in settings:
+            # Check if this setting is sensitive and should be decrypted
+            metadata = get_setting_metadata(setting.key)
+            if metadata.get("sensitive", False):
+                from app.utils.encryption import decrypt_value
+                result[setting.key] = decrypt_value(setting.value)
+            else:
+                result[setting.key] = setting.value
+        
+        return result
     except SQLAlchemyError as e:
         logger.error(f"Error retrieving all settings from database: {e}")
         return {}

app/utils/setup_wizard.py

@@ -0,0 +1,212 @@
+"""
+Setup wizard utilities for first-time system configuration.
+
+Detects if the system needs initial setup and provides required settings list.
+"""
+
+import logging
+from typing import List, Dict, Any
+from app.config import settings
+
+logger = logging.getLogger(__name__)
+
+
+def get_required_settings() -> List[Dict[str, Any]]:
+    """
+    Get list of settings that are absolutely required for the system to operate.
+    
+    Returns:
+        List of required setting definitions with metadata
+    """
+    return [
+        {
+            "key": "database_url",
+            "label": "Database URL",
+            "description": "Database connection string (e.g., sqlite:///./app/database.db)",
+            "type": "string",
+            "sensitive": False,
+            "default": "sqlite:///./app/database.db",
+            "wizard_step": 1,
+            "wizard_category": "Core Infrastructure"
+        },
+        {
+            "key": "redis_url",
+            "label": "Redis URL",
+            "description": "Redis connection for task queue (e.g., redis://localhost:6379/0)",
+            "type": "string",
+            "sensitive": False,
+            "default": "redis://localhost:6379/0",
+            "wizard_step": 1,
+            "wizard_category": "Core Infrastructure"
+        },
+        {
+            "key": "workdir",
+            "label": "Working Directory",
+            "description": "Directory for temporary file storage and processing",
+            "type": "string",
+            "sensitive": False,
+            "default": "/workdir",
+            "wizard_step": 1,
+            "wizard_category": "Core Infrastructure"
+        },
+        {
+            "key": "gotenberg_url",
+            "label": "Gotenberg URL",
+            "description": "Gotenberg service URL for document conversion",
+            "type": "string",
+            "sensitive": False,
+            "default": "http://gotenberg:3000",
+            "wizard_step": 1,
+            "wizard_category": "Core Infrastructure"
+        },
+        {
+            "key": "session_secret",
+            "label": "Session Secret",
+            "description": "Secret key for session encryption (min 32 characters, auto-generate recommended)",
+            "type": "string",
+            "sensitive": True,
+            "default": None,  # Should be generated
+            "wizard_step": 2,
+            "wizard_category": "Security"
+        },
+        {
+            "key": "admin_username",
+            "label": "Admin Username",
+            "description": "Username for the admin account",
+            "type": "string",
+            "sensitive": False,
+            "default": "admin",
+            "wizard_step": 2,
+            "wizard_category": "Security"
+        },
+        {
+            "key": "admin_password",
+            "label": "Admin Password",
+            "description": "Password for the admin account",
+            "type": "string",
+            "sensitive": True,
+            "default": None,  # Must be set
+            "wizard_step": 2,
+            "wizard_category": "Security"
+        },
+        {
+            "key": "openai_api_key",
+            "label": "OpenAI API Key",
+            "description": "API key for OpenAI services (metadata extraction)",
+            "type": "string",
+            "sensitive": True,
+            "default": None,
+            "wizard_step": 3,
+            "wizard_category": "AI Services"
+        },
+        {
+            "key": "azure_ai_key",
+            "label": "Azure AI Key",
+            "description": "Azure AI key for document intelligence (OCR)",
+            "type": "string",
+            "sensitive": True,
+            "default": None,
+            "wizard_step": 3,
+            "wizard_category": "AI Services"
+        },
+        {
+            "key": "azure_region",
+            "label": "Azure Region",
+            "description": "Azure region for AI services (e.g., eastus)",
+            "type": "string",
+            "sensitive": False,
+            "default": "eastus",
+            "wizard_step": 3,
+            "wizard_category": "AI Services"
+        },
+        {
+            "key": "azure_endpoint",
+            "label": "Azure Endpoint",
+            "description": "Azure AI endpoint URL",
+            "type": "string",
+            "sensitive": False,
+            "default": None,
+            "wizard_step": 3,
+            "wizard_category": "AI Services"
+        },
+    ]
+
+
+def is_setup_required() -> bool:
+    """
+    Check if the system requires initial setup.
+    
+    Returns True if any critical required settings are missing or have placeholder values.
+    
+    Returns:
+        True if setup wizard should be shown, False otherwise
+    """
+    try:
+        # Critical settings that must be configured
+        critical_settings = [
+            ("session_secret", ["INSECURE_DEFAULT_FOR_DEVELOPMENT_ONLY_DO_NOT_USE_IN_PRODUCTION_MINIMUM_32_CHARS"]),
+            ("admin_password", [None, "", "your_secure_password", "changeme", "admin"]),
+            ("openai_api_key", [None, "", "<OPENAI_API_KEY>", "test-key"]),
+            ("azure_ai_key", [None, "", "<AZURE_AI_KEY>", "test-key"]),
+        ]
+        
+        for setting_key, invalid_values in critical_settings:
+            value = getattr(settings, setting_key, None)
+            if value in invalid_values:
+                logger.warning(f"Setup required: {setting_key} has placeholder or missing value")
+                return True
+        
+        # All critical settings are configured
+        return False
+        
+    except Exception as e:
+        logger.error(f"Error checking if setup required: {e}")
+        # If we can't check, assume setup is not required (fail open)
+        return False
+
+
+def get_missing_required_settings() -> List[str]:
+    """
+    Get list of required settings that are missing or have placeholder values.
+    
+    Returns:
+        List of setting keys that need to be configured
+    """
+    missing = []
+    
+    for required_setting in get_required_settings():
+        key = required_setting["key"]
+        value = getattr(settings, key, None)
+        
+        # Check if value is missing or is a placeholder
+        placeholder_values = [
+            None, "", 
+            f"<{key.upper()}>",
+            "test-key",
+            "your_secure_password",
+            "changeme",
+            "INSECURE_DEFAULT_FOR_DEVELOPMENT_ONLY_DO_NOT_USE_IN_PRODUCTION_MINIMUM_32_CHARS"
+        ]
+        
+        if value in placeholder_values:
+            missing.append(key)
+    
+    return missing
+
+
+def get_wizard_steps() -> Dict[int, List[Dict[str, Any]]]:
+    """
+    Get setup wizard steps organized by step number.
+    
+    Returns:
+        Dictionary mapping step number to list of settings in that step
+    """
+    steps = {}
+    
+    for setting in get_required_settings():
+        step_num = setting.get("wizard_step", 1)
+        if step_num not in steps:
+            steps[step_num] = []
+        steps[step_num].append(setting)
+    
+    return steps

app/views/__init__.py

@@ -11,9 +11,11 @@ from app.views.dropbox import router as dropbox_router
 from app.views.google_drive import router as google_drive_router
 from app.views.license_routes import router as license_router  # Add the license router
 from app.views.settings import router as settings_router
+from app.views.wizard import router as wizard_router
 
 # Create a main router that includes all the view routers
 router = APIRouter()
+router.include_router(wizard_router)  # Wizard first (for /setup)
 router.include_router(general_router)
 router.include_router(status_router)
 router.include_router(onedrive_router)

app/views/general.py

@@ -14,7 +14,25 @@ router = APIRouter()
 
 @router.get("/", include_in_schema=False)
 async def serve_index(request: Request, db: Session = Depends(get_db)):
-    """Serve the index/home page."""
+    """
+    Serve the index/home page.
+    
+    If the system requires initial setup, redirect to the setup wizard.
+    """
+    # Check if setup wizard is needed
+    from app.utils.setup_wizard import is_setup_required
+    from app.utils.settings_service import get_setting_from_db
+    
+    # Check if setup was explicitly skipped
+    setup_skipped = get_setting_from_db(db, "_setup_wizard_skipped")
+    
+    # Check setup completion query param
+    setup_complete = request.query_params.get("setup") == "complete"
+    
+    if not setup_skipped and not setup_complete and is_setup_required():
+        logger.info("System requires initial setup, redirecting to wizard")
+        return RedirectResponse(url="/setup?step=1", status_code=303)
+    
     # Get provider information from config validator
     providers = get_provider_status()
     

app/views/wizard.py

@@ -0,0 +1,133 @@
+"""
+Setup wizard views for initial system configuration.
+"""
+
+import os
+import logging
+import secrets
+from fastapi import Request, Depends, Form
+from fastapi.responses import RedirectResponse
+from sqlalchemy.orm import Session
+
+from app.views.base import APIRouter, templates, get_db
+from app.utils.setup_wizard import (
+    is_setup_required,
+    get_required_settings,
+    get_wizard_steps,
+    get_missing_required_settings
+)
+from app.utils.settings_service import save_setting_to_db
+
+logger = logging.getLogger(__name__)
+router = APIRouter()
+
+
+@router.get("/setup")
+async def setup_wizard(request: Request, step: int = 1):
+    """
+    Setup wizard for first-time configuration.
+    
+    This wizard guides users through configuring essential settings
+    needed for the system to operate properly.
+    """
+    # Get wizard steps
+    wizard_steps = get_wizard_steps()
+    max_step = max(wizard_steps.keys())
+    
+    # Validate step number
+    if step < 1:
+        step = 1
+    elif step > max_step:
+        step = max_step
+    
+    # Get settings for current step
+    current_settings = wizard_steps.get(step, [])
+    
+    # Get step category (all settings in a step should have same category)
+    step_category = current_settings[0].get("wizard_category", "Configuration") if current_settings else "Configuration"
+    
+    return templates.TemplateResponse(
+        "setup_wizard.html",
+        {
+            "request": request,
+            "current_step": step,
+            "max_step": max_step,
+            "settings": current_settings,
+            "step_category": step_category,
+            "progress_percent": int((step / max_step) * 100)
+        }
+    )
+
+
+@router.post("/setup")
+async def setup_wizard_save(
+    request: Request,
+    step: int = Form(...),
+    db: Session = Depends(get_db)
+):
+    """
+    Save settings from the current wizard step.
+    """
+    try:
+        # Get form data
+        form_data = await request.form()
+        
+        # Get settings for current step
+        wizard_steps = get_wizard_steps()
+        current_settings = wizard_steps.get(step, [])
+        
+        # Save each setting from the form
+        saved_count = 0
+        for setting in current_settings:
+            key = setting["key"]
+            value = form_data.get(key)
+            
+            # Skip empty values unless it's explicitly allowed
+            if value and value.strip():
+                # Auto-generate session_secret if needed
+                if key == "session_secret" and value == "auto-generate":
+                    value = secrets.token_hex(32)
+                    logger.info("Auto-generated session secret")
+                
+                # Save to database
+                if save_setting_to_db(db, key, value):
+                    saved_count += 1
+                    logger.info(f"Setup wizard: Saved {key}")
+        
+        logger.info(f"Setup wizard step {step}: Saved {saved_count} settings")
+        
+        # Determine next step
+        max_step = max(wizard_steps.keys())
+        next_step = step + 1
+        
+        if next_step > max_step:
+            # Setup complete, redirect to home
+            return RedirectResponse(url="/?setup=complete", status_code=303)
+        else:
+            # Go to next step
+            return RedirectResponse(url=f"/setup?step={next_step}", status_code=303)
+            
+    except Exception as e:
+        logger.error(f"Error saving wizard settings: {e}")
+        return RedirectResponse(url=f"/setup?step={step}&error=save_failed", status_code=303)
+
+
+@router.get("/setup/skip")
+async def setup_wizard_skip(request: Request):
+    """
+    Skip the setup wizard (for advanced users).
+    
+    Creates a marker to indicate setup was skipped.
+    """
+    try:
+        db = next(get_db())
+        try:
+            # Save a marker to indicate setup was skipped
+            save_setting_to_db(db, "_setup_wizard_skipped", "true")
+            logger.info("Setup wizard skipped by user")
+            return RedirectResponse(url="/", status_code=303)
+        finally:
+            db.close()
+    except Exception as e:
+        logger.error(f"Error skipping setup wizard: {e}")
+        return RedirectResponse(url="/", status_code=303)

frontend/templates/settings.html

@@ -29,7 +29,8 @@
       <p class="font-bold">⚠️ Important Notes:</p>
       <ul class="list-disc list-inside ml-4 mt-2">
         <li>Settings marked with <span class="text-red-600">*</span> require an application restart to take effect.</li>
-        <li>Sensitive values (passwords, API keys) are masked for security.</li>
+        <li>Sensitive values (passwords, API keys) are <strong>encrypted at rest</strong> in the database <i class="fas fa-lock text-xs"></i>.</li>
+        <li>Use the <i class="fas fa-eye"></i> icon to temporarily show/hide sensitive values.</li>
         <li>Saving a setting here stores it in the database and overrides environment variables.</li>
         <li>Only administrators can access and modify these settings.</li>
         <li>All fields are optional - you can save just the settings you want to override.</li>
@@ -108,22 +109,35 @@
               <!-- Text Input -->
               <div class="relative">
                 {% if setting.metadata.sensitive %}
+                <!-- Sensitive Field with Show/Hide Toggle -->
+                <div class="relative">
                   <input 
                     :type="showPassword['{{ setting.key }}'] ? 'text' : 'password'"
                     id="{{ setting.key }}" 
                     name="{{ setting.key }}"
                     x-model="formData['{{ setting.key }}']"
-                  class="setting-input w-full px-3 py-2 pr-10 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-blue-500 focus:border-blue-500"
+                    class="setting-input w-full px-3 py-2 pr-24 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-blue-500 focus:border-blue-500 font-mono text-sm"
                     placeholder="{{ setting.metadata.description }}"
+                    autocomplete="off"
                   />
+                  <div class="absolute inset-y-0 right-0 flex items-center pr-3 space-x-2">
+                    <!-- Encrypted indicator -->
+                    <span class="text-xs text-gray-400" title="Value is encrypted at rest in database">
+                      <i class="fas fa-lock"></i>
+                    </span>
+                    <!-- Show/Hide Toggle -->
                     <button 
                       type="button"
                       @click="togglePassword('{{ setting.key }}')"
-                  class="absolute inset-y-0 right-0 pr-3 flex items-center text-gray-400 hover:text-gray-600"
+                      class="text-gray-400 hover:text-gray-600 focus:outline-none"
+                      title="Show/hide value"
                     >
                       <i :class="showPassword['{{ setting.key }}'] ? 'fas fa-eye-slash' : 'fas fa-eye'"></i>
                     </button>
+                  </div>
+                </div>
                 {% else %}
+                <!-- Non-Sensitive Field -->
                 <input 
                   type="text" 
                   id="{{ setting.key }}" 

frontend/templates/setup_wizard.html

@@ -0,0 +1,218 @@
+{% extends "base.html" %}
+{% block title %}Setup Wizard - DocuElevate{% endblock %}
+
+{% block head_extra %}
+<style>
+  .wizard-input {
+    font-family: 'Courier New', monospace;
+  }
+  .progress-step {
+    transition: all 0.3s ease;
+  }
+  .progress-step.active {
+    background-color: #3b82f6;
+    color: white;
+  }
+  .progress-step.completed {
+    background-color: #10b981;
+    color: white;
+  }
+</style>
+{% endblock %}
+
+{% block content %}
+<div class="min-h-screen bg-gradient-to-br from-blue-50 to-indigo-100 py-12 px-4 sm:px-6 lg:px-8">
+  <div class="max-w-3xl mx-auto">
+    
+    <!-- Wizard Header -->
+    <div class="text-center mb-8">
+      <h1 class="text-4xl font-bold text-gray-900 mb-2">
+        <i class="fas fa-magic text-indigo-600"></i>
+        DocuElevate Setup Wizard
+      </h1>
+      <p class="text-lg text-gray-600">
+        Welcome! Let's configure your system in just a few steps.
+      </p>
+    </div>
+
+    <!-- Progress Bar -->
+    <div class="mb-8">
+      <div class="flex justify-between items-center mb-2">
+        <span class="text-sm font-medium text-gray-700">Step {{ current_step }} of {{ max_step }}</span>
+        <span class="text-sm font-medium text-gray-700">{{ progress_percent }}% Complete</span>
+      </div>
+      <div class="w-full bg-gray-200 rounded-full h-3">
+        <div class="bg-indigo-600 h-3 rounded-full transition-all duration-500" style="width: {{ progress_percent }}%"></div>
+      </div>
+      
+      <!-- Step Indicators -->
+      <div class="flex justify-between mt-4">
+        {% for step_num in range(1, max_step + 1) %}
+        <div class="flex flex-col items-center progress-step {% if step_num < current_step %}completed{% elif step_num == current_step %}active{% endif %}">
+          <div class="w-10 h-10 rounded-full flex items-center justify-center border-2 {% if step_num < current_step %}bg-green-500 border-green-500 text-white{% elif step_num == current_step %}bg-indigo-600 border-indigo-600 text-white{% else %}bg-white border-gray-300 text-gray-500{% endif %}">
+            {% if step_num < current_step %}
+            <i class="fas fa-check"></i>
+            {% else %}
+            {{ step_num }}
+            {% endif %}
+          </div>
+          <span class="text-xs mt-1 {% if step_num == current_step %}text-indigo-600 font-semibold{% else %}text-gray-500{% endif %}">
+            {% if step_num == 1 %}Infrastructure{% elif step_num == 2 %}Security{% elif step_num == 3 %}AI Services{% endif %}
+          </span>
+        </div>
+        {% endfor %}
+      </div>
+    </div>
+
+    <!-- Wizard Card -->
+    <div class="bg-white rounded-lg shadow-xl overflow-hidden">
+      
+      <!-- Card Header -->
+      <div class="bg-indigo-600 px-6 py-4">
+        <h2 class="text-2xl font-bold text-white">
+          <i class="fas fa-cog mr-2"></i>
+          {{ step_category }}
+        </h2>
+        <p class="text-indigo-100 mt-1">Configure essential settings for this category</p>
+      </div>
+
+      <!-- Card Body -->
+      <form method="post" action="/setup" class="px-6 py-8">
+        <input type="hidden" name="step" value="{{ current_step }}">
+        
+        {% if request.query_params.get('error') == 'save_failed' %}
+        <div class="mb-6 bg-red-100 border-l-4 border-red-500 text-red-700 p-4" role="alert">
+          <p class="font-bold">⚠️ Error</p>
+          <p>Failed to save settings. Please try again.</p>
+        </div>
+        {% endif %}
+
+        <div class="space-y-6">
+          {% for setting in settings %}
+          <div class="border-b border-gray-200 pb-6 last:border-b-0">
+            <label for="{{ setting.key }}" class="block text-sm font-medium text-gray-900 mb-1">
+              {{ setting.label }}
+              {% if setting.default is none or setting.key in ['admin_password', 'openai_api_key', 'azure_ai_key', 'azure_endpoint'] %}
+              <span class="text-red-600">*</span>
+              {% endif %}
+            </label>
+            
+            <p class="text-xs text-gray-500 mb-3">
+              {{ setting.description }}
+            </p>
+
+            {% if setting.key == 'session_secret' %}
+            <!-- Special handling for session_secret with auto-generate -->
+            <div class="space-y-2">
+              <div class="flex items-center space-x-4">
+                <label class="inline-flex items-center">
+                  <input type="radio" name="session_secret_mode" value="auto" checked 
+                         class="form-radio text-indigo-600" 
+                         onchange="document.getElementById('session_secret').value = 'auto-generate'; document.getElementById('session_secret').disabled = true;">
+                  <span class="ml-2 text-sm">Auto-generate (recommended)</span>
+                </label>
+                <label class="inline-flex items-center">
+                  <input type="radio" name="session_secret_mode" value="manual"
+                         class="form-radio text-indigo-600"
+                         onchange="document.getElementById('session_secret').value = ''; document.getElementById('session_secret').disabled = false; document.getElementById('session_secret').focus();">
+                  <span class="ml-2 text-sm">Enter manually</span>
+                </label>
+              </div>
+              <input 
+                type="{% if setting.sensitive %}password{% else %}text{% endif %}"
+                id="{{ setting.key }}"
+                name="{{ setting.key }}"
+                value="auto-generate"
+                disabled
+                class="wizard-input w-full px-4 py-3 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-transparent disabled:bg-gray-100"
+                placeholder="Will be auto-generated"
+              />
+            </div>
+            {% else %}
+            <!-- Regular input -->
+            <input 
+              type="{% if setting.sensitive %}password{% else %}text{% endif %}"
+              id="{{ setting.key }}"
+              name="{{ setting.key }}"
+              value="{{ setting.default if setting.default else '' }}"
+              class="wizard-input w-full px-4 py-3 border border-gray-300 rounded-md shadow-sm focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-transparent"
+              placeholder="{{ setting.description }}"
+              {% if setting.default is none or setting.key in ['admin_password', 'openai_api_key', 'azure_ai_key', 'azure_endpoint'] %}required{% endif %}
+            />
+            {% endif %}
+            
+            {% if setting.key == 'admin_password' %}
+            <p class="mt-2 text-xs text-amber-600">
+              <i class="fas fa-exclamation-triangle"></i>
+              <strong>Important:</strong> Choose a strong password. This cannot be recovered if lost.
+            </p>
+            {% elif setting.sensitive %}
+            <p class="mt-2 text-xs text-gray-500">
+              <i class="fas fa-lock"></i>
+              This value will be encrypted at rest in the database.
+            </p>
+            {% endif %}
+          </div>
+          {% endfor %}
+        </div>
+
+        <!-- Navigation Buttons -->
+        <div class="flex justify-between items-center mt-8 pt-6 border-t border-gray-200">
+          <div>
+            {% if current_step == 1 %}
+            <a href="/setup/skip" class="text-sm text-gray-600 hover:text-gray-900">
+              <i class="fas fa-forward"></i>
+              Skip setup (advanced users)
+            </a>
+            {% endif %}
+          </div>
+          
+          <div class="flex space-x-4">
+            {% if current_step > 1 %}
+            <a href="/setup?step={{ current_step - 1 }}" 
+               class="px-6 py-3 border border-gray-300 text-gray-700 rounded-md hover:bg-gray-50 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500">
+              <i class="fas fa-arrow-left mr-2"></i>
+              Previous
+            </a>
+            {% endif %}
+            
+            <button type="submit" 
+                    class="px-6 py-3 bg-indigo-600 text-white rounded-md hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 shadow-lg">
+              {% if current_step < max_step %}
+              Next Step
+              <i class="fas fa-arrow-right ml-2"></i>
+              {% else %}
+              Complete Setup
+              <i class="fas fa-check ml-2"></i>
+              {% endif %}
+            </button>
+          </div>
+        </div>
+      </form>
+    </div>
+
+    <!-- Help Text -->
+    <div class="mt-6 text-center">
+      <p class="text-sm text-gray-600">
+        <i class="fas fa-info-circle"></i>
+        All settings can be changed later in the Settings page.
+      </p>
+      <p class="text-xs text-gray-500 mt-2">
+        Fields marked with <span class="text-red-600">*</span> are required.
+      </p>
+    </div>
+
+  </div>
+</div>
+
+<script>
+// Auto-submit form when session_secret mode changes
+document.querySelectorAll('input[name="session_secret_mode"]').forEach(radio => {
+  radio.addEventListener('change', function() {
+    if (this.value === 'auto') {
+      document.getElementById('session_secret').value = 'auto-generate';
+    }
+  });
+});
+</script>
+{% endblock %}

requirements.txt

@@ -4,6 +4,7 @@ celery  # Task queue
 redis  # Message broker for Celery
 sqlalchemy  # Database ORM
 pydantic  # Data validation
+cryptography>=41.0.0  # Encryption for sensitive settings in database
 openai  # GPT integration for metadata extraction
 PyPDF2>=3.0.0  # PDF processing for text extraction, metadata editing and rotation (replaces PyMuPDF)
 requests  # HTTP client

test_form_prefilling.py

@@ -1,183 +0,0 @@
-#!/usr/bin/env python3
-"""
-Test to verify settings form pre-filling and source detection
-"""
-import os
-import sys
-import tempfile
-
-# Set up test environment
-os.environ["DATABASE_URL"] = f"sqlite:///{tempfile.gettempdir()}/test_source.db"
-os.environ["REDIS_URL"] = "redis://localhost:6379/1"
-os.environ["OPENAI_API_KEY"] = "test-key-from-env"
-os.environ["AZURE_AI_KEY"] = "test-key"
-os.environ["AZURE_REGION"] = "test"
-os.environ["AZURE_ENDPOINT"] = "https://test.example.com"
-os.environ["GOTENBERG_URL"] = "http://localhost:3000"
-os.environ["WORKDIR"] = tempfile.gettempdir()
-os.environ["AUTH_ENABLED"] = "true"
-os.environ["SESSION_SECRET"] = "test_secret_key_for_testing_must_be_at_least_32_characters_long"
-os.environ["ADMIN_USERNAME"] = "admin"
-os.environ["ADMIN_PASSWORD"] = "admin123"
-os.environ["DEBUG"] = "true"  # Set via environment
-
-from app.config import settings
-from app.database import init_db, SessionLocal
-from app.utils.settings_service import save_setting_to_db, get_all_settings_from_db
-from app.utils.config_loader import load_settings_from_db
-
-def test_settings_source_detection():
-    """Test that we can detect the source of each setting"""
-    print("=" * 60)
-    print("Testing Settings Source Detection")
-    print("=" * 60)
-    
-    # Initialize database
-    init_db()
-    db = SessionLocal()
-    
-    try:
-        # 1. Test DEFAULT source
-        print("\n1. Testing DEFAULT source:")
-        # allow_file_delete has a default value and no env var set
-        print(f"   allow_file_delete = {settings.allow_file_delete}")
-        print(f"   Source: DEFAULT (no env var or DB entry)")
-        
-        # 2. Test ENVIRONMENT source
-        print("\n2. Testing ENVIRONMENT source:")
-        print(f"   debug = {settings.debug}")
-        print(f"   DEBUG env var = {os.environ.get('DEBUG')}")
-        db_settings = get_all_settings_from_db(db)
-        if "debug" in db_settings:
-            print(f"   Source: DATABASE (overriding env)")
-        else:
-            print(f"   Source: ENVIRONMENT (from env var)")
-        
-        # 3. Test DATABASE source (save to DB and reload)
-        print("\n3. Testing DATABASE source:")
-        save_setting_to_db(db, "openai_model", "gpt-4-custom")
-        load_settings_from_db(settings, db)
-        print(f"   openai_model = {settings.openai_model}")
-        db_settings = get_all_settings_from_db(db)
-        if "openai_model" in db_settings:
-            print(f"   Source: DATABASE (explicitly saved)")
-        
-        # 4. Simulate what the view does
-        print("\n4. Simulating view source detection:")
-        db_settings = get_all_settings_from_db(db)
-        
-        test_keys = ["database_url", "debug", "openai_model", "allow_file_delete"]
-        for key in test_keys:
-            value = getattr(settings, key, None)
-            
-            if key in db_settings:
-                source = "DATABASE"
-                color = "green"
-            elif key.upper() in os.environ or key in os.environ:
-                source = "ENVIRONMENT"
-                color = "blue"
-            else:
-                source = "DEFAULT"
-                color = "gray"
-            
-            value_str = str(value)[:50] if value else "None"
-            print(f"   {key:25} = {value_str:30} [{color.upper()} {source}]")
-        
-        print("\n✓ Source detection works correctly!")
-        print()
-        
-    finally:
-        db.close()
-
-def test_form_prefilling():
-    """Test that form would be pre-filled with current values"""
-    print("=" * 60)
-    print("Testing Form Pre-filling Logic")
-    print("=" * 60)
-    
-    db = SessionLocal()
-    try:
-        # Get all settings from DB
-        db_settings = get_all_settings_from_db(db)
-        
-        # Simulate building form data (what the view does)
-        form_data = {}
-        test_keys = ["database_url", "debug", "openai_api_key", "openai_model", "allow_file_delete"]
-        
-        for key in test_keys:
-            # Get current value (with precedence already applied)
-            value = getattr(settings, key, None)
-            
-            # Determine source
-            if key in db_settings:
-                source = "DB"
-            elif key.upper() in os.environ or key in os.environ:
-                source = "ENV"
-            else:
-                source = "DEFAULT"
-            
-            # This would be passed to the template
-            form_data[key] = {
-                "value": value,
-                "source": source
-            }
-            
-            # Show what would be in the form
-            value_display = str(value)[:40] if value else ""
-            print(f"   {key:25} [{source:8}]: {value_display}")
-        
-        print("\n✓ Form data prepared correctly!")
-        print("✓ All fields would be pre-filled with current values")
-        print("✓ Source indicators would be shown")
-        print()
-        
-    finally:
-        db.close()
-
-def test_optional_fields():
-    """Test that fields are not required for submission"""
-    print("=" * 60)
-    print("Testing Optional Fields (No HTML 'required')")
-    print("=" * 60)
-    
-    # The template should NOT have 'required' attributes on inputs
-    # This means users can save just the settings they want to change
-    
-    print("✓ HTML 'required' attributes removed from template")
-    print("✓ Users can leave fields empty")
-    print("✓ Only changed values are submitted")
-    print("✓ Server-side validation handles actual requirements")
-    print()
-
-def main():
-    print("\n" + "=" * 60)
-    print("SETTINGS FORM PRE-FILLING AND SOURCE DETECTION TEST")
-    print("=" * 60 + "\n")
-    
-    try:
-        test_settings_source_detection()
-        test_form_prefilling()
-        test_optional_fields()
-        
-        print("=" * 60)
-        print("ALL TESTS PASSED! ✓")
-        print("=" * 60)
-        print("\nSummary:")
-        print("  ✓ Settings form is pre-filled with current values")
-        print("  ✓ Values come from DB > ENV > DEFAULT (precedence order)")
-        print("  ✓ Source of each setting is detected correctly")
-        print("  ✓ UI shows badges: DB (green), ENV (blue), DEFAULT (gray)")
-        print("  ✓ All fields are optional (no HTML 'required' attribute)")
-        print("  ✓ Users can save just the settings they want to change")
-        return 0
-        
-    except Exception as e:
-        print("\n" + "=" * 60)
-        print(f"TEST FAILED: {e}")
-        print("=" * 60)
-        import traceback
-        traceback.print_exc()
-        return 1
-
-if __name__ == "__main__":
-    sys.exit(main())