Add Docker and manual installation guides, and comprehensive settings documentation for DMARQ

.readthedocs.yaml

@@ -10,7 +10,4 @@ mkdocs:
 
 python:
   install:
-    - requirements: docs/readthedocs/requirements.txt
+    - requirements: docs/readthedocs/requirements.txt
-
-sphinx:
-  fail_on_warning: true

docs/deployment/docker.md

@@ -0,0 +1,295 @@
+# Docker Setup
+
+This guide covers how to deploy DMARQ using Docker, which is the recommended deployment method.
+
+## Prerequisites
+
+Before deploying DMARQ with Docker, ensure you have:
+
+- Docker Engine 20.10.0 or later
+- Docker Compose v2.0.0 or later
+- 2GB RAM minimum (4GB recommended)
+- 20GB storage space
+
+## Quick Start
+
+The fastest way to get DMARQ running is to use Docker Compose:
+
+1. **Clone the repository**
+
+   ```bash
+   git clone https://github.com/yourusername/dmarq.git
+   cd dmarq
+   ```
+
+2. **Configure environment variables**
+
+   Create a `.env` file in the project root:
+
+   ```
+   # Database Configuration
+   DB_TYPE=sqlite  # or postgres for production
+   DB_PATH=./data/dmarq.db  # for SQLite
+   # For PostgreSQL:
+   # DB_HOST=postgres
+   # DB_PORT=5432
+   # DB_USER=dmarq
+   # DB_PASS=secure_password
+   # DB_NAME=dmarq
+
+   # IMAP Configuration (optional)
+   IMAP_ENABLED=false
+   # IMAP_SERVER=mail.example.com
+   # IMAP_PORT=993
+   # IMAP_USERNAME=dmarc@example.com
+   # IMAP_PASSWORD=your_secure_password
+   # IMAP_USE_SSL=true
+   # IMAP_POLLING_INTERVAL=60
+
+   # Security Settings
+   SECRET_KEY=generate_a_secure_random_key
+   ALLOWED_HOSTS=localhost,127.0.0.1
+   ```
+
+   Generate a secure random key for `SECRET_KEY`:
+
+   ```bash
+   openssl rand -hex 32
+   ```
+
+3. **Start the containers**
+
+   ```bash
+   docker-compose up -d
+   ```
+
+4. **Access the application**
+
+   Open your browser and navigate to `http://localhost:8000`
+
+## Understanding the Docker Setup
+
+The `docker-compose.yml` file defines the following services:
+
+- **backend**: The FastAPI application that handles API requests, processes reports, and serves the web interface
+- **db**: A PostgreSQL database container (when using Postgres instead of SQLite)
+
+### Docker Compose File Structure
+
+The `docker-compose.yml` file looks like this:
+
+```yaml
+version: '3.8'
+
+services:
+  backend:
+    build: 
+      context: ./backend
+    ports:
+      - "8000:8000"
+    volumes:
+      - ./data:/app/data
+    environment:
+      - DB_TYPE=${DB_TYPE:-sqlite}
+      - DB_PATH=${DB_PATH:-./data/dmarq.db}
+      - DB_HOST=${DB_HOST:-postgres}
+      - DB_PORT=${DB_PORT:-5432}
+      - DB_USER=${DB_USER:-dmarq}
+      - DB_PASS=${DB_PASS:-dmarqpassword}
+      - DB_NAME=${DB_NAME:-dmarq}
+      - IMAP_ENABLED=${IMAP_ENABLED:-false}
+      - IMAP_SERVER=${IMAP_SERVER:-}
+      - IMAP_PORT=${IMAP_PORT:-993}
+      - IMAP_USERNAME=${IMAP_USERNAME:-}
+      - IMAP_PASSWORD=${IMAP_PASSWORD:-}
+      - IMAP_USE_SSL=${IMAP_USE_SSL:-true}
+      - IMAP_POLLING_INTERVAL=${IMAP_POLLING_INTERVAL:-60}
+      - SECRET_KEY=${SECRET_KEY:-insecure_key_change_me_in_production}
+      - ALLOWED_HOSTS=${ALLOWED_HOSTS:-localhost,127.0.0.1}
+    depends_on:
+      - db
+    restart: unless-stopped
+
+  db:
+    image: postgres:14-alpine
+    volumes:
+      - postgres_data:/var/lib/postgresql/data
+    environment:
+      - POSTGRES_USER=${DB_USER:-dmarq}
+      - POSTGRES_PASSWORD=${DB_PASS:-dmarqpassword}
+      - POSTGRES_DB=${DB_NAME:-dmarq}
+    restart: unless-stopped
+
+volumes:
+  postgres_data:
+```
+
+## Configuration Options
+
+### Environment Variables
+
+All configuration in the Docker setup is done via environment variables, either directly in the `docker-compose.yml` file or through a separate `.env` file. See the [Configuration](configuration.md) page for detailed information about all available variables.
+
+### Volumes
+
+The Docker Compose setup uses these volumes:
+
+- **./data**: Local directory mapped to `/app/data` in the container, stores SQLite database (if used) and other persistent data
+- **postgres_data**: Docker volume for PostgreSQL data (when using Postgres)
+
+## Production Deployment
+
+For production deployments, consider these additional steps:
+
+### Using a Reverse Proxy
+
+In production, it's recommended to use a reverse proxy like Nginx or Traefik in front of DMARQ:
+
+```yaml
+version: '3.8'
+
+services:
+  # ...existing services...
+  
+  nginx:
+    image: nginx:alpine
+    ports:
+      - "80:80"
+      - "443:443"
+    volumes:
+      - ./nginx/conf.d:/etc/nginx/conf.d
+      - ./nginx/ssl:/etc/nginx/ssl
+    depends_on:
+      - backend
+    restart: unless-stopped
+```
+
+Example Nginx configuration:
+
+```nginx
+server {
+    listen 80;
+    server_name dmarq.example.com;
+    return 301 https://$server_name$request_uri;
+}
+
+server {
+    listen 443 ssl;
+    server_name dmarq.example.com;
+
+    ssl_certificate /etc/nginx/ssl/cert.pem;
+    ssl_certificate_key /etc/nginx/ssl/key.pem;
+
+    location / {
+        proxy_pass http://backend:8000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+    }
+}
+```
+
+### Docker Compose Profiles
+
+For more complex deployments, you can use Docker Compose profiles:
+
+```yaml
+version: '3.8'
+
+services:
+  backend:
+    # ...existing config...
+    profiles: [app, all]
+  
+  db:
+    # ...existing config...
+    profiles: [app, all]
+  
+  nginx:
+    # ...nginx config...
+    profiles: [production, all]
+```
+
+Then start only specific profiles:
+
+```bash
+docker-compose --profile production up -d
+```
+
+## Updating DMARQ
+
+To update to a newer version:
+
+```bash
+# Pull the latest code
+git pull
+
+# Stop the containers
+docker-compose down
+
+# Rebuild and start
+docker-compose up -d --build
+```
+
+## Monitoring and Maintenance
+
+### Viewing Logs
+
+To view logs from the containers:
+
+```bash
+# All logs
+docker-compose logs
+
+# Just backend logs
+docker-compose logs backend
+
+# Follow logs in real-time
+docker-compose logs -f
+```
+
+### Container Health Checks
+
+Monitor the health of your containers:
+
+```bash
+docker-compose ps
+```
+
+### Database Backups
+
+For PostgreSQL backups:
+
+```bash
+# Create a backup
+docker-compose exec db pg_dump -U dmarq dmarq > backup_$(date +%Y%m%d).sql
+
+# Restore from a backup
+cat backup_file.sql | docker-compose exec -T db psql -U dmarq dmarq
+```
+
+## Troubleshooting
+
+### Container Won't Start
+
+If containers fail to start:
+
+1. Check logs: `docker-compose logs backend`
+2. Verify environment variables: `docker-compose config`
+3. Check disk space: `df -h`
+4. Ensure ports aren't already in use: `netstat -tuln | grep 8000`
+
+### Database Connection Issues
+
+If the application can't connect to the database:
+
+1. Check the DB environment variables in `.env`
+2. For Postgres, ensure the `db` service is running: `docker-compose ps db`
+3. Try connecting manually: `docker-compose exec db psql -U dmarq dmarq`
+
+### Mounting Issues
+
+If you encounter volume mounting problems:
+
+1. Check file permissions on the host
+2. Use absolute paths in your volume mappings
+3. On Windows, ensure you've enabled Docker file sharing for the relevant drives

docs/deployment/manual.md

@@ -0,0 +1,333 @@
+# Manual Installation
+
+This guide covers how to deploy DMARQ without Docker, using a traditional installation method.
+
+## Prerequisites
+
+Before proceeding with a manual installation, ensure you have:
+
+- Python 3.9 or higher
+- pip and virtualenv
+- Node.js 16+ (if modifying frontend assets)
+- PostgreSQL (recommended for production) or SQLite
+- A web server like Nginx (for production)
+
+## Installation Steps
+
+### 1. Set Up the Environment
+
+First, clone the repository and set up a virtual environment:
+
+```bash
+# Clone the repository
+git clone https://github.com/yourusername/dmarq.git
+cd dmarq
+
+# Create and activate a virtual environment
+python -m venv venv
+source venv/bin/activate  # On Windows: venv\Scripts\activate
+```
+
+### 2. Install Dependencies
+
+Install the required Python packages:
+
+```bash
+cd backend
+pip install -r requirements.txt
+```
+
+### 3. Configure Environment Variables
+
+Create a `.env` file in the backend directory with your configuration:
+
+```
+# Database Configuration
+DB_TYPE=sqlite  # or postgres for production
+DB_PATH=./data/dmarq.db  # for SQLite
+# For PostgreSQL:
+# DB_HOST=localhost
+# DB_PORT=5432
+# DB_USER=dmarq
+# DB_PASS=secure_password
+# DB_NAME=dmarq
+
+# IMAP Configuration (optional)
+IMAP_ENABLED=false
+# IMAP_SERVER=mail.example.com
+# IMAP_PORT=993
+# IMAP_USERNAME=dmarc@example.com
+# IMAP_PASSWORD=your_secure_password
+# IMAP_USE_SSL=true
+# IMAP_POLLING_INTERVAL=60
+
+# Security Settings
+SECRET_KEY=generate_a_secure_random_key
+ALLOWED_HOSTS=localhost,127.0.0.1
+```
+
+Generate a secure random key for `SECRET_KEY`:
+
+```bash
+openssl rand -hex 32
+```
+
+### 4. Initialize the Database
+
+For SQLite:
+
+```bash
+# Create the data directory
+mkdir -p data
+
+# Initialize the database
+cd app
+python -m alembic upgrade head
+```
+
+For PostgreSQL:
+
+```bash
+# Create the database and user in PostgreSQL
+sudo -u postgres psql -c "CREATE USER dmarq WITH PASSWORD 'secure_password';"
+sudo -u postgres psql -c "CREATE DATABASE dmarq OWNER dmarq;"
+
+# Initialize the database
+cd app
+python -m alembic upgrade head
+```
+
+### 5. Start the Application (Development)
+
+For development or testing, you can run the application directly with Uvicorn:
+
+```bash
+cd app
+uvicorn main:app --host 0.0.0.0 --port 8000 --reload
+```
+
+### 6. Production Deployment with Systemd
+
+For a production environment, it's recommended to use a process manager like systemd:
+
+1. Create a systemd service file:
+
+```bash
+sudo nano /etc/systemd/system/dmarq.service
+```
+
+2. Add the following configuration:
+
+```
+[Unit]
+Description=DMARQ Application
+After=network.target
+
+[Service]
+User=dmarq
+WorkingDirectory=/path/to/dmarq/backend/app
+ExecStart=/path/to/dmarq/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
+Restart=always
+Environment="PATH=/path/to/dmarq/venv/bin"
+EnvironmentFile=/path/to/dmarq/backend/.env
+
+[Install]
+WantedBy=multi-user.target
+```
+
+3. Start and enable the service:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl start dmarq
+sudo systemctl enable dmarq
+```
+
+### 7. Set Up Nginx as a Reverse Proxy
+
+For production, it's recommended to use Nginx as a reverse proxy:
+
+1. Install Nginx:
+
+```bash
+sudo apt install nginx
+```
+
+2. Create a Nginx configuration file:
+
+```bash
+sudo nano /etc/nginx/sites-available/dmarq
+```
+
+3. Add the following configuration:
+
+```nginx
+server {
+    listen 80;
+    server_name dmarq.example.com;
+    
+    location / {
+        proxy_pass http://127.0.0.1:8000;
+        proxy_set_header Host $host;
+        proxy_set_header X-Real-IP $remote_addr;
+        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+        proxy_set_header X-Forwarded-Proto $scheme;
+    }
+}
+```
+
+4. Enable the site and reload Nginx:
+
+```bash
+sudo ln -s /etc/nginx/sites-available/dmarq /etc/nginx/sites-enabled/
+sudo nginx -t
+sudo systemctl reload nginx
+```
+
+### 8. Set Up HTTPS with Let's Encrypt
+
+For production, you should secure your site with HTTPS:
+
+```bash
+sudo apt install certbot python3-certbot-nginx
+sudo certbot --nginx -d dmarq.example.com
+```
+
+## Background Tasks
+
+DMARQ requires background tasks for IMAP polling and report processing. For simple deployments, the built-in background task system in FastAPI is sufficient. 
+
+For more complex deployments, you might want to set up Celery:
+
+1. Install Celery:
+
+```bash
+pip install celery redis
+```
+
+2. Create a Celery service file:
+
+```bash
+sudo nano /etc/systemd/system/dmarq-celery.service
+```
+
+3. Add the following configuration:
+
+```
+[Unit]
+Description=DMARQ Celery Worker
+After=network.target
+
+[Service]
+User=dmarq
+WorkingDirectory=/path/to/dmarq/backend/app
+ExecStart=/path/to/dmarq/venv/bin/celery -A worker worker --loglevel=info
+Restart=always
+Environment="PATH=/path/to/dmarq/venv/bin"
+EnvironmentFile=/path/to/dmarq/backend/.env
+
+[Install]
+WantedBy=multi-user.target
+```
+
+4. Start and enable the service:
+
+```bash
+sudo systemctl daemon-reload
+sudo systemctl start dmarq-celery
+sudo systemctl enable dmarq-celery
+```
+
+## Updating DMARQ
+
+To update to a newer version:
+
+```bash
+# Pull the latest code
+cd /path/to/dmarq
+git pull
+
+# Activate the virtual environment
+source venv/bin/activate
+
+# Update dependencies
+cd backend
+pip install -r requirements.txt
+
+# Apply any database migrations
+cd app
+python -m alembic upgrade head
+
+# Restart the service
+sudo systemctl restart dmarq
+```
+
+## Troubleshooting
+
+### Application Won't Start
+
+If the application fails to start:
+
+1. Check the systemd logs: `sudo journalctl -u dmarq`
+2. Verify the environment variables in your `.env` file
+3. Check that all Python dependencies are installed: `pip list | grep -E 'fastapi|uvicorn'`
+
+### Database Connection Issues
+
+If the application can't connect to the database:
+
+1. Check the DB environment variables in `.env`
+2. For PostgreSQL, verify the database exists: `sudo -u postgres psql -c "\l" | grep dmarq`
+3. Check if you can connect manually: `psql -U dmarq -h localhost dmarq`
+
+### Nginx Configuration Issues
+
+If Nginx isn't serving the application:
+
+1. Check Nginx error logs: `sudo tail -f /var/log/nginx/error.log`
+2. Verify the Nginx configuration: `sudo nginx -t`
+3. Make sure the application is running: `curl http://localhost:8000`
+
+## Monitoring and Maintenance
+
+### Checking Application Status
+
+Check if the application is running:
+
+```bash
+sudo systemctl status dmarq
+```
+
+### Viewing Application Logs
+
+View logs from the application:
+
+```bash
+# System logs
+sudo journalctl -u dmarq
+
+# Application logs (if configured to log to file)
+tail -f /path/to/dmarq/logs/dmarq.log
+```
+
+### Database Backups
+
+For PostgreSQL backups:
+
+```bash
+# Create a backup
+pg_dump -U dmarq dmarq > dmarq_backup_$(date +%Y%m%d).sql
+
+# Restore from a backup
+psql -U dmarq dmarq < dmarq_backup_file.sql
+```
+
+For SQLite backups:
+
+```bash
+# Create a backup
+sqlite3 /path/to/data/dmarq.db .dump > dmarq_backup_$(date +%Y%m%d).sql
+
+# Restore from a backup
+cat dmarq_backup_file.sql | sqlite3 /path/to/data/dmarq.db
+```

docs/user_guide/settings.md

@@ -0,0 +1,142 @@
+# Settings
+
+This guide covers the various settings and configuration options available in DMARQ.
+
+## General Settings
+
+### User Profile
+
+To manage your user profile:
+
+1. Click on your username in the top-right corner
+2. Select **Profile Settings**
+3. Here you can:
+   - Update your name and email address
+   - Change your password
+   - Set your timezone and date format preferences
+   - Configure UI theme preferences (light/dark mode)
+
+### System Settings
+
+System-wide settings are available to administrators:
+
+1. Navigate to **Settings** > **System**
+2. Configure the following options:
+   - **Instance Name**: Custom name for your DMARQ instance
+   - **Logo**: Upload a custom logo for branding
+   - **Session Timeout**: How long before inactive users are logged out
+   - **Default Language**: Set the default interface language
+
+## Notification Settings
+
+### Email Notifications
+
+Configure how you receive email notifications:
+
+1. Navigate to **Settings** > **Notifications** > **Email**
+2. Configure the following:
+   - **Email Address**: Where notifications will be sent
+   - **Notification Frequency**: Immediate, daily digest, or weekly summary
+   - **Notification Types**: Select which events trigger notifications
+
+### Alert Thresholds
+
+Set thresholds for when alerts are triggered:
+
+1. Navigate to **Settings** > **Notifications** > **Thresholds**
+2. Configure thresholds for:
+   - **Compliance Rate Drop**: Alert when compliance falls below a threshold
+   - **New Sending Sources**: Alert when new IPs/servers send email as your domain
+   - **Authentication Failures**: Alert when failures exceed a certain number
+   - **Report Processing Issues**: Alert on report processing errors
+
+### Integration Notifications
+
+If you've enabled additional notification channels through Apprise:
+
+1. Navigate to **Settings** > **Notifications** > **Integrations**
+2. Configure each integration separately (Slack, Teams, Discord, etc.)
+3. Set which notification types go to each channel
+
+## API Access
+
+DMARQ provides an API for integration with other systems:
+
+1. Navigate to **Settings** > **API Access**
+2. Here you can:
+   - Generate API keys
+   - View and revoke existing keys
+   - Set permissions and access levels for each key
+   - View API usage statistics
+
+## Integrations
+
+### Cloudflare Integration
+
+If you use Cloudflare for DNS management:
+
+1. Navigate to **Settings** > **Integrations** > **Cloudflare**
+2. Configure:
+   - **API Token**: Your Cloudflare API token
+   - **Zone ID**: The Cloudflare Zone ID for your domain
+   - **Permissions**: What actions DMARQ can take on your DNS records
+
+### Other Integrations
+
+DMARQ supports additional integrations:
+
+1. Navigate to **Settings** > **Integrations**
+2. Select the integration you wish to configure
+3. Follow the specific setup instructions for that integration
+
+## Backup and Data Management
+
+### Data Retention
+
+Configure how long DMARQ keeps data:
+
+1. Navigate to **Settings** > **Data Management**
+2. Configure retention periods for:
+   - **Aggregate Reports**: How long to keep aggregate report data
+   - **Forensic Reports**: How long to keep forensic report data
+   - **Activity Logs**: How long to keep system activity logs
+
+### Backup Configuration
+
+Set up automated backups:
+
+1. Navigate to **Settings** > **Data Management** > **Backups**
+2. Configure:
+   - **Backup Schedule**: How often to create backups
+   - **Backup Location**: Where to store backups (local, S3, etc.)
+   - **Retention**: How many backups to keep
+
+### Data Export
+
+Configure scheduled data exports:
+
+1. Navigate to **Settings** > **Data Management** > **Exports**
+2. Configure scheduled exports to CSV or JSON format
+3. Set delivery methods (download, email, FTP, etc.)
+
+## System Logs
+
+View and manage system logs:
+
+1. Navigate to **Settings** > **Logs**
+2. Filter logs by:
+   - **Log Level**: Error, Warning, Info, Debug
+   - **Component**: API, Parser, IMAP, Authentication, etc.
+   - **Time Range**: When the logs were generated
+3. Download logs for external analysis if needed
+
+## Advanced Settings
+
+Advanced configuration options (administrators only):
+
+1. Navigate to **Settings** > **Advanced**
+2. Configure:
+   - **Database Connection**: Change database settings
+   - **Worker Configuration**: Configure background processing settings
+   - **Caching**: Adjust cache settings for performance
+   - **Debug Mode**: Enable additional logging for troubleshooting