diff --git a/.readthedocs.yaml b/.readthedocs.yaml index b8d4089..ecf807c 100644 --- a/.readthedocs.yaml +++ b/.readthedocs.yaml @@ -10,7 +10,4 @@ mkdocs: python: install: - - requirements: docs/readthedocs/requirements.txt - -sphinx: - fail_on_warning: true \ No newline at end of file + - requirements: docs/readthedocs/requirements.txt \ No newline at end of file diff --git a/docs/deployment/docker.md b/docs/deployment/docker.md new file mode 100644 index 0000000..ec6ac47 --- /dev/null +++ b/docs/deployment/docker.md @@ -0,0 +1,295 @@ +# Docker Setup + +This guide covers how to deploy DMARQ using Docker, which is the recommended deployment method. + +## Prerequisites + +Before deploying DMARQ with Docker, ensure you have: + +- Docker Engine 20.10.0 or later +- Docker Compose v2.0.0 or later +- 2GB RAM minimum (4GB recommended) +- 20GB storage space + +## Quick Start + +The fastest way to get DMARQ running is to use Docker Compose: + +1. **Clone the repository** + + ```bash + git clone https://github.com/yourusername/dmarq.git + cd dmarq + ``` + +2. **Configure environment variables** + + Create a `.env` file in the project root: + + ``` + # Database Configuration + DB_TYPE=sqlite # or postgres for production + DB_PATH=./data/dmarq.db # for SQLite + # For PostgreSQL: + # DB_HOST=postgres + # DB_PORT=5432 + # DB_USER=dmarq + # DB_PASS=secure_password + # DB_NAME=dmarq + + # IMAP Configuration (optional) + IMAP_ENABLED=false + # IMAP_SERVER=mail.example.com + # IMAP_PORT=993 + # IMAP_USERNAME=dmarc@example.com + # IMAP_PASSWORD=your_secure_password + # IMAP_USE_SSL=true + # IMAP_POLLING_INTERVAL=60 + + # Security Settings + SECRET_KEY=generate_a_secure_random_key + ALLOWED_HOSTS=localhost,127.0.0.1 + ``` + + Generate a secure random key for `SECRET_KEY`: + + ```bash + openssl rand -hex 32 + ``` + +3. **Start the containers** + + ```bash + docker-compose up -d + ``` + +4. **Access the application** + + Open your browser and navigate to `http://localhost:8000` + +## Understanding the Docker Setup + +The `docker-compose.yml` file defines the following services: + +- **backend**: The FastAPI application that handles API requests, processes reports, and serves the web interface +- **db**: A PostgreSQL database container (when using Postgres instead of SQLite) + +### Docker Compose File Structure + +The `docker-compose.yml` file looks like this: + +```yaml +version: '3.8' + +services: + backend: + build: + context: ./backend + ports: + - "8000:8000" + volumes: + - ./data:/app/data + environment: + - DB_TYPE=${DB_TYPE:-sqlite} + - DB_PATH=${DB_PATH:-./data/dmarq.db} + - DB_HOST=${DB_HOST:-postgres} + - DB_PORT=${DB_PORT:-5432} + - DB_USER=${DB_USER:-dmarq} + - DB_PASS=${DB_PASS:-dmarqpassword} + - DB_NAME=${DB_NAME:-dmarq} + - IMAP_ENABLED=${IMAP_ENABLED:-false} + - IMAP_SERVER=${IMAP_SERVER:-} + - IMAP_PORT=${IMAP_PORT:-993} + - IMAP_USERNAME=${IMAP_USERNAME:-} + - IMAP_PASSWORD=${IMAP_PASSWORD:-} + - IMAP_USE_SSL=${IMAP_USE_SSL:-true} + - IMAP_POLLING_INTERVAL=${IMAP_POLLING_INTERVAL:-60} + - SECRET_KEY=${SECRET_KEY:-insecure_key_change_me_in_production} + - ALLOWED_HOSTS=${ALLOWED_HOSTS:-localhost,127.0.0.1} + depends_on: + - db + restart: unless-stopped + + db: + image: postgres:14-alpine + volumes: + - postgres_data:/var/lib/postgresql/data + environment: + - POSTGRES_USER=${DB_USER:-dmarq} + - POSTGRES_PASSWORD=${DB_PASS:-dmarqpassword} + - POSTGRES_DB=${DB_NAME:-dmarq} + restart: unless-stopped + +volumes: + postgres_data: +``` + +## Configuration Options + +### Environment Variables + +All configuration in the Docker setup is done via environment variables, either directly in the `docker-compose.yml` file or through a separate `.env` file. See the [Configuration](configuration.md) page for detailed information about all available variables. + +### Volumes + +The Docker Compose setup uses these volumes: + +- **./data**: Local directory mapped to `/app/data` in the container, stores SQLite database (if used) and other persistent data +- **postgres_data**: Docker volume for PostgreSQL data (when using Postgres) + +## Production Deployment + +For production deployments, consider these additional steps: + +### Using a Reverse Proxy + +In production, it's recommended to use a reverse proxy like Nginx or Traefik in front of DMARQ: + +```yaml +version: '3.8' + +services: + # ...existing services... + + nginx: + image: nginx:alpine + ports: + - "80:80" + - "443:443" + volumes: + - ./nginx/conf.d:/etc/nginx/conf.d + - ./nginx/ssl:/etc/nginx/ssl + depends_on: + - backend + restart: unless-stopped +``` + +Example Nginx configuration: + +```nginx +server { + listen 80; + server_name dmarq.example.com; + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl; + server_name dmarq.example.com; + + ssl_certificate /etc/nginx/ssl/cert.pem; + ssl_certificate_key /etc/nginx/ssl/key.pem; + + location / { + proxy_pass http://backend:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + } +} +``` + +### Docker Compose Profiles + +For more complex deployments, you can use Docker Compose profiles: + +```yaml +version: '3.8' + +services: + backend: + # ...existing config... + profiles: [app, all] + + db: + # ...existing config... + profiles: [app, all] + + nginx: + # ...nginx config... + profiles: [production, all] +``` + +Then start only specific profiles: + +```bash +docker-compose --profile production up -d +``` + +## Updating DMARQ + +To update to a newer version: + +```bash +# Pull the latest code +git pull + +# Stop the containers +docker-compose down + +# Rebuild and start +docker-compose up -d --build +``` + +## Monitoring and Maintenance + +### Viewing Logs + +To view logs from the containers: + +```bash +# All logs +docker-compose logs + +# Just backend logs +docker-compose logs backend + +# Follow logs in real-time +docker-compose logs -f +``` + +### Container Health Checks + +Monitor the health of your containers: + +```bash +docker-compose ps +``` + +### Database Backups + +For PostgreSQL backups: + +```bash +# Create a backup +docker-compose exec db pg_dump -U dmarq dmarq > backup_$(date +%Y%m%d).sql + +# Restore from a backup +cat backup_file.sql | docker-compose exec -T db psql -U dmarq dmarq +``` + +## Troubleshooting + +### Container Won't Start + +If containers fail to start: + +1. Check logs: `docker-compose logs backend` +2. Verify environment variables: `docker-compose config` +3. Check disk space: `df -h` +4. Ensure ports aren't already in use: `netstat -tuln | grep 8000` + +### Database Connection Issues + +If the application can't connect to the database: + +1. Check the DB environment variables in `.env` +2. For Postgres, ensure the `db` service is running: `docker-compose ps db` +3. Try connecting manually: `docker-compose exec db psql -U dmarq dmarq` + +### Mounting Issues + +If you encounter volume mounting problems: + +1. Check file permissions on the host +2. Use absolute paths in your volume mappings +3. On Windows, ensure you've enabled Docker file sharing for the relevant drives \ No newline at end of file diff --git a/docs/deployment/manual.md b/docs/deployment/manual.md new file mode 100644 index 0000000..14385a4 --- /dev/null +++ b/docs/deployment/manual.md @@ -0,0 +1,333 @@ +# Manual Installation + +This guide covers how to deploy DMARQ without Docker, using a traditional installation method. + +## Prerequisites + +Before proceeding with a manual installation, ensure you have: + +- Python 3.9 or higher +- pip and virtualenv +- Node.js 16+ (if modifying frontend assets) +- PostgreSQL (recommended for production) or SQLite +- A web server like Nginx (for production) + +## Installation Steps + +### 1. Set Up the Environment + +First, clone the repository and set up a virtual environment: + +```bash +# Clone the repository +git clone https://github.com/yourusername/dmarq.git +cd dmarq + +# Create and activate a virtual environment +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate +``` + +### 2. Install Dependencies + +Install the required Python packages: + +```bash +cd backend +pip install -r requirements.txt +``` + +### 3. Configure Environment Variables + +Create a `.env` file in the backend directory with your configuration: + +``` +# Database Configuration +DB_TYPE=sqlite # or postgres for production +DB_PATH=./data/dmarq.db # for SQLite +# For PostgreSQL: +# DB_HOST=localhost +# DB_PORT=5432 +# DB_USER=dmarq +# DB_PASS=secure_password +# DB_NAME=dmarq + +# IMAP Configuration (optional) +IMAP_ENABLED=false +# IMAP_SERVER=mail.example.com +# IMAP_PORT=993 +# IMAP_USERNAME=dmarc@example.com +# IMAP_PASSWORD=your_secure_password +# IMAP_USE_SSL=true +# IMAP_POLLING_INTERVAL=60 + +# Security Settings +SECRET_KEY=generate_a_secure_random_key +ALLOWED_HOSTS=localhost,127.0.0.1 +``` + +Generate a secure random key for `SECRET_KEY`: + +```bash +openssl rand -hex 32 +``` + +### 4. Initialize the Database + +For SQLite: + +```bash +# Create the data directory +mkdir -p data + +# Initialize the database +cd app +python -m alembic upgrade head +``` + +For PostgreSQL: + +```bash +# Create the database and user in PostgreSQL +sudo -u postgres psql -c "CREATE USER dmarq WITH PASSWORD 'secure_password';" +sudo -u postgres psql -c "CREATE DATABASE dmarq OWNER dmarq;" + +# Initialize the database +cd app +python -m alembic upgrade head +``` + +### 5. Start the Application (Development) + +For development or testing, you can run the application directly with Uvicorn: + +```bash +cd app +uvicorn main:app --host 0.0.0.0 --port 8000 --reload +``` + +### 6. Production Deployment with Systemd + +For a production environment, it's recommended to use a process manager like systemd: + +1. Create a systemd service file: + +```bash +sudo nano /etc/systemd/system/dmarq.service +``` + +2. Add the following configuration: + +``` +[Unit] +Description=DMARQ Application +After=network.target + +[Service] +User=dmarq +WorkingDirectory=/path/to/dmarq/backend/app +ExecStart=/path/to/dmarq/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000 +Restart=always +Environment="PATH=/path/to/dmarq/venv/bin" +EnvironmentFile=/path/to/dmarq/backend/.env + +[Install] +WantedBy=multi-user.target +``` + +3. Start and enable the service: + +```bash +sudo systemctl daemon-reload +sudo systemctl start dmarq +sudo systemctl enable dmarq +``` + +### 7. Set Up Nginx as a Reverse Proxy + +For production, it's recommended to use Nginx as a reverse proxy: + +1. Install Nginx: + +```bash +sudo apt install nginx +``` + +2. Create a Nginx configuration file: + +```bash +sudo nano /etc/nginx/sites-available/dmarq +``` + +3. Add the following configuration: + +```nginx +server { + listen 80; + server_name dmarq.example.com; + + location / { + proxy_pass http://127.0.0.1:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +4. Enable the site and reload Nginx: + +```bash +sudo ln -s /etc/nginx/sites-available/dmarq /etc/nginx/sites-enabled/ +sudo nginx -t +sudo systemctl reload nginx +``` + +### 8. Set Up HTTPS with Let's Encrypt + +For production, you should secure your site with HTTPS: + +```bash +sudo apt install certbot python3-certbot-nginx +sudo certbot --nginx -d dmarq.example.com +``` + +## Background Tasks + +DMARQ requires background tasks for IMAP polling and report processing. For simple deployments, the built-in background task system in FastAPI is sufficient. + +For more complex deployments, you might want to set up Celery: + +1. Install Celery: + +```bash +pip install celery redis +``` + +2. Create a Celery service file: + +```bash +sudo nano /etc/systemd/system/dmarq-celery.service +``` + +3. Add the following configuration: + +``` +[Unit] +Description=DMARQ Celery Worker +After=network.target + +[Service] +User=dmarq +WorkingDirectory=/path/to/dmarq/backend/app +ExecStart=/path/to/dmarq/venv/bin/celery -A worker worker --loglevel=info +Restart=always +Environment="PATH=/path/to/dmarq/venv/bin" +EnvironmentFile=/path/to/dmarq/backend/.env + +[Install] +WantedBy=multi-user.target +``` + +4. Start and enable the service: + +```bash +sudo systemctl daemon-reload +sudo systemctl start dmarq-celery +sudo systemctl enable dmarq-celery +``` + +## Updating DMARQ + +To update to a newer version: + +```bash +# Pull the latest code +cd /path/to/dmarq +git pull + +# Activate the virtual environment +source venv/bin/activate + +# Update dependencies +cd backend +pip install -r requirements.txt + +# Apply any database migrations +cd app +python -m alembic upgrade head + +# Restart the service +sudo systemctl restart dmarq +``` + +## Troubleshooting + +### Application Won't Start + +If the application fails to start: + +1. Check the systemd logs: `sudo journalctl -u dmarq` +2. Verify the environment variables in your `.env` file +3. Check that all Python dependencies are installed: `pip list | grep -E 'fastapi|uvicorn'` + +### Database Connection Issues + +If the application can't connect to the database: + +1. Check the DB environment variables in `.env` +2. For PostgreSQL, verify the database exists: `sudo -u postgres psql -c "\l" | grep dmarq` +3. Check if you can connect manually: `psql -U dmarq -h localhost dmarq` + +### Nginx Configuration Issues + +If Nginx isn't serving the application: + +1. Check Nginx error logs: `sudo tail -f /var/log/nginx/error.log` +2. Verify the Nginx configuration: `sudo nginx -t` +3. Make sure the application is running: `curl http://localhost:8000` + +## Monitoring and Maintenance + +### Checking Application Status + +Check if the application is running: + +```bash +sudo systemctl status dmarq +``` + +### Viewing Application Logs + +View logs from the application: + +```bash +# System logs +sudo journalctl -u dmarq + +# Application logs (if configured to log to file) +tail -f /path/to/dmarq/logs/dmarq.log +``` + +### Database Backups + +For PostgreSQL backups: + +```bash +# Create a backup +pg_dump -U dmarq dmarq > dmarq_backup_$(date +%Y%m%d).sql + +# Restore from a backup +psql -U dmarq dmarq < dmarq_backup_file.sql +``` + +For SQLite backups: + +```bash +# Create a backup +sqlite3 /path/to/data/dmarq.db .dump > dmarq_backup_$(date +%Y%m%d).sql + +# Restore from a backup +cat dmarq_backup_file.sql | sqlite3 /path/to/data/dmarq.db +``` \ No newline at end of file diff --git a/docs/user_guide/settings.md b/docs/user_guide/settings.md new file mode 100644 index 0000000..db23a91 --- /dev/null +++ b/docs/user_guide/settings.md @@ -0,0 +1,142 @@ +# Settings + +This guide covers the various settings and configuration options available in DMARQ. + +## General Settings + +### User Profile + +To manage your user profile: + +1. Click on your username in the top-right corner +2. Select **Profile Settings** +3. Here you can: + - Update your name and email address + - Change your password + - Set your timezone and date format preferences + - Configure UI theme preferences (light/dark mode) + +### System Settings + +System-wide settings are available to administrators: + +1. Navigate to **Settings** > **System** +2. Configure the following options: + - **Instance Name**: Custom name for your DMARQ instance + - **Logo**: Upload a custom logo for branding + - **Session Timeout**: How long before inactive users are logged out + - **Default Language**: Set the default interface language + +## Notification Settings + +### Email Notifications + +Configure how you receive email notifications: + +1. Navigate to **Settings** > **Notifications** > **Email** +2. Configure the following: + - **Email Address**: Where notifications will be sent + - **Notification Frequency**: Immediate, daily digest, or weekly summary + - **Notification Types**: Select which events trigger notifications + +### Alert Thresholds + +Set thresholds for when alerts are triggered: + +1. Navigate to **Settings** > **Notifications** > **Thresholds** +2. Configure thresholds for: + - **Compliance Rate Drop**: Alert when compliance falls below a threshold + - **New Sending Sources**: Alert when new IPs/servers send email as your domain + - **Authentication Failures**: Alert when failures exceed a certain number + - **Report Processing Issues**: Alert on report processing errors + +### Integration Notifications + +If you've enabled additional notification channels through Apprise: + +1. Navigate to **Settings** > **Notifications** > **Integrations** +2. Configure each integration separately (Slack, Teams, Discord, etc.) +3. Set which notification types go to each channel + +## API Access + +DMARQ provides an API for integration with other systems: + +1. Navigate to **Settings** > **API Access** +2. Here you can: + - Generate API keys + - View and revoke existing keys + - Set permissions and access levels for each key + - View API usage statistics + +## Integrations + +### Cloudflare Integration + +If you use Cloudflare for DNS management: + +1. Navigate to **Settings** > **Integrations** > **Cloudflare** +2. Configure: + - **API Token**: Your Cloudflare API token + - **Zone ID**: The Cloudflare Zone ID for your domain + - **Permissions**: What actions DMARQ can take on your DNS records + +### Other Integrations + +DMARQ supports additional integrations: + +1. Navigate to **Settings** > **Integrations** +2. Select the integration you wish to configure +3. Follow the specific setup instructions for that integration + +## Backup and Data Management + +### Data Retention + +Configure how long DMARQ keeps data: + +1. Navigate to **Settings** > **Data Management** +2. Configure retention periods for: + - **Aggregate Reports**: How long to keep aggregate report data + - **Forensic Reports**: How long to keep forensic report data + - **Activity Logs**: How long to keep system activity logs + +### Backup Configuration + +Set up automated backups: + +1. Navigate to **Settings** > **Data Management** > **Backups** +2. Configure: + - **Backup Schedule**: How often to create backups + - **Backup Location**: Where to store backups (local, S3, etc.) + - **Retention**: How many backups to keep + +### Data Export + +Configure scheduled data exports: + +1. Navigate to **Settings** > **Data Management** > **Exports** +2. Configure scheduled exports to CSV or JSON format +3. Set delivery methods (download, email, FTP, etc.) + +## System Logs + +View and manage system logs: + +1. Navigate to **Settings** > **Logs** +2. Filter logs by: + - **Log Level**: Error, Warning, Info, Debug + - **Component**: API, Parser, IMAP, Authentication, etc. + - **Time Range**: When the logs were generated +3. Download logs for external analysis if needed + +## Advanced Settings + +Advanced configuration options (administrators only): + +1. Navigate to **Settings** > **Advanced** +2. Configure: + - **Database Connection**: Change database settings + - **Worker Configuration**: Configure background processing settings + - **Caching**: Adjust cache settings for performance + - **Debug Mode**: Enable additional logging for troubleshooting \ No newline at end of file