gitea-admin/gh-christianlouis-dmarq
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Docker and manual installation guides, and comprehensive settings documentation for DMARQ

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-04-21 01:53:38 +02:00
parent f7a51d499a
commit e7843790cf
4 changed files with 771 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.readthedocs.yaml
+1 -4
View File
@@ -10,7 +10,4 @@ mkdocs:
python:
install:
- requirements: docs/readthedocs/requirements.txt
sphinx:
fail_on_warning: true
- requirements: docs/readthedocs/requirements.txt
docs/deployment/docker.md
+295
View File
@@ -0,0 +1,295 @@
# Docker Setup
This guide covers how to deploy DMARQ using Docker, which is the recommended deployment method.
## Prerequisites
Before deploying DMARQ with Docker, ensure you have:
- Docker Engine 20.10.0 or later
- Docker Compose v2.0.0 or later
- 2GB RAM minimum (4GB recommended)
- 20GB storage space
## Quick Start
The fastest way to get DMARQ running is to use Docker Compose:
1. **Clone the repository**
```bash
git clone https://github.com/yourusername/dmarq.git
cd dmarq
```
2. **Configure environment variables**
Create a `.env` file in the project root:
```
# Database Configuration
DB_TYPE=sqlite # or postgres for production
DB_PATH=./data/dmarq.db # for SQLite
# For PostgreSQL:
# DB_HOST=postgres
# DB_PORT=5432
# DB_USER=dmarq
# DB_PASS=secure_password
# DB_NAME=dmarq
# IMAP Configuration (optional)
IMAP_ENABLED=false
# IMAP_SERVER=mail.example.com
# IMAP_PORT=993
# IMAP_USERNAME=dmarc@example.com
# IMAP_PASSWORD=your_secure_password
# IMAP_USE_SSL=true
# IMAP_POLLING_INTERVAL=60
# Security Settings
SECRET_KEY=generate_a_secure_random_key
ALLOWED_HOSTS=localhost,127.0.0.1
```
Generate a secure random key for `SECRET_KEY`:
```bash
openssl rand -hex 32
```
3. **Start the containers**
```bash
docker-compose up -d
```
4. **Access the application**
Open your browser and navigate to `http://localhost:8000`
## Understanding the Docker Setup
The `docker-compose.yml` file defines the following services:
- **backend**: The FastAPI application that handles API requests, processes reports, and serves the web interface
- **db**: A PostgreSQL database container (when using Postgres instead of SQLite)
### Docker Compose File Structure
The `docker-compose.yml` file looks like this:
```yaml
version: '3.8'
services:
backend:
build:
context: ./backend
ports:
- "8000:8000"
volumes:
- ./data:/app/data
environment:
- DB_TYPE=${DB_TYPE:-sqlite}
- DB_PATH=${DB_PATH:-./data/dmarq.db}
- DB_HOST=${DB_HOST:-postgres}
- DB_PORT=${DB_PORT:-5432}
- DB_USER=${DB_USER:-dmarq}
- DB_PASS=${DB_PASS:-dmarqpassword}
- DB_NAME=${DB_NAME:-dmarq}
- IMAP_ENABLED=${IMAP_ENABLED:-false}
- IMAP_SERVER=${IMAP_SERVER:-}
- IMAP_PORT=${IMAP_PORT:-993}
- IMAP_USERNAME=${IMAP_USERNAME:-}
- IMAP_PASSWORD=${IMAP_PASSWORD:-}
- IMAP_USE_SSL=${IMAP_USE_SSL:-true}
- IMAP_POLLING_INTERVAL=${IMAP_POLLING_INTERVAL:-60}
- SECRET_KEY=${SECRET_KEY:-insecure_key_change_me_in_production}
- ALLOWED_HOSTS=${ALLOWED_HOSTS:-localhost,127.0.0.1}
depends_on:
- db
restart: unless-stopped
db:
image: postgres:14-alpine
volumes:
- postgres_data:/var/lib/postgresql/data
environment:
- POSTGRES_USER=${DB_USER:-dmarq}
- POSTGRES_PASSWORD=${DB_PASS:-dmarqpassword}
- POSTGRES_DB=${DB_NAME:-dmarq}
restart: unless-stopped
volumes:
postgres_data:
```
## Configuration Options
### Environment Variables
All configuration in the Docker setup is done via environment variables, either directly in the `docker-compose.yml` file or through a separate `.env` file. See the [Configuration](configuration.md) page for detailed information about all available variables.
### Volumes
The Docker Compose setup uses these volumes:
- **./data**: Local directory mapped to `/app/data` in the container, stores SQLite database (if used) and other persistent data
- **postgres_data**: Docker volume for PostgreSQL data (when using Postgres)
## Production Deployment
For production deployments, consider these additional steps:
### Using a Reverse Proxy
In production, it's recommended to use a reverse proxy like Nginx or Traefik in front of DMARQ:
```yaml
version: '3.8'
services:
# ...existing services...
nginx:
image: nginx:alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx/conf.d:/etc/nginx/conf.d
- ./nginx/ssl:/etc/nginx/ssl
depends_on:
- backend
restart: unless-stopped
```
Example Nginx configuration:
```nginx
server {
listen 80;
server_name dmarq.example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl;
server_name dmarq.example.com;
ssl_certificate /etc/nginx/ssl/cert.pem;
ssl_certificate_key /etc/nginx/ssl/key.pem;
location / {
proxy_pass http://backend:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
```
### Docker Compose Profiles
For more complex deployments, you can use Docker Compose profiles:
```yaml
version: '3.8'
services:
backend:
# ...existing config...
profiles: [app, all]
db:
# ...existing config...
profiles: [app, all]
nginx:
# ...nginx config...
profiles: [production, all]
```
Then start only specific profiles:
```bash
docker-compose --profile production up -d
```
## Updating DMARQ
To update to a newer version:
```bash
# Pull the latest code
git pull
# Stop the containers
docker-compose down
# Rebuild and start
docker-compose up -d --build
```
## Monitoring and Maintenance
### Viewing Logs
To view logs from the containers:
```bash
# All logs
docker-compose logs
# Just backend logs
docker-compose logs backend
# Follow logs in real-time
docker-compose logs -f
```
### Container Health Checks
Monitor the health of your containers:
```bash
docker-compose ps
```
### Database Backups
For PostgreSQL backups:
```bash
# Create a backup
docker-compose exec db pg_dump -U dmarq dmarq > backup_$(date +%Y%m%d).sql
# Restore from a backup
cat backup_file.sql | docker-compose exec -T db psql -U dmarq dmarq
```
## Troubleshooting
### Container Won't Start
If containers fail to start:
1. Check logs: `docker-compose logs backend`
2. Verify environment variables: `docker-compose config`
3. Check disk space: `df -h`
4. Ensure ports aren't already in use: `netstat -tuln | grep 8000`
### Database Connection Issues
If the application can't connect to the database:
1. Check the DB environment variables in `.env`
2. For Postgres, ensure the `db` service is running: `docker-compose ps db`
3. Try connecting manually: `docker-compose exec db psql -U dmarq dmarq`
### Mounting Issues
If you encounter volume mounting problems:
1. Check file permissions on the host
2. Use absolute paths in your volume mappings
3. On Windows, ensure you've enabled Docker file sharing for the relevant drives
docs/deployment/manual.md
+333
View File
@@ -0,0 +1,333 @@
# Manual Installation
This guide covers how to deploy DMARQ without Docker, using a traditional installation method.
## Prerequisites
Before proceeding with a manual installation, ensure you have:
- Python 3.9 or higher
- pip and virtualenv
- Node.js 16+ (if modifying frontend assets)
- PostgreSQL (recommended for production) or SQLite
- A web server like Nginx (for production)
## Installation Steps
### 1. Set Up the Environment
First, clone the repository and set up a virtual environment:
```bash
# Clone the repository
git clone https://github.com/yourusername/dmarq.git
cd dmarq
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
### 2. Install Dependencies
Install the required Python packages:
```bash
cd backend
pip install -r requirements.txt
```
### 3. Configure Environment Variables
Create a `.env` file in the backend directory with your configuration:
```
# Database Configuration
DB_TYPE=sqlite # or postgres for production
DB_PATH=./data/dmarq.db # for SQLite
# For PostgreSQL:
# DB_HOST=localhost
# DB_PORT=5432
# DB_USER=dmarq
# DB_PASS=secure_password
# DB_NAME=dmarq
# IMAP Configuration (optional)
IMAP_ENABLED=false
# IMAP_SERVER=mail.example.com
# IMAP_PORT=993
# IMAP_USERNAME=dmarc@example.com
# IMAP_PASSWORD=your_secure_password
# IMAP_USE_SSL=true
# IMAP_POLLING_INTERVAL=60
# Security Settings
SECRET_KEY=generate_a_secure_random_key
ALLOWED_HOSTS=localhost,127.0.0.1
```
Generate a secure random key for `SECRET_KEY`:
```bash
openssl rand -hex 32
```
### 4. Initialize the Database
For SQLite:
```bash
# Create the data directory
mkdir -p data
# Initialize the database
cd app
python -m alembic upgrade head
```
For PostgreSQL:
```bash
# Create the database and user in PostgreSQL
sudo -u postgres psql -c "CREATE USER dmarq WITH PASSWORD 'secure_password';"
sudo -u postgres psql -c "CREATE DATABASE dmarq OWNER dmarq;"
# Initialize the database
cd app
python -m alembic upgrade head
```
### 5. Start the Application (Development)
For development or testing, you can run the application directly with Uvicorn:
```bash
cd app
uvicorn main:app --host 0.0.0.0 --port 8000 --reload
```
### 6. Production Deployment with Systemd
For a production environment, it's recommended to use a process manager like systemd:
1. Create a systemd service file:
```bash
sudo nano /etc/systemd/system/dmarq.service
```
2. Add the following configuration:
```
[Unit]
Description=DMARQ Application
After=network.target
[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env
[Install]
WantedBy=multi-user.target
```
3. Start and enable the service:
```bash
sudo systemctl daemon-reload
sudo systemctl start dmarq
sudo systemctl enable dmarq
```
### 7. Set Up Nginx as a Reverse Proxy
For production, it's recommended to use Nginx as a reverse proxy:
1. Install Nginx:
```bash
sudo apt install nginx
```
2. Create a Nginx configuration file:
```bash
sudo nano /etc/nginx/sites-available/dmarq
```
3. Add the following configuration:
```nginx
server {
listen 80;
server_name dmarq.example.com;
location / {
proxy_pass http://127.0.0.1:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
4. Enable the site and reload Nginx:
```bash
sudo ln -s /etc/nginx/sites-available/dmarq /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```
### 8. Set Up HTTPS with Let's Encrypt
For production, you should secure your site with HTTPS:
```bash
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d dmarq.example.com
```
## Background Tasks
DMARQ requires background tasks for IMAP polling and report processing. For simple deployments, the built-in background task system in FastAPI is sufficient.
For more complex deployments, you might want to set up Celery:
1. Install Celery:
```bash
pip install celery redis
```
2. Create a Celery service file:
```bash
sudo nano /etc/systemd/system/dmarq-celery.service
```
3. Add the following configuration:
```
[Unit]
Description=DMARQ Celery Worker
After=network.target
[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/celery -A worker worker --loglevel=info
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env
[Install]
WantedBy=multi-user.target
```
4. Start and enable the service:
```bash
sudo systemctl daemon-reload
sudo systemctl start dmarq-celery
sudo systemctl enable dmarq-celery
```
## Updating DMARQ
To update to a newer version:
```bash
# Pull the latest code
cd /path/to/dmarq
git pull
# Activate the virtual environment
source venv/bin/activate
# Update dependencies
cd backend
pip install -r requirements.txt
# Apply any database migrations
cd app
python -m alembic upgrade head
# Restart the service
sudo systemctl restart dmarq
```
## Troubleshooting
### Application Won't Start
If the application fails to start:
1. Check the systemd logs: `sudo journalctl -u dmarq`
2. Verify the environment variables in your `.env` file
3. Check that all Python dependencies are installed: `pip list | grep -E 'fastapi|uvicorn'`
### Database Connection Issues
If the application can't connect to the database:
1. Check the DB environment variables in `.env`
2. For PostgreSQL, verify the database exists: `sudo -u postgres psql -c "\l" | grep dmarq`
3. Check if you can connect manually: `psql -U dmarq -h localhost dmarq`
### Nginx Configuration Issues
If Nginx isn't serving the application:
1. Check Nginx error logs: `sudo tail -f /var/log/nginx/error.log`
2. Verify the Nginx configuration: `sudo nginx -t`
3. Make sure the application is running: `curl http://localhost:8000`
## Monitoring and Maintenance
### Checking Application Status
Check if the application is running:
```bash
sudo systemctl status dmarq
```
### Viewing Application Logs
View logs from the application:
```bash
# System logs
sudo journalctl -u dmarq
# Application logs (if configured to log to file)
tail -f /path/to/dmarq/logs/dmarq.log
```
### Database Backups
For PostgreSQL backups:
```bash
# Create a backup
pg_dump -U dmarq dmarq > dmarq_backup_$(date +%Y%m%d).sql
# Restore from a backup
psql -U dmarq dmarq < dmarq_backup_file.sql
```
For SQLite backups:
```bash
# Create a backup
sqlite3 /path/to/data/dmarq.db .dump > dmarq_backup_$(date +%Y%m%d).sql
# Restore from a backup
cat dmarq_backup_file.sql | sqlite3 /path/to/data/dmarq.db
```
docs/user_guide/settings.md
+142
View File
@@ -0,0 +1,142 @@
# Settings
This guide covers the various settings and configuration options available in DMARQ.
## General Settings
### User Profile
To manage your user profile:
1. Click on your username in the top-right corner
2. Select **Profile Settings**
3. Here you can:
- Update your name and email address
- Change your password
- Set your timezone and date format preferences
- Configure UI theme preferences (light/dark mode)
### System Settings
System-wide settings are available to administrators:
1. Navigate to **Settings** > **System**
2. Configure the following options:
- **Instance Name**: Custom name for your DMARQ instance
- **Logo**: Upload a custom logo for branding
- **Session Timeout**: How long before inactive users are logged out
- **Default Language**: Set the default interface language
## Notification Settings
### Email Notifications
Configure how you receive email notifications:
1. Navigate to **Settings** > **Notifications** > **Email**
2. Configure the following:
- **Email Address**: Where notifications will be sent
- **Notification Frequency**: Immediate, daily digest, or weekly summary
- **Notification Types**: Select which events trigger notifications
### Alert Thresholds
Set thresholds for when alerts are triggered:
1. Navigate to **Settings** > **Notifications** > **Thresholds**
2. Configure thresholds for:
- **Compliance Rate Drop**: Alert when compliance falls below a threshold
- **New Sending Sources**: Alert when new IPs/servers send email as your domain
- **Authentication Failures**: Alert when failures exceed a certain number
- **Report Processing Issues**: Alert on report processing errors
### Integration Notifications
If you've enabled additional notification channels through Apprise:
1. Navigate to **Settings** > **Notifications** > **Integrations**
2. Configure each integration separately (Slack, Teams, Discord, etc.)
3. Set which notification types go to each channel
## API Access
DMARQ provides an API for integration with other systems:
1. Navigate to **Settings** > **API Access**
2. Here you can:
- Generate API keys
- View and revoke existing keys
- Set permissions and access levels for each key
- View API usage statistics
## Integrations
### Cloudflare Integration
If you use Cloudflare for DNS management:
1. Navigate to **Settings** > **Integrations** > **Cloudflare**
2. Configure:
- **API Token**: Your Cloudflare API token
- **Zone ID**: The Cloudflare Zone ID for your domain
- **Permissions**: What actions DMARQ can take on your DNS records
### Other Integrations
DMARQ supports additional integrations:
1. Navigate to **Settings** > **Integrations**
2. Select the integration you wish to configure
3. Follow the specific setup instructions for that integration
## Backup and Data Management
### Data Retention
Configure how long DMARQ keeps data:
1. Navigate to **Settings** > **Data Management**
2. Configure retention periods for:
- **Aggregate Reports**: How long to keep aggregate report data
- **Forensic Reports**: How long to keep forensic report data
- **Activity Logs**: How long to keep system activity logs
### Backup Configuration
Set up automated backups:
1. Navigate to **Settings** > **Data Management** > **Backups**
2. Configure:
- **Backup Schedule**: How often to create backups
- **Backup Location**: Where to store backups (local, S3, etc.)
- **Retention**: How many backups to keep
### Data Export
Configure scheduled data exports:
1. Navigate to **Settings** > **Data Management** > **Exports**
2. Configure scheduled exports to CSV or JSON format
3. Set delivery methods (download, email, FTP, etc.)
## System Logs
View and manage system logs:
1. Navigate to **Settings** > **Logs**
2. Filter logs by:
- **Log Level**: Error, Warning, Info, Debug
- **Component**: API, Parser, IMAP, Authentication, etc.
- **Time Range**: When the logs were generated
3. Download logs for external analysis if needed
## Advanced Settings
Advanced configuration options (administrators only):
1. Navigate to **Settings** > **Advanced**
2. Configure:
- **Database Connection**: Change database settings
- **Worker Configuration**: Configure background processing settings
- **Caching**: Adjust cache settings for performance
- **Debug Mode**: Enable additional logging for troubleshooting