gh-christianlouis-dmarq/docs/deployment/manual.md

Manual Installation

This guide covers how to deploy DMARQ without Docker, using a traditional installation method.

For production operations after installation, use the Operator Runbook for verification, upgrades, rollback, and routine checks. Use Troubleshooting Playbooks when auth, ingestion, DNS, database, or notification workflows fail.

Prerequisites

Before proceeding with a manual installation, ensure you have:

• Python 3.13 or higher
• pip and virtualenv
• Node.js 16+ (if modifying frontend assets)
• PostgreSQL (recommended for production) or SQLite
• A web server like Nginx (for production)

Installation Steps

1. Set Up the Environment

First, clone the repository and set up a virtual environment:

# Clone the repository
git clone https://github.com/yourusername/dmarq.git
cd dmarq

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Dependencies

Install the required Python packages:

cd backend
pip install -r requirements.txt

3. Configure Environment Variables

Create a .env file in the backend directory with your configuration. For production, prefer a 1Password-mounted .env file; see Secret Handling with 1Password.

# Database Configuration
DB_TYPE=sqlite  # or postgres for production
DB_PATH=./data/dmarq.db  # for SQLite
# For PostgreSQL:
# DB_HOST=localhost
# DB_PORT=5432
# DB_USER=dmarq
# DB_PASS=secure_password
# DB_NAME=dmarq

# IMAP Configuration (optional)
IMAP_ENABLED=false
# IMAP_SERVER=mail.example.com
# IMAP_PORT=993
# IMAP_USERNAME=dmarc@example.com
# IMAP_PASSWORD=your_secure_password
# IMAP_USE_SSL=true
# IMAP_POLLING_INTERVAL=60

# Security Settings
SECRET_KEY=generate_a_secure_random_key
ALLOWED_HOSTS=localhost,127.0.0.1

Generate a secure random key for SECRET_KEY:

openssl rand -hex 32

4. Initialize the Database

For SQLite:

# Create the data directory
mkdir -p data

# Initialize the database
cd app
python -m alembic upgrade head

For PostgreSQL:

# Create the database and user in PostgreSQL
sudo -u postgres psql -c "CREATE USER dmarq WITH PASSWORD 'secure_password';"
sudo -u postgres psql -c "CREATE DATABASE dmarq OWNER dmarq;"

# Initialize the database
cd app
python -m alembic upgrade head

5. Start the Application (Development)

For development or testing, you can run the application directly with Uvicorn:

cd app
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

6. Production Deployment with Systemd

For a production environment, it's recommended to use a process manager like systemd:

1. Create a systemd service file:

sudo nano /etc/systemd/system/dmarq.service

2. Add the following configuration:

[Unit]
Description=DMARQ Application
After=network.target

[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env

[Install]
WantedBy=multi-user.target

3. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start dmarq
sudo systemctl enable dmarq

7. Set Up Nginx as a Reverse Proxy

For production, it's recommended to use Nginx as a reverse proxy:

1. Install Nginx:

sudo apt install nginx

2. Create a Nginx configuration file:

sudo nano /etc/nginx/sites-available/dmarq

3. Add the following configuration:

server {
    listen 80;
    server_name dmarq.example.com;
    
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

4. Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/dmarq /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

8. Set Up HTTPS with Let's Encrypt

For production, you should secure your site with HTTPS:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d dmarq.example.com

Background Tasks

DMARQ requires background tasks for IMAP polling and report processing. For simple deployments, the built-in background task system in FastAPI is sufficient.

For more complex deployments, you might want to set up Celery:

1. Install Celery:

pip install celery redis

2. Create a Celery service file:

sudo nano /etc/systemd/system/dmarq-celery.service

3. Add the following configuration:

[Unit]
Description=DMARQ Celery Worker
After=network.target

[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/celery -A worker worker --loglevel=info
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env

[Install]
WantedBy=multi-user.target

4. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start dmarq-celery
sudo systemctl enable dmarq-celery

Updating DMARQ

Before updating a production deployment, follow the Release Checklist and create a database backup. The full operator upgrade sequence is documented in Operator Runbook.

To update to a newer version:

# Pull the latest code
cd /path/to/dmarq
git pull

# Activate the virtual environment
source venv/bin/activate

# Update dependencies
cd backend
pip install -r requirements.txt

# Apply any database migrations
cd app
python -m alembic upgrade head

# Restart the service
sudo systemctl restart dmarq

Troubleshooting

Application Won't Start

If the application fails to start:

1. Check the systemd logs: sudo journalctl -u dmarq
2. Verify the environment variables in your .env file
3. Check that all Python dependencies are installed: pip list | grep -E 'fastapi|uvicorn'

Database Connection Issues

If the application can't connect to the database:

1. Check the DB environment variables in .env
2. For PostgreSQL, verify the database exists: sudo -u postgres psql -c "\l" | grep dmarq
3. Check if you can connect manually: psql -U dmarq -h localhost dmarq

Nginx Configuration Issues

If Nginx isn't serving the application:

1. Check Nginx error logs: sudo tail -f /var/log/nginx/error.log
2. Verify the Nginx configuration: sudo nginx -t
3. Make sure the application is running: curl http://localhost:8000

Monitoring and Maintenance

Checking Application Status

Check if the application is running:

sudo systemctl status dmarq

Viewing Application Logs

View logs from the application:

# System logs
sudo journalctl -u dmarq

# Application logs (if configured to log to file)
tail -f /path/to/dmarq/logs/dmarq.log

Database Backups

Back up the database before upgrades and on a regular schedule. See Database Backup and Restore for SQLite and PostgreSQL backup, restore, and verification commands.