gh-christianlouis-dmarq/docs/deployment/manual.md

Manual Installation

This guide covers how to deploy DMARQ without Docker, using a traditional installation method.

Prerequisites

Before proceeding with a manual installation, ensure you have:

• Python 3.13 or higher
• pip and virtualenv
• Node.js 16+ (if modifying frontend assets)
• PostgreSQL (recommended for production) or SQLite
• A web server like Nginx (for production)

Installation Steps

1. Set Up the Environment

First, clone the repository and set up a virtual environment:

# Clone the repository
git clone https://github.com/yourusername/dmarq.git
cd dmarq

# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

2. Install Dependencies

Install the required Python packages:

cd backend
pip install -r requirements.txt

3. Configure Environment Variables

Create a .env file in the backend directory with your configuration. For production, prefer a 1Password-mounted .env file; see Secret Handling with 1Password.

# Database Configuration
DB_TYPE=sqlite  # or postgres for production
DB_PATH=./data/dmarq.db  # for SQLite
# For PostgreSQL:
# DB_HOST=localhost
# DB_PORT=5432
# DB_USER=dmarq
# DB_PASS=secure_password
# DB_NAME=dmarq

# IMAP Configuration (optional)
IMAP_ENABLED=false
# IMAP_SERVER=mail.example.com
# IMAP_PORT=993
# IMAP_USERNAME=dmarc@example.com
# IMAP_PASSWORD=your_secure_password
# IMAP_USE_SSL=true
# IMAP_POLLING_INTERVAL=60

# Security Settings
SECRET_KEY=generate_a_secure_random_key
ALLOWED_HOSTS=localhost,127.0.0.1

Generate a secure random key for SECRET_KEY:

openssl rand -hex 32

4. Initialize the Database

For SQLite:

# Create the data directory
mkdir -p data

# Initialize the database
cd app
python -m alembic upgrade head

For PostgreSQL:

# Create the database and user in PostgreSQL
sudo -u postgres psql -c "CREATE USER dmarq WITH PASSWORD 'secure_password';"
sudo -u postgres psql -c "CREATE DATABASE dmarq OWNER dmarq;"

# Initialize the database
cd app
python -m alembic upgrade head

5. Start the Application (Development)

For development or testing, you can run the application directly with Uvicorn:

cd app
uvicorn main:app --host 0.0.0.0 --port 8000 --reload

6. Production Deployment with Systemd

For a production environment, it's recommended to use a process manager like systemd:

1. Create a systemd service file:

sudo nano /etc/systemd/system/dmarq.service

2. Add the following configuration:

[Unit]
Description=DMARQ Application
After=network.target

[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env

[Install]
WantedBy=multi-user.target

3. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start dmarq
sudo systemctl enable dmarq

7. Set Up Nginx as a Reverse Proxy

For production, it's recommended to use Nginx as a reverse proxy:

1. Install Nginx:

sudo apt install nginx

2. Create a Nginx configuration file:

sudo nano /etc/nginx/sites-available/dmarq

3. Add the following configuration:

server {
    listen 80;
    server_name dmarq.example.com;
    
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

4. Enable the site and reload Nginx:

sudo ln -s /etc/nginx/sites-available/dmarq /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx

8. Set Up HTTPS with Let's Encrypt

For production, you should secure your site with HTTPS:

sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d dmarq.example.com

Background Tasks

DMARQ requires background tasks for IMAP polling and report processing. For simple deployments, the built-in background task system in FastAPI is sufficient.

For more complex deployments, you might want to set up Celery:

1. Install Celery:

pip install celery redis

2. Create a Celery service file:

sudo nano /etc/systemd/system/dmarq-celery.service

3. Add the following configuration:

[Unit]
Description=DMARQ Celery Worker
After=network.target

[Service]
User=dmarq
WorkingDirectory=/path/to/dmarq/backend/app
ExecStart=/path/to/dmarq/venv/bin/celery -A worker worker --loglevel=info
Restart=always
Environment="PATH=/path/to/dmarq/venv/bin"
EnvironmentFile=/path/to/dmarq/backend/.env

[Install]
WantedBy=multi-user.target

4. Start and enable the service:

sudo systemctl daemon-reload
sudo systemctl start dmarq-celery
sudo systemctl enable dmarq-celery

Updating DMARQ

Before updating a production deployment, follow the Release Checklist and create a database backup.

To update to a newer version:

# Pull the latest code
cd /path/to/dmarq
git pull

# Activate the virtual environment
source venv/bin/activate

# Update dependencies
cd backend
pip install -r requirements.txt

# Apply any database migrations
cd app
python -m alembic upgrade head

# Restart the service
sudo systemctl restart dmarq

Troubleshooting

Application Won't Start

If the application fails to start:

1. Check the systemd logs: sudo journalctl -u dmarq
2. Verify the environment variables in your .env file
3. Check that all Python dependencies are installed: pip list | grep -E 'fastapi|uvicorn'

Database Connection Issues

If the application can't connect to the database:

1. Check the DB environment variables in .env
2. For PostgreSQL, verify the database exists: sudo -u postgres psql -c "\l" | grep dmarq
3. Check if you can connect manually: psql -U dmarq -h localhost dmarq

Nginx Configuration Issues

If Nginx isn't serving the application:

1. Check Nginx error logs: sudo tail -f /var/log/nginx/error.log
2. Verify the Nginx configuration: sudo nginx -t
3. Make sure the application is running: curl http://localhost:8000

Monitoring and Maintenance

Checking Application Status

Check if the application is running:

sudo systemctl status dmarq

Viewing Application Logs

View logs from the application:

# System logs
sudo journalctl -u dmarq

# Application logs (if configured to log to file)
tail -f /path/to/dmarq/logs/dmarq.log

Database Backups

Back up the database before upgrades and on a regular schedule. See Database Backup and Restore for SQLite and PostgreSQL backup, restore, and verification commands.