Merge branch 'main' into copilot/fix-password-length-issue
This commit is contained in:
Christian Krakau-Louis
@@ -0,0 +1,22 @@
|
||||
# EditorConfig — https://editorconfig.org
|
||||
root = true
|
||||
|
||||
[*]
|
||||
indent_style = space
|
||||
indent_size = 4
|
||||
end_of_line = lf
|
||||
charset = utf-8
|
||||
trim_trailing_whitespace = true
|
||||
insert_final_newline = true
|
||||
|
||||
[*.{js,jsx,ts,tsx,json,css,scss,yml,yaml}]
|
||||
indent_size = 2
|
||||
|
||||
[*.md]
|
||||
trim_trailing_whitespace = false
|
||||
|
||||
[Makefile]
|
||||
indent_style = tab
|
||||
|
||||
[Dockerfile*]
|
||||
indent_size = 4
|
||||
@@ -24,6 +24,13 @@ cd frontend
|
||||
npm run lint
|
||||
```
|
||||
|
||||
## Documentation Requirements
|
||||
|
||||
When making changes, always update the following files:
|
||||
|
||||
- **`docs/TODO.md`**: Update task checkboxes, progress percentages, and status indicators to reflect completed work and any new items discovered.
|
||||
- **`CHANGELOG.md`**: Add entries under the `[Unreleased]` section using the appropriate category (`Added`, `Changed`, `Fixed`, `Security`, `Removed`, `Deprecated`).
|
||||
|
||||
## Conventions
|
||||
|
||||
- **Python**: Follow PEP 8. Use `black` for formatting. All ruff and mypy errors must be resolved before committing.
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
version: 2
|
||||
updates:
|
||||
# Python dependencies (backend)
|
||||
- package-ecosystem: "pip"
|
||||
directory: "/backend"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
open-pull-requests-limit: 10
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "python"
|
||||
commit-message:
|
||||
prefix: "chore(deps):"
|
||||
|
||||
# GitHub Actions
|
||||
- package-ecosystem: "github-actions"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "github-actions"
|
||||
commit-message:
|
||||
prefix: "chore(ci):"
|
||||
|
||||
# npm dependencies (frontend)
|
||||
- package-ecosystem: "npm"
|
||||
directory: "/frontend"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
open-pull-requests-limit: 10
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "javascript"
|
||||
commit-message:
|
||||
prefix: "chore(deps):"
|
||||
|
||||
# Docker dependencies
|
||||
- package-ecosystem: "docker"
|
||||
directory: "/"
|
||||
schedule:
|
||||
interval: "weekly"
|
||||
day: "monday"
|
||||
open-pull-requests-limit: 5
|
||||
labels:
|
||||
- "dependencies"
|
||||
- "docker"
|
||||
commit-message:
|
||||
prefix: "chore(docker):"
|
||||
+10
-2
@@ -22,6 +22,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
- Rate limiting per user/tier
|
||||
- Comprehensive test infrastructure setup
|
||||
- CI/CD pipeline for testing and security scanning
|
||||
- Dependabot configuration for automated dependency updates (pip, npm, GitHub Actions, Docker)
|
||||
- Copilot instructions requiring TODO.md and CHANGELOG.md updates
|
||||
- Unit tests for security middleware (SecurityHeadersMiddleware, CSRFProtectionMiddleware)
|
||||
- Unit tests for JWT token lifecycle (access tokens, refresh tokens, decode, edge cases)
|
||||
- Unit tests for credential encryption edge cases (empty, long, unicode, special chars)
|
||||
- Unit tests for FastAPI application factory and core endpoints (root, health, OpenAPI)
|
||||
- Unit tests for Pydantic schema validation (users, mail accounts, notifications, subscriptions)
|
||||
- Reached 57% test coverage (up from 54%)
|
||||
|
||||
### Changed
|
||||
- Reorganized documentation into `docs/` directory
|
||||
@@ -113,5 +121,5 @@ Use these standard categories:
|
||||
|
||||
---
|
||||
|
||||
**Maintained by**: Development Team
|
||||
**Last Updated**: 2026-02-06
|
||||
**Maintained by**: Development Team
|
||||
**Last Updated**: 2026-03-23
|
||||
|
||||
+28
-23
@@ -20,7 +20,7 @@ Be respectful and inclusive. We welcome contributions from everyone.
|
||||
|
||||
### Suggesting Features
|
||||
|
||||
1. Check the [Roadmap](ROADMAP.md) to see if it's already planned
|
||||
1. Check the [Roadmap](docs/ROADMAP.md) to see if it's already planned
|
||||
2. Open an issue with the "enhancement" label
|
||||
3. Describe the feature and its use case
|
||||
4. Explain why it would be useful
|
||||
@@ -40,14 +40,14 @@ Be respectful and inclusive. We welcome contributions from everyone.
|
||||
|
||||
4. **Test your changes**
|
||||
```bash
|
||||
# Test Python syntax
|
||||
python3 -m py_compile pop3_forwarder.py
|
||||
|
||||
# Run the test suite
|
||||
make test
|
||||
|
||||
# Or run linting + formatting + tests together
|
||||
make quick-test
|
||||
|
||||
# Test Docker build
|
||||
docker build -t pop3-test .
|
||||
|
||||
# Test with your configuration
|
||||
docker-compose up
|
||||
```
|
||||
|
||||
5. **Commit your changes**
|
||||
@@ -80,19 +80,18 @@ Be respectful and inclusive. We welcome contributions from everyone.
|
||||
git clone https://github.com/YOUR-USERNAME/pop_puller_to_gmail.git
|
||||
cd pop_puller_to_gmail
|
||||
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
# Install all development dependencies
|
||||
make install-dev
|
||||
|
||||
# Copy example config
|
||||
cp .env.example .env
|
||||
# Edit .env with test credentials
|
||||
|
||||
# Run locally
|
||||
# Run the legacy forwarder script directly
|
||||
python pop3_forwarder.py
|
||||
|
||||
# Or start the SaaS backend in dev mode
|
||||
make run-dev
|
||||
```
|
||||
|
||||
### Docker Development
|
||||
@@ -112,23 +111,29 @@ docker run --env-file .env pop3-dev
|
||||
- Add docstrings to functions and classes
|
||||
- Keep functions focused and small
|
||||
- Handle errors gracefully
|
||||
- Run `make format` to auto-format with Black and Ruff
|
||||
|
||||
## Testing
|
||||
|
||||
Before submitting a PR:
|
||||
|
||||
1. **Syntax check**
|
||||
1. **Run the test suite**
|
||||
```bash
|
||||
python3 -m py_compile pop3_forwarder.py
|
||||
make test
|
||||
```
|
||||
|
||||
2. **Docker build**
|
||||
2. **Run linting**
|
||||
```bash
|
||||
make lint
|
||||
```
|
||||
|
||||
3. **Docker build**
|
||||
```bash
|
||||
docker build -t pop3-test .
|
||||
```
|
||||
|
||||
3. **Manual testing**
|
||||
- Test with real POP3 account (or mock)
|
||||
4. **Manual testing** (if applicable)
|
||||
- Test with a real POP3 account or mock
|
||||
- Verify emails are forwarded correctly
|
||||
- Check error handling
|
||||
- Review logs
|
||||
@@ -143,9 +148,9 @@ Update documentation when:
|
||||
|
||||
Files to update:
|
||||
- `README.md` - Main documentation
|
||||
- `QUICKSTART.md` - If setup changes
|
||||
- `MVP.md` - If MVP scope changes
|
||||
- `ROADMAP.md` - If adding future plans
|
||||
- `docs/QUICKSTART.md` - If setup changes
|
||||
- `docs/MVP.md` - If MVP scope changes
|
||||
- `docs/ROADMAP.md` - If adding future plans
|
||||
|
||||
## Security
|
||||
|
||||
@@ -158,7 +163,7 @@ Files to update:
|
||||
|
||||
**Do NOT open public issues for security vulnerabilities.**
|
||||
|
||||
Email security concerns to the maintainers privately.
|
||||
Please see [SECURITY.md](SECURITY.md) for responsible disclosure instructions.
|
||||
|
||||
## Questions?
|
||||
|
||||
|
||||
@@ -1,135 +1,61 @@
|
||||
# POP3 to Gmail Forwarder
|
||||
|
||||
[](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/test.yml)
|
||||
[](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/lint.yml)
|
||||
[](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/security.yml)
|
||||
[](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/docker-build.yml)
|
||||
[](https://opensource.org/licenses/MIT)
|
||||
[](https://www.python.org/downloads/)
|
||||
[](https://www.docker.com/)
|
||||
|
||||
A Docker-based solution that automatically fetches emails from POP3 mailboxes and forwards them to Gmail, replacing Google's discontinued POP3 import feature.
|
||||
|
||||
## Features
|
||||
|
||||
- ✅ **Multiple POP3 Accounts**: Support for unlimited POP3 mailboxes via environment variables
|
||||
- ✅ **Automatic Forwarding**: Sends emails to your Gmail account via SMTP
|
||||
- ✅ **Smart Throttling**: Rate limiting to avoid Gmail quotas (configurable emails per minute)
|
||||
- ✅ **Error Reporting**: Email notifications via Postmarkapp when issues occur
|
||||
- ✅ **Scheduled Polling**: Configurable check intervals (default: every 5 minutes)
|
||||
- ✅ **Docker Ready**: Fully containerized with docker-compose support
|
||||
- ✅ **Secure**: Runs as non-root user, uses SSL/TLS for connections
|
||||
- ✅ **Production Ready**: Comprehensive logging, error handling, and best practices
|
||||
- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables
|
||||
- **Automatic Forwarding** — sends emails to your Gmail account via SMTP
|
||||
- **Smart Throttling** — configurable rate limiting to stay within Gmail quotas
|
||||
- **Error Reporting** — notifications via Postmarkapp when issues occur
|
||||
- **Scheduled Polling** — configurable check intervals (default: every 5 minutes)
|
||||
- **Docker Ready** — fully containerized with Docker Compose support
|
||||
- **Secure** — runs as non-root user, SSL/TLS connections
|
||||
|
||||
### SaaS Platform (in development)
|
||||
|
||||
The repository also includes a multi-tenant SaaS backend built with FastAPI, PostgreSQL, Redis, and Celery. It adds multi-user support, OAuth2 authentication, POP3/IMAP protocol support, encrypted credential storage, and background job processing. See the [SaaS README](docs/README_SAAS.md) for details.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Docker and Docker Compose installed
|
||||
- A Gmail account with [App Password](https://support.google.com/accounts/answer/185833) enabled
|
||||
- POP3 account credentials
|
||||
- (Optional) Postmarkapp account for error notifications
|
||||
|
||||
### Option 1: Using Pre-built Docker Image (Recommended)
|
||||
|
||||
The Docker images are automatically built and published to GitHub Container Registry.
|
||||
|
||||
1. **Create configuration file**
|
||||
```bash
|
||||
# Download the docker-compose.yml and .env.example
|
||||
curl -O https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/docker-compose.yml
|
||||
curl -o .env https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/.env.example
|
||||
|
||||
# Edit .env with your credentials
|
||||
nano .env
|
||||
```
|
||||
|
||||
2. **Update docker-compose.yml to use the pre-built image**
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
pop3-forwarder:
|
||||
image: ghcr.io/christianlouis/pop_puller_to_gmail:latest
|
||||
container_name: pop3-gmail-forwarder
|
||||
restart: unless-stopped
|
||||
env_file:
|
||||
- .env
|
||||
```
|
||||
|
||||
3. **Run the container**
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Option 2: Building from Source
|
||||
|
||||
1. **Clone the repository**
|
||||
```bash
|
||||
git clone https://github.com/christianlouis/pop_puller_to_gmail.git
|
||||
cd pop_puller_to_gmail
|
||||
```
|
||||
|
||||
2. **Configure environment variables**
|
||||
```bash
|
||||
cp .env.example .env
|
||||
# Edit .env with your credentials
|
||||
nano .env
|
||||
```
|
||||
|
||||
3. **Essential Configuration**
|
||||
|
||||
Edit `.env` and set:
|
||||
|
||||
```bash
|
||||
# Your POP3 account(s)
|
||||
POP3_ACCOUNT_1_HOST=pop.yourprovider.com
|
||||
POP3_ACCOUNT_1_PORT=995
|
||||
POP3_ACCOUNT_1_USER=your-email@provider.com
|
||||
POP3_ACCOUNT_1_PASSWORD=your-password
|
||||
|
||||
# Your Gmail SMTP settings
|
||||
SMTP_USER=your-gmail@gmail.com
|
||||
SMTP_PASSWORD=your-app-password # Generate at myaccount.google.com/apppasswords
|
||||
GMAIL_DESTINATION=your-gmail@gmail.com
|
||||
|
||||
# Optional: Postmarkapp for error notifications
|
||||
POSTMARK_API_TOKEN=your-token
|
||||
POSTMARK_FROM_EMAIL=errors@yourdomain.com
|
||||
POSTMARK_TO_EMAIL=admin@yourdomain.com
|
||||
```
|
||||
|
||||
4. **Run with Docker Compose**
|
||||
```bash
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
5. **Check logs**
|
||||
```bash
|
||||
docker-compose logs -f
|
||||
```
|
||||
|
||||
## Using Pre-built Docker Images
|
||||
|
||||
Docker images are automatically built and published to GitHub Container Registry for every release and commit to the main branch.
|
||||
|
||||
### Available Image Tags
|
||||
|
||||
- `ghcr.io/christianlouis/pop_puller_to_gmail:latest` - Latest build from main branch
|
||||
- `ghcr.io/christianlouis/pop_puller_to_gmail:v1.0.0` - Specific version tags
|
||||
- `ghcr.io/christianlouis/pop_puller_to_gmail:main` - Main branch builds
|
||||
|
||||
### Pull and Run
|
||||
### Using a Pre-built Docker Image (Recommended)
|
||||
|
||||
```bash
|
||||
# Pull the latest image
|
||||
docker pull ghcr.io/christianlouis/pop_puller_to_gmail:latest
|
||||
# Pull and configure
|
||||
curl -O https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/docker-compose.yml
|
||||
curl -o .env https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/.env.example
|
||||
|
||||
# Run directly with Docker
|
||||
docker run -d \
|
||||
--name pop3-forwarder \
|
||||
--env-file .env \
|
||||
--restart unless-stopped \
|
||||
ghcr.io/christianlouis/pop_puller_to_gmail:latest
|
||||
# Edit .env with your credentials
|
||||
nano .env
|
||||
|
||||
# Start
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
### Building from Source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/christianlouis/pop_puller_to_gmail.git
|
||||
cd pop_puller_to_gmail
|
||||
cp .env.example .env # then edit .env
|
||||
docker-compose up -d
|
||||
```
|
||||
|
||||
See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions.
|
||||
|
||||
## Configuration
|
||||
|
||||
### POP3 Accounts
|
||||
|
||||
Add multiple POP3 accounts by incrementing the account number:
|
||||
Add multiple POP3 accounts by incrementing the account number in your `.env`:
|
||||
|
||||
```bash
|
||||
POP3_ACCOUNT_1_HOST=pop.provider1.com
|
||||
@@ -139,199 +65,115 @@ POP3_ACCOUNT_1_PASSWORD=password1
|
||||
POP3_ACCOUNT_2_HOST=pop.provider2.com
|
||||
POP3_ACCOUNT_2_USER=user2@provider2.com
|
||||
POP3_ACCOUNT_2_PASSWORD=password2
|
||||
|
||||
# ... add more as needed
|
||||
```
|
||||
|
||||
### Gmail App Password
|
||||
|
||||
1. Go to your Google Account: https://myaccount.google.com/
|
||||
2. Select Security
|
||||
3. Under "Signing in to Google," select App Passwords
|
||||
4. Generate a new app password for "Mail"
|
||||
5. Use this password in `SMTP_PASSWORD`
|
||||
1. Go to your [Google Account Security](https://myaccount.google.com/security)
|
||||
2. Under "Signing in to Google," select **App Passwords**
|
||||
3. Generate a new app password for "Mail"
|
||||
4. Use this password as `SMTP_PASSWORD`
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Required | Default | Description |
|
||||
|----------|----------|---------|-------------|
|
||||
| `POP3_ACCOUNT_N_HOST` | Yes | - | POP3 server hostname |
|
||||
| `POP3_ACCOUNT_N_PORT` | No | 995 | POP3 server port |
|
||||
| `POP3_ACCOUNT_N_USER` | Yes | - | POP3 username |
|
||||
| `POP3_ACCOUNT_N_PASSWORD` | Yes | - | POP3 password |
|
||||
| `POP3_ACCOUNT_N_USE_SSL` | No | true | Use SSL/TLS |
|
||||
| `SMTP_HOST` | No | smtp.gmail.com | SMTP server |
|
||||
| `SMTP_PORT` | No | 587 | SMTP port |
|
||||
| `SMTP_USER` | Yes | - | SMTP username |
|
||||
| `SMTP_PASSWORD` | Yes | - | SMTP password (App Password) |
|
||||
| `SMTP_USE_TLS` | No | true | Use STARTTLS |
|
||||
| `GMAIL_DESTINATION` | Yes | - | Destination Gmail address |
|
||||
| `CHECK_INTERVAL_MINUTES` | No | 5 | How often to check for new mail |
|
||||
| `MAX_EMAILS_PER_RUN` | No | 50 | Max emails to process per account per run |
|
||||
| `THROTTLE_EMAILS_PER_MINUTE` | No | 10 | Rate limit for sending emails |
|
||||
| `POSTMARK_API_TOKEN` | No | - | Postmarkapp API token |
|
||||
| `POSTMARK_FROM_EMAIL` | No | - | Error notification sender |
|
||||
| `POSTMARK_TO_EMAIL` | No | - | Error notification recipient |
|
||||
| `LOG_LEVEL` | No | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) |
|
||||
| `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname |
|
||||
| `POP3_ACCOUNT_N_PORT` | No | `995` | POP3 server port |
|
||||
| `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username |
|
||||
| `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password |
|
||||
| `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS |
|
||||
| `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server |
|
||||
| `SMTP_PORT` | No | `587` | SMTP port |
|
||||
| `SMTP_USER` | Yes | — | SMTP username |
|
||||
| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) |
|
||||
| `SMTP_USE_TLS` | No | `true` | Use STARTTLS |
|
||||
| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address |
|
||||
| `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval |
|
||||
| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run |
|
||||
| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit |
|
||||
| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token |
|
||||
| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender |
|
||||
| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient |
|
||||
| `LOG_LEVEL` | No | `INFO` | Logging level |
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Polling**: The application checks configured POP3 mailboxes at regular intervals
|
||||
2. **Fetching**: Retrieves new emails from each POP3 account
|
||||
3. **Forwarding**: Sends emails to your Gmail account via SMTP with original metadata preserved
|
||||
4. **Cleanup**: Deletes emails from POP3 server after successful forwarding
|
||||
5. **Throttling**: Respects rate limits to avoid Gmail quota issues
|
||||
6. **Error Handling**: Sends notifications via Postmarkapp if issues occur
|
||||
|
||||
## Email Format
|
||||
|
||||
Forwarded emails include:
|
||||
- Original sender information in the subject line: `[Fwd from user@provider.com] Original Subject`
|
||||
- Header section with original From, Date, Subject, and source account
|
||||
- Original email body preserved
|
||||
|
||||
## Monitoring and Logs
|
||||
|
||||
### View logs
|
||||
```bash
|
||||
docker-compose logs -f pop3-forwarder
|
||||
```
|
||||
|
||||
### Check container status
|
||||
```bash
|
||||
docker-compose ps
|
||||
```
|
||||
|
||||
### Restart the service
|
||||
```bash
|
||||
docker-compose restart
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Gmail Authentication Issues
|
||||
|
||||
**Problem**: "Username and Password not accepted"
|
||||
|
||||
**Solution**:
|
||||
- Ensure 2FA is enabled on your Google account
|
||||
- Generate an App Password (don't use your regular Gmail password)
|
||||
- Use the 16-character app password without spaces
|
||||
|
||||
### POP3 Connection Issues
|
||||
|
||||
**Problem**: "Connection refused" or "SSL error"
|
||||
|
||||
**Solution**:
|
||||
- Verify POP3 server hostname and port
|
||||
- Check if POP3 is enabled in your email provider settings
|
||||
- Try with `POP3_ACCOUNT_N_USE_SSL=false` for non-SSL connections (port 110)
|
||||
|
||||
### No Emails Being Forwarded
|
||||
|
||||
**Problem**: Container runs but no emails are forwarded
|
||||
|
||||
**Solution**:
|
||||
- Check if there are emails in your POP3 mailbox
|
||||
- Review logs for errors: `docker-compose logs -f`
|
||||
- Verify `GMAIL_DESTINATION` is correct
|
||||
- Check Gmail spam folder
|
||||
|
||||
### Rate Limiting
|
||||
|
||||
**Problem**: "Too many requests" or quota errors
|
||||
|
||||
**Solution**:
|
||||
- Increase `CHECK_INTERVAL_MINUTES`
|
||||
- Decrease `THROTTLE_EMAILS_PER_MINUTE`
|
||||
- Reduce `MAX_EMAILS_PER_RUN`
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
1. **Never commit `.env` file** - It contains sensitive credentials
|
||||
2. **Use App Passwords** - Don't use your main Gmail password
|
||||
3. **Rotate credentials regularly** - Update passwords periodically
|
||||
4. **Enable 2FA** - On all email accounts
|
||||
5. **Review logs** - Monitor for suspicious activity
|
||||
6. **Use SSL/TLS** - Keep `USE_SSL` and `USE_TLS` enabled
|
||||
7. **Limit network access** - Use firewall rules if needed
|
||||
|
||||
## Development
|
||||
|
||||
### Local Development (without Docker)
|
||||
|
||||
```bash
|
||||
# Create virtual environment
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate # On Windows: venv\Scripts\activate
|
||||
|
||||
# Install dependencies
|
||||
pip install -r requirements.txt
|
||||
|
||||
# Copy and configure .env
|
||||
cp .env.example .env
|
||||
# Edit .env with your settings
|
||||
|
||||
# Run the application
|
||||
python pop3_forwarder.py
|
||||
```
|
||||
|
||||
### Building the Docker Image
|
||||
|
||||
```bash
|
||||
docker build -t pop3-gmail-forwarder .
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Run with verbose logging
|
||||
LOG_LEVEL=DEBUG docker-compose up
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ POP3 Server 1 │
|
||||
└────────┬────────┘
|
||||
│
|
||||
│ (Fetch emails)
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐ ┌──────────────┐ ┌─────────────┐
|
||||
│ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail │
|
||||
└─────────────────┘ │ Container │ │ (SMTP) │
|
||||
│ └──────┬───────┘ └─────────────┘
|
||||
│ │
|
||||
┌────────▼────────┐ │
|
||||
│ POP3 Server N │ │
|
||||
└─────────────────┘ │
|
||||
│ (Error notifications)
|
||||
▼
|
||||
┌─────────────────┐
|
||||
┌────────▼────────┐ │ (Error notifications)
|
||||
│ POP3 Server N │ ▼
|
||||
└─────────────────┘ ┌─────────────────┐
|
||||
│ Postmarkapp │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
1. **Polling** — checks POP3 mailboxes at the configured interval
|
||||
2. **Fetching** — retrieves new emails from each account
|
||||
3. **Forwarding** — delivers to Gmail with original metadata preserved
|
||||
4. **Cleanup** — deletes from POP3 after successful forwarding
|
||||
5. **Throttling** — respects rate limits to avoid quota issues
|
||||
6. **Error Handling** — sends notifications if something goes wrong
|
||||
|
||||
## Development
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
make install-dev
|
||||
|
||||
# Run linting & formatting
|
||||
make lint
|
||||
make format
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Start backend in dev mode
|
||||
make run-dev
|
||||
```
|
||||
|
||||
See the [Testing Guide](docs/TESTING_GUIDE.md) for the full test workflow.
|
||||
|
||||
## Documentation
|
||||
|
||||
Detailed documentation lives in the [`docs/`](docs/) directory:
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [Architecture](docs/ARCHITECTURE.md) | System design and component overview |
|
||||
| [Quick Start](docs/QUICKSTART.md) | Step-by-step setup guide |
|
||||
| [Migration Guide](docs/MIGRATION_GUIDE.md) | Upgrading from v1 to v2 |
|
||||
| [Deployment Checklist](docs/DEPLOYMENT_CHECKLIST.md) | Production deployment guide |
|
||||
| [Roadmap](docs/ROADMAP.md) | Planned features and milestones |
|
||||
| [Testing Guide](docs/TESTING_GUIDE.md) | How to run and write tests |
|
||||
| [Coding Patterns](docs/CODING_PATTERNS.md) | Code style and conventions |
|
||||
| [SaaS README](docs/README_SAAS.md) | Multi-tenant SaaS platform details |
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please:
|
||||
Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Submit a pull request
|
||||
- Reporting bugs and suggesting features
|
||||
- Development setup and code style
|
||||
- Pull request process
|
||||
|
||||
## Security
|
||||
|
||||
To report a vulnerability, please see [SECURITY.md](SECURITY.md). **Do not open public issues for security concerns.**
|
||||
|
||||
## License
|
||||
|
||||
MIT License - See LICENSE file for details
|
||||
This project is licensed under the MIT License — see [LICENSE](LICENSE) for details.
|
||||
|
||||
## Support
|
||||
|
||||
- **Issues**: https://github.com/christianlouis/pop_puller_to_gmail/issues
|
||||
- **Discussions**: https://github.com/christianlouis/pop_puller_to_gmail/discussions
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Built to replace Gmail's discontinued POP3 import feature. Uses industry-standard Python libraries for email handling and Docker for easy deployment.
|
||||
- [Issue Tracker](https://github.com/christianlouis/pop_puller_to_gmail/issues)
|
||||
- [Discussions](https://github.com/christianlouis/pop_puller_to_gmail/discussions)
|
||||
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
# Security Policy
|
||||
|
||||
## Supported Versions
|
||||
|
||||
| Version | Supported |
|
||||
|---------|--------------------|
|
||||
| 2.x | ✅ Yes |
|
||||
| 1.x | ❌ No |
|
||||
| < 1.0 | ❌ No |
|
||||
|
||||
## Reporting a Vulnerability
|
||||
|
||||
**Please do NOT open public issues for security vulnerabilities.**
|
||||
|
||||
If you discover a security vulnerability, please report it responsibly:
|
||||
|
||||
1. **Email**: Send details to the repository maintainer via the email listed on the [GitHub profile](https://github.com/christianlouis).
|
||||
2. **GitHub Private Vulnerability Reporting**: Use [GitHub's security advisory feature](https://github.com/christianlouis/pop_puller_to_gmail/security/advisories/new) to report privately.
|
||||
|
||||
### What to Include
|
||||
|
||||
- A description of the vulnerability
|
||||
- Steps to reproduce the issue
|
||||
- Potential impact
|
||||
- Suggested fix (if any)
|
||||
|
||||
### Response Timeline
|
||||
|
||||
- **Acknowledgment**: Within 48 hours
|
||||
- **Initial Assessment**: Within 1 week
|
||||
- **Fix & Disclosure**: Coordinated with the reporter
|
||||
|
||||
## Security Best Practices for Users
|
||||
|
||||
- **Never commit `.env` files** containing credentials
|
||||
- **Use App Passwords** for Gmail instead of your main password
|
||||
- **Enable 2FA** on all email accounts
|
||||
- **Rotate credentials** regularly
|
||||
- **Use SSL/TLS** for all mail connections
|
||||
- **Run containers as non-root** (default in provided Dockerfile)
|
||||
- **Keep dependencies updated** — Dependabot is enabled on this repository
|
||||
@@ -0,0 +1,108 @@
|
||||
"""
|
||||
Unit tests for the FastAPI application factory and core endpoints.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
from app.main import create_application
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def app():
|
||||
"""Create a fresh application instance for testing."""
|
||||
return create_application()
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestRootEndpoint:
|
||||
"""Test root endpoint"""
|
||||
|
||||
async def test_root_returns_200(self, app):
|
||||
"""Test that root endpoint returns 200"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
async def test_root_returns_api_info(self, app):
|
||||
"""Test that root returns API information"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/")
|
||||
|
||||
data = response.json()
|
||||
assert "message" in data
|
||||
assert "version" in data
|
||||
assert "docs" in data
|
||||
assert data["docs"] == "/api/docs"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestHealthEndpoint:
|
||||
"""Test health check endpoint"""
|
||||
|
||||
async def test_health_returns_200(self, app):
|
||||
"""Test that health endpoint returns 200"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health")
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
async def test_health_returns_healthy(self, app):
|
||||
"""Test that health endpoint returns healthy status"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health")
|
||||
|
||||
data = response.json()
|
||||
assert data["status"] == "healthy"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestSecurityHeaders:
|
||||
"""Test that security headers are present in responses"""
|
||||
|
||||
async def test_security_headers_on_root(self, app):
|
||||
"""Test security headers on root endpoint"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/")
|
||||
|
||||
assert response.headers["X-Frame-Options"] == "DENY"
|
||||
assert response.headers["X-Content-Type-Options"] == "nosniff"
|
||||
assert response.headers["X-XSS-Protection"] == "1; mode=block"
|
||||
|
||||
async def test_security_headers_on_health(self, app):
|
||||
"""Test security headers on health endpoint"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/health")
|
||||
|
||||
assert response.headers["X-Frame-Options"] == "DENY"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestApplicationFactory:
|
||||
"""Test the create_application factory"""
|
||||
|
||||
async def test_app_title(self, app):
|
||||
"""Test that app has correct title"""
|
||||
assert app.title == "POP3 Forwarder SaaS"
|
||||
|
||||
async def test_app_version(self, app):
|
||||
"""Test that app has a version"""
|
||||
assert app.version is not None
|
||||
assert len(app.version) > 0
|
||||
|
||||
async def test_openapi_endpoint(self, app):
|
||||
"""Test that OpenAPI schema is available"""
|
||||
transport = ASGITransport(app=app)
|
||||
async with AsyncClient(transport=transport, base_url="http://test") as client:
|
||||
response = await client.get("/api/openapi.json")
|
||||
|
||||
assert response.status_code == 200
|
||||
schema = response.json()
|
||||
assert "openapi" in schema
|
||||
assert "info" in schema
|
||||
@@ -0,0 +1,127 @@
|
||||
"""
|
||||
Unit tests for security middleware.
|
||||
"""
|
||||
|
||||
from starlette.testclient import TestClient
|
||||
from starlette.applications import Starlette
|
||||
from starlette.responses import PlainTextResponse
|
||||
from starlette.routing import Route
|
||||
|
||||
from app.core.middleware import SecurityHeadersMiddleware, CSRFProtectionMiddleware
|
||||
|
||||
|
||||
def _make_app(middleware_classes):
|
||||
"""Helper to build a Starlette app with given middleware."""
|
||||
|
||||
async def homepage(request):
|
||||
return PlainTextResponse("OK")
|
||||
|
||||
app = Starlette(
|
||||
routes=[
|
||||
Route("/", homepage, methods=["GET", "HEAD", "POST", "OPTIONS"]),
|
||||
Route("/api/v1/auth/login", homepage, methods=["GET", "POST"]),
|
||||
]
|
||||
)
|
||||
for cls in middleware_classes:
|
||||
app.add_middleware(cls)
|
||||
return app
|
||||
|
||||
|
||||
class TestSecurityHeadersMiddleware:
|
||||
"""Test security headers added to all responses"""
|
||||
|
||||
def setup_method(self):
|
||||
app = _make_app([SecurityHeadersMiddleware])
|
||||
self.client = TestClient(app)
|
||||
|
||||
def test_x_frame_options_header(self):
|
||||
"""Test X-Frame-Options is set to DENY"""
|
||||
response = self.client.get("/")
|
||||
assert response.headers["X-Frame-Options"] == "DENY"
|
||||
|
||||
def test_x_content_type_options_header(self):
|
||||
"""Test X-Content-Type-Options is set to nosniff"""
|
||||
response = self.client.get("/")
|
||||
assert response.headers["X-Content-Type-Options"] == "nosniff"
|
||||
|
||||
def test_x_xss_protection_header(self):
|
||||
"""Test X-XSS-Protection header is set"""
|
||||
response = self.client.get("/")
|
||||
assert response.headers["X-XSS-Protection"] == "1; mode=block"
|
||||
|
||||
def test_content_security_policy_header(self):
|
||||
"""Test Content-Security-Policy header is present"""
|
||||
response = self.client.get("/")
|
||||
csp = response.headers["Content-Security-Policy"]
|
||||
assert "default-src 'self'" in csp
|
||||
assert "script-src" in csp
|
||||
|
||||
def test_referrer_policy_header(self):
|
||||
"""Test Referrer-Policy header"""
|
||||
response = self.client.get("/")
|
||||
assert response.headers["Referrer-Policy"] == "strict-origin-when-cross-origin"
|
||||
|
||||
def test_permissions_policy_header(self):
|
||||
"""Test Permissions-Policy header"""
|
||||
response = self.client.get("/")
|
||||
policy = response.headers["Permissions-Policy"]
|
||||
assert "geolocation=()" in policy
|
||||
assert "microphone=()" in policy
|
||||
assert "camera=()" in policy
|
||||
|
||||
def test_no_hsts_for_localhost(self):
|
||||
"""Test that HSTS header check depends on hostname"""
|
||||
# The HSTS header is only skipped when hostname is localhost or 127.0.0.1.
|
||||
# TestClient uses 'testserver' as hostname, which is not in the skip list,
|
||||
# so HSTS will be set. Verify the logic works with a direct check.
|
||||
response = self.client.get("/")
|
||||
# TestClient hostname is 'testserver', not localhost, so HSTS IS set
|
||||
assert "Strict-Transport-Security" in response.headers
|
||||
|
||||
|
||||
class TestCSRFProtectionMiddleware:
|
||||
"""Test CSRF protection middleware"""
|
||||
|
||||
def setup_method(self):
|
||||
app = _make_app([CSRFProtectionMiddleware])
|
||||
self.client = TestClient(app)
|
||||
|
||||
def test_get_requests_pass_through(self):
|
||||
"""Test that GET requests are not blocked"""
|
||||
response = self.client.get("/")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_head_requests_pass_through(self):
|
||||
"""Test that HEAD requests are not blocked"""
|
||||
response = self.client.head("/")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_options_requests_pass_through(self):
|
||||
"""Test that OPTIONS requests are not blocked"""
|
||||
response = self.client.options("/")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_exempt_paths_pass_through(self):
|
||||
"""Test that exempt paths are not CSRF-checked for POST"""
|
||||
response = self.client.post("/api/v1/auth/login")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_post_to_non_exempt_path_passes(self):
|
||||
"""Test that POST to non-exempt path also passes (JWT provides CSRF protection)"""
|
||||
response = self.client.post("/")
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_generate_csrf_token(self):
|
||||
"""Test CSRF token generation produces valid token"""
|
||||
token = CSRFProtectionMiddleware._generate_csrf_token()
|
||||
assert isinstance(token, str)
|
||||
assert len(token) == 43 # token_urlsafe(32) produces 43 chars
|
||||
|
||||
def test_validate_csrf_token_valid(self):
|
||||
"""Test CSRF token validation with valid token"""
|
||||
token = CSRFProtectionMiddleware._generate_csrf_token()
|
||||
assert CSRFProtectionMiddleware._validate_csrf_token(token) is True
|
||||
|
||||
def test_validate_csrf_token_invalid(self):
|
||||
"""Test CSRF token validation with invalid token"""
|
||||
assert CSRFProtectionMiddleware._validate_csrf_token("short") is False
|
||||
@@ -0,0 +1,277 @@
|
||||
"""
|
||||
Unit tests for Pydantic schema validation.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from pydantic import ValidationError
|
||||
from app.models.schemas import (
|
||||
UserCreate,
|
||||
UserUpdate,
|
||||
MailAccountCreate,
|
||||
MailAccountUpdate,
|
||||
MailAccountTestRequest,
|
||||
MailAccountAutoDetectRequest,
|
||||
MailProtocol,
|
||||
DeliveryMethod,
|
||||
SubscriptionTier,
|
||||
AccountStatus,
|
||||
NotificationChannel,
|
||||
Token,
|
||||
TokenPayload,
|
||||
GoogleAuthRequest,
|
||||
NotificationConfigCreate,
|
||||
SubscriptionCheckoutRequest,
|
||||
ProviderPreset,
|
||||
)
|
||||
|
||||
|
||||
class TestEnums:
|
||||
"""Test enum values"""
|
||||
|
||||
def test_subscription_tiers(self):
|
||||
"""Test all subscription tier values"""
|
||||
assert SubscriptionTier.FREE == "free"
|
||||
assert SubscriptionTier.BASIC == "basic"
|
||||
assert SubscriptionTier.PRO == "pro"
|
||||
assert SubscriptionTier.ENTERPRISE == "enterprise"
|
||||
|
||||
def test_mail_protocols(self):
|
||||
"""Test all mail protocol values"""
|
||||
assert MailProtocol.POP3 == "pop3"
|
||||
assert MailProtocol.POP3_SSL == "pop3_ssl"
|
||||
assert MailProtocol.IMAP == "imap"
|
||||
assert MailProtocol.IMAP_SSL == "imap_ssl"
|
||||
|
||||
def test_account_status(self):
|
||||
"""Test all account status values"""
|
||||
assert AccountStatus.ACTIVE == "active"
|
||||
assert AccountStatus.INACTIVE == "inactive"
|
||||
assert AccountStatus.ERROR == "error"
|
||||
assert AccountStatus.TESTING == "testing"
|
||||
|
||||
def test_delivery_method(self):
|
||||
"""Test all delivery method values"""
|
||||
assert DeliveryMethod.SMTP == "smtp"
|
||||
assert DeliveryMethod.GMAIL_API == "gmail_api"
|
||||
|
||||
def test_notification_channels(self):
|
||||
"""Test all notification channel values"""
|
||||
assert NotificationChannel.EMAIL == "email"
|
||||
assert NotificationChannel.TELEGRAM == "telegram"
|
||||
assert NotificationChannel.WEBHOOK == "webhook"
|
||||
assert NotificationChannel.SLACK == "slack"
|
||||
assert NotificationChannel.DISCORD == "discord"
|
||||
|
||||
|
||||
class TestUserSchemas:
|
||||
"""Test user-related schemas"""
|
||||
|
||||
def test_user_create_with_email(self):
|
||||
"""Test UserCreate with valid email"""
|
||||
user = UserCreate(email="test@example.com", password="password123")
|
||||
assert user.email == "test@example.com"
|
||||
assert user.password == "password123"
|
||||
|
||||
def test_user_create_without_password(self):
|
||||
"""Test UserCreate without password (OAuth users)"""
|
||||
user = UserCreate(email="test@example.com")
|
||||
assert user.password is None
|
||||
|
||||
def test_user_create_with_full_name(self):
|
||||
"""Test UserCreate with full name"""
|
||||
user = UserCreate(
|
||||
email="test@example.com", full_name="Test User", password="pass"
|
||||
)
|
||||
assert user.full_name == "Test User"
|
||||
|
||||
def test_user_create_invalid_email(self):
|
||||
"""Test UserCreate rejects invalid email"""
|
||||
with pytest.raises(ValidationError):
|
||||
UserCreate(email="not-an-email", password="pass")
|
||||
|
||||
def test_user_update_partial(self):
|
||||
"""Test UserUpdate with partial data"""
|
||||
update = UserUpdate(full_name="New Name")
|
||||
assert update.full_name == "New Name"
|
||||
assert update.email is None
|
||||
|
||||
|
||||
class TestTokenSchemas:
|
||||
"""Test token schemas"""
|
||||
|
||||
def test_token_schema(self):
|
||||
"""Test Token schema"""
|
||||
token = Token(
|
||||
access_token="abc123", refresh_token="def456", token_type="bearer"
|
||||
)
|
||||
assert token.access_token == "abc123"
|
||||
assert token.token_type == "bearer"
|
||||
|
||||
def test_token_payload_schema(self):
|
||||
"""Test TokenPayload schema"""
|
||||
payload = TokenPayload(sub=42, type="access")
|
||||
assert payload.sub == 42
|
||||
assert payload.type == "access"
|
||||
|
||||
def test_google_auth_request(self):
|
||||
"""Test GoogleAuthRequest schema"""
|
||||
req = GoogleAuthRequest(
|
||||
code="auth-code-123", redirect_uri="http://localhost:3000/callback"
|
||||
)
|
||||
assert req.code == "auth-code-123"
|
||||
|
||||
|
||||
class TestMailAccountSchemas:
|
||||
"""Test mail account schemas"""
|
||||
|
||||
def test_mail_account_create_valid(self):
|
||||
"""Test creating a valid mail account"""
|
||||
account = MailAccountCreate(
|
||||
name="Test Account",
|
||||
email_address="user@example.com",
|
||||
host="imap.example.com",
|
||||
port=993,
|
||||
username="user@example.com",
|
||||
password="secret",
|
||||
forward_to="me@gmail.com",
|
||||
)
|
||||
assert account.name == "Test Account"
|
||||
assert account.protocol == MailProtocol.POP3_SSL # default
|
||||
assert account.use_ssl is True
|
||||
|
||||
def test_mail_account_create_invalid_port(self):
|
||||
"""Test that invalid port is rejected"""
|
||||
with pytest.raises(ValidationError):
|
||||
MailAccountCreate(
|
||||
name="Test",
|
||||
email_address="user@example.com",
|
||||
host="imap.example.com",
|
||||
port=0, # invalid
|
||||
username="user@example.com",
|
||||
password="secret",
|
||||
forward_to="me@gmail.com",
|
||||
)
|
||||
|
||||
def test_mail_account_create_port_too_high(self):
|
||||
"""Test that port above 65535 is rejected"""
|
||||
with pytest.raises(ValidationError):
|
||||
MailAccountCreate(
|
||||
name="Test",
|
||||
email_address="user@example.com",
|
||||
host="imap.example.com",
|
||||
port=70000, # invalid
|
||||
username="user@example.com",
|
||||
password="secret",
|
||||
forward_to="me@gmail.com",
|
||||
)
|
||||
|
||||
def test_mail_account_update_partial(self):
|
||||
"""Test partial mail account update"""
|
||||
update = MailAccountUpdate(is_enabled=False)
|
||||
assert update.is_enabled is False
|
||||
assert update.name is None
|
||||
assert update.password is None
|
||||
|
||||
def test_mail_account_test_request(self):
|
||||
"""Test mail account test connection schema"""
|
||||
req = MailAccountTestRequest(
|
||||
host="imap.gmail.com",
|
||||
port=993,
|
||||
protocol=MailProtocol.IMAP_SSL,
|
||||
username="user@gmail.com",
|
||||
password="app-password",
|
||||
)
|
||||
assert req.host == "imap.gmail.com"
|
||||
|
||||
def test_auto_detect_request(self):
|
||||
"""Test auto-detect request schema"""
|
||||
req = MailAccountAutoDetectRequest(email_address="user@gmail.com")
|
||||
assert req.email_address == "user@gmail.com"
|
||||
|
||||
def test_auto_detect_invalid_email(self):
|
||||
"""Test auto-detect rejects invalid email"""
|
||||
with pytest.raises(ValidationError):
|
||||
MailAccountAutoDetectRequest(email_address="not-email")
|
||||
|
||||
|
||||
class TestNotificationSchemas:
|
||||
"""Test notification schemas"""
|
||||
|
||||
def test_notification_config_create(self):
|
||||
"""Test creating notification config"""
|
||||
config = NotificationConfigCreate(
|
||||
channel=NotificationChannel.TELEGRAM,
|
||||
config={"bot_token": "123:abc", "chat_id": "456"},
|
||||
)
|
||||
assert config.channel == NotificationChannel.TELEGRAM
|
||||
assert config.notify_on_errors is True # default
|
||||
assert config.notify_on_success is False # default
|
||||
|
||||
def test_notification_config_threshold_validation(self):
|
||||
"""Test notification threshold validation"""
|
||||
with pytest.raises(ValidationError):
|
||||
NotificationConfigCreate(
|
||||
channel=NotificationChannel.EMAIL,
|
||||
config={},
|
||||
notify_threshold=0, # must be > 0
|
||||
)
|
||||
|
||||
|
||||
class TestSubscriptionSchemas:
|
||||
"""Test subscription schemas"""
|
||||
|
||||
def test_subscription_checkout_request_monthly(self):
|
||||
"""Test subscription checkout with monthly billing"""
|
||||
req = SubscriptionCheckoutRequest(
|
||||
tier=SubscriptionTier.PRO,
|
||||
billing_period="monthly",
|
||||
success_url="https://example.com/success",
|
||||
cancel_url="https://example.com/cancel",
|
||||
)
|
||||
assert req.tier == SubscriptionTier.PRO
|
||||
assert req.billing_period == "monthly"
|
||||
|
||||
def test_subscription_checkout_request_yearly(self):
|
||||
"""Test subscription checkout with yearly billing"""
|
||||
req = SubscriptionCheckoutRequest(
|
||||
tier=SubscriptionTier.BASIC,
|
||||
billing_period="yearly",
|
||||
success_url="https://example.com/success",
|
||||
cancel_url="https://example.com/cancel",
|
||||
)
|
||||
assert req.billing_period == "yearly"
|
||||
|
||||
def test_subscription_checkout_invalid_period(self):
|
||||
"""Test that invalid billing period is rejected"""
|
||||
with pytest.raises(ValidationError):
|
||||
SubscriptionCheckoutRequest(
|
||||
tier=SubscriptionTier.BASIC,
|
||||
billing_period="quarterly", # invalid
|
||||
success_url="https://example.com/success",
|
||||
cancel_url="https://example.com/cancel",
|
||||
)
|
||||
|
||||
|
||||
class TestProviderPresetSchema:
|
||||
"""Test provider preset schema"""
|
||||
|
||||
def test_provider_preset_with_imap(self):
|
||||
"""Test provider preset with IMAP config"""
|
||||
preset = ProviderPreset(
|
||||
id="gmail",
|
||||
name="Gmail",
|
||||
domains=["gmail.com", "googlemail.com"],
|
||||
imap_ssl={"host": "imap.gmail.com", "port": 993},
|
||||
)
|
||||
assert preset.id == "gmail"
|
||||
assert "gmail.com" in preset.domains
|
||||
|
||||
def test_provider_preset_without_pop3(self):
|
||||
"""Test provider preset without POP3 (IMAP only)"""
|
||||
preset = ProviderPreset(
|
||||
id="posteo",
|
||||
name="Posteo",
|
||||
domains=["posteo.de"],
|
||||
imap_ssl={"host": "posteo.de", "port": 993},
|
||||
)
|
||||
assert preset.pop3_ssl is None
|
||||
@@ -0,0 +1,188 @@
|
||||
"""
|
||||
Unit tests for extended security module functionality.
|
||||
"""
|
||||
|
||||
from datetime import timedelta
|
||||
from app.core.security import (
|
||||
create_access_token,
|
||||
create_refresh_token,
|
||||
decode_token,
|
||||
generate_random_token,
|
||||
encrypt_credential,
|
||||
decrypt_credential,
|
||||
CredentialEncryption,
|
||||
)
|
||||
|
||||
|
||||
class TestAccessToken:
|
||||
"""Test JWT access token creation and decoding"""
|
||||
|
||||
def test_create_access_token_with_custom_expiry(self):
|
||||
"""Test creating an access token with custom expiry"""
|
||||
data = {"sub": "test@example.com"}
|
||||
token = create_access_token(data, expires_delta=timedelta(hours=1))
|
||||
|
||||
assert isinstance(token, str)
|
||||
assert token.count(".") == 2
|
||||
|
||||
def test_decode_valid_access_token(self):
|
||||
"""Test decoding a valid access token"""
|
||||
data = {"sub": "user123"}
|
||||
token = create_access_token(data)
|
||||
|
||||
payload = decode_token(token)
|
||||
assert payload is not None
|
||||
assert payload["sub"] == "user123"
|
||||
assert payload["type"] == "access"
|
||||
|
||||
def test_decode_invalid_token_returns_none(self):
|
||||
"""Test that decoding an invalid token returns None"""
|
||||
result = decode_token("invalid.token.string")
|
||||
assert result is None
|
||||
|
||||
def test_decode_empty_token_returns_none(self):
|
||||
"""Test that decoding an empty string returns None"""
|
||||
result = decode_token("")
|
||||
assert result is None
|
||||
|
||||
def test_access_token_contains_type(self):
|
||||
"""Test that access token payload contains type 'access'"""
|
||||
data = {"sub": "user@example.com"}
|
||||
token = create_access_token(data)
|
||||
payload = decode_token(token)
|
||||
|
||||
assert payload is not None
|
||||
assert payload["type"] == "access"
|
||||
|
||||
def test_access_token_contains_expiry(self):
|
||||
"""Test that access token payload contains expiry"""
|
||||
data = {"sub": "user@example.com"}
|
||||
token = create_access_token(data)
|
||||
payload = decode_token(token)
|
||||
|
||||
assert payload is not None
|
||||
assert "exp" in payload
|
||||
|
||||
|
||||
class TestRefreshToken:
|
||||
"""Test JWT refresh token creation and decoding"""
|
||||
|
||||
def test_create_refresh_token(self):
|
||||
"""Test refresh token creation"""
|
||||
data = {"sub": "test@example.com"}
|
||||
token = create_refresh_token(data)
|
||||
|
||||
assert isinstance(token, str)
|
||||
assert token.count(".") == 2
|
||||
|
||||
def test_decode_refresh_token(self):
|
||||
"""Test decoding a valid refresh token"""
|
||||
data = {"sub": "user456"}
|
||||
token = create_refresh_token(data)
|
||||
|
||||
payload = decode_token(token)
|
||||
assert payload is not None
|
||||
assert payload["sub"] == "user456"
|
||||
assert payload["type"] == "refresh"
|
||||
|
||||
def test_refresh_token_different_from_access(self):
|
||||
"""Test that refresh and access tokens are different"""
|
||||
data = {"sub": "test@example.com"}
|
||||
access = create_access_token(data)
|
||||
refresh = create_refresh_token(data)
|
||||
|
||||
assert access != refresh
|
||||
|
||||
|
||||
class TestRandomToken:
|
||||
"""Test random token generation"""
|
||||
|
||||
def test_generate_random_token_default_length(self):
|
||||
"""Test generating a random token with default length"""
|
||||
token = generate_random_token()
|
||||
assert isinstance(token, str)
|
||||
assert len(token) > 0
|
||||
|
||||
def test_generate_random_token_custom_length(self):
|
||||
"""Test generating a random token with custom length"""
|
||||
token = generate_random_token(64)
|
||||
assert isinstance(token, str)
|
||||
assert len(token) > 0
|
||||
|
||||
def test_random_tokens_are_unique(self):
|
||||
"""Test that generated tokens are unique"""
|
||||
tokens = {generate_random_token() for _ in range(10)}
|
||||
assert len(tokens) == 10
|
||||
|
||||
|
||||
class TestGlobalEncryptionFunctions:
|
||||
"""Test global encryption convenience functions"""
|
||||
|
||||
def test_encrypt_credential_returns_string(self):
|
||||
"""Test that encrypt_credential returns a non-empty string"""
|
||||
encrypted = encrypt_credential("my-password")
|
||||
assert isinstance(encrypted, str)
|
||||
assert len(encrypted) > 0
|
||||
|
||||
def test_decrypt_credential_roundtrip(self):
|
||||
"""Test encrypt/decrypt roundtrip with global functions"""
|
||||
original = "super-secret-password-123"
|
||||
encrypted = encrypt_credential(original)
|
||||
decrypted = decrypt_credential(encrypted)
|
||||
assert decrypted == original
|
||||
|
||||
def test_encrypt_credential_is_not_plaintext(self):
|
||||
"""Test that encrypted credential differs from plaintext"""
|
||||
password = "my-password"
|
||||
encrypted = encrypt_credential(password)
|
||||
assert encrypted != password
|
||||
|
||||
|
||||
class TestCredentialEncryptionEdgeCases:
|
||||
"""Test edge cases in credential encryption"""
|
||||
|
||||
def test_encrypt_empty_string(self):
|
||||
"""Test encrypting an empty string"""
|
||||
encryptor = CredentialEncryption(user_id=1)
|
||||
encrypted = encryptor.encrypt("")
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == ""
|
||||
|
||||
def test_encrypt_long_string(self):
|
||||
"""Test encrypting a very long string"""
|
||||
long_password = "a" * 10000
|
||||
encryptor = CredentialEncryption(user_id=1)
|
||||
encrypted = encryptor.encrypt(long_password)
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == long_password
|
||||
|
||||
def test_encrypt_special_characters(self):
|
||||
"""Test encrypting a string with special characters"""
|
||||
special = "p@$$w0rd!#%^&*()_+-=[]{}|;':\",./<>?"
|
||||
encryptor = CredentialEncryption(user_id=1)
|
||||
encrypted = encryptor.encrypt(special)
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == special
|
||||
|
||||
def test_encrypt_unicode(self):
|
||||
"""Test encrypting unicode characters"""
|
||||
unicode_str = "密码テスト🔒"
|
||||
encryptor = CredentialEncryption(user_id=1)
|
||||
encrypted = encryptor.encrypt(unicode_str)
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == unicode_str
|
||||
|
||||
def test_custom_key(self):
|
||||
"""Test encryption with a custom key"""
|
||||
key = "custom-encryption-key-that-is-at-least-32-chars-long"
|
||||
encryptor = CredentialEncryption(key=key, user_id=1)
|
||||
encrypted = encryptor.encrypt("test-data")
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == "test-data"
|
||||
|
||||
def test_system_salt_without_user_id(self):
|
||||
"""Test encryption with system salt (no user_id)"""
|
||||
encryptor = CredentialEncryption()
|
||||
encrypted = encryptor.encrypt("system-data")
|
||||
decrypted = encryptor.decrypt(encrypted)
|
||||
assert decrypted == "system-data"
|
||||
@@ -316,9 +316,9 @@ alembic downgrade -1
|
||||
## 📚 Additional Documentation
|
||||
|
||||
- [API Documentation](http://localhost:8000/api/docs) - Interactive API docs
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) - Contribution guidelines
|
||||
- [CONTRIBUTING.md](../CONTRIBUTING.md) - Contribution guidelines
|
||||
- [ROADMAP.md](ROADMAP.md) - Future development plans
|
||||
- [SECURITY.md](SECURITY.md) - Security policies
|
||||
- [SECURITY.md](../SECURITY.md) - Security policies
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
@@ -331,7 +331,7 @@ Contributions welcome! Please:
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT License - See [LICENSE](LICENSE) file
|
||||
MIT License - See [LICENSE](../LICENSE) file
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -312,7 +312,7 @@ Helm charts and Kubernetes manifests will be provided for production deployment.
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
||||
Contributions are welcome! Please see [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelines.
|
||||
|
||||
### Areas for Contribution
|
||||
|
||||
@@ -325,7 +325,7 @@ Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for gui
|
||||
|
||||
## 📜 License
|
||||
|
||||
MIT License - See [LICENSE](LICENSE) file for details.
|
||||
MIT License - See [LICENSE](../LICENSE) file for details.
|
||||
|
||||
## 🆘 Support
|
||||
|
||||
@@ -357,16 +357,19 @@ Built with these amazing open-source projects:
|
||||
| Background Jobs | ✅ Complete | 100% |
|
||||
| Documentation | ✅ Complete | 100% |
|
||||
| Stripe Integration | 🚧 In Progress | 60% |
|
||||
| Frontend Dashboard | 📋 Planned | 0% |
|
||||
| Notification System | 📋 Planned | 40% |
|
||||
| Testing Suite | 📋 Planned | 20% |
|
||||
| Frontend Dashboard | 🚧 In Progress | 70% |
|
||||
| Notification System | 🚧 In Progress | 40% |
|
||||
| Testing Suite | 🚧 In Progress | 30% |
|
||||
|
||||
> **Note:** The frontend pages and components are implemented but the API client
|
||||
> layer (`lib/api.ts`) is not yet wired up, so the dashboard does not function
|
||||
> end-to-end yet.
|
||||
|
||||
## 🔮 Roadmap
|
||||
|
||||
See [ROADMAP.md](ROADMAP.md) for detailed future plans, including:
|
||||
|
||||
- Complete web dashboard
|
||||
- Mobile app (iOS/Android)
|
||||
- Advanced email filtering
|
||||
- Email archiving
|
||||
- Multi-destination forwarding
|
||||
@@ -374,12 +377,6 @@ See [ROADMAP.md](ROADMAP.md) for detailed future plans, including:
|
||||
- Kubernetes deployment
|
||||
- High availability setup
|
||||
|
||||
## ⭐ Star History
|
||||
|
||||
If you find this project useful, please consider giving it a star! ⭐
|
||||
|
||||
---
|
||||
|
||||
**Version**: 2.0.0 | **Status**: Production Ready (Backend) | **Updated**: 2026-02-01
|
||||
|
||||
Made with ❤️ for the community
|
||||
**Status**: In Development | **Backend**: Production-ready | **Frontend**: In Progress
|
||||
@@ -243,7 +243,7 @@ We welcome contributions! Areas where help is needed:
|
||||
5. **Performance**: Optimize slow operations
|
||||
6. **Security**: Security audits and improvements
|
||||
|
||||
See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
|
||||
See [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelines.
|
||||
|
||||
---
|
||||
|
||||
+52
-13
@@ -43,8 +43,8 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
- [x] Add `.secrets.baseline` for detect-secrets
|
||||
|
||||
### In Progress 🔨
|
||||
- [x] Reorganize documentation into `docs/` directory
|
||||
- [ ] Complete ADR documentation (add ADR-003 through ADR-010)
|
||||
- [ ] Reorganize documentation into `docs/` directory
|
||||
- [ ] Create GitHub Projects board for task management
|
||||
|
||||
### Not Started 📋
|
||||
@@ -65,6 +65,11 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
- [x] Add `backend/pytest.ini` configuration
|
||||
- [x] Create sample unit tests (test_security.py, test_config.py)
|
||||
- [x] Add user and mail account factory fixtures
|
||||
- [x] Write unit tests for security module (100% coverage)
|
||||
- [x] Write unit tests for middleware (98% coverage)
|
||||
- [x] Write unit tests for schemas and validation
|
||||
- [x] Write unit tests for application factory and core endpoints
|
||||
- [x] Reach 50%+ test coverage (currently 57%)
|
||||
|
||||
### In Progress 🔨
|
||||
- [ ] Write unit tests for authentication (target 80%+ coverage)
|
||||
@@ -88,6 +93,7 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
- [x] Create `.github/workflows/lint.yml` for code quality checks
|
||||
- [x] Create `.github/workflows/security.yml` for security scanning
|
||||
- [x] Existing `.github/workflows/docker-build.yml` for Docker images
|
||||
- [x] Set up automatic dependency updates (Dependabot)
|
||||
|
||||
### In Progress 🔨
|
||||
- [ ] Configure branch protection rules
|
||||
@@ -95,7 +101,6 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
|
||||
### Not Started 📋
|
||||
- [ ] Add deployment workflow (staging/production)
|
||||
- [ ] Set up automatic dependency updates (Dependabot)
|
||||
- [ ] Add release workflow with automated changelog
|
||||
- [ ] Configure status checks for PRs
|
||||
- [ ] Add performance regression detection
|
||||
@@ -169,12 +174,46 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
- [ ] Add advanced email filtering
|
||||
- [ ] Implement OAuth2 for Gmail (instead of App Passwords)
|
||||
- [ ] Add attachment handling improvements
|
||||
- [ ] Build frontend dashboard (React/Next.js)
|
||||
- [ ] Add email archiving feature
|
||||
- [ ] Implement webhook support for external integrations
|
||||
|
||||
---
|
||||
|
||||
## 🖥️ High Priority - Frontend Completion
|
||||
|
||||
The Next.js frontend has pages and components implemented but is **not functional**
|
||||
because the API client layer is missing.
|
||||
|
||||
### Critical Blockers 🔴
|
||||
- [ ] Create `frontend/src/lib/api.ts` — API client using axios
|
||||
- Must export: `authApi`, `mailAccountsApi`, `processingRunsApi`, `userApi`
|
||||
- Must export types: `User`, `MailAccount`, `MailAccountCreate`
|
||||
- 8 files import from `@/lib/api` and will fail to compile without it:
|
||||
`AuthGuard.tsx`, `AddMailAccountModal.tsx`, `authStore.ts`,
|
||||
`login/page.tsx`, `register/page.tsx`, `auth/callback/page.tsx`,
|
||||
`dashboard/page.tsx`, `accounts/page.tsx`
|
||||
|
||||
### Existing Pages (UI done, need API wiring) 🔨
|
||||
- [x] Landing page (`app/page.tsx`)
|
||||
- [x] Login page with email/password + Google OAuth
|
||||
- [x] Registration page
|
||||
- [x] OAuth callback handler
|
||||
- [x] Dashboard with stats cards and processing runs table
|
||||
- [x] Mail accounts list with CRUD operations
|
||||
- [x] Settings page
|
||||
- [x] `AddMailAccountModal` component (auto-detect, test connection)
|
||||
- [x] `DashboardLayout` with responsive sidebar
|
||||
- [x] `AuthGuard` for protected routes
|
||||
|
||||
### Not Started 📋
|
||||
- [ ] End-to-end testing of frontend against backend API
|
||||
- [ ] Error boundary components
|
||||
- [ ] Loading skeletons / proper loading states
|
||||
- [ ] Notification preferences UI
|
||||
- [ ] Subscription management / billing UI
|
||||
|
||||
---
|
||||
|
||||
## 📅 Milestone Timeline
|
||||
|
||||
### Milestone 1: Security & Infrastructure (Week 1-2) 🔴
|
||||
@@ -253,31 +292,31 @@ Comprehensive task breakdown for repository improvements and production readines
|
||||
| Category | Progress | Status |
|
||||
|----------|----------|--------|
|
||||
| Security | 60% | 🟡 In Progress |
|
||||
| Agentic Infrastructure | 80% | 🟢 Near Complete |
|
||||
| Testing | 30% | 🔴 Needs Work |
|
||||
| CI/CD | 70% | 🟡 In Progress |
|
||||
| Agentic Infrastructure | 95% | 🟢 Near Complete |
|
||||
| Testing | 57% | 🟡 In Progress |
|
||||
| CI/CD | 80% | 🟢 Near Complete |
|
||||
| Code Quality | 40% | 🔴 Needs Work |
|
||||
| Production Ready | 20% | 🔴 Needs Work |
|
||||
| Observability | 10% | 🔴 Needs Work |
|
||||
| Features | 70% | 🟡 In Progress |
|
||||
| Backend Features | 80% | 🟢 Near Complete |
|
||||
| Frontend | 30% | 🔴 Blocked (missing lib/api.ts) |
|
||||
|
||||
**Overall Repository Readiness**: 47% ⚠️
|
||||
**Overall Repository Readiness**: 52% ⚠️
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Next Actions (Priority Order)
|
||||
|
||||
1. **Immediate** (Today):
|
||||
- [ ] Create `frontend/src/lib/api.ts` (frontend is broken without it)
|
||||
- [ ] Fix remaining security issues (bare excepts, datetime, redirect_uri)
|
||||
- [ ] Write 10 more unit tests
|
||||
- [ ] Test security validators work correctly
|
||||
|
||||
2. **This Week**:
|
||||
- [ ] Enable rate limiting
|
||||
- [ ] Add audit logging
|
||||
- [ ] Reach 50% test coverage
|
||||
- [ ] Write more unit tests (target 70% coverage)
|
||||
- [ ] Complete ADR documentation
|
||||
- [ ] Reorganize docs into docs/ directory
|
||||
- [ ] End-to-end test frontend against backend
|
||||
|
||||
3. **Next Week**:
|
||||
- [ ] Kubernetes manifests
|
||||
@@ -319,6 +358,6 @@ Must complete before production:
|
||||
|
||||
---
|
||||
|
||||
**Last Updated**: 2026-02-06
|
||||
**Last Updated**: 2026-03-23
|
||||
**Maintained By**: Development Team
|
||||
**Review Frequency**: Weekly
|
||||
@@ -156,7 +156,7 @@ def get_or_create_user_salt(user_id: int) -> bytes:
|
||||
## Related Decisions
|
||||
|
||||
- See ADR-006 for key management in production
|
||||
- See SECURITY_REPORT.md for security analysis
|
||||
- See ../SECURITY_REPORT.md for security analysis
|
||||
|
||||
## References
|
||||
|
||||
|
||||
Reference in New Issue
Block a user