From 88a44dfe738537ebff4742113492aa5989844cb1 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 10:26:30 +0000 Subject: [PATCH 1/7] Initial plan From b3a0c4bfd8255f3978d250073e1881f3cffefd97 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 10:57:31 +0000 Subject: [PATCH 2/7] Clean up repo: move docs to docs/, add SECURITY.md, .editorconfig, update README with badges, fix cross-references, correct documentation to reflect actual project state Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/71f26285-5584-42b2-8255-8ad2c9e9ecb4 --- .editorconfig | 22 + CONTRIBUTING.md | 51 ++- README.md | 392 ++++++------------ SECURITY.md | 41 ++ ARCHITECTURE.md => docs/ARCHITECTURE.md | 6 +- .../DEPLOYMENT_CHECKLIST.md | 0 FEATURE_SUMMARY.md => docs/FEATURE_SUMMARY.md | 0 .../IMPLEMENTATION_COMPLETE.md | 0 .../IMPLEMENTATION_GUIDE.md | 0 .../IMPROVEMENTS_SUMMARY.md | 0 MIGRATION_GUIDE.md => docs/MIGRATION_GUIDE.md | 0 MVP.md => docs/MVP.md | 0 QUICKSTART.md => docs/QUICKSTART.md | 0 README.NEW.md => docs/README_SAAS.md | 23 +- ROADMAP.md => docs/ROADMAP.md | 2 +- SECURITY_REPORT.md => docs/SECURITY_REPORT.md | 0 .../SECURITY_SUMMARY.md | 0 TESTING_GUIDE.md => docs/TESTING_GUIDE.md | 0 TODO.md => docs/TODO.md | 0 .../UI_DOCUMENTATION.md | 0 .../WEB_INTERFACE_GUIDE.md | 0 docs/adr/002-fernet-encryption.md | 2 +- 22 files changed, 223 insertions(+), 316 deletions(-) create mode 100644 .editorconfig create mode 100644 SECURITY.md rename ARCHITECTURE.md => docs/ARCHITECTURE.md (98%) rename DEPLOYMENT_CHECKLIST.md => docs/DEPLOYMENT_CHECKLIST.md (100%) rename FEATURE_SUMMARY.md => docs/FEATURE_SUMMARY.md (100%) rename IMPLEMENTATION_COMPLETE.md => docs/IMPLEMENTATION_COMPLETE.md (100%) rename IMPLEMENTATION_GUIDE.md => docs/IMPLEMENTATION_GUIDE.md (100%) rename IMPROVEMENTS_SUMMARY.md => docs/IMPROVEMENTS_SUMMARY.md (100%) rename MIGRATION_GUIDE.md => docs/MIGRATION_GUIDE.md (100%) rename MVP.md => docs/MVP.md (100%) rename QUICKSTART.md => docs/QUICKSTART.md (100%) rename README.NEW.md => docs/README_SAAS.md (95%) rename ROADMAP.md => docs/ROADMAP.md (99%) rename SECURITY_REPORT.md => docs/SECURITY_REPORT.md (100%) rename SECURITY_SUMMARY.md => docs/SECURITY_SUMMARY.md (100%) rename TESTING_GUIDE.md => docs/TESTING_GUIDE.md (100%) rename TODO.md => docs/TODO.md (100%) rename UI_DOCUMENTATION.md => docs/UI_DOCUMENTATION.md (100%) rename WEB_INTERFACE_GUIDE.md => docs/WEB_INTERFACE_GUIDE.md (100%) diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 0000000..1c38801 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,22 @@ +# EditorConfig — https://editorconfig.org +root = true + +[*] +indent_style = space +indent_size = 4 +end_of_line = lf +charset = utf-8 +trim_trailing_whitespace = true +insert_final_newline = true + +[*.{js,jsx,ts,tsx,json,css,scss,yml,yaml}] +indent_size = 2 + +[*.md] +trim_trailing_whitespace = false + +[Makefile] +indent_style = tab + +[Dockerfile*] +indent_size = 4 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e704f79..10c64bc 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -20,7 +20,7 @@ Be respectful and inclusive. We welcome contributions from everyone. ### Suggesting Features -1. Check the [Roadmap](ROADMAP.md) to see if it's already planned +1. Check the [Roadmap](docs/ROADMAP.md) to see if it's already planned 2. Open an issue with the "enhancement" label 3. Describe the feature and its use case 4. Explain why it would be useful @@ -40,14 +40,14 @@ Be respectful and inclusive. We welcome contributions from everyone. 4. **Test your changes** ```bash - # Test Python syntax - python3 -m py_compile pop3_forwarder.py - + # Run the test suite + make test + + # Or run linting + formatting + tests together + make quick-test + # Test Docker build docker build -t pop3-test . - - # Test with your configuration - docker-compose up ``` 5. **Commit your changes** @@ -80,19 +80,18 @@ Be respectful and inclusive. We welcome contributions from everyone. git clone https://github.com/YOUR-USERNAME/pop_puller_to_gmail.git cd pop_puller_to_gmail -# Create virtual environment -python3 -m venv venv -source venv/bin/activate # On Windows: venv\Scripts\activate - -# Install dependencies -pip install -r requirements.txt +# Install all development dependencies +make install-dev # Copy example config cp .env.example .env # Edit .env with test credentials -# Run locally +# Run the legacy forwarder script directly python pop3_forwarder.py + +# Or start the SaaS backend in dev mode +make run-dev ``` ### Docker Development @@ -112,23 +111,29 @@ docker run --env-file .env pop3-dev - Add docstrings to functions and classes - Keep functions focused and small - Handle errors gracefully +- Run `make format` to auto-format with Black and Ruff ## Testing Before submitting a PR: -1. **Syntax check** +1. **Run the test suite** ```bash - python3 -m py_compile pop3_forwarder.py + make test ``` -2. **Docker build** +2. **Run linting** + ```bash + make lint + ``` + +3. **Docker build** ```bash docker build -t pop3-test . ``` -3. **Manual testing** - - Test with real POP3 account (or mock) +4. **Manual testing** (if applicable) + - Test with a real POP3 account or mock - Verify emails are forwarded correctly - Check error handling - Review logs @@ -143,9 +148,9 @@ Update documentation when: Files to update: - `README.md` - Main documentation -- `QUICKSTART.md` - If setup changes -- `MVP.md` - If MVP scope changes -- `ROADMAP.md` - If adding future plans +- `docs/QUICKSTART.md` - If setup changes +- `docs/MVP.md` - If MVP scope changes +- `docs/ROADMAP.md` - If adding future plans ## Security @@ -158,7 +163,7 @@ Files to update: **Do NOT open public issues for security vulnerabilities.** -Email security concerns to the maintainers privately. +Please see [SECURITY.md](SECURITY.md) for responsible disclosure instructions. ## Questions? diff --git a/README.md b/README.md index 9d60def..55f21c8 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,61 @@ # POP3 to Gmail Forwarder +[![CI Tests](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/test.yml/badge.svg)](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/test.yml) +[![Lint](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/lint.yml/badge.svg)](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/lint.yml) +[![Security Scan](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/security.yml/badge.svg)](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/security.yml) +[![Docker Build](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/docker-build.yml/badge.svg)](https://github.com/christianlouis/pop_puller_to_gmail/actions/workflows/docker-build.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) +[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/) +[![Docker](https://img.shields.io/badge/docker-ready-blue.svg)](https://www.docker.com/) + A Docker-based solution that automatically fetches emails from POP3 mailboxes and forwards them to Gmail, replacing Google's discontinued POP3 import feature. ## Features -- ✅ **Multiple POP3 Accounts**: Support for unlimited POP3 mailboxes via environment variables -- ✅ **Automatic Forwarding**: Sends emails to your Gmail account via SMTP -- ✅ **Smart Throttling**: Rate limiting to avoid Gmail quotas (configurable emails per minute) -- ✅ **Error Reporting**: Email notifications via Postmarkapp when issues occur -- ✅ **Scheduled Polling**: Configurable check intervals (default: every 5 minutes) -- ✅ **Docker Ready**: Fully containerized with docker-compose support -- ✅ **Secure**: Runs as non-root user, uses SSL/TLS for connections -- ✅ **Production Ready**: Comprehensive logging, error handling, and best practices +- **Multiple POP3 Accounts** — support for unlimited POP3 mailboxes via environment variables +- **Automatic Forwarding** — sends emails to your Gmail account via SMTP +- **Smart Throttling** — configurable rate limiting to stay within Gmail quotas +- **Error Reporting** — notifications via Postmarkapp when issues occur +- **Scheduled Polling** — configurable check intervals (default: every 5 minutes) +- **Docker Ready** — fully containerized with Docker Compose support +- **Secure** — runs as non-root user, SSL/TLS connections + +### SaaS Platform (in development) + +The repository also includes a multi-tenant SaaS backend built with FastAPI, PostgreSQL, Redis, and Celery. It adds multi-user support, OAuth2 authentication, POP3/IMAP protocol support, encrypted credential storage, and background job processing. See the [SaaS README](docs/README_SAAS.md) for details. ## Quick Start -### Prerequisites - -- Docker and Docker Compose installed -- A Gmail account with [App Password](https://support.google.com/accounts/answer/185833) enabled -- POP3 account credentials -- (Optional) Postmarkapp account for error notifications - -### Option 1: Using Pre-built Docker Image (Recommended) - -The Docker images are automatically built and published to GitHub Container Registry. - -1. **Create configuration file** - ```bash - # Download the docker-compose.yml and .env.example - curl -O https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/docker-compose.yml - curl -o .env https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/.env.example - - # Edit .env with your credentials - nano .env - ``` - -2. **Update docker-compose.yml to use the pre-built image** - ```yaml - version: '3.8' - - services: - pop3-forwarder: - image: ghcr.io/christianlouis/pop_puller_to_gmail:latest - container_name: pop3-gmail-forwarder - restart: unless-stopped - env_file: - - .env - ``` - -3. **Run the container** - ```bash - docker-compose up -d - ``` - -### Option 2: Building from Source - -1. **Clone the repository** - ```bash - git clone https://github.com/christianlouis/pop_puller_to_gmail.git - cd pop_puller_to_gmail - ``` - -2. **Configure environment variables** - ```bash - cp .env.example .env - # Edit .env with your credentials - nano .env - ``` - -3. **Essential Configuration** - - Edit `.env` and set: - - ```bash - # Your POP3 account(s) - POP3_ACCOUNT_1_HOST=pop.yourprovider.com - POP3_ACCOUNT_1_PORT=995 - POP3_ACCOUNT_1_USER=your-email@provider.com - POP3_ACCOUNT_1_PASSWORD=your-password - - # Your Gmail SMTP settings - SMTP_USER=your-gmail@gmail.com - SMTP_PASSWORD=your-app-password # Generate at myaccount.google.com/apppasswords - GMAIL_DESTINATION=your-gmail@gmail.com - - # Optional: Postmarkapp for error notifications - POSTMARK_API_TOKEN=your-token - POSTMARK_FROM_EMAIL=errors@yourdomain.com - POSTMARK_TO_EMAIL=admin@yourdomain.com - ``` - -4. **Run with Docker Compose** - ```bash - docker-compose up -d - ``` - -5. **Check logs** - ```bash - docker-compose logs -f - ``` - -## Using Pre-built Docker Images - -Docker images are automatically built and published to GitHub Container Registry for every release and commit to the main branch. - -### Available Image Tags - -- `ghcr.io/christianlouis/pop_puller_to_gmail:latest` - Latest build from main branch -- `ghcr.io/christianlouis/pop_puller_to_gmail:v1.0.0` - Specific version tags -- `ghcr.io/christianlouis/pop_puller_to_gmail:main` - Main branch builds - -### Pull and Run +### Using a Pre-built Docker Image (Recommended) ```bash -# Pull the latest image -docker pull ghcr.io/christianlouis/pop_puller_to_gmail:latest +# Pull and configure +curl -O https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/docker-compose.yml +curl -o .env https://raw.githubusercontent.com/christianlouis/pop_puller_to_gmail/main/.env.example -# Run directly with Docker -docker run -d \ - --name pop3-forwarder \ - --env-file .env \ - --restart unless-stopped \ - ghcr.io/christianlouis/pop_puller_to_gmail:latest +# Edit .env with your credentials +nano .env + +# Start +docker-compose up -d ``` +### Building from Source + +```bash +git clone https://github.com/christianlouis/pop_puller_to_gmail.git +cd pop_puller_to_gmail +cp .env.example .env # then edit .env +docker-compose up -d +``` + +See the [Quick Start Guide](docs/QUICKSTART.md) for detailed instructions. + ## Configuration ### POP3 Accounts -Add multiple POP3 accounts by incrementing the account number: +Add multiple POP3 accounts by incrementing the account number in your `.env`: ```bash POP3_ACCOUNT_1_HOST=pop.provider1.com @@ -139,199 +65,115 @@ POP3_ACCOUNT_1_PASSWORD=password1 POP3_ACCOUNT_2_HOST=pop.provider2.com POP3_ACCOUNT_2_USER=user2@provider2.com POP3_ACCOUNT_2_PASSWORD=password2 - -# ... add more as needed ``` ### Gmail App Password -1. Go to your Google Account: https://myaccount.google.com/ -2. Select Security -3. Under "Signing in to Google," select App Passwords -4. Generate a new app password for "Mail" -5. Use this password in `SMTP_PASSWORD` +1. Go to your [Google Account Security](https://myaccount.google.com/security) +2. Under "Signing in to Google," select **App Passwords** +3. Generate a new app password for "Mail" +4. Use this password as `SMTP_PASSWORD` ### Environment Variables | Variable | Required | Default | Description | |----------|----------|---------|-------------| -| `POP3_ACCOUNT_N_HOST` | Yes | - | POP3 server hostname | -| `POP3_ACCOUNT_N_PORT` | No | 995 | POP3 server port | -| `POP3_ACCOUNT_N_USER` | Yes | - | POP3 username | -| `POP3_ACCOUNT_N_PASSWORD` | Yes | - | POP3 password | -| `POP3_ACCOUNT_N_USE_SSL` | No | true | Use SSL/TLS | -| `SMTP_HOST` | No | smtp.gmail.com | SMTP server | -| `SMTP_PORT` | No | 587 | SMTP port | -| `SMTP_USER` | Yes | - | SMTP username | -| `SMTP_PASSWORD` | Yes | - | SMTP password (App Password) | -| `SMTP_USE_TLS` | No | true | Use STARTTLS | -| `GMAIL_DESTINATION` | Yes | - | Destination Gmail address | -| `CHECK_INTERVAL_MINUTES` | No | 5 | How often to check for new mail | -| `MAX_EMAILS_PER_RUN` | No | 50 | Max emails to process per account per run | -| `THROTTLE_EMAILS_PER_MINUTE` | No | 10 | Rate limit for sending emails | -| `POSTMARK_API_TOKEN` | No | - | Postmarkapp API token | -| `POSTMARK_FROM_EMAIL` | No | - | Error notification sender | -| `POSTMARK_TO_EMAIL` | No | - | Error notification recipient | -| `LOG_LEVEL` | No | INFO | Logging level (DEBUG, INFO, WARNING, ERROR) | +| `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname | +| `POP3_ACCOUNT_N_PORT` | No | `995` | POP3 server port | +| `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username | +| `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password | +| `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL/TLS | +| `SMTP_HOST` | No | `smtp.gmail.com` | SMTP server | +| `SMTP_PORT` | No | `587` | SMTP port | +| `SMTP_USER` | Yes | — | SMTP username | +| `SMTP_PASSWORD` | Yes | — | SMTP password (App Password) | +| `SMTP_USE_TLS` | No | `true` | Use STARTTLS | +| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address | +| `CHECK_INTERVAL_MINUTES` | No | `5` | Polling interval | +| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per run | +| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit | +| `POSTMARK_API_TOKEN` | No | — | Postmarkapp API token | +| `POSTMARK_FROM_EMAIL` | No | — | Error notification sender | +| `POSTMARK_TO_EMAIL` | No | — | Error notification recipient | +| `LOG_LEVEL` | No | `INFO` | Logging level | ## How It Works -1. **Polling**: The application checks configured POP3 mailboxes at regular intervals -2. **Fetching**: Retrieves new emails from each POP3 account -3. **Forwarding**: Sends emails to your Gmail account via SMTP with original metadata preserved -4. **Cleanup**: Deletes emails from POP3 server after successful forwarding -5. **Throttling**: Respects rate limits to avoid Gmail quota issues -6. **Error Handling**: Sends notifications via Postmarkapp if issues occur - -## Email Format - -Forwarded emails include: -- Original sender information in the subject line: `[Fwd from user@provider.com] Original Subject` -- Header section with original From, Date, Subject, and source account -- Original email body preserved - -## Monitoring and Logs - -### View logs -```bash -docker-compose logs -f pop3-forwarder -``` - -### Check container status -```bash -docker-compose ps -``` - -### Restart the service -```bash -docker-compose restart -``` - -## Troubleshooting - -### Gmail Authentication Issues - -**Problem**: "Username and Password not accepted" - -**Solution**: -- Ensure 2FA is enabled on your Google account -- Generate an App Password (don't use your regular Gmail password) -- Use the 16-character app password without spaces - -### POP3 Connection Issues - -**Problem**: "Connection refused" or "SSL error" - -**Solution**: -- Verify POP3 server hostname and port -- Check if POP3 is enabled in your email provider settings -- Try with `POP3_ACCOUNT_N_USE_SSL=false` for non-SSL connections (port 110) - -### No Emails Being Forwarded - -**Problem**: Container runs but no emails are forwarded - -**Solution**: -- Check if there are emails in your POP3 mailbox -- Review logs for errors: `docker-compose logs -f` -- Verify `GMAIL_DESTINATION` is correct -- Check Gmail spam folder - -### Rate Limiting - -**Problem**: "Too many requests" or quota errors - -**Solution**: -- Increase `CHECK_INTERVAL_MINUTES` -- Decrease `THROTTLE_EMAILS_PER_MINUTE` -- Reduce `MAX_EMAILS_PER_RUN` - -## Security Best Practices - -1. **Never commit `.env` file** - It contains sensitive credentials -2. **Use App Passwords** - Don't use your main Gmail password -3. **Rotate credentials regularly** - Update passwords periodically -4. **Enable 2FA** - On all email accounts -5. **Review logs** - Monitor for suspicious activity -6. **Use SSL/TLS** - Keep `USE_SSL` and `USE_TLS` enabled -7. **Limit network access** - Use firewall rules if needed - -## Development - -### Local Development (without Docker) - -```bash -# Create virtual environment -python3 -m venv venv -source venv/bin/activate # On Windows: venv\Scripts\activate - -# Install dependencies -pip install -r requirements.txt - -# Copy and configure .env -cp .env.example .env -# Edit .env with your settings - -# Run the application -python pop3_forwarder.py -``` - -### Building the Docker Image - -```bash -docker build -t pop3-gmail-forwarder . -``` - -### Running Tests - -```bash -# Run with verbose logging -LOG_LEVEL=DEBUG docker-compose up -``` - -## Architecture - ``` ┌─────────────────┐ │ POP3 Server 1 │ └────────┬────────┘ - │ │ (Fetch emails) - │ ▼ ┌─────────────────┐ ┌──────────────┐ ┌─────────────┐ │ POP3 Server 2 │─────▶│ Forwarder │─────▶│ Gmail │ └─────────────────┘ │ Container │ │ (SMTP) │ │ └──────┬───────┘ └─────────────┘ - │ │ -┌────────▼────────┐ │ -│ POP3 Server N │ │ -└─────────────────┘ │ - │ (Error notifications) - ▼ - ┌─────────────────┐ +┌────────▼────────┐ │ (Error notifications) +│ POP3 Server N │ ▼ +└─────────────────┘ ┌─────────────────┐ │ Postmarkapp │ └─────────────────┘ ``` +1. **Polling** — checks POP3 mailboxes at the configured interval +2. **Fetching** — retrieves new emails from each account +3. **Forwarding** — delivers to Gmail with original metadata preserved +4. **Cleanup** — deletes from POP3 after successful forwarding +5. **Throttling** — respects rate limits to avoid quota issues +6. **Error Handling** — sends notifications if something goes wrong + +## Development + +```bash +# Install dependencies +make install-dev + +# Run linting & formatting +make lint +make format + +# Run tests +make test + +# Start backend in dev mode +make run-dev +``` + +See the [Testing Guide](docs/TESTING_GUIDE.md) for the full test workflow. + +## Documentation + +Detailed documentation lives in the [`docs/`](docs/) directory: + +| Document | Description | +|----------|-------------| +| [Architecture](docs/ARCHITECTURE.md) | System design and component overview | +| [Quick Start](docs/QUICKSTART.md) | Step-by-step setup guide | +| [Migration Guide](docs/MIGRATION_GUIDE.md) | Upgrading from v1 to v2 | +| [Deployment Checklist](docs/DEPLOYMENT_CHECKLIST.md) | Production deployment guide | +| [Roadmap](docs/ROADMAP.md) | Planned features and milestones | +| [Testing Guide](docs/TESTING_GUIDE.md) | How to run and write tests | +| [Coding Patterns](docs/CODING_PATTERNS.md) | Code style and conventions | +| [SaaS README](docs/README_SAAS.md) | Multi-tenant SaaS platform details | + ## Contributing -Contributions are welcome! Please: +Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on: -1. Fork the repository -2. Create a feature branch -3. Make your changes -4. Submit a pull request +- Reporting bugs and suggesting features +- Development setup and code style +- Pull request process + +## Security + +To report a vulnerability, please see [SECURITY.md](SECURITY.md). **Do not open public issues for security concerns.** ## License -MIT License - See LICENSE file for details +This project is licensed under the MIT License — see [LICENSE](LICENSE) for details. ## Support -- **Issues**: https://github.com/christianlouis/pop_puller_to_gmail/issues -- **Discussions**: https://github.com/christianlouis/pop_puller_to_gmail/discussions - -## Acknowledgments - -Built to replace Gmail's discontinued POP3 import feature. Uses industry-standard Python libraries for email handling and Docker for easy deployment. +- [Issue Tracker](https://github.com/christianlouis/pop_puller_to_gmail/issues) +- [Discussions](https://github.com/christianlouis/pop_puller_to_gmail/discussions) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..eb88245 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,41 @@ +# Security Policy + +## Supported Versions + +| Version | Supported | +|---------|--------------------| +| 2.x | ✅ Yes | +| 1.x | ❌ No | +| < 1.0 | ❌ No | + +## Reporting a Vulnerability + +**Please do NOT open public issues for security vulnerabilities.** + +If you discover a security vulnerability, please report it responsibly: + +1. **Email**: Send details to the repository maintainer via the email listed on the [GitHub profile](https://github.com/christianlouis). +2. **GitHub Private Vulnerability Reporting**: Use [GitHub's security advisory feature](https://github.com/christianlouis/pop_puller_to_gmail/security/advisories/new) to report privately. + +### What to Include + +- A description of the vulnerability +- Steps to reproduce the issue +- Potential impact +- Suggested fix (if any) + +### Response Timeline + +- **Acknowledgment**: Within 48 hours +- **Initial Assessment**: Within 1 week +- **Fix & Disclosure**: Coordinated with the reporter + +## Security Best Practices for Users + +- **Never commit `.env` files** containing credentials +- **Use App Passwords** for Gmail instead of your main password +- **Enable 2FA** on all email accounts +- **Rotate credentials** regularly +- **Use SSL/TLS** for all mail connections +- **Run containers as non-root** (default in provided Dockerfile) +- **Keep dependencies updated** — Dependabot is enabled on this repository diff --git a/ARCHITECTURE.md b/docs/ARCHITECTURE.md similarity index 98% rename from ARCHITECTURE.md rename to docs/ARCHITECTURE.md index 612afd2..fe92608 100644 --- a/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -316,9 +316,9 @@ alembic downgrade -1 ## 📚 Additional Documentation - [API Documentation](http://localhost:8000/api/docs) - Interactive API docs -- [CONTRIBUTING.md](CONTRIBUTING.md) - Contribution guidelines +- [CONTRIBUTING.md](../CONTRIBUTING.md) - Contribution guidelines - [ROADMAP.md](ROADMAP.md) - Future development plans -- [SECURITY.md](SECURITY.md) - Security policies +- [SECURITY.md](../SECURITY.md) - Security policies ## 🤝 Contributing @@ -331,7 +331,7 @@ Contributions welcome! Please: ## 📄 License -MIT License - See [LICENSE](LICENSE) file +MIT License - See [LICENSE](../LICENSE) file ## 🆘 Support diff --git a/DEPLOYMENT_CHECKLIST.md b/docs/DEPLOYMENT_CHECKLIST.md similarity index 100% rename from DEPLOYMENT_CHECKLIST.md rename to docs/DEPLOYMENT_CHECKLIST.md diff --git a/FEATURE_SUMMARY.md b/docs/FEATURE_SUMMARY.md similarity index 100% rename from FEATURE_SUMMARY.md rename to docs/FEATURE_SUMMARY.md diff --git a/IMPLEMENTATION_COMPLETE.md b/docs/IMPLEMENTATION_COMPLETE.md similarity index 100% rename from IMPLEMENTATION_COMPLETE.md rename to docs/IMPLEMENTATION_COMPLETE.md diff --git a/IMPLEMENTATION_GUIDE.md b/docs/IMPLEMENTATION_GUIDE.md similarity index 100% rename from IMPLEMENTATION_GUIDE.md rename to docs/IMPLEMENTATION_GUIDE.md diff --git a/IMPROVEMENTS_SUMMARY.md b/docs/IMPROVEMENTS_SUMMARY.md similarity index 100% rename from IMPROVEMENTS_SUMMARY.md rename to docs/IMPROVEMENTS_SUMMARY.md diff --git a/MIGRATION_GUIDE.md b/docs/MIGRATION_GUIDE.md similarity index 100% rename from MIGRATION_GUIDE.md rename to docs/MIGRATION_GUIDE.md diff --git a/MVP.md b/docs/MVP.md similarity index 100% rename from MVP.md rename to docs/MVP.md diff --git a/QUICKSTART.md b/docs/QUICKSTART.md similarity index 100% rename from QUICKSTART.md rename to docs/QUICKSTART.md diff --git a/README.NEW.md b/docs/README_SAAS.md similarity index 95% rename from README.NEW.md rename to docs/README_SAAS.md index 6baaf59..36d37ff 100644 --- a/README.NEW.md +++ b/docs/README_SAAS.md @@ -312,7 +312,7 @@ Helm charts and Kubernetes manifests will be provided for production deployment. ## 🤝 Contributing -Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. +Contributions are welcome! Please see [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelines. ### Areas for Contribution @@ -325,7 +325,7 @@ Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for gui ## 📜 License -MIT License - See [LICENSE](LICENSE) file for details. +MIT License - See [LICENSE](../LICENSE) file for details. ## 🆘 Support @@ -357,16 +357,19 @@ Built with these amazing open-source projects: | Background Jobs | ✅ Complete | 100% | | Documentation | ✅ Complete | 100% | | Stripe Integration | 🚧 In Progress | 60% | -| Frontend Dashboard | 📋 Planned | 0% | -| Notification System | 📋 Planned | 40% | -| Testing Suite | 📋 Planned | 20% | +| Frontend Dashboard | 🚧 In Progress | 70% | +| Notification System | 🚧 In Progress | 40% | +| Testing Suite | 🚧 In Progress | 30% | + +> **Note:** The frontend pages and components are implemented but the API client +> layer (`lib/api.ts`) is not yet wired up, so the dashboard does not function +> end-to-end yet. ## 🔮 Roadmap See [ROADMAP.md](ROADMAP.md) for detailed future plans, including: - Complete web dashboard -- Mobile app (iOS/Android) - Advanced email filtering - Email archiving - Multi-destination forwarding @@ -374,12 +377,6 @@ See [ROADMAP.md](ROADMAP.md) for detailed future plans, including: - Kubernetes deployment - High availability setup -## ⭐ Star History - -If you find this project useful, please consider giving it a star! ⭐ - --- -**Version**: 2.0.0 | **Status**: Production Ready (Backend) | **Updated**: 2026-02-01 - -Made with ❤️ for the community +**Status**: In Development | **Backend**: Production-ready | **Frontend**: In Progress diff --git a/ROADMAP.md b/docs/ROADMAP.md similarity index 99% rename from ROADMAP.md rename to docs/ROADMAP.md index de1ed6e..da39ac5 100644 --- a/ROADMAP.md +++ b/docs/ROADMAP.md @@ -243,7 +243,7 @@ We welcome contributions! Areas where help is needed: 5. **Performance**: Optimize slow operations 6. **Security**: Security audits and improvements -See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. +See [CONTRIBUTING.md](../CONTRIBUTING.md) for guidelines. --- diff --git a/SECURITY_REPORT.md b/docs/SECURITY_REPORT.md similarity index 100% rename from SECURITY_REPORT.md rename to docs/SECURITY_REPORT.md diff --git a/SECURITY_SUMMARY.md b/docs/SECURITY_SUMMARY.md similarity index 100% rename from SECURITY_SUMMARY.md rename to docs/SECURITY_SUMMARY.md diff --git a/TESTING_GUIDE.md b/docs/TESTING_GUIDE.md similarity index 100% rename from TESTING_GUIDE.md rename to docs/TESTING_GUIDE.md diff --git a/TODO.md b/docs/TODO.md similarity index 100% rename from TODO.md rename to docs/TODO.md diff --git a/UI_DOCUMENTATION.md b/docs/UI_DOCUMENTATION.md similarity index 100% rename from UI_DOCUMENTATION.md rename to docs/UI_DOCUMENTATION.md diff --git a/WEB_INTERFACE_GUIDE.md b/docs/WEB_INTERFACE_GUIDE.md similarity index 100% rename from WEB_INTERFACE_GUIDE.md rename to docs/WEB_INTERFACE_GUIDE.md diff --git a/docs/adr/002-fernet-encryption.md b/docs/adr/002-fernet-encryption.md index 007fff4..07be933 100644 --- a/docs/adr/002-fernet-encryption.md +++ b/docs/adr/002-fernet-encryption.md @@ -156,7 +156,7 @@ def get_or_create_user_salt(user_id: int) -> bytes: ## Related Decisions - See ADR-006 for key management in production -- See SECURITY_REPORT.md for security analysis +- See ../SECURITY_REPORT.md for security analysis ## References From 1263ed5266c0e34d0f27cb3539467eaa5c9fe907 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 11:02:09 +0000 Subject: [PATCH 3/7] Update TODO.md: document missing frontend API client, mark completed items, update progress tracking Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/71f26285-5584-42b2-8255-8ad2c9e9ecb4 --- docs/TODO.md | 51 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 43 insertions(+), 8 deletions(-) diff --git a/docs/TODO.md b/docs/TODO.md index d2afb90..4ed44ef 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -43,8 +43,8 @@ Comprehensive task breakdown for repository improvements and production readines - [x] Add `.secrets.baseline` for detect-secrets ### In Progress 🔨 +- [x] Reorganize documentation into `docs/` directory - [ ] Complete ADR documentation (add ADR-003 through ADR-010) -- [ ] Reorganize documentation into `docs/` directory - [ ] Create GitHub Projects board for task management ### Not Started 📋 @@ -169,12 +169,46 @@ Comprehensive task breakdown for repository improvements and production readines - [ ] Add advanced email filtering - [ ] Implement OAuth2 for Gmail (instead of App Passwords) - [ ] Add attachment handling improvements -- [ ] Build frontend dashboard (React/Next.js) - [ ] Add email archiving feature - [ ] Implement webhook support for external integrations --- +## 🖥️ High Priority - Frontend Completion + +The Next.js frontend has pages and components implemented but is **not functional** +because the API client layer is missing. + +### Critical Blockers 🔴 +- [ ] Create `frontend/src/lib/api.ts` — API client using axios + - Must export: `authApi`, `mailAccountsApi`, `processingRunsApi`, `userApi` + - Must export types: `User`, `MailAccount`, `MailAccountCreate` + - 8 files import from `@/lib/api` and will fail to compile without it: + `AuthGuard.tsx`, `AddMailAccountModal.tsx`, `authStore.ts`, + `login/page.tsx`, `register/page.tsx`, `auth/callback/page.tsx`, + `dashboard/page.tsx`, `accounts/page.tsx` + +### Existing Pages (UI done, need API wiring) 🔨 +- [x] Landing page (`app/page.tsx`) +- [x] Login page with email/password + Google OAuth +- [x] Registration page +- [x] OAuth callback handler +- [x] Dashboard with stats cards and processing runs table +- [x] Mail accounts list with CRUD operations +- [x] Settings page +- [x] `AddMailAccountModal` component (auto-detect, test connection) +- [x] `DashboardLayout` with responsive sidebar +- [x] `AuthGuard` for protected routes + +### Not Started 📋 +- [ ] End-to-end testing of frontend against backend API +- [ ] Error boundary components +- [ ] Loading skeletons / proper loading states +- [ ] Notification preferences UI +- [ ] Subscription management / billing UI + +--- + ## 📅 Milestone Timeline ### Milestone 1: Security & Infrastructure (Week 1-2) 🔴 @@ -253,31 +287,32 @@ Comprehensive task breakdown for repository improvements and production readines | Category | Progress | Status | |----------|----------|--------| | Security | 60% | 🟡 In Progress | -| Agentic Infrastructure | 80% | 🟢 Near Complete | +| Agentic Infrastructure | 90% | 🟢 Near Complete | | Testing | 30% | 🔴 Needs Work | | CI/CD | 70% | 🟡 In Progress | | Code Quality | 40% | 🔴 Needs Work | | Production Ready | 20% | 🔴 Needs Work | | Observability | 10% | 🔴 Needs Work | -| Features | 70% | 🟡 In Progress | +| Backend Features | 80% | 🟢 Near Complete | +| Frontend | 30% | 🔴 Blocked (missing lib/api.ts) | -**Overall Repository Readiness**: 47% ⚠️ +**Overall Repository Readiness**: 48% ⚠️ --- ## 🎯 Next Actions (Priority Order) 1. **Immediate** (Today): + - [ ] Create `frontend/src/lib/api.ts` (frontend is broken without it) - [ ] Fix remaining security issues (bare excepts, datetime, redirect_uri) - [ ] Write 10 more unit tests - - [ ] Test security validators work correctly 2. **This Week**: - [ ] Enable rate limiting - [ ] Add audit logging - [ ] Reach 50% test coverage - [ ] Complete ADR documentation - - [ ] Reorganize docs into docs/ directory + - [ ] End-to-end test frontend against backend 3. **Next Week**: - [ ] Kubernetes manifests @@ -319,6 +354,6 @@ Must complete before production: --- -**Last Updated**: 2026-02-06 +**Last Updated**: 2026-03-23 **Maintained By**: Development Team **Review Frequency**: Weekly From 3d48bc6f663cfc38cd47828dc093ab3adb90b200 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 11:12:14 +0000 Subject: [PATCH 4/7] Initial plan From 1402afb9ce4b45efba69b4240f7c9e6c60eed51e Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 11:13:45 +0000 Subject: [PATCH 5/7] Initial plan From a84a4fe033143c11cb41e07bcc2e6a3cbb3bb079 Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 11:18:33 +0000 Subject: [PATCH 6/7] Add comprehensive deployment guide (docs/DEPLOYMENT_GUIDE.md) Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/0204699a-6f9d-4481-a23c-b543fee5e3fc --- docs/DEPLOYMENT_GUIDE.md | 1038 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 1038 insertions(+) create mode 100644 docs/DEPLOYMENT_GUIDE.md diff --git a/docs/DEPLOYMENT_GUIDE.md b/docs/DEPLOYMENT_GUIDE.md new file mode 100644 index 0000000..cefeba1 --- /dev/null +++ b/docs/DEPLOYMENT_GUIDE.md @@ -0,0 +1,1038 @@ +# Deployment Guide + +This guide walks you through deploying **POP3 to Gmail Forwarder** from scratch — whether you just want a single container pulling emails, a full multi-service SaaS stack with Docker Compose, or a production-grade Kubernetes setup. + +--- + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Deployment Options at a Glance](#deployment-options-at-a-glance) +- [Option 1 — Legacy Single-Container Deployment](#option-1--legacy-single-container-deployment) +- [Option 2 — Full SaaS Stack with Docker Compose](#option-2--full-saas-stack-with-docker-compose) +- [Option 3 — Kubernetes Deployment](#option-3--kubernetes-deployment) +- [Google OAuth and Gmail API Setup](#google-oauth-and-gmail-api-setup) +- [Reverse Proxy and TLS](#reverse-proxy-and-tls) +- [Environment Variable Reference](#environment-variable-reference) +- [Upgrading](#upgrading) +- [Troubleshooting](#troubleshooting) + +--- + +## Prerequisites + +| Requirement | Minimum | Recommended | +|---|---|---| +| **Docker** | 20.10+ | Latest stable | +| **Docker Compose** | v2.0+ | Latest stable | +| **RAM** | 1 GB (legacy) / 4 GB (SaaS) | 8 GB (SaaS) | +| **Disk** | 10 GB (legacy) / 40 GB (SaaS) | 80 GB (SaaS) | +| **CPU** | 1 vCPU (legacy) / 2 vCPU (SaaS) | 4 vCPU (SaaS) | + +You will also need: + +- A **Gmail account** with [2-Step Verification](https://myaccount.google.com/signinoptions/two-step-verification) enabled and an [App Password](https://myaccount.google.com/apppasswords) generated (for SMTP delivery). +- Credentials for one or more **POP3 mailboxes** you want to pull email from. +- *(SaaS stack only)* A **Google Cloud project** with OAuth 2.0 credentials if you want Google sign-in or Gmail API injection (see [Google OAuth and Gmail API Setup](#google-oauth-and-gmail-api-setup)). + +--- + +## Deployment Options at a Glance + +| | Legacy | SaaS (Docker Compose) | SaaS (Kubernetes) | +|---|---|---|---| +| **Services** | 1 container | 6 containers | 6+ pods | +| **Database** | None | PostgreSQL | PostgreSQL | +| **Queue** | None | Redis + Celery | Redis + Celery | +| **Web UI** | None | Next.js frontend | Next.js frontend | +| **Multi-user** | No | Yes | Yes | +| **Best for** | Personal / single mailbox | Small teams / self-hosted | Production / scale | + +--- + +## Option 1 — Legacy Single-Container Deployment + +The legacy mode runs a single Python script (`pop3_forwarder.py`) that polls POP3 mailboxes and forwards email via SMTP. No database, no web UI — just a container and an `.env` file. + +### 1. Create the environment file + +```bash +git clone https://github.com/christianlouis/pop_puller_to_gmail.git +cd pop_puller_to_gmail +cp .env.example .env +``` + +Edit `.env` with your credentials: + +```ini +# POP3 source mailbox +POP3_ACCOUNT_1_HOST=pop.example.com +POP3_ACCOUNT_1_PORT=995 +POP3_ACCOUNT_1_USER=user@example.com +POP3_ACCOUNT_1_PASSWORD=your_password +POP3_ACCOUNT_1_USE_SSL=true + +# Gmail SMTP destination +SMTP_HOST=smtp.gmail.com +SMTP_PORT=587 +SMTP_USER=you@gmail.com +SMTP_PASSWORD=xxxx-xxxx-xxxx-xxxx # Gmail App Password +SMTP_USE_TLS=true +GMAIL_DESTINATION=you@gmail.com + +# Tuning (optional) +CHECK_INTERVAL_MINUTES=5 +MAX_EMAILS_PER_RUN=50 +THROTTLE_EMAILS_PER_MINUTE=10 +LOG_LEVEL=INFO +``` + +> **Tip:** Add more accounts by duplicating the `POP3_ACCOUNT_*` block with an incremented number (`POP3_ACCOUNT_2_*`, `POP3_ACCOUNT_3_*`, etc.). + +### 2. Docker Compose file + +The repository ships `docker-compose.yml` for this mode. Here is the content for reference: + +```yaml +version: "3.8" + +services: + pop3-forwarder: + # Build from source + build: . + # Or use the pre-built image: + # image: ghcr.io/christianlouis/pop_puller_to_gmail:latest + container_name: pop3-gmail-forwarder + restart: unless-stopped + env_file: + - .env + environment: + - LOG_LEVEL=${LOG_LEVEL:-INFO} + - CHECK_INTERVAL_MINUTES=${CHECK_INTERVAL_MINUTES:-5} + - MAX_EMAILS_PER_RUN=${MAX_EMAILS_PER_RUN:-50} + - THROTTLE_EMAILS_PER_MINUTE=${THROTTLE_EMAILS_PER_MINUTE:-10} + volumes: + - ./logs:/app/logs + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "3" +``` + +### 3. Start + +```bash +docker compose up -d +docker compose logs -f # watch the output +``` + +The forwarder will check for new mail every 5 minutes (configurable) and forward messages to your Gmail inbox. + +--- + +## Option 2 — Full SaaS Stack with Docker Compose + +The SaaS stack gives you a multi-user web application with a React frontend, FastAPI backend, PostgreSQL database, Redis cache, and Celery workers for background email processing. + +### 1. Generate secrets + +```bash +# Generate a 64-character hex secret for JWT signing +openssl rand -hex 32 +# Generate a separate key for encrypting stored credentials +openssl rand -hex 32 +``` + +Save both values — you will need them below. + +### 2. Create the backend environment file + +```bash +cd pop_puller_to_gmail +cp backend/.env.example backend/.env +``` + +Edit `backend/.env`: + +```ini +# ── Database ────────────────────────────────────────────── +DATABASE_URL=postgresql+asyncpg://postgres:change-me@postgres:5432/pop3_forwarder + +# ── Security (paste the values you generated above) ────── +SECRET_KEY= +ENCRYPTION_KEY= + +# ── Redis / Celery ─────────────────────────────────────── +REDIS_URL=redis://redis:6379/0 +CELERY_BROKER_URL=redis://redis:6379/0 +CELERY_RESULT_BACKEND=redis://redis:6379/0 + +# ── Google OAuth (optional — see setup section below) ──── +# GOOGLE_CLIENT_ID= +# GOOGLE_CLIENT_SECRET= +# GOOGLE_REDIRECT_URI=https://your-domain.com/auth/callback/google + +# ── CORS (include your frontend URL) ──────────────────── +CORS_ORIGINS=http://localhost:3000 + +# ── Admin account (created on first startup) ───────────── +ADMIN_EMAIL=admin@example.com +ADMIN_PASSWORD=change-this-to-a-strong-password + +# ── Application ───────────────────────────────────────── +DEBUG=false +LOG_LEVEL=INFO +HOST=0.0.0.0 +PORT=8000 +``` + +### 3. Production Docker Compose file + +Below is a production-ready `docker-compose.prod.yml`. It is based on the `docker-compose.new.yml` that ships with the repository, hardened for production use: + +```yaml +version: "3.8" + +services: + # ── PostgreSQL ─────────────────────────────────────────── + postgres: + image: postgres:15-alpine + container_name: pop3-postgres + restart: unless-stopped + environment: + POSTGRES_USER: postgres + POSTGRES_PASSWORD: change-me # must match DATABASE_URL + POSTGRES_DB: pop3_forwarder + volumes: + - postgres_data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U postgres"] + interval: 10s + timeout: 5s + retries: 5 + # Do NOT expose the port in production unless you need + # external access — keep it on the internal network only. + # ports: + # - "5432:5432" + + # ── Redis ──────────────────────────────────────────────── + redis: + image: redis:7-alpine + container_name: pop3-redis + restart: unless-stopped + command: redis-server --appendonly yes + volumes: + - redis_data:/data + healthcheck: + test: ["CMD", "redis-cli", "ping"] + interval: 10s + timeout: 5s + retries: 5 + + # ── FastAPI Backend ────────────────────────────────────── + backend: + build: + context: ./backend + dockerfile: Dockerfile + container_name: pop3-backend + restart: unless-stopped + ports: + - "8000:8000" + env_file: + - ./backend/.env + depends_on: + postgres: + condition: service_healthy + redis: + condition: service_healthy + command: > + sh -c "alembic upgrade head && + uvicorn app.main:app --host 0.0.0.0 --port 8000" + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + # ── Celery Worker ──────────────────────────────────────── + celery-worker: + build: + context: ./backend + dockerfile: Dockerfile + container_name: pop3-celery-worker + restart: unless-stopped + env_file: + - ./backend/.env + depends_on: + - backend + command: celery -A app.workers.celery_app worker --loglevel=info --concurrency=2 + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + # ── Celery Beat (scheduler) ───────────────────────────── + celery-beat: + build: + context: ./backend + dockerfile: Dockerfile + container_name: pop3-celery-beat + restart: unless-stopped + env_file: + - ./backend/.env + depends_on: + - backend + command: celery -A app.workers.celery_app beat --loglevel=info + logging: + driver: "json-file" + options: + max-size: "10m" + max-file: "5" + + # ── Next.js Frontend ──────────────────────────────────── + frontend: + build: + context: ./frontend + dockerfile: Dockerfile + container_name: pop3-frontend + restart: unless-stopped + ports: + - "3000:3000" + environment: + - NEXT_PUBLIC_API_URL=http://backend:8000 + depends_on: + - backend + +volumes: + postgres_data: + redis_data: +``` + +### 4. Build and start + +```bash +# Build all images +docker compose -f docker-compose.prod.yml build + +# Start in detached mode +docker compose -f docker-compose.prod.yml up -d + +# Verify all services are healthy +docker compose -f docker-compose.prod.yml ps +``` + +### 5. Verify + +```bash +# Backend health check +curl http://localhost:8000/health + +# Open the frontend +open http://localhost:3000 + +# Watch logs +docker compose -f docker-compose.prod.yml logs -f +``` + +### 6. Create the first admin account + +If you set `ADMIN_EMAIL` and `ADMIN_PASSWORD` in `backend/.env`, an admin account is created automatically on first startup. Otherwise you can register via the API: + +```bash +curl -X POST http://localhost:8000/api/v1/auth/register \ + -H "Content-Type: application/json" \ + -d '{"email":"you@example.com","password":"your-password","full_name":"Your Name"}' +``` + +--- + +## Option 3 — Kubernetes Deployment + +Below is a set of example Kubernetes manifests to get you started. Adapt namespaces, resource limits, and Ingress rules to your cluster. + +### Namespace + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: pop3-forwarder +``` + +### Secrets + +Store sensitive values in a Kubernetes Secret. In production, consider using an external secret manager (e.g., HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets). + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: pop3-forwarder-secrets + namespace: pop3-forwarder +type: Opaque +stringData: + SECRET_KEY: "" + ENCRYPTION_KEY: "" + DATABASE_URL: "postgresql+asyncpg://postgres:change-me@postgres:5432/pop3_forwarder" + REDIS_URL: "redis://redis:6379/0" + CELERY_BROKER_URL: "redis://redis:6379/0" + CELERY_RESULT_BACKEND: "redis://redis:6379/0" + ADMIN_EMAIL: "admin@example.com" + ADMIN_PASSWORD: "change-this-to-a-strong-password" + POSTGRES_PASSWORD: "change-me" + # Optional + # GOOGLE_CLIENT_ID: "" + # GOOGLE_CLIENT_SECRET: "" +``` + +### PostgreSQL + +For production, consider a managed database (RDS, Cloud SQL, etc.) or an operator such as [CloudNativePG](https://cloudnative-pg.io/). The manifest below is a simple single-instance deployment for getting started: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: postgres + namespace: pop3-forwarder +spec: + replicas: 1 + selector: + matchLabels: + app: postgres + template: + metadata: + labels: + app: postgres + spec: + containers: + - name: postgres + image: postgres:15-alpine + ports: + - containerPort: 5432 + env: + - name: POSTGRES_USER + value: postgres + - name: POSTGRES_DB + value: pop3_forwarder + - name: POSTGRES_PASSWORD + valueFrom: + secretKeyRef: + name: pop3-forwarder-secrets + key: POSTGRES_PASSWORD + volumeMounts: + - name: pgdata + mountPath: /var/lib/postgresql/data + readinessProbe: + exec: + command: ["pg_isready", "-U", "postgres"] + initialDelaySeconds: 5 + periodSeconds: 10 + volumes: + - name: pgdata + persistentVolumeClaim: + claimName: postgres-pvc +--- +apiVersion: v1 +kind: Service +metadata: + name: postgres + namespace: pop3-forwarder +spec: + selector: + app: postgres + ports: + - port: 5432 + targetPort: 5432 +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: postgres-pvc + namespace: pop3-forwarder +spec: + accessModes: [ReadWriteOnce] + resources: + requests: + storage: 10Gi +``` + +### Redis + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: redis + namespace: pop3-forwarder +spec: + replicas: 1 + selector: + matchLabels: + app: redis + template: + metadata: + labels: + app: redis + spec: + containers: + - name: redis + image: redis:7-alpine + command: ["redis-server", "--appendonly", "yes"] + ports: + - containerPort: 6379 + readinessProbe: + exec: + command: ["redis-cli", "ping"] + initialDelaySeconds: 5 + periodSeconds: 10 +--- +apiVersion: v1 +kind: Service +metadata: + name: redis + namespace: pop3-forwarder +spec: + selector: + app: redis + ports: + - port: 6379 + targetPort: 6379 +``` + +### Backend API + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: backend + namespace: pop3-forwarder +spec: + replicas: 2 + selector: + matchLabels: + app: backend + template: + metadata: + labels: + app: backend + spec: + initContainers: + - name: run-migrations + image: ghcr.io/christianlouis/pop_puller_to_gmail-backend:latest + command: ["alembic", "upgrade", "head"] + envFrom: + - secretRef: + name: pop3-forwarder-secrets + env: + - name: DEBUG + value: "false" + containers: + - name: backend + image: ghcr.io/christianlouis/pop_puller_to_gmail-backend:latest + ports: + - containerPort: 8000 + envFrom: + - secretRef: + name: pop3-forwarder-secrets + env: + - name: HOST + value: "0.0.0.0" + - name: PORT + value: "8000" + - name: DEBUG + value: "false" + - name: LOG_LEVEL + value: "INFO" + - name: CORS_ORIGINS + value: "https://your-domain.com" + readinessProbe: + httpGet: + path: /health + port: 8000 + initialDelaySeconds: 10 + periodSeconds: 10 + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: "1" + memory: 512Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: backend + namespace: pop3-forwarder +spec: + selector: + app: backend + ports: + - port: 8000 + targetPort: 8000 +``` + +### Celery Worker + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: celery-worker + namespace: pop3-forwarder +spec: + replicas: 2 + selector: + matchLabels: + app: celery-worker + template: + metadata: + labels: + app: celery-worker + spec: + containers: + - name: worker + image: ghcr.io/christianlouis/pop_puller_to_gmail-backend:latest + command: + - celery + - -A + - app.workers.celery_app + - worker + - --loglevel=info + - --concurrency=2 + envFrom: + - secretRef: + name: pop3-forwarder-secrets + resources: + requests: + cpu: 250m + memory: 256Mi + limits: + cpu: "1" + memory: 512Mi +``` + +### Celery Beat (scheduler) + +Only one replica should run Celery Beat at a time: + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: celery-beat + namespace: pop3-forwarder +spec: + replicas: 1 # Must be exactly 1 + strategy: + type: Recreate # Avoid two schedulers running simultaneously + selector: + matchLabels: + app: celery-beat + template: + metadata: + labels: + app: celery-beat + spec: + containers: + - name: beat + image: ghcr.io/christianlouis/pop_puller_to_gmail-backend:latest + command: + - celery + - -A + - app.workers.celery_app + - beat + - --loglevel=info + envFrom: + - secretRef: + name: pop3-forwarder-secrets + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 250m + memory: 256Mi +``` + +### Frontend + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: frontend + namespace: pop3-forwarder +spec: + replicas: 2 + selector: + matchLabels: + app: frontend + template: + metadata: + labels: + app: frontend + spec: + containers: + - name: frontend + image: ghcr.io/christianlouis/pop_puller_to_gmail-frontend:latest + ports: + - containerPort: 3000 + env: + - name: NEXT_PUBLIC_API_URL + value: "https://api.your-domain.com" + resources: + requests: + cpu: 100m + memory: 128Mi + limits: + cpu: 500m + memory: 256Mi +--- +apiVersion: v1 +kind: Service +metadata: + name: frontend + namespace: pop3-forwarder +spec: + selector: + app: frontend + ports: + - port: 3000 + targetPort: 3000 +``` + +### Ingress + +The Ingress below assumes you have an Ingress controller installed (e.g., [ingress-nginx](https://kubernetes.github.io/ingress-nginx/)) and [cert-manager](https://cert-manager.io/) for automatic TLS certificates: + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: pop3-forwarder-ingress + namespace: pop3-forwarder + annotations: + cert-manager.io/cluster-issuer: letsencrypt-prod + nginx.ingress.kubernetes.io/proxy-body-size: "10m" +spec: + ingressClassName: nginx + tls: + - hosts: + - your-domain.com + - api.your-domain.com + secretName: pop3-forwarder-tls + rules: + - host: your-domain.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: frontend + port: + number: 3000 + - host: api.your-domain.com + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: backend + port: + number: 8000 +``` + +### Helm chart idea + +If you manage many environments (staging, production, etc.) consider wrapping the manifests above into a Helm chart: + +```text +helm/pop3-forwarder/ +├── Chart.yaml +├── values.yaml # defaults for all environments +├── values-staging.yaml +├── values-production.yaml +└── templates/ + ├── namespace.yaml + ├── secret.yaml + ├── postgres.yaml + ├── redis.yaml + ├── backend-deployment.yaml + ├── backend-service.yaml + ├── celery-worker.yaml + ├── celery-beat.yaml + ├── frontend-deployment.yaml + ├── frontend-service.yaml + └── ingress.yaml +``` + +Key values to parameterize in `values.yaml`: + +```yaml +replicaCount: + backend: 2 + celeryWorker: 2 + frontend: 2 + +image: + backend: ghcr.io/christianlouis/pop_puller_to_gmail-backend + frontend: ghcr.io/christianlouis/pop_puller_to_gmail-frontend + tag: latest + +ingress: + enabled: true + host: your-domain.com + apiHost: api.your-domain.com + tls: true + clusterIssuer: letsencrypt-prod + +resources: + backend: + requests: { cpu: 250m, memory: 256Mi } + limits: { cpu: "1", memory: 512Mi } + +postgres: + # Set to false when using an external/managed database + enabled: true + storage: 10Gi + +redis: + enabled: true +``` + +--- + +## Google OAuth and Gmail API Setup + +If you want Google sign-in or direct Gmail API email injection (instead of SMTP), follow these steps: + +### 1. Create a Google Cloud project + +1. Go to the [Google Cloud Console](https://console.cloud.google.com/). +2. Create a new project (or select an existing one). +3. Navigate to **APIs & Services → Library**. +4. Enable the **Gmail API**. + +### 2. Configure the OAuth consent screen + +1. Go to **APIs & Services → OAuth consent screen**. +2. Choose **External** (or **Internal** if you have a Google Workspace org). +3. Fill in the required fields (app name, user-support email, developer contact). +4. Under **Scopes**, add: + - `openid` + - `email` + - `profile` + - `https://www.googleapis.com/auth/gmail.insert` *(for Gmail API injection)* + - `https://www.googleapis.com/auth/gmail.labels` + +### 3. Create OAuth 2.0 credentials + +1. Go to **APIs & Services → Credentials**. +2. Click **Create Credentials → OAuth client ID**. +3. Application type: **Web application**. +4. Add **Authorized redirect URIs**: + - Development: `http://localhost:3000/auth/callback/google` + - Production: `https://your-domain.com/auth/callback/google` +5. Copy the **Client ID** and **Client Secret**. + +### 4. Configure the application + +Add these to `backend/.env`: + +```ini +GOOGLE_CLIENT_ID=your-client-id.apps.googleusercontent.com +GOOGLE_CLIENT_SECRET=your-client-secret +GOOGLE_REDIRECT_URI=https://your-domain.com/auth/callback/google +GMAIL_API_ENABLED=true +``` + +--- + +## Reverse Proxy and TLS + +In production you should place a reverse proxy in front of the backend and frontend to handle TLS termination. + +### Example: nginx + +```nginx +# /etc/nginx/sites-available/pop3-forwarder + +# Frontend +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem; + + location / { + proxy_pass http://127.0.0.1:3000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} + +# Backend API +server { + listen 443 ssl http2; + server_name api.your-domain.com; + + ssl_certificate /etc/letsencrypt/live/api.your-domain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/api.your-domain.com/privkey.pem; + + location / { + proxy_pass http://127.0.0.1:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +Generate certificates with Let's Encrypt: + +```bash +sudo apt-get install certbot python3-certbot-nginx +sudo certbot --nginx -d your-domain.com -d api.your-domain.com +``` + +### Example: Traefik (Docker Compose add-on) + +If you prefer Traefik, add it as a service in your Compose file and use labels on the `backend` and `frontend` services. Traefik handles TLS via Let's Encrypt automatically. + +--- + +## Environment Variable Reference + +### Legacy mode (`.env`) + +| Variable | Required | Default | Description | +|---|---|---|---| +| `POP3_ACCOUNT_N_HOST` | Yes | — | POP3 server hostname (N = 1, 2, 3…) | +| `POP3_ACCOUNT_N_PORT` | No | `995` | POP3 server port | +| `POP3_ACCOUNT_N_USER` | Yes | — | POP3 username | +| `POP3_ACCOUNT_N_PASSWORD` | Yes | — | POP3 password | +| `POP3_ACCOUNT_N_USE_SSL` | No | `true` | Use SSL for POP3 | +| `SMTP_HOST` | Yes | — | SMTP server (e.g., `smtp.gmail.com`) | +| `SMTP_PORT` | Yes | — | SMTP port (e.g., `587`) | +| `SMTP_USER` | Yes | — | SMTP username | +| `SMTP_PASSWORD` | Yes | — | SMTP password / App Password | +| `SMTP_USE_TLS` | No | `true` | Use TLS for SMTP | +| `GMAIL_DESTINATION` | Yes | — | Destination Gmail address | +| `CHECK_INTERVAL_MINUTES` | No | `5` | Minutes between polling cycles | +| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails forwarded per cycle | +| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit | +| `LOG_LEVEL` | No | `INFO` | Logging level | +| `POSTMARK_API_TOKEN` | No | — | Postmark token for error alerts | +| `POSTMARK_FROM_EMAIL` | No | — | Sender for error alerts | +| `POSTMARK_TO_EMAIL` | No | — | Recipient for error alerts | + +### SaaS mode (`backend/.env`) + +| Variable | Required | Default | Description | +|---|---|---|---| +| `DATABASE_URL` | Yes | — | PostgreSQL connection string | +| `SECRET_KEY` | Yes | — | JWT signing key (≥ 32 chars) | +| `ENCRYPTION_KEY` | Yes | — | Credential encryption key (≥ 32 chars) | +| `REDIS_URL` | Yes | `redis://localhost:6379/0` | Redis connection string | +| `CELERY_BROKER_URL` | Yes | `redis://localhost:6379/0` | Celery broker URL | +| `CELERY_RESULT_BACKEND` | Yes | `redis://localhost:6379/0` | Celery result backend URL | +| `CORS_ORIGINS` | Yes | `http://localhost:3000` | Comma-separated allowed origins | +| `ADMIN_EMAIL` | No | — | Auto-created admin email | +| `ADMIN_PASSWORD` | No | — | Auto-created admin password | +| `GOOGLE_CLIENT_ID` | No | — | Google OAuth client ID | +| `GOOGLE_CLIENT_SECRET` | No | — | Google OAuth client secret | +| `GOOGLE_REDIRECT_URI` | No | `http://localhost:3000/auth/callback/google` | OAuth redirect URI | +| `GMAIL_API_ENABLED` | No | `true` | Enable Gmail API injection | +| `DEBUG` | No | `false` | Enable debug mode | +| `LOG_LEVEL` | No | `INFO` | Logging level | +| `HOST` | No | `0.0.0.0` | Bind address | +| `PORT` | No | `8000` | Bind port | +| `STRIPE_API_KEY` | No | — | Stripe API key | +| `STRIPE_WEBHOOK_SECRET` | No | — | Stripe webhook secret | +| `MAX_EMAILS_PER_RUN` | No | `50` | Max emails per account per cycle | +| `CHECK_INTERVAL_MINUTES` | No | `5` | Minutes between polling cycles | +| `THROTTLE_EMAILS_PER_MINUTE` | No | `10` | Rate limit | + +--- + +## Upgrading + +```bash +cd pop_puller_to_gmail + +# Pull latest code +git pull origin main + +# Rebuild and restart +docker compose -f docker-compose.prod.yml build +docker compose -f docker-compose.prod.yml up -d + +# The backend init container / startup command runs migrations automatically. +# To run them manually: +docker compose -f docker-compose.prod.yml exec backend alembic upgrade head +``` + +--- + +## Troubleshooting + +### Container won't start + +```bash +# Check logs for the failing service +docker compose -f docker-compose.prod.yml logs backend + +# Common causes: +# - DATABASE_URL is wrong or PostgreSQL isn't ready yet +# - SECRET_KEY or ENCRYPTION_KEY is shorter than 32 characters +# - Port conflict on the host +``` + +### Frontend can't reach the backend (CORS errors) + +1. Ensure `CORS_ORIGINS` in `backend/.env` includes the frontend URL exactly (protocol + host + port). +2. Verify the backend is reachable from the frontend container: `docker compose exec frontend wget -qO- http://backend:8000/health`. + +### Emails aren't being forwarded + +1. Check Celery worker logs: `docker compose -f docker-compose.prod.yml logs celery-worker`. +2. Verify Redis is running: `docker compose -f docker-compose.prod.yml exec redis redis-cli ping`. +3. Confirm POP3 credentials are correct by testing manually. + +### Database migration errors + +```bash +# Check current migration state +docker compose -f docker-compose.prod.yml exec backend alembic current + +# Show migration history +docker compose -f docker-compose.prod.yml exec backend alembic history + +# If stuck, you may need to stamp the current head +docker compose -f docker-compose.prod.yml exec backend alembic stamp head +``` + +### OAuth redirect mismatch + +The `GOOGLE_REDIRECT_URI` in `backend/.env` must **exactly** match one of the authorized redirect URIs configured in the Google Cloud Console (including protocol, host, port, and path). + +--- + +## Further Reading + +- [QUICKSTART.md](QUICKSTART.md) — Get running in under 10 minutes (legacy mode) +- [DEPLOYMENT_CHECKLIST.md](DEPLOYMENT_CHECKLIST.md) — Pre-deployment and post-deployment checklists +- [ARCHITECTURE.md](ARCHITECTURE.md) — System architecture and component overview +- [MIGRATION_GUIDE.md](MIGRATION_GUIDE.md) — Upgrading from v1 (legacy) to v2 (SaaS) +- [SECURITY_SUMMARY.md](SECURITY_SUMMARY.md) — Security best practices From d66af4d8eff1db04fb08f5026ecb658010453ede Mon Sep 17 00:00:00 2001 From: "copilot-swe-agent[bot]" <198982749+Copilot@users.noreply.github.com> Date: Mon, 23 Mar 2026 11:23:37 +0000 Subject: [PATCH 7/7] feat: Milestone 1 Security & Infrastructure improvements - Add Dependabot configuration for pip, npm, GitHub Actions, Docker - Update copilot instructions to require TODO.md and CHANGELOG.md updates - Add unit tests for middleware, security, app factory, and schemas (125 total tests, 57% coverage) - Update TODO.md with current progress - Update CHANGELOG.md with all changes Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/pop_puller_to_gmail/sessions/2f15a52f-6812-4586-9d5c-a144226c842b --- .github/copilot-instructions.md | 7 + .github/dependabot.yml | 53 ++++ CHANGELOG.md | 12 +- backend/tests/unit/test_app.py | 108 ++++++++ backend/tests/unit/test_middleware.py | 127 +++++++++ backend/tests/unit/test_schemas.py | 277 +++++++++++++++++++ backend/tests/unit/test_security_extended.py | 188 +++++++++++++ docs/TODO.md | 18 +- 8 files changed, 781 insertions(+), 9 deletions(-) create mode 100644 .github/dependabot.yml create mode 100644 backend/tests/unit/test_app.py create mode 100644 backend/tests/unit/test_middleware.py create mode 100644 backend/tests/unit/test_schemas.py create mode 100644 backend/tests/unit/test_security_extended.py diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 8a420cc..096eb47 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -24,6 +24,13 @@ cd frontend npm run lint ``` +## Documentation Requirements + +When making changes, always update the following files: + +- **`docs/TODO.md`**: Update task checkboxes, progress percentages, and status indicators to reflect completed work and any new items discovered. +- **`CHANGELOG.md`**: Add entries under the `[Unreleased]` section using the appropriate category (`Added`, `Changed`, `Fixed`, `Security`, `Removed`, `Deprecated`). + ## Conventions - **Python**: Follow PEP 8. Use `black` for formatting. All ruff and mypy errors must be resolved before committing. diff --git a/.github/dependabot.yml b/.github/dependabot.yml new file mode 100644 index 0000000..503ce52 --- /dev/null +++ b/.github/dependabot.yml @@ -0,0 +1,53 @@ +version: 2 +updates: + # Python dependencies (backend) + - package-ecosystem: "pip" + directory: "/backend" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 10 + labels: + - "dependencies" + - "python" + commit-message: + prefix: "chore(deps):" + + # GitHub Actions + - package-ecosystem: "github-actions" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + labels: + - "dependencies" + - "github-actions" + commit-message: + prefix: "chore(ci):" + + # npm dependencies (frontend) + - package-ecosystem: "npm" + directory: "/frontend" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 10 + labels: + - "dependencies" + - "javascript" + commit-message: + prefix: "chore(deps):" + + # Docker dependencies + - package-ecosystem: "docker" + directory: "/" + schedule: + interval: "weekly" + day: "monday" + open-pull-requests-limit: 5 + labels: + - "dependencies" + - "docker" + commit-message: + prefix: "chore(docker):" diff --git a/CHANGELOG.md b/CHANGELOG.md index ed92480..ae11c87 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Rate limiting per user/tier - Comprehensive test infrastructure setup - CI/CD pipeline for testing and security scanning +- Dependabot configuration for automated dependency updates (pip, npm, GitHub Actions, Docker) +- Copilot instructions requiring TODO.md and CHANGELOG.md updates +- Unit tests for security middleware (SecurityHeadersMiddleware, CSRFProtectionMiddleware) +- Unit tests for JWT token lifecycle (access tokens, refresh tokens, decode, edge cases) +- Unit tests for credential encryption edge cases (empty, long, unicode, special chars) +- Unit tests for FastAPI application factory and core endpoints (root, health, OpenAPI) +- Unit tests for Pydantic schema validation (users, mail accounts, notifications, subscriptions) +- Reached 57% test coverage (up from 54%) ### Changed - Reorganized documentation into `docs/` directory @@ -113,5 +121,5 @@ Use these standard categories: --- -**Maintained by**: Development Team -**Last Updated**: 2026-02-06 +**Maintained by**: Development Team +**Last Updated**: 2026-03-23 diff --git a/backend/tests/unit/test_app.py b/backend/tests/unit/test_app.py new file mode 100644 index 0000000..f868886 --- /dev/null +++ b/backend/tests/unit/test_app.py @@ -0,0 +1,108 @@ +""" +Unit tests for the FastAPI application factory and core endpoints. +""" + +import pytest +from httpx import AsyncClient, ASGITransport +from app.main import create_application + + +@pytest.fixture +def app(): + """Create a fresh application instance for testing.""" + return create_application() + + +@pytest.mark.asyncio +class TestRootEndpoint: + """Test root endpoint""" + + async def test_root_returns_200(self, app): + """Test that root endpoint returns 200""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/") + + assert response.status_code == 200 + + async def test_root_returns_api_info(self, app): + """Test that root returns API information""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/") + + data = response.json() + assert "message" in data + assert "version" in data + assert "docs" in data + assert data["docs"] == "/api/docs" + + +@pytest.mark.asyncio +class TestHealthEndpoint: + """Test health check endpoint""" + + async def test_health_returns_200(self, app): + """Test that health endpoint returns 200""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/health") + + assert response.status_code == 200 + + async def test_health_returns_healthy(self, app): + """Test that health endpoint returns healthy status""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/health") + + data = response.json() + assert data["status"] == "healthy" + + +@pytest.mark.asyncio +class TestSecurityHeaders: + """Test that security headers are present in responses""" + + async def test_security_headers_on_root(self, app): + """Test security headers on root endpoint""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/") + + assert response.headers["X-Frame-Options"] == "DENY" + assert response.headers["X-Content-Type-Options"] == "nosniff" + assert response.headers["X-XSS-Protection"] == "1; mode=block" + + async def test_security_headers_on_health(self, app): + """Test security headers on health endpoint""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/health") + + assert response.headers["X-Frame-Options"] == "DENY" + + +@pytest.mark.asyncio +class TestApplicationFactory: + """Test the create_application factory""" + + async def test_app_title(self, app): + """Test that app has correct title""" + assert app.title == "POP3 Forwarder SaaS" + + async def test_app_version(self, app): + """Test that app has a version""" + assert app.version is not None + assert len(app.version) > 0 + + async def test_openapi_endpoint(self, app): + """Test that OpenAPI schema is available""" + transport = ASGITransport(app=app) + async with AsyncClient(transport=transport, base_url="http://test") as client: + response = await client.get("/api/openapi.json") + + assert response.status_code == 200 + schema = response.json() + assert "openapi" in schema + assert "info" in schema diff --git a/backend/tests/unit/test_middleware.py b/backend/tests/unit/test_middleware.py new file mode 100644 index 0000000..a6ac2a4 --- /dev/null +++ b/backend/tests/unit/test_middleware.py @@ -0,0 +1,127 @@ +""" +Unit tests for security middleware. +""" + +from starlette.testclient import TestClient +from starlette.applications import Starlette +from starlette.responses import PlainTextResponse +from starlette.routing import Route + +from app.core.middleware import SecurityHeadersMiddleware, CSRFProtectionMiddleware + + +def _make_app(middleware_classes): + """Helper to build a Starlette app with given middleware.""" + + async def homepage(request): + return PlainTextResponse("OK") + + app = Starlette( + routes=[ + Route("/", homepage, methods=["GET", "HEAD", "POST", "OPTIONS"]), + Route("/api/v1/auth/login", homepage, methods=["GET", "POST"]), + ] + ) + for cls in middleware_classes: + app.add_middleware(cls) + return app + + +class TestSecurityHeadersMiddleware: + """Test security headers added to all responses""" + + def setup_method(self): + app = _make_app([SecurityHeadersMiddleware]) + self.client = TestClient(app) + + def test_x_frame_options_header(self): + """Test X-Frame-Options is set to DENY""" + response = self.client.get("/") + assert response.headers["X-Frame-Options"] == "DENY" + + def test_x_content_type_options_header(self): + """Test X-Content-Type-Options is set to nosniff""" + response = self.client.get("/") + assert response.headers["X-Content-Type-Options"] == "nosniff" + + def test_x_xss_protection_header(self): + """Test X-XSS-Protection header is set""" + response = self.client.get("/") + assert response.headers["X-XSS-Protection"] == "1; mode=block" + + def test_content_security_policy_header(self): + """Test Content-Security-Policy header is present""" + response = self.client.get("/") + csp = response.headers["Content-Security-Policy"] + assert "default-src 'self'" in csp + assert "script-src" in csp + + def test_referrer_policy_header(self): + """Test Referrer-Policy header""" + response = self.client.get("/") + assert response.headers["Referrer-Policy"] == "strict-origin-when-cross-origin" + + def test_permissions_policy_header(self): + """Test Permissions-Policy header""" + response = self.client.get("/") + policy = response.headers["Permissions-Policy"] + assert "geolocation=()" in policy + assert "microphone=()" in policy + assert "camera=()" in policy + + def test_no_hsts_for_localhost(self): + """Test that HSTS header check depends on hostname""" + # The HSTS header is only skipped when hostname is localhost or 127.0.0.1. + # TestClient uses 'testserver' as hostname, which is not in the skip list, + # so HSTS will be set. Verify the logic works with a direct check. + response = self.client.get("/") + # TestClient hostname is 'testserver', not localhost, so HSTS IS set + assert "Strict-Transport-Security" in response.headers + + +class TestCSRFProtectionMiddleware: + """Test CSRF protection middleware""" + + def setup_method(self): + app = _make_app([CSRFProtectionMiddleware]) + self.client = TestClient(app) + + def test_get_requests_pass_through(self): + """Test that GET requests are not blocked""" + response = self.client.get("/") + assert response.status_code == 200 + + def test_head_requests_pass_through(self): + """Test that HEAD requests are not blocked""" + response = self.client.head("/") + assert response.status_code == 200 + + def test_options_requests_pass_through(self): + """Test that OPTIONS requests are not blocked""" + response = self.client.options("/") + assert response.status_code == 200 + + def test_exempt_paths_pass_through(self): + """Test that exempt paths are not CSRF-checked for POST""" + response = self.client.post("/api/v1/auth/login") + assert response.status_code == 200 + + def test_post_to_non_exempt_path_passes(self): + """Test that POST to non-exempt path also passes (JWT provides CSRF protection)""" + response = self.client.post("/") + assert response.status_code == 200 + + def test_generate_csrf_token(self): + """Test CSRF token generation produces valid token""" + token = CSRFProtectionMiddleware._generate_csrf_token() + assert isinstance(token, str) + assert len(token) == 43 # token_urlsafe(32) produces 43 chars + + def test_validate_csrf_token_valid(self): + """Test CSRF token validation with valid token""" + token = CSRFProtectionMiddleware._generate_csrf_token() + assert CSRFProtectionMiddleware._validate_csrf_token(token) is True + + def test_validate_csrf_token_invalid(self): + """Test CSRF token validation with invalid token""" + assert CSRFProtectionMiddleware._validate_csrf_token("short") is False diff --git a/backend/tests/unit/test_schemas.py b/backend/tests/unit/test_schemas.py new file mode 100644 index 0000000..d937c4a --- /dev/null +++ b/backend/tests/unit/test_schemas.py @@ -0,0 +1,277 @@ +""" +Unit tests for Pydantic schema validation. +""" + +import pytest +from pydantic import ValidationError +from app.models.schemas import ( + UserCreate, + UserUpdate, + MailAccountCreate, + MailAccountUpdate, + MailAccountTestRequest, + MailAccountAutoDetectRequest, + MailProtocol, + DeliveryMethod, + SubscriptionTier, + AccountStatus, + NotificationChannel, + Token, + TokenPayload, + GoogleAuthRequest, + NotificationConfigCreate, + SubscriptionCheckoutRequest, + ProviderPreset, +) + + +class TestEnums: + """Test enum values""" + + def test_subscription_tiers(self): + """Test all subscription tier values""" + assert SubscriptionTier.FREE == "free" + assert SubscriptionTier.BASIC == "basic" + assert SubscriptionTier.PRO == "pro" + assert SubscriptionTier.ENTERPRISE == "enterprise" + + def test_mail_protocols(self): + """Test all mail protocol values""" + assert MailProtocol.POP3 == "pop3" + assert MailProtocol.POP3_SSL == "pop3_ssl" + assert MailProtocol.IMAP == "imap" + assert MailProtocol.IMAP_SSL == "imap_ssl" + + def test_account_status(self): + """Test all account status values""" + assert AccountStatus.ACTIVE == "active" + assert AccountStatus.INACTIVE == "inactive" + assert AccountStatus.ERROR == "error" + assert AccountStatus.TESTING == "testing" + + def test_delivery_method(self): + """Test all delivery method values""" + assert DeliveryMethod.SMTP == "smtp" + assert DeliveryMethod.GMAIL_API == "gmail_api" + + def test_notification_channels(self): + """Test all notification channel values""" + assert NotificationChannel.EMAIL == "email" + assert NotificationChannel.TELEGRAM == "telegram" + assert NotificationChannel.WEBHOOK == "webhook" + assert NotificationChannel.SLACK == "slack" + assert NotificationChannel.DISCORD == "discord" + + +class TestUserSchemas: + """Test user-related schemas""" + + def test_user_create_with_email(self): + """Test UserCreate with valid email""" + user = UserCreate(email="test@example.com", password="password123") + assert user.email == "test@example.com" + assert user.password == "password123" + + def test_user_create_without_password(self): + """Test UserCreate without password (OAuth users)""" + user = UserCreate(email="test@example.com") + assert user.password is None + + def test_user_create_with_full_name(self): + """Test UserCreate with full name""" + user = UserCreate( + email="test@example.com", full_name="Test User", password="pass" + ) + assert user.full_name == "Test User" + + def test_user_create_invalid_email(self): + """Test UserCreate rejects invalid email""" + with pytest.raises(ValidationError): + UserCreate(email="not-an-email", password="pass") + + def test_user_update_partial(self): + """Test UserUpdate with partial data""" + update = UserUpdate(full_name="New Name") + assert update.full_name == "New Name" + assert update.email is None + + +class TestTokenSchemas: + """Test token schemas""" + + def test_token_schema(self): + """Test Token schema""" + token = Token( + access_token="abc123", refresh_token="def456", token_type="bearer" + ) + assert token.access_token == "abc123" + assert token.token_type == "bearer" + + def test_token_payload_schema(self): + """Test TokenPayload schema""" + payload = TokenPayload(sub=42, type="access") + assert payload.sub == 42 + assert payload.type == "access" + + def test_google_auth_request(self): + """Test GoogleAuthRequest schema""" + req = GoogleAuthRequest( + code="auth-code-123", redirect_uri="http://localhost:3000/callback" + ) + assert req.code == "auth-code-123" + + +class TestMailAccountSchemas: + """Test mail account schemas""" + + def test_mail_account_create_valid(self): + """Test creating a valid mail account""" + account = MailAccountCreate( + name="Test Account", + email_address="user@example.com", + host="imap.example.com", + port=993, + username="user@example.com", + password="secret", + forward_to="me@gmail.com", + ) + assert account.name == "Test Account" + assert account.protocol == MailProtocol.POP3_SSL # default + assert account.use_ssl is True + + def test_mail_account_create_invalid_port(self): + """Test that invalid port is rejected""" + with pytest.raises(ValidationError): + MailAccountCreate( + name="Test", + email_address="user@example.com", + host="imap.example.com", + port=0, # invalid + username="user@example.com", + password="secret", + forward_to="me@gmail.com", + ) + + def test_mail_account_create_port_too_high(self): + """Test that port above 65535 is rejected""" + with pytest.raises(ValidationError): + MailAccountCreate( + name="Test", + email_address="user@example.com", + host="imap.example.com", + port=70000, # invalid + username="user@example.com", + password="secret", + forward_to="me@gmail.com", + ) + + def test_mail_account_update_partial(self): + """Test partial mail account update""" + update = MailAccountUpdate(is_enabled=False) + assert update.is_enabled is False + assert update.name is None + assert update.password is None + + def test_mail_account_test_request(self): + """Test mail account test connection schema""" + req = MailAccountTestRequest( + host="imap.gmail.com", + port=993, + protocol=MailProtocol.IMAP_SSL, + username="user@gmail.com", + password="app-password", + ) + assert req.host == "imap.gmail.com" + + def test_auto_detect_request(self): + """Test auto-detect request schema""" + req = MailAccountAutoDetectRequest(email_address="user@gmail.com") + assert req.email_address == "user@gmail.com" + + def test_auto_detect_invalid_email(self): + """Test auto-detect rejects invalid email""" + with pytest.raises(ValidationError): + MailAccountAutoDetectRequest(email_address="not-email") + + +class TestNotificationSchemas: + """Test notification schemas""" + + def test_notification_config_create(self): + """Test creating notification config""" + config = NotificationConfigCreate( + channel=NotificationChannel.TELEGRAM, + config={"bot_token": "123:abc", "chat_id": "456"}, + ) + assert config.channel == NotificationChannel.TELEGRAM + assert config.notify_on_errors is True # default + assert config.notify_on_success is False # default + + def test_notification_config_threshold_validation(self): + """Test notification threshold validation""" + with pytest.raises(ValidationError): + NotificationConfigCreate( + channel=NotificationChannel.EMAIL, + config={}, + notify_threshold=0, # must be > 0 + ) + + +class TestSubscriptionSchemas: + """Test subscription schemas""" + + def test_subscription_checkout_request_monthly(self): + """Test subscription checkout with monthly billing""" + req = SubscriptionCheckoutRequest( + tier=SubscriptionTier.PRO, + billing_period="monthly", + success_url="https://example.com/success", + cancel_url="https://example.com/cancel", + ) + assert req.tier == SubscriptionTier.PRO + assert req.billing_period == "monthly" + + def test_subscription_checkout_request_yearly(self): + """Test subscription checkout with yearly billing""" + req = SubscriptionCheckoutRequest( + tier=SubscriptionTier.BASIC, + billing_period="yearly", + success_url="https://example.com/success", + cancel_url="https://example.com/cancel", + ) + assert req.billing_period == "yearly" + + def test_subscription_checkout_invalid_period(self): + """Test that invalid billing period is rejected""" + with pytest.raises(ValidationError): + SubscriptionCheckoutRequest( + tier=SubscriptionTier.BASIC, + billing_period="quarterly", # invalid + success_url="https://example.com/success", + cancel_url="https://example.com/cancel", + ) + + +class TestProviderPresetSchema: + """Test provider preset schema""" + + def test_provider_preset_with_imap(self): + """Test provider preset with IMAP config""" + preset = ProviderPreset( + id="gmail", + name="Gmail", + domains=["gmail.com", "googlemail.com"], + imap_ssl={"host": "imap.gmail.com", "port": 993}, + ) + assert preset.id == "gmail" + assert "gmail.com" in preset.domains + + def test_provider_preset_without_pop3(self): + """Test provider preset without POP3 (IMAP only)""" + preset = ProviderPreset( + id="posteo", + name="Posteo", + domains=["posteo.de"], + imap_ssl={"host": "posteo.de", "port": 993}, + ) + assert preset.pop3_ssl is None diff --git a/backend/tests/unit/test_security_extended.py b/backend/tests/unit/test_security_extended.py new file mode 100644 index 0000000..5c3d7e8 --- /dev/null +++ b/backend/tests/unit/test_security_extended.py @@ -0,0 +1,188 @@ +""" +Unit tests for extended security module functionality. +""" + +from datetime import timedelta +from app.core.security import ( + create_access_token, + create_refresh_token, + decode_token, + generate_random_token, + encrypt_credential, + decrypt_credential, + CredentialEncryption, +) + + +class TestAccessToken: + """Test JWT access token creation and decoding""" + + def test_create_access_token_with_custom_expiry(self): + """Test creating an access token with custom expiry""" + data = {"sub": "test@example.com"} + token = create_access_token(data, expires_delta=timedelta(hours=1)) + + assert isinstance(token, str) + assert token.count(".") == 2 + + def test_decode_valid_access_token(self): + """Test decoding a valid access token""" + data = {"sub": "user123"} + token = create_access_token(data) + + payload = decode_token(token) + assert payload is not None + assert payload["sub"] == "user123" + assert payload["type"] == "access" + + def test_decode_invalid_token_returns_none(self): + """Test that decoding an invalid token returns None""" + result = decode_token("invalid.token.string") + assert result is None + + def test_decode_empty_token_returns_none(self): + """Test that decoding an empty string returns None""" + result = decode_token("") + assert result is None + + def test_access_token_contains_type(self): + """Test that access token payload contains type 'access'""" + data = {"sub": "user@example.com"} + token = create_access_token(data) + payload = decode_token(token) + + assert payload is not None + assert payload["type"] == "access" + + def test_access_token_contains_expiry(self): + """Test that access token payload contains expiry""" + data = {"sub": "user@example.com"} + token = create_access_token(data) + payload = decode_token(token) + + assert payload is not None + assert "exp" in payload + + +class TestRefreshToken: + """Test JWT refresh token creation and decoding""" + + def test_create_refresh_token(self): + """Test refresh token creation""" + data = {"sub": "test@example.com"} + token = create_refresh_token(data) + + assert isinstance(token, str) + assert token.count(".") == 2 + + def test_decode_refresh_token(self): + """Test decoding a valid refresh token""" + data = {"sub": "user456"} + token = create_refresh_token(data) + + payload = decode_token(token) + assert payload is not None + assert payload["sub"] == "user456" + assert payload["type"] == "refresh" + + def test_refresh_token_different_from_access(self): + """Test that refresh and access tokens are different""" + data = {"sub": "test@example.com"} + access = create_access_token(data) + refresh = create_refresh_token(data) + + assert access != refresh + + +class TestRandomToken: + """Test random token generation""" + + def test_generate_random_token_default_length(self): + """Test generating a random token with default length""" + token = generate_random_token() + assert isinstance(token, str) + assert len(token) > 0 + + def test_generate_random_token_custom_length(self): + """Test generating a random token with custom length""" + token = generate_random_token(64) + assert isinstance(token, str) + assert len(token) > 0 + + def test_random_tokens_are_unique(self): + """Test that generated tokens are unique""" + tokens = {generate_random_token() for _ in range(10)} + assert len(tokens) == 10 + + +class TestGlobalEncryptionFunctions: + """Test global encryption convenience functions""" + + def test_encrypt_credential_returns_string(self): + """Test that encrypt_credential returns a non-empty string""" + encrypted = encrypt_credential("my-password") + assert isinstance(encrypted, str) + assert len(encrypted) > 0 + + def test_decrypt_credential_roundtrip(self): + """Test encrypt/decrypt roundtrip with global functions""" + original = "super-secret-password-123" + encrypted = encrypt_credential(original) + decrypted = decrypt_credential(encrypted) + assert decrypted == original + + def test_encrypt_credential_is_not_plaintext(self): + """Test that encrypted credential differs from plaintext""" + password = "my-password" + encrypted = encrypt_credential(password) + assert encrypted != password + + +class TestCredentialEncryptionEdgeCases: + """Test edge cases in credential encryption""" + + def test_encrypt_empty_string(self): + """Test encrypting an empty string""" + encryptor = CredentialEncryption(user_id=1) + encrypted = encryptor.encrypt("") + decrypted = encryptor.decrypt(encrypted) + assert decrypted == "" + + def test_encrypt_long_string(self): + """Test encrypting a very long string""" + long_password = "a" * 10000 + encryptor = CredentialEncryption(user_id=1) + encrypted = encryptor.encrypt(long_password) + decrypted = encryptor.decrypt(encrypted) + assert decrypted == long_password + + def test_encrypt_special_characters(self): + """Test encrypting a string with special characters""" + special = "p@$$w0rd!#%^&*()_+-=[]{}|;':\",./<>?" + encryptor = CredentialEncryption(user_id=1) + encrypted = encryptor.encrypt(special) + decrypted = encryptor.decrypt(encrypted) + assert decrypted == special + + def test_encrypt_unicode(self): + """Test encrypting unicode characters""" + unicode_str = "密码テスト🔒" + encryptor = CredentialEncryption(user_id=1) + encrypted = encryptor.encrypt(unicode_str) + decrypted = encryptor.decrypt(encrypted) + assert decrypted == unicode_str + + def test_custom_key(self): + """Test encryption with a custom key""" + key = "custom-encryption-key-that-is-at-least-32-chars-long" + encryptor = CredentialEncryption(key=key, user_id=1) + encrypted = encryptor.encrypt("test-data") + decrypted = encryptor.decrypt(encrypted) + assert decrypted == "test-data" + + def test_system_salt_without_user_id(self): + """Test encryption with system salt (no user_id)""" + encryptor = CredentialEncryption() + encrypted = encryptor.encrypt("system-data") + decrypted = encryptor.decrypt(encrypted) + assert decrypted == "system-data" diff --git a/docs/TODO.md b/docs/TODO.md index 4ed44ef..85b07bc 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -65,6 +65,11 @@ Comprehensive task breakdown for repository improvements and production readines - [x] Add `backend/pytest.ini` configuration - [x] Create sample unit tests (test_security.py, test_config.py) - [x] Add user and mail account factory fixtures +- [x] Write unit tests for security module (100% coverage) +- [x] Write unit tests for middleware (98% coverage) +- [x] Write unit tests for schemas and validation +- [x] Write unit tests for application factory and core endpoints +- [x] Reach 50%+ test coverage (currently 57%) ### In Progress 🔨 - [ ] Write unit tests for authentication (target 80%+ coverage) @@ -88,6 +93,7 @@ Comprehensive task breakdown for repository improvements and production readines - [x] Create `.github/workflows/lint.yml` for code quality checks - [x] Create `.github/workflows/security.yml` for security scanning - [x] Existing `.github/workflows/docker-build.yml` for Docker images +- [x] Set up automatic dependency updates (Dependabot) ### In Progress 🔨 - [ ] Configure branch protection rules @@ -95,7 +101,6 @@ Comprehensive task breakdown for repository improvements and production readines ### Not Started 📋 - [ ] Add deployment workflow (staging/production) -- [ ] Set up automatic dependency updates (Dependabot) - [ ] Add release workflow with automated changelog - [ ] Configure status checks for PRs - [ ] Add performance regression detection @@ -287,16 +292,16 @@ because the API client layer is missing. | Category | Progress | Status | |----------|----------|--------| | Security | 60% | 🟡 In Progress | -| Agentic Infrastructure | 90% | 🟢 Near Complete | -| Testing | 30% | 🔴 Needs Work | -| CI/CD | 70% | 🟡 In Progress | +| Agentic Infrastructure | 95% | 🟢 Near Complete | +| Testing | 57% | 🟡 In Progress | +| CI/CD | 80% | 🟢 Near Complete | | Code Quality | 40% | 🔴 Needs Work | | Production Ready | 20% | 🔴 Needs Work | | Observability | 10% | 🔴 Needs Work | | Backend Features | 80% | 🟢 Near Complete | | Frontend | 30% | 🔴 Blocked (missing lib/api.ts) | -**Overall Repository Readiness**: 48% ⚠️ +**Overall Repository Readiness**: 52% ⚠️ --- @@ -305,12 +310,11 @@ because the API client layer is missing. 1. **Immediate** (Today): - [ ] Create `frontend/src/lib/api.ts` (frontend is broken without it) - [ ] Fix remaining security issues (bare excepts, datetime, redirect_uri) - - [ ] Write 10 more unit tests 2. **This Week**: - [ ] Enable rate limiting - [ ] Add audit logging - - [ ] Reach 50% test coverage + - [ ] Write more unit tests (target 70% coverage) - [ ] Complete ADR documentation - [ ] End-to-end test frontend against backend