gh-christianlouis-inboxconverge/docs/TODO.md

TODO & Milestones

Comprehensive task breakdown for repository improvements and production readiness.

✅ Recently Completed

• Improved test coverage for mail_processor.py: Added 46 new unit tests covering POP3 connection testing, POP3 email fetching, IMAP edge cases, email forwarding (STARTTLS/SSL), and fetch_emails/test_connection routing. Coverage increased from ~42% to 98%.

• Log noise reduction: Suppressed ignored untagged response INFO messages from aioimaplib in Celery workers (set logger to WARNING). Eliminated repeated file_cache is only supported with oauth2client<4.0.0 warnings from the Gmail API client by passing cache_discovery=False to googleapiclient.discovery.build().

• ESLint fix: Converted frontend/jest.config.js to jest.config.mjs (ES module syntax) to resolve @typescript-eslint/no-require-imports lint error.

• Codecov integration: Added Codecov coverage reporting with CODECOV_TOKEN authentication. Set up Jest for frontend tests with lcov coverage, updated CI to collect and upload both backend (XML via pytest-cov) and frontend (lcov via Jest) coverage reports to Codecov with separate backend and frontend flags.

• IMAP: fix all emails appearing empty — aioimaplib stores RFC822 literal data as bytearray, not bytes. The extraction loop was checking isinstance(line, bytes) which returns False for bytearray, so every email body was silently skipped. Fixed to accept both types and convert to bytes. Affected T-Online, GMX, and all IMAP accounts.

• CI: fix safety scan EOF error — replaced safety scan --json (Safety CLI v3 requires interactive login) with pip-audit (no auth required, maintained by PyPA).

• Security: upgrade fastapi/starlette and fix safety CI command — Upgraded fastapi to 0.135.2 (pulls in starlette>=1.0.0) fixing 4 DoS CVEs in starlette<=0.35.1; replaced deprecated safety check with safety scan; added .safety-policy.yml to suppress unfixable ecdsa side-channel CVEs (maintainers won't fix).

• IMAP RFC 3501 flag syntax & aioimaplib UID SEARCH fix: _fetch_imap_emails now uses a plain SEARCH UNSEEN + FETCH (UID) to resolve sequence numbers to stable UIDs (aioimaplib blocks uid("search")), and wraps all flag names in parentheses (+FLAGS (\Seen), +FLAGS (\Deleted)) as required by RFC 3501 to prevent T-Online and other strict servers from dropping the connection with "Too many invalid IMAP commands".

• CI pipeline fixes: Added FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true to ci.yml (Node.js 20 deprecation), fixed Codecov file: → files: invalid input, replaced <img> with <Image /> from next/image in ProviderWizard.tsx (ESLint no-img-element).

• IMAP reliability: switched to UID-based commands — _fetch_imap_emails now uses UID SEARCH, UID FETCH, and UID STORE throughout. Sequence numbers are volatile (they shift on expunge), causing "Too many invalid IMAP commands" on strict servers (e.g. T-Online). UIDs are stable. The per-message STORE +FLAGS \Seen (redundant — RFC822 sets it implicitly) and per-message STORE +FLAGS \Deleted are replaced with single batch commands. Stale already-seen UIDs are re-marked \Seen in one command. Logout is now in a finally block so a mid-session BYE is handled gracefully.

• Fixed timezone display bug in Mailbox Activity and Admin Logs pages: ISO timestamps without a Z suffix were parsed as local time by JavaScript, shifting "Xm ago" / "Xh ago" displays and absolute dates by the client's UTC offset.

• Fixed worker send_user_notification using rolled-back DB session causing greenlet_spawn has not been called errors; status/last_check_at now always committed before sending notifications via a fresh session.

• Dashboard redesign: Replaced noisy "Recent Processing Runs" table with a per-account "Mailbox Status" view showing last-check status (OK/Error/Pending), relative timestamp, error messages, and lifetime counters. Stats cards updated to show all-time processed count and accounts-with-errors count.

• Provider logos now saved on account creation: provider_name field added to MailAccountCreate and MailAccountUpdate schemas (backend and frontend). ProviderWizard now passes provider_name in its onSelect callback; AddMailAccountModal stores it so logos are displayed correctly on the accounts page.

• Domain-based logo fallback: ProviderLogoBanner now falls back to email-domain matching when provider_name is absent, so all known providers (GMX, WEB.DE, T-Online, etc.) show their logo even on legacy accounts.

• Fetch button UX improvements: The "fetch emails" button on the accounts page now shows a "Fetch" text label for clarity, a tooltip explaining its purpose, a spinning "Fetching…" state during the API call, and a brief green "Queued!" confirmation after success.

• Pull Now: Added "Pull Now" button on Accounts page that immediately queues a process_mail_account Celery task via POST /mail-accounts/{id}/pull-now. Button shows spinner while in flight and is disabled for inactive accounts.

• Fixed 21 mypy type errors: Column[T] vs native type mismatches in notification_service.py, mail_processor.py, auth.py, tasks.py, providers.py, mail_accounts.py, and main.py (lifespan parameter rename).

• Provider logos rework: Logos now displayed as full-width banner strips at the top of each account card using next/image fill + object-contain. Handles all aspect ratios (1:1 square to 6:1 wordmark) without distortion. Proton Mail added.

• Proton Mail provider: Added Proton Mail preset in backend and ProviderWizard frontend. Domains: proton.me, protonmail.com, protonmail.ch, pm.me. Auto-detect and IMAP/POP3 Bridge settings included.

• Redesigned user-facing Logs page to mailbox-centric "Mailbox Activity" view: shows last check status per account + only successful pulls, suppressing noise from empty polling cycles.

• Added has_emails filter to GET /processing-runs and GET /mail-accounts/{id}/processing-runs API endpoints.

• Rename entire project to InboxConverge: all user-visible strings, Docker container/image names, DB defaults, monitoring, and docs updated.

• Domain updated to inboxconverge.com; contact email defaults to christian@inboxconverge.com.

• New configurable env vars: CONTACT_EMAIL, APP_URL, NEXT_PUBLIC_APP_NAME.

• Fixed Black formatting failure in CI (admin.py reformatted).

• Fixed /processing-runs endpoint 404s caused by duplicate path prefix in logs.py.

• Added Semantic Release workflow (release.yml) for automatic versioning and GitHub Releases.

• Added pyproject.toml with [tool.semantic_release] configuration.

• Fixed GitOps update-k8s-manifest job: corrected image tag computation and yq patterns to use registry.cklnet.com (private registry) matching the actual k8s manifest image references, so SHA-pinned tags are properly applied on each deploy.

• Added GitOps auto-deployment step in ci.yml to update preprod k8s manifest in k8s-cluster-state repo.

• Fixed GitOps update-k8s-manifest job: added PAT availability check to skip gracefully when GH_PAT secret is not configured, fixing 403 "Write access to repository not granted" pipeline failure.

• Fixed Celery TypeError: can't subtract offset-naive and offset-aware datetimes in process_all_enabled_accounts — all mail accounts were silently skipped on every scheduled run.

• Fixed Test Connection always reporting success regardless of authentication outcome.

• Added POST /mail-accounts/{account_id}/test endpoint to test existing accounts with stored credentials.

🔴 Critical - Security (In Progress)

Completed ✅

• Add SECRET_KEY validation on startup
• Add ENCRYPTION_KEY validation on startup
• Implement security headers middleware (X-Frame-Options, CSP, HSTS, etc.)
• Implement CSRF protection middleware
• Document all error codes in docs/ERRORS.md
• Create security ADR (Architecture Decision Records)
• Upgrade python-jose 3.3.0 → 3.5.0 (algorithm confusion with OpenSSH ECDSA keys, CVE, affected < 3.4.0)

In Progress 🔨

• Enable rate limiting per user/tier
• Fix bare exception handlers throughout codebase
• Update datetime usage to timezone-aware (DateTime(timezone=True) columns and lambda: datetime.now(timezone.utc) defaults; fixes DBAPIError from asyncpg on timezone-naive columns)
• Fix Exception terminating connection in Celery workers: call await engine.dispose() inside task coroutine so pooled asyncpg connections are closed before the event loop is torn down
• Validate redirect_uri to prevent open redirect vulnerabilities
• Add per-user random salt for encryption (currently deterministic)

Not Started 📋

• Implement audit logging middleware
• Add 2FA support
• Implement API key authentication
• Set up secrets management (HashiCorp Vault or AWS Secrets Manager)
• Professional security audit/penetration testing

---

🤖 High Priority - Agentic Coding Infrastructure

Completed ✅

• Create .github/ISSUE_TEMPLATE/ (bug_report.md, feature_request.md, test_needed.md)
• Create .github/PULL_REQUEST_TEMPLATE.md
• Create docs/CODING_PATTERNS.md with best practices
• Create docs/ERRORS.md documenting error codes
• Create docs/adr/ for Architecture Decision Records
• Add Makefile with common development tasks
• Add .pre-commit-config.yaml with black, ruff, mypy
• Create CHANGELOG.md with version history
• Add .yamllint.yml configuration
• Add .secrets.baseline for detect-secrets

In Progress 🔨

• Reorganize documentation into docs/ directory
• Complete ADR documentation (add ADR-003 through ADR-010)
• Create GitHub Projects board for task management

Not Started 📋

• Add commitlint.config.js for conventional commits
• Create video tutorials for setup
• Add interactive setup wizard
• Document migration path from legacy script
• Create performance benchmarks baseline
• Set up Discord/Slack community

---

🧪 High Priority - Testing Infrastructure

Completed ✅

• Create backend/tests/ directory structure (unit, integration, e2e)
• Add backend/tests/conftest.py with fixtures
• Add backend/pytest.ini configuration
• Create sample unit tests (test_security.py, test_config.py)
• Add user and mail account factory fixtures
• Write unit tests for security module (100% coverage)
• Write unit tests for middleware (98% coverage)
• Write unit tests for schemas and validation
• Write unit tests for application factory and core endpoints
• Reach 50%+ test coverage (currently 59%)
• Write tests for Celery tasks (96% coverage for tasks.py)
• Write unit tests for admin endpoints (87 tests, 100% coverage on admin.py)
• Frontend test coverage: Added 113 new tests across 7 new test suites covering all components and utility functions. Installed @testing-library/react, @testing-library/jest-dom, @testing-library/user-event. New suites: date-utils (30 tests), API interceptors (9 tests), AuthGuard (6 tests), QueryProvider (2 tests), DashboardLayout (14 tests), NotificationWizard (32 tests), ProviderWizard (20 tests). Total frontend: 119 tests across 8 suites.

In Progress 🔨

• Write unit tests for authentication (target 80%+ coverage)
• Write unit tests for mail processing
• Write integration tests for API endpoints

Not Started 📋

• Add end-to-end tests
• Add performance/load tests
• Create mock POP3/IMAP server for testing
• Add test data seeding scripts
• Reach 80%+ code coverage

---

🔄 High Priority - CI/CD Pipeline

Completed ✅

• Create .github/workflows/test.yml for automated testing
• Create .github/workflows/lint.yml for code quality checks
• Create .github/workflows/security.yml for security scanning
• Existing .github/workflows/docker-build.yml for Docker images
• Set up automatic dependency updates (Dependabot)
• Merge Dependabot dependency updates (PRs #26–#49)
• Remove CodeQL checks from CI (was blocking builds)
• Upgrade SQLAlchemy to 2.0.48 to fix Python 3.14 test failures
• Fix Docker build failure: wrap useSearchParams() in Suspense boundary in /auth/callback page
• Fix frontend API URL hardcoded to localhost:8000 in production: replaced build-time NEXT_PUBLIC_API_URL with a runtime Next.js Route Handler proxy (/api/v1/[...path]) reading BACKEND_URL at server startup
• Log BACKEND_URL at frontend server startup and include target URL in per-request proxy error messages
• Fix UndefinedTableError on first boot: lifespan event now runs Base.metadata.create_all() so tables are created automatically when no migrations have been applied
• Fix ProgrammingError (cached statement plan is invalid) during startup: set prepared_statement_cache_size=0 on the asyncpg engine to prevent plan invalidation when CREATE TYPE DDL runs at startup

In Progress 🔨

• Configure branch protection rules
• Set up Codecov integration

Not Started 📋

• Add deployment workflow (dual-registry: GHCR + private registry)
• Add release workflow with automated changelog
• Configure status checks for PRs
• Add performance regression detection

---

🟡 Medium Priority - Code Quality

Completed ✅

• Create coding patterns documentation
• Define error code structure

In Progress 🔨

• Add comprehensive type hints to all functions
• Add docstrings to all public methods
• Move magic numbers to constants
• Improve error messages with context

Not Started 📋

• Add database indexes for performance
• Complete database migration scripts
• Implement retry logic for Celery tasks
• Add structured JSON logging
• Refactor mixed async/blocking code in mail processor
• Complete API documentation with examples

---

📦 Medium Priority - Production Readiness

Completed ✅

• Basic health check endpoint exists
• Database-backed configuration (AppSetting model + ConfigService)
• Admin API for managing settings (/api/v1/settings)
• Default settings seeded on first startup

In Progress 🔨

• Improve health checks (DB/Redis connectivity)

Not Started 📋

• Create production docker-compose.yml
• Add Kubernetes manifests (deployment, service, ingress)
• Create Helm chart for easy deployment
• Add nginx reverse proxy configuration
• Document backup strategy
• Create comprehensive deployment guide
• Set up log aggregation (ELK/Loki)
• Configure alerting system

---

📊 Medium Priority - Observability

Completed ✅

• Add Prometheus metrics endpoint (/metrics) to FastAPI backend
• Instrument HTTP layer (request count + latency histograms per method/endpoint/status)
• Instrument mail-processing tasks (runs, emails fetched/forwarded/failed, duration)
• Instrument Gmail API operations (inject, verify, get_profile, get_label — count + latency)
• Track OAuth token refreshes and credential invalidation events
• Instrument auth endpoints (logins, registrations, OAuth callbacks — by method/status)
• Instrument Celery tasks (count + duration per task name)
• Add Prometheus scrape config (monitoring/prometheus.yml)
• Add Grafana auto-provisioned datasource and pre-built dashboard (monitoring/grafana/)
• Add Prometheus + Grafana services to docker-compose.new.yml (Grafana on port 3001)

Not Started 📋

• Integrate Sentry for error tracking
• Add structured logging with correlation IDs (per-email ProcessingLog entries now captured in DB)
• Add APM (Application Performance Monitoring)
• Set up uptime monitoring
• Create runbook for common issues

---

✨ Low Priority - Feature Completion

Not Started 📋

• Implement Stripe webhook handling
• Add scheduled Celery tasks for email processing
• Account enable/disable toggle (UX + backend)
• Per-user SMTP configuration (UX + backend)
• Gmail API one-click OAuth grant flow with token refresh and revocation handling
• Configurable Gmail import labels (default {{source_email}} + imported, editable in Settings with reset-to-default action)
• Decoupled Google Sign-In from Gmail API permissions: login now requests only basic profile scopes; Gmail access is granted separately via Settings
• Message deduplication (POP3 UIDL + IMAP \Seen flag + DB tracking)
• Debug email: "Send Debug Email" button in Settings injects a test message (from christian@docuelevate.org, dated today, labelled test + imported, placed in inbox) to verify end-to-end Gmail API delivery
• Logging & reporting: per-email ProcessingLog capture in worker; user /logs page; admin /admin/logs page; GDPR masking utilities (gdpr.py)
• Implement GDPR data export endpoint
• Complete notification service integration (Apprise)
• Add advanced email filtering
• Implement OAuth2 for Gmail (instead of App Passwords)
• Add attachment handling improvements
• Add email archiving feature
• Implement webhook support for external integrations

---

🖥️ High Priority - Frontend Completion

The Next.js frontend has pages and components implemented but is not functional because the API client layer is missing.

Critical Blockers 🔴

• Create frontend/src/lib/api.ts — API client using axios
  • Exports: authApi, mailAccountsApi, processingRunsApi, userApi
  • Exports types: User, MailAccount, MailAccountCreate, ProcessingRun
  • 8 files import from @/lib/api — all compilation errors resolved
• Fix infinite spinner on home page: isLoading now initialises based on token presence; home page performs auth check when token exists

Existing Pages (UI done, need API wiring) 🔨

• Landing page (app/page.tsx)
• Login page with email/password + Google OAuth
• Registration page
• OAuth callback handler
• Dashboard with stats cards and processing runs table
• Mail accounts list with CRUD operations + enable/disable toggle
• Settings page — Profile, Gmail API connection, SMTP relay, Account info, Security
• AddMailAccountModal component (auto-detect, test connection, all required fields, is_enabled checkbox)
• Fix AddMailAccountModal edit mode: backend now returns username in MailAccountResponse; all fields (including protocol, host, port, use_ssl, username) are editable in edit mode and pre-populated from the stored account; Auto-Detect is shown in edit mode too; only password is omitted from the update payload when left blank
• DashboardLayout with responsive sidebar
• AuthGuard for protected routes
• Fix wizard grey screen (Tailwind v4 bg-opacity → /75 syntax, modal restructure)
• /auth/gmail-callback page for Gmail OAuth one-click flow
• /logs page — user processing history: paginated runs table with expandable per-email log panel (subject, sender, size, status)
• Dashboard — "Recent Processing Runs" now wired to real /processing-runs endpoint; shows account name and links to /logs

Not Started 📋

• End-to-end testing of frontend against backend API
• Error boundary components
• Loading skeletons / proper loading states
• Notification preferences UI
• Notification channels page (/notifications) with full CRUD, wizard, and test button
• Apprise-powered notification wizard for Telegram, Discord, Slack, Email, Webhook, and custom URLs
• Admin system alert channels section (/admin page) with full CRUD and test
• Subscription management / billing UI

Admin Interface ✅

• Admin section in sidebar (visible to superusers only)
• Admin overview page (/admin) with system-wide stats
• User management page (/admin/users) — list, edit, delete users; assign plans; promote/demote admin
• Plan management page (/admin/plans) — full CRUD for subscription plans (mailboxes, emails/day, interval, pricing)
• ADMIN_EMAIL env var with default christian@inboxconverge.com; admin auto-promoted on login and on every application startup (fixes pre-existing accounts)
• is_superuser exposed in /users/me response
• Admin badge (purple shield) shown in top bar for superusers
• Fix blank page on direct navigation to /admin*: moved superuser guard inside <AuthGuard> so auth check always runs on fresh load
• /admin/logs page — system-wide processing activity: expandable run table + flat per-email log table with GDPR-masked sender addresses; filterable by user ID, status, log level

---

📅 Milestone Timeline

Milestone 1: Security & Infrastructure (Week 1-2) 🔴

Goal: Make repository secure and AI-agent friendly

Tasks:

• Complete all security hardening
• Finish agentic coding infrastructure
• Set up CI/CD pipeline
• Reach 50% test coverage

Success Criteria:

• All security validators passing
• CI/CD running on all PRs
• Issue/PR templates in use
• Pre-commit hooks working

---

Milestone 2: Testing & Quality (Week 3-4) 🧪

Goal: Establish quality baseline

Tasks:

• Write comprehensive test suite
• Reach 80% code coverage
• Fix all linting issues
• Complete API documentation

Success Criteria:

• 80%+ test coverage
• All tests passing
• Zero critical security issues
• API docs complete

---

Milestone 3: Production Readiness (Week 5-6) 📦

Goal: Ready for production deployment

Tasks:

• Complete observability setup
• Add Kubernetes manifests
• Implement rate limiting
• Add audit logging
• Complete deployment documentation

Success Criteria:

• Can deploy to Kubernetes
• Monitoring and alerting active
• Health checks comprehensive
• Deployment documented

---

Milestone 4: Feature Completion (Week 7-8) ✨

Goal: Complete remaining features

Tasks:

• Implement Stripe webhooks
• Add Celery scheduled tasks
• Complete notification integration
• Build basic frontend

Success Criteria:

• Stripe integration working
• Scheduled tasks running
• Notifications functional
• Basic UI available

---

📊 Progress Tracking

Overall Progress by Category

Category	Progress	Status
Security	60%	🟡 In Progress
Agentic Infrastructure	98%	🟢 Near Complete
Testing	59%	🟡 In Progress
CI/CD	80%	🟢 Near Complete
Code Quality	40%	🔴 Needs Work
Production Ready	30%	🔴 Needs Work
Observability	10%	🔴 Needs Work
Backend Features	85%	🟢 Near Complete
Frontend	65%	🟢 Near Complete

Overall Repository Readiness: 55% ⚠️

---

🎯 Next Actions (Priority Order)

1. Immediate (Today):

   • Create frontend/src/lib/api.ts (frontend is broken without it)
   • Fix remaining security issues (bare excepts, datetime, redirect_uri)
   • Add backend endpoint for processing runs (needed by dashboard)
   • Build logging & reporting: per-email ProcessingLog capture, user /logs page, admin /admin/logs page, GDPR masking

2. This Week:

   • Enable rate limiting
   • Add audit logging
   • Write more unit tests (target 70% coverage)
   • Complete ADR documentation
   • End-to-end test frontend against backend

3. Next Week:

   • Kubernetes manifests
   • Prometheus metrics
   • Sentry integration
   • Production docker-compose

4. This Month:

   • 80% test coverage
   • Complete all documentation
   • Professional security audit
   • First production deployment

---

📝 Notes

Dependencies Between Tasks

• Security hardening must complete before production deployment
• Test infrastructure needed before reaching coverage goals
• CI/CD needed before enforcing quality standards
• Observability needed before production monitoring

AI Agent Readiness

After Milestone 1 completes, AI agents will have:

• Clear issue templates to report bugs
• Coding patterns to follow
• Test fixtures to write tests
• CI/CD to validate changes
• Pre-commit hooks to enforce quality

Production Blockers

Must complete before production:

1. All critical security issues
2. Basic monitoring/alerting
3. Backup strategy
4. Incident response plan
5. 50%+ test coverage

---

Last Updated: 2026-03-25 Maintained By: Development Team Review Frequency: Weekly