# TODO & Milestones Comprehensive task breakdown for repository improvements and production readiness. ## ✅ Recently Completed - [x] **Improved test coverage for `mail_processor.py`**: Added 46 new unit tests covering POP3 connection testing, POP3 email fetching, IMAP edge cases, email forwarding (STARTTLS/SSL), and `fetch_emails`/`test_connection` routing. Coverage increased from ~42% to 98%. - [x] **Log noise reduction**: Suppressed `ignored untagged response` INFO messages from `aioimaplib` in Celery workers (set logger to WARNING). Eliminated repeated `file_cache is only supported with oauth2client<4.0.0` warnings from the Gmail API client by passing `cache_discovery=False` to `googleapiclient.discovery.build()`. - [x] **ESLint fix**: Converted `frontend/jest.config.js` to `jest.config.mjs` (ES module syntax) to resolve `@typescript-eslint/no-require-imports` lint error. - [x] **Codecov integration**: Added Codecov coverage reporting with `CODECOV_TOKEN` authentication. Set up Jest for frontend tests with lcov coverage, updated CI to collect and upload both backend (XML via pytest-cov) and frontend (lcov via Jest) coverage reports to Codecov with separate `backend` and `frontend` flags. - [x] **IMAP: fix all emails appearing empty** — `aioimaplib` stores RFC822 literal data as `bytearray`, not `bytes`. The extraction loop was checking `isinstance(line, bytes)` which returns `False` for `bytearray`, so every email body was silently skipped. Fixed to accept both types and convert to `bytes`. Affected T-Online, GMX, and all IMAP accounts. - [x] **CI: fix safety scan EOF error** — replaced `safety scan --json` (Safety CLI v3 requires interactive login) with `pip-audit` (no auth required, maintained by PyPA). - [x] **Security: upgrade fastapi/starlette and fix safety CI command** — Upgraded `fastapi` to `0.135.2` (pulls in `starlette>=1.0.0`) fixing 4 DoS CVEs in `starlette<=0.35.1`; replaced deprecated `safety check` with `safety scan`; added `.safety-policy.yml` to suppress unfixable `ecdsa` side-channel CVEs (maintainers won't fix). - [x] **IMAP RFC 3501 flag syntax & aioimaplib UID SEARCH fix**: `_fetch_imap_emails` now uses a plain `SEARCH UNSEEN` + `FETCH (UID)` to resolve sequence numbers to stable UIDs (aioimaplib blocks `uid("search")`), and wraps all flag names in parentheses (`+FLAGS (\Seen)`, `+FLAGS (\Deleted)`) as required by RFC 3501 to prevent T-Online and other strict servers from dropping the connection with "Too many invalid IMAP commands". - [x] **CI pipeline fixes**: Added `FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true` to `ci.yml` (Node.js 20 deprecation), fixed Codecov `file:` → `files:` invalid input, replaced `` with `` from `next/image` in `ProviderWizard.tsx` (ESLint no-img-element). - [x] **IMAP reliability: switched to UID-based commands** — `_fetch_imap_emails` now uses `UID SEARCH`, `UID FETCH`, and `UID STORE` throughout. Sequence numbers are volatile (they shift on expunge), causing "Too many invalid IMAP commands" on strict servers (e.g. T-Online). UIDs are stable. The per-message `STORE +FLAGS \Seen` (redundant — RFC822 sets it implicitly) and per-message `STORE +FLAGS \Deleted` are replaced with single batch commands. Stale already-seen UIDs are re-marked `\Seen` in one command. Logout is now in a `finally` block so a mid-session `BYE` is handled gracefully. - [x] Fixed timezone display bug in Mailbox Activity and Admin Logs pages: ISO timestamps without a `Z` suffix were parsed as local time by JavaScript, shifting "Xm ago" / "Xh ago" displays and absolute dates by the client's UTC offset. - [x] Fixed worker `send_user_notification` using rolled-back DB session causing `greenlet_spawn has not been called` errors; status/`last_check_at` now always committed before sending notifications via a fresh session. - [x] **Dashboard redesign**: Replaced noisy "Recent Processing Runs" table with a per-account "Mailbox Status" view showing last-check status (OK/Error/Pending), relative timestamp, error messages, and lifetime counters. Stats cards updated to show all-time processed count and accounts-with-errors count. - [x] **Provider logos now saved on account creation**: `provider_name` field added to `MailAccountCreate` and `MailAccountUpdate` schemas (backend and frontend). `ProviderWizard` now passes `provider_name` in its `onSelect` callback; `AddMailAccountModal` stores it so logos are displayed correctly on the accounts page. - [x] **Domain-based logo fallback**: `ProviderLogoBanner` now falls back to email-domain matching when `provider_name` is absent, so all known providers (GMX, WEB.DE, T-Online, etc.) show their logo even on legacy accounts. - [x] **Fetch button UX improvements**: The "fetch emails" button on the accounts page now shows a "Fetch" text label for clarity, a tooltip explaining its purpose, a spinning "Fetching…" state during the API call, and a brief green "Queued!" confirmation after success. - [x] **Pull Now**: Added "Pull Now" button on Accounts page that immediately queues a `process_mail_account` Celery task via `POST /mail-accounts/{id}/pull-now`. Button shows spinner while in flight and is disabled for inactive accounts. - [x] Fixed 21 mypy type errors: `Column[T]` vs native type mismatches in `notification_service.py`, `mail_processor.py`, `auth.py`, `tasks.py`, `providers.py`, `mail_accounts.py`, and `main.py` (`lifespan` parameter rename). - [x] **Provider logos rework**: Logos now displayed as full-width banner strips at the top of each account card using `next/image fill + object-contain`. Handles all aspect ratios (1:1 square to 6:1 wordmark) without distortion. Proton Mail added. - [x] **Proton Mail provider**: Added Proton Mail preset in backend and ProviderWizard frontend. Domains: proton.me, protonmail.com, protonmail.ch, pm.me. Auto-detect and IMAP/POP3 Bridge settings included. - [x] Redesigned user-facing Logs page to mailbox-centric "Mailbox Activity" view: shows last check status per account + only successful pulls, suppressing noise from empty polling cycles. - [x] Added `has_emails` filter to `GET /processing-runs` and `GET /mail-accounts/{id}/processing-runs` API endpoints. - [x] Rename entire project to **InboxConverge**: all user-visible strings, Docker container/image names, DB defaults, monitoring, and docs updated. - [x] Domain updated to `inboxconverge.com`; contact email defaults to `christian@inboxconverge.com`. - [x] New configurable env vars: `CONTACT_EMAIL`, `APP_URL`, `NEXT_PUBLIC_APP_NAME`. - [x] Fixed Black formatting failure in CI (`admin.py` reformatted). - [x] Fixed `/processing-runs` endpoint 404s caused by duplicate path prefix in `logs.py`. - [x] Added Semantic Release workflow (`release.yml`) for automatic versioning and GitHub Releases. - [x] Added `pyproject.toml` with `[tool.semantic_release]` configuration. - [x] Fixed GitOps `update-k8s-manifest` job: corrected image tag computation and `yq` patterns to use `registry.cklnet.com` (private registry) matching the actual k8s manifest image references, so SHA-pinned tags are properly applied on each deploy. - [x] Added GitOps auto-deployment step in `ci.yml` to update preprod k8s manifest in `k8s-cluster-state` repo. - [x] Fixed GitOps `update-k8s-manifest` job: added PAT availability check to skip gracefully when `GH_PAT` secret is not configured, fixing 403 "Write access to repository not granted" pipeline failure. - [x] Fixed Celery `TypeError: can't subtract offset-naive and offset-aware datetimes` in `process_all_enabled_accounts` — all mail accounts were silently skipped on every scheduled run. - [x] Fixed **Test Connection** always reporting success regardless of authentication outcome. - [x] Added `POST /mail-accounts/{account_id}/test` endpoint to test existing accounts with stored credentials. ## 🔴 Critical - Security (In Progress) ### Completed ✅ - [x] Add SECRET_KEY validation on startup - [x] Add ENCRYPTION_KEY validation on startup - [x] Implement security headers middleware (X-Frame-Options, CSP, HSTS, etc.) - [x] Implement CSRF protection middleware - [x] Document all error codes in docs/ERRORS.md - [x] Create security ADR (Architecture Decision Records) - [x] Upgrade `python-jose` 3.3.0 → 3.5.0 (algorithm confusion with OpenSSH ECDSA keys, CVE, affected < 3.4.0) ### In Progress 🔨 - [ ] Enable rate limiting per user/tier - [x] Fix bare exception handlers throughout codebase - [x] Update datetime usage to timezone-aware (`DateTime(timezone=True)` columns and `lambda: datetime.now(timezone.utc)` defaults; fixes `DBAPIError` from asyncpg on timezone-naive columns) - [x] Fix `Exception terminating connection` in Celery workers: call `await engine.dispose()` inside task coroutine so pooled asyncpg connections are closed before the event loop is torn down - [ ] Validate redirect_uri to prevent open redirect vulnerabilities - [ ] Add per-user random salt for encryption (currently deterministic) ### Not Started 📋 - [ ] Implement audit logging middleware - [ ] Add 2FA support - [ ] Implement API key authentication - [ ] Set up secrets management (HashiCorp Vault or AWS Secrets Manager) - [ ] Professional security audit/penetration testing --- ## 🤖 High Priority - Agentic Coding Infrastructure ### Completed ✅ - [x] Create `.github/ISSUE_TEMPLATE/` (bug_report.md, feature_request.md, test_needed.md) - [x] Create `.github/PULL_REQUEST_TEMPLATE.md` - [x] Create `docs/CODING_PATTERNS.md` with best practices - [x] Create `docs/ERRORS.md` documenting error codes - [x] Create `docs/adr/` for Architecture Decision Records - [x] Add `Makefile` with common development tasks - [x] Add `.pre-commit-config.yaml` with black, ruff, mypy - [x] Create `CHANGELOG.md` with version history - [x] Add `.yamllint.yml` configuration - [x] Add `.secrets.baseline` for detect-secrets ### In Progress 🔨 - [x] Reorganize documentation into `docs/` directory - [x] Complete ADR documentation (add ADR-003 through ADR-010) - [ ] Create GitHub Projects board for task management ### Not Started 📋 - [ ] Add `commitlint.config.js` for conventional commits - [ ] Create video tutorials for setup - [ ] Add interactive setup wizard - [ ] Document migration path from legacy script - [ ] Create performance benchmarks baseline - [ ] Set up Discord/Slack community --- ## 🧪 High Priority - Testing Infrastructure ### Completed ✅ - [x] Create `backend/tests/` directory structure (unit, integration, e2e) - [x] Add `backend/tests/conftest.py` with fixtures - [x] Add `backend/pytest.ini` configuration - [x] Create sample unit tests (test_security.py, test_config.py) - [x] Add user and mail account factory fixtures - [x] Write unit tests for security module (100% coverage) - [x] Write unit tests for middleware (98% coverage) - [x] Write unit tests for schemas and validation - [x] Write unit tests for application factory and core endpoints - [x] Reach 50%+ test coverage (currently 59%) - [x] Write tests for Celery tasks (96% coverage for `tasks.py`) - [x] Write unit tests for admin endpoints (87 tests, 100% coverage on admin.py) - [x] **Frontend test coverage**: Added 113 new tests across 7 new test suites covering all components and utility functions. Installed `@testing-library/react`, `@testing-library/jest-dom`, `@testing-library/user-event`. New suites: `date-utils` (30 tests), API interceptors (9 tests), `AuthGuard` (6 tests), `QueryProvider` (2 tests), `DashboardLayout` (14 tests), `NotificationWizard` (32 tests), `ProviderWizard` (20 tests). Total frontend: 119 tests across 8 suites. ### In Progress 🔨 - [ ] Write unit tests for authentication (target 80%+ coverage) - [ ] Write unit tests for mail processing - [ ] Write integration tests for API endpoints ### Not Started 📋 - [ ] Add end-to-end tests - [ ] Add performance/load tests - [ ] Create mock POP3/IMAP server for testing - [ ] Add test data seeding scripts - [ ] Reach 80%+ code coverage --- ## 🔄 High Priority - CI/CD Pipeline ### Completed ✅ - [x] Create `.github/workflows/test.yml` for automated testing - [x] Create `.github/workflows/lint.yml` for code quality checks - [x] Create `.github/workflows/security.yml` for security scanning - [x] Existing `.github/workflows/docker-build.yml` for Docker images - [x] Set up automatic dependency updates (Dependabot) - [x] Merge Dependabot dependency updates (PRs #26–#49) - [x] Remove CodeQL checks from CI (was blocking builds) - [x] Upgrade SQLAlchemy to 2.0.48 to fix Python 3.14 test failures - [x] Fix Docker build failure: wrap `useSearchParams()` in Suspense boundary in `/auth/callback` page - [x] Fix frontend API URL hardcoded to `localhost:8000` in production: replaced build-time `NEXT_PUBLIC_API_URL` with a runtime Next.js Route Handler proxy (`/api/v1/[...path]`) reading `BACKEND_URL` at server startup - [x] Log `BACKEND_URL` at frontend server startup and include target URL in per-request proxy error messages - [x] Fix `UndefinedTableError` on first boot: lifespan event now runs `Base.metadata.create_all()` so tables are created automatically when no migrations have been applied - [x] Fix `ProgrammingError` (cached statement plan is invalid) during startup: set `prepared_statement_cache_size=0` on the asyncpg engine to prevent plan invalidation when `CREATE TYPE` DDL runs at startup ### In Progress 🔨 - [ ] Configure branch protection rules - [ ] Set up Codecov integration ### Not Started 📋 - [x] Add deployment workflow (dual-registry: GHCR + private registry) - [ ] Add release workflow with automated changelog - [ ] Configure status checks for PRs - [ ] Add performance regression detection --- ## 🟡 Medium Priority - Code Quality ### Completed ✅ - [x] Create coding patterns documentation - [x] Define error code structure ### In Progress 🔨 - [ ] Add comprehensive type hints to all functions - [ ] Add docstrings to all public methods - [ ] Move magic numbers to constants - [ ] Improve error messages with context ### Not Started 📋 - [ ] Add database indexes for performance - [ ] Complete database migration scripts - [ ] Implement retry logic for Celery tasks - [ ] Add structured JSON logging - [ ] Refactor mixed async/blocking code in mail processor - [ ] Complete API documentation with examples --- ## 📦 Medium Priority - Production Readiness ### Completed ✅ - [x] Basic health check endpoint exists - [x] Database-backed configuration (`AppSetting` model + `ConfigService`) - [x] Admin API for managing settings (`/api/v1/settings`) - [x] Default settings seeded on first startup ### In Progress 🔨 - [ ] Improve health checks (DB/Redis connectivity) ### Not Started 📋 - [ ] Create production docker-compose.yml - [ ] Add Kubernetes manifests (deployment, service, ingress) - [ ] Create Helm chart for easy deployment - [ ] Add nginx reverse proxy configuration - [ ] Document backup strategy - [ ] Create comprehensive deployment guide - [ ] Set up log aggregation (ELK/Loki) - [ ] Configure alerting system --- ## 📊 Medium Priority - Observability ### Completed ✅ - [x] Add Prometheus metrics endpoint (`/metrics`) to FastAPI backend - [x] Instrument HTTP layer (request count + latency histograms per method/endpoint/status) - [x] Instrument mail-processing tasks (runs, emails fetched/forwarded/failed, duration) - [x] Instrument Gmail API operations (inject, verify, get_profile, get_label — count + latency) - [x] Track OAuth token refreshes and credential invalidation events - [x] Instrument auth endpoints (logins, registrations, OAuth callbacks — by method/status) - [x] Instrument Celery tasks (count + duration per task name) - [x] Add Prometheus scrape config (`monitoring/prometheus.yml`) - [x] Add Grafana auto-provisioned datasource and pre-built dashboard (`monitoring/grafana/`) - [x] Add Prometheus + Grafana services to `docker-compose.new.yml` (Grafana on port 3001) ### Not Started 📋 - [ ] Integrate Sentry for error tracking - [x] Add structured logging with correlation IDs (per-email ProcessingLog entries now captured in DB) - [ ] Add APM (Application Performance Monitoring) - [ ] Set up uptime monitoring - [ ] Create runbook for common issues --- ## ✨ Low Priority - Feature Completion ### Not Started 📋 - [ ] Implement Stripe webhook handling - [ ] Add scheduled Celery tasks for email processing - [x] Account enable/disable toggle (UX + backend) - [x] Per-user SMTP configuration (UX + backend) - [x] Gmail API one-click OAuth grant flow with token refresh and revocation handling - [x] Configurable Gmail import labels (default `{{source_email}}` + `imported`, editable in Settings with reset-to-default action) - [x] Decoupled Google Sign-In from Gmail API permissions: login now requests only basic profile scopes; Gmail access is granted separately via Settings - [x] Message deduplication (POP3 UIDL + IMAP \Seen flag + DB tracking) - [x] **Debug email**: "Send Debug Email" button in Settings injects a test message (from christian@docuelevate.org, dated today, labelled `test` + `imported`, placed in inbox) to verify end-to-end Gmail API delivery - [x] **Logging & reporting**: per-email ProcessingLog capture in worker; user `/logs` page; admin `/admin/logs` page; GDPR masking utilities (`gdpr.py`) - [ ] Implement GDPR data export endpoint - [x] Complete notification service integration (Apprise) - [ ] Add advanced email filtering - [ ] Implement OAuth2 for Gmail (instead of App Passwords) - [ ] Add attachment handling improvements - [ ] Add email archiving feature - [ ] Implement webhook support for external integrations --- ## 🖥️ High Priority - Frontend Completion The Next.js frontend has pages and components implemented but is **not functional** because the API client layer is missing. ### Critical Blockers 🔴 - [x] Create `frontend/src/lib/api.ts` — API client using axios - Exports: `authApi`, `mailAccountsApi`, `processingRunsApi`, `userApi` - Exports types: `User`, `MailAccount`, `MailAccountCreate`, `ProcessingRun` - 8 files import from `@/lib/api` — all compilation errors resolved - [x] Fix infinite spinner on home page: `isLoading` now initialises based on token presence; home page performs auth check when token exists ### Existing Pages (UI done, need API wiring) 🔨 - [x] Landing page (`app/page.tsx`) - [x] Login page with email/password + Google OAuth - [x] Registration page - [x] OAuth callback handler - [x] Dashboard with stats cards and processing runs table - [x] Mail accounts list with CRUD operations + enable/disable toggle - [x] Settings page — Profile, Gmail API connection, SMTP relay, Account info, Security - [x] `AddMailAccountModal` component (auto-detect, test connection, all required fields, is_enabled checkbox) - [x] Fix `AddMailAccountModal` edit mode: backend now returns `username` in `MailAccountResponse`; all fields (including protocol, host, port, use\_ssl, username) are editable in edit mode and pre-populated from the stored account; Auto-Detect is shown in edit mode too; only password is omitted from the update payload when left blank - [x] `DashboardLayout` with responsive sidebar - [x] `AuthGuard` for protected routes - [x] Fix wizard grey screen (Tailwind v4 `bg-opacity` → `/75` syntax, modal restructure) - [x] `/auth/gmail-callback` page for Gmail OAuth one-click flow - [x] **`/logs` page** — user processing history: paginated runs table with expandable per-email log panel (subject, sender, size, status) - [x] **Dashboard** — "Recent Processing Runs" now wired to real `/processing-runs` endpoint; shows account name and links to `/logs` ### Not Started 📋 - [ ] End-to-end testing of frontend against backend API - [ ] Error boundary components - [ ] Loading skeletons / proper loading states - [ ] Notification preferences UI - [x] Notification channels page (`/notifications`) with full CRUD, wizard, and test button - [x] Apprise-powered notification wizard for Telegram, Discord, Slack, Email, Webhook, and custom URLs - [x] Admin system alert channels section (`/admin` page) with full CRUD and test - [ ] Subscription management / billing UI ### Admin Interface ✅ - [x] Admin section in sidebar (visible to superusers only) - [x] Admin overview page (`/admin`) with system-wide stats - [x] User management page (`/admin/users`) — list, edit, delete users; assign plans; promote/demote admin - [x] Plan management page (`/admin/plans`) — full CRUD for subscription plans (mailboxes, emails/day, interval, pricing) - [x] `ADMIN_EMAIL` env var with default `christian@inboxconverge.com`; admin auto-promoted on login and on every application startup (fixes pre-existing accounts) - [x] `is_superuser` exposed in `/users/me` response - [x] Admin badge (purple shield) shown in top bar for superusers - [x] Fix blank page on direct navigation to `/admin*`: moved superuser guard inside `` so auth check always runs on fresh load - [x] **`/admin/logs` page** — system-wide processing activity: expandable run table + flat per-email log table with GDPR-masked sender addresses; filterable by user ID, status, log level --- ## 📅 Milestone Timeline ### Milestone 1: Security & Infrastructure (Week 1-2) 🔴 **Goal**: Make repository secure and AI-agent friendly **Tasks**: - Complete all security hardening - Finish agentic coding infrastructure - Set up CI/CD pipeline - Reach 50% test coverage **Success Criteria**: - All security validators passing - CI/CD running on all PRs - Issue/PR templates in use - Pre-commit hooks working --- ### Milestone 2: Testing & Quality (Week 3-4) 🧪 **Goal**: Establish quality baseline **Tasks**: - Write comprehensive test suite - Reach 80% code coverage - Fix all linting issues - Complete API documentation **Success Criteria**: - 80%+ test coverage - All tests passing - Zero critical security issues - API docs complete --- ### Milestone 3: Production Readiness (Week 5-6) 📦 **Goal**: Ready for production deployment **Tasks**: - Complete observability setup - Add Kubernetes manifests - Implement rate limiting - Add audit logging - Complete deployment documentation **Success Criteria**: - Can deploy to Kubernetes - Monitoring and alerting active - Health checks comprehensive - Deployment documented --- ### Milestone 4: Feature Completion (Week 7-8) ✨ **Goal**: Complete remaining features **Tasks**: - Implement Stripe webhooks - Add Celery scheduled tasks - Complete notification integration - Build basic frontend **Success Criteria**: - Stripe integration working - Scheduled tasks running - Notifications functional - Basic UI available --- ## 📊 Progress Tracking ### Overall Progress by Category | Category | Progress | Status | |----------|----------|--------| | Security | 60% | 🟡 In Progress | | Agentic Infrastructure | 98% | 🟢 Near Complete | | Testing | 59% | 🟡 In Progress | | CI/CD | 80% | 🟢 Near Complete | | Code Quality | 40% | 🔴 Needs Work | | Production Ready | 30% | 🔴 Needs Work | | Observability | 10% | 🔴 Needs Work | | Backend Features | 85% | 🟢 Near Complete | | Frontend | 65% | 🟢 Near Complete | **Overall Repository Readiness**: 55% ⚠️ --- ## 🎯 Next Actions (Priority Order) 1. **Immediate** (Today): - [x] Create `frontend/src/lib/api.ts` (frontend is broken without it) - [x] Fix remaining security issues (bare excepts, datetime, redirect_uri) - [x] Add backend endpoint for processing runs (needed by dashboard) - [x] Build logging & reporting: per-email ProcessingLog capture, user `/logs` page, admin `/admin/logs` page, GDPR masking 2. **This Week**: - [ ] Enable rate limiting - [ ] Add audit logging - [ ] Write more unit tests (target 70% coverage) - [x] Complete ADR documentation - [ ] End-to-end test frontend against backend 3. **Next Week**: - [ ] Kubernetes manifests - [ ] Prometheus metrics - [ ] Sentry integration - [ ] Production docker-compose 4. **This Month**: - [ ] 80% test coverage - [ ] Complete all documentation - [ ] Professional security audit - [ ] First production deployment --- ## 📝 Notes ### Dependencies Between Tasks - Security hardening must complete before production deployment - Test infrastructure needed before reaching coverage goals - CI/CD needed before enforcing quality standards - Observability needed before production monitoring ### AI Agent Readiness After Milestone 1 completes, AI agents will have: - Clear issue templates to report bugs - Coding patterns to follow - Test fixtures to write tests - CI/CD to validate changes - Pre-commit hooks to enforce quality ### Production Blockers Must complete before production: 1. All critical security issues 2. Basic monitoring/alerting 3. Backup strategy 4. Incident response plan 5. 50%+ test coverage --- **Last Updated**: 2026-03-25 **Maintained By**: Development Team **Review Frequency**: Weekly