Merge branch 'main' into copilot/fix-mailbox-parameter-editing

This commit is contained in:
Christian Krakau-Louis
2026-03-26 11:47:37 +01:00
committed by GitHub
9 changed files with 302 additions and 37 deletions
+9
View File
@@ -9,12 +9,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Security
- Upgraded `python-jose` from 3.3.0 to 3.5.0 to fix CVE: algorithm confusion vulnerability with OpenSSH ECDSA keys (affected versions < 3.4.0).
- Upgraded `python-jose[cryptography]` from `3.3.0` to `3.4.0` to fix an algorithm-confusion vulnerability with OpenSSH ECDSA keys (CVE affects all versions < 3.4.0).
### Added
- **Gmail Debug Email**: New "Send Debug Email" button in the Gmail API settings section. When clicked, it injects a test email into the user's Gmail inbox via the Gmail API. The message appears to be from `christian@docuelevate.org`, includes the current date in the subject line, and is automatically labelled with `test` and `imported` (labels are created on first use) and placed in the inbox. Useful for verifying end-to-end Gmail API delivery without requiring a full mail-account polling cycle.
- `GmailService.get_or_create_label()` async method: lists the user's Gmail labels and returns the matching label ID, creating the label if it does not yet exist.
- `GmailService.inject_debug_email()` async method: builds a properly formatted RFC 2822 test message and calls `inject_email()` with the INBOX, `test`, and `imported` label IDs.
- `POST /providers/gmail/debug-email` backend endpoint: requires a valid Gmail credential, injects the debug email, and persists any auto-refreshed access token.
- `gmailApi.sendDebugEmail()` frontend API helper and `GmailDebugEmailResponse` TypeScript interface.
- **Unified Google OAuth flow**: Google Sign-In now requests all Gmail API scopes (`gmail.insert`, `gmail.labels`, `gmail.readonly`) in the same consent screen, so users no longer need a separate "Connect Gmail" step after signing in with Google. Gmail credentials are stored automatically on successful sign-in.
- `include_granted_scopes=true` added to both the login and Gmail authorize URLs so scope additions take effect for users who previously connected.
### Changed
- `providers.py` now imports both `encrypt_credential` and `decrypt_credential` from `app.core.security`.
- `gmail_service.py` now imports `textwrap`, `MIMEText`, `format_datetime`, and `datetime`/`timezone` for the debug email builder.
- `GMAIL_API_SCOPES` (providers endpoint) and `GMAIL_SCOPES` (GmailService) now include `gmail.readonly`, required for `users().getProfile()` access verification (fixes 403 insufficientPermissions errors).
- Google Sign-In authorize URL (`GET /auth/google/authorize-url`) now requests all six scopes with `access_type=offline`, `prompt=consent`, and `include_granted_scopes=true` so a refresh token is always issued.
- Gmail "Connect Gmail" button in Settings now redirects to `/auth/callback?state=gmail_connect` instead of the dedicated `/auth/gmail-callback` page, reducing the number of redirect URIs that must be registered in Google Cloud Console to one (`{origin}/auth/callback`).
@@ -24,6 +32,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Fixed mailbox edit form: username field was marked `required` but was never pre-populated (the backend intentionally excludes credentials from responses), making it impossible to save edits without re-entering the username. The backend now returns `username` in `MailAccountResponse` so the edit form can pre-populate it. All connection fields (protocol, host, port, use\_ssl, username) are now fully editable in edit mode. The Auto-Detect button is also shown in edit mode to re-detect server settings after a protocol change.
- Fixed mailbox edit form silently overwriting stored credentials with an empty string: when the password field was left blank during an edit the frontend sent `password: ""`, which the backend encrypted and stored, locking the user out. The frontend now only includes `password` in the update payload when it is non-empty, and the backend additionally guards against empty-string passwords.
- Fixed mailbox edit form sending all `MailAccountCreate` fields (including immutable ones like `username`, `host`, `port`) on update requests. The submit handler now builds a `MailAccountUpdate` payload containing all editable fields; `MailAccountUpdate` now covers every field in `MailAccountBase` (protocol, host, port, use\_ssl, use\_tls, username, email\_address, and the previously supported subset).
- Fixed `Exception terminating connection` error logged by Celery workers after every task run. The error was caused by `asyncio.run()` closing the event loop while the asyncpg connection pool still held open idle connections. The fix calls `await engine.dispose()` inside the task's `_run()` coroutine (within the same event loop) so all pooled connections are closed cleanly before the loop is torn down.
- Gmail API `verify_access()` returning 403 for tokens that lacked a read-capable scope: added `gmail.readonly` to all scope lists.
- Fixed three ESLint errors that caused CI to fail: removed unused `_setUser` store binding and unused `useAuthStore` import from `login/page.tsx`; replaced unused `_err` catch binding with a bare `catch {}` in `login/page.tsx`; removed a `useEffect` in `settings/page.tsx` that called `setProfileForm` synchronously (flagged by `react-hooks/set-state-in-effect`) — the effect was redundant because `useState` already initialises the form from the auth store's `user` object, which is the same value passed as `initialData` to `useQuery`.
+73 -2
View File
@@ -11,7 +11,7 @@ import logging
from app.core.database import get_db
from app.core.deps import get_current_active_user
from app.core.security import encrypt_credential
from app.core.security import encrypt_credential, decrypt_credential
from app.core.config import settings
from app.models.database_models import User, GmailCredential
from app.models.schemas import (
@@ -22,7 +22,7 @@ from app.models.schemas import (
GmailAuthorizeResponse,
GmailCallbackRequest,
)
from app.services.gmail_service import GmailService, GMAIL_SCOPES
from app.services.gmail_service import GmailService, GmailInjectionError, GMAIL_SCOPES
router = APIRouter()
logger = logging.getLogger(__name__)
@@ -319,6 +319,77 @@ async def get_gmail_authorize_url(
return GmailAuthorizeResponse(authorization_url=url)
@router.post("/gmail/debug-email", status_code=status.HTTP_200_OK)
async def send_gmail_debug_email(
current_user: User = Depends(get_current_active_user),
db: AsyncSession = Depends(get_db),
):
"""
Inject a debug/test email into the current user's Gmail inbox.
The message appears to have been sent by christian@docuelevate.org,
carries today's date in the subject, and is tagged with the custom
labels "test" and "imported" as well as placed in the inbox.
Useful for verifying that Gmail API delivery is working end-to-end
without requiring an active mail-account polling cycle.
"""
result = await db.execute(
select(GmailCredential).where(
GmailCredential.user_id == current_user.id,
GmailCredential.is_valid == True, # noqa: E712
)
)
credential = result.scalar_one_or_none()
if not credential:
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
detail="No valid Gmail credentials found. Connect Gmail first.",
)
access_token = decrypt_credential(credential.encrypted_access_token) # type: ignore[arg-type]
refresh_token = (
decrypt_credential(credential.encrypted_refresh_token) # type: ignore[arg-type]
if credential.encrypted_refresh_token
else None
)
gmail_service = GmailService(
access_token=access_token,
refresh_token=refresh_token,
client_id=settings.GOOGLE_CLIENT_ID,
client_secret=settings.GOOGLE_CLIENT_SECRET,
)
try:
inject_result = await gmail_service.inject_debug_email(
recipient_email=credential.gmail_email, # type: ignore[arg-type]
)
except GmailInjectionError as exc:
raise HTTPException(
status_code=status.HTTP_502_BAD_GATEWAY,
detail=f"Gmail injection failed: {exc}",
)
# Persist refreshed token if the google-auth library renewed it
refreshed = gmail_service.get_refreshed_token()
if refreshed:
credential.encrypted_access_token = encrypt_credential( # type: ignore[assignment]
refreshed["access_token"]
)
if refreshed.get("expiry"):
credential.token_expiry = refreshed["expiry"] # type: ignore[assignment]
await db.commit()
return {
"message": "Debug email injected successfully",
"message_id": inject_result.get("message_id"),
"thread_id": inject_result.get("thread_id"),
"label_ids": inject_result.get("label_ids", []),
}
@router.post(
"/gmail/callback",
response_model=GmailCredentialResponse,
+117
View File
@@ -9,6 +9,10 @@ This is preferred over SMTP forwarding as it doesn't modify the email.
import asyncio
import base64
import logging
import textwrap
from datetime import datetime, timezone
from email.mime.text import MIMEText
from email.utils import format_datetime
from typing import Optional, Dict, Any
from google.oauth2.credentials import Credentials
@@ -184,6 +188,119 @@ class GmailService:
logger.error(f"Failed to get Gmail email address: {e}")
return None
async def get_or_create_label(self, name: str) -> str:
"""
Return the Gmail label ID for a label with the given name.
Lists the user's existing labels and returns the ID of the first
match (case-insensitive). If no matching label is found, a new
label is created and its ID is returned.
Args:
name: Human-readable label name (e.g. "test", "imported").
Returns:
Gmail label ID string (e.g. "Label_1234567890").
Raises:
GmailInjectionError: If the Gmail API call fails.
"""
loop = asyncio.get_event_loop()
try:
labels_resp = await loop.run_in_executor(
None,
lambda: self.service.users().labels().list(userId="me").execute(),
)
for label in labels_resp.get("labels", []):
if label.get("name", "").lower() == name.lower():
return label["id"]
# Label not found create it
created = await loop.run_in_executor(
None,
lambda: self.service.users()
.labels()
.create(userId="me", body={"name": name})
.execute(),
)
logger.info(f"Created Gmail label '{name}' with id={created['id']}")
return created["id"]
except HttpError as e:
error_msg = f"Gmail API error while managing label '{name}': {e.reason if hasattr(e, 'reason') else str(e)}"
logger.error(error_msg)
raise GmailInjectionError(error_msg)
except Exception as e:
error_msg = f"Failed to get/create Gmail label '{name}': {str(e)}"
logger.error(error_msg)
raise GmailInjectionError(error_msg)
async def inject_debug_email(
self,
recipient_email: str,
) -> Dict[str, Any]:
"""
Inject a debug/test email into the user's Gmail inbox.
The message is made to appear as if it was sent by
christian@docuelevate.org on the current date. It is placed in
the inbox and tagged with the custom labels "test" and "imported"
so it is easy to identify and clean up.
Args:
recipient_email: The Gmail address to deliver the message to
(the authenticated user's address).
Returns:
Dict with message_id, thread_id, and label_ids.
Raises:
GmailInjectionError: If injection or label management fails.
"""
now = datetime.now(timezone.utc)
date_str = now.strftime("%d %B %Y") # e.g. "25 March 2026"
subject = f"Test Import {date_str}"
body = textwrap.dedent(f"""\
Hi there,
This is an automated test message injected via the Gmail API to
confirm that the import pipeline is working correctly.
Date: {date_str}
Source: DocuElevate Integration Test
If you can see this message in your inbox it means that Gmail API
delivery is functioning as expected. Feel free to delete it.
Best regards,
Christian Loris
DocuElevate
""")
msg = MIMEText(body, "plain", "utf-8")
msg["From"] = "Christian Loris <christian@docuelevate.org>"
msg["To"] = recipient_email
msg["Subject"] = subject
msg["Date"] = format_datetime(now)
msg["Message-ID"] = f"<debug-{now.strftime('%Y%m%d%H%M%S')}@docuelevate.org>"
raw_bytes = msg.as_bytes()
# Resolve label IDs (create labels if they don't exist yet)
test_label_id = await self.get_or_create_label("test")
imported_label_id = await self.get_or_create_label("imported")
label_ids = ["INBOX", test_label_id, imported_label_id]
return await self.inject_email(
raw_email=raw_bytes,
label_ids=label_ids,
source_account_name="debug",
)
def get_refreshed_token(self) -> Optional[Dict[str, Any]]:
"""
Return the current access token and expiry if the token was refreshed
+15 -3
View File
@@ -8,7 +8,7 @@ from celery import Task
import logging
from app.workers.celery_app import celery_app
from app.core.database import async_session_maker
from app.core.database import async_session_maker, engine
from app.core.security import decrypt_credential, encrypt_credential
from app.models.database_models import (
MailAccount,
@@ -34,8 +34,20 @@ class AsyncTask(Task):
def __call__(self, *args, **kwargs):
"""Run async task in event loop"""
# Use asyncio.run() for better event loop management
return asyncio.run(self.run(*args, **kwargs))
async def _run():
try:
return await self.run(*args, **kwargs)
finally:
# Dispose the connection pool before the event loop closes.
# Each asyncio.run() creates a fresh event loop; if pooled
# asyncpg connections are still open when the loop is torn
# down, asyncpg raises "Exception terminating connection".
# Disposing the engine here closes those connections cleanly
# inside the same loop, before asyncio.run() shuts it down.
await engine.dispose()
return asyncio.run(_run())
@celery_app.task(base=AsyncTask, name="app.workers.tasks.process_mail_account")
+1
View File
@@ -11,6 +11,7 @@ psycopg2-binary==2.9.11
asyncpg==0.31.0
# Authentication
python-jose[cryptography]==3.5.0 # Updated: Fixed algorithm confusion with OpenSSH ECDSA keys (was 3.3.0)
bcrypt==4.3.0
python-multipart==0.0.22 # Updated: Fixed multiple vulnerabilities (was 0.0.6)
+2
View File
@@ -17,6 +17,7 @@ Comprehensive task breakdown for repository improvements and production readines
- [ ] Enable rate limiting per user/tier
- [x] Fix bare exception handlers throughout codebase
- [x] Update datetime usage to timezone-aware (`DateTime(timezone=True)` columns and `lambda: datetime.now(timezone.utc)` defaults; fixes `DBAPIError` from asyncpg on timezone-naive columns)
- [x] Fix `Exception terminating connection` in Celery workers: call `await engine.dispose()` inside task coroutine so pooled asyncpg connections are closed before the event loop is torn down
- [ ] Validate redirect_uri to prevent open redirect vulnerabilities
- [ ] Add per-user random salt for encryption (currently deterministic)
@@ -185,6 +186,7 @@ Comprehensive task breakdown for repository improvements and production readines
- [x] Gmail API one-click OAuth grant flow with token refresh and revocation handling
- [x] Unified Google OAuth flow: sign-in requests all Gmail scopes; single `/auth/callback` redirect URI needed in Google Console
- [x] Message deduplication (POP3 UIDL + IMAP \Seen flag + DB tracking)
- [x] **Debug email**: "Send Debug Email" button in Settings injects a test message (from christian@docuelevate.org, dated today, labelled `test` + `imported`, placed in inbox) to verify end-to-end Gmail API delivery
- [ ] Implement GDPR data export endpoint
- [ ] Complete notification service integration (Apprise)
- [ ] Add advanced email filtering
+6 -6
View File
@@ -5625,9 +5625,9 @@
"license": "ISC"
},
"node_modules/picomatch": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
"version": "2.3.2",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==",
"dev": true,
"license": "MIT",
"engines": {
@@ -6432,9 +6432,9 @@
}
},
"node_modules/tinyglobby/node_modules/picomatch": {
"version": "4.0.3",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz",
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
"dev": true,
"license": "MIT",
"engines": {
+66 -26
View File
@@ -15,6 +15,7 @@ import {
Server,
AlertTriangle,
XCircle,
Bug,
} from 'lucide-react';
export default function SettingsPage() {
@@ -104,6 +105,19 @@ function SettingsContent() {
},
});
const [debugEmailResult, setDebugEmailResult] = useState<string | null>(null);
const sendDebugEmailMutation = useMutation({
mutationFn: gmailApi.sendDebugEmail,
onSuccess: () => {
setDebugEmailResult('success');
setTimeout(() => setDebugEmailResult(null), 5000);
},
onError: () => {
setDebugEmailResult('error');
setTimeout(() => setDebugEmailResult(null), 5000);
},
});
const saveSmtpMutation = useMutation({
mutationFn: smtpApi.save,
onSuccess: () => {
@@ -302,35 +316,61 @@ function SettingsContent() {
)}
{!gmailLoading && gmailConnected && (
<div className="flex items-center justify-between flex-wrap gap-4">
<div className="flex items-center gap-2">
<CheckCircle className="h-5 w-5 text-green-500" />
<div>
<p className="text-sm font-medium text-gray-900">
Connected as <span className="font-semibold">{gmailCredential.gmail_email}</span>
</p>
{gmailCredential.last_verified_at && (
<p className="text-xs text-gray-500">
Last verified: {new Date(gmailCredential.last_verified_at).toLocaleString()}
<div className="flex flex-col gap-4">
<div className="flex items-center justify-between flex-wrap gap-4">
<div className="flex items-center gap-2">
<CheckCircle className="h-5 w-5 text-green-500" />
<div>
<p className="text-sm font-medium text-gray-900">
Connected as <span className="font-semibold">{gmailCredential.gmail_email}</span>
</p>
)}
{gmailCredential.last_verified_at && (
<p className="text-xs text-gray-500">
Last verified: {new Date(gmailCredential.last_verified_at).toLocaleString()}
</p>
)}
</div>
</div>
<div className="flex gap-2 flex-wrap">
<button
onClick={() => sendDebugEmailMutation.mutate()}
disabled={sendDebugEmailMutation.isPending}
title="Inject a test email into your Gmail inbox"
className="flex items-center gap-1.5 px-4 py-2 text-sm bg-amber-50 text-amber-700 rounded-md hover:bg-amber-100 transition-colors disabled:opacity-50"
>
{sendDebugEmailMutation.isPending ? (
<><Loader2 className="h-4 w-4 animate-spin" />Sending</>
) : (
<><Bug className="h-4 w-4" />Send Debug Email</>
)}
</button>
<button
onClick={handleConnectGmail}
className="px-4 py-2 text-sm bg-blue-50 text-blue-700 rounded-md hover:bg-blue-100 transition-colors"
>
Re-authorise
</button>
<button
onClick={handleDisconnectGmail}
disabled={disconnectGmailMutation.isPending}
className="px-4 py-2 text-sm bg-red-50 text-red-700 rounded-md hover:bg-red-100 transition-colors disabled:opacity-50"
>
Disconnect
</button>
</div>
</div>
<div className="flex gap-2">
<button
onClick={handleConnectGmail}
className="px-4 py-2 text-sm bg-blue-50 text-blue-700 rounded-md hover:bg-blue-100 transition-colors"
>
Re-authorise
</button>
<button
onClick={handleDisconnectGmail}
disabled={disconnectGmailMutation.isPending}
className="px-4 py-2 text-sm bg-red-50 text-red-700 rounded-md hover:bg-red-100 transition-colors disabled:opacity-50"
>
Disconnect
</button>
</div>
{debugEmailResult === 'success' && (
<div className="flex items-center gap-2 text-sm text-green-700 bg-green-50 border border-green-200 rounded-md px-3 py-2">
<CheckCircle className="h-4 w-4 flex-shrink-0" />
Debug email injected successfully. Check your Gmail inbox it should be labelled <strong className="mx-1">test</strong> and <strong className="mx-1">imported</strong>.
</div>
)}
{debugEmailResult === 'error' && (
<div className="flex items-center gap-2 text-sm text-red-700 bg-red-50 border border-red-200 rounded-md px-3 py-2">
<AlertTriangle className="h-4 w-4 flex-shrink-0" />
Failed to inject debug email. Check that Gmail API access is still valid.
</div>
)}
</div>
)}
+13
View File
@@ -145,6 +145,13 @@ export interface GmailCredential {
updated_at: string;
}
export interface GmailDebugEmailResponse {
message: string;
message_id: string | null;
thread_id: string | null;
label_ids: string[];
}
export interface UserSmtpConfig {
id: number;
user_id: number;
@@ -322,6 +329,12 @@ export const gmailApi = {
async disconnect(): Promise<void> {
await api.delete('/providers/gmail-credential');
},
/** Inject a debug test email into the user's Gmail inbox. */
async sendDebugEmail(): Promise<GmailDebugEmailResponse> {
const response = await api.post<GmailDebugEmailResponse>('/providers/gmail/debug-email');
return response.data;
},
};
// ── SMTP Config API ─────────────────────────────────────────────────────