fix: merge main, address code review feedback for security fix PR #816
- Merge origin/main into branch (resolve conflict in integrations_dashboard.html) - Add defensive JSON parsing with try/except for integration.config - Wrap tester() call in try/except to prevent 500 errors from bad config - Add i18n key integrations.connection_test_failed_fallback in en.json - Reference i18n key in template JS fallback message - Update SECURITY_AUDIT.md: add fix date (2026-03-23), update doc date - Remove accidental revert.sh file - Fix missing MagicMock/patch imports in test file - Add tests for invalid JSON config and tester exception error paths Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com> Agent-Logs-Url: https://github.com/christianlouis/DocuElevate/sessions/daebb70e-059a-4601-8864-88eef49f99cf
This commit is contained in:
@@ -287,6 +287,27 @@ alembic revision --autogenerate -m "describe your change"
|
||||
|
||||
Review the generated file in `migrations/versions/` before applying it.
|
||||
|
||||
> **Tip:** For detailed guidance on naming conventions, idempotent patterns, parallel-branch workflows, and resolving merge conflicts, see the [Migration Workflow Guide](MigrationWorkflow.md).
|
||||
|
||||
### Validating the Migration Chain
|
||||
|
||||
A CI check and pre-commit hook validate that the migration chain has no broken
|
||||
references, duplicate revisions, or diverged heads. Run the check locally:
|
||||
|
||||
```bash
|
||||
python scripts/check_alembic_migrations.py
|
||||
python scripts/check_alembic_migrations.py --verbose # extra detail
|
||||
```
|
||||
|
||||
If you see **"Multiple migration heads detected"**, two branches added
|
||||
migrations from the same parent. Create a merge migration:
|
||||
|
||||
```bash
|
||||
alembic merge heads -m "merge_parallel_branches"
|
||||
```
|
||||
|
||||
For a complete walk-through, see the [Migration Workflow Guide](MigrationWorkflow.md).
|
||||
|
||||
### Automating Migrations in Docker Compose
|
||||
|
||||
Add a short-lived `migrate` service that runs before the API and Worker:
|
||||
@@ -316,18 +337,26 @@ The Helm chart includes a pre-install and pre-upgrade Job hook that runs `alembi
|
||||
|
||||
## Connection Pooling
|
||||
|
||||
SQLAlchemy manages a connection pool automatically. The defaults are suitable for most deployments. For high-concurrency or Kubernetes deployments you may want to tune:
|
||||
SQLAlchemy manages a connection pool automatically. DocuElevate selects the pool
|
||||
strategy based on the database backend:
|
||||
|
||||
- **SQLite** — uses `NullPool` (a fresh connection per request, closed immediately).
|
||||
This avoids the `QueuePool limit reached` `TimeoutError` that can occur under
|
||||
concurrent load because SQLite does not benefit from persistent connection pooling.
|
||||
- **PostgreSQL / MySQL** — uses a bounded `QueuePool` whose size is configurable
|
||||
via environment variables.
|
||||
|
||||
```bash
|
||||
# Optional — these are set via environment variables if you extend app/database.py
|
||||
# Typical production values:
|
||||
DB_POOL_SIZE=10 # Number of persistent connections per worker
|
||||
DB_MAX_OVERFLOW=20 # Additional connections allowed beyond pool_size
|
||||
DB_POOL_TIMEOUT=30 # Seconds to wait for a connection from the pool
|
||||
DB_POOL_RECYCLE=1800 # Recycle connections after 30 minutes (avoids stale connections)
|
||||
# Tune these for PostgreSQL / MySQL (ignored when using SQLite):
|
||||
DB_POOL_SIZE=10 # Number of persistent connections per worker (default: 10)
|
||||
DB_MAX_OVERFLOW=20 # Additional connections allowed beyond pool_size (default: 20)
|
||||
DB_POOL_TIMEOUT=30 # Seconds to wait for a connection from the pool (default: 30)
|
||||
DB_POOL_RECYCLE=1800 # Recycle connections after 30 minutes (default: 1800)
|
||||
```
|
||||
|
||||
> **Note:** These environment variables are not exposed in the default `app/config.py`. If you need to tune them, extend the database engine creation in `app/database.py`.
|
||||
All backends also enable `pool_pre_ping`, which sends a lightweight health-check
|
||||
before each connection is handed out. This detects stale or dropped connections
|
||||
and transparently reconnects.
|
||||
|
||||
For **PgBouncer** (external connection pooling), point `DATABASE_URL` at your PgBouncer instance and use transaction-mode pooling:
|
||||
|
||||
@@ -491,4 +520,13 @@ Then retry `alembic upgrade head`.
|
||||
|
||||
Either increase `max_connections` in `postgresql.conf` or add PgBouncer in front of PostgreSQL. The default PostgreSQL `max_connections` is `100`; reduce `DB_POOL_SIZE` per worker to stay within this limit.
|
||||
|
||||
### "QueuePool limit reached" TimeoutError (SQLite)
|
||||
|
||||
If you see `TimeoutError: QueuePool limit of size 5 overflow 10 reached`, your
|
||||
deployment is still running an older version of DocuElevate that used a bounded
|
||||
connection pool for SQLite. Upgrade to the latest release — SQLite now uses
|
||||
`NullPool`, which eliminates this error entirely. If you are already on the
|
||||
latest version and are still seeing pool exhaustion, ensure you are not
|
||||
overriding the engine creation manually.
|
||||
|
||||
For more help, see the [Troubleshooting Guide](Troubleshooting.md).
|
||||
|
||||
Reference in New Issue
Block a user