gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Adjust rate limits per feedback - increase uploads to 600/minute, remove processing limit

Browse Source 
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-02-10 20:29:26 +00:00
parent 7480c70de4
commit 403cd02428
5 changed files with 28 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.demo
+6 -7
View File
@@ -59,19 +59,18 @@ RATE_LIMIT_DEFAULT=100/minute
# Rate limit for file upload endpoints
# Allows faster uploads while still preventing abuse
# Default: 10 uploads per second per IP/user (600/minute)
RATE_LIMIT_UPLOAD=10/second
# Rate limit for document processing endpoints (OCR, metadata extraction)
# These operations are resource-intensive
# Default: 30 requests per minute per IP/user
RATE_LIMIT_PROCESS=30/minute
# Default: 600 uploads per minute per IP/user
RATE_LIMIT_UPLOAD=600/minute
# Rate limit for authentication endpoints
# Strict limit to prevent brute force attacks
# Default: 10 attempts per minute per IP
RATE_LIMIT_AUTH=10/minute
# Note: Processing endpoints (OCR, metadata extraction) use built-in queue throttling
# via Celery task queue to control processing rates and prevent upstream API overloads.
# No additional API-level rate limit is needed for processing endpoints.
# **Authentication**
AUTH_ENABLED=true
# Generate a secure random string, for example:
app/config.py
+1 -5
View File
@@ -224,13 +224,9 @@ class Settings(BaseSettings):
description="Default rate limit for all endpoints (format: 'count/period', e.g., '100/minute', '1000/hour').",
)
rate_limit_upload: str = Field(
default="10/second",
default="600/minute",
description="Rate limit for file upload endpoints to prevent resource exhaustion.",
)
rate_limit_process: str = Field(
default="30/minute",
description="Rate limit for document processing endpoints (OCR, metadata extraction).",
)
rate_limit_auth: str = Field(
default="10/minute",
description="Stricter rate limit for authentication endpoints to prevent brute force attacks.",
docs/API.md
+4 -4
View File
@@ -24,10 +24,11 @@ DocuElevate implements rate limiting to protect against abuse and DoS attacks. R
### Default Limits
- **Default endpoints**: 100 requests per minute
- **File upload**: 20 requests per minute
- **Document processing**: 30 requests per minute
- **File upload**: 600 requests per minute
- **Authentication**: 10 requests per minute
**Note**: Document processing endpoints (OCR, metadata extraction) use built-in queue throttling to control processing rates and prevent upstream API overloads. No additional API-level rate limit is applied to processing endpoints.
### Rate Limit Headers
When a rate limit is exceeded, the API returns a `429 Too Many Requests` response:
@@ -47,8 +48,7 @@ Rate limits can be configured via environment variables:
```bash
RATE_LIMITING_ENABLED=true
RATE_LIMIT_DEFAULT=100/minute
RATE_LIMIT_UPLOAD=20/minute
RATE_LIMIT_PROCESS=30/minute
RATE_LIMIT_UPLOAD=600/minute
RATE_LIMIT_AUTH=10/minute
```
docs/ConfigurationGuide.md
+7 -10
View File
@@ -121,10 +121,11 @@ Rate limits are specified in the format `count/period`, where:
| **Variable** | **Description** | **Default** | **Applies To** |
|------------------------|----------------------------------------------------------------------|------------------|-----------------------------------------|
| `RATE_LIMIT_DEFAULT` | Default rate limit for all API endpoints | `100/minute` | Most API endpoints |
| `RATE_LIMIT_UPLOAD` | Rate limit for file upload endpoints (prevents resource exhaustion) | `10/second` | `/api/ui-upload` and similar |
| `RATE_LIMIT_PROCESS` | Rate limit for processing endpoints (OCR, metadata extraction) | `30/minute` | `/api/process`, OCR endpoints |
| `RATE_LIMIT_UPLOAD` | Rate limit for file upload endpoints (prevents resource exhaustion) | `600/minute` | `/api/ui-upload` and similar |
| `RATE_LIMIT_AUTH` | Stricter rate limit for authentication (prevents brute force) | `10/minute` | Login, authentication endpoints |
**Note**: Processing endpoints (OCR, metadata extraction) use built-in queue throttling via Celery to control processing rates and prevent upstream API overloads. No additional API-level rate limit is configured for processing endpoints.
#### How Rate Limiting Works
1. **Per-User Tracking**: For authenticated requests, limits are enforced per user ID
@@ -144,8 +145,7 @@ REDIS_URL=redis://redis:6379/0
# Customize rate limits
RATE_LIMIT_DEFAULT=100/minute # 100 requests per minute per user/IP
RATE_LIMIT_UPLOAD=20/minute # 20 uploads per minute
RATE_LIMIT_PROCESS=30/minute # 30 processing requests per minute
RATE_LIMIT_UPLOAD=600/minute # 600 uploads per minute
RATE_LIMIT_AUTH=10/minute # 10 auth attempts per minute (brute force protection)
```
@@ -154,24 +154,21 @@ RATE_LIMIT_AUTH=10/minute # 10 auth attempts per minute (brute force pro
**Small Deployment (1-10 users)**:
```bash
RATE_LIMIT_DEFAULT=200/minute
RATE_LIMIT_UPLOAD=50/minute
RATE_LIMIT_PROCESS=50/minute
RATE_LIMIT_UPLOAD=1200/minute
RATE_LIMIT_AUTH=20/minute
```
**Medium Deployment (10-100 users)**:
```bash
RATE_LIMIT_DEFAULT=100/minute
RATE_LIMIT_UPLOAD=20/minute
RATE_LIMIT_PROCESS=30/minute
RATE_LIMIT_UPLOAD=600/minute
RATE_LIMIT_AUTH=10/minute
```
**Large Deployment (100+ users)**:
```bash
RATE_LIMIT_DEFAULT=50/minute
RATE_LIMIT_UPLOAD=10/minute
RATE_LIMIT_PROCESS=15/minute
RATE_LIMIT_UPLOAD=300/minute
RATE_LIMIT_AUTH=5/minute
```
docs/RateLimitingStrategy.md
+10 -40
View File
@@ -11,10 +11,11 @@ DocuElevate implements rate limiting using [SlowAPI](https://github.com/laurents
All API endpoints are protected with a default rate limit unless explicitly exempted or configured otherwise:
- **Default**: 100 requests per minute per IP/user
- **File Upload**: 20 requests per minute (resource-intensive)
- **Document Processing**: 30 requests per minute (CPU/API-intensive)
- **File Upload**: 600 requests per minute per IP/user
- **Authentication**: 10 requests per minute (brute force protection)
**Note**: Document processing endpoints use built-in queue throttling via Celery to control processing rates and prevent upstream API overloads. No additional API-level rate limit is configured for processing endpoints.
## Endpoint Categories
### 1. Authentication Endpoints (Stricter Limits)
@@ -35,52 +36,21 @@ All API endpoints are protected with a default rate limit unless explicitly exem
### 2. File Upload Endpoints (Resource Protection)
**Rate Limit**: 20 requests per minute
**Rate Limit**: 600 requests per minute
**Endpoints**:
- `POST /api/ui-upload` - Web UI file upload
- `POST /api/upload` - API file upload
**Rationale**: File uploads are resource-intensive operations that consume:
- Network bandwidth
- Disk I/O
- Processing time
- Storage space
**Rationale**: File uploads consume network bandwidth, disk I/O, and storage space. A limit of 600 uploads per minute allows fast batch uploads while preventing resource exhaustion and abuse.
Limiting to 20 uploads per minute prevents resource exhaustion while allowing legitimate batch uploads.
**Implementation Status**: Configured via `RATE_LIMIT_UPLOAD` (default: `600/minute`)
**Implementation Status**: Configured via `RATE_LIMIT_UPLOAD` (default: `20/minute`)
---
### 3. Document Processing Endpoints (API/CPU Protection)
**Rate Limit**: 30 requests per minute
**Endpoints**:
- `POST /api/process/` - Trigger document processing
- `POST /api/send_to_dropbox/` - Send to Dropbox
- `POST /api/send_to_google_drive/` - Send to Google Drive
- `POST /api/send_to_onedrive/` - Send to OneDrive
- `POST /api/send_to_paperless/` - Send to Paperless
- `POST /api/send_to_all/` - Send to all destinations
- `POST /api/processall/` - Batch process all files
- `POST /api/azure/process/` - Azure OCR processing
- `POST /api/openai/extract-metadata/` - OpenAI metadata extraction
**Rationale**: These endpoints trigger:
- External API calls (OpenAI, Azure)
- CPU-intensive operations (OCR, PDF processing)
- Network requests to cloud storage
- Background Celery tasks
A limit of 30 requests per minute protects both internal resources and prevents excessive API costs from external services.
**Implementation Status**: Configured via `RATE_LIMIT_PROCESS` (default: `30/minute`)
---
### 4. Read-Only API Endpoints (Default Limits)
### 3. Read-Only API Endpoints (Default Limits)
**Rate Limit**: 100 requests per minute
@@ -99,7 +69,7 @@ A limit of 30 requests per minute protects both internal resources and prevents
---
### 5. Frontend Routes (Default Limits)
### 4. Frontend Routes (Default Limits)
**Rate Limit**: 100 requests per minute
@@ -116,7 +86,7 @@ A limit of 30 requests per minute protects both internal resources and prevents
---
### 6. Webhook/Callback Endpoints (Higher Limits)
### 5. Webhook/Callback Endpoints (Higher Limits)
**Rate Limit**: Consider exemption or very high limits
@@ -129,7 +99,7 @@ A limit of 30 requests per minute protects both internal resources and prevents
---
### 7. Health/Diagnostic Endpoints (Exempt or High Limits)
### 6. Health/Diagnostic Endpoints (Exempt or High Limits)
**Rate Limit**: Potentially exempt for monitoring