diff --git a/.env.demo b/.env.demo index 5f69b5f0..5351c11a 100644 --- a/.env.demo +++ b/.env.demo @@ -59,19 +59,18 @@ RATE_LIMIT_DEFAULT=100/minute # Rate limit for file upload endpoints # Allows faster uploads while still preventing abuse -# Default: 10 uploads per second per IP/user (600/minute) -RATE_LIMIT_UPLOAD=10/second - -# Rate limit for document processing endpoints (OCR, metadata extraction) -# These operations are resource-intensive -# Default: 30 requests per minute per IP/user -RATE_LIMIT_PROCESS=30/minute +# Default: 600 uploads per minute per IP/user +RATE_LIMIT_UPLOAD=600/minute # Rate limit for authentication endpoints # Strict limit to prevent brute force attacks # Default: 10 attempts per minute per IP RATE_LIMIT_AUTH=10/minute +# Note: Processing endpoints (OCR, metadata extraction) use built-in queue throttling +# via Celery task queue to control processing rates and prevent upstream API overloads. +# No additional API-level rate limit is needed for processing endpoints. + # **Authentication** AUTH_ENABLED=true # Generate a secure random string, for example: diff --git a/app/config.py b/app/config.py index 12af0fe3..4db06fed 100644 --- a/app/config.py +++ b/app/config.py @@ -224,13 +224,9 @@ class Settings(BaseSettings): description="Default rate limit for all endpoints (format: 'count/period', e.g., '100/minute', '1000/hour').", ) rate_limit_upload: str = Field( - default="10/second", + default="600/minute", description="Rate limit for file upload endpoints to prevent resource exhaustion.", ) - rate_limit_process: str = Field( - default="30/minute", - description="Rate limit for document processing endpoints (OCR, metadata extraction).", - ) rate_limit_auth: str = Field( default="10/minute", description="Stricter rate limit for authentication endpoints to prevent brute force attacks.", diff --git a/docs/API.md b/docs/API.md index 3c038b1a..ea7d00bd 100644 --- a/docs/API.md +++ b/docs/API.md @@ -24,10 +24,11 @@ DocuElevate implements rate limiting to protect against abuse and DoS attacks. R ### Default Limits - **Default endpoints**: 100 requests per minute -- **File upload**: 20 requests per minute -- **Document processing**: 30 requests per minute +- **File upload**: 600 requests per minute - **Authentication**: 10 requests per minute +**Note**: Document processing endpoints (OCR, metadata extraction) use built-in queue throttling to control processing rates and prevent upstream API overloads. No additional API-level rate limit is applied to processing endpoints. + ### Rate Limit Headers When a rate limit is exceeded, the API returns a `429 Too Many Requests` response: @@ -47,8 +48,7 @@ Rate limits can be configured via environment variables: ```bash RATE_LIMITING_ENABLED=true RATE_LIMIT_DEFAULT=100/minute -RATE_LIMIT_UPLOAD=20/minute -RATE_LIMIT_PROCESS=30/minute +RATE_LIMIT_UPLOAD=600/minute RATE_LIMIT_AUTH=10/minute ``` diff --git a/docs/ConfigurationGuide.md b/docs/ConfigurationGuide.md index f31c6976..76ce9bb7 100644 --- a/docs/ConfigurationGuide.md +++ b/docs/ConfigurationGuide.md @@ -121,10 +121,11 @@ Rate limits are specified in the format `count/period`, where: | **Variable** | **Description** | **Default** | **Applies To** | |------------------------|----------------------------------------------------------------------|------------------|-----------------------------------------| | `RATE_LIMIT_DEFAULT` | Default rate limit for all API endpoints | `100/minute` | Most API endpoints | -| `RATE_LIMIT_UPLOAD` | Rate limit for file upload endpoints (prevents resource exhaustion) | `10/second` | `/api/ui-upload` and similar | -| `RATE_LIMIT_PROCESS` | Rate limit for processing endpoints (OCR, metadata extraction) | `30/minute` | `/api/process`, OCR endpoints | +| `RATE_LIMIT_UPLOAD` | Rate limit for file upload endpoints (prevents resource exhaustion) | `600/minute` | `/api/ui-upload` and similar | | `RATE_LIMIT_AUTH` | Stricter rate limit for authentication (prevents brute force) | `10/minute` | Login, authentication endpoints | +**Note**: Processing endpoints (OCR, metadata extraction) use built-in queue throttling via Celery to control processing rates and prevent upstream API overloads. No additional API-level rate limit is configured for processing endpoints. + #### How Rate Limiting Works 1. **Per-User Tracking**: For authenticated requests, limits are enforced per user ID @@ -144,8 +145,7 @@ REDIS_URL=redis://redis:6379/0 # Customize rate limits RATE_LIMIT_DEFAULT=100/minute # 100 requests per minute per user/IP -RATE_LIMIT_UPLOAD=20/minute # 20 uploads per minute -RATE_LIMIT_PROCESS=30/minute # 30 processing requests per minute +RATE_LIMIT_UPLOAD=600/minute # 600 uploads per minute RATE_LIMIT_AUTH=10/minute # 10 auth attempts per minute (brute force protection) ``` @@ -154,24 +154,21 @@ RATE_LIMIT_AUTH=10/minute # 10 auth attempts per minute (brute force pro **Small Deployment (1-10 users)**: ```bash RATE_LIMIT_DEFAULT=200/minute -RATE_LIMIT_UPLOAD=50/minute -RATE_LIMIT_PROCESS=50/minute +RATE_LIMIT_UPLOAD=1200/minute RATE_LIMIT_AUTH=20/minute ``` **Medium Deployment (10-100 users)**: ```bash RATE_LIMIT_DEFAULT=100/minute -RATE_LIMIT_UPLOAD=20/minute -RATE_LIMIT_PROCESS=30/minute +RATE_LIMIT_UPLOAD=600/minute RATE_LIMIT_AUTH=10/minute ``` **Large Deployment (100+ users)**: ```bash RATE_LIMIT_DEFAULT=50/minute -RATE_LIMIT_UPLOAD=10/minute -RATE_LIMIT_PROCESS=15/minute +RATE_LIMIT_UPLOAD=300/minute RATE_LIMIT_AUTH=5/minute ``` diff --git a/docs/RateLimitingStrategy.md b/docs/RateLimitingStrategy.md index b6c49a76..80def7e4 100644 --- a/docs/RateLimitingStrategy.md +++ b/docs/RateLimitingStrategy.md @@ -11,10 +11,11 @@ DocuElevate implements rate limiting using [SlowAPI](https://github.com/laurents All API endpoints are protected with a default rate limit unless explicitly exempted or configured otherwise: - **Default**: 100 requests per minute per IP/user -- **File Upload**: 20 requests per minute (resource-intensive) -- **Document Processing**: 30 requests per minute (CPU/API-intensive) +- **File Upload**: 600 requests per minute per IP/user - **Authentication**: 10 requests per minute (brute force protection) +**Note**: Document processing endpoints use built-in queue throttling via Celery to control processing rates and prevent upstream API overloads. No additional API-level rate limit is configured for processing endpoints. + ## Endpoint Categories ### 1. Authentication Endpoints (Stricter Limits) @@ -35,52 +36,21 @@ All API endpoints are protected with a default rate limit unless explicitly exem ### 2. File Upload Endpoints (Resource Protection) -**Rate Limit**: 20 requests per minute +**Rate Limit**: 600 requests per minute **Endpoints**: - `POST /api/ui-upload` - Web UI file upload - `POST /api/upload` - API file upload -**Rationale**: File uploads are resource-intensive operations that consume: -- Network bandwidth -- Disk I/O -- Processing time -- Storage space +**Rationale**: File uploads consume network bandwidth, disk I/O, and storage space. A limit of 600 uploads per minute allows fast batch uploads while preventing resource exhaustion and abuse. -Limiting to 20 uploads per minute prevents resource exhaustion while allowing legitimate batch uploads. +**Implementation Status**: Configured via `RATE_LIMIT_UPLOAD` (default: `600/minute`) **Implementation Status**: Configured via `RATE_LIMIT_UPLOAD` (default: `20/minute`) --- -### 3. Document Processing Endpoints (API/CPU Protection) - -**Rate Limit**: 30 requests per minute - -**Endpoints**: -- `POST /api/process/` - Trigger document processing -- `POST /api/send_to_dropbox/` - Send to Dropbox -- `POST /api/send_to_google_drive/` - Send to Google Drive -- `POST /api/send_to_onedrive/` - Send to OneDrive -- `POST /api/send_to_paperless/` - Send to Paperless -- `POST /api/send_to_all/` - Send to all destinations -- `POST /api/processall/` - Batch process all files -- `POST /api/azure/process/` - Azure OCR processing -- `POST /api/openai/extract-metadata/` - OpenAI metadata extraction - -**Rationale**: These endpoints trigger: -- External API calls (OpenAI, Azure) -- CPU-intensive operations (OCR, PDF processing) -- Network requests to cloud storage -- Background Celery tasks - -A limit of 30 requests per minute protects both internal resources and prevents excessive API costs from external services. - -**Implementation Status**: Configured via `RATE_LIMIT_PROCESS` (default: `30/minute`) - ---- - -### 4. Read-Only API Endpoints (Default Limits) +### 3. Read-Only API Endpoints (Default Limits) **Rate Limit**: 100 requests per minute @@ -99,7 +69,7 @@ A limit of 30 requests per minute protects both internal resources and prevents --- -### 5. Frontend Routes (Default Limits) +### 4. Frontend Routes (Default Limits) **Rate Limit**: 100 requests per minute @@ -116,7 +86,7 @@ A limit of 30 requests per minute protects both internal resources and prevents --- -### 6. Webhook/Callback Endpoints (Higher Limits) +### 5. Webhook/Callback Endpoints (Higher Limits) **Rate Limit**: Consider exemption or very high limits @@ -129,7 +99,7 @@ A limit of 30 requests per minute protects both internal resources and prevents --- -### 7. Health/Diagnostic Endpoints (Exempt or High Limits) +### 6. Health/Diagnostic Endpoints (Exempt or High Limits) **Rate Limit**: Potentially exempt for monitoring