diff --git a/docs/API.md b/docs/API.md index f113731b..45636a13 100644 --- a/docs/API.md +++ b/docs/API.md @@ -101,6 +101,29 @@ Reprocess a specific file. } ``` +### Batch Processing + +**POST** `/api/processall` + +Process all PDF files in the configured workdir directory. + +**Throttling**: For large batches (>20 files by default), tasks are automatically staggered to prevent overwhelming downstream APIs. The throttling behavior can be configured via environment variables: + +- `PROCESSALL_THROTTLE_THRESHOLD`: Number of files above which throttling is applied (default: 20) +- `PROCESSALL_THROTTLE_DELAY`: Delay in seconds between each task submission when throttling (default: 3) + +**Example**: When processing 25 files with default settings, the first file is queued immediately, the second after 3 seconds, the third after 6 seconds, etc., spreading the load over 72 seconds total. + +**Response**: +```json +{ + "message": "Enqueued 25 PDFs for processing (throttled over 72 seconds)", + "pdf_files": ["file1.pdf", "file2.pdf", ...], + "task_ids": ["a1b2c3...", "d4e5f6...", ...], + "throttled": true +} +``` + **POST** `/send_to_google_drive/` Send a processed file to Google Drive. diff --git a/docs/ConfigurationGuide.md b/docs/ConfigurationGuide.md index 04fc0edb..0aef4ee6 100644 --- a/docs/ConfigurationGuide.md +++ b/docs/ConfigurationGuide.md @@ -17,6 +17,20 @@ Configuration is primarily done through environment variables specified in a `.e | `EXTERNAL_HOSTNAME` | The external hostname for the application. | `docuelevate.example.com` | | `ALLOW_FILE_DELETE` | Enable file deletion in the web interface (`true`/`false`). | `true` | +### Batch Processing Settings + +Control how the `/processall` endpoint handles large batches of files to prevent overwhelming downstream APIs. + +| **Variable** | **Description** | **Default** | +|-----------------------------------|----------------------------------------------------------------------------------------------------|-------------| +| `PROCESSALL_THROTTLE_THRESHOLD` | Number of files above which throttling is applied. Files <= threshold are processed immediately. | `20` | +| `PROCESSALL_THROTTLE_DELAY` | Delay in seconds between each task submission when throttling is active. | `3` | + +**Example Usage**: When processing 25 files with default settings: +- Files are staggered: file 0 at 0s, file 1 at 3s, file 2 at 6s, etc. +- Total queue time: (25-1) × 3 = 72 seconds +- Prevents API rate limit issues and ensures smooth processing + ### IMAP Configuration DocuElevate can monitor multiple IMAP mailboxes for document attachments. Each mailbox uses a numbered prefix (e.g., `IMAP1_`, `IMAP2_`).