feat(tasks): add 5 additional scheduled jobs (expire_shared_links, prune_processing_logs, prune_old_notifications, backfill_missing_metadata, sync_search_index)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
@@ -0,0 +1,226 @@
|
||||
# Scheduled Jobs
|
||||
|
||||
DocuElevate includes an admin-managed **Scheduled Jobs** system that runs
|
||||
recurring maintenance and processing tasks automatically via Celery Beat.
|
||||
|
||||
All jobs are visible and configurable through the admin UI at
|
||||
**Admin → Scheduled Jobs** (`/admin/scheduled-jobs`).
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
Scheduled jobs replace the need to run manual batch operations. Each job:
|
||||
|
||||
- Runs automatically on its configured cron or interval schedule.
|
||||
- Can be **enabled** or **disabled** without restarting the worker.
|
||||
- Can be triggered **immediately** with the *Run Now* button.
|
||||
- Reports its last-run time, status (`success`, `failed`, `running`), and
|
||||
a short detail message back to the UI.
|
||||
|
||||
> **Note:** Schedule changes (cron expressions, interval values, enable/disable
|
||||
> toggles) are persisted to the database immediately. However, Celery Beat
|
||||
> reads the schedule only at worker startup — so changes take effect after
|
||||
> **restarting the Celery worker**.
|
||||
> The *Run Now* button dispatches a job immediately and does **not** require
|
||||
> a restart.
|
||||
|
||||
---
|
||||
|
||||
## Built-in Jobs
|
||||
|
||||
### 1. Process New Documents
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.process_new_documents` |
|
||||
| **Default schedule** | Every hour (cron `0 */1 * * *`) |
|
||||
| **Purpose** | Scans for documents that have been uploaded but never processed, then queues them through the full pipeline. |
|
||||
|
||||
A document is considered *new* when it has no `FileProcessingStep` rows
|
||||
matching the core pipeline steps. Only files whose `local_filename` still
|
||||
exists on disk are queued; others are skipped and counted.
|
||||
|
||||
---
|
||||
|
||||
### 2. Reprocess Failed Documents
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.reprocess_failed_documents` |
|
||||
| **Default schedule** | Every 6 hours (cron `30 */6 * * *`) |
|
||||
| **Purpose** | Finds documents whose last processing step has status `failure` and re-queues them. Skips files that are currently in-progress. |
|
||||
|
||||
---
|
||||
|
||||
### 3. Clean Up Temporary Files
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.cleanup_temp_files` |
|
||||
| **Default schedule** | Daily at 03:30 UTC (cron `30 3 * * *`) |
|
||||
| **Purpose** | Removes stale files from `workdir/tmp`. Only files older than 24 hours that are not referenced by any active processing job are deleted. |
|
||||
|
||||
---
|
||||
|
||||
### 4. Expire Stale Shared Links
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.expire_shared_links` |
|
||||
| **Default schedule** | Daily at 01:00 UTC (cron `0 1 * * *`) |
|
||||
| **Purpose** | Marks shared document links as inactive when their `expires_at` timestamp has passed. |
|
||||
|
||||
Access is already blocked at request time, but this task keeps the management
|
||||
UI counts and statuses accurate.
|
||||
|
||||
---
|
||||
|
||||
### 5. Prune Old Processing Logs
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.prune_processing_logs` |
|
||||
| **Default schedule** | Weekly on Sunday at 04:00 UTC (cron `0 4 * * 0`) |
|
||||
| **Purpose** | Deletes `processing_logs` and `settings_audit_log` rows older than 30 days to prevent unbounded table growth. |
|
||||
|
||||
---
|
||||
|
||||
### 6. Prune Old Notifications
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.prune_old_notifications` |
|
||||
| **Default schedule** | Weekly on Sunday at 04:30 UTC (cron `30 4 * * 0`) |
|
||||
| **Purpose** | Deletes *read* `in_app_notifications` rows older than 30 days. Unread notifications are never deleted. |
|
||||
|
||||
---
|
||||
|
||||
### 7. Backfill Missing AI Metadata
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.backfill_missing_metadata` |
|
||||
| **Default schedule** | Every 6 hours (cron `0 */6 * * *`) |
|
||||
| **Purpose** | Re-triggers AI metadata extraction for documents that have OCR text but no `ai_metadata`. Processes up to 50 documents per run. |
|
||||
|
||||
This is particularly useful when:
|
||||
- An AI provider (OpenAI / Azure) was configured after documents were already
|
||||
ingested.
|
||||
- A previous metadata extraction attempt failed.
|
||||
|
||||
---
|
||||
|
||||
### 8. Sync Search Index
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Task** | `app.tasks.batch_tasks.sync_search_index` |
|
||||
| **Default schedule** | Hourly (cron `15 */1 * * *`) |
|
||||
| **Purpose** | Indexes documents in Meilisearch that have OCR text or AI metadata but are not yet present in the search index. Processes up to 100 documents per run. |
|
||||
|
||||
This is useful after:
|
||||
- Enabling Meilisearch for the first time on an existing installation.
|
||||
- Recovering from a Meilisearch index wipe or migration.
|
||||
|
||||
---
|
||||
|
||||
## Managing Schedules
|
||||
|
||||
### Editing a schedule
|
||||
|
||||
1. Navigate to **Admin → Scheduled Jobs**.
|
||||
2. Click **Edit Schedule** on the job card.
|
||||
3. Choose **Cron** or **Interval** and fill in the fields.
|
||||
4. Click **Save Changes**.
|
||||
|
||||
**Restart the worker** for the new schedule to take effect:
|
||||
|
||||
```bash
|
||||
# Docker Compose
|
||||
docker compose restart worker
|
||||
|
||||
# Kubernetes
|
||||
kubectl rollout restart deployment/docuelevate-worker
|
||||
```
|
||||
|
||||
### Cron format
|
||||
|
||||
Cron expressions follow the standard 5-field format:
|
||||
|
||||
```
|
||||
minute hour day-of-month month day-of-week
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
| Expression | Meaning |
|
||||
|---|---|
|
||||
| `0 * * * *` | Every hour on the hour |
|
||||
| `0 2 * * *` | Daily at 02:00 UTC |
|
||||
| `30 4 * * 0` | Every Sunday at 04:30 UTC |
|
||||
| `*/15 * * * *` | Every 15 minutes |
|
||||
|
||||
### Interval format
|
||||
|
||||
For interval schedules, provide the number of **seconds**:
|
||||
|
||||
| Seconds | Meaning |
|
||||
|---|---|
|
||||
| `60` | Every minute |
|
||||
| `3600` | Every hour |
|
||||
| `86400` | Every day |
|
||||
|
||||
The minimum interval is **60 seconds**.
|
||||
|
||||
---
|
||||
|
||||
## Enabling / Disabling a job
|
||||
|
||||
Click the toggle on the job card to enable or disable the job. The change
|
||||
is saved immediately, but the updated schedule only takes effect after the
|
||||
worker restarts.
|
||||
|
||||
Disabling a job does **not** cancel an in-flight execution — it only prevents
|
||||
Celery Beat from scheduling new executions.
|
||||
|
||||
---
|
||||
|
||||
## Run Now
|
||||
|
||||
Click **Run Now** on any job card to dispatch it immediately, regardless of
|
||||
its schedule or enabled status. The task is sent to the `default` Celery
|
||||
queue and will be picked up by the next available worker.
|
||||
|
||||
The UI shows a spinner while the dispatch request is in-flight and displays
|
||||
the Celery task ID on success. Refresh the page after a few seconds to see
|
||||
the updated last-run status.
|
||||
|
||||
---
|
||||
|
||||
## Adding custom jobs
|
||||
|
||||
Custom batch jobs can be added by:
|
||||
|
||||
1. Creating a new Celery task in `app/tasks/` (or any existing tasks module).
|
||||
2. Adding a row to `DEFAULT_JOBS` in `app/api/scheduled_jobs.py`.
|
||||
3. Running the app — the new job will be seeded automatically on startup.
|
||||
|
||||
The task function should call `_update_job_status(job_name, status, detail)`
|
||||
at the end of its execution so the admin UI reflects the outcome.
|
||||
|
||||
---
|
||||
|
||||
## Architecture notes
|
||||
|
||||
- **Seeding**: Default jobs are seeded into the `scheduled_jobs` database
|
||||
table when the FastAPI application starts up (via the lifespan handler in
|
||||
`app/main.py`). Seeding is idempotent — re-running does not create
|
||||
duplicate rows.
|
||||
- **Beat integration**: `app/celery_worker.py` calls `_load_db_scheduled_jobs()`
|
||||
at import time to extend `celery.conf.beat_schedule` with enabled DB jobs.
|
||||
Static hardcoded entries always take precedence over DB entries with the
|
||||
same key.
|
||||
- **Status tracking**: Each task calls `_update_job_status()` in
|
||||
`app/tasks/batch_tasks.py` to persist `last_run_at`, `last_run_status`, and
|
||||
`last_run_detail` back to the `scheduled_jobs` table.
|
||||
Reference in New Issue
Block a user