gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: merge main branch and renumber migration 027→037

Browse Source 
Resolve 3 merge conflicts and renumber the automation_hooks migration
to follow main's migration chain (036_add_document_translation_fields).

Conflicts resolved:
- app/api/__init__.py: add automation_router alongside main's new routers
- app/utils/settings_service.py: add automation_hooks_enabled alongside compliance_enabled
- tests/conftest.py: add AutomationHook alongside AuditLog/ComplianceTemplate imports

Migration renumbered:
- 027_add_automation_hooks → 037_add_automation_hooks
- down_revision: 026_add_scheduled_jobs → 036_add_document_translation_fields

Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
This commit is contained in:
copilot-swe-agent[bot]
2026-03-16 22:40:15 +00:00
parent 6a83d51d88
commit 204000aabc
318 changed files with 317516 additions and 2858 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+203 -97
View File
@@ -24,121 +24,154 @@
</div>
<div align="center">
<a href="https://www.docuelevate.org"><img src="frontend/static/hero.png" alt="DocuElevate Logo" width="80%" /></a>
<a href="https://www.docuelevate.org"><img src="frontend/static/hero.png" alt="DocuElevate Hero" width="80%" /></a>
</div>
## Overview
DocuElevate automates the handling, extraction, and processing of documents using a variety of services, including:
DocuElevate is an intelligent document processing system that automates the ingestion, OCR, AI-powered metadata extraction, and distribution of documents. It supports a wide range of AI providers, OCR engines, and cloud storage destinations out of the box.
- **AI Provider** (pluggable – OpenAI, Anthropic, Gemini, Ollama, OpenRouter, Portkey, and more) for metadata extraction and text refinement.
- **Dropbox**, **Nextcloud**, and **Google Drive** for file storage and uploads.
- **Paperless NGX** for document indexing and management.
- **Azure Document Intelligence** for OCR on PDFs.
- **Gotenberg** for file-to-PDF conversions.
- **Authentik** for authentication and user management.
**Key capabilities:**
It is designed for flexibility and configurability through environment variables, making it easily customizable for different workflows. The system can fetch documents from multiple IMAP mailboxes, process them (OCR, metadata extraction, PDF conversion), and store them in the desired destinations.
- **AI-Powered Metadata Extraction** — pluggable AI providers including OpenAI, Anthropic Claude, Google Gemini, Ollama (local), OpenRouter, Portkey, and Azure OpenAI via LiteLLM
- **Multi-Engine OCR** — Azure Document Intelligence, Tesseract, EasyOCR, Mistral OCR, Google Cloud Document AI, and AWS Textract with configurable merge strategies
- **12 Storage Destinations** — Dropbox, Google Drive, OneDrive, Amazon S3, Nextcloud, WebDAV, FTP, SFTP, iCloud Drive, Email (SMTP), Paperless-ngx, and Rclone
- **Multi-Channel Ingestion** — web upload, browser extension, mobile app, CLI, REST API, IMAP email, and watched folders (local, cloud, FTP/SFTP)
- **Processing Pipelines** — customizable multi-step workflows with conditional routing rules
- **Full-Text Search** — powered by Meilisearch for instant document discovery
- **Multi-User with SSO** — local accounts, OAuth2/OIDC (Authentik), and social login (Google, Microsoft, Apple, Dropbox)
The project includes a **UI** for uploading and managing files, and an API documentation page is available at `/docs` (powered by **FastAPI**).
## Documentation Index
- [User Guide](docs/UserGuide.md) - How to use DocuElevate
- [Browser Extension Guide](docs/BrowserExtension.md) - Install and use the browser extension
- [API Documentation](docs/API.md) - API reference
- [Deployment Guide](docs/DeploymentGuide.md) - How to deploy DocuElevate
- [Configuration Guide](docs/ConfigurationGuide.md) - Available configuration options
- [Build Metadata](docs/BuildMetadata.md) - Automated version and build information
- [CI/CD Tools Guide](docs/CIToolsGuide.md) - CI/CD pipeline and tool documentation
- [CI Workflow Guide](docs/CIWorkflow.md) - Detailed workflow documentation
- [Development Guide](CONTRIBUTING.md) - How to contribute to DocuElevate
- [Troubleshooting](docs/Troubleshooting.md) - Common issues and solutions
The project ships with a web UI, a REST + GraphQL API, a CLI tool, a native mobile app (iOS & Android), a browser extension, and Helm charts for Kubernetes deployment.
## Screenshots
<div align="center">
<img src="docs/upload-view.png" alt="DocuElevate Upload Interface" width="80%" />
<p><em>Upload interface for adding new documents</em></p>
<p><em>Upload interface — drag-and-drop file upload with real-time progress</em></p>
<img src="docs/files-view.png" alt="DocuElevate Files View" width="80%" />
<p><em>Files view with processed documents and metadata</em></p>
<p><em>Files view — processed documents with AI-extracted metadata</em></p>
<img src="docs/status-view.png" alt="DocuElevate Status View" width="80%" />
<p><em>Status view — system health and service monitoring</em></p>
</div>
> **Note:** Screenshots may not reflect the very latest UI. For the most current look, visit [docuelevate.org](https://www.docuelevate.org).
## Workflow Process
DocuElevate follows a streamlined document processing workflow:
## Workflow
<div align="center">
<img src="docs/workflow-diagram.png" alt="DocuElevate Workflow" width="90%" />
</div>
### Document Ingestion
Documents enter DocuElevate through four possible channels:
1. **Web Upload**: Users manually upload files via the web interface
2. **Browser Extension**: Send files directly from your browser with one click
3. **Email Attachments**: Automatic polling of configured IMAP mailboxes (supports multiple accounts)
4. **API**: Direct programmatic uploads via the REST API
### Ingestion
Documents enter DocuElevate through multiple channels:
| Channel | Description |
|---------|-------------|
| **Web Upload** | Drag-and-drop interface with real-time progress (up to 1 GB per file) |
| **Browser Extension** | Clip web pages or send files from Chrome, Firefox, or Edge |
| **Mobile App** | Capture documents with the device camera or upload from the photo library |
| **CLI** | Batch uploads and scripted workflows via the `docuelevate` command-line tool |
| **REST API** | Programmatic uploads with full API-token authentication |
| **Email (IMAP)** | Automatic polling of multiple mailboxes with attachment filtering |
| **Watched Folders** | Monitor local paths, FTP, SFTP, S3, Dropbox, Google Drive, OneDrive, Nextcloud, or WebDAV for new files |
### Processing Pipeline
Every document goes through the following steps:
1. **PDF Conversion**: Non-PDF files are converted to PDF format using Gotenberg
2. **OCR Processing**: Azure Document Intelligence extracts text from images/scans
3. **Metadata Extraction**: The configured AI provider analyzes document content to identify:
- Document type (invoice, receipt, contract, etc.)
- Key entities (dates, names, amounts, account numbers)
- Important data points specific to the document type
4. **Enrichment**: Metadata is attached to the document in a structured format
Each document passes through a configurable set of steps:
1. **PDF Conversion** — Non-PDF files are converted using Gotenberg, with optional PDF/A archival conversion
2. **OCR** — Text extraction via one or more OCR engines (Azure, Tesseract, EasyOCR, Mistral, Google Document AI, AWS Textract) with configurable merge strategies
3. **AI Metadata Extraction** — The configured AI provider classifies the document and extracts structured metadata (type, dates, amounts, entities)
4. **Enrichment** — Metadata is embedded into the PDF and stored alongside the document
5. **Embedding Generation** — Vector embeddings for similarity search and duplicate detection
Steps can be customized using **Pipelines** and **Routing Rules** for conditional processing.
### Distribution
Processed documents with their metadata can be automatically sent to:
- **Dropbox**: For cloud storage and sharing
- **Nextcloud**: For self-hosted file storage
- **Google Drive**: For Google Workspace integration
- **Paperless-NGX**: For advanced document management with search capabilities
Users can choose to send documents to any combination of these destinations through configuration settings or manual selection.
Processed documents are distributed to any combination of configured destinations:
| Destination | Type |
|------------|------|
| **Dropbox** | Cloud storage |
| **Google Drive** | Cloud storage |
| **OneDrive** | Cloud storage |
| **Amazon S3** | Object storage |
| **Nextcloud** | Self-hosted cloud |
| **WebDAV** | Protocol-based |
| **FTP / SFTP** | File transfer |
| **iCloud Drive** | Apple cloud |
| **Email (SMTP)** | Send as attachment |
| **Paperless-ngx** | Document management system |
| **Rclone** | 70+ cloud providers via Rclone |
## Features
- **Intuitive File Upload**:
- Drag-and-drop file upload on both Upload and Files pages—upload anywhere on the Files page
- Real-time upload progress with validation
- Support for PDF, Office documents, images, and more (up to 500MB per file)
- **Browser Extension**:
- Send files directly from your browser to DocuElevate with one click
- Compatible with Chrome, Firefox, Edge, and other Chromium-based browsers
- Context menu integration for quick access
- See [Browser Extension Guide](docs/BrowserExtension.md) for installation and usage
- **Document Upload & Storage**:
- Manual uploads (via API or UI) to Dropbox, Nextcloud, Google Drive, or Paperless
- **OCR Processing (Azure)**:
- Extract text from scanned PDFs using Azure Document Intelligence
- **Metadata Extraction (AI Provider)**:
- Use any supported AI provider (OpenAI, Anthropic, Gemini, Ollama, etc.) to classify, label, or otherwise enrich the text with structured metadata
- **PDF Conversion (Gotenberg)**:
- Convert non-PDF attachments (e.g., Word docs, images) into PDFs
- **Document Management (Paperless NGX)**:
- Store processed documents and metadata in a Paperless NGX instance
- **IMAP Integration**:
- Fetch documents from multiple mailboxes (including Gmail) and automatically enqueue them for processing
- **Authentication**:
- Secure access to the system using **Authentik** for OAuth2-based login
### Document Processing
- **Multi-engine OCR** with quality checks and configurable merge strategies (AI merge, longest, primary)
- **AI metadata extraction** using any supported provider (OpenAI, Anthropic, Gemini, Ollama, OpenRouter, Portkey, Azure OpenAI)
- **PDF conversion** via Gotenberg with optional PDF/A archival format
- **Duplicate detection** — exact (SHA-256) and near-duplicate (content similarity with vector embeddings)
- **Customizable pipelines** — define multi-step processing workflows with conditional routing rules
## Frameworks Used
### Document Management
- **Full-text search** powered by Meilisearch with saved searches
- **File detail view** with metadata, text preview, processing history, and similarity analysis
- **Shared links** for public document access with expiration controls
- **Bulk operations** — reprocess, delete, or reassign documents in batch
- **FastAPI**: High-performance web framework for APIs.
- **Celery**: Task queue for asynchronous processing.
- **Redis**: Message broker and result backend.
- **SQLAlchemy**: ORM for database interactions.
- **Tailwind CSS**: Utility-first CSS framework.
- **Docker**: Containerization for easy deployment.
### Multi-Channel Ingestion
- **Web UI** — drag-and-drop upload with real-time progress
- **Browser extension** — clip web pages or send files from Chrome, Firefox, Edge ([guide](docs/BrowserExtension.md))
- **Mobile app** — iOS and Android with camera capture, push notifications, and SSO ([guide](docs/MobileApp.md))
- **CLI tool** — batch uploads, downloads, search, and API-token management ([guide](docs/CLIGuide.md))
- **REST API & GraphQL** — full programmatic access with Swagger documentation at `/docs`
- **IMAP email** — poll multiple mailboxes with attachment filtering and auto-processing
- **Watched folders** — local filesystem, FTP, SFTP, and cloud storage providers
### Administration
- **Multi-user mode** with per-user document isolation and ownership
- **Subscription & billing** — Stripe integration with configurable plans and quotas
- **Scheduled jobs** — IMAP polling, watched folder scans, automated backups, uptime monitoring
- **Audit logging** with SIEM integration support
- **Compliance templates** — GDPR, HIPAA, SOC 2
- **Admin dashboard** — user management, queue monitoring, credential management, backup/restore
### Authentication & Security
- **Local accounts** with self-service registration and password reset
- **OAuth2/OIDC** via Authentik or any OIDC provider
- **Social login** — Google, Microsoft, Apple, Dropbox
- **API tokens** for CLI, mobile, and automation access
- **Security headers** — HSTS, CSP, X-Frame-Options, X-Content-Type-Options
- **Rate limiting** with configurable per-endpoint controls
### Notifications
- **100+ notification backends** via Apprise — Discord, Telegram, Slack, Microsoft Teams, Email, webhooks, and more
- **Configurable events** — task failures, credential issues, file processed, user signup, payment issues
- **In-app notification inbox** with per-user preferences
- **Webhooks** — push events to external systems with HMAC signature verification and retry
## Tech Stack
| Component | Technology |
|-----------|-----------|
| **Backend** | FastAPI, Celery, Redis, SQLAlchemy, Alembic |
| **Frontend** | Jinja2, Tailwind CSS |
| **Search** | Meilisearch |
| **Mobile** | React Native (Expo) — iOS & Android |
| **AI** | LiteLLM (OpenAI, Anthropic, Gemini, Ollama, OpenRouter, Portkey) |
| **OCR** | Azure Document Intelligence, Tesseract, EasyOCR, Mistral, Google Doc AI, AWS Textract |
| **PDF** | Gotenberg, pypdf |
| **Auth** | Authlib (OAuth2/OIDC), MSAL, social providers |
| **Infrastructure** | Docker, Docker Compose, Helm/Kubernetes |
| **Docs** | MkDocs Material |
## Quick Start
For detailed installation and deployment instructions, please refer to the [Deployment Guide](docs/DeploymentGuide.md).
For detailed installation and deployment instructions, see the [Deployment Guide](docs/DeploymentGuide.md).
```bash
# Clone the repository
@@ -147,20 +180,96 @@ cd DocuElevate
# Configure environment variables
cp .env.demo .env
# Edit .env with your settings
# Edit .env with your settings (see Configuration Guide for all options)
# Run with Docker Compose
docker-compose up -d
docker compose up -d
```
The API will be available at **`http://localhost:8000`**, and the API documentation is available at **`http://localhost:8000/docs`**.
The web UI is available at **`http://localhost:8000`** and the interactive API documentation at **`http://localhost:8000/docs`**.
### Kubernetes / Helm
```bash
helm repo add docuelevate https://christianlouis.github.io/DocuElevate
helm install docuelevate docuelevate/docuelevate -f values.yaml
```
See the [Kubernetes Deployment Guide](docs/KubernetesDeployment.md) for full details.
## Documentation
### Getting Started
| Guide | Description |
|-------|-------------|
| [Setup Wizard](docs/SetupWizard.md) | Interactive first-run setup |
| [User Guide](docs/UserGuide.md) | How to use DocuElevate |
| [Browser Extension](docs/BrowserExtension.md) | Install and use the browser extension |
| [Mobile App](docs/MobileApp.md) | iOS and Android mobile app |
| [CLI Guide](docs/CLIGuide.md) | Command-line tool for automation |
### How-To Guides
| Guide | Description |
|-------|-------------|
| [How-To Overview](docs/HowToGuides.md) | Index of all how-to guides |
| [Email Ingestion](docs/howto/EmailIngestion.md) | Set up IMAP email polling |
| [Watched Folder](docs/howto/WatchedFolderSetup.md) | Monitor local or remote folders |
| [Mobile Scanning](docs/howto/MobileScanning.md) | Scan documents with your phone |
### Reference
| Guide | Description |
|-------|-------------|
| [API Documentation](docs/API.md) | REST & GraphQL API reference |
| [Configuration Guide](docs/ConfigurationGuide.md) | All environment variables |
| [Configuration Master](docs/ConfigurationMaster.md) | Configuration overview |
| [Settings Management](docs/SettingsManagement.md) | Runtime settings UI |
### Deployment & Operations
| Guide | Description |
|-------|-------------|
| [Deployment Guide](docs/DeploymentGuide.md) | Docker Compose deployment |
| [Kubernetes / Helm](docs/KubernetesDeployment.md) | Kubernetes deployment with Helm charts |
| [Production Readiness](docs/ProductionReadiness.md) | Checklist for production environments |
| [Database Configuration](docs/DatabaseConfiguration.md) | Database setup and migration |
| [Backup & Restore](docs/ConfigurationGuide.md#backup--restore) | Automated backup configuration |
### Storage Integration Setup
| Guide | Description |
|-------|-------------|
| [Dropbox](docs/DropboxSetup.md) | Dropbox OAuth setup |
| [Google Drive](docs/GoogleDriveSetup.md) | Google Drive service account / OAuth |
| [OneDrive](docs/OneDriveSetup.md) | Microsoft OneDrive setup |
| [Amazon S3](docs/AmazonS3Setup.md) | S3 bucket configuration |
| [Authentication](docs/AuthenticationSetup.md) | OAuth2, OIDC, and social login |
| [Notifications](docs/NotificationsSetup.md) | Notification backend setup |
### Security & Compliance
| Guide | Description |
|-------|-------------|
| [Credential Rotation](docs/CredentialRotationGuide.md) | Rotate secrets safely |
| [Licensing Compliance](docs/LicensingCompliance.md) | Dependency licenses |
| [Privacy & GDPR](docs/PrivacyCompliance.md) | Privacy compliance |
### Development
| Guide | Description |
|-------|-------------|
| [Contributing](CONTRIBUTING.md) | Code style, commits, and PR process |
| [Troubleshooting](docs/Troubleshooting.md) | Common issues and solutions |
| [Configuration Troubleshooting](docs/ConfigurationTroubleshooting.md) | Configuration-specific issues |
| [Build Metadata](docs/BuildMetadata.md) | Version and build information |
| [Internationalization](docs/InternationalizationGuide.md) | Translation and localization |
## Development & Testing
### Running Tests
DocuElevate includes comprehensive test coverage. To run tests:
```bash
# Install development dependencies
pip install -r requirements-dev.txt
@@ -175,21 +284,21 @@ pytest --cov=app --cov-report=term-missing
pytest -m unit
```
Tests are automatically configured with the necessary environment variables - **no manual setup required!**
Tests are automatically configured with the necessary environment variables — **no manual setup required!**
For detailed testing information, including integration tests with Docker and authentication testing, see the [Contributing Guide](CONTRIBUTING.md#running-tests).
For detailed testing information, see the [Contributing Guide](CONTRIBUTING.md#running-tests).
### Contributing
We welcome contributions! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for:
- Code style guidelines
- Code style guidelines (Ruff for formatting and linting)
- Commit message format (Conventional Commits)
- Testing requirements
- Pull request process
## License
This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details.
This project is licensed under the Apache License 2.0 — see the [LICENSE](LICENSE) file for details.
## Third-Party Software
@@ -214,13 +323,10 @@ The following is a summary of the licenses used by our direct dependencies:
| Uvicorn | BSD |
| SQLAlchemy | MIT |
| Pydantic | MIT |
| openai | MIT |
| litellm | MIT |
| pypdf | BSD |
| Requests | Apache 2.0 |
| puremagic | MIT |
| filetype | MIT |
| Dropbox | MIT |
| Dropbox SDK | MIT |
| Azure AI Document Intelligence | MIT |
| Authlib | BSD |
| Starlette | BSD |
@@ -229,15 +335,15 @@ The following is a summary of the licenses used by our direct dependencies:
| Microsoft Graph Core | MIT |
| MSAL | MIT |
| Boto3 | Apache 2.0 |
| Paramiko | LGPL-2.1|
| Paramiko | LGPL-2.1 |
| Apprise | MIT |
| Redis | BSD |
| Gotenberg | MIT |
| Redis (py) | BSD |
| Gotenberg Client | MIT |
| Meilisearch | MIT |
For a comprehensive list of all dependencies and their licenses, run:
```
```bash
pip install pip-licenses
pip-licenses
```