gitea-admin/gh-christianlouis-docuelevate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

added a favicon, updated the README.md file

Browse Source
This commit is contained in:
Christian Krakau-Louis
2025-03-25 10:48:55 +01:00
parent 50ba7989c5
commit 19c763e595
33 changed files with 2434 additions and 2374 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+115 -60
View File
@@ -2,95 +2,144 @@
## Overview ## Overview
This project automates the handling, extraction, and processing of documents using a combination of services such as OpenAI, Dropbox, Nextcloud, and Paperless NGX. The system extracts metadata, processes document contents, and stores the results efficiently. It is designed for flexibility and configurability through environment variables, making it easily customizable for different workflows. This project automates the handling, extraction, and processing of documents using a variety of services, including:
- **OpenAI** for metadata extraction and text refinement.
- **Dropbox** and **Nextcloud** for file storage and uploads.
- **Paperless NGX** for document indexing and management.
- **Azure Document Intelligence** (optional) for OCR on PDFs (replacing Textract).
- **Gotenberg** for file-to-PDF conversions.
- **AWS S3** (currently implemented but may be removed in the future).
It is designed for flexibility and configurability through environment variables, making it easily customizable for different workflows. The system can fetch documents from multiple IMAP mailboxes, process them (OCR, metadata extraction, PDF conversion), and store them in the desired destinations.
## Features ## Features
- **Document Upload & Storage**: Upload and manage documents via Dropbox and Nextcloud. - **Document Upload & Storage**:
- **OCR Processing**: Extract text from scanned documents. - Manual uploads (via API) to S3, or direct uploads to Dropbox/Nextcloud/Paperless.
- **Metadata Extraction**: Automatically extract key information using OpenAI's API. - **OCR Processing (Azure)**:
- **Document Management**: Store processed documents and metadata in Paperless NGX for easy retrieval. - Extract text from scanned PDFs using Azure Document Intelligence.
- **IMAP Integration**: Fetch documents from multiple IMAP email accounts for processing. - **Metadata Extraction (OpenAI)**:
- Use GPT to classify, label, or otherwise enrich the text with structured metadata.
- **PDF Conversion (Gotenberg)**:
- Convert non-PDF attachments (e.g., Word docs, images) into PDFs.
- **Document Management (Paperless NGX)**:
- Store processed documents and metadata in a Paperless NGX instance.
- **IMAP Integration**:
- Fetch documents from multiple mailboxes (including Gmail) and automatically enqueue them for processing.
## Environment Variables ## Environment Variables
The project is configured via the `.env` file, where credentials and settings for different services are defined. Below is a breakdown of key configuration variables: The `.env` file drives all configuration. This table breaks down key variables—some are optional, depending on which services you actually use.
### General Configuration ### Core Settings
| **Variable** | **Description** | **How to Obtain** | | **Variable** | **Description** | **Example** |
|-------------|----------------|-------------------| |------------------------|----------------------------------------------------------|--------------------------------|
| `DATABASE_URL` | Path to the SQLite database. | Example: `sqlite:///./app/database.db` | | `DATABASE_URL` | Path/URL to the SQLite database (or other SQL backend). | `sqlite:///./app/database.db` |
| `REDIS_URL` | URL for Redis connection. | Example: `redis://redis:6379/0` | | `REDIS_URL` | URL for Redis, used by Celery for broker & result store. | `redis://redis:6379/0` |
| `WORKDIR` | Working directory for the application. | Example: `/workdir` | | `WORKDIR` | Working directory for the application. | `/workdir` |
| `NEXTCLOUD_UPLOAD_URL` | Nextcloud WebDAV upload URL. | Example: `https://nextcloud.example.com/remote.php/dav/files/<USERNAME>` | | `GOTENBERG_URL` | Gotenberg PDF processing URL. | `http://gotenberg:3000` |
| `NEXTCLOUD_FOLDER` | Folder in Nextcloud for file uploads. | Example: `/Documents/Uploads` |
| `PAPERLESS_NGX_URL` | Paperless NGX API endpoint. | Example: `https://paperless.example.com/api/documents/post_document/` |
| `PAPERLESS_HOST` | Root URL for Paperless NGX. | Example: `https://paperless.example.com` |
### Tokens/API Credentials ### IMAP Configuration (Multiple Mailboxes)
| **Variable** | **Description** | **How to Obtain** | | **Variable** | **Description** | **Example** |
|-------------|----------------|-------------------| |-------------------------------|--------------------------------------------------------------|-------------------|
| `OPENAI_API_KEY` | API key for OpenAI services. | Get from [OpenAI platform](https://platform.openai.com/account/api-keys). | | `IMAP1_HOST` | Hostname for first IMAP server. | `mail.example.com`|
| `PAPERLESS_NGX_API_TOKEN` | API token for Paperless NGX. | Obtain from your Paperless NGX instance. | | `IMAP1_PORT` | Port number (usually `993`). | `993` |
| `DROPBOX_APP_KEY` | Dropbox API key. | Generate from the [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create). | | `IMAP1_USERNAME` | IMAP login (first mailbox). | `user@example.com`|
| `DROPBOX_APP_SECRET` | Dropbox API secret. | Available in the Dropbox Developer Console. | | `IMAP1_PASSWORD` | IMAP password (first mailbox). | `*******` |
| `DROPBOX_REFRESH_TOKEN` | Dropbox OAuth refresh token. | Obtain by following Dropbox's OAuth flow. | | `IMAP1_SSL` | Use SSL (`true`/`false`). | `true` |
| `IMAP1_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll for new mail. | `5` |
| `IMAP1_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`). | `false` |
| `IMAP2_HOST` | Hostname for second IMAP server (optional). | `imap.gmail.com` |
| `IMAP2_PORT` | Port number for second mailbox. | `993` |
| `IMAP2_USERNAME` | IMAP login for second mailbox. | `you@gmail.com` |
| `IMAP2_PASSWORD` | IMAP password for second mailbox. | `*******` |
| `IMAP2_SSL` | Use SSL for second mailbox (`true`/`false`). | `true` |
| `IMAP2_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll second mailbox. | `10` |
| `IMAP2_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`) for mailbox.| `false` |
### User Credentials ### OpenAI & Azure Document Intelligence
| **Variable** | **Description** | | **Variable** | **Description** | **How to Obtain** |
|-------------|----------------| |-----------------------|--------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------|
| `ADMIN_USERNAME` | Admin username for system access. | | `OPENAI_API_KEY` | API key for OpenAI services (used for metadata extraction/refinement). | [OpenAI platform](https://platform.openai.com/account/api-keys) |
| `ADMIN_PASSWORD` | Admin password for system access. | | `AZURE_AI_KEY` | Azure Document Intelligence key (for OCR). | [Azure Portal](https://portal.azure.com/) |
| `NEXTCLOUD_USERNAME` | Username for Nextcloud authentication. | | `AZURE_REGION` | Azure region of your Document Intelligence instance. | e.g. `eastus`, `westeurope` |
| `NEXTCLOUD_PASSWORD` | Password for Nextcloud authentication. | | `AZURE_ENDPOINT` | Endpoint URL for Document Intelligence. | e.g. `https://<yourendpoint>.cognitiveservices.azure.com/` |
### IMAP Configuration ### Paperless NGX
| **Variable** | **Description** | | **Variable** | **Description** |
|-------------|----------------| |-------------------------------|-----------------------------------------------------|
| `IMAP1_USERNAME` | IMAP username for the first email account. | | `PAPERLESS_NGX_API_TOKEN` | API token for Paperless NGX. |
| `IMAP1_PASSWORD` | IMAP password for the first email account. | | `PAPERLESS_HOST` | Root URL for Paperless NGX (e.g. `https://paperless.example.com`). |
| `IMAP1_HOST` | Hostname of the first IMAP server. |
| `IMAP1_PORT` | IMAP server port (typically `993`). |
| `IMAP1_SSL` | Enable SSL (`true` or `false`). |
| `IMAP1_POLL_INTERVAL_MINUTES` | Polling interval for IMAP server. |
| `IMAP1_DELETE_AFTER_PROCESS` | Delete emails after processing (`true` or `false`). |
### Additional Services ### Dropbox
| **Variable** | **Description** | | **Variable** | **Description** | **How to Obtain** |
|-------------|----------------| |-------------------------|--------------------------------------------------|------------------------------------------------------------------------------------|
| `GOTENBERG_URL` | URL for Gotenberg PDF processing. | | `DROPBOX_APP_KEY` | Dropbox API app key. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) |
| `DROPBOX_APP_SECRET` | Dropbox API app secret. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) |
| `DROPBOX_REFRESH_TOKEN` | OAuth2 refresh token for Dropbox. | Follow Dropbox OAuth flow to retrieve |
| `DROPBOX_FOLDER` | Default folder path for Dropbox uploads. | e.g. `"/Documents/Uploads"` |
### Nextcloud
| **Variable** | **Description** |
|-------------------------|---------------------------------------------------------------|
| `NEXTCLOUD_UPLOAD_URL` | Nextcloud WebDAV URL (e.g. `https://nc.example.com/remote.php/dav/files/<USERNAME>`). |
| `NEXTCLOUD_USERNAME` | Nextcloud login username. |
| `NEXTCLOUD_PASSWORD` | Nextcloud login password. |
| `NEXTCLOUD_FOLDER` | Destination folder in Nextcloud (e.g. `"/Documents/Uploads"`). |
### AWS S3
| **Variable** | **Description** |
|------------------------|---------------------------------------------------------------|
| `AWS_ACCESS_KEY_ID` | AWS Access Key (used for S3 upload). |
| `AWS_SECRET_ACCESS_KEY`| AWS Secret Key (used for S3 upload). |
| `AWS_REGION` | AWS region for S3. |
| `S3_BUCKET_NAME` | Default S3 bucket name if using S3 upload. |
### General Admin Credentials
| **Variable** | **Description** |
|--------------------|---------------------------------------------|
| `ADMIN_USERNAME` | Admin username for system access. |
| `ADMIN_PASSWORD` | Admin password for system access. |
## Running as a Docker Container ## Running as a Docker Container
This project includes a `docker-compose.yml` file that allows for easy deployment using Docker. The following services are defined: This project uses Celery (with Redis) for asynchronous task management and Gotenberg for PDF conversion. The `docker-compose.yml` file defines these services:
- **API Service**: Runs the document processing API using `uvicorn`. - **API Service**: Runs the FastAPI application via `uvicorn`.
- **Worker Service**: Runs the Celery worker for handling document processing tasks. - **Worker Service**: Runs the Celery worker for processing tasks (PDF conversions, OCR, etc.).
- **Redis**: Used as a message broker for Celery. - **Redis**: Provides the message broker & result backend for Celery.
- **Gotenberg**: Provides PDF processing capabilities. - **Gotenberg**: Offers PDF conversion capabilities.
### Running the Application with Docker Compose ### Running the Application with Docker Compose
1. **Ensure Docker and Docker Compose are installed**. 1. **Install Docker and Docker Compose** on your system.
2. **Clone the repository and navigate to the directory**: 2. **Clone the repository** and navigate into it:
```bash ```bash
git clone <repository_url> git clone <repository_url>
cd <repository_name> cd <repository_name>
``` ```
3. **Create and configure the `.env` file**. 3. **Create and configure the `.env` file**:
4. **Start the services**: - Fill in the variables from the tables above.
- (At minimum, you need `DATABASE_URL`, `REDIS_URL`, `WORKDIR`, plus whichever service creds you plan to use.)
4. **Launch the services**:
```bash ```bash
docker-compose up -d docker-compose up -d
``` ```
5. The API will be available at `http://localhost:8000`. 5. The API will be available at **`http://localhost:8000`**.
### Services in `docker-compose.yml` ### Services in `docker-compose.yml`
Below is the default structure (simplified):
```yaml ```yaml
services: services:
api: api:
@@ -137,8 +186,14 @@ services:
## To-Do List ## To-Do List
- Refactor AWS-related code to Azure. - **Refactor AWS-related code** to rely on Azure or remove if no longer needed.
- Remove unnecessary environment variables. - **Remove unnecessary environment variables** once final service usage is determined.
- Make upload targets configurable. - **Make upload targets configurable** (e.g., easily choose only Dropbox, Nextcloud, or Paperless).
- Remove S3 upload functionality. - **Potentially remove or consolidate S3 upload code** if Azure is the preferred cloud option.
---
**Questions or Issues?**
- Feel free to open an issue or pull request.
- For local testing or development, use `docker-compose up` and watch the logs via `docker-compose logs -f`.
- Ensure your `.env` aligns with the environment variables listed above. If you see unexpected errors, check for typos or missing values.
app/frontend.py
+5
View File
@@ -17,3 +17,8 @@ router.mount("/static", StaticFiles(directory=frontend_folder), name="static")
@router.get("/ui", response_class=FileResponse) @router.get("/ui", response_class=FileResponse)
def serve_ui(): def serve_ui():
return os.path.join(frontend_folder, "index.html") return os.path.join(frontend_folder, "index.html")
# 3) Serve favicon.ico from the frontend folder
@router.get("/favicon.ico", response_class=FileResponse)
def favicon():
return os.path.join(frontend_folder, "favicon.ico")
frontend/favicon.ico
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 14 KiB