diff --git a/README.md b/README.md index fbc52b35..42431050 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,10 @@
- DocuNova Logo -

DocuNova

+ DocuNova Logo

Intelligent Document Processing & Management

+# DocuNova + ## Overview DocuNova automates the handling, extraction, and processing of documents using a variety of services, including: @@ -19,6 +20,15 @@ It is designed for flexibility and configurability through environment variables The project includes a **UI** for uploading and managing files, and an API documentation page is available at `/docs` (powered by **FastAPI**). +## Documentation Index + +- [User Guide](docs/UserGuide.md) - How to use DocuNova +- [API Documentation](docs/API.md) - API reference +- [Deployment Guide](docs/DeploymentGuide.md) - How to deploy DocuNova +- [Configuration Guide](docs/ConfigurationGuide.md) - Available configuration options +- [Development Guide](CONTRIBUTING.md) - How to contribute to DocuNova +- [Troubleshooting](docs/Troubleshooting.md) - Common issues and solutions + ## Screenshots
@@ -80,171 +90,32 @@ Users can choose to send documents to any combination of these destinations thro ## Frameworks Used -- **FastAPI**: A modern, fast (high-performance) web framework for building APIs with Python. -- **Celery**: A distributed task queue for asynchronous processing. -- **SQLAlchemy**: A powerful ORM for database interactions. -- **Jinja2**: A templating engine for rendering HTML pages. -- **Tailwind CSS**: A utility-first CSS framework for styling the UI. +- **FastAPI**: High-performance web framework for APIs. +- **Celery**: Task queue for asynchronous processing. +- **Redis**: Message broker and result backend. +- **SQLAlchemy**: ORM for database interactions. +- **Tailwind CSS**: Utility-first CSS framework. +- **Docker**: Containerization for easy deployment. -## Environment Variables +## Quick Start -The `.env` file drives all configuration. This table breaks down key variables—some are optional, depending on which services you actually use. +For detailed installation and deployment instructions, please refer to the [Deployment Guide](docs/DeploymentGuide.md). -### Core Settings +```bash +# Clone the repository +git clone +cd document-processor -| **Variable** | **Description** | **Example** | -|------------------------|----------------------------------------------------------|--------------------------------| -| `DATABASE_URL` | Path/URL to the SQLite database (or other SQL backend). | `sqlite:///./app/database.db` | -| `REDIS_URL` | URL for Redis, used by Celery for broker & result store. | `redis://redis:6379/0` | -| `WORKDIR` | Working directory for the application. | `/workdir` | -| `GOTENBERG_URL` | Gotenberg PDF processing URL. | `http://gotenberg:3000` | -| `EXTERNAL_HOSTNAME` | The external hostname for the application. | `docunova.example.com` | +# Configure environment variables +cp .env.example .env +# Edit .env with your settings -### IMAP Configuration (Multiple Mailboxes) - -| **Variable** | **Description** | **Example** | -|-------------------------------|--------------------------------------------------------------|-------------------| -| `IMAP1_HOST` | Hostname for first IMAP server. | `mail.example.com`| -| `IMAP1_PORT` | Port number (usually `993`). | `993` | -| `IMAP1_USERNAME` | IMAP login (first mailbox). | `user@example.com`| -| `IMAP1_PASSWORD` | IMAP password (first mailbox). | `*******` | -| `IMAP1_SSL` | Use SSL (`true`/`false`). | `true` | -| `IMAP1_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll for new mail. | `5` | -| `IMAP1_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`). | `false` | -| `IMAP2_HOST` | Hostname for second IMAP server (optional). | `imap.gmail.com` | -| `IMAP2_PORT` | Port number for second mailbox. | `993` | -| `IMAP2_USERNAME` | IMAP login for second mailbox. | `you@gmail.com` | -| `IMAP2_PASSWORD` | IMAP password for second mailbox. | `*******` | -| `IMAP2_SSL` | Use SSL for second mailbox (`true`/`false`). | `true` | -| `IMAP2_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll second mailbox. | `10` | -| `IMAP2_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`) for mailbox.| `false` | - -### OpenAI & Azure Document Intelligence - -| **Variable** | **Description** | **How to Obtain** | -|-----------------------|--------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| -| `OPENAI_API_KEY` | API key for OpenAI services (used for metadata extraction/refinement). | [OpenAI platform](https://platform.openai.com/account/api-keys) | -| `OPENAI_BASE_URL` | Base URL for OpenAI API (optional, defaults to OpenAI's endpoint). | `https://api.openai.com/v1` | -| `OPENAI_MODEL` | OpenAI model to use for tasks (e.g., GPT-4). | `gpt-4` | -| `AZURE_AI_KEY` | Azure Document Intelligence key (for OCR). | [Azure Portal](https://portal.azure.com/) | -| `AZURE_REGION` | Azure region of your Document Intelligence instance. | e.g. `eastus`, `westeurope` | -| `AZURE_ENDPOINT` | Endpoint URL for Document Intelligence. | e.g. `https://.cognitiveservices.azure.com/` | - -### Authentik - -| **Variable** | **Description** | -|-------------------------|---------------------------------------------------------------| -| `AUTH_ENABLED` | Enable or disable authentication (`true`/`false`). | -| `AUTHENTIK_CLIENT_ID` | Client ID for Authentik OAuth2. | -| `AUTHENTIK_CLIENT_SECRET` | Client secret for Authentik OAuth2. | -| `AUTHENTIK_CONFIG_URL` | Configuration URL for Authentik OpenID Connect. | - -### Paperless NGX - -| **Variable** | **Description** | -|-------------------------------|-----------------------------------------------------| -| `PAPERLESS_NGX_API_TOKEN` | API token for Paperless NGX. | -| `PAPERLESS_HOST` | Root URL for Paperless NGX (e.g. `https://paperless.example.com`). | - -### Dropbox - -| **Variable** | **Description** | **How to Obtain** | -|-------------------------|--------------------------------------------------|------------------------------------------------------------------------------------| -| `DROPBOX_APP_KEY` | Dropbox API app key. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) | -| `DROPBOX_APP_SECRET` | Dropbox API app secret. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) | -| `DROPBOX_REFRESH_TOKEN` | OAuth2 refresh token for Dropbox. | Follow Dropbox OAuth flow to retrieve | -| `DROPBOX_FOLDER` | Default folder path for Dropbox uploads. | e.g. `"/Documents/Uploads"` | - -### Nextcloud - -| **Variable** | **Description** | -|-------------------------|---------------------------------------------------------------| -| `NEXTCLOUD_UPLOAD_URL` | Nextcloud WebDAV URL (e.g. `https://nc.example.com/remote.php/dav/files/`). | -| `NEXTCLOUD_USERNAME` | Nextcloud login username. | -| `NEXTCLOUD_PASSWORD` | Nextcloud login password. | -| `NEXTCLOUD_FOLDER` | Destination folder in Nextcloud (e.g. `"/Documents/Uploads"`). | - -## Running as a Docker Container - -This project uses Celery (with Redis) for asynchronous task management and Gotenberg for PDF conversion. The `docker-compose.yml` file defines these services: - -- **API Service**: Runs the FastAPI application via `uvicorn`. -- **Worker Service**: Runs the Celery worker for processing tasks (PDF conversions, OCR, etc.). -- **Redis**: Provides the message broker & result backend for Celery. -- **Gotenberg**: Offers PDF conversion capabilities. - -### Running the Application with Docker Compose - -1. **Install Docker and Docker Compose** on your system. -2. **Clone the repository** and navigate into it: - ```bash - git clone - cd - ``` -3. **Create and configure the `.env` file**: - - Fill in the variables from the tables above. - - (At minimum, you need `DATABASE_URL`, `REDIS_URL`, `WORKDIR`, plus whichever service creds you plan to use.) -4. **Launch the services**: - ```bash - docker-compose up -d - ``` -5. The API will be available at **`http://localhost:8000`**, and the API documentation is available at **`http://localhost:8000/docs`**. - -### Services in `docker-compose.yml` - -Below is the default structure: - -```yaml -services: - api: - image: christianlouis/document-processor:latest - container_name: document_api - working_dir: /workdir - command: ["sh", "-c", "cd /app && uvicorn app.main:app --host 0.0.0.0 --port 8000 --proxy-headers"] - environment: - - PYTHONPATH=/app - env_file: - - .env - ports: - - "8000:8000" - depends_on: - - redis - - worker - volumes: - - /var/docparse/workdir:/workdir - - worker: - image: christianlouis/document-processor:latest - container_name: document_worker - working_dir: /workdir - command: ["celery", "-A", "app.celery_worker", "worker", "-B", "--loglevel=info", "-Q", "document_processor,default,celery"] - env_file: - - .env - environment: - - PYTHONPATH=/app - depends_on: - - redis - - gotenberg - volumes: - - /var/docparse/workdir:/workdir - - gotenberg: - image: gotenberg/gotenberg:latest - container_name: gotenberg - - redis: - image: redis:alpine - container_name: document_redis - restart: always +# Run with Docker Compose +docker-compose up -d ``` -## To-Do List +The API will be available at **`http://localhost:8000`**, and the API documentation is available at **`http://localhost:8000/docs`**. -- **Make upload targets configurable** (e.g., easily choose only Dropbox, Nextcloud, or Paperless). +## License ---- - -**Questions or Issues?** -- Feel free to open an issue or pull request. -- For local testing or development, use `docker-compose up` and watch the logs via `docker-compose logs -f`. -- Ensure your `.env` aligns with the environment variables listed above. If you see unexpected errors, check for typos or missing values. \ No newline at end of file +This project is licensed under the Apache License 2.0 - see the [LICENSE](LICENSE) file for details. \ No newline at end of file diff --git a/docs/ConfigurationGuide.md b/docs/ConfigurationGuide.md new file mode 100644 index 00000000..c0af39a3 --- /dev/null +++ b/docs/ConfigurationGuide.md @@ -0,0 +1,141 @@ +# Configuration Guide + +DocuNova is designed to be highly configurable through environment variables. This guide explains all available configuration options and how to use them effectively. + +## Environment Variables + +Configuration is primarily done through environment variables specified in a `.env` file. + +### Core Settings + +| **Variable** | **Description** | **Example** | +|------------------------|----------------------------------------------------------|--------------------------------| +| `DATABASE_URL` | Path/URL to the SQLite database (or other SQL backend). | `sqlite:///./app/database.db` | +| `REDIS_URL` | URL for Redis, used by Celery for broker & result store. | `redis://redis:6379/0` | +| `WORKDIR` | Working directory for the application. | `/workdir` | +| `GOTENBERG_URL` | Gotenberg PDF processing URL. | `http://gotenberg:3000` | +| `EXTERNAL_HOSTNAME` | The external hostname for the application. | `docunova.example.com` | + +### IMAP Configuration + +DocuNova can monitor multiple IMAP mailboxes for document attachments. Each mailbox uses a numbered prefix (e.g., `IMAP1_`, `IMAP2_`). + +| **Variable** | **Description** | **Example** | +|-------------------------------|--------------------------------------------------------------|-------------------| +| `IMAP1_HOST` | Hostname for first IMAP server. | `mail.example.com`| +| `IMAP1_PORT` | Port number (usually `993`). | `993` | +| `IMAP1_USERNAME` | IMAP login (first mailbox). | `user@example.com`| +| `IMAP1_PASSWORD` | IMAP password (first mailbox). | `*******` | +| `IMAP1_SSL` | Use SSL (`true`/`false`). | `true` | +| `IMAP1_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll for new mail. | `5` | +| `IMAP1_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`). | `false` | + +To configure additional mailboxes, use `IMAP2_`, `IMAP3_`, etc. + +### OpenAI & Azure Document Intelligence + +| **Variable** | **Description** | **How to Obtain** | +|---------------------------------|------------------------------------------|--------------------------------------------------------------------------| +| `OPENAI_API_KEY` | OpenAI API key for GPT metadata extraction. | [OpenAI API keys](https://platform.openai.com/account/api-keys) | +| `AZURE_DOCUMENT_INTELLIGENCE_KEY` | Azure Document Intelligence API key for OCR. | [Azure Portal](https://portal.azure.com/) | +| `AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT` | Endpoint URL for Azure Doc Intelligence API. | [Azure Portal](https://portal.azure.com/) | + +### Authentication + +| **Variable** | **Description** | +|-------------------------|---------------------------------------------------------------| +| `AUTH_ENABLED` | Enable or disable authentication (`true`/`false`). | +| `AUTHENTIK_CLIENT_ID` | Client ID for Authentik OAuth2. | +| `AUTHENTIK_CLIENT_SECRET` | Client secret for Authentik OAuth2. | +| `AUTHENTIK_CONFIG_URL` | Configuration URL for Authentik OpenID Connect. | + +### Paperless NGX + +| **Variable** | **Description** | +|-------------------------------|-----------------------------------------------------| +| `PAPERLESS_NGX_API_TOKEN` | API token for Paperless NGX. | +| `PAPERLESS_HOST` | Root URL for Paperless NGX (e.g. `https://paperless.example.com`). | + +### Dropbox + +| **Variable** | **Description** | **How to Obtain** | +|-------------------------|--------------------------------------------------|------------------------------------------------------------------------------------| +| `DROPBOX_APP_KEY` | Dropbox API app key. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) | +| `DROPBOX_APP_SECRET` | Dropbox API app secret. | [Dropbox Developer Console](https://www.dropbox.com/developers/apps/create) | +| `DROPBOX_REFRESH_TOKEN` | OAuth2 refresh token for Dropbox. | Follow Dropbox OAuth flow to retrieve | +| `DROPBOX_FOLDER` | Default folder path for Dropbox uploads. | e.g. `"/Documents/Uploads"` | + +### Nextcloud + +| **Variable** | **Description** | +|-------------------------|---------------------------------------------------------------| +| `NEXTCLOUD_UPLOAD_URL` | Nextcloud WebDAV URL (e.g. `https://nc.example.com/remote.php/dav/files/`). | +| `NEXTCLOUD_USERNAME` | Nextcloud login username. | +| `NEXTCLOUD_PASSWORD` | Nextcloud login password. | +| `NEXTCLOUD_FOLDER` | Destination folder in Nextcloud (e.g. `"/Documents/Uploads"`). | + +## Configuration Examples + +### Minimal Configuration + +This is the minimal configuration needed to run DocuNova with local storage only: + +```dotenv +DATABASE_URL=sqlite:///./app/database.db +REDIS_URL=redis://redis:6379/0 +WORKDIR=/workdir +GOTENBERG_URL=http://gotenberg:3000 +``` + +### Full Configuration with All Services + +```dotenv +# Core settings +DATABASE_URL=sqlite:///./app/database.db +REDIS_URL=redis://redis:6379/0 +WORKDIR=/workdir +GOTENBERG_URL=http://gotenberg:3000 +EXTERNAL_HOSTNAME=docunova.example.com + +# IMAP settings +IMAP1_HOST=mail.example.com +IMAP1_PORT=993 +IMAP1_USERNAME=user@example.com +IMAP1_PASSWORD=password +IMAP1_SSL=true +IMAP1_POLL_INTERVAL_MINUTES=5 +IMAP1_DELETE_AFTER_PROCESS=false + +# AI services +OPENAI_API_KEY=sk-... +AZURE_DOCUMENT_INTELLIGENCE_KEY=... +AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT=https://... + +# Authentication +AUTH_ENABLED=true +AUTHENTIK_CLIENT_ID=... +AUTHENTIK_CLIENT_SECRET=... +AUTHENTIK_CONFIG_URL=https://auth.example.com/.well-known/openid-configuration + +# Storage services +PAPERLESS_NGX_API_TOKEN=... +PAPERLESS_HOST=https://paperless.example.com + +DROPBOX_APP_KEY=... +DROPBOX_APP_SECRET=... +DROPBOX_REFRESH_TOKEN=... +DROPBOX_FOLDER=/Documents/Uploads + +NEXTCLOUD_UPLOAD_URL=https://nc.example.com/remote.php/dav/files/username +NEXTCLOUD_USERNAME=username +NEXTCLOUD_PASSWORD=password +NEXTCLOUD_FOLDER=/Documents/Uploads +``` + +## Selective Service Configuration + +You can choose which document storage services to use by only including the relevant environment variables. For example, if you only want to use Dropbox, include only the Dropbox variables and omit the Paperless NGX and Nextcloud variables. + +## Configuration File Location + +The `.env` file should be placed at the root of the project directory. When using Docker Compose, you can reference it with the `env_file` directive in your `docker-compose.yml`. diff --git a/docs/ecosystem.svg b/docs/ecosystem.svg new file mode 100644 index 00000000..1384f44f --- /dev/null +++ b/docs/ecosystem.svg @@ -0,0 +1,73 @@ + + + + + + + + + DocuNova + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/frontend/static/logo_writing.svg b/frontend/static/logo_writing.svg new file mode 100644 index 00000000..f9353842 --- /dev/null +++ b/frontend/static/logo_writing.svg @@ -0,0 +1,21 @@ + + + + + + + + + + DocuNova + diff --git a/migrations/README b/migrations/README new file mode 100644 index 00000000..98e4f9c4 --- /dev/null +++ b/migrations/README @@ -0,0 +1 @@ +Generic single-database configuration. \ No newline at end of file diff --git a/requirements.txt b/requirements.txt index 879cf836..28b66b30 100644 --- a/requirements.txt +++ b/requirements.txt @@ -11,4 +11,5 @@ dropbox # Dropbox integration azure-ai-documentintelligence # Azure OCR service authlib # Authentication python-dotenv # Environment variables -starlette # ASGI toolkit (used by FastAPI) \ No newline at end of file +starlette # ASGI toolkit (used by FastAPI) +alembic # Database migrations \ No newline at end of file