From 5aff586fb00a480d4051c2e6f60b8f1d1b378d57 Mon Sep 17 00:00:00 2001 From: Christian Krakau-Louis Date: Mon, 31 Mar 2025 10:58:01 +0200 Subject: [PATCH] feat: add comprehensive documentation including user guide, API reference, and deployment instructions --- CONTRIBUTING.md | 80 +++++++++++++++++ docs/API.md | 127 +++++++++++++++++++++++++++ docs/ConfigurationGuide.md | 19 ++-- docs/DeploymentGuide.md | 142 ++++++++++++++++++++++++++++++ docs/README.md | 34 +++++++ docs/Troubleshooting.md | 175 +++++++++++++++++++++++++++++++++++++ docs/UserGuide.md | 69 +++++++++++++++ 7 files changed, 635 insertions(+), 11 deletions(-) create mode 100644 CONTRIBUTING.md create mode 100644 docs/API.md create mode 100644 docs/DeploymentGuide.md create mode 100644 docs/README.md create mode 100644 docs/Troubleshooting.md create mode 100644 docs/UserGuide.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..a5b37c7a --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,80 @@ +# Contributing to DocuNova + +Thank you for your interest in contributing to DocuNova! This document provides guidelines and instructions for contributing to the project. + +## Code of Conduct + +By participating in this project, you agree to abide by the [Code of Conduct](CODE_OF_CONDUCT.md). + +## How to Contribute + +### Reporting Bugs + +If you find a bug in the codebase, please submit an issue on GitHub with: + +1. A clear title and description +2. Steps to reproduce the issue +3. Expected behavior +4. Actual behavior +5. Environment information (OS, Docker version, etc.) + +### Feature Requests + +We welcome feature requests! Please submit an issue with: + +1. A clear title and description +2. The problem the feature would solve +3. Any ideas you have for implementing the feature + +### Pull Requests + +1. Fork the repository +2. Create a new branch for your changes +3. Make your changes +4. Run the tests to ensure everything works +5. Submit a pull request with a clear description of the changes + +## Development Environment + +### Setting Up Your Environment + +```bash +# Clone the repository +git clone https://github.com/christianlouis/document-processor.git +cd document-processor + +# Create a virtual environment +python -m venv venv +source venv/bin/activate # On Windows: venv\Scripts\activate + +# Install dependencies +pip install -r requirements.txt +pip install -r requirements-dev.txt +``` + +### Running Tests + +```bash +pytest +``` + +### Code Style + +We use: +- Black for Python code formatting +- Flake8 for linting +- isort for import sorting + +```bash +# Format code +black . + +# Check linting +flake8 + +# Sort imports +isort . +``` + +## Project Structure + diff --git a/docs/API.md b/docs/API.md new file mode 100644 index 00000000..55b5f797 --- /dev/null +++ b/docs/API.md @@ -0,0 +1,127 @@ +# API Documentation + +DocuNova provides a powerful REST API for programmatic access to all its features. This document serves as a reference for the available endpoints and their usage. + +## API Overview + +- Base URL: `http:///api` +- Authentication: OAuth2 (when enabled) +- Response Format: JSON + +## Interactive API Documentation + +The most up-to-date and interactive API documentation is available at: + +`http:///docs` + +This Swagger UI provides a complete reference with the ability to try out API calls directly from your browser. + +## Authentication + +When authentication is enabled, you must include an authentication token in your requests: + +```bash +curl -X GET "http:///api/files" \ + -H "Authorization: Bearer " +``` + +## Common Endpoints + +### Document Upload + +**POST** `/api/upload` + +Upload one or more files for processing. + +**Request**: +- Multipart form data with file(s) + +**Response**: +```json +{ + "success": true, + "file_ids": [123, 124], + "message": "Files uploaded and queued for processing" +} +``` + +### Get Files + +**GET** `/api/files` + +Retrieve a list of processed files. + +**Parameters**: +- `limit` (optional): Maximum number of files to return +- `offset` (optional): Pagination offset +- `search` (optional): Search term + +**Response**: +```json +[ + { + "id": 123, + "original_filename": "invoice.pdf", + "file_size": 1024000, + "mime_type": "application/pdf", + "created_at": "2023-04-15T12:30:45Z" + }, + ... +] +``` + +### File Metadata + +**GET** `/api/files/{file_id}/metadata` + +Retrieve metadata for a specific file. + +**Response**: +```json +{ + "document_type": "invoice", + "date": "2023-04-10", + "vendor": "Acme Corp", + "amount": "$1,234.56", + "extracted_text": "..." +} +``` + +### Process Control + +**POST** `/api/files/{file_id}/reprocess` + +Reprocess a specific file. + +**Response**: +```json +{ + "success": true, + "message": "File queued for reprocessing" +} +``` + +## Error Handling + +Errors follow standard HTTP status codes with descriptive messages: + +```json +{ + "detail": "File not found", + "status_code": 404 +} +``` + +## Rate Limiting + +The API implements rate limiting to ensure system stability. If you exceed the limits, you'll receive a `429 Too Many Requests` response. + +## Client Libraries + +For easy integration with your applications, consider using one of our client libraries: + +- Python: [github.com/christianlouis/docunova-python-client](https://github.com/christianlouis/docunova-python-client) + +## Further Assistance + +For additional help with the API, please contact our support team or refer to the [Development Guide](../CONTRIBUTING.md). diff --git a/docs/ConfigurationGuide.md b/docs/ConfigurationGuide.md index c0af39a3..688bdb1e 100644 --- a/docs/ConfigurationGuide.md +++ b/docs/ConfigurationGuide.md @@ -28,17 +28,6 @@ DocuNova can monitor multiple IMAP mailboxes for document attachments. Each mail | `IMAP1_PASSWORD` | IMAP password (first mailbox). | `*******` | | `IMAP1_SSL` | Use SSL (`true`/`false`). | `true` | | `IMAP1_POLL_INTERVAL_MINUTES` | Frequency in minutes to poll for new mail. | `5` | -| `IMAP1_DELETE_AFTER_PROCESS` | Delete emails after processing (`true`/`false`). | `false` | - -To configure additional mailboxes, use `IMAP2_`, `IMAP3_`, etc. - -### OpenAI & Azure Document Intelligence - -| **Variable** | **Description** | **How to Obtain** | -|---------------------------------|------------------------------------------|--------------------------------------------------------------------------| -| `OPENAI_API_KEY` | OpenAI API key for GPT metadata extraction. | [OpenAI API keys](https://platform.openai.com/account/api-keys) | -| `AZURE_DOCUMENT_INTELLIGENCE_KEY` | Azure Document Intelligence API key for OCR. | [Azure Portal](https://portal.azure.com/) | -| `AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT` | Endpoint URL for Azure Doc Intelligence API. | [Azure Portal](https://portal.azure.com/) | ### Authentication @@ -49,6 +38,14 @@ To configure additional mailboxes, use `IMAP2_`, `IMAP3_`, etc. | `AUTHENTIK_CLIENT_SECRET` | Client secret for Authentik OAuth2. | | `AUTHENTIK_CONFIG_URL` | Configuration URL for Authentik OpenID Connect. | +### OpenAI & Azure Document Intelligence + +| **Variable** | **Description** | **How to Obtain** | +|---------------------------------|------------------------------------------|--------------------------------------------------------------------------| +| `OPENAI_API_KEY` | OpenAI API key for GPT metadata extraction. | [OpenAI API keys](https://platform.openai.com/account/api-keys) | +| `AZURE_DOCUMENT_INTELLIGENCE_KEY` | Azure Document Intelligence API key for OCR. | [Azure Portal](https://portal.azure.com/) | +| `AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT` | Endpoint URL for Azure Doc Intelligence API. | [Azure Portal](https://portal.azure.com/) | + ### Paperless NGX | **Variable** | **Description** | diff --git a/docs/DeploymentGuide.md b/docs/DeploymentGuide.md new file mode 100644 index 00000000..e85fb104 --- /dev/null +++ b/docs/DeploymentGuide.md @@ -0,0 +1,142 @@ +# Deployment Guide + +This guide provides instructions for deploying DocuNova in various environments. + +## Prerequisites + +- Docker and Docker Compose +- Access to required external services (if configured): + - OpenAI API + - Azure Document Intelligence + - Dropbox API + - Nextcloud instance + - Paperless NGX instance + - SMTP server (for email notifications) + - IMAP server(s) (for email attachment processing) + +## Docker Deployment + +Docker is the recommended deployment method for DocuNova. + +### Step 1: Clone the Repository + +```bash +git clone https://github.com/christianlouis/document-processor.git +cd document-processor +``` + +### Step 2: Configure Environment Variables + +Create a `.env` file based on the example: + +```bash +cp .env.example .env +``` + +Edit the `.env` file with your configuration settings. See the [Configuration Guide](ConfigurationGuide.md) for details. + +### Step 3: Run with Docker Compose + +```bash +docker-compose up -d +``` + +This will start: +- The DocuNova API server +- A worker for background tasks +- Redis for message broker and result storage +- Gotenberg for PDF processing + +### Step 4: Verify the Installation + +Access the web interface at `http://localhost:8000` and the API documentation at `http://localhost:8000/docs`. + +## Production Considerations + +### Reverse Proxy Setup + +For production use, we recommend setting up a reverse proxy (like Nginx or Traefik) to handle HTTPS and domain routing: + +```nginx +server { + listen 80; + server_name docunova.example.com; + + location / { + proxy_pass http://localhost:8000; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +### Persistent Storage + +The Docker setup uses volumes for persistent storage. For production, consider: + +```yaml +volumes: + - /path/to/persistent/storage:/workdir +``` + +### Security + +1. **Always use HTTPS** in production +2. Enable authentication by setting `AUTH_ENABLED=true` +3. Use strong passwords for all services +4. Limit access to the Docker host +5. Regularly update the application and dependencies + +## Scaling + +For high-volume deployments: + +1. Increase worker processes by adding more worker containers: + +```yaml +worker: + image: christianlouis/document-processor:latest + deploy: + replicas: 3 +``` + +2. Consider using dedicated Redis and database servers +3. Monitor system performance and adjust resources as needed + +## Monitoring + +Monitor your DocuNova deployment using: + +- Docker's built-in logging: `docker-compose logs -f` +- Container metrics: `docker stats` +- External monitoring tools like Prometheus and Grafana + +## Backup Procedures + +Regularly back up the following: + +1. The `/workdir` directory containing all processed documents +2. The database file (if using SQLite) or database contents (if using another DBMS) +3. The `.env` configuration file + +## Updates + +To update DocuNova to a newer version: + +```bash +# Pull the latest changes +git pull + +# Pull the latest Docker images +docker-compose pull + +# Restart the services +docker-compose down +docker-compose up -d +``` + +## Troubleshooting + +See the [Troubleshooting](Troubleshooting.md) guide for common deployment issues and solutions. diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 00000000..c2e695c8 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,34 @@ +# DocuNova Documentation + +Welcome to the DocuNova documentation. This directory contains comprehensive guides to help you install, configure, and use DocuNova effectively. + +## Available Documentation + +- [User Guide](UserGuide.md) - How to use DocuNova's features and interface +- [API Documentation](API.md) - Complete API reference for developers +- [Deployment Guide](DeploymentGuide.md) - How to deploy DocuNova in various environments +- [Configuration Guide](ConfigurationGuide.md) - All available configuration options +- [Troubleshooting](Troubleshooting.md) - Solutions to common issues + +## Additional Resources + +- [Contributing Guide](../CONTRIBUTING.md) - How to contribute to DocuNova development +- [License Information](../LICENSE) - Apache License 2.0 details + +## Support + +If you need additional assistance beyond what's covered in these guides: + +1. Check the [GitHub repository](https://github.com/christianlouis/document-processor) for updates and issues +2. Contact the developer through the information provided on the About page + +## Screenshots + +![Upload Interface](upload-view.png) +*DocuNova's upload interface* + +![Files View](files-view.png) +*Document management interface* + +![Workflow Diagram](workflow-diagram.png) +*DocuNova processing workflow* diff --git a/docs/Troubleshooting.md b/docs/Troubleshooting.md new file mode 100644 index 00000000..03237492 --- /dev/null +++ b/docs/Troubleshooting.md @@ -0,0 +1,175 @@ +# Troubleshooting Guide + +This document provides solutions to common problems encountered when using DocuNova. + +## Common Issues + +### Application Won't Start + +#### Symptoms +- Docker containers exit immediately +- Web interface not accessible +- Error logs show startup failures + +#### Possible Solutions +1. **Check environment variables** + ```bash + docker-compose config + ``` + Ensure all required variables are set properly in your `.env` file. + +2. **Verify permissions** + ```bash + ls -la /path/to/workdir + ``` + Make sure the application has write permissions to the working directory. + +3. **Check port availability** + ```bash + netstat -tuln | grep 8000 + ``` + Ensure the port isn't already in use by another application. + +### Document Upload Fails + +#### Symptoms +- Error messages during upload +- Files appear to upload but aren't processed +- Browser console errors + +#### Possible Solutions +1. **Check file size limits** + - Default maximum file size is 100MB + - Adjust `client_max_body_size` in your reverse proxy configuration + +2. **Verify storage space** + ```bash + df -h + ``` + Ensure there's sufficient disk space available. + +3. **Check worker process** + ```bash + docker-compose logs worker + ``` + Verify the Celery worker is running and processing tasks. + +### OCR or Text Extraction Issues + +#### Symptoms +- Documents upload but text isn't extracted +- Poor quality text extraction +- API errors related to Azure services + +#### Possible Solutions +1. **Verify API credentials** + Check the Azure Document Intelligence API key and endpoint in your `.env` file. + +2. **Check document quality** + - Ensure documents are clearly scanned + - Try preprocessing images to improve quality before upload + +3. **Test API connectivity** + ```bash + curl -X GET -H "Ocp-Apim-Subscription-Key: YOUR_KEY" "YOUR_ENDPOINT" + ``` + Ensure the API is accessible from your server. + +### Email Integration Problems + +#### Symptoms +- Email attachments aren't being processed +- IMAP connection errors in logs +- Authentication failures + +#### Possible Solutions +1. **Verify IMAP settings** + Check host, port, username, and password in your configuration. + +2. **Test IMAP connectivity** + ```bash + telnet mail.example.com 993 + ``` + Ensure the IMAP server is accessible. + +3. **Enable less secure apps** + For Gmail and some providers, you may need to enable access for less secure apps or use app-specific passwords. + +4. **Check firewall settings** + Ensure your server can make outbound connections to the mail server. + +### Storage Integration Issues + +#### Symptoms +- Files aren't appearing in Dropbox/Nextcloud/Paperless +- Authentication errors in logs +- API rate limiting errors + +#### Possible Solutions +1. **Verify API credentials** + Double-check all API keys, tokens, and secrets. + +2. **Check access permissions** + Ensure the application has write permissions to the specified folders. + +3. **Refresh tokens** + For OAuth-based services like Dropbox, try generating new refresh tokens. + +4. **Examine detailed logs** + ```bash + docker-compose logs worker | grep -i dropbox + ``` + Look for specific error messages related to the service. + +## Database Issues + +#### Symptoms +- Application errors related to database connections +- Missing or corrupt data +- Slow performance + +#### Possible Solutions +1. **Check database connection string** + Verify the `DATABASE_URL` variable in your `.env` file. + +2. **Inspect database integrity** + ```bash + sqlite3 database.db "PRAGMA integrity_check;" + ``` + (For SQLite databases) + +3. **Perform database migrations** + ```bash + docker-compose exec api alembic upgrade head + ``` + Ensure the database schema is up-to-date. + +## Authentication Problems + +#### Symptoms +- Unable to log in +- Redirect loops during authentication +- OAuth errors + +#### Possible Solutions +1. **Verify Authentik configuration** + Check client ID, client secret, and configuration URL. + +2. **Check callback URLs** + Ensure the redirect URIs are correctly configured in your OAuth provider. + +3. **Clear browser cookies and cache** + Authentication issues can sometimes be resolved by clearing browser data. + +## Getting Additional Help + +If you continue to experience issues after trying these solutions: + +1. **Check the logs** for detailed error messages + ```bash + docker-compose logs --tail=100 + ``` + +2. **Open an issue** on the [GitHub repository](https://github.com/christianlouis/document-processor/issues) + +3. **Contact the developer** via the information provided on the About page diff --git a/docs/UserGuide.md b/docs/UserGuide.md new file mode 100644 index 00000000..9de2e124 --- /dev/null +++ b/docs/UserGuide.md @@ -0,0 +1,69 @@ +# User Guide + +This guide helps you get started with using DocuNova for document management and processing. + +## Getting Started + +DocuNova offers an intuitive web interface for uploading, managing, and processing your documents. + +### Accessing the System + +1. Navigate to your DocuNova instance (typically at `http://your-server-address:8000`) +2. If authentication is enabled, you'll be prompted to log in using your credentials + +### Main Interface + +DocuNova features a simple navigation system with the following main sections: +- **Home**: Dashboard and overview +- **Upload**: For adding new documents to the system +- **Files**: For viewing and managing processed documents +- **About**: Information about DocuNova + +## Uploading Documents + +### Web Upload + +1. Navigate to the **Upload** page +2. Drag and drop files onto the upload area, or click to browse your files +3. Select files to upload (supported formats include PDF, Word documents, images, etc.) +4. Click "Upload" to begin the process +5. Your documents will be processed automatically according to the system configuration + +### Email Attachments + +If configured, DocuNova can automatically fetch documents from email attachments: + +1. Send an email with attachments to the configured email account +2. DocuNova will poll the mailbox at the configured interval +3. Attachments will be automatically downloaded and processed +4. No further action is required + +## Managing Documents + +The **Files** page provides access to all processed documents: + +1. Navigate to the **Files** page +2. Use the search box to find specific documents +3. Click on any file to view its details +4. Sort the list by any column by clicking on the column header + +## Document Processing Features + +Depending on the system configuration, DocuNova can perform: + +- **OCR** - Extract text from images and scanned PDFs +- **Metadata Extraction** - Automatically identify document types, dates, and other key information +- **PDF Conversion** - Convert various file formats to PDF +- **Document Distribution** - Store documents in Dropbox, Nextcloud, or Paperless NGX + +## API Access + +For programmatic access, DocuNova provides a comprehensive REST API: + +1. Navigate to `/docs` on your DocuNova instance +2. The interactive Swagger documentation allows you to test API endpoints directly +3. Obtain an API token if authentication is enabled + +## Troubleshooting + +If you encounter issues while using DocuNova, please refer to the [Troubleshooting Guide](Troubleshooting.md).