From e46f9b9a21e5838f883eca4370445e0f1b57e6c9 Mon Sep 17 00:00:00 2001 From: Christian Krakau-Louis Date: Sat, 23 May 2026 02:29:46 +0200 Subject: [PATCH] docs: extend product roadmap and milestones --- MILESTONES.md | 184 ++++++++++++++++++++++++--------------- ROADMAP.md | 236 ++++++++++++++++++++++---------------------------- 2 files changed, 218 insertions(+), 202 deletions(-) diff --git a/MILESTONES.md b/MILESTONES.md index f80a04f1..e818e038 100644 --- a/MILESTONES.md +++ b/MILESTONES.md @@ -1,6 +1,6 @@ # DocuElevate Milestones -**Last Updated:** 2026-02-08 +**Last Updated:** 2026-05-23 This document outlines the release milestones, versioning strategy, and detailed feature breakdown for DocuElevate. @@ -19,17 +19,15 @@ DocuElevate follows [Semantic Versioning 2.0.0](https://semver.org/): --- -## Current Release: v0.5.0 (February 2026) +## Current State (Continuous Releases) -### Status: Stable -- Production-ready document processing -- Multi-provider storage support -- **Database-backed settings management with encryption** -- **Setup wizard for first-time configuration** -- **Admin UI for runtime configuration** -- **Automated semantic versioning and releases** -- OAuth2 authentication with admin group support -- Basic web UI and REST API +DocuElevate ships continuously via automated semantic versioning. Use **GitHub Releases** for the latest build artifacts and **GitHub Milestones** (below) for roadmap tracking. + +### Last Shipped Milestone: v0.5.0 (Released February 8, 2026) +- Database-backed settings management with encryption +- Setup wizard for first-time configuration +- Admin UI for runtime configuration +- Release automation via semantic-release ### Important Note on Versioning As of February 2026, DocuElevate uses **automated semantic versioning**: @@ -130,87 +128,78 @@ As of February 2026, DocuElevate uses **automated semantic versioning**: ## Upcoming Milestones -### v0.6.0 - Enhanced Search & UI Improvements (April 2026) -**Target Date:** April 1, 2026 +### v0.6.0 - Clarity: Enhanced Search & UI (Target: July 31, 2026) +**Target Date:** July 31, 2026 **Status:** 📋 Planned -**Theme:** User Experience, Search, Performance +**Theme:** Search, Discovery, Modern UX +**Epic:** #863 #### Goals -- Implement full-text search across documents -- Responsive mobile interface -- Dark mode support -- Document preview in browser -- Performance optimizations -- Improved error handling and user feedback +- Hybrid discovery: keyword + semantic search, fast filtering, saved searches +- Preview-first UX (open, skim, and act quickly) +- Modern UX polish (accessibility, responsiveness, performance) #### Deliverables -- Full-text search API and UI -- Advanced filtering capabilities -- Responsive CSS framework integration -- Dark mode toggle -- In-browser document viewer -- Loading states and progress indicators -- Performance benchmarks -- Mobile-optimized interface +- Semantic search foundation (vectorization + ranking signals) +- Saved searches / smart views +- In-browser preview + “quick actions” (tag, route, export) +- Bulk operations and pagination improvements +- UX polish (dark mode/accessibility where applicable) #### Breaking Changes -- API response format changes for search endpoints (documented) +- Potential pagination/search response changes (must be versioned and documented) #### Migration Path -- Search endpoint changes will be versioned (/api/v1/search → /api/v2/search) -- Old endpoints deprecated but functional for 2 releases +- Version endpoints where needed and keep previous versions working for at least 2 minor milestones --- -### v0.4.5 - Workflow Automation (June 2026) -**Target Date:** June 1, 2026 +### v0.7.0 - Conductor: Workflow Automation & Integrations (Target: September 30, 2026) +**Target Date:** September 30, 2026 **Status:** 📋 Planned **Theme:** Automation, Integration, Webhooks +**Epic:** #864 #### Goals -- Custom processing pipelines -- Conditional routing based on document type -- Webhook support for external integrations -- Rule-based classification -- Scheduled batch processing +- First-class workflow model (steps, state, retries) that matches what the system actually executes +- Workflow-aware UI status, retries, and observability +- Webhooks + event-driven automation foundations #### Deliverables -- Pipeline configuration UI -- Webhook management interface -- Rule engine for document routing -- Batch processing scheduler -- Integration examples and templates -- Webhook payload documentation +- Workflow object model and storage +- Workflow-aware file detail view + status dashboard +- Scheduling primitives (recurring jobs / delayed runs) +- Webhook system (outbound events + inbound triggers) +- Integration templates and documentation --- -### v0.7.0 - Advanced AI & Multi-language (August 2026) -**Target Date:** August 1, 2026 +### v0.8.0 - Signal: AI Quality, RAG, and Multi-language (Target: November 30, 2026) +**Target Date:** November 30, 2026 **Status:** 📋 Planned -**Theme:** AI Enhancement, Internationalization +**Theme:** AI Quality, Retrieval, Internationalization +**Epic:** #865 #### Goals -- Custom AI model support -- Multi-language OCR -- Document similarity detection -- Duplicate detection -- UI internationalization (i18n) -- API localization +- “Chat with Library” foundations (retrieval + UI) +- Local AI options for privacy-sensitive setups +- Measurable AI quality (confidence + human review loop) +- Expand multilingual capability across OCR + UI #### Deliverables -- Custom model integration API -- Multi-language OCR configuration -- Similarity algorithm implementation -- Duplicate detection service -- Translation framework (10+ languages) -- Localized documentation +- Vector DB integration and embeddings pipeline +- Chat UI foundations and retrieval API +- Confidence scoring + human review/edit loop for extracted fields +- Multi-language OCR configuration improvements +- Expanded i18n coverage + localized docs --- -### v1.0.0 - Enterprise Edition (November 2026) -**Target Date:** November 1, 2026 +### v1.0.0 - Summit: Enterprise Edition (Target: March 31, 2027) +**Target Date:** March 31, 2027 **Status:** 📋 Planned **Theme:** Enterprise Features, Scalability, Multi-tenancy +**Epic:** #866 This is our first major release, marking production-ready enterprise capabilities. @@ -244,6 +233,60 @@ This is our first major release, marking production-ready enterprise capabilitie --- +### v2.0.0 - Horizon: Platform Expansion (Target: September 30, 2027) +**Target Date:** September 30, 2027 +**Status:** 📋 Planned +**Theme:** Ecosystem, Platform, Distribution +**Epic:** #867 + +#### Goals +- Make DocuElevate extensible by design (plugins + templates) +- Expand integrations and developer experience +- Harden multi-surface experiences (web, mobile, extension, CLI) as a cohesive product + +#### Deliverables +- Plugin system foundations and public extension points +- Template library for pipelines/workflows + “starter kits” +- Integration hub patterns (webhooks, events, connectors) +- SDK + documentation for extensions + +--- + +### v2.1.0 - Sentinel: Governance & Policy (Target: March 31, 2028) +**Target Date:** March 31, 2028 +**Status:** 📋 Planned +**Theme:** Governance, Compliance, Policy-driven Automation +**Epic:** #868 + +#### Goals +- Make governance first-class (retention, legal hold, PII workflows) +- Provide tamper-evident auditing and admin controls +- Introduce policy-driven approvals for sensitive automation + +#### Deliverables +- Retention policies + legal hold primitives +- PII detection + redaction workflows +- Tamper-evident audit trails + admin activity feed +- Policy-as-code concepts for workflows (with approval gates) + +--- + +### v3.0.0 - Constellation: Integration Hub & Agent Platform (Target: September 30, 2028) +**Target Date:** September 30, 2028 +**Status:** 📋 Planned +**Theme:** Ecosystem, Agents, Interoperability +**Epic:** #869 + +#### Goals +- Make DocuElevate the “system of record” for document intelligence in an organization +- Support external automation ecosystems (Zapier/Make/n8n) and agent runtimes +- Provide a clean interoperability layer for modern AI tools + +#### Deliverables +- DocuElevate MCP server (search, retrieve, summarize, route) and documentation +- Connector marketplace concepts (curated + community) +- Event stream + webhooks at scale (delivery guarantees, retries, signing) + ## Release Process ### Automated Semantic Versioning (v0.6.0+) @@ -255,15 +298,15 @@ Starting with v0.6.0, releases are fully automated using `python-semantic-releas 4. **Automatic Updates**: - Updates `VERSION` file - Generates/updates `CHANGELOG.md` - - Creates Git tag (e.g., `v0.6.0`) + - Creates Git tag (e.g., `v0.173.1`) - Creates GitHub Release with notes - Triggers Docker image builds 5. **No Manual Steps**: VERSION and CHANGELOG are never edited manually ### Version Bump Rules -- `feat:` commits → Minor version (0.5.0 → 0.6.0) -- `fix:`, `perf:` → Patch version (0.5.0 → 0.5.1) -- `feat!:`, `BREAKING CHANGE:` → Major version (0.5.0 → 1.0.0) +- `feat:` commits → Minor version (e.g., 0.173.1 → 0.174.0) +- `fix:`, `perf:` → Patch version (e.g., 0.173.1 → 0.173.2) +- `feat!:`, `BREAKING CHANGE:` → Major version (e.g., 0.173.1 → 1.0.0) - Other types (docs, chore, etc.) → No version bump ### Pre-release Checklist (Automated) @@ -297,10 +340,13 @@ Starting with v0.6.0, releases are fully automated using `python-semantic-releas | v0.3.2 | 2026-02-06 | Security Updates | Released | | v0.3.3 | 2026-02-08 | Drag-and-Drop Upload | Released | | v0.5.0 | 2026-02-08 | **Settings & Encryption** | **Released** | -| v0.6.0 | 2026-04 | Search & UX | Planned | -| v0.7.0 | 2026-08 | Advanced AI | Planned | -| v1.0.0 | 2026-11 | Enterprise | Planned | -| v2.0.0 | 2027-Q3 | Platform Expansion | Future | +| v0.6.0 | 2026-07-31 | **Clarity:** Search & UX | Planned | +| v0.7.0 | 2026-09-30 | **Conductor:** Workflows & Integrations | Planned | +| v0.8.0 | 2026-11-30 | **Signal:** AI Quality, RAG, Multi-language | Planned | +| v1.0.0 | 2027-03-31 | **Summit:** Enterprise | Planned | +| v2.0.0 | 2027-09-30 | **Horizon:** Platform Expansion | Future | +| v2.1.0 | 2028-03-31 | **Sentinel:** Governance & Policy | Future | +| v3.0.0 | 2028-09-30 | **Constellation:** Integration Hub & Agents | Future | --- diff --git a/ROADMAP.md b/ROADMAP.md index d5eb0a83..ecc7ef64 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,169 +1,139 @@ # DocuElevate Roadmap -**Last Updated:** 2026-02-08 -**Version:** 1.0 +**Last Updated:** 2026-05-23 +**Version:** 2.0 ## Vision DocuElevate aims to be the premier open-source intelligent document processing platform, providing seamless integration with cloud storage providers, advanced AI-powered metadata extraction, and enterprise-grade security and scalability. +## How to Read This Roadmap + +DocuElevate ships frequently (automated semantic versioning), so this roadmap is organized around **milestone outcomes** and **themes**, not exact build numbers. + +- **P0** = required for the milestone to feel “done” +- **P1** = strongly desired; may slip if needed +- **P2** = nice-to-have / opportunistic + +For the detailed milestone breakdown and target dates, see [MILESTONES.md](MILESTONES.md). + ## Release Naming Each major milestone release carries a codename to anchor key project moments. These names appear in the status dashboard, build metadata, and changelog. For details, see [docs/ReleaseNaming.md](docs/ReleaseNaming.md). -| Version Range | Codename | Theme | -|---------------|---------------|--------------------------------------------------| -| 0.5.x | **Foundation** | Core platform, multi-provider storage, AI, UI | -| 0.6.x | **Clarity** | Enhanced search, filtering, UI/UX improvements | -| 0.7.x | **Conductor** | Workflow automation, pipelines, rule-based logic | -| 1.0.x | **Summit** | Enterprise features, multi-tenancy, RBAC | -| 1.1.x | **Bridge** | Collaboration, sharing, analytics | -| 2.0.x | **Horizon** | On-premise AI, platform expansion | +| Milestone | Codename | Theme | +|----------|------------------|-------| +| v0.6.0 | **Clarity** | Search, discovery, and modern UX | +| v0.7.0 | **Conductor** | Workflows, orchestration, and integrations | +| v0.8.0 | **Signal** | AI quality, multilingual, and “Chat with Library” foundations | +| v1.0.0 | **Summit** | Enterprise readiness (multi-tenancy, RBAC, scaling) | +| v2.0.0 | **Horizon** | Platform expansion and ecosystem maturity | +| v2.1.0+ | **Sentinel** | Governance, compliance, and policy-driven automation | +| v3.0.0 | **Constellation**| Integration hub, agents, and interoperability | -## Current Status (v0.5.0 "Foundation") +## Current Product Capabilities (Today) ### Core Features ✅ -- Multi-provider document storage (Dropbox, Google Drive, OneDrive, Nextcloud, S3, etc.) -- IMAP email integration for document ingestion -- OCR processing via Azure Document Intelligence -- AI-powered metadata extraction via OpenAI -- PDF conversion via Gotenberg -- Web UI for document upload and management -- **Database-backed settings management with admin UI** -- **Fernet encryption for sensitive configuration** -- **Setup wizard for first-time installation** -- REST API with OpenAPI documentation -- Celery-based async task processing -- OAuth2 authentication via Authentik with admin group support +- Multi-channel ingestion (web upload, IMAP email, watched folders, mobile, CLI, API) +- Multi-engine OCR + AI extraction with configurable providers +- Customizable processing pipelines and routing rules +- Full-text search and document discovery +- Multi-destination distribution (cloud providers, DMS, protocols, email) +- Admin UI for configuration (database-backed settings, encryption, setup wizard) +- Production hardening building blocks (CI/CD, security docs, deployment guides) -## Short-term Goals (Q1-Q2 2026) - v0.4.x to v0.5.x "Foundation" +## Feature Landscape (Themes) -### Quality & Stability 🎯 -- **Test Coverage** (High Priority) - - [ ] Achieve 80% code coverage for core modules - - [ ] Add integration tests for all storage providers - - [ ] Add end-to-end workflow tests - - [ ] Performance benchmarks and load testing +### 1) Search & Discovery +- **P0:** hybrid search (keyword + semantic), fast faceted filtering, saved searches +- **P1:** “explain results” (why a document matched), query suggestions, pinned results +- **P2:** entity search (people/companies/amounts/dates) and graph-style exploration -- **Code Quality** (High Priority) - - [ ] Enable strict linting in CI/CD - - [ ] Refactor large modules for better maintainability - - [ ] Add comprehensive type hints - - [ ] Improve error handling and user feedback +### 2) AI Quality & Trust +- **P0:** confidence scoring, human review/edit loop, extraction evaluation harness +- **P1:** per-document-type schemas/templates, active learning (feedback improves extraction) +- **P2:** multi-model routing (choose model by cost/latency/accuracy per step) -- **Security** (Critical Priority) - - [x] Fix known vulnerabilities in dependencies - - [ ] Implement rate limiting on API endpoints - - [ ] Add CSRF protection - - [ ] Security audit by external party - - [ ] Implement API key rotation - - [ ] Add audit logging for sensitive operations +### 3) Workflow Automation & Orchestration +- **P0:** first-class workflow model (steps, state, retries), workflow-aware UI status +- **P1:** visual workflow builder, scheduling, webhooks, and event-driven triggers +- **P2:** agentic workflows (“autopilot” suggestions with approval gates) -- **Release Automation** (Completed ✅) - - [x] Implement semantic-release for automated versioning - - [x] Add conventional commit validation - - [x] Automate CHANGELOG generation - - [x] Integrate Docker builds with releases +### 4) Integrations & Ecosystem (Including MCP) +- **P0:** stable webhooks + outbound actions (Slack/Teams, email, DMS), bi-directional sync where supported +- **P1:** “Integration Hub” (Zapier/Make/n8n style), connector templates, secrets handling patterns +- **P2:** **MCP**: ship a DocuElevate MCP server (search, retrieve, summarize, route) + allow MCP tools as pipeline steps -### Features - v0.4.0 -- **Enhanced Search & Filtering** → _preparing for v0.6.0 "Clarity"_ - - [ ] Full-text search across documents - - [ ] Advanced filtering by metadata, tags, date ranges - - [ ] Saved search queries - - [ ] Bulk operations on search results +### 5) Governance, Compliance, and Security +- **P0:** audit trails, tamper-evident logs, API key lifecycle/rotation, admin activity feed +- **P1:** retention policies, legal hold, PII detection + redaction, data residency controls +- **P2:** compliance packs (SOC2/GDPR/HIPAA), BYOK/KMS integration paths -- **Improved UI/UX** - - [ ] Responsive mobile interface - - [ ] Dark mode support - - [ ] Document preview in browser - - [ ] Drag-and-drop file upload - - [ ] Progress indicators for long-running tasks - - [ ] Real-time notifications via WebSocket +### 6) Enterprise & Scale +- **P0:** multi-tenancy, RBAC, horizontal scaling reference architecture +- **P1:** SCIM provisioning, SAML/Okta/Azure AD hardening, quotas/billing at org level +- **P2:** multi-region deployment patterns and disaster recovery playbooks -### Features - v0.5.0 "Foundation" -- **Workflow Automation** → _evolving into v0.7.0 "Conductor"_ - - [ ] Custom processing pipelines - - [ ] Conditional routing based on document type - - [ ] Scheduled batch processing - - [ ] Webhook support for external integrations - - [ ] Rule-based document classification +## Release Plan (Extended) -- **Advanced AI Features** - - [ ] Custom AI models for specialized document types - - [ ] Multi-language OCR support - - [ ] Document similarity detection - - [ ] Automatic duplicate detection - - [ ] Intelligent document splitting +This plan extends the existing milestones with a clearer thematic arc and a forward-looking “beyond v2.0” horizon. Each milestone links to an epic issue that owns scope and sub-issues. -## Medium-term Goals (Q3-Q4 2026) - v1.0.x "Summit" +### v0.6.0 — Clarity (Search & UX) +- **Outcome:** users can reliably find, preview, and act on documents in seconds +- **P0:** semantic search + hybrid ranking, saved searches, fast filters, preview-first UX +- **P1:** bulk operations, query suggestions, accessibility/dark mode polish +- **Tracking:** GitHub milestone `v0.6.0 - Enhanced Search & UI` (epic #863) -### Enterprise Features - v1.0.0 "Summit" -- **Multi-tenancy** - - [ ] Organization/team management - - [ ] Role-based access control (RBAC) - - [ ] Per-tenant configuration - - [ ] Resource quotas and limits - - [ ] Audit logs per organization +### v0.7.0 — Conductor (Workflows & Integrations) +- **Outcome:** workflows are explicit, inspectable, and automatable end-to-end +- **P0:** workflow object model + workflow-aware UI status, retries, pipeline definitions +- **P1:** workflow builder, scheduling, inbound/outbound webhooks +- **P2:** integration templates + “connector marketplace” concepts +- **Tracking:** GitHub milestone `v0.7.0 - Workflow Automation` (epic #864) -- **Scalability** - - [ ] Horizontal scaling support - - [ ] Distributed task processing - - [ ] Caching layer (Redis/Memcached) - - [ ] Database connection pooling - - [ ] Message queue optimization +### v0.8.0 — Signal (AI Quality + “Chat with Library” Foundations) +- **Outcome:** AI features are measurable, reviewable, and safe to trust +- **P0:** vector DB + embeddings pipeline, chat UI foundations, local AI options +- **P1:** confidence scoring and review loop, extraction evaluation harness +- **P2:** multilingual UX + localization expansion +- **Tracking:** GitHub milestone `v0.8.0 - Advanced AI & Multi-language` (epic #865) -- **Advanced Integrations** - - [ ] Microsoft SharePoint integration - - [ ] Slack/Teams bot integration - - [ ] Zapier/Make.com integration - - [ ] Custom webhook receivers - - [ ] GraphQL API +### v1.0.0 — Summit (Enterprise Readiness) +- **Outcome:** teams can run DocuElevate with strong isolation, access control, and scale +- **P0:** multi-tenancy, RBAC, audit logging, scaling guidance +- **P1:** SSO hardening (SAML/LDAP), org-level quotas and billing hooks +- **P2:** enterprise admin experience (policies, approvals, reporting) +- **Tracking:** GitHub milestone `v1.0.0 - Enterprise Edition` (epic #866) -### Features - v1.1.0 "Bridge" -- **Collaboration** - - [ ] Document sharing with expiring links - - [ ] Comments and annotations - - [ ] Version history and rollback - - [ ] Real-time collaborative editing metadata - - [ ] Activity feed +### v2.0.0 — Horizon (Platform Expansion) +- **Outcome:** DocuElevate becomes an extensible platform with a thriving ecosystem +- **P0:** plugin system foundations, SDK + templates, deeper integrations +- **P1:** marketplace patterns, app distribution, mobile/extension maturity +- **P2:** multi-workspace experiences (personal + org) +- **Tracking:** GitHub milestone `v2.0.0 - Platform Expansion` (epic #867) -- **Reporting & Analytics** - - [ ] Processing statistics dashboard - - [ ] Storage usage analytics - - [ ] AI confidence scores and accuracy tracking - - [ ] Cost analysis per provider - - [ ] Export reports (PDF, CSV, Excel) +### v2.1.0+ — Sentinel (Governance & Policy) +- **Outcome:** governance becomes a first-class layer (policy-driven automation) +- **P0:** retention + legal hold, PII detection/redaction, tamper-evident audit trails +- **P1:** BYOK/KMS integration patterns, advanced access policies, compliance reporting +- **P2:** “policy as code” for workflows + approvals (change management) +- **Tracking:** GitHub milestone `v2.1.0 - Governance & Policy (Sentinel)` (epic #868) -## Long-term Goals (2027+) - v2.0+ "Horizon" +### v3.0.0 — Constellation (Integration Hub & Agent Platform) +- **Outcome:** DocuElevate plugs into modern automation and AI ecosystems as a first-class system of record +- **P0:** MCP server, durable event stream + production-grade webhooks +- **P1:** connector templates + curated catalog, agent-friendly permissioning and auditing +- **P2:** bring-your-own-agent patterns (sandboxing, scoped credentials) +- **Tracking:** GitHub milestone `v3.0.0 - Integration Hub & Agent Platform (Constellation)` (epic #869) -### Strategic Initiatives -- **On-Premise AI Models** - - [ ] Self-hosted OCR (Tesseract, EasyOCR) - - [ ] Local LLM integration (Ollama, LLaMA) - - [ ] GPU acceleration support - - [ ] Model fine-tuning interface - - [ ] Hybrid cloud/on-premise processing +## Research Bets (Optional / Experimental) -- **Advanced Document Management** - - [ ] Document lifecycle management - - [ ] Retention policies and auto-deletion - - [ ] Compliance templates (GDPR, HIPAA, SOC2) - - [ ] Digital signature support - - [ ] Encryption at rest and in transit +These are longer-horizon bets that should only be productized if they prove real user value. -- **Platform Expansion** - - [ ] Desktop applications (Electron) - - [ ] Mobile apps (iOS/Android) - - [ ] Browser extensions - - [ ] Command-line interface (CLI) - - [ ] VS Code extension for developers - -### Research & Innovation -- [ ] Machine learning for custom document types -- [ ] Blockchain for document provenance -- [ ] Federated learning for privacy-preserving AI -- [ ] Edge computing support -- [ ] Quantum-resistant encryption +- Knowledge graph over extracted entities (contracts ↔ vendors ↔ invoices) +- Auto-generated “case files” (collections) from intent (“tax 2025”, “project alpha”) +- Privacy-preserving learning (federated patterns) to improve extraction quality +- Document provenance (signing, attestations) and tamper detection ## Community & Ecosystem