Subscription quota is now checked before the file is written to disk,
so users who have exceeded their quota do not waste bandwidth or disk
I/O. The post-write cleanup path for quota rejections is no longer
needed and has been removed.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add SubscriptionPlan model and subscription_plans table (migration 015)
- Add billing cycle/period/allow_overage fields to UserProfile (migration 016)
- Add subscription_overage_percent config field (replaces overage_factor)
- Rewrite check_upload_allowed: use overage_percent, yearly carry-over, no daily cap
- Add seed_default_plans(), _plan_to_dict(), get_year_file_count(), _months_elapsed()
- Update get_tier/get_all_tiers to be DB-first with TIER_DEFAULTS fallback
- Add TIER_DEFAULTS alias (TIERS kept for backward compat)
- New /api/plans/ CRUD endpoints (admin-only except list/get)
- New /admin/plans Plan Designer page with Alpine.js UI
- Add Plan Designer link to admin navigation in base.html
- Remove 'Files per day' row from pricing comparison table
- Add billing cycle + period start to admin users edit modal
- Seed default plans on startup in lifespan handler
- Rewrite docs/SubscriptionTiers.md with full plan/overage/API docs
- Fix all tests in test_subscription.py (remove daily cap tests, add overage/carry-over tests)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Use shared _require_admin from admin_users in subscriptions API endpoint
- Remove unnecessary Alpine.js hidden-div workaround in pricing.html
- Replace fragile string replace for OCR page count with proper Jinja {:,} format
- Improve comment wording in upload quota cleanup code
- Extract _scalar_count() helper in subscription.py to reduce repetition
- Add aria-valuemin='0' to all progressbar elements in subscription/index templates
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add Free / Starter / Professional / Business tiers with lifetime, daily, and monthly
file limits (app/utils/subscription.py)
- Add subscription_tier column to UserProfile model + migration 014
- Enforce quotas at upload time (HTTP 402 on violation) in /api/ui-upload
- New REST API: GET /api/subscriptions/tiers, /my, /platform (admin)
- New pages: /pricing (marketing, public) and /subscription (per-user status)
- Enhanced dashboard: SaaS stats (files today/month, OCR count, active users)
in multi-user mode; original single-user layout preserved
- Admin users page: show Plan badge, allow tier editing via dropdown
- Navigation: add Pricing link + subscription icon in user header
- Tests: 23 unit tests for subscription tier logic
- Docs: docs/SubscriptionTiers.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add UserProfile model (app/models.py) with per-user settings: display_name, daily_upload_limit, notes, is_blocked
- Add Alembic migration 013_add_user_profiles for the new table
- Add REST API at /api/admin/users/ with list, get, upsert (PUT), delete endpoints (admin-only)
- Add HTML template admin_users.html with Alpine.js: filterable user list, paginated table, edit/create modal, delete confirmation modal
- Add view handler at /admin/users (admin-only redirect guard)
- Register routers in app/api/__init__.py and app/views/__init__.py
- Add 'Users' link to admin nav dropdown in base.html (desktop + mobile)
- Add 27 tests covering auth, list, get, upsert, delete, and model constraints
- Register UserProfile in conftest.py model imports
- Document new endpoints in docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add inline safety comment for noqa: S608 (table_name from inspect)
- Fix HTTPException detail to be a string (not dict)
- Add aria-label to migration progress bar
- Rename _noop to _NoOpContextManager in tests
- Add explanatory comment for zip(strict=False)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add a guided database configuration wizard and a data migration tool that
allows users to:
- Build database connection strings through a step-by-step UI
- Test database connections before applying
- Preview and execute data migrations from SQLite to PostgreSQL/MySQL
- Copy to clipboard for easy .env file updates
New files:
- app/utils/db_wizard.py — connection string builder, parser, and tester
- app/utils/db_migrate.py — table-by-table data migration utility
- app/api/database.py — REST API endpoints for wizard operations
- app/views/db_wizard.py — view route for the wizard page
- frontend/templates/db_wizard.html — multi-tab wizard UI
- tests/test_db_wizard.py — unit tests for db_wizard utilities
- tests/test_db_migrate.py — unit tests for db_migrate utilities
- tests/test_db_wizard_api.py — integration tests for API and views
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add GET /api/similarity/pairs endpoint for corpus-wide pair discovery
- Add /similarity view route and similarity_dashboard.html template
- Add Similarity link to desktop and mobile nav menus
- Register compute_embedding as a tracked FileProcessingStep
- Update compute_embedding task with update_step_status calls
- Add compute_embedding to flow visualization in _compute_processing_flow
- Add backfill_missing_embeddings periodic beat task (every 5 min)
- Return clear message when embedding not yet computed in similar docs API
- Fix all tests to use pre-computed embeddings (no lazy API calls)
- Add tests for similarity pairs, backfill task, and embedding-not-computed
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add near_duplicate_threshold config setting (default 0.85)
- New GET /api/duplicates endpoint listing all exact-duplicate groups
- New GET /api/files/{id}/duplicates endpoint returning exact + near-duplicates
- POST /api/ui-upload now returns immediate exact-duplicate warning (respects ENABLE_DEDUPLICATION)
- New /duplicates management UI with Exact Duplicates tab and Near-Duplicate Finder tab
- Add Duplicates link in admin nav menu (desktop + mobile)
- Document new config options in ConfigurationGuide.md and .env.demo
- 20 new tests covering all acceptance criteria
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Add WebhookConfig model, CRUD API endpoints, HMAC-SHA256 signed delivery,
and Celery-based async dispatch with retry/backoff for document events
(document.uploaded, document.processed, document.failed).
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add health check endpoint at GET /api/diagnostic/health
- Auth-protected via @require_login (no-op when AUTH_ENABLED=False)
- Checks database (SELECT 1) and Redis (ping) with 2s timeouts
- Returns healthy/degraded/unhealthy with per-check detail
- Returns HTTP 503 when database is down, 200 otherwise
- 7 new unit tests covering all status scenarios
- Update docs/API.md with Grafana/monitoring integration notes
- Fixes test_cors_headers_absent_when_disabled CI timeout
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default per_page from 50 to 25
- Rename total_items → total, total_pages → pages in pagination response
- Add next/previous URL fields to pagination response
- Update view and template to use new field names
- Update tests and API docs
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add tags, sender, text_quality filters to search API and Meilisearch client
- Add sender and ocr_text_length to Meilisearch filterable attributes
- Expand saved search allowed filter keys to include q, document_type, language, sender, text_quality
- Add filters panel and saved searches UI to the Search view template
- Add tests for new search filters, saved search keys, and search view elements
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add date range (date_from/date_to), storage provider, and tags filters to GET /api/files
- Add SavedSearch model and migration (005_add_saved_searches)
- Add CRUD API endpoints for saved searches at /api/saved-searches
- Update files.html template with new filter controls and saved searches UI
- Update files view to pass new filter parameters to template
- Add comprehensive tests for all new functionality (26 tests)
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Change default `version` param from 'original' to 'processed' so
GET /api/files/{id}/download (no param) returns the processed file
- Update docstring to reflect new default
- Add tests: ?version=processed, default→processed, invalid→400
- Fix test_file_download_missing_mime_type to use explicit ?version=original
- Add File Download section to docs/API.md
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
The upload retry logic now checks file_record.processed_file_path first
(the GPT-suggested filename stored during finalization), before falling
back to legacy hash-based and original-filename-based path patterns.
This fixes the case where the processed file has a different name than
the original (e.g., '2023-10-01_Unknown.pdf' vs 'cable_graphic.pdf')
and the retry couldn't find the file on disk.
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
Both get_file_preview and download_file were ignoring the DB-stored
original_file_path and processed_file_path fields, instead relying on
local_filename (a temp path that may be gone) and guessing patterns for
the processed file. This caused "Processed file not found" and
potentially "Original file not found" in the /files/{id} view even when
the files existed at their stored paths.
- version=original: check original_file_path first, fall back to local_filename
- version=processed: check processed_file_path first, fall back to
hash/filename guessing patterns
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Update _compute_processing_flow to recognize process_with_ocr as the OCR
stage and remap legacy process_with_azure_document_intelligence log entries
for backward compatibility
- Normalize legacy OCR step name in _compute_step_summary log fallback
- Add process_with_ocr to REAL_MAIN_STEPS/REAL_STEPS in step_manager,
file_status, and file_queries (keeping legacy name for old DB entries)
- Update retry logic in api/files.py to retry failed OCR via process_with_ocr
(handles both step names as aliases)
- Fix process_document.py to log process_with_ocr as skipped (not azure step)
for the local text extraction path
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
- Add app/utils/ocr_language_manager.py: detects tessdata dir, downloads
missing .traineddata files via wget/curl from tessdata_fast GitHub repo,
pre-downloads EasyOCR models, exposes async background-thread helper
- TesseractOCRProvider.process() calls ensure_tesseract_languages() before
running pytesseract; raises clear error if languages remain unavailable
- EasyOCRProvider.process() logs informational message when models download
- app/main.py: calls ensure_ocr_languages_async() at startup
- app/utils/settings_sync.py: triggers language re-check after every
settings reload so UI changes take effect without container restart
- app/api/settings.py: adds POST /api/settings/install-ocr-languages
endpoint for on-demand language installation from the admin UI
- Dockerfile: adds wget for runtime tessdata downloads
- docs/ConfigurationGuide.md: documents automatic language download
- tests/test_ocr_language_manager.py: 29 unit tests
Co-authored-by: christianlouis <361235+christianlouis@users.noreply.github.com>
copilot-swe-agent[bot]