Commit Graph
4 Commits
Author SHA1 Message Date
Richie c7cd63f8e4 feat(ebook): migrate to async DB/HTTP and parallelize phrase pipeline
Convert the ebook-search web app to async end to end and add concurrency
to the protected-phrase extraction and judging pipeline so large books no
longer block the event loop or the UI.

ORM / infra:
- Add get_async_postgres_engine and factor shared URL/connect_args building
  into build_postgres_url (reused by the sync and async engine builders)
- Add async FastAPI session helpers (get_async_db, AsyncDbSession) with
  expire_on_commit=False to avoid implicit IO under asyncio

App:
- Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings,
  answer, rerank and LLM calls; convert handlers to async
- Share a single httpx.AsyncClient in app state for LLM requests; size the
  connection pool for concurrent phrase-judging workers
- Add judge_tasks: run per-book judging as tracked background tasks so a
  book already being judged isn't double-queued

Protected phrases:
- Add a process pool (pool.py) and worker-count config
  (extraction/judge book/phrase workers) to parallelize candidate generation
  and judging
- Split admin actions into all/missing variants for generation and judging

Config:
- Add protected_phrase_extraction_workers, phrase_judge_book_workers,
  phrase_judge_phrase_workers
2026-07-09 11:04:59 -04:00
Richie d9115f7c91 feat(ebook): add admin and book-detail UI for protected phrase pipeline
Expose the protected phrase extraction pipeline through the web UI:

- Admin routes: POST /admin/build-phrases, /admin/generate-ngrams, and
  /admin/judge-ngrams, each wrapping the protected_phrases.lib backfill
  helpers, committing on success, rolling back and rendering an error
  partial on failure, and reporting per-book/candidate/mention counts.
- Book detail page: show candidate, judged, and protected phrase counts,
  list top candidate n-grams (with kept/rejected status) and protected
  phrases, and add a POST /books/{id}/recalculate-phrases action that
  clears and regenerates candidates, then redirects back with a status
  message.
- Admin template: add Generate/Judge n-gram buttons.

Also reflows admin.html to 2-space HTML formatting.
2026-07-09 11:04:59 -04:00
Richie 03f610feed feat(ui): refactor HTML templates for consistent structure and improved styling
treefmt / nix fmt (pull_request) Successful in 5s
pytest / pytest (pull_request) Successful in 37s
build_systems / build-leviathan (pull_request) Successful in 1m20s
build_systems / build-jeeves (pull_request) Successful in 2m57s
build_systems / build-brain (pull_request) Successful in 3m3s
build_systems / build-rhapsody-in-green (pull_request) Successful in 3m25s
build_systems / build-bob (pull_request) Successful in 32m7s
treefmt / nix fmt (push) Successful in 5s
pytest / pytest (push) Successful in 26s
build_systems / build-brain (push) Successful in 31s
build_systems / build-bob (push) Successful in 34s
build_systems / build-leviathan (push) Successful in 41s
build_systems / build-rhapsody-in-green (push) Successful in 46s
build_systems / build-jeeves (push) Successful in 2m10s
2026-06-29 10:23:13 -04:00
Richie 6715bbf0a5 build api and frountend 2026-06-14 15:40:04 -04:00