Commit Graph
9 Commits
Author SHA1 Message Date
Richie bfb3463fd0 feat(ebook): migrate to async DB/HTTP and parallelize phrase pipeline
Convert the ebook-search web app to async end to end and add concurrency
to the protected-phrase extraction and judging pipeline so large books no
longer block the event loop or the UI.

ORM / infra:
- Add get_async_postgres_engine and factor shared URL/connect_args building
  into build_postgres_url (reused by the sync and async engine builders)
- Add async FastAPI session helpers (get_async_db, AsyncDbSession) with
  expire_on_commit=False to avoid implicit IO under asyncio

App:
- Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings,
  answer, rerank and LLM calls; convert handlers to async
- Share a single httpx.AsyncClient in app state for LLM requests; size the
  connection pool for concurrent phrase-judging workers
- Add judge_tasks: run per-book judging as tracked background tasks so a
  book already being judged isn't double-queued

Protected phrases:
- Add a process pool (pool.py) and worker-count config
  (extraction/judge book/phrase workers) to parallelize candidate generation
  and judging
- Split admin actions into all/missing variants for generation and judging

Config:
- Add protected_phrase_extraction_workers, phrase_judge_book_workers,
  phrase_judge_phrase_workers
2026-07-12 17:51:06 -04:00
Richie bcb8b7b169 Add models and database persistence for protected phrase extraction
- Introduced dataclasses for phrase candidates, judgments, and matches in `models.py`.
- Implemented database operations for candidate and protected phrases in `store.py`, including loading, saving, and deleting phrases.
- Enhanced text normalization functions in `text_normalization.py` with detailed docstrings.
- Refactored search functionality to utilize new models and methods for detecting protected phrases.
2026-07-12 17:49:52 -04:00
Richie 222ae5755e feat(ebook): add phrase matching display and update search result structure 2026-07-12 17:49:52 -04:00
Richie d03daeb66d feat(ebook): implement phrase matching functionality and UI enhancements 2026-07-12 17:49:52 -04:00
Richie 6bc30115d9 fix(ebook-search): code clean up to impove reliablty and readabilty 2026-06-18 12:45:56 -04:00
Richie 68b3a38b81 converting to pydantic-settings 2026-06-14 21:29:45 -04:00
Richie 2995a75748 fixed test
treefmt / nix fmt (pull_request) Failing after 5s
pytest / pytest (pull_request) Successful in 28s
build_systems / build-bob (pull_request) Successful in 48s
build_systems / build-brain (pull_request) Successful in 53s
build_systems / build-leviathan (pull_request) Successful in 54s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m1s
build_systems / build-jeeves (pull_request) Successful in 2m43s
2026-06-14 15:41:09 -04:00
Richie 11cbe31152 improved queary for vector search 2026-06-14 15:40:31 -04:00
Richie aed1e14d95 built rag search setup 2026-06-14 15:40:04 -04:00