fix(protected-phrases): isolate phrase generation per book

Run full-book candidate generation inside worker-owned sessions so each book commits independently during backfills. Abort recalculation when a book has no indexed chapters to preserve existing phrase data, and update admin/UI tests for the new generation flow.
feat(admin): simplify phrase generation by removing missing phrases endpoint
2026-07-12 13:34:04 -04:00 · 2026-07-12 13:34:04 -04:00 · 2026-07-12 13:34:04 -04:00 · 2026-07-12 13:34:04 -04:00 · 2026-07-12 13:34:04 -04:00 · 2026-07-12 13:34:04 -04:00
1 changed files with 18 additions and 18 deletions
@@ -8,11 +8,11 @@
      },
      "locked": {
        "dir": "pkgs/firefox-addons",
-        "lastModified": 1782964936,
-        "narHash": "sha256-wXEBDr7/dFQYhVpDwCKc9fkrYQQE4x0bdirX1bsLBGA=",
+        "lastModified": 1783828963,
+        "narHash": "sha256-eTytzcUJCaDUZ3/9EF0+V3fvlikQMQBwiX1Sx4Gy+No=",
        "owner": "rycee",
        "repo": "nur-expressions",
-        "rev": "64feee871e0373dd6121e412c3fb12e372d1bfb5",
+        "rev": "8d61e9afde605cd6c22dab68b83d7a71f0a6c5b2",
        "type": "gitlab"
      },
      "original": {
@@ -29,11 +29,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1783005591,
-        "narHash": "sha256-NcLHV5uBAeggDUE2wPbKszjfyaSLsoqaYt7izOphkZw=",
+        "lastModified": 1783823409,
+        "narHash": "sha256-OI4IkRjRXa1e7hYmCGJDPDq5H/kPwhsyoS80cNUF9fI=",
        "owner": "nix-community",
        "repo": "home-manager",
-        "rev": "f469c79b955609d6a8fdd9e689be76a93b1621d7",
+        "rev": "7566825d4652a1b885bd4ce65bd9e8def432fec9",
        "type": "github"
      },
      "original": {
@@ -47,11 +47,11 @@
        "nixpkgs": "nixpkgs"
      },
      "locked": {
-        "lastModified": 1782562157,
-        "narHash": "sha256-a7+T6QSeowynwZ1ZJJbP8T8ntAytvrui8kFGJmIZt2c=",
+        "lastModified": 1783792734,
+        "narHash": "sha256-50rvY9GdFvpYDcMLcD/4cWSi0hVxArT5wsGlVsHy8eY=",
        "owner": "nixos",
        "repo": "nixos-hardware",
-        "rev": "a9cf7546a938c737b079e738de73934a13de9784",
+        "rev": "8efb4337e857949f4cfac86d12ef1066f417f31f",
        "type": "github"
      },
      "original": {
@@ -76,11 +76,11 @@
    },
    "nixpkgs-master": {
      "locked": {
-        "lastModified": 1783021952,
-        "narHash": "sha256-8PghAtSGGZ0umfVI8Qbd7ZbFrfZPiH1UwtVbgLeikDA=",
+        "lastModified": 1783874024,
+        "narHash": "sha256-Fd8rPvyBv6JjcO/nZxZiFQan6Fww/jAF4TYj0Th/Yfo=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "f136374c679c54171a3ace589d15e9e79a8bd086",
+        "rev": "0b4f03c64b236e4ba4252414274e92796c300124",
        "type": "github"
      },
      "original": {
@@ -108,11 +108,11 @@
    },
    "nixpkgs_2": {
      "locked": {
-        "lastModified": 1782723713,
-        "narHash": "sha256-oPXCU/SSUokcGaJREHibG1CBX3+s/W7orDWQOZDsEeQ=",
+        "lastModified": 1783776592,
+        "narHash": "sha256-UgCQzxeWI75XM8G+hPrPh+MKzEPjG3SpAj7dtqSbksA=",
        "owner": "nixos",
        "repo": "nixpkgs",
-        "rev": "b5aa0fbd538984f6e3d201be0005b4463d8b09f8",
+        "rev": "e7a3ca8092b61ff85b6a45bf863ea2b2d6a661b3",
        "type": "github"
      },
      "original": {
@@ -141,11 +141,11 @@
        ]
      },
      "locked": {
-        "lastModified": 1782165805,
-        "narHash": "sha256-478kKQBvK6SYTOdN2h9jhKJv94nbXRbFMfuL1WshErg=",
+        "lastModified": 1783174389,
+        "narHash": "sha256-aCWC8ngycU7OdJrU2+Je3qf+1a2ykuBvpPhZT/9tXMc=",
        "owner": "Mic92",
        "repo": "sops-nix",
-        "rev": "56b24064fdcaedca53553b1a6d607fd23b613a24",
+        "rev": "f1406619a3884cd5c47992a70b8b35c9c0fcb4c9",
        "type": "github"
      },
      "original": {
Author	SHA1	Message	Date
Richie	1841ac69fa	fix(protected-phrases): isolate phrase generation per book treefmt / nix fmt (pull_request) Failing after 5s Details pytest / pytest (pull_request) Failing after 36s Details build_systems / build-brain (pull_request) Successful in 51s Details build_systems / build-bob (pull_request) Successful in 55s Details build_systems / build-rhapsody-in-green (pull_request) Successful in 1m30s Details build_systems / build-jeeves (pull_request) Successful in 3m9s Details Run full-book candidate generation inside worker-owned sessions so each book commits independently during backfills. Abort recalculation when a book has no indexed chapters to preserve existing phrase data, and update admin/UI tests for the new generation flow.	2026-07-12 13:34:04 -04:00
Richie	b22357f215	feat(admin): simplify phrase generation by removing missing phrases endpoint	2026-07-12 13:34:04 -04:00
Richie	cf23a04d36	refactor(ebook): remove spaCy-ner attributes from PhraseCandidate and related functions	2026-07-12 13:34:04 -04:00
Richie	728f4a143c	ran treefmt	2026-07-12 13:34:04 -04:00
Richie	73f08f31cf	feat(dependencies): update sqlalchemy to use asyncio and add aiosqlite and pytest-asyncio to dev dependencies	2026-07-12 13:34:04 -04:00
Richie	ef0638b23b	test(ebook): cover protected phrases and migrate suite to async Add test_protected_phrases.py covering phrase-matching behavior in the RAG engine, and update the existing ebook_search tests to use the async SQLAlchemy engine/session (create_async_engine, AsyncSession) and async HTTP paths.	2026-07-12 13:34:04 -04:00
Richie	e9d1a911df	feat(ebook): add additional tokens to junk tokens configuration	2026-07-12 13:34:04 -04:00
Richie	e4d7fe07eb	feat(vscode): add new words to spell checker configuration	2026-07-12 13:34:04 -04:00
Richie	271de355fe	feat(ebook): improve search UX with grid actions and Enter-to-submit Add a two-column grid layout for the admin protected-phrases actions and submit the search form on Enter (Shift+Enter for newline).	2026-07-12 13:34:04 -04:00
Richie	76a9c70a47	feat(ebook): migrate to async DB/HTTP and parallelize phrase pipeline Convert the ebook-search web app to async end to end and add concurrency to the protected-phrase extraction and judging pipeline so large books no longer block the event loop or the UI. ORM / infra: - Add get_async_postgres_engine and factor shared URL/connect_args building into build_postgres_url (reused by the sync and async engine builders) - Add async FastAPI session helpers (get_async_db, AsyncDbSession) with expire_on_commit=False to avoid implicit IO under asyncio App: - Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings, answer, rerank and LLM calls; convert handlers to async - Share a single httpx.AsyncClient in app state for LLM requests; size the connection pool for concurrent phrase-judging workers - Add judge_tasks: run per-book judging as tracked background tasks so a book already being judged isn't double-queued Protected phrases: - Add a process pool (pool.py) and worker-count config (extraction/judge book/phrase workers) to parallelize candidate generation and judging - Split admin actions into all/missing variants for generation and judging Config: - Add protected_phrase_extraction_workers, phrase_judge_book_workers, phrase_judge_phrase_workers	2026-07-12 13:34:04 -04:00
RichieandClaude Fable 5	903ad749b0	feat(ebook): install sqlalchemy[asyncio] in the ebook-search container The async engine needs greenlet at runtime, which the asyncio extra provides. Test-only deps (aiosqlite, pytest-asyncio) stay out of the image. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-07-12 13:34:04 -04:00
Richie	5978373f8f	feat(python-env): remove unused dependencies and clean up package list	2026-07-12 13:34:04 -04:00
Richie	cdba465550	feat(extraction): add cached YAKE extractor for improved performance	2026-07-12 13:34:04 -04:00
Richie	f82bac2165	feat(ebook): add junk tokens for improved phrase matching	2026-07-12 13:34:04 -04:00
Richie	decd0a6165	feat(orm): add pool_size parameter to get_postgres_engine for connection management	2026-07-12 13:34:04 -04:00
Richie	338ff60f80	Add models and database persistence for protected phrase extraction - Introduced dataclasses for phrase candidates, judgments, and matches in `models.py`. - Implemented database operations for candidate and protected phrases in `store.py`, including loading, saving, and deleting phrases. - Enhanced text normalization functions in `text_normalization.py` with detailed docstrings. - Refactored search functionality to utilize new models and methods for detecting protected phrases.	2026-07-12 13:34:04 -04:00
Richie	2c30e5ee5a	feat(ebook): add phrase matching display and update search result structure	2026-07-12 13:34:04 -04:00
Richie	609ab75658	refactor(protected-phrases): extract config and text normalization helpers	2026-07-12 13:34:04 -04:00
Richie	2a1764c4df	feat(ebook): update protected phrases with additional tokens and phrases	2026-07-12 13:34:04 -04:00
Richie	34512611bb	feat(ebook): enhance phrase judgment logging with failure tracking	2026-07-12 13:34:04 -04:00
Richie	5ca8e4d591	feat(ebook): add Docker packaging and lifecycle tooling Add a self-contained docker/ package for running the ebook search app against the existing Postgres database on jeeves: - Dockerfile: python:3.14-slim image, non-root user, runs the FastAPI app on port 8070 - docker-compose.yml: service definition with library volume mount, BM25 index volume, .env loading, and a /health healthcheck - containers.py: Typer CLI (ebook-search-containers) for build/start/ stop/restart/logs/ps lifecycle management - README.md: usage and configuration docs	2026-07-12 13:34:04 -04:00
Richie	a105fe3016	feat(common): add get_repo_dir function and corresponding tests	2026-07-12 13:34:04 -04:00
Richie	120469d697	updated dependencies and added .dockerignore	2026-07-12 13:34:04 -04:00
Richie	365384aab1	feat(ebook): implement phrase matching functionality and UI enhancements	2026-07-12 13:34:04 -04:00
Richie	fff50084c1	feat(ebook): add admin and book-detail UI for protected phrase pipeline Expose the protected phrase extraction pipeline through the web UI: - Admin routes: POST /admin/build-phrases, /admin/generate-ngrams, and /admin/judge-ngrams, each wrapping the protected_phrases.lib backfill helpers, committing on success, rolling back and rendering an error partial on failure, and reporting per-book/candidate/mention counts. - Book detail page: show candidate, judged, and protected phrase counts, list top candidate n-grams (with kept/rejected status) and protected phrases, and add a POST /books/{id}/recalculate-phrases action that clears and regenerates candidates, then redirects back with a status message. - Admin template: add Generate/Judge n-gram buttons. Also reflows admin.html to 2-space HTML formatting.	2026-07-12 13:34:04 -04:00
Richie	dd3c3b8f3c	feat(ebook): add protected phrase extraction library with config-driven tuning Refactor protected phrase handling from a single module into a python/ebook_search/protected_phrases package covering extraction, storage, and runtime matching. Phrase filtering is now data-driven via bundled TOML files: ignored_phrases, bad_starts, bad_ends, and most_common_words. Add phrase-tuning settings to EbookSearchConfig so candidate generation, scoring, LLM judging, and matching are configurable rather than hardcoded: token bounds, entity token limit, raw n-gram min count, frequency and chapter-spread score thresholds, candidate/LLM/target caps, confidence threshold, nesting defaults, and the phrase hit boost.	2026-07-12 13:34:04 -04:00
Richie	54ed90bfd4	fix(ebook): enhance EPUB ingestion with error handling and incrmental commits	2026-07-12 13:34:04 -04:00
Richie	bcf386a51c	feat(ebook): add phrase metadata tables for protected phrase matching Introduce four ORM models and their Alembic migration to support phrase-based query matching in the ebook RAG engine: - EbookCandidatePhrase: high-recall phrase candidates extracted per book, with source flags (ngram/yake/spacy/capitalized/metadata), scoring, and LLM judge results. - EbookProtectedPhrase: phrases accepted by the LLM judge, with canonical id, importance, and nesting controls. - EbookPhraseAlias: normalized aliases mapping to protected phrases. - EbookChunkPhraseMention: precomputed phrase occurrences within chunks. Export the new models from python.orm.richie and add a JSON_DOCUMENT helper (JSON with JSONB postgres variant) for storing sample contexts.	2026-07-12 13:34:04 -04:00
Richie	80a521f297	refactor: extract signal_alert into its own module treefmt / nix fmt (pull_request) Successful in 5s Details pytest / pytest (pull_request) Successful in 24s Details build_systems / build-bob (pull_request) Successful in 45s Details build_systems / build-brain (pull_request) Successful in 45s Details build_systems / build-rhapsody-in-green (pull_request) Successful in 58s Details build_systems / build-jeeves (pull_request) Successful in 2m15s Details treefmt / nix fmt (push) Successful in 5s Details build_systems / build-jeeves (push) Successful in 9s Details pytest / pytest (push) Successful in 24s Details build_systems / build-brain (push) Successful in 30s Details build_systems / build-bob (push) Successful in 32s Details build_systems / build-rhapsody-in-green (push) Successful in 44s Details Move signal_alert out of python/common.py into a dedicated python/signal_alert.py module and update its importers (validate_system.py, snapshot_manager.py) to the new path. Relocate the signal_alert tests from tests/test_common.py into tests/test_signal_alert.py, repatching python.signal_alert.logger and python.signal_alert.Apprise to match the new module.	2026-07-12 13:16:28 -04:00
Richie	4a410dbdf8	feat(postgres): add trust authentication for richie on 172.16.0.0/12 treefmt / nix fmt (pull_request) Successful in 5s Details pytest / pytest (pull_request) Successful in 23s Details build_systems / build-brain (pull_request) Successful in 44s Details build_systems / build-bob (pull_request) Successful in 45s Details build_systems / build-jeeves (pull_request) Successful in 2m20s Details build_systems / build-rhapsody-in-green (pull_request) Successful in 56s Details treefmt / nix fmt (push) Successful in 5s Details build_systems / build-rhapsody-in-green (push) Successful in 11s Details pytest / pytest (push) Successful in 24s Details build_systems / build-brain (push) Successful in 30s Details build_systems / build-bob (push) Successful in 33s Details build_systems / build-jeeves (push) Successful in 2m7s Details	2026-07-12 13:06:44 -04:00
Richie	138b79bf97	feat(flake): update lock file treefmt / nix fmt (pull_request) Successful in 5s Details pytest / pytest (pull_request) Successful in 1m12s Details build_systems / build-brain (pull_request) Successful in 2m46s Details build_systems / build-bob (pull_request) Successful in 3m4s Details build_systems / build-rhapsody-in-green (pull_request) Successful in 7m42s Details build_systems / build-jeeves (pull_request) Successful in 7m44s Details treefmt / nix fmt (push) Successful in 5s Details pytest / pytest (push) Successful in 23s Details build_systems / build-brain (push) Successful in 30s Details build_systems / build-bob (push) Successful in 32s Details build_systems / build-rhapsody-in-green (push) Successful in 45s Details build_systems / build-jeeves (push) Successful in 2m10s Details	2026-07-12 12:49:57 -04:00