Move protected phrase detection into the retrieval gather so it runs
concurrently with vector and BM25 candidates instead of sequentially
before them. Make the search API accept real bool form fields for
rerank/phrase_matching, gate phrase matching on both the request and
config kill switch, and reflow log f-strings for readability.
Run full-book candidate generation inside worker-owned sessions so each book commits independently during backfills. Abort recalculation when a book has no indexed chapters to preserve existing phrase data, and update admin/UI tests for the new generation flow.
Convert the ebook-search web app to async end to end and add concurrency
to the protected-phrase extraction and judging pipeline so large books no
longer block the event loop or the UI.
ORM / infra:
- Add get_async_postgres_engine and factor shared URL/connect_args building
into build_postgres_url (reused by the sync and async engine builders)
- Add async FastAPI session helpers (get_async_db, AsyncDbSession) with
expire_on_commit=False to avoid implicit IO under asyncio
App:
- Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings,
answer, rerank and LLM calls; convert handlers to async
- Share a single httpx.AsyncClient in app state for LLM requests; size the
connection pool for concurrent phrase-judging workers
- Add judge_tasks: run per-book judging as tracked background tasks so a
book already being judged isn't double-queued
Protected phrases:
- Add a process pool (pool.py) and worker-count config
(extraction/judge book/phrase workers) to parallelize candidate generation
and judging
- Split admin actions into all/missing variants for generation and judging
Config:
- Add protected_phrase_extraction_workers, phrase_judge_book_workers,
phrase_judge_phrase_workers
- Introduced dataclasses for phrase candidates, judgments, and matches in `models.py`.
- Implemented database operations for candidate and protected phrases in `store.py`, including loading, saving, and deleting phrases.
- Enhanced text normalization functions in `text_normalization.py` with detailed docstrings.
- Refactored search functionality to utilize new models and methods for detecting protected phrases.
Expose the protected phrase extraction pipeline through the web UI:
- Admin routes: POST /admin/build-phrases, /admin/generate-ngrams, and
/admin/judge-ngrams, each wrapping the protected_phrases.lib backfill
helpers, committing on success, rolling back and rendering an error
partial on failure, and reporting per-book/candidate/mention counts.
- Book detail page: show candidate, judged, and protected phrase counts,
list top candidate n-grams (with kept/rejected status) and protected
phrases, and add a POST /books/{id}/recalculate-phrases action that
clears and regenerates candidates, then redirects back with a status
message.
- Admin template: add Generate/Judge n-gram buttons.
Also reflows admin.html to 2-space HTML formatting.
Refactor protected phrase handling from a single module into a
python/ebook_search/protected_phrases package covering extraction,
storage, and runtime matching. Phrase filtering is now data-driven via
bundled TOML files: ignored_phrases, bad_starts, bad_ends, and
most_common_words.
Add phrase-tuning settings to EbookSearchConfig so candidate generation,
scoring, LLM judging, and matching are configurable rather than hardcoded:
token bounds, entity token limit, raw n-gram min count, frequency and
chapter-spread score thresholds, candidate/LLM/target caps, confidence
threshold, nesting defaults, and the phrase hit boost.
Introduce four ORM models and their Alembic migration to support
phrase-based query matching in the ebook RAG engine:
- EbookCandidatePhrase: high-recall phrase candidates extracted per book,
with source flags (ngram/yake/spacy/capitalized/metadata), scoring, and
LLM judge results.
- EbookProtectedPhrase: phrases accepted by the LLM judge, with canonical
id, importance, and nesting controls.
- EbookPhraseAlias: normalized aliases mapping to protected phrases.
- EbookChunkPhraseMention: precomputed phrase occurrences within chunks.
Export the new models from python.orm.richie and add a JSON_DOCUMENT
helper (JSON with JSONB postgres variant) for storing sample contexts.
- implement FastAPI and HTMX lobby and game interfaces
- support up to four human and AI-controlled players
- add configurable rules, victory conditions, and expansion modules
- support validated JSON cards, patrons, objectives, and outposts
- Deleted base template (base.html) and all related contact templates (contact_detail.html, contact_form.html, contact_list.html, graph.html, need_list.html).
- Removed partial templates for managing contacts and needs (contact_table.html, manage_needs.html, manage_relationships.html, need_items.html).
- Eliminated contact API service configuration (contact_api.nix) from the NixOS setup.
The gitea runner containers have no docker access, so build the test
env with uv from the existing lockfile and run pytest directly:
- test_ebook_search workflow: uv sync --locked + uv run pytest, with
UV_PYTHON_DOWNLOADS=never so uv uses the nix-provided python 3.14
- add uv to the runner hostPackages (needs a jeeves rebuild to apply)
- ignore nested **/.venv in .dockerignore (uv sync creates one in
python/ebook_search/docker)
- document the uv test commands in the docker README; the docker test
image remains for validating the image itself
Add a self-contained docker/ package for running the ebook search app
against the existing Postgres database on jeeves:
- Dockerfile: python:3.14-slim image, non-root user, runs the FastAPI
app on port 8070
- docker-compose.yml: service definition with library volume mount,
BM25 index volume, .env loading, and a /health healthcheck
- containers.py: Typer CLI (ebook-search-containers) for build/start/
stop/restart/logs/ps lifecycle management
- README.md: usage and configuration docs
Move signal_alert out of python/common.py into a dedicated
python/signal_alert.py module and update its importers
(validate_system.py, snapshot_manager.py) to the new path.
Relocate the signal_alert tests from tests/test_common.py into
tests/test_signal_alert.py, repatching python.signal_alert.logger and
python.signal_alert.Apprise to match the new module.
Replace old_installer.py with a curses TUI installer packaged as a
one-file PyInstaller binary (python/installer/build.py, wrapped by
python/installer/package.nix). The default .#installer is patched to run
on foreign Linux live media; the new .#installer-nixos variant keeps its
Nix store interpreter so it runs on NixOS.
Add systems/iso, a minimal NixOS install CD with kernel 6.18 and ZFS 2.4
matching the deployed systems and the installer on PATH; build it with
nix build .#iso. Shared logging and subprocess helpers move out of
common.py into python/logging_config.py and python/process.py.
Add a pipeline to load HAProxy `option httplog` lines into the Richie
database so bot/crawler traffic can be analyzed.
- model: HaproxyRequest mirroring the httplog format, with a unique
line_hash dedup key and indexes on common filter columns
- migration: create the haproxy_request table (unique line_hash + indexes)
- haproxy_logs package:
- parser: httplog line -> columns, strips the journald prefix and
hashes the normalized line
- ingest: batched, idempotent insert that skips rows whose line_hash
already exists, so re-ingesting the same logs is a no-op
- cli: ingest-only `haproxy-logs` command reading stdin or a file
- tests: parsing of a real GPTBot line and idempotent re-ingestion
load_gold_queries now skips blank and `//` comment lines so the committed
section separator in queries.jsonl no longer breaks dataset/load-test loading.
Update tests left stale by the search refactor (6bc3011):
- pass the now-required rank_constant to reciprocal_rank_fusion
- expect bm25_candidates to receive the full query and drop the removed
"BM25 query preparation" timing step
- assert reranking is enabled by default
Add a Typer CLI script that drives POST /search on a running server at a
configurable concurrency and reports latency percentiles (p50/p90/p95/p99),
throughput, and HTTP status distribution. Queries are drawn from the shared
eval JSONL set so load testing and evaluation exercise the same questions.
Richie