Introduce four ORM models and their Alembic migration to support
phrase-based query matching in the ebook RAG engine:
- EbookCandidatePhrase: high-recall phrase candidates extracted per book,
with source flags (ngram/yake/spacy/capitalized/metadata), scoring, and
LLM judge results.
- EbookProtectedPhrase: phrases accepted by the LLM judge, with canonical
id, importance, and nesting controls.
- EbookPhraseAlias: normalized aliases mapping to protected phrases.
- EbookChunkPhraseMention: precomputed phrase occurrences within chunks.
Export the new models from python.orm.richie and add a JSON_DOCUMENT
helper (JSON with JSONB postgres variant) for storing sample contexts.
Add a pipeline to load HAProxy `option httplog` lines into the Richie
database so bot/crawler traffic can be analyzed.
- model: HaproxyRequest mirroring the httplog format, with a unique
line_hash dedup key and indexes on common filter columns
- migration: create the haproxy_request table (unique line_hash + indexes)
- haproxy_logs package:
- parser: httplog line -> columns, strips the journald prefix and
hashes the normalized line
- ingest: batched, idempotent insert that skips rows whose line_hash
already exists, so re-ingesting the same logs is a no-op
- cli: ingest-only `haproxy-logs` command reading stdin or a file
- tests: parsing of a real GPTBot line and idempotent re-ingestion
Richie