100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Richie	e259526c38	feat(settings): add configuration to disable YAML schema detection for specific workflow files	2026-07-25 18:44:13 -04:00
Richie	e57895cc6e	patch cuda	2026-07-25 14:21:07 -04:00
Richie	89e24c45a0	chore: remove poetry from Python packages list	2026-07-25 13:59:24 -04:00
Richie	2e7b51ce1d	flake update	2026-07-25 13:47:36 -04:00
Richie	8eee5faf72	feat(chat): add optional response format parameter to request_chat_completion	2026-07-24 11:38:51 -04:00
Richie	c135821534	feat(ingest): ensure session rollback on ingestion error and log mention count	2026-07-24 11:38:51 -04:00
Richie	62bcc4e156	feat(judge_ngrams): enhance phrase judgment with alias filtering and strict boolean handling	2026-07-24 11:38:51 -04:00
Richie	31ecad881f	feat(extraction): add deduplication limit parameter to YAKE extractor	2026-07-24 11:38:51 -04:00
Richie	68c9693711	feat(config): add new scoring parameters for phrase extraction and matching	2026-07-24 11:38:51 -04:00
Richie	6a0e71a30d	feat(ebook-search): implement async file path resolution and EPUB discovery	2026-07-24 11:38:51 -04:00
Richie	8073144e2b	feat(prompts): implement TOML-backed prompt loading and refactor message generation	2026-07-24 11:38:51 -04:00
Richie	94f18722e4	feat(search): enhance phrase matching and reranking logic with improved async handling	2026-07-24 11:38:51 -04:00
Richie	e510c94b95	feat(dependencies): update database engine dependencies to support async operations	2026-07-24 11:38:51 -04:00
Richie	9f126fedc7	feat(ebook-search): implement async phrase matching for chunks and add ChunkPhraseHit model	2026-07-24 11:38:51 -04:00
Richie	e010756e09	refactor(ebook-search): simplify search and phrase matching	2026-07-24 11:38:51 -04:00
Richie	0028237579	perf(ebook-search): run phrase detection in parallel with retrieval ... Move protected phrase detection into the retrieval gather so it runs concurrently with vector and BM25 candidates instead of sequentially before them. Make the search API accept real bool form fields for rerank/phrase_matching, gate phrase matching on both the request and config kill switch, and reflow log f-strings for readability.	2026-07-24 11:38:51 -04:00
Richie	8f1a69529c	Refactor logging statements to use f-strings for improved readability and consistency across the codebase. This change enhances the clarity of log messages by directly embedding variable values, making it easier to trace and debug application behavior.	2026-07-24 11:38:51 -04:00
Richie	5210f00587	feat(dependencies): update sqlalchemy to use asyncio and add aiosqlite and pytest-asyncio	2026-07-24 11:38:51 -04:00
Richie	58be234d7f	fix(protected-phrases): isolate phrase generation per book ... Run full-book candidate generation inside worker-owned sessions so each book commits independently during backfills. Abort recalculation when a book has no indexed chapters to preserve existing phrase data, and update admin/UI tests for the new generation flow.	2026-07-24 11:38:50 -04:00
Richie	29a51eb1b8	feat(admin): simplify phrase generation by removing missing phrases endpoint	2026-07-24 11:38:50 -04:00
Richie	da78914a9f	refactor(ebook): remove spaCy-ner attributes from PhraseCandidate and related functions	2026-07-24 11:38:50 -04:00
Richie	c852f9136a	feat(dependencies): update sqlalchemy to use asyncio and add aiosqlite and pytest-asyncio to dev dependencies	2026-07-24 11:38:50 -04:00
Richie	565119ee45	test(ebook): cover protected phrases and migrate suite to async ... Add test_protected_phrases.py covering phrase-matching behavior in the RAG engine, and update the existing ebook_search tests to use the async SQLAlchemy engine/session (create_async_engine, AsyncSession) and async HTTP paths.	2026-07-24 11:38:50 -04:00
Richie	56c9bb2520	feat(ebook): add additional tokens to junk tokens configuration	2026-07-24 11:38:50 -04:00
Richie	f69c84e7b6	feat(vscode): add new words to spell checker configuration	2026-07-24 11:38:50 -04:00
Richie	5c826088c5	feat(ebook): improve search UX with grid actions and Enter-to-submit ... Add a two-column grid layout for the admin protected-phrases actions and submit the search form on Enter (Shift+Enter for newline).	2026-07-24 11:38:50 -04:00
Richie	2706c4417d	feat(ebook): migrate to async DB/HTTP and parallelize phrase pipeline ... Convert the ebook-search web app to async end to end and add concurrency to the protected-phrase extraction and judging pipeline so large books no longer block the event loop or the UI. ORM / infra: - Add get_async_postgres_engine and factor shared URL/connect_args building into build_postgres_url (reused by the sync and async engine builders) - Add async FastAPI session helpers (get_async_db, AsyncDbSession) with expire_on_commit=False to avoid implicit IO under asyncio App: - Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings, answer, rerank and LLM calls; convert handlers to async - Share a single httpx.AsyncClient in app state for LLM requests; size the connection pool for concurrent phrase-judging workers - Add judge_tasks: run per-book judging as tracked background tasks so a book already being judged isn't double-queued Protected phrases: - Add a process pool (pool.py) and worker-count config (extraction/judge book/phrase workers) to parallelize candidate generation and judging - Split admin actions into all/missing variants for generation and judging Config: - Add protected_phrase_extraction_workers, phrase_judge_book_workers, phrase_judge_phrase_workers	2026-07-24 11:38:50 -04:00
Richie	38c01ec121	feat(extraction): add cached YAKE extractor for improved performance	2026-07-24 11:38:50 -04:00
Richie	a9311a2f9e	feat(ebook): add junk tokens for improved phrase matching	2026-07-24 11:38:50 -04:00
Richie	584b209dfa	feat(orm): add pool_size parameter to get_postgres_engine for connection management	2026-07-24 11:38:50 -04:00
Richie	4861f58f27	Add models and database persistence for protected phrase extraction ... - Introduced dataclasses for phrase candidates, judgments, and matches in `models.py`. - Implemented database operations for candidate and protected phrases in `store.py`, including loading, saving, and deleting phrases. - Enhanced text normalization functions in `text_normalization.py` with detailed docstrings. - Refactored search functionality to utilize new models and methods for detecting protected phrases.	2026-07-24 11:38:50 -04:00
Richie	34e7823517	feat(ebook): add phrase matching display and update search result structure	2026-07-24 11:38:50 -04:00
Richie	11b5d5db3c	refactor(protected-phrases): extract config and text normalization helpers	2026-07-24 11:38:50 -04:00
Richie	8e2ca365c2	feat(ebook): update protected phrases with additional tokens and phrases	2026-07-24 11:38:50 -04:00
Richie	4c71508e73	feat(ebook): enhance phrase judgment logging with failure tracking	2026-07-24 11:38:50 -04:00
Richie	70bf8627a2	feat(ebook): implement phrase matching functionality and UI enhancements	2026-07-24 11:38:50 -04:00
Richie	a12e7461c5	feat(ebook): add admin and book-detail UI for protected phrase pipeline ... Expose the protected phrase extraction pipeline through the web UI: - Admin routes: POST /admin/build-phrases, /admin/generate-ngrams, and /admin/judge-ngrams, each wrapping the protected_phrases.lib backfill helpers, committing on success, rolling back and rendering an error partial on failure, and reporting per-book/candidate/mention counts. - Book detail page: show candidate, judged, and protected phrase counts, list top candidate n-grams (with kept/rejected status) and protected phrases, and add a POST /books/{id}/recalculate-phrases action that clears and regenerates candidates, then redirects back with a status message. - Admin template: add Generate/Judge n-gram buttons. Also reflows admin.html to 2-space HTML formatting.	2026-07-24 11:38:50 -04:00
Richie	6c8a4bfea7	feat(ebook): add protected phrase extraction library with config-driven tuning ... Refactor protected phrase handling from a single module into a python/ebook_search/protected_phrases package covering extraction, storage, and runtime matching. Phrase filtering is now data-driven via bundled TOML files: ignored_phrases, bad_starts, bad_ends, and most_common_words. Add phrase-tuning settings to EbookSearchConfig so candidate generation, scoring, LLM judging, and matching are configurable rather than hardcoded: token bounds, entity token limit, raw n-gram min count, frequency and chapter-spread score thresholds, candidate/LLM/target caps, confidence threshold, nesting defaults, and the phrase hit boost.	2026-07-24 11:38:50 -04:00
Richie	fbb1ebfd56	fix(ebook): enhance EPUB ingestion with error handling and incrmental commits	2026-07-24 11:38:50 -04:00
Richie	9ba8200673	feat(ebook): add phrase metadata tables for protected phrase matching ... Introduce four ORM models and their Alembic migration to support phrase-based query matching in the ebook RAG engine: - EbookCandidatePhrase: high-recall phrase candidates extracted per book, with source flags (ngram/yake/spacy/capitalized/metadata), scoring, and LLM judge results. - EbookProtectedPhrase: phrases accepted by the LLM judge, with canonical id, importance, and nesting controls. - EbookPhraseAlias: normalized aliases mapping to protected phrases. - EbookChunkPhraseMention: precomputed phrase occurrences within chunks. Export the new models from python.orm.richie and add a JSON_DOCUMENT helper (JSON with JSONB postgres variant) for storing sample contexts.	2026-07-24 11:38:50 -04:00
Richie	add7a6a848	feat(gems): add multiplayer gem game with custom content packs ... - implement FastAPI and HTMX lobby and game interfaces - support up to four human and AI-controlled players - add configurable rules, victory conditions, and expansion modules - support validated JSON cards, patrons, objectives, and outposts	2026-07-22 20:47:03 -04:00
Richie	247e951a27	Remove legacy HTML templates and API service configuration for contact management system ... - Deleted base template (base.html) and all related contact templates (contact_detail.html, contact_form.html, contact_list.html, graph.html, need_list.html). - Removed partial templates for managing contacts and needs (contact_table.html, manage_needs.html, manage_relationships.html, need_items.html). - Eliminated contact API service configuration (contact_api.nix) from the NixOS setup.	2026-07-15 15:28:15 -04:00
Richie	8c3de690c9	feat(ebook-search): set up uv cache management and pruning service	2026-07-12 17:26:55 -04:00
Richie	91575e3ab2	feat(ebook-search): update test workflow to use nix for dependency management	2026-07-12 17:10:47 -04:00
Richie	4799054601	ci(ebook-search): run tests via uv on the runner instead of docker ... The gitea runner containers have no docker access, so build the test env with uv from the existing lockfile and run pytest directly: - test_ebook_search workflow: uv sync --locked + uv run pytest, with UV_PYTHON_DOWNLOADS=never so uv uses the nix-provided python 3.14 - add uv to the runner hostPackages (needs a jeeves rebuild to apply) - ignore nested **/.venv in .dockerignore (uv sync creates one in python/ebook_search/docker) - document the uv test commands in the docker README; the docker test image remains for validating the image itself	2026-07-12 16:40:56 -04:00
Richie	e9636d19de	feat(ebook-search): add containerized test environment	2026-07-12 16:12:40 -04:00
Richie	2518ec8551	feat(python-env): remove unused dependencies and clean up package list	2026-07-12 16:12:40 -04:00
Richie	f5ec88f1e5	feat(ebook): add Docker packaging and lifecycle tooling ... Add a self-contained docker/ package for running the ebook search app against the existing Postgres database on jeeves: - Dockerfile: python:3.14-slim image, non-root user, runs the FastAPI app on port 8070 - docker-compose.yml: service definition with library volume mount, BM25 index volume, .env loading, and a /health healthcheck - containers.py: Typer CLI (ebook-search-containers) for build/start/ stop/restart/logs/ps lifecycle management - README.md: usage and configuration docs	2026-07-12 16:12:40 -04:00
Richie	f4c4b11ff8	updated dependencies and added .dockerignore	2026-07-12 16:12:40 -04:00
Richie	37f37c41ac	feat(common): add get_repo_dir function and corresponding tests	2026-07-12 14:24:05 -04:00
Richie	80a521f297	refactor: extract signal_alert into its own module ... Move signal_alert out of python/common.py into a dedicated python/signal_alert.py module and update its importers (validate_system.py, snapshot_manager.py) to the new path. Relocate the signal_alert tests from tests/test_common.py into tests/test_signal_alert.py, repatching python.signal_alert.logger and python.signal_alert.Apprise to match the new module.	2026-07-12 13:16:28 -04:00
Richie	4a410dbdf8	feat(postgres): add trust authentication for richie on 172.16.0.0/12	2026-07-12 13:06:44 -04:00
Richie	138b79bf97	feat(flake): update lock file	2026-07-12 12:49:57 -04:00
Richie	47f77443dd	feat(networking): configure unused network ports to prevent errors and optimize boot time	2026-07-09 10:45:08 -04:00
Richie	8548f4ff64	feat(programs): add codebase-memory-mcp to home packages	2026-07-02 16:08:12 -04:00
Richie	682960b4c9	feat(installer): refactor NixOS configurations to dynamically generate host systems	2026-07-02 14:33:45 -04:00
Richie	ddea13b93a	feat(installer): enhance NixOS installer with SSH access and encryption password input	2026-07-02 14:17:49 -04:00
Richie	b252b0bf6e	feat(installer): add one-file installer build and custom install ISO ... Replace old_installer.py with a curses TUI installer packaged as a one-file PyInstaller binary (python/installer/build.py, wrapped by python/installer/package.nix). The default .#installer is patched to run on foreign Linux live media; the new .#installer-nixos variant keeps its Nix store interpreter so it runs on NixOS. Add systems/iso, a minimal NixOS install CD with kernel 6.18 and ZFS 2.4 matching the deployed systems and the installer on PATH; build it with nix build .#iso. Shared logging and subprocess helpers move out of common.py into python/logging_config.py and python/process.py.	2026-07-02 14:17:49 -04:00
Richie	fbf288649a	feat(haproxy): simplify Gitea rate limiting and fix rule ordering ... - Drop the session-cookie rate-limit bypass: Gitea hands the cookie to anonymous visitors too, so any cookie-accepting crawler could exempt itself after one request - Loosen the general Gitea limit from 10 to 50 req/10s since logged-in users are now rate-limited as well - Serve robots.txt above the rate-limit rules so throttled crawlers can always read the Crawl-delay, and capture Host/User-Agent before the deny so 429s are logged with them - Match the compare/diff cap on every repo case-insensitively instead of only /Richie/dotfiles - Remove the Jellyfin health check; it never ran (server had no check flag) and is not wanted	2026-07-01 16:54:07 -04:00
Richie	db199f884d	feat(comms): add Slack to communication packages	2026-07-01 12:14:23 -04:00
Richie	c6645eb37c	feat(home-assistant): remove status indicator configuration and related YAML file	2026-07-01 00:21:05 -04:00
Richie	3984081753	feat(cleanup): remove leviathan and elise configurations and references	2026-06-29 16:24:21 -04:00
Richie	03f610feed	feat(ui): refactor HTML templates for consistent structure and improved styling	2026-06-29 10:23:13 -04:00
Richie	8804a8df83	added rate limits	2026-06-27 23:01:03 -04:00
Richie	8daac4d98b	feat(grafana): add HAProxy requests dashboard and Richie Postgres datasource ... Visualize the ingested HAProxy request logs in Grafana. - add a `richie-postgres` datasource connecting to the Richie DB over the local Unix socket (/run/postgresql; pg_hba trust for user richie) - order grafana after postgresql.service - add the "HAProxy Requests" dashboard: request rate by backend, response -time percentiles (p50-p99), top user-agents/IPs/endpoints, status mix	2026-06-27 23:01:03 -04:00
Richie	7606ddc514	feat(haproxy-logs): ingest HAProxy request logs into Richie DB ... Add a pipeline to load HAProxy `option httplog` lines into the Richie database so bot/crawler traffic can be analyzed. - model: HaproxyRequest mirroring the httplog format, with a unique line_hash dedup key and indexes on common filter columns - migration: create the haproxy_request table (unique line_hash + indexes) - haproxy_logs package: - parser: httplog line -> columns, strips the journald prefix and hashes the normalized line - ingest: batched, idempotent insert that skips rows whose line_hash already exists, so re-ingesting the same logs is a no-op - cli: ingest-only `haproxy-logs` command reading stdin or a file - tests: parsing of a real GPTBot line and idempotent re-ingestion	2026-06-27 23:01:03 -04:00
Richie	137bd8ed2f	adding robots.txt	2026-06-27 23:01:03 -04:00
Richie	f936253ad5	flake update	2026-06-22 11:31:47 -04:00
Richie	fc01fb9d0b	Merge branch 'main' into feature/book-search-engine	2026-06-21 15:04:43 -04:00
Richie	09d963ba34	fix(ebook-search): skip comment lines in gold query loader and realign tests ... load_gold_queries now skips blank and `//` comment lines so the committed section separator in queries.jsonl no longer breaks dataset/load-test loading. Update tests left stale by the search refactor (6bc3011): - pass the now-required rank_constant to reciprocal_rank_fusion - expect bm25_candidates to receive the full query and drop the removed "BM25 query preparation" timing step - assert reranking is enabled by default	2026-06-21 14:45:31 -04:00
Richie	855af0fdff	updating 10-Primary to enp97s0f1	2026-06-21 14:25:02 -04:00
Richie	fcb69cc68b	adding words to spell check	2026-06-18 12:55:46 -04:00
Richie	6bc30115d9	fix(ebook-search): code clean up to impove reliablty and readabilty	2026-06-18 12:45:56 -04:00
Richie	6ae1ff1f5c	feat(ebook-search): add load-test CLI for the search service ... Add a Typer CLI script that drives POST /search on a running server at a configurable concurrency and reports latency percentiles (p50/p90/p95/p99), throughput, and HTTP status distribution. Queries are drawn from the shared eval JSONL set so load testing and evaluation exercise the same questions.	2026-06-18 12:39:55 -04:00
Richie	dbc6b5b53b	test(ebook-search): organize tests under dedicated package ... Move ebook search tests into tests/ebook_search and standardize mocking on pytest-mock.	2026-06-16 21:47:40 -04:00
Richie	a9daa60c17	moved TC001 expetop out of pyproject.toml	2026-06-16 18:31:40 -04:00
Richie	241a42b20d	updated front end	2026-06-16 00:00:13 -04:00
Richie	4640ebf8ce	converted addmin.py and page.py to DbSession and AppConfig	2026-06-15 22:01:42 -04:00
Richie	0c9583c1cc	moved config = load_config() to lifespan	2026-06-15 21:59:51 -04:00
Richie	f71ae7d2c6	added guardrails.py to constrain responses and added validation to config.py	2026-06-15 21:57:38 -04:00
Richie	2e68c83021	added health endpoints	2026-06-15 21:21:01 -04:00
Richie	b126987b63	added AppConfig and AppEngine	2026-06-15 21:08:55 -04:00
Richie	68b3a38b81	converting to pydantic-settings	2026-06-14 21:29:45 -04:00
Richie	a5d7c3be4f	fixed fomat issue	2026-06-14 15:42:05 -04:00
Richie	2995a75748	fixed test	2026-06-14 15:41:09 -04:00
Richie	dce1838163	opning ports for testing	2026-06-14 15:41:09 -04:00
Richie	121eb979a4	added a index for the VEctor DB	2026-06-14 15:41:09 -04:00
Richie	c88315e9b6	improved BM25 write	2026-06-14 15:41:09 -04:00
Richie	50795ab7fc	added ZstdMiddleware to ebook_search	2026-06-14 15:41:09 -04:00
Richie	e9a80a0308	added vector_engine to fix name postgres name space issue	2026-06-14 15:41:09 -04:00
Richie	eb76edb740	reworked ebook_search routers	2026-06-14 15:41:09 -04:00
Richie	c53afb3c70	made fastapi tools	2026-06-14 15:41:09 -04:00
Richie	8301db39e5	added proper cache invalidation to load_bm25_corpus	2026-06-14 15:40:31 -04:00
Richie	45a9e90524	updated tests	2026-06-14 15:40:31 -04:00
Richie	dd67de3993	improved reranking weights	2026-06-14 15:40:31 -04:00
Richie	6e3635ca01	fixed duplicat enrichment	2026-06-14 15:40:31 -04:00
Richie	11cbe31152	improved queary for vector search	2026-06-14 15:40:31 -04:00
Richie	5544dfa61c	add .ebook_search_bm25 to gitignore	2026-06-14 15:40:31 -04:00
Richie	911df63513	updated python	2026-06-14 15:40:31 -04:00
Richie	07eb170b34	setup tests	2026-06-14 15:40:04 -04:00