fix(protected-phrases): isolate phrase generation per book
treefmt / nix fmt (pull_request) Failing after 5s
pytest / pytest (pull_request) Failing after 36s
build_systems / build-brain (pull_request) Successful in 51s
build_systems / build-bob (pull_request) Successful in 55s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m30s
build_systems / build-jeeves (pull_request) Successful in 3m9s
treefmt / nix fmt (pull_request) Failing after 5s
pytest / pytest (pull_request) Failing after 36s
build_systems / build-brain (pull_request) Successful in 51s
build_systems / build-bob (pull_request) Successful in 55s
build_systems / build-rhapsody-in-green (pull_request) Successful in 1m30s
build_systems / build-jeeves (pull_request) Successful in 3m9s
Run full-book candidate generation inside worker-owned sessions so each book commits independently during backfills. Abort recalculation when a book has no indexed chapters to preserve existing phrase data, and update admin/UI tests for the new generation flow.
This commit is contained in:
@@ -9,21 +9,11 @@ or server threads.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import multiprocessing
|
||||
import os
|
||||
from concurrent.futures import ProcessPoolExecutor
|
||||
from threading import Lock
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from python.ebook_search.protected_phrases.extraction import extract_phrase_candidates_for_book
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Mapping, Sequence
|
||||
|
||||
from python.ebook_search.config import EbookSearchConfig
|
||||
from python.ebook_search.protected_phrases.models import PhraseCandidate
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@@ -66,36 +56,3 @@ def shutdown_extraction_pool() -> None:
|
||||
_extraction_pool.pool.shutdown(wait=False, cancel_futures=True)
|
||||
_extraction_pool.pool = None
|
||||
logger.info("ebook_phrase_extraction_pool_shutdown")
|
||||
|
||||
|
||||
async def extract_phrase_candidates_in_pool(
|
||||
book_text: str,
|
||||
chapters: Sequence[str],
|
||||
config: EbookSearchConfig,
|
||||
*,
|
||||
metadata: Mapping[str, object] | None,
|
||||
) -> list[PhraseCandidate]:
|
||||
"""Run book phrase extraction in a worker process and await the result.
|
||||
|
||||
Only the CPU-bound extraction runs in the worker; the caller keeps all database work in the
|
||||
request process. The spaCy pipeline is not supported here because it is not picklable, so
|
||||
this always runs the non-spaCy extraction path.
|
||||
|
||||
Args:
|
||||
book_text (str): Full book text used for extraction.
|
||||
chapters (Sequence[str]): Chapter-like text blocks used for frequency counts.
|
||||
config (EbookSearchConfig): Runtime phrase-tuning settings.
|
||||
metadata (Mapping[str, object] | None): Optional book metadata used as a candidate source.
|
||||
|
||||
Returns:
|
||||
list[PhraseCandidate]: Scored candidates sorted best-first and capped per book.
|
||||
"""
|
||||
pool = get_extraction_pool(config.protected_phrase_extraction_workers)
|
||||
future = pool.submit(
|
||||
extract_phrase_candidates_for_book,
|
||||
book_text,
|
||||
list(chapters),
|
||||
config,
|
||||
metadata=dict(metadata) if metadata is not None else None,
|
||||
)
|
||||
return await asyncio.wrap_future(future)
|
||||
|
||||
Reference in New Issue
Block a user