feat(ebook): add protected phrase extraction library with config-driven tuning
Refactor protected phrase handling from a single module into a python/ebook_search/protected_phrases package covering extraction, storage, and runtime matching. Phrase filtering is now data-driven via bundled TOML files: ignored_phrases, bad_starts, bad_ends, and most_common_words. Add phrase-tuning settings to EbookSearchConfig so candidate generation, scoring, LLM judging, and matching are configurable rather than hardcoded: token bounds, entity token limit, raw n-gram min count, frequency and chapter-spread score thresholds, candidate/LLM/target caps, confidence threshold, nesting defaults, and the phrase hit boost.
This commit is contained in:
@@ -0,0 +1,30 @@
|
||||
tokens = [
|
||||
"a",
|
||||
"an",
|
||||
"and",
|
||||
"any",
|
||||
"as",
|
||||
"at",
|
||||
"be",
|
||||
"because",
|
||||
"but",
|
||||
"by",
|
||||
"can",
|
||||
"could",
|
||||
"do",
|
||||
"for",
|
||||
"from",
|
||||
"have",
|
||||
"if",
|
||||
"of",
|
||||
"or",
|
||||
"some",
|
||||
"than",
|
||||
"the",
|
||||
"these",
|
||||
"this",
|
||||
"to",
|
||||
"will",
|
||||
"with",
|
||||
"would",
|
||||
]
|
||||
Reference in New Issue
Block a user