feat(ebook): add protected phrase extraction library with config-driven tuning
Refactor protected phrase handling from a single module into a python/ebook_search/protected_phrases package covering extraction, storage, and runtime matching. Phrase filtering is now data-driven via bundled TOML files: ignored_phrases, bad_starts, bad_ends, and most_common_words. Add phrase-tuning settings to EbookSearchConfig so candidate generation, scoring, LLM judging, and matching are configurable rather than hardcoded: token bounds, entity token limit, raw n-gram min count, frequency and chapter-spread score thresholds, candidate/LLM/target caps, confidence threshold, nesting defaults, and the phrase hit boost.
This commit is contained in:
@@ -0,0 +1,134 @@
|
||||
phrases = [
|
||||
"ended up",
|
||||
"had a",
|
||||
"across the",
|
||||
"anyone in",
|
||||
"are you",
|
||||
"around him",
|
||||
"around the",
|
||||
"as much",
|
||||
"as soon",
|
||||
"before the",
|
||||
"behind him",
|
||||
"between the",
|
||||
"did you",
|
||||
"end up",
|
||||
"he asked",
|
||||
"he concluded",
|
||||
"he continued",
|
||||
"he couldn't",
|
||||
"he did",
|
||||
"he didn't",
|
||||
"he felt",
|
||||
"he had",
|
||||
"he hadn't",
|
||||
"he knew",
|
||||
"he noted",
|
||||
"he pointed",
|
||||
"he realized",
|
||||
"he replied",
|
||||
"he said",
|
||||
"he saw",
|
||||
"he tapped",
|
||||
"he told",
|
||||
"he was",
|
||||
"he wasn't",
|
||||
"his body",
|
||||
"his chair",
|
||||
"his feet",
|
||||
"his hands",
|
||||
"his head",
|
||||
"his office",
|
||||
"his own",
|
||||
"his pc",
|
||||
"his power",
|
||||
"his shield",
|
||||
"his sight",
|
||||
"his voice",
|
||||
"his wrist",
|
||||
"how many",
|
||||
"i am",
|
||||
"i don't",
|
||||
"i was",
|
||||
"i wouldn't",
|
||||
"i'm not",
|
||||
"is in",
|
||||
"is not",
|
||||
"is that",
|
||||
"is the",
|
||||
"it had",
|
||||
"it had",
|
||||
"it was",
|
||||
"it wasn't",
|
||||
"it wasn't",
|
||||
"of course",
|
||||
"of force",
|
||||
"of magic",
|
||||
"of marines",
|
||||
"of power",
|
||||
"of those",
|
||||
"old man",
|
||||
"older man",
|
||||
"set up",
|
||||
"she admitted",
|
||||
"she asked",
|
||||
"she had",
|
||||
"she replied",
|
||||
"she said",
|
||||
"she snapped",
|
||||
"she told",
|
||||
"she was",
|
||||
"she'd been",
|
||||
"shook his",
|
||||
"sure he",
|
||||
"tell you",
|
||||
"that had",
|
||||
"that is",
|
||||
"that was",
|
||||
"there are",
|
||||
"there was",
|
||||
"there were",
|
||||
"they are",
|
||||
"they had",
|
||||
"they were",
|
||||
"they weren't",
|
||||
"this is",
|
||||
"this place",
|
||||
"though he",
|
||||
"through his",
|
||||
"through the",
|
||||
"to find",
|
||||
"to keep",
|
||||
"to stay",
|
||||
"to stop",
|
||||
"to tell",
|
||||
"to try",
|
||||
"told her",
|
||||
"told him",
|
||||
"under his",
|
||||
"was a",
|
||||
"was enough",
|
||||
"was going",
|
||||
"was in",
|
||||
"was no",
|
||||
"was not",
|
||||
"was now",
|
||||
"was on",
|
||||
"was one",
|
||||
"was only",
|
||||
"was still",
|
||||
"was that",
|
||||
"was the",
|
||||
"was there",
|
||||
"were in",
|
||||
"what had",
|
||||
"what happened",
|
||||
"what was",
|
||||
"where the",
|
||||
"while i",
|
||||
"you are",
|
||||
"you can't",
|
||||
"you don't",
|
||||
"you need",
|
||||
"you were",
|
||||
]
|
||||
Reference in New Issue
Block a user