Richie
1f5e527cbc
Convert the ebook-search web app to async end to end and add concurrency to the protected-phrase extraction and judging pipeline so large books no longer block the event loop or the UI. ORM / infra: - Add get_async_postgres_engine and factor shared URL/connect_args building into build_postgres_url (reused by the sync and async engine builders) - Add async FastAPI session helpers (get_async_db, AsyncDbSession) with expire_on_commit=False to avoid implicit IO under asyncio App: - Use AsyncEngine/AsyncSession throughout routes, search, ingest, embeddings, answer, rerank and LLM calls; convert handlers to async - Share a single httpx.AsyncClient in app state for LLM requests; size the connection pool for concurrent phrase-judging workers - Add judge_tasks: run per-book judging as tracked background tasks so a book already being judged isn't double-queued Protected phrases: - Add a process pool (pool.py) and worker-count config (extraction/judge book/phrase workers) to parallelize candidate generation and judging - Split admin actions into all/missing variants for generation and judging Config: - Add protected_phrase_extraction_workers, phrase_judge_book_workers, phrase_judge_phrase_workers
111 lines
2.8 KiB
HTML
111 lines
2.8 KiB
HTML
{% extends "base.html" %} {% block title %}EPUB Admin{% endblock %} {% block
|
|
head %}
|
|
<script src="https://unpkg.com/htmx.org@2.0.4"></script>
|
|
{% endblock %} {% block content %}
|
|
<h1>Admin</h1>
|
|
<section id="admin-status"></section>
|
|
<section class="actions">
|
|
<form hx-post="/admin/scan" hx-target="#admin-status" hx-swap="innerHTML">
|
|
<button type="submit">Scan</button>
|
|
</form>
|
|
</section>
|
|
<section>
|
|
<h2>Embeddings</h2>
|
|
<section class="actions">
|
|
<form
|
|
hx-post="/admin/embed-missing"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Embed</button>
|
|
</form>
|
|
<form
|
|
hx-post="/admin/embed-all"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Embed all</button>
|
|
</form>
|
|
</section>
|
|
<table>
|
|
<thead>
|
|
<tr>
|
|
<th>Model</th>
|
|
<th>Dimensions</th>
|
|
<th>Embedded</th>
|
|
<th>Missing</th>
|
|
<th>Total chunks</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
{% for item in stats %}
|
|
<tr>
|
|
<td>{{ item.model_name }}</td>
|
|
<td>{{ item.dimension }}</td>
|
|
<td>{{ item.embedded_chunks }}</td>
|
|
<td>{{ item.missing_chunks }}</td>
|
|
<td>{{ item.total_chunks }}</td>
|
|
</tr>
|
|
{% endfor %}
|
|
</tbody>
|
|
</table>
|
|
</section>
|
|
<section>
|
|
<h2>Protected phrases</h2>
|
|
<section class="actions">
|
|
<form
|
|
hx-post="/admin/phrases/generate-all"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Regenerate all phrases</button>
|
|
</form>
|
|
<form
|
|
hx-post="/admin/phrases/generate-missing"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Add missing phrases</button>
|
|
</form>
|
|
<form
|
|
hx-post="/admin/phrases/judge-all"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Judge all phrases</button>
|
|
</form>
|
|
<form
|
|
hx-post="/admin/phrases/judge-missing"
|
|
hx-target="#admin-status"
|
|
hx-swap="innerHTML"
|
|
>
|
|
<button type="submit">Judge missing phrases</button>
|
|
</form>
|
|
</section>
|
|
<table>
|
|
<thead>
|
|
<tr>
|
|
<th>Candidates</th>
|
|
<th>Judged</th>
|
|
<th>Unjudged</th>
|
|
<th>Protected</th>
|
|
<th>Books indexed</th>
|
|
<th>Books generated</th>
|
|
<th>Books fully judged</th>
|
|
</tr>
|
|
</thead>
|
|
<tbody>
|
|
<tr>
|
|
<td>{{ phrase_stats.candidate_phrases }}</td>
|
|
<td>{{ phrase_stats.judged_candidates }}</td>
|
|
<td>{{ phrase_stats.unjudged_candidates }}</td>
|
|
<td>{{ phrase_stats.protected_phrases }}</td>
|
|
<td>{{ phrase_stats.total_books }}</td>
|
|
<td>{{ phrase_stats.books_with_candidates }}</td>
|
|
<td>{{ phrase_stats.books_fully_judged }}</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</section>
|
|
{% endblock %}
|