Expand README with project constraints, search tools, and contribution areas
Add sections explaining why Parakeet is the current default (MacBook Pro
constraint, better quality than SpeechAnalyzer, other backends too heavy),
what features are missing (contextual hinting, diarization), and the two
CLI search tools (FTS5 full-text, sentence-transformers semantic). Clarify
that only rendered text is committed while JSON intermediates stay local.
Assisted-by: Claude Opus 4.6 via pi
diff --git a/README.md b/README.md
index 6ae33bf..705b567 100644
--- a/README.md
+++ b/README.md
@@ -24,6 +24,24 @@ Set the backend with `TRANSCRIBER=name`.
| `mlx-diarize` | mlx-whisper + pyannote | Yes | Speaker labels. Requires HF token. |
| `whisperx` | WhisperX | Yes | CPU-based with pyannote diarization. |
+## Why Parakeet
+
+I run everything locally on a MacBook Pro, which is the primary constraint. That rules out most heavyweight transcription pipelines and anything that requires more GPU than Apple Silicon provides.
+
+Apple's SpeechAnalyzer (`sous_chef`) is the fastest option, but Parakeet produces noticeably better transcripts — based on informal comparison of a handful of episodes, not a rigorous benchmark. Most of the other backends (WhisperX, mlx-whisper with diarization) were too resource-intensive to run comfortably on the same machine.
+
+There are multiple transcriber backends wired up so it's easy to re-evaluate as models and hardware improve. The transcriber table above reflects what's implemented, not what's practical to run at scale on a single laptop.
+
+Solutions don't need to stay local — cloud APIs, hosted inference, and CI-based pipelines are all welcome. I'm not looking to purchase or rent hardware for this, but I'm open to other approaches.
+
+## What's missing
+
+Two features would meaningfully improve transcript quality but aren't available with the current setup.
+
+Contextual hinting would let the transcriber bias toward domain-specific vocabulary — recurring guest names, cooking terminology (rotovap, Searzall, hydrocolloids), show-specific jargon.
+
+Diarization so that transcripts could denote speakers. Pyannote-based diarization works (`mlx-diarize`, `whisperx`) but wasn't performant enough to run locally alongside transcription. The `whisper-cpp-tdrz` backend offers lightweight tinydiarize, but it uses a smaller model with lower transcription quality.
+
## Usage
```sh
@@ -57,24 +75,46 @@ export HUGGING_FACE_TOKEN=hf_...
## Output
-Intermediate JSON goes to `cache/<transcriber>/`. Rendered text is committed to `transcripts/<transcriber>/`, one file per episode:
+Only the rendered text files are committed to the repo, one per episode in `transcripts/<transcriber>/`:
```
transcripts/parakeet/001-episode-1-cooking-issues-debuts.txt
```
+The intermediate JSON from each transcriber lives in `cache/<transcriber>/` and is ignored. These files contain word- or sentence-level timestamps and are substantially larger than the rendered text. I have them locally for re-rendering or analysis but they're not worth carrying in the repo.
+
The text renderer groups segments into paragraphs. The heuristic varies by transcriber — silence gaps for most, sentence counts for Parakeet (whose chunking overlap produces overlapping timestamps).
## Static site
`rake pages` generates a static site in `pages/` with one HTML page per episode and a Pagefind search index. `rake serve` builds and serves it locally on port 8000. GitHub Actions deploys it to GitHub Pages on push to main.
+## Search
+
+Beyond the Pagefind-powered static site, there are two CLI search tools for querying transcripts directly. Both index from the rendered text files and store their databases in `cache/`.
+
+Full-text search uses SQLite FTS5 with BM25 ranking:
+
+```sh
+bin/fts-search index [transcripts/parakeet]
+bin/fts-search search "agar clarification" -n 5
+```
+
+Semantic search uses sentence-transformers (`all-MiniLM-L6-v2`) with sqlite-vec for nearest-neighbor lookup:
+
+```sh
+bin/semantic-search index [transcripts/parakeet]
+bin/semantic-search search "how to clarify juice" -n 5
+```
+
+Full-text search is fast and needs only SQLite. Semantic search is better for fuzzy or conceptual queries but requires downloading the embedding model on first run.
+
## Requirements
- Ruby 3.x with Bundler
- Python 3.10+ with [uv](https://github.com/astral-sh/uv) (for Parakeet, MLX, and Pagefind)
- At least one transcriber's dependencies:
- - **parakeet**: uv (dependencies managed automatically)
- - **whisper-cpp**: `whisper-cli` (Homebrew or build from source)
- - **sous_chef**: macOS 26+, Xcode 26+ (builds automatically)
- - **mlx / mlx-diarize / whisperx**: uv
+ - parakeet — uv (dependencies managed automatically)
+ - whisper-cpp — `whisper-cli` (Homebrew or build from source)
+ - sous_chef — macOS 26+, Xcode 26+ (builds automatically)
+ - mlx, mlx-diarize, whisperx — uv