Prompt is now a template (fixes the hardcoded en/es leftovers in rules
2, 3, 15, 16, 26, 31). pipeline.ts replaces the pseudocode: wordlist
normalization, skip-already-staged idempotency, batches of 20 against
gemini-3.6-flash with responseSchema, raw responses persisted per batch,
per-entry validation with rejection log, one transaction per word into
db/staging.db. Flags: --langs --pos --max-batches --delay-ms --dry-run.
Smoke run: 40/40 words staged (de+es, one batch each), 0 rejections.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add stage-1-extract/scripts/extract.ts — streams Kaikki JSONL,
filters to supported POS and languages, skips abbreviations and
senses with no translations in supported languages
- Rewrite db/import.ts for Kaikki flat model — tracks sense_index
offsets per headword+pos to handle duplicate JSONL entries
- Rewrite db/schema.sql for Kaikki model — entries, translations,
LLM vote tables, resolved tables
- Add extract and db:import scripts to package.json
- Sample mode hardcoded to 500 entries for development
- Remove foreign key on run_status.source_id to support sentinel rows
for tracking one-time pipeline steps (compile_candidates, compile_votes,
merge, compare)
- Add stage-3-enrich/config.ts with all provider configurations,
ALL_PROVIDERS ordered local-first, and validateProviderKey() for
startup key checks
- Add .env.example with required API keys for OpenRouter and Anthropic
- Add pipeline:run script to package.json using --env-file .env
- Add .env to root .gitignore coverage for data-pipeline/.env
