updating documentation, prettier format
This commit is contained in:
lila
parent
039ed50567
commit
e534b98bc5
16 changed files with 1271 additions and 1070 deletions
|
|
@ -1,173 +1,151 @@
|
|||
# 05 — Data Pipeline
|
||||
|
||||
> **Purpose:** Condensed reference for LLMs working on the Kaikki data pipeline. Covers stages, data flow, and current blockers. For full operational details (llama.cpp setup, provider configs, hardware specs), see the human-readable DATA_PIPELINE.md.
|
||||
> **Last updated:** 2026-05-15
|
||||
> **Depends on:** 00-project-overview.md
|
||||
> **Purpose:** Condensed reference for LLMs working on the vocabulary data pipeline. Covers the flow, what exists, and what is still unwritten. Full detail: `documentation/DATA_PIPELINE.md`, `documentation/pipeline/design-doc.md`, `documentation/pipeline/roadmap.md`.
|
||||
> **Last updated:** 2026-08-01
|
||||
> **Depends on:** 00-project-overview.md, 02-data-model.md
|
||||
|
||||
---
|
||||
|
||||
## Read this first
|
||||
|
||||
The pipeline was **completely rewritten**. The old Kaikki/local-LLM architecture — six stages, `pipeline.db`, `stage-1-extract/`, `stage-3-enrich/`, multi-model CEFR voters, llama.cpp — is **gone from the codebase**. Any reference you see to those stages, directories, or the voter strategy is historical (`documentation/archive/`), not something you can call or modify.
|
||||
|
||||
The current pipeline is Gemini-only, and most of it **is not written yet**. Do not assume a module exists because a doc names it.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Overview
|
||||
|
||||
```
|
||||
Kaikki JSONL (Wiktionary extracts)
|
||||
source-data/{lang}/{pos} frequency wordlists, one word per line, UTF-8
|
||||
↓
|
||||
Stage 1: Extract → Parse into pipeline.db (SQLite)
|
||||
Gemini API batches of 20 words, one language at a time
|
||||
↓
|
||||
Stage 2: Reverse Link → Insert missing reverse translations
|
||||
validation per entry; rejects go to a log, never to the DB
|
||||
↓
|
||||
Stage 3: Enrich → LLMs review glosses, examples, translations, assign CEFR
|
||||
db/staging.db SQLite staging (words, senses, translations)
|
||||
↓
|
||||
Stage 4: Merge → Resolve LLM votes into final values
|
||||
import script SQLite → PostgreSQL via Drizzle, transaction per batch
|
||||
↓
|
||||
Stage 4b: Tiebreak → Run unused models on flagged entries
|
||||
↓
|
||||
Stage 5: Compare / QA → Generate COVERAGE.md quality report
|
||||
↓
|
||||
Stage 6: Sync → Upsert resolved records into production PostgreSQL
|
||||
PostgreSQL dev (:5432), then production
|
||||
```
|
||||
|
||||
**Current state:** Stage 1 and 2 complete on sample data. Stage 3 enrich script being rewritten for sub-stage architecture. Stages 4–6 not started.
|
||||
Each language is processed independently so that definitions and examples are written **in that language**. Only translations cross language boundaries.
|
||||
|
||||
The app always reads PostgreSQL. SQLite is a staging file only, so re-runs and prompt tweaks never touch a real database.
|
||||
|
||||
**Current state:** Phases 1–2 complete (schema, wordlists, databases, prompt). Phase 3 in progress — validation module, pipeline script, and first real run are all still to be written. Phases 4–7 not started.
|
||||
|
||||
---
|
||||
|
||||
## Stage 1: Extract
|
||||
## What exists in `data-pipeline/`
|
||||
|
||||
**Input:** `data-pipeline/stage-1-extract/sources/*.jsonl` (Kaikki files, not in git)
|
||||
**Output:** `pipeline.db` — `vocabulary_entries` and `entry_translations` tables
|
||||
| Path | State |
|
||||
| --------------------------- | ------------------------------------------------------------------ |
|
||||
| `source-data/{lang}/{pos}/` | ✅ Noun lists for `de`, `en`, `es`, `fr`, `it` |
|
||||
| `prompt` | ✅ Gemini system prompt — a plain UTF-8 text file, not a TS module |
|
||||
| `db/schema.sql` | ✅ SQLite staging schema |
|
||||
| `db/staging.db` | ✅ Tables created, **0 rows**, gitignored |
|
||||
| `pipeline.ts` | 🚧 Design pseudocode in comments only — no executable code |
|
||||
| validation module | ❌ Not written (rules in design-doc §6.4) |
|
||||
| SQLite → PostgreSQL import | ❌ Not written |
|
||||
| `kaikki-source-files/` | ⚠️ Leftover JSONL dumps; nothing reads them |
|
||||
| `worddata/english/nouns/` | ⚠️ Empty leftover output dir from the old per-word-JSON design |
|
||||
|
||||
**What it does:**
|
||||
Directory names use the codes from `packages/shared/src/constants.ts` (`de/noun`, not `german/nouns`) so no mapping layer is needed.
|
||||
|
||||
- Parses Kaikki JSONL for all 5 languages (en, de, es, fr, it)
|
||||
- Filters to 4 POS: noun, verb, adjective, adverb
|
||||
- Each Kaikki sense becomes one `vocabulary_entries` row
|
||||
- Translations stored in `entry_translations` with sense hints
|
||||
|
||||
**Key design:** Kaikki is structured per word sense. Each headword has multiple senses, and translations are linked to a specific sense. This prevents the sense-disambiguation problems of OpenWordNet/OMW.
|
||||
`data-pipeline/vitest.config.ts` looks for `tests/**/*.test.ts`; that directory does not exist yet.
|
||||
|
||||
---
|
||||
|
||||
## Stage 2: Reverse Link Sync
|
||||
## Gemini output contract
|
||||
|
||||
**Pure script, no LLMs.**
|
||||
The model returns a JSON array, one object per input word, in input order — no markdown fences, comments, or trailing commas.
|
||||
|
||||
For each translation pair (e.g., English "thrill" → German "begeistern"), checks if the reverse exists (German "begeistern" → English "thrill"). If the German entry exists but lacks the English back-link, inserts it automatically.
|
||||
<!-- prettier-ignore -->
|
||||
```json
|
||||
[
|
||||
{
|
||||
"headword": "Haus",
|
||||
"language": "de",
|
||||
"pos": "noun",
|
||||
"senses": [
|
||||
{
|
||||
"sense_index": 0,
|
||||
"difficulty": "easy",
|
||||
"definitions": ["Ein Gebäude zum Wohnen."],
|
||||
"examples": ["Sie kauften ein Haus in der Stadt."],
|
||||
"translations": [
|
||||
{ "target_language": "en", "word": "house", "gender": null, "difficulty": "easy" },
|
||||
{ "target_language": "es", "word": "casa", "gender": "feminine", "difficulty": "easy" }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
**Why:** Ensures LLMs in Stage 3 only generate translations that are genuinely missing — not translations findable by simple reverse lookup.
|
||||
Prompt rules that the output depends on:
|
||||
|
||||
- Definitions and examples in the **source** language, not English.
|
||||
- Gender required for `de` (masculine/feminine/neuter) and `it`/`es`/`fr` (masculine/feminine); always `null` for `en`.
|
||||
- German nouns capitalized; Romance-language nouns lowercase unless proper nouns.
|
||||
- Base dictionary form, no articles or determiners.
|
||||
- 1–3 senses per word, most words 1; no rare, archaic, or technical senses.
|
||||
- Max 2 translations per target language per sense, only genuine synonyms or difficulty variants.
|
||||
- A translation's difficulty is never lower than its sense's difficulty.
|
||||
- A word that is not a valid noun in that language returns `"senses": []`.
|
||||
|
||||
⚠️ The checked-in `prompt` file still has hardcoded English leftovers (rules 2, 3, and 31 say `"en"` / "English noun" while the header says Spanish) and its target-language list disagrees with its own header. It is also pinned to one sample batch rather than templated. Fix when implementing `pipeline.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3: Enrich (In Progress — Being Rewritten)
|
||||
## Validation rules (design-doc §6.4)
|
||||
|
||||
**Current blocker:** The original single-prompt design had problems (skipped invalid translations, triggered reasoning mode, 20% manual review). Being rewritten as four ordered sub-stages.
|
||||
Run per entry before anything is written to SQLite. Invalid entries go to a rejection log for review, not to the database. Target reject rate: under 10%.
|
||||
|
||||
### Sub-Stage Architecture
|
||||
|
||||
Each model processes every entry through four sub-stages in order:
|
||||
|
||||
1. **`round1_gloss`** — Review existing gloss. Confirm if clear, generate better one if not.
|
||||
2. **`round1_example`** — Review examples. Confirm if natural, generate one better sentence.
|
||||
3. **`round1_translations`** — Validate translations with verified gloss as context. Confirm valid, reject invalid, generate missing.
|
||||
4. **`round1_cefr`** — Assign CEFR level (A1–C2) to headword and each confirmed translation.
|
||||
|
||||
**Why this order:** CEFR sub-stage only sees clean, verified data. Bad translations are rejected before reaching CEFR assignment.
|
||||
|
||||
**Voter strategy:** Multiple models vote independently. Each model = one vote per sub-stage. Current plan:
|
||||
|
||||
- Primary: Local Qwen3.5-9B (overnight runs, unlimited)
|
||||
- Secondary: Groq Llama 3.3 70B (cloud, batched)
|
||||
- Tertiary: Gemini AI Studio (cloud, batched)
|
||||
|
||||
**Context enrichment:** Before calling models for gloss/example, pipeline queries Wiktionary API for the headword. Full entry (all senses, usage notes) added to prompt. Fixes category header glosses and short ambiguous glosses.
|
||||
- `headword` non-empty; `language` in the 5 supported codes; `pos` in the 4 supported values
|
||||
- at least one sense; each sense has ≥1 definition, ≥1 example, ≥1 translation
|
||||
- `sense_index` a non-negative integer, starting at 0 and increasing by 1
|
||||
- `difficulty` in `easy | medium | hard` on both senses and translations
|
||||
- `target_language` supported and never equal to the word's own language
|
||||
- `gender` valid for the target language (see above); `null` for English
|
||||
- no duplicate (headword, language, pos, sense_index)
|
||||
|
||||
---
|
||||
|
||||
## Stage 4: Merge
|
||||
## Constants
|
||||
|
||||
Resolves LLM votes into final values per entry.
|
||||
| Constant | Values | Source |
|
||||
| ---------- | ------------------------------------- | -------------------------- |
|
||||
| Languages | `en`, `it`, `de`, `es`, `fr` | `SUPPORTED_LANGUAGE_CODES` |
|
||||
| POS | `noun`, `verb`, `adjective`, `adverb` | `SUPPORTED_POS` |
|
||||
| Difficulty | `easy`, `medium`, `hard` | `DIFFICULTY_LEVELS` |
|
||||
| Gender | `masculine`, `feminine`, `neuter` | `NOUN_GENDERS` |
|
||||
|
||||
**Rules:**
|
||||
All live in `packages/shared/src/constants.ts` and are CHECK-constrained in the PostgreSQL schema. Adding a value means updating the constant **and** a Drizzle migration before re-running the pipeline.
|
||||
|
||||
- Kaikki source data wins automatically (never overridden)
|
||||
- CEFR: level with most votes wins
|
||||
- Text fields (gloss, example, translation): candidate with most votes wins
|
||||
- No majority → flag for tiebreaker
|
||||
|
||||
**Difficulty mapping:**
|
||||
| CEFR | Difficulty |
|
||||
|------|-----------|
|
||||
| A1, A2 | easy |
|
||||
| B1, B2 | intermediate |
|
||||
| C1, C2 | hard |
|
||||
CEFR levels still exist as a constant and as columns on the old `vocabulary_entries` tables, but the new pipeline does not produce them — it produces the three-level difficulty directly. The prompt calibrates difficulty against CEFR bands internally (easy ≈ A1/A2, medium ≈ B1/B2, hard ≈ C1/C2) but is explicitly told not to emit CEFR levels.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4b: Tiebreak
|
||||
## Running it
|
||||
|
||||
Runs automatically after merge if flagged entries remain. Queries unused models (not yet voted) and re-runs merge. Repeats until resolved or no unused models remain.
|
||||
```bash
|
||||
docker compose up -d pipeline-database # dedicated PostgreSQL on :5433
|
||||
pnpm --filter @lila/pipeline pipeline:run # tsx --env-file .env pipeline.ts (currently a no-op)
|
||||
pnpm --filter @lila/pipeline test
|
||||
```
|
||||
|
||||
**If still unresolved:** Sync is blocked. Add more models to config and re-run.
|
||||
Env vars come from the repo-root `.env`: `GEMINI_API_KEY`, `PIPELINE_POSTGRES_USER`, `PIPELINE_POSTGRES_PASSWORD`, `PIPELINE_POSTGRES_DB`, `PIPELINE_DATABASE_URL`. The pipeline database is deliberately separate from the app database (`:5432`).
|
||||
|
||||
---
|
||||
|
||||
## Stage 5: Compare / QA
|
||||
|
||||
Read-only. Generates `COVERAGE.md` with per-language breakdown:
|
||||
|
||||
- Total entries, POS distribution
|
||||
- Translation coverage per language pair
|
||||
- CEFR coverage and difficulty breakdown
|
||||
- Gloss/example coverage by source (Kaikki vs LLM)
|
||||
- Per-model contribution stats
|
||||
|
||||
Run this before syncing to production.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6: Sync
|
||||
|
||||
Upserts all `status = "final"` entries from `pipeline.db` to production PostgreSQL.
|
||||
|
||||
**Behavior:**
|
||||
|
||||
- Missing → insert
|
||||
- Present but changed → update
|
||||
- Present and unchanged → skip
|
||||
|
||||
**Idempotent.** Safe to re-run.
|
||||
|
||||
---
|
||||
|
||||
## Key Constraints
|
||||
|
||||
| Constant | Values |
|
||||
| ---------- | ------------------------------------- |
|
||||
| Languages | `en`, `it`, `de`, `es`, `fr` |
|
||||
| POS | `noun`, `verb`, `adjective`, `adverb` |
|
||||
| CEFR | `A1`, `A2`, `B1`, `B2`, `C1`, `C2` |
|
||||
| Difficulty | `easy`, `intermediate`, `hard` |
|
||||
|
||||
Adding a new value requires updating `packages/shared/src/constants.ts` AND a database migration before re-running the pipeline.
|
||||
Implementation notes from the roadmap: `better-sqlite3` for staging (synchronous), `crypto.randomUUID()` for ids, `JSON.stringify` for the definitions/examples arrays (SQLite has no array type — the import script parses them back into PostgreSQL `text[]`), batches of 20 with a 1s sleep between calls.
|
||||
|
||||
---
|
||||
|
||||
## Current Blockers
|
||||
|
||||
1. **Enrich sub-stage rewrite** — Stage 3 script needs redesign and testing
|
||||
2. **Cloud provider integration** — Groq and Gemini not yet wired into pipeline
|
||||
3. **Batching prompt design** — 5–10 entries per API call for efficiency; not yet designed
|
||||
4. **Full dataset scale unknown** — Currently running on 500-entry samples. Full Kaikki English file has ~1.3M entries. Exact filtered count and runtime estimate not yet known.
|
||||
|
||||
---
|
||||
|
||||
## Key Files
|
||||
|
||||
| File | Purpose |
|
||||
| ------------------------------------------------------------ | --------------------------------------------------------- |
|
||||
| `data-pipeline/pipeline.ts` | Orchestrator — runs stages in order, handles resumability |
|
||||
| `data-pipeline/stage-1-extract/scripts/extract.ts` | Parse Kaikki JSONL |
|
||||
| `data-pipeline/stage-2-reverse-link/scripts/reverse-link.ts` | Insert reverse translations |
|
||||
| `data-pipeline/stage-3-enrich/scripts/enrich.ts` | LLM enrichment (being rewritten) |
|
||||
| `data-pipeline/stage-3-enrich/config.ts` | Provider configs (local, OpenRouter, etc.) |
|
||||
| `data-pipeline/db/schema.sql` | pipeline.db schema |
|
||||
| `data-pipeline/db/import.ts` | Import stage 1 output into pipeline.db |
|
||||
| `packages/shared/src/constants.ts` | Language codes, POS, CEFR, difficulty constants |
|
||||
1. **Validation module and `pipeline.ts` are unwritten** — this is Phase 3, the active work.
|
||||
2. **Prompt is not templated** — source language, POS, target languages, and the word batch are hardcoded for one sample run.
|
||||
3. **No import script** — nothing moves staging rows into PostgreSQL yet (Phase 4).
|
||||
4. **App still reads the old schema** — `termModel.ts` queries `vocabulary_entries`/`entry_translations`. Until Phase 5 rewrites it, pipeline output is invisible to the app.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue