lila/documentation/ai-context/05-data-pipeline.md

05 — Data Pipeline

Purpose: Condensed reference for LLMs working on the vocabulary data pipeline. Covers the flow, what exists, and what is still unwritten. Full detail: documentation/DATA_PIPELINE.md, documentation/pipeline/design-doc.md, documentation/pipeline/roadmap.md. Last updated: 2026-08-01 Depends on: 00-project-overview.md, 02-data-model.md

---

Read this first

The pipeline was completely rewritten. The old Kaikki/local-LLM architecture — six stages, pipeline.db, stage-1-extract/, stage-3-enrich/, multi-model CEFR voters, llama.cpp — is gone from the codebase. Any reference you see to those stages, directories, or the voter strategy is historical (documentation/archive/), not something you can call or modify.

The current pipeline is Gemini-only, and most of it is not written yet. Do not assume a module exists because a doc names it.

---

Pipeline Overview

source-data/{lang}/{pos}   frequency wordlists, one word per line, UTF-8
     ↓
Gemini API                 batches of 20 words, one language at a time
     ↓
validation                 per entry; rejects go to a log, never to the DB
     ↓
db/staging.db              SQLite staging (words, senses, translations)
     ↓
import script              SQLite → PostgreSQL via Drizzle, transaction per batch
     ↓
PostgreSQL                 dev (:5432), then production

Each language is processed independently so that definitions and examples are written in that language. Only translations cross language boundaries.

The app always reads PostgreSQL. SQLite is a staging file only, so re-runs and prompt tweaks never touch a real database.

Current state: Phases 1–2 complete (schema, wordlists, databases, prompt). Phase 3 in progress — validation module, pipeline script, and first real run are all still to be written. Phases 4–7 not started.

---

What exists in data-pipeline/

Path	State
source-data/{lang}/{pos}/	✅ Noun lists for de, en, es, fr, it
prompt	✅ Gemini system prompt — a plain UTF-8 text file, not a TS module
db/schema.sql	✅ SQLite staging schema
db/staging.db	✅ Tables created, 0 rows, gitignored
pipeline.ts	🚧 Design pseudocode in comments only — no executable code
validation module	❌ Not written (rules in design-doc §6.4)
SQLite → PostgreSQL import	❌ Not written
kaikki-source-files/	⚠️ Leftover JSONL dumps; nothing reads them
worddata/english/nouns/	⚠️ Empty leftover output dir from the old per-word-JSON design

Directory names use the codes from packages/shared/src/constants.ts (de/noun, not german/nouns) so no mapping layer is needed.

data-pipeline/vitest.config.ts looks for tests/**/*.test.ts; that directory does not exist yet.

---

Gemini output contract

The model returns a JSON array, one object per input word, in input order — no markdown fences, comments, or trailing commas.

[
  {
    "headword": "Haus",
    "language": "de",
    "pos": "noun",
    "senses": [
      {
        "sense_index": 0,
        "difficulty": "easy",
        "definitions": ["Ein Gebäude zum Wohnen."],
        "examples": ["Sie kauften ein Haus in der Stadt."],
        "translations": [
          { "target_language": "en", "word": "house", "gender": null, "difficulty": "easy" },
          { "target_language": "es", "word": "casa", "gender": "feminine", "difficulty": "easy" }
        ]
      }
    ]
  }
]

Prompt rules that the output depends on:

• Definitions and examples in the source language, not English.
• Gender required for de (masculine/feminine/neuter) and it/es/fr (masculine/feminine); always null for en.
• German nouns capitalized; Romance-language nouns lowercase unless proper nouns.
• Base dictionary form, no articles or determiners.
• 1–3 senses per word, most words 1; no rare, archaic, or technical senses.
• Max 2 translations per target language per sense, only genuine synonyms or difficulty variants.
• A translation's difficulty is never lower than its sense's difficulty.
• A word that is not a valid noun in that language returns "senses": [].

⚠️ The checked-in prompt file still has hardcoded English leftovers (rules 2, 3, and 31 say "en" / "English noun" while the header says Spanish) and its target-language list disagrees with its own header. It is also pinned to one sample batch rather than templated. Fix when implementing pipeline.ts.

---

Validation rules (design-doc §6.4)

Run per entry before anything is written to SQLite. Invalid entries go to a rejection log for review, not to the database. Target reject rate: under 10%.

• headword non-empty; language in the 5 supported codes; pos in the 4 supported values
• at least one sense; each sense has ≥1 definition, ≥1 example, ≥1 translation
• sense_index a non-negative integer, starting at 0 and increasing by 1
• difficulty in easy | medium | hard on both senses and translations
• target_language supported and never equal to the word's own language
• gender valid for the target language (see above); null for English
• no duplicate (headword, language, pos, sense_index)

---

Constants

Constant	Values	Source
Languages	en, it, de, es, fr	SUPPORTED_LANGUAGE_CODES
POS	noun, verb, adjective, adverb	SUPPORTED_POS
Difficulty	easy, medium, hard	DIFFICULTY_LEVELS
Gender	masculine, feminine, neuter	NOUN_GENDERS

All live in packages/shared/src/constants.ts and are CHECK-constrained in the PostgreSQL schema. Adding a value means updating the constant and a Drizzle migration before re-running the pipeline.

CEFR levels still exist as a constant and as columns on the old vocabulary_entries tables, but the new pipeline does not produce them — it produces the three-level difficulty directly. The prompt calibrates difficulty against CEFR bands internally (easy ≈ A1/A2, medium ≈ B1/B2, hard ≈ C1/C2) but is explicitly told not to emit CEFR levels.

---

Running it

docker compose up -d pipeline-database      # dedicated PostgreSQL on :5433
pnpm --filter @lila/pipeline pipeline:run   # tsx --env-file .env pipeline.ts (currently a no-op)
pnpm --filter @lila/pipeline test

Env vars come from the repo-root .env: GEMINI_API_KEY, PIPELINE_POSTGRES_USER, PIPELINE_POSTGRES_PASSWORD, PIPELINE_POSTGRES_DB, PIPELINE_DATABASE_URL. The pipeline database is deliberately separate from the app database (:5432).

Implementation notes from the roadmap: better-sqlite3 for staging (synchronous), crypto.randomUUID() for ids, JSON.stringify for the definitions/examples arrays (SQLite has no array type — the import script parses them back into PostgreSQL text[]), batches of 20 with a 1s sleep between calls.

---

Current Blockers

1. Validation module and pipeline.ts are unwritten — this is Phase 3, the active work.
2. Prompt is not templated — source language, POS, target languages, and the word batch are hardcoded for one sample run.
3. No import script — nothing moves staging rows into PostgreSQL yet (Phase 4).
4. App still reads the old schema — termModel.ts queries vocabulary_entries/entry_translations. Until Phase 5 rewrites it, pipeline output is invisible to the app.