# 05 — Data Pipeline

> **Purpose:** Condensed reference for LLMs working on the vocabulary data pipeline. Covers the flow, what exists, and what is still unwritten. Full detail: `documentation/DATA_PIPELINE.md`, `documentation/pipeline/design-doc.md`, `documentation/pipeline/roadmap.md`.
> **Last updated:** 2026-08-01
> **Depends on:** 00-project-overview.md, 02-data-model.md

---

## Read this first

The pipeline was **completely rewritten**. The old Kaikki/local-LLM architecture — six stages, `pipeline.db`, `stage-1-extract/`, `stage-3-enrich/`, multi-model CEFR voters, llama.cpp — is **gone from the codebase**. Any reference you see to those stages, directories, or the voter strategy is historical (`documentation/archive/`), not something you can call or modify.

The current pipeline is Gemini-only, and most of it **is not written yet**. Do not assume a module exists because a doc names it.

---

## Pipeline Overview

```
source-data/{lang}/{pos}   frequency wordlists, one word per line, UTF-8
     ↓
Gemini API                 batches of 20 words, one language at a time
     ↓
validation                 per entry; rejects go to a log, never to the DB
     ↓
db/staging.db              SQLite staging (words, senses, translations)
     ↓
import script              SQLite → PostgreSQL via Drizzle, transaction per batch
     ↓
PostgreSQL                 dev (:5432), then production
```

Each language is processed independently so that definitions and examples are written **in that language**. Only translations cross language boundaries.

The app always reads PostgreSQL. SQLite is a staging file only, so re-runs and prompt tweaks never touch a real database.

**Current state:** Phases 1–2 complete (schema, wordlists, databases, prompt). Phase 3 in progress — validation module, pipeline script, and first real run are all still to be written. Phases 4–7 not started.

---

## What exists in `data-pipeline/`

| Path                        | State                                                              |
| --------------------------- | ------------------------------------------------------------------ |
| `source-data/{lang}/{pos}/` | ✅ Noun lists for `de`, `en`, `es`, `fr`, `it`                     |
| `prompt`                    | ✅ Gemini system prompt — a plain UTF-8 text file, not a TS module |
| `db/schema.sql`             | ✅ SQLite staging schema                                           |
| `db/staging.db`             | ✅ Tables created, **0 rows**, gitignored                          |
| `pipeline.ts`               | 🚧 Design pseudocode in comments only — no executable code         |
| validation module           | ❌ Not written (rules in design-doc §6.4)                          |
| SQLite → PostgreSQL import  | ❌ Not written                                                     |
| `kaikki-source-files/`      | ⚠️ Leftover JSONL dumps; nothing reads them                        |
| `worddata/english/nouns/`   | ⚠️ Empty leftover output dir from the old per-word-JSON design     |

Directory names use the codes from `packages/shared/src/constants.ts` (`de/noun`, not `german/nouns`) so no mapping layer is needed.

`data-pipeline/vitest.config.ts` looks for `tests/**/*.test.ts`; that directory does not exist yet.

---

## Gemini output contract

The model returns a JSON array, one object per input word, in input order — no markdown fences, comments, or trailing commas.

<!-- prettier-ignore -->
```json
[
  {
    "headword": "Haus",
    "language": "de",
    "pos": "noun",
    "senses": [
      {
        "sense_index": 0,
        "difficulty": "easy",
        "definitions": ["Ein Gebäude zum Wohnen."],
        "examples": ["Sie kauften ein Haus in der Stadt."],
        "translations": [
          { "target_language": "en", "word": "house", "gender": null, "difficulty": "easy" },
          { "target_language": "es", "word": "casa", "gender": "feminine", "difficulty": "easy" }
        ]
      }
    ]
  }
]
```

Prompt rules that the output depends on:

- Definitions and examples in the **source** language, not English.
- Gender required for `de` (masculine/feminine/neuter) and `it`/`es`/`fr` (masculine/feminine); always `null` for `en`.
- German nouns capitalized; Romance-language nouns lowercase unless proper nouns.
- Base dictionary form, no articles or determiners.
- 1–3 senses per word, most words 1; no rare, archaic, or technical senses.
- Max 2 translations per target language per sense, only genuine synonyms or difficulty variants.
- A translation's difficulty is never lower than its sense's difficulty.
- A word that is not a valid noun in that language returns `"senses": []`.

⚠️ The checked-in `prompt` file still has hardcoded English leftovers (rules 2, 3, and 31 say `"en"` / "English noun" while the header says Spanish) and its target-language list disagrees with its own header. It is also pinned to one sample batch rather than templated. Fix when implementing `pipeline.ts`.

---

## Validation rules (design-doc §6.4)

Run per entry before anything is written to SQLite. Invalid entries go to a rejection log for review, not to the database. Target reject rate: under 10%.

- `headword` non-empty; `language` in the 5 supported codes; `pos` in the 4 supported values
- at least one sense; each sense has ≥1 definition, ≥1 example, ≥1 translation
- `sense_index` a non-negative integer, starting at 0 and increasing by 1
- `difficulty` in `easy | medium | hard` on both senses and translations
- `target_language` supported and never equal to the word's own language
- `gender` valid for the target language (see above); `null` for English
- no duplicate (headword, language, pos, sense_index)

---

## Constants

| Constant   | Values                                | Source                     |
| ---------- | ------------------------------------- | -------------------------- |
| Languages  | `en`, `it`, `de`, `es`, `fr`          | `SUPPORTED_LANGUAGE_CODES` |
| POS        | `noun`, `verb`, `adjective`, `adverb` | `SUPPORTED_POS`            |
| Difficulty | `easy`, `medium`, `hard`              | `DIFFICULTY_LEVELS`        |
| Gender     | `masculine`, `feminine`, `neuter`     | `NOUN_GENDERS`             |

All live in `packages/shared/src/constants.ts` and are CHECK-constrained in the PostgreSQL schema. Adding a value means updating the constant **and** a Drizzle migration before re-running the pipeline.

CEFR levels still exist as a constant and as columns on the old `vocabulary_entries` tables, but the new pipeline does not produce them — it produces the three-level difficulty directly. The prompt calibrates difficulty against CEFR bands internally (easy ≈ A1/A2, medium ≈ B1/B2, hard ≈ C1/C2) but is explicitly told not to emit CEFR levels.

---

## Running it

```bash
docker compose up -d pipeline-database      # dedicated PostgreSQL on :5433
pnpm --filter @lila/pipeline pipeline:run   # tsx --env-file .env pipeline.ts (currently a no-op)
pnpm --filter @lila/pipeline test
```

Env vars come from the repo-root `.env`: `GEMINI_API_KEY`, `PIPELINE_POSTGRES_USER`, `PIPELINE_POSTGRES_PASSWORD`, `PIPELINE_POSTGRES_DB`, `PIPELINE_DATABASE_URL`. The pipeline database is deliberately separate from the app database (`:5432`).

Implementation notes from the roadmap: `better-sqlite3` for staging (synchronous), `crypto.randomUUID()` for ids, `JSON.stringify` for the definitions/examples arrays (SQLite has no array type — the import script parses them back into PostgreSQL `text[]`), batches of 20 with a 1s sleep between calls.

---

## Current Blockers

1. **Validation module and `pipeline.ts` are unwritten** — this is Phase 3, the active work.
2. **Prompt is not templated** — source language, POS, target languages, and the word batch are hardcoded for one sample run.
3. **No import script** — nothing moves staging rows into PostgreSQL yet (Phase 4).
4. **App still reads the old schema** — `termModel.ts` queries `vocabulary_entries`/`entry_translations`. Until Phase 5 rewrites it, pipeline output is invisible to the app.