forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

updating documentation, prettier format

Browse source
This commit is contained in:
lila 2026-08-02 01:01:33 +02:00
parent 039ed50567
commit e534b98bc5
16 changed files with 1271 additions and 1070 deletions
Download patch file Download diff file Expand all files Collapse all files

16
documentation/ai-context/00-project-overview.md
View file

@ -31,7 +31,7 @@ Live at [lilastudy.com](https://lilastudy.com).
### What's In Progress / Blocked
- **Kaikki data pipeline migration** — Replacing OpenWordNet/OMW with sense-disambiguated Kaikki data. Stage 1 (extract) and Stage 2 (reverse link) complete on sample data. Stage 3 (enrich) being rewritten for sub-stage architecture.
- **Data pipeline rewrite** — The Kaikki/local-LLM pipeline was replaced by a Gemini-only pipeline writing a new sense-based schema (`words` → `senses` → `translations`). Schema and prompt are done; the pipeline script itself is not written yet. See 05-data-pipeline.md.
- **Guest play** — No try-before-signup flow yet. Auth required for all game routes.
- **Game session store** — Still in-memory. Valkey container exists locally but not wired up.
- **Media ingestion** — Not started. No pipeline for subtitles/lyrics → vocab extraction yet.
@ -40,7 +40,7 @@ Live at [lilastudy.com](https://lilastudy.com).
The app is currently a **generic vocabulary quiz**. The media-based practice feature (the differentiator) does not exist yet. It depends on:
1. Kaikki pipeline reaching production (fixes translation quality)
1. The new data pipeline reaching production (fixes translation quality)
2. A media ingestion prototype (subtitles/lyrics → text → vocab extraction → quiz)
---
@ -50,7 +50,7 @@ The app is currently a **generic vocabulary quiz**. The media-based practice fea
| Layer | Technology |
| ------------- | -------------------------------------------------------------- |
| Monorepo | pnpm workspaces |
| Frontend | React 18, Vite, TanStack Router, TanStack Query, Tailwind CSS |
| Frontend | React 19, Vite, TanStack Router, Tailwind CSS |
| Backend | Node.js, Express, TypeScript, WebSockets (`ws` library) |
| Database | PostgreSQL + Drizzle ORM |
| Auth | Better Auth (Google + GitHub) |
@ -58,7 +58,7 @@ The app is currently a **generic vocabulary quiz**. The media-based practice fea
| Testing | Vitest, supertest |
| Deployment | Docker Compose, Caddy, Hetzner VPS |
| CI/CD | Forgejo Actions |
| Data Pipeline | Kaikki (Wiktionary) → SQLite (`pipeline.db`) → PostgreSQL |
| Data Pipeline | Gemini API → SQLite staging (`db/staging.db`) → PostgreSQL |
---
@ -72,7 +72,7 @@ lila/
├── packages/
│ ├── shared/ — Zod schemas + constants (API/web contract)
│ └── db/ — Drizzle schema, migrations, models, seeding
├── data-pipeline/ — Kaikki extraction → enrichment → PostgreSQL sync
├── data-pipeline/ — Gemini generation → SQLite staging → PostgreSQL
└── documentation/ — Project docs (human + AI-context branches)
```
@ -86,7 +86,7 @@ lila/
2. **Server-side answer evaluation** — The correct answer is never sent to the frontend. All evaluation happens server-side.
3. **Zod discriminated unions for WebSockets** — All WS messages are typed via Zod schemas in `packages/shared`. The router switches on the `type` field.
4. **GameSessionStore abstraction** — Session state is stored through an interface (`InMemoryGameSessionStore` now, `ValkeyGameSessionStore` planned).
5. **Language-neutral data model** — `terms` are concepts; `translations` are per-language words. Adding a language requires no schema changes.
5. **Sense-based data model** — `words` have `senses`, and translations hang off a sense. Adding a language requires no schema changes. Note the app still queries the older `vocabulary_entries` tables — see 02-data-model.md.
---
@ -98,7 +98,7 @@ lila/
| WebSocket | `ws` library, not Socket.io | 2–4 players, explicit Zod protocol sufficient |
| Auth | Better Auth, not Keycloak | Embedded middleware, no separate service |
| Answer eval | Server-side only | Correct answer never sent to frontend |
| Data source | Kaikki, not OMW | Sense-disambiguated translations |
| Data source | Gemini-generated, not OMW | Sense-disambiguated, language-native glosses |
---
@ -110,7 +110,7 @@ lila/
| [02-data-model.md](02-data-model.md) | Database schema, tables, relationships, constraints |
| [03-api-contract.md](03-api-contract.md) | REST endpoints, request/response schemas, Zod types |
| [04-websocket-protocol.md](04-websocket-protocol.md) | WS message types, game flow, auth, state management |
| [05-data-pipeline.md](05-data-pipeline.md) | Kaikki pipeline stages, enrich sub-stages, sync |
| [05-data-pipeline.md](05-data-pipeline.md) | Gemini pipeline flow, output contract, validation, blockers |
| [06-deployment.md](06-deployment.md) | Docker, Caddy, CI/CD, backups |
| [prompts/meta.md](prompts/meta.md) | How to work with LLMs on this codebase |
| [99-current-task.md](99-current-task.md) | Template: fill this out before giving a task to an LLM |

2
documentation/ai-context/01-architecture.md
View file

@ -16,7 +16,7 @@ lila/
├── packages/
│ ├── shared/ — Zod schemas, constants, derived types. THE CONTRACT.
│ └── db/ — Drizzle schema, migrations, models (termModel, lobbyModel), seeding
├── data-pipeline/ — Kaikki extraction → enrichment → sync to PostgreSQL
├── data-pipeline/ — Gemini generation → SQLite staging → sync to PostgreSQL
└── documentation/ — Human docs + ai-context/
```

266
documentation/ai-context/02-data-model.md
View file

@ -1,136 +1,123 @@
# 02 — Data Model
> **Purpose:** Database schema reference for LLMs working on features that query or modify data. Concatenate with 00-project-overview.md and 99-current-task.md.
> **Last updated:** 2026-05-15
> **Last updated:** 2026-08-01
> **Depends on:** 00-project-overview.md
> **Source of truth:** `packages/db/src/db/schema.ts`. If this file and the schema disagree, the schema wins.
---
## Core Tables
## Two vocabulary schemas exist right now
### `terms` — Language-neutral concepts
The database is mid-migration and contains **both** vocabulary schemas. This is the most important thing to know before writing a query.
| Column | Type | Constraints | Notes |
| ------------ | --------- | -------------------------------------------- | ------------------------------------------------------ |
| `id` | uuid | PK | |
| `pos` | varchar | CHECK: `noun`, `verb`, `adjective`, `adverb` | Part of speech |
| `source` | varchar | | Pipeline that created this term (e.g. `kaikki`, `omw`) |
| `source_id` | varchar | UNIQUE(`source`, `source_id`) | Idempotency key for imports |
| `synset_id` | varchar | nullable | WordNet synset ID. Nullable for non-WordNet terms. |
| `created_at` | timestamp | default now() | |
| Schema | Status |
| ------------------------------------------- | --------------------------------------------------------------------------------------- |
| `vocabulary_entries` + `entry_translations` | **Live.** What the app queries today (`packages/db/src/models/termModel.ts`). |
| `words` → `senses` → `translations` | **Target.** Migrated and empty. The new pipeline writes here; no app code reads it yet. |
**Rule:** One row per concept. The word "cat" (animal) and "cat" (nautical) are separate rows because they have different `source_id` values.
The `terms` / `term_glosses` / `decks` / `deck_terms` tables described in earlier versions of this doc **no longer exist**.
Migration path: the pipeline fills `words`/`senses`/`translations`, then `termModel.ts` is rewritten against it (roadmap Phase 5), then the `vocabulary_entries` tables are dropped.
---
### `translations` — Per-language words
## Live schema (what the app queries)
| Column | Type | Constraints | Notes |
| --------------- | ---------- | ----------------------------------- | ---------------------------------------- |
| `id` | uuid | PK | |
| `term_id` | uuid | FK → terms.id | |
| `language_code` | varchar(2) | CHECK: `en`, `it`, `de`, `es`, `fr` | |
| `text` | varchar | | The actual word |
| `cefr_level` | varchar(2) | nullable, CHECK: `A1`–`C2` | Difficulty of THIS word in THIS language |
| `created_at` | timestamp | default now() | |
### `vocabulary_entries` — one row per word sense
**Unique constraint:** (`term_id`, `language_code`, `text`) — allows synonyms (e.g. "dog" and "hound" for same term), prevents exact duplicates.
| Column | Type | Constraints | Notes |
| --------------- | ----------- | ----------------------------------- | ----------------------------------------- |
| `id` | uuid | PK, default random | |
| `headword` | text | NOT NULL | The word itself |
| `language_code` | varchar(10) | CHECK `SUPPORTED_LANGUAGE_CODES` | |
| `pos` | varchar(20) | CHECK `SUPPORTED_POS` | |
| `sense_index` | smallint | NOT NULL, default 0 | Distinguishes senses of the same headword |
| `gloss` | text | nullable | Definition |
| `examples` | text[] | NOT NULL, default `[]` | |
| `cefr_level` | varchar(2) | nullable, CHECK `A1`–`C2` | |
| `difficulty` | varchar(20) | nullable, CHECK `DIFFICULTY_LEVELS` | |
| `source` | varchar(50) | NOT NULL, default `"kaikki"` | |
| `created_at` | timestamptz | NOT NULL, default now() | |
**Key design:** `cefr_level` is on `translations`, not `terms`. "House" in English is A1; "domicile" is also English but B2 — same concept, different words, different difficulty.
UNIQUE (`headword`, `language_code`, `pos`, `sense_index`) · INDEX (`language_code`, `pos`, `difficulty`)
### `entry_translations`
| Column | Type | Constraints |
| ---------------------- | ----------- | ----------------------------------------------- |
| `id` | uuid | PK |
| `entry_id` | uuid | FK → `vocabulary_entries.id`, ON DELETE CASCADE |
| `target_language_code` | varchar(10) | CHECK `SUPPORTED_LANGUAGE_CODES` |
| `translation` | text | NOT NULL |
| `sense_hint` | text | nullable |
| `cefr_level` | varchar(2) | nullable, CHECK `A1`–`C2` |
| `difficulty` | varchar(20) | nullable, CHECK `DIFFICULTY_LEVELS` |
| `source` | varchar(50) | NOT NULL, default `"kaikki"` |
| `created_at` | timestamptz | NOT NULL, default now() |
UNIQUE (`entry_id`, `target_language_code`, `translation`) · INDEX (`target_language_code`, `difficulty`, `entry_id`)
---
### `term_glosses` — Definitions per language
## Target schema (what the new pipeline writes)
| Column | Type | Constraints | Notes |
| --------------- | ---------- | ----------------------------------- | ---------------------- |
| `id` | uuid | PK | |
| `term_id` | uuid | FK → terms.id | |
| `language_code` | varchar(2) | CHECK: `en`, `it`, `de`, `es`, `fr` | |
| `text` | text | | Definition/explanation |
| `created_at` | timestamp | default now() | |
Three levels: a **word** has **senses**, and translations hang off a **sense**, not off the word. That is the point of the redesign — a quiz question is tied to one specific meaning.
**Unique constraint:** (`term_id`, `language_code`) — one gloss per term per language. Prevents left joins from multiplying question rows.
### `words`
**Note:** Italian gloss coverage is sparse (~2% of terms have Italian glosses). UI falls back to English gloss when no gloss exists for the user's language.
| Column | Type | Constraints |
| --------------- | ----------- | -------------------------------- |
| `id` | uuid | PK |
| `headword` | text | NOT NULL |
| `language_code` | varchar(10) | CHECK `SUPPORTED_LANGUAGE_CODES` |
| `pos` | varchar(20) | CHECK `SUPPORTED_POS` |
| `created_at` | timestamptz | NOT NULL, default now() |
---
UNIQUE `unique_word_per_language_and_pos` (`headword`, `language_code`, `pos`) · INDEX `idx_language_code_pos`
### `decks` — Curated wordlists
### `senses`
| Column | Type | Constraints | Notes |
| --------------------- | ------------ | ------------------------------------------------- | ------------------------------------------------------- |
| `id` | uuid | PK | |
| `name` | varchar | | e.g. `en-core-1000` |
| `source_language` | varchar(2) | CHECK | Language the wordlist was built from |
| `validated_languages` | varchar(2)[] | CHECK: source_language NOT IN validated_languages | Languages with complete translations for all deck terms |
| `description` | text | nullable | |
| `created_at` | timestamp | default now() | |
| Column | Type | Constraints |
| ------------- | ----------- | ----------------------------------- |
| `id` | uuid | PK |
| `word_id` | uuid | FK → `words.id`, ON DELETE CASCADE |
| `sense_index` | smallint | NOT NULL, default 0 |
| `difficulty` | varchar(20) | NOT NULL, CHECK `DIFFICULTY_LEVELS` |
| `definitions` | text[] | NOT NULL, default `[]` |
| `examples` | text[] | NOT NULL, default `[]` |
| `created_at` | timestamptz | NOT NULL, default now() |
**Design:** One deck per frequency tier per source language. POS, difficulty, and category are query filters, not separate decks. Decks must not overlap — each term appears in exactly one tier.
UNIQUE `unique_sense_per_word` (`word_id`, `sense_index`) · INDEX `idx_word_sense_difficulty`
**Source:** SUBTLEX frequency lists (per-language editions, same methodology).
### `translations`
---
| Column | Type | Constraints |
| ---------------------- | ----------- | -------------------------------------- |
| `id` | uuid | PK |
| `sense_id` | uuid | FK → `senses.id`, ON DELETE CASCADE |
| `target_language_code` | varchar(10) | CHECK `SUPPORTED_LANGUAGE_CODES` |
| `translation` | text | NOT NULL |
| `gender` | varchar(20) | nullable, CHECK NULL or `NOUN_GENDERS` |
| `difficulty` | varchar(20) | NOT NULL, CHECK `DIFFICULTY_LEVELS` |
| `created_at` | timestamptz | NOT NULL, default now() |
### `deck_terms` — Junction table
UNIQUE `unique_translation_per_sense` (`sense_id`, `target_language_code`, `translation`) · INDEX `idx_translations_sense_language_difficulty`
| Column | Type | Constraints | Notes |
| ------------ | --------- | ------------- | ----- |
| `deck_id` | uuid | FK → decks.id | |
| `term_id` | uuid | FK → terms.id | |
| `created_at` | timestamp | default now() | |
**Two difficulty columns, two meanings.** `senses.difficulty` = is this _meaning_ appropriate for the level. `translations.difficulty` = is this _word_ an acceptable answer. Queries use sense difficulty as a ceiling and translation difficulty as the target; a translation's difficulty is never lower than its sense's. See `documentation/pipeline/design-doc.md` §4.
**PK:** (`deck_id`, `term_id`)
**Note:** `DIFFICULTY_LEVELS` is `easy | medium | hard`. `"intermediate"` was renamed to `"medium"` and no longer exists anywhere.
---
## Auth Tables (managed by Better Auth)
Better Auth creates and owns these tables. Do not modify directly.
Better Auth creates and owns `user`, `session`, `account`, and `verification`. Do not modify them directly — changes come from Better Auth config. `user.id` is `text`, not uuid, so foreign keys to it must also be `text`.
### `user`
| Column | Type | Notes |
| ---------------- | --------- | -------------------- |
| `id` | varchar | PK |
| `name` | varchar | Display name |
| `email` | varchar | |
| `email_verified` | boolean | |
| `image` | varchar | nullable, avatar URL |
| `created_at` | timestamp | |
| `updated_at` | timestamp | |
### `session`
| Column | Type | Notes |
| ------------ | --------- | ------------- |
| `id` | varchar | PK |
| `user_id` | varchar | FK → user.id |
| `token` | varchar | Session token |
| `expires_at` | timestamp | |
| `ip_address` | varchar | nullable |
| `user_agent` | text | nullable |
| `created_at` | timestamp | |
### `account` — Social provider links
| Column | Type | Notes |
| --------------- | --------- | -------------------- |
| `id` | varchar | PK |
| `user_id` | varchar | FK → user.id |
| `account_id` | varchar | Provider's user ID |
| `provider_id` | varchar | `google` or `github` |
| `access_token` | text | nullable |
| `refresh_token` | text | nullable |
| `id_token` | text | nullable |
| `expires_at` | timestamp | nullable |
**Note:** One user can have multiple accounts (Google + GitHub linked to same user).
### `verification`
Email verification tokens. Unused for social-only auth but managed by Better Auth.
- `user` — `id`, `name`, `email` (unique), `email_verified`, `image`, timestamps
- `session` — `id`, `user_id`, `token`, `expires_at`, `ip_address`, `user_agent`
- `account` — social provider links; one user can have both Google and GitHub
- `verification` — email verification tokens; managed but unused for social-only auth
---
@ -138,84 +125,61 @@ Email verification tokens. Unused for social-only auth but managed by Better Aut
### `lobbies`
| Column | Type | Constraints | Notes |
| ------------- | --------- | ------------------------------------------- | -------------------------------------------- |
| `id` | uuid | PK | |
| `code` | varchar | UNIQUE | Human-readable room code (e.g. `WOLF-42`) |
| `host_id` | varchar | FK → user.id | |
| `status` | varchar | CHECK: `waiting`, `in_progress`, `finished` | |
| `max_players` | integer | default 4 | |
| `settings` | jsonb | nullable | Game mode, round count, timer duration, etc. |
| `created_at` | timestamp | default now() | |
| `updated_at` | timestamp | default now() | Used for stale recovery |
| Column | Type | Constraints |
| -------------- | ----------- | --------------------------------------------------- |
| `id` | uuid | PK |
| `code` | varchar(10) | NOT NULL, UNIQUE — room code |
| `host_user_id` | text | FK → `user.id`, ON DELETE CASCADE |
| `status` | varchar(20) | NOT NULL, default `waiting`, CHECK `LOBBY_STATUSES` |
| `created_at` | timestamptz | NOT NULL, default now() |
### `lobby_players`
| Column | Type | Constraints | Notes |
| -------------- | --------- | --------------- | ---------------------------- |
| `id` | uuid | PK | |
| `lobby_id` | uuid | FK → lobbies.id | |
| `user_id` | varchar | FK → user.id | |
| `display_name` | varchar | | Player's shown name in lobby |
| `is_host` | boolean | default false | |
| `joined_at` | timestamp | default now() | |
| Column | Type | Constraints |
| ----------- | ----------- | ------------------------------------ |
| `lobby_id` | uuid | FK → `lobbies.id`, ON DELETE CASCADE |
| `user_id` | text | FK → `user.id`, ON DELETE CASCADE |
| `score` | integer | NOT NULL, default 0 |
| `joined_at` | timestamptz | NOT NULL, default now() |
**Unique constraint:** (`lobby_id`, `user_id`) — one entry per player per lobby.
**Composite PK:** (`lobby_id`, `user_id`) — no surrogate `id` column, one row per player per lobby.
Only lobby _membership_ is persisted. Live game state (questions, timers, per-round answers) lives in the in-memory stores in `apps/api`, not in these tables. Max players is the `MAX_LOBBY_PLAYERS` constant in `packages/shared`, not a column.
---
## Key Relationships
```
terms (1) ←──→ (N) translations
terms (1) ←──→ (N) term_glosses
terms (N) ←──→ (N) decks via deck_terms
user (1) ←──→ (N) sessions
user (1) ←──→ (N) accounts
vocabulary_entries (1) ←──→ (N) entry_translations ← live
words (1) ←──→ (N) senses (1) ←──→ (N) translations ← target
user (1) ←──→ (N) session
user (1) ←──→ (N) account
user (1) ←──→ (N) lobbies (as host)
user (1) ←──→ (N) lobby_players
lobbies (1) ←──→ (N) lobby_players
lobbies (1) ←──→ (N) lobby_players (N) ←──→ (1) user
```
---
## Query Patterns
### Get quiz terms (singleplayer)
All queries live in `packages/db/src/models/` — `apps/api` never imports `drizzle-orm`.
```sql
SELECT t.id, t.pos, src.text AS source_text, tgt.text AS target_text, g.text AS gloss
FROM terms t
JOIN translations src ON src.term_id = t.id AND src.language_code = ?
JOIN translations tgt ON tgt.term_id = t.id AND tgt.language_code = ?
LEFT JOIN term_glosses g ON g.term_id = t.id AND g.language_code = ?
WHERE t.pos = ? AND tgt.cefr_level IN (?)
LIMIT ?
```
`termModel.ts` queries the live schema: it self-joins `vocabulary_entries` (aliased source and target), joins `entry_translations` for the answer, and fetches distractors with a separate query per question that excludes both the current entry id and the correct answer text — different entries can share a translation string.
### Get distractors
The distractor query is N+1, one round trip per question. Batching is a known BACKLOG item.
```sql
SELECT text FROM translations
WHERE language_code = ? AND pos = ? AND cefr_level IN (?)
AND term_id != ? AND text != ?
ORDER BY RANDOM()
LIMIT 3
```
**Note:** This is the N+1 query mentioned in BACKLOG.md. Each question fetches 3 distractors separately. Batching is planned.
Phase 5 rewrites these against `words`/`senses`/`translations`, which changes the shape: filter on `senses.difficulty` as a ceiling, then select `translations` at the requested difficulty. Target queries are sketched in `documentation/pipeline/design-doc.md` §5.
---
## Deferred Schema Extensions (Not Yet Implemented)
These tables are planned but do not exist yet. All are additive — they reference existing `terms` rows via FK.
Planned, additive, and keyed off the **target** schema:
| Table | Purpose | Trigger |
| --------------------- | ----------------------------------------------- | ----------------------- |
| `noun_forms` | Gender, singular, plural, articles per language | Grammar quiz mode |
| `verb_forms` | Conjugation tables per language | Grammar quiz mode |
| `term_pronunciations` | IPA + audio URLs per language | Pronunciation quiz mode |
| `user_decks` | Which decks a user studies | User customization |
| `user_term_progress` | Spaced repetition state per user/term/language | SRS review queue |
| `quiz_answers` | Answer history for stats/analytics | User stats dashboard |
| Table | Purpose | Trigger |
| -------------------- | ---------------------------------------------- | ----------------------- |
| `inflection_forms` | Gender, plural, conjugation/declension tables | Grammar quiz mode |
| `pronunciations` | IPA + audio URLs per language | Pronunciation quiz mode |
| `user_word_progress` | Spaced repetition state per user/word/language | SRS review queue |
| `quiz_answers` | Answer history for stats/analytics | User stats dashboard |

2
documentation/ai-context/03-api-contract.md
View file

@ -33,7 +33,7 @@ POST /api/v1/game/start
source_language: SupportedLanguageCode, // "en" | "it" | "de" | "es" | "fr"
target_language: SupportedLanguageCode,
pos: SupportedPos, // "noun" | "verb" | "adjective" | "adverb"
difficulty: DifficultyLevel, // "easy" | "intermediate" | "hard"
difficulty: DifficultyLevel, // "easy" | "medium" | "hard"
rounds: GameRounds // "3" | "10" (string enum, converted to number in service)
}
```

226
documentation/ai-context/05-data-pipeline.md
View file

@ -1,173 +1,151 @@
# 05 — Data Pipeline
> **Purpose:** Condensed reference for LLMs working on the Kaikki data pipeline. Covers stages, data flow, and current blockers. For full operational details (llama.cpp setup, provider configs, hardware specs), see the human-readable DATA_PIPELINE.md.
> **Last updated:** 2026-05-15
> **Depends on:** 00-project-overview.md
> **Purpose:** Condensed reference for LLMs working on the vocabulary data pipeline. Covers the flow, what exists, and what is still unwritten. Full detail: `documentation/DATA_PIPELINE.md`, `documentation/pipeline/design-doc.md`, `documentation/pipeline/roadmap.md`.
> **Last updated:** 2026-08-01
> **Depends on:** 00-project-overview.md, 02-data-model.md
---
## Read this first
The pipeline was **completely rewritten**. The old Kaikki/local-LLM architecture — six stages, `pipeline.db`, `stage-1-extract/`, `stage-3-enrich/`, multi-model CEFR voters, llama.cpp — is **gone from the codebase**. Any reference you see to those stages, directories, or the voter strategy is historical (`documentation/archive/`), not something you can call or modify.
The current pipeline is Gemini-only, and most of it **is not written yet**. Do not assume a module exists because a doc names it.
---
## Pipeline Overview
```
Kaikki JSONL (Wiktionary extracts)
source-data/{lang}/{pos} frequency wordlists, one word per line, UTF-8
↓
Stage 1: Extract → Parse into pipeline.db (SQLite)
Gemini API batches of 20 words, one language at a time
↓
Stage 2: Reverse Link → Insert missing reverse translations
validation per entry; rejects go to a log, never to the DB
↓
Stage 3: Enrich → LLMs review glosses, examples, translations, assign CEFR
db/staging.db SQLite staging (words, senses, translations)
↓
Stage 4: Merge → Resolve LLM votes into final values
import script SQLite → PostgreSQL via Drizzle, transaction per batch
↓
Stage 4b: Tiebreak → Run unused models on flagged entries
↓
Stage 5: Compare / QA → Generate COVERAGE.md quality report
↓
Stage 6: Sync → Upsert resolved records into production PostgreSQL
PostgreSQL dev (:5432), then production
```
**Current state:** Stage 1 and 2 complete on sample data. Stage 3 enrich script being rewritten for sub-stage architecture. Stages 4–6 not started.
Each language is processed independently so that definitions and examples are written **in that language**. Only translations cross language boundaries.
The app always reads PostgreSQL. SQLite is a staging file only, so re-runs and prompt tweaks never touch a real database.
**Current state:** Phases 1–2 complete (schema, wordlists, databases, prompt). Phase 3 in progress — validation module, pipeline script, and first real run are all still to be written. Phases 4–7 not started.
---
## Stage 1: Extract
## What exists in `data-pipeline/`
**Input:** `data-pipeline/stage-1-extract/sources/*.jsonl` (Kaikki files, not in git)
**Output:** `pipeline.db` — `vocabulary_entries` and `entry_translations` tables
| Path | State |
| --------------------------- | ------------------------------------------------------------------ |
| `source-data/{lang}/{pos}/` | ✅ Noun lists for `de`, `en`, `es`, `fr`, `it` |
| `prompt` | ✅ Gemini system prompt — a plain UTF-8 text file, not a TS module |
| `db/schema.sql` | ✅ SQLite staging schema |
| `db/staging.db` | ✅ Tables created, **0 rows**, gitignored |
| `pipeline.ts` | 🚧 Design pseudocode in comments only — no executable code |
| validation module | ❌ Not written (rules in design-doc §6.4) |
| SQLite → PostgreSQL import | ❌ Not written |
| `kaikki-source-files/` | ⚠️ Leftover JSONL dumps; nothing reads them |
| `worddata/english/nouns/` | ⚠️ Empty leftover output dir from the old per-word-JSON design |
**What it does:**
Directory names use the codes from `packages/shared/src/constants.ts` (`de/noun`, not `german/nouns`) so no mapping layer is needed.
- Parses Kaikki JSONL for all 5 languages (en, de, es, fr, it)
- Filters to 4 POS: noun, verb, adjective, adverb
- Each Kaikki sense becomes one `vocabulary_entries` row
- Translations stored in `entry_translations` with sense hints
**Key design:** Kaikki is structured per word sense. Each headword has multiple senses, and translations are linked to a specific sense. This prevents the sense-disambiguation problems of OpenWordNet/OMW.
`data-pipeline/vitest.config.ts` looks for `tests/**/*.test.ts`; that directory does not exist yet.
---
## Stage 2: Reverse Link Sync
## Gemini output contract
**Pure script, no LLMs.**
The model returns a JSON array, one object per input word, in input order — no markdown fences, comments, or trailing commas.
For each translation pair (e.g., English "thrill" → German "begeistern"), checks if the reverse exists (German "begeistern" → English "thrill"). If the German entry exists but lacks the English back-link, inserts it automatically.
<!-- prettier-ignore -->
```json
[
{
"headword": "Haus",
"language": "de",
"pos": "noun",
"senses": [
{
"sense_index": 0,
"difficulty": "easy",
"definitions": ["Ein Gebäude zum Wohnen."],
"examples": ["Sie kauften ein Haus in der Stadt."],
"translations": [
{ "target_language": "en", "word": "house", "gender": null, "difficulty": "easy" },
{ "target_language": "es", "word": "casa", "gender": "feminine", "difficulty": "easy" }
]
}
]
}
]
```
**Why:** Ensures LLMs in Stage 3 only generate translations that are genuinely missing — not translations findable by simple reverse lookup.
Prompt rules that the output depends on:
- Definitions and examples in the **source** language, not English.
- Gender required for `de` (masculine/feminine/neuter) and `it`/`es`/`fr` (masculine/feminine); always `null` for `en`.
- German nouns capitalized; Romance-language nouns lowercase unless proper nouns.
- Base dictionary form, no articles or determiners.
- 1–3 senses per word, most words 1; no rare, archaic, or technical senses.
- Max 2 translations per target language per sense, only genuine synonyms or difficulty variants.
- A translation's difficulty is never lower than its sense's difficulty.
- A word that is not a valid noun in that language returns `"senses": []`.
⚠️ The checked-in `prompt` file still has hardcoded English leftovers (rules 2, 3, and 31 say `"en"` / "English noun" while the header says Spanish) and its target-language list disagrees with its own header. It is also pinned to one sample batch rather than templated. Fix when implementing `pipeline.ts`.
---
## Stage 3: Enrich (In Progress — Being Rewritten)
## Validation rules (design-doc §6.4)
**Current blocker:** The original single-prompt design had problems (skipped invalid translations, triggered reasoning mode, 20% manual review). Being rewritten as four ordered sub-stages.
Run per entry before anything is written to SQLite. Invalid entries go to a rejection log for review, not to the database. Target reject rate: under 10%.
### Sub-Stage Architecture
Each model processes every entry through four sub-stages in order:
1. **`round1_gloss`** — Review existing gloss. Confirm if clear, generate better one if not.
2. **`round1_example`** — Review examples. Confirm if natural, generate one better sentence.
3. **`round1_translations`** — Validate translations with verified gloss as context. Confirm valid, reject invalid, generate missing.
4. **`round1_cefr`** — Assign CEFR level (A1–C2) to headword and each confirmed translation.
**Why this order:** CEFR sub-stage only sees clean, verified data. Bad translations are rejected before reaching CEFR assignment.
**Voter strategy:** Multiple models vote independently. Each model = one vote per sub-stage. Current plan:
- Primary: Local Qwen3.5-9B (overnight runs, unlimited)
- Secondary: Groq Llama 3.3 70B (cloud, batched)
- Tertiary: Gemini AI Studio (cloud, batched)
**Context enrichment:** Before calling models for gloss/example, pipeline queries Wiktionary API for the headword. Full entry (all senses, usage notes) added to prompt. Fixes category header glosses and short ambiguous glosses.
- `headword` non-empty; `language` in the 5 supported codes; `pos` in the 4 supported values
- at least one sense; each sense has ≥1 definition, ≥1 example, ≥1 translation
- `sense_index` a non-negative integer, starting at 0 and increasing by 1
- `difficulty` in `easy | medium | hard` on both senses and translations
- `target_language` supported and never equal to the word's own language
- `gender` valid for the target language (see above); `null` for English
- no duplicate (headword, language, pos, sense_index)
---
## Stage 4: Merge
## Constants
Resolves LLM votes into final values per entry.
| Constant | Values | Source |
| ---------- | ------------------------------------- | -------------------------- |
| Languages | `en`, `it`, `de`, `es`, `fr` | `SUPPORTED_LANGUAGE_CODES` |
| POS | `noun`, `verb`, `adjective`, `adverb` | `SUPPORTED_POS` |
| Difficulty | `easy`, `medium`, `hard` | `DIFFICULTY_LEVELS` |
| Gender | `masculine`, `feminine`, `neuter` | `NOUN_GENDERS` |
**Rules:**
All live in `packages/shared/src/constants.ts` and are CHECK-constrained in the PostgreSQL schema. Adding a value means updating the constant **and** a Drizzle migration before re-running the pipeline.
- Kaikki source data wins automatically (never overridden)
- CEFR: level with most votes wins
- Text fields (gloss, example, translation): candidate with most votes wins
- No majority → flag for tiebreaker
**Difficulty mapping:**
| CEFR | Difficulty |
|------|-----------|
| A1, A2 | easy |
| B1, B2 | intermediate |
| C1, C2 | hard |
CEFR levels still exist as a constant and as columns on the old `vocabulary_entries` tables, but the new pipeline does not produce them — it produces the three-level difficulty directly. The prompt calibrates difficulty against CEFR bands internally (easy ≈ A1/A2, medium ≈ B1/B2, hard ≈ C1/C2) but is explicitly told not to emit CEFR levels.
---
## Stage 4b: Tiebreak
## Running it
Runs automatically after merge if flagged entries remain. Queries unused models (not yet voted) and re-runs merge. Repeats until resolved or no unused models remain.
```bash
docker compose up -d pipeline-database # dedicated PostgreSQL on :5433
pnpm --filter @lila/pipeline pipeline:run # tsx --env-file .env pipeline.ts (currently a no-op)
pnpm --filter @lila/pipeline test
```
**If still unresolved:** Sync is blocked. Add more models to config and re-run.
Env vars come from the repo-root `.env`: `GEMINI_API_KEY`, `PIPELINE_POSTGRES_USER`, `PIPELINE_POSTGRES_PASSWORD`, `PIPELINE_POSTGRES_DB`, `PIPELINE_DATABASE_URL`. The pipeline database is deliberately separate from the app database (`:5432`).
---
## Stage 5: Compare / QA
Read-only. Generates `COVERAGE.md` with per-language breakdown:
- Total entries, POS distribution
- Translation coverage per language pair
- CEFR coverage and difficulty breakdown
- Gloss/example coverage by source (Kaikki vs LLM)
- Per-model contribution stats
Run this before syncing to production.
---
## Stage 6: Sync
Upserts all `status = "final"` entries from `pipeline.db` to production PostgreSQL.
**Behavior:**
- Missing → insert
- Present but changed → update
- Present and unchanged → skip
**Idempotent.** Safe to re-run.
---
## Key Constraints
| Constant | Values |
| ---------- | ------------------------------------- |
| Languages | `en`, `it`, `de`, `es`, `fr` |
| POS | `noun`, `verb`, `adjective`, `adverb` |
| CEFR | `A1`, `A2`, `B1`, `B2`, `C1`, `C2` |
| Difficulty | `easy`, `intermediate`, `hard` |
Adding a new value requires updating `packages/shared/src/constants.ts` AND a database migration before re-running the pipeline.
Implementation notes from the roadmap: `better-sqlite3` for staging (synchronous), `crypto.randomUUID()` for ids, `JSON.stringify` for the definitions/examples arrays (SQLite has no array type — the import script parses them back into PostgreSQL `text[]`), batches of 20 with a 1s sleep between calls.
---
## Current Blockers
1. **Enrich sub-stage rewrite** — Stage 3 script needs redesign and testing
2. **Cloud provider integration** — Groq and Gemini not yet wired into pipeline
3. **Batching prompt design** — 5–10 entries per API call for efficiency; not yet designed
4. **Full dataset scale unknown** — Currently running on 500-entry samples. Full Kaikki English file has ~1.3M entries. Exact filtered count and runtime estimate not yet known.
---
## Key Files
| File | Purpose |
| ------------------------------------------------------------ | --------------------------------------------------------- |
| `data-pipeline/pipeline.ts` | Orchestrator — runs stages in order, handles resumability |
| `data-pipeline/stage-1-extract/scripts/extract.ts` | Parse Kaikki JSONL |
| `data-pipeline/stage-2-reverse-link/scripts/reverse-link.ts` | Insert reverse translations |
| `data-pipeline/stage-3-enrich/scripts/enrich.ts` | LLM enrichment (being rewritten) |
| `data-pipeline/stage-3-enrich/config.ts` | Provider configs (local, OpenRouter, etc.) |
| `data-pipeline/db/schema.sql` | pipeline.db schema |
| `data-pipeline/db/import.ts` | Import stage 1 output into pipeline.db |
| `packages/shared/src/constants.ts` | Language codes, POS, CEFR, difficulty constants |
1. **Validation module and `pipeline.ts` are unwritten** — this is Phase 3, the active work.
2. **Prompt is not templated** — source language, POS, target languages, and the word batch are hardcoded for one sample run.
3. **No import script** — nothing moves staging rows into PostgreSQL yet (Phase 4).
4. **App still reads the old schema** — `termModel.ts` queries `vocabulary_entries`/`entry_translations`. Until Phase 5 rewrites it, pipeline output is invisible to the app.