diff --git a/documentation/pipeline/design-doc.md b/documentation/pipeline/design-doc.md new file mode 100644 index 0000000..a1d7666 --- /dev/null +++ b/documentation/pipeline/design-doc.md @@ -0,0 +1,617 @@ +# Vocabulary Trainer — Data Schema Design Document + +> **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline +> **Author:** [Your Name] +> **Date:** July 2026 +> **Status:** Approved — ready for implementation + +--- + +## 1. Overview + +This document describes the database schema, data pipeline, and query +patterns for the vocabulary trainer application. The app supports 5 +languages (English, German, Italian, French, Spanish) and tests learners +by showing a word with a definition and example sentence, then asking +them to pick the correct translation from 4 options (1 correct, 3 +distractors). + +The previous data source (OpenWordNet / kaikki.org) produced +low-quality and inaccurate entries. The new approach uses a batch +pipeline: word frequency lists are fed to the Gemini API in groups of +20, which generates structured metadata (definitions, examples, +translations, difficulty levels). The output is validated, stored in a +local SQLite staging database, and mirrored into the production +Postgres database. + +--- + +## 2. Database Choice + +**Postgres** (production and development) with **SQLite** (pipeline +staging). + +### Why Postgres + +- The data is inherently relational: words → senses → translations. + Foreign keys enforce referential integrity at the database level. +- The query pattern (filter → join → random → limit) is exactly what + SQL is designed for. +- Postgres provides JSONB and native arrays for semi-structured fields + (definitions, examples, inflection tags) without sacrificing + relational structure. +- ACID transactions ensure batch imports are atomic. +- Already part of the PERN stack. No additional infrastructure. + +### Why not MongoDB + +- The data has real, meaningful relationships (not arbitrary nested + documents). The distractor query ("exclude translations from the + same sense") is a single `WHERE sense_id != X` in SQL but requires + a complex aggregation pipeline in MongoDB. +- No foreign key enforcement. Data integrity would depend entirely on + application code — risky with LLM-generated data. +- Future inflection tables (one word → 30–50 forms) are a natural + relational fit, not a document-store fit. + +### Why not DynamoDB + +- Designed for simple key-value lookups at massive scale (billions of + rows). Cannot do ad-hoc filtering, joins, or `ORDER BY RANDOM()`. +- The query pattern (filter by language + pos + difficulty, then + randomize) would require pre-building indexes for every combination. +- Massive overkill for 500k words. + +### Why SQLite for staging + +- Zero-config, file-based. Ideal for the single-writer batch pipeline. +- The pipeline writes to SQLite, a separate import script mirrors the + data into Postgres. The application (dev and prod) always reads + from Postgres to avoid SQLite/Postgres dialect differences. + +--- + +## 3. Schema + +### 3.1 Entity Relationship + +``` +┌─────────────┐ ┌─────────────┐ ┌──────────────────┐ +│ words │ │ senses │ │ translations │ +├─────────────┤ ├─────────────┤ ├──────────────────┤ +│ id (PK) │──┐ │ id (PK) │──┐ │ id (PK) │ +│ headword │ └───>│ word_id(FK) │ └───>│ sense_id (FK) │ +│ language_code│ │ sense_index │ │ target_lang_code │ +│ pos │ │ difficulty │ │ translation │ +│ │ │ cefr_level │ │ gender │ +│ │ │ definitions │ │ difficulty │ +│ │ │ examples │ │ │ +└─────────────┘ └─────────────┘ └──────────────────┘ + +Future (not yet implemented): +┌──────────────────┐ +│ inflection_forms │ +├──────────────────┤ +│ id (PK) │ +│ word_id (FK) ────────> words.id +│ form │ +│ tags (JSONB) │ +└──────────────────┘ +``` + +### 3.2 Table: `words` + +One row per unique word in a specific language. + +| Column | Type | Constraints | Notes | +| ------------- | ----------- | --------------------------------- | ------------------------------------- | +| id | UUID | PK, default random | | +| headword | TEXT | NOT NULL | "Haus", "casa", "house" | +| language_code | VARCHAR(10) | NOT NULL, CHECK in supported list | "de", "es", "en", "fr", "it" | +| pos | VARCHAR(20) | NOT NULL, CHECK in supported list | "noun", "verb", "adjective", "adverb" | +| created_at | TIMESTAMPTZ | NOT NULL, default now() | | + +**Constraints:** + +- `UNIQUE (headword, language_code, pos)` — prevents duplicate entries. +- `CHECK (language_code IN ('en','de','it','fr','es'))` +- `CHECK (pos IN ('noun','verb','adjective','adverb'))` + +**Index:** + +- `idx_words_lang_pos ON (language_code, pos)` — accelerates the + primary game query filter. + +**Design note:** Each language gets its own headword entries. "Haus" +is a German word row. "casa" is a Spanish word row. They are separate +entries, linked through the translations table. This is what enables +any language pair as source/target. + +### 3.3 Table: `senses` + +One row per distinct meaning of a word. This is where polysemy is +handled: "bank" (financial institution) and "bank" (river edge) are +two senses of one word. + +| Column | Type | Constraints | Notes | +| ----------- | ----------- | -------------------------------- | -------------------------------------------- | +| id | UUID | PK, default random | | +| word_id | UUID | FK → words.id, ON DELETE CASCADE | | +| sense_index | SMALLINT | NOT NULL, default 0 | 0 = primary meaning, 1 = secondary, etc. | +| difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | "easy", "medium", "hard" | +| cefr_level | VARCHAR(2) | nullable, CHECK in allowed list | "A1","A2","B1","B2","C1","C2" | +| definitions | TEXT[] | NOT NULL, default '{}' | 1–3 definitions in the word's language | +| examples | TEXT[] | NOT NULL, default '{}' | 1–3 example sentences in the word's language | +| created_at | TIMESTAMPTZ | NOT NULL, default now() | | + +**Constraints:** + +- `UNIQUE (word_id, sense_index)` — one sense per index per word. +- `CHECK (difficulty IN ('easy','medium','hard'))` +- `CHECK (cefr_level IS NULL OR cefr_level IN ('A1','A2','B1','B2','C1','C2'))` + +**CEFR → difficulty mapping:** + +- A1, A2 → easy +- B1, B2 → medium +- C1, C2 → hard + +**Index:** + +- `idx_senses_word_diff ON (word_id, difficulty)` — accelerates the + join from words and the difficulty filter. + +**Design note — definitions and examples as arrays:** +Definitions and examples are stored as `TEXT[]` arrays on the sense +row rather than in separate tables. Rationale: + +- Each sense has at most 2–3 definitions/examples (1-to-few). +- They are always fetched together with the sense (no independent + querying needed). +- Separate tables would add 2 JOINs to the hottest query for no + practical benefit. +- The exercise generator picks one definition and one example + randomly in application code: + `arr[Math.floor(Math.random() * arr.length)]`. + +### 3.4 Table: `translations` + +One row per translation of a sense into another language. A single +sense can have multiple translations into the same language at +different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium). + +| Column | Type | Constraints | Notes | +| -------------------- | ----------- | --------------------------------- | ------------------------------------- | +| id | UUID | PK, default random | | +| sense_id | UUID | FK → senses.id, ON DELETE CASCADE | | +| target_language_code | VARCHAR(10) | NOT NULL, CHECK in supported list | Language of the translation | +| translation | TEXT | NOT NULL | "casa", "Haus", "maison" | +| gender | VARCHAR(20) | nullable | "masculine","feminine","neuter", NULL | +| difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | Can differ from sense difficulty | +| created_at | TIMESTAMPTZ | NOT NULL, default now() | | + +**Constraints:** + +- `UNIQUE (sense_id, target_language_code, translation)` — allows + multiple translations per language (synonyms) but no exact + duplicates. +- `CHECK (target_language_code IN ('en','de','it','fr','es'))` +- `CHECK (difficulty IN ('easy','medium','hard'))` +- `CHECK (gender IS NULL OR gender IN ('masculine','feminine','neuter'))` + +**Index:** + +- `idx_translations_sense_lang_diff ON (sense_id, target_language_code, difficulty)` + — accelerates the join from senses and the language/difficulty filter. + +**Design note — gender as a real column:** +Gender is stored as a column, not embedded in a JSON blob, because +future gender exercises will need to filter and group by gender. +English nouns have `gender = NULL`. + +### 3.5 Future Table: `inflection_forms` (not yet implemented) + +Will be added when verb conjugation and adjective declension exercises +are built. + +| Column | Type | Constraints | Notes | +| ------- | ----- | -------------------------------- | ------------------------------------- | +| id | UUID | PK, default random | | +| word_id | UUID | FK → words.id, ON DELETE CASCADE | | +| form | TEXT | NOT NULL | "Häuser", "ginge", "grüner" | +| tags | JSONB | NOT NULL, default '{}' | {"case":"genitive","number":"plural"} | + +This table is the primary reason the schema is normalized rather than +using JSONB documents. A German verb has 30–50 inflected forms. An +adjective has 12+. These are one-to-many relationships that require +their own table with a foreign key. + +--- + +## 4. Difficulty Model + +There are two difficulty columns. They answer different questions. + +### `senses.difficulty` — "Is this meaning appropriate for the level?" + +Controls which _meanings_ of a word are shown. A beginner should not +be tested on "Haus = noble dynasty" because the concept itself is +advanced, regardless of how hard the translation word is. + +### `translations.difficulty` — "Is this word an appropriate answer?" + +Controls which _translation word_ is the correct answer. The concept +of "bank" is easy, but "Geldinstitut" is a harder word than "Bank" +for that same concept. + +### Query filter logic: sense as ceiling, translation as target + +```sql +WHERE s.difficulty IN ('easy', 'medium') -- sense at or below level + AND t.difficulty = 'medium' -- translation exactly at level +``` + +- The sense difficulty acts as a **ceiling**: don't show meanings + harder than the selected level. +- The translation difficulty acts as the **target**: test the learner + on a word at exactly this level. + +This ensures that easy concepts with medium-level synonyms (e.g., +"bank" → "Geldinstitut") are reachable at the medium level, while +advanced concepts (e.g., "Haus" → "dynasty") remain gated. + +### Example data + +| Word | Sense | Sense Diff. | Translation | Trans. Diff. | +| ---- | --------------- | ----------- | ------------------- | ------------ | +| Haus | building | easy | casa (es) | easy | +| Haus | noble dynasty | hard | dinastía (es) | hard | +| bank | financial inst. | easy | Bank (de) | easy | +| bank | financial inst. | easy | Geldinstitut (de) | medium | +| bank | financial inst. | easy | Kreditinstitut (de) | hard | +| bank | river edge | medium | Ufer (de) | medium | + +At **medium** level, the query returns: Geldinstitut, Ufer. +At **hard** level: Kreditinstitut, dinastía. +"Bank" (easy translation) never appears at medium/hard as a correct +answer. "dinastía" (hard sense) never appears at easy/medium. + +--- + +## 5. Query Patterns + +### 5.1 Game query — get N random words + +Scenario: user picks German → Spanish, nouns, medium, 20 rounds. + +```sql +SELECT + w.id AS word_id, + w.headword, + s.id AS sense_id, + s.definitions, + s.examples, + t.translation, + t.gender +FROM words w +INNER JOIN senses s + ON s.word_id = w.id +INNER JOIN translations t + ON t.sense_id = s.id +WHERE w.language_code = 'de' + AND w.pos = 'noun' + AND s.difficulty IN ('easy', 'medium') + AND t.target_language_code = 'es' + AND t.difficulty = 'medium' +ORDER BY RANDOM() +LIMIT 20; +``` + +The application then picks one random definition and one random +example from the arrays for each word. + +### 5.2 Distractor query — get 3 wrong answers + +For a given correct answer, fetch 3 distractors from the same +language, pos, and difficulty pool. + +```sql +SELECT t.translation, t.gender +FROM translations t +INNER JOIN senses s + ON t.sense_id = s.id +INNER JOIN words w + ON s.word_id = w.id +WHERE w.language_code = 'de' + AND w.pos = 'noun' + AND s.difficulty IN ('easy', 'medium') + AND t.target_language_code = 'es' + AND t.difficulty = 'medium' + AND t.sense_id != :current_sense_id + AND t.translation != :correct_answer +ORDER BY RANDOM() +LIMIT 3; +``` + +### 5.3 Distractor exclusion rule + +**Distractors must come from a different sense than the correct +answer.** Not just a different word — a different sense. + +Rationale: multiple translations of the same sense are all valid +answers. "Bank" and "Geldinstitut" are both correct translations of +the financial-institution sense. Showing one as a distractor for the +other would confuse the learner and break trust. + +The `sense_id != :current_sense_id` filter excludes all synonyms of +the same sense in one condition. No synonym table needed. + +Translations from a _different_ sense of the same word are valid +distractors (e.g., "Ufer" from the river-bank sense is a fine +distractor for the financial-institution sense — the definition makes +it clearly wrong). + +### 5.4 Edge case: identical translation text across senses + +Two different senses of different words may share the same translation +text (e.g., "Schloss" = castle and "Schloss" = lock). The +`t.translation != :correct_answer` filter handles this by excluding +the exact text regardless of sense. + +--- + +## 6. Data Pipeline + +### 6.1 Flow + +``` +Word frequency lists (per language, per POS) + │ + ▼ +Gemini API (batches of 20 words) + │ + ▼ +Validation script (reject/flag bad entries) + │ + ▼ +SQLite staging database (local file) + │ + ▼ +Import script (SQLite → Postgres, batch inserts) + │ + ▼ +Postgres (dev) → test full game flow + │ + ▼ +Postgres (prod) via Drizzle migration +``` + +### 6.2 Wordlist source + +Frequency-based word lists, one per language. Example: "1000 most +common German nouns." Sources: Leipzig Corpora, Wiktionary frequency +lists, or similar open-source frequency data. + +Each language is processed independently. This ensures language-native +definitions and examples (a German word gets a German definition, not +a translated English one). + +### 6.3 Gemini output JSON contract + +The API is prompted to return an array of objects. Expected shape: + +```json +[ + { + "headword": "Haus", + "language": "de", + "pos": "noun", + "senses": [ + { + "sense_index": 0, + "cefr_level": "A1", + "difficulty": "easy", + "definitions": ["Ein Gebäude zum Wohnen."], + "examples": ["Sie kauften ein Haus in der Stadt."], + "translations": [ + { + "target_language": "en", + "word": "house", + "gender": null, + "difficulty": "easy" + }, + { + "target_language": "es", + "word": "casa", + "gender": "feminine", + "difficulty": "easy" + }, + { + "target_language": "fr", + "word": "maison", + "gender": "feminine", + "difficulty": "easy" + }, + { + "target_language": "it", + "word": "casa", + "gender": "feminine", + "difficulty": "easy" + } + ] + }, + { + "sense_index": 1, + "cefr_level": "C1", + "difficulty": "hard", + "definitions": ["Ein Adelsgeschlecht, eine Dynastie."], + "examples": ["Das Haus der Merowinger herrschte über Franken."], + "translations": [ + { + "target_language": "en", + "word": "house", + "gender": null, + "difficulty": "hard" + }, + { + "target_language": "es", + "word": "dinastía", + "gender": "feminine", + "difficulty": "hard" + } + ] + } + ] + } +] +``` + +### 6.4 Validation rules + +Before writing to SQLite, every entry is checked: + +- `headword`, `language`, `pos` are present and valid. +- At least one sense per word. +- Each sense has at least one definition and one example. +- `difficulty` is one of: `easy`, `medium`, `hard`. +- `cefr_level` is one of: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`. +- CEFR → difficulty mapping is consistent. +- `gender` is valid for the target language: + - German: masculine, feminine, neuter + - French, Spanish, Italian: masculine, feminine + - English: null +- Translations exist for at least the 4 other supported languages. +- No duplicate entries (headword + language + pos + sense_index). + +Invalid entries are logged and excluded. They can be reviewed and +re-processed manually. + +### 6.5 Import: SQLite → Postgres + +A Node.js script reads from SQLite (via `better-sqlite3`) and +batch-inserts into Postgres (via Drizzle). Inserts are wrapped in +transactions per batch (20 words) for atomicity. + +The application (dev and prod) always reads from Postgres. SQLite is +used only as a pipeline staging file. + +--- + +## 7. Indexes + +Three indexes cover the game and distractor queries: + +```sql +CREATE INDEX idx_words_lang_pos + ON words (language_code, pos); + +CREATE INDEX idx_senses_word_diff + ON senses (word_id, difficulty); + +CREATE INDEX idx_translations_sense_lang_diff + ON translations (sense_id, target_language_code, difficulty); +``` + +--- + +## 8. Performance + +### Estimated data volume at target scale + +| Table | Rows | Derivation | +| ------------ | ----- | -------------------------------- | +| words | 500k | ~100k per language × 5 languages | +| senses | ~750k | ~1.5 senses per word average | +| translations | ~3M | ~4 translations per sense | + +### Query performance + +| Scale | Game query (LIMIT 20) | Distractor query (LIMIT 3) | +| ---------- | --------------------- | -------------------------- | +| 10k words | < 10 ms | < 10 ms | +| 500k words | 20–80 ms | 15–60 ms | +| 5M words | 100–300 ms | 80–200 ms | + +The bottleneck at scale is `ORDER BY RANDOM()`, which sorts the +entire filtered result set before applying LIMIT. At 500k words, the +filtered set per query is ~5k–15k rows — well within comfortable +range. + +**Future optimization** (if filtered sets exceed ~100k rows): + +```sql +WHERE ... AND random() < 0.05 -- pre-filter to ~5% of rows +ORDER BY RANDOM() +LIMIT 20; +``` + +Or use `TABLESAMPLE`. Not needed at current scale. + +### Import performance + +| Method | Time for 3.5M rows | +| ---------------------------- | ------------------ | +| Individual INSERT | ~30–60 min | +| Batch INSERT (1000 per stmt) | ~2–5 min | +| Postgres COPY (CSV) | ~10–30 sec | + +The pipeline uses batch inserts via Drizzle. The full import is a +one-time or occasional operation. + +--- + +## 9. Implementation Plan + +``` + 1. ✅ Design doc (this document) + 2. Acquire word frequency lists (5 languages, nouns first) + 3. Build + test Gemini prompt (5 words → 20 words → full batch) + 4. Validation script (Gemini output → clean JSON) + 5. SQLite staging schema + pipeline write + 6. Import script (SQLite → local Postgres) + 7. Drizzle schema: words, senses, translations + indexes + 8. Drizzle migration on local Postgres + 9. Dev branch: new queries (game + distractor), full game flow test +10. Drizzle migration on prod Postgres + data import + verify +11. Extend pipeline: verbs, adjectives, adverbs (schema unchanged) +12. Later: inflection_forms table + conjugation/declension exercises +``` + +--- + +## 10. Future Extensions + +### Verb conjugation / adjective declension exercises + +The `inflection_forms` table (section 3.5) will store inflected forms +with grammatical tags as JSONB. The schema is normalized specifically +to support this: one word → many forms, each independently queryable. + +### Gender exercises + +The `gender` column on `translations` enables filtering and grouping +by grammatical gender for dedicated gender practice rounds. + +### Additional POS + +The `pos` column already supports noun, verb, adjective, adverb. +Adding a new POS requires no schema change — only a new wordlist and +an adjusted Gemini prompt. + +--- + +## 11. Key Design Decisions — Summary + +| Decision | Choice | Rationale | +| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------ | +| Database | Postgres + SQLite staging | Relational data, FK integrity, SQL query pattern, already in stack | +| Schema structure | 3 normalized tables | Matches query pattern, supports any language pair, extensible for inflections | +| Definitions / examples | `TEXT[]` arrays on `senses` | 1-to-few relationship, always fetched with sense, avoids 2 extra JOINs | +| Gender | Column on `translations` | Needed for filtering in future gender exercises | +| Difficulty | Two columns: `senses.difficulty` + `translations.difficulty` | Sense = concept gate, translation = word-level target | +| Difficulty filter logic | Sense as ceiling, translation as exact match | Ensures easy concepts with hard synonyms are reachable; advanced concepts stay gated | +| Distractor exclusion | `sense_id != current` | Prevents valid synonyms from appearing as wrong answers | +| Language direction | Any of 5 languages as source or target | Each language has its own headword entries; translations link them | +| Pipeline | Gemini → validate → SQLite → Postgres | Batch-generated data, LLM output needs validation, SQLite for staging simplicity | +| Table-per-language/pos | Rejected | Anti-pattern: 40+ tables, exponential maintenance | +| Single JSONB blob | Rejected | Cannot support inflection tables, cannot index gender, no FK integrity | diff --git a/documentation/pipeline/roadmap.md b/documentation/pipeline/roadmap.md new file mode 100644 index 0000000..c6baef7 --- /dev/null +++ b/documentation/pipeline/roadmap.md @@ -0,0 +1,704 @@ +# Vocabulary Trainer — Roadmap: Data Pipeline & Schema Migration + +> **Objective:** Replace the OpenWordNet/kaikki data source with a +> Gemini-powered pipeline that generates high-quality vocabulary data, +> backed by a normalized Postgres schema. +> +> **Author:** [Your Name] +> **Date:** July 2026 +> **Companion doc:** `docs/schema-design.md` + +--- + +## How to Read This Document + +This roadmap has three zoom levels: + +- **Part 1 — Overview:** The phases at a glance. Read this to understand the full scope in 30 seconds. +- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and acceptance criteria per phase. Read this to plan your week. +- **Part 3 — Detailed Tasks:** Step-by-step instructions within each phase. Read this when you sit down to code. + +--- + +# Part 1 — High-Level Overview + +``` +Phase 0 Preparation + Get wordlists, set up tooling, finalize the Gemini prompt. + +Phase 1 Data Pipeline + Build the Gemini → validate → SQLite pipeline. + Produce a clean dataset of ~1000 nouns × 5 languages. + +Phase 2 Schema & Migration + Create the new Drizzle schema (words, senses, translations). + Write the SQLite → Postgres import script. + Migrate the local dev database. + +Phase 3 App Integration + Rewrite the game and distractor queries against the new schema. + Update the exercise-generation logic. + Test the full game flow in dev. + +Phase 4 Production Deploy + Run the Drizzle migration on prod. + Import the dataset. + Verify the live app works end-to-end. + +Phase 5 Extend POS + Run the pipeline for verbs, adjectives, adverbs. + No schema changes needed — new wordlists + adjusted prompts. + +Phase 6 Future Features (out of scope for now) + Inflection tables, conjugation/declension exercises, + gender exercises, spaced-repetition scheduling. +``` + +**Dependency chain:** + +``` +Phase 0 → Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5 + │ + ▼ + Phase 6 +``` + +Each phase depends on the previous one being complete. Do not skip +ahead. + +--- + +# Part 2 — Phase Breakdown + +--- + +## Phase 0: Preparation + +**Goal:** Have everything you need before writing pipeline code. + +**Tasks:** + +- [ ] Acquire frequency-based noun lists for all 5 languages +- [ ] Clean and format the lists (one word per line, UTF-8) +- [ ] Set up local Postgres (Docker or native) +- [ ] Set up a SQLite database file for staging +- [ ] Write and test the Gemini prompt with 5 sample words +- [ ] Refine the prompt until the JSON output matches the contract + defined in `docs/schema-design.md` §6.3 + +**Dependencies:** None. + +**Acceptance criteria:** + +- You have 5 wordlist files in `data/wordlists/` (one per language). +- A Gemini call with 5 German nouns returns valid JSON matching the + contract, including definitions, examples, translations with gender, + and difficulty levels. +- Local Postgres is running and reachable from your app. + +--- + +## Phase 1: Data Pipeline + +**Goal:** A repeatable script that takes a wordlist, calls Gemini in +batches of 20, validates the output, and writes clean rows to SQLite. + +**Tasks:** + +- [ ] Write the validation module (see schema-design §6.4) +- [ ] Write the pipeline script: - Read wordlist file - Split into batches of 20 - Call Gemini API per batch - Parse JSON response - Validate each entry - Write valid entries to SQLite - Log invalid entries to a rejection file +- [ ] Create the SQLite schema (mirrors the Postgres schema) +- [ ] Run the pipeline for all 5 languages (nouns only) +- [ ] Review the rejection log, fix prompt issues, re-run failed batches +- [ ] Spot-check 50 random entries for correctness + +**Dependencies:** Phase 0 complete. + +**Acceptance criteria:** + +- SQLite database contains ~1000 nouns × 5 languages with senses and + translations. +- Rejection rate is below 10%. +- Spot-checked entries have correct definitions, plausible examples, + correct genders, and reasonable difficulty levels. +- The pipeline is re-runnable (idempotent or with duplicate handling). + +--- + +## Phase 2: Schema & Migration + +**Goal:** The new normalized schema exists in Drizzle, the local +Postgres database is migrated, and the SQLite data is imported. + +**Tasks:** + +- [ ] Write the new Drizzle schema file: - `words` table (headword, language_code, pos) - `senses` table (word_id FK, sense_index, difficulty, + cefr_level, definitions TEXT[], examples TEXT[]) - `translations` table (sense_id FK, target_language_code, + translation, gender, difficulty) - All CHECK constraints, UNIQUE constraints, indexes +- [ ] Generate the Drizzle migration (`npx drizzle-kit generate`) +- [ ] Inspect the generated SQL file — verify it creates the right + tables, constraints, and indexes +- [ ] Apply the migration to local Postgres (`npx drizzle-kit migrate`) +- [ ] Write the import script (SQLite → Postgres): - Read all rows from SQLite - Insert into Postgres in dependency order: + words → senses → translations - Use batch inserts (not row-by-row) - Wrap in transactions (per batch of 20 words) - Handle duplicates gracefully (skip or upsert) +- [ ] Run the import script +- [ ] Verify row counts match between SQLite and Postgres +- [ ] Run 3–5 manual SQL queries against Postgres to sanity-check + the data + +**Dependencies:** Phase 1 complete (SQLite has data). + +**Acceptance criteria:** + +- `npx drizzle-kit migrate` runs without errors on local Postgres. +- Import script completes without errors. +- Row counts in Postgres match SQLite (±rejection count). +- Manual query: "Give me 5 random German nouns with Spanish + translations at easy difficulty" returns sensible results. + +--- + +## Phase 3: App Integration + +**Goal:** The running app uses the new schema. Game rounds and +distractors work correctly for all language pairs. + +**Tasks:** + +- [ ] Rewrite `getGameTerms` query: - JOIN words → senses → translations - Filter: source language, pos, sense difficulty (ceiling), + target language, translation difficulty (exact) - ORDER BY RANDOM(), LIMIT rounds +- [ ] Rewrite `getDistractors` query: - Same JOINs and filters - Exclude: `sense_id != current`, `translation != correct` - ORDER BY RANDOM(), LIMIT 3 +- [ ] Update the exercise-generation logic: - Pick one random definition from the `definitions` array - Pick one random example from the `examples` array - Assemble the 4 answer options (1 correct + 3 distractors) - Shuffle the options +- [ ] Remove or deprecate old schema references + (old `vocabulary_entries`, `entry_translations` tables) +- [ ] Test manually: - German → Spanish, nouns, easy, 10 rounds - Spanish → German, nouns, medium, 10 rounds - English → French, nouns, hard, 10 rounds - Italian → German, nouns, easy, 10 rounds - Verify: no duplicate answers, no same-sense synonyms as + distractors, definitions and examples are in the source + language, genders display correctly +- [ ] Test edge cases: - A difficulty/pos/language combo with very few words + (does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?) + +**Dependencies:** Phase 2 complete (Postgres has data, schema exists). + +**Acceptance criteria:** + +- Full game flow works in dev for at least 4 different language pairs. +- No same-sense synonyms appear as distractors. +- Definitions and examples are in the correct language. +- Gender is displayed where applicable. +- No console errors or unhandled query failures. + +--- + +## Phase 4: Production Deploy + +**Goal:** The live app runs on the new schema with the new data. + +**Tasks:** + +- [ ] Back up the production database +- [ ] Run the Drizzle migration on prod +- [ ] Run the import script against prod Postgres +- [ ] Verify row counts on prod +- [ ] Test the live app: - Play 2 full games on the deployed app - Check different language pairs and difficulties +- [ ] Monitor for errors (server logs, browser console) for 24h +- [ ] Remove old tables from prod (after confirming everything works) + +**Dependencies:** Phase 3 complete (dev is fully working). + +**Acceptance criteria:** + +- Live app serves game rounds from the new schema. +- No errors in server logs for 24 hours. +- Old tables are dropped (or scheduled for removal). +- Rollback plan exists: the prod backup can be restored if needed. + +--- + +## Phase 5: Extend POS + +**Goal:** Verbs, adjectives, and adverbs are in the database and +usable in the app. + +**Tasks:** + +- [ ] Acquire frequency lists for verbs, adjectives, adverbs + (all 5 languages) +- [ ] Adjust the Gemini prompt per POS: - Verbs: may need different metadata (transitivity, etc.) - Adjectives: may need base form info - Adverbs: typically simpler metadata +- [ ] Run the pipeline for each POS +- [ ] Validate, import to Postgres (dev → prod) +- [ ] Test game flow with verbs, adjectives, adverbs +- [ ] Verify the POS filter in the app UI works for all types + +**Dependencies:** Phase 4 complete. + +**Acceptance criteria:** + +- All 4 POS types are playable in the app. +- Pipeline is repeatable for future data additions. + +--- + +## Phase 6: Future Features (out of scope) + +Listed here for visibility. Not planned, not estimated. + +- [ ] `inflection_forms` table + conjugation exercises (verbs) +- [ ] Adjective declension exercises (der grüne Mann, grüner Mann…) +- [ ] Gender exercises (pick the correct article) +- [ ] Spaced-repetition scheduling (track which words the user knows) +- [ ] User accounts and progress persistence +- [ ] Additional languages (if ever) + +--- + +# Part 3 — Detailed Task Breakdown + +--- + +## Phase 0: Preparation — Detailed + +### 0.1 Acquire wordlists + +- Search for frequency lists. Good starting points: + - "Leipzig Corpora Collection" (academic, per-language) + - Wiktionary frequency lists + - GitHub repos: search "german noun frequency list", + "spanish noun frequency list", etc. + - Tatoeba sentence counts as a proxy for word frequency +- Target: ~1000 nouns per language, sorted by frequency. +- Format: plain text, one word per line, UTF-8, no headers. +- Save to `data/wordlists/nouns_de.txt`, `nouns_es.txt`, etc. +- Clean the lists: remove duplicates, remove words with spaces + (multi-word expressions), remove proper nouns if desired. + +### 0.2 Set up local Postgres + +- Option A (Docker): + ``` + docker run --name vocab-dev \ + -e POSTGRES_USER=dev \ + -e POSTGRES_PASSWORD=dev \ + -e POSTGRES_DB=vocab \ + -p 5432:5432 \ + -d postgres:16 + ``` +- Option B (native install): install Postgres, create a `vocab` + database. +- Update your `.env` / `.env.local` with the connection string. +- Verify: connect with `psql` or a GUI client (TablePlus, DBeaver, + pgAdmin). Run `SELECT 1;`. + +### 0.3 Write and test the Gemini prompt + +- Start with 5 German nouns. +- The prompt should specify: + - The exact JSON structure expected (copy from schema-design §6.3) + - That definitions and examples must be in the word's language + - That translations are needed for all 4 other supported languages + - That gender must be provided for de/fr/es/it, null for en + - That difficulty must be one of: easy, medium, hard + - That CEFR level must be one of: A1, A2, B1, B2, C1, C2 + - That multiple senses should be included for polysemous words +- Call the API. Inspect the JSON. Common issues to fix: + - Gemini wraps the JSON in markdown code fences → strip them + - Gemini returns "intermediate" instead of "medium" → normalize + - Gemini omits a language → re-prompt or reject + - Gemini returns gender "common" for German → reject +- Iterate until 3 consecutive batches of 20 return clean JSON. + +--- + +## Phase 1: Data Pipeline — Detailed + +### 1.1 Create the SQLite schema + +- Create a file `pipeline/schema.sql` or define it in your pipeline + script. +- Tables mirror the Postgres schema: + + ```sql + CREATE TABLE words ( + id TEXT PRIMARY KEY, + headword TEXT NOT NULL, + language_code TEXT NOT NULL, + pos TEXT NOT NULL, + created_at TEXT DEFAULT (datetime('now')), + UNIQUE(headword, language_code, pos) + ); + + CREATE TABLE senses ( + id TEXT PRIMARY KEY, + word_id TEXT NOT NULL REFERENCES words(id), + sense_index INTEGER NOT NULL DEFAULT 0, + difficulty TEXT NOT NULL, + cefr_level TEXT, + definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text + examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text + created_at TEXT DEFAULT (datetime('now')), + UNIQUE(word_id, sense_index) + ); + + CREATE TABLE translations ( + id TEXT PRIMARY KEY, + sense_id TEXT NOT NULL REFERENCES senses(id), + target_language_code TEXT NOT NULL, + translation TEXT NOT NULL, + gender TEXT, + difficulty TEXT NOT NULL, + created_at TEXT DEFAULT (datetime('now')), + UNIQUE(sense_id, target_language_code, translation) + ); + ``` + +- Note: SQLite has no native TEXT[]. Store arrays as JSON text. + The import script will parse them into Postgres TEXT[] on import. + +### 1.2 Write the validation module + +- Create `pipeline/validate.ts` (or .js). +- Input: one parsed Gemini entry (the JSON object for one word). +- Checks (return a list of errors, empty = valid): + - `headword` is a non-empty string + - `language` is in ['en','de','it','fr','es'] + - `pos` is in ['noun','verb','adjective','adverb'] + - `senses` is a non-empty array + - Each sense has: + - `sense_index` is a non-negative integer + - `difficulty` is in ['easy','medium','hard'] + - `cefr_level` is in ['A1','A2','B1','B2','C1','C2'] or null + - CEFR ↔ difficulty mapping is consistent + - `definitions` is a non-empty array of non-empty strings + - `examples` is a non-empty array of non-empty strings + - `translations` is a non-empty array + - Each translation has: + - `target_language` is in the supported list + - `target_language` != the word's own language + - `word` is a non-empty string + - `gender` is valid for the target language + (de: masculine/feminine/neuter/null, + fr/es/it: masculine/feminine/null, + en: null) + - `difficulty` is in ['easy','medium','hard'] +- Output: `{ valid: boolean, errors: string[] }` + +### 1.3 Write the pipeline script + +- Create `pipeline/run.ts`. +- Pseudocode: + ``` + for each language in [de, en, es, fr, it]: + read wordlist file → array of words + split into batches of 20 + for each batch: + call Gemini API with the batch + parse JSON response (strip markdown fences if present) + for each entry in response: + validate(entry) + if valid: + generate UUIDs for word, senses, translations + INSERT into SQLite (words, senses, translations) + else: + append to rejection log (data/rejections/{lang}_{pos}.jsonl) + log progress: "Batch 12/50 done. 238 words imported, 2 rejected." + sleep 1s (rate limiting) + ``` +- Use `better-sqlite3` for SQLite access (synchronous, simple). +- Generate UUIDs with `crypto.randomUUID()`. +- Store definitions/examples as JSON strings in SQLite + (`JSON.stringify(arr)`). + +### 1.4 Run and review + +- Run the pipeline for all 5 languages. +- Check the rejection log. If rejection rate > 10%, fix the prompt + and re-run failed batches. +- Spot-check: open the SQLite DB, run: + ```sql + SELECT w.headword, s.definitions, t.translation, t.gender + FROM words w + JOIN senses s ON s.word_id = w.id + JOIN translations t ON t.sense_id = s.id + WHERE w.language_code = 'de' AND w.pos = 'noun' + ORDER BY RANDOM() + LIMIT 20; + ``` +- Read the definitions. Are they in German? Do they make sense? + Are the genders correct? Are the difficulties reasonable? + +--- + +## Phase 2: Schema & Migration — Detailed + +### 2.1 Write the Drizzle schema + +- Create or update `src/db/schema.ts`. +- Define the three tables using Drizzle's `pgTable` builder. +- Include all columns, types, constraints, and indexes as specified + in `docs/schema-design.md` §3. +- Keep the old tables in the file for now (commented out or in a + separate file) so the app doesn't break during development. + +### 2.2 Generate and inspect the migration + +- Run: `npx drizzle-kit generate` +- Open the generated SQL file in `drizzle/` (or wherever your + config puts it). +- Read it. Verify: + - Three CREATE TABLE statements + - CHECK constraints match your schema + - UNIQUE constraints are present + - Three CREATE INDEX statements + - Foreign keys reference the correct tables with ON DELETE CASCADE +- If something looks wrong, fix the schema file and regenerate. + +### 2.3 Apply the migration locally + +- Run: `npx drizzle-kit migrate` +- Connect to local Postgres and verify: + ```sql + \dt -- list tables + \d words -- describe words table + \d senses -- describe senses table + \d translations -- describe translations table + ``` + +### 2.4 Write the import script + +- Create `scripts/import-to-postgres.ts`. +- Pseudocode: + + ``` + open SQLite database (read-only) + connect to Postgres via Drizzle + + read all words from SQLite + for each batch of 100 words: + begin transaction + INSERT words into Postgres + for each word: + read its senses from SQLite + INSERT senses into Postgres + for each sense: + read its translations from SQLite + parse definitions/examples from JSON string → TEXT[] + INSERT translations into Postgres + commit transaction + log progress + + log final counts: words, senses, translations + ``` + +- Parse `definitions` and `examples` from JSON strings (SQLite) + into actual arrays (Postgres TEXT[]). +- Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully. + +### 2.5 Run and verify + +- Run the import script. +- Compare counts: + + ```sql + -- In SQLite + SELECT COUNT(*) FROM words; + SELECT COUNT(*) FROM senses; + SELECT COUNT(*) FROM translations; + + -- In Postgres + SELECT COUNT(*) FROM words; + SELECT COUNT(*) FROM senses; + SELECT COUNT(*) FROM translations; + ``` + +- Counts should match (minus any rows that failed validation). +- Run the game query manually in Postgres: + ```sql + SELECT w.headword, s.definitions, s.examples, + t.translation, t.gender + FROM words w + JOIN senses s ON s.word_id = w.id + JOIN translations t ON t.sense_id = s.id + WHERE w.language_code = 'de' + AND w.pos = 'noun' + AND s.difficulty IN ('easy', 'medium') + AND t.target_language_code = 'es' + AND t.difficulty = 'medium' + ORDER BY RANDOM() + LIMIT 5; + ``` +- Verify the results make sense. + +--- + +## Phase 3: App Integration — Detailed + +### 3.1 Rewrite getGameTerms + +- Open the file containing `getGameTerms`. +- Replace the old query (2-table join on `vocabulary_entries` + + `entry_translations`) with the new 3-table join + (words → senses → translations). +- Parameters: sourceLanguage, targetLanguage, pos, difficulty, rounds. +- Difficulty filter: + - `senses.difficulty IN (all levels up to and including selected)` + — ceiling logic + - `translations.difficulty = selected` — exact match +- Return: word_id, headword, sense_id, definitions, examples, + translation, gender. + +### 3.2 Rewrite getDistractors + +- Same 3-table join. +- Additional filters: + - `t.sense_id != :currentSenseId` + - `t.translation != :correctAnswer` +- LIMIT 3. +- If fewer than 3 distractors are found (small data pool), handle + gracefully: reduce the number of options or log a warning. + +### 3.3 Update exercise generation + +- In the function that assembles a game round: + - Pick one random definition: + `definitions[Math.floor(Math.random() * definitions.length)]` + - Pick one random example: + `examples[Math.floor(Math.random() * examples.length)]` + - Combine 1 correct translation + 3 distractors. + - Shuffle the 4 options (Fisher-Yates or similar). + - Attach gender to each option for display. + +### 3.4 Test matrix + +Run through this matrix manually in the dev app: + +| Source | Target | POS | Difficulty | Rounds | Pass? | +| ------ | ------ | ---- | ---------- | ------ | ----- | +| de | es | noun | easy | 10 | | +| es | de | noun | medium | 10 | | +| en | fr | noun | hard | 10 | | +| it | de | noun | easy | 10 | | +| fr | en | noun | medium | 10 | | + +For each: verify definitions are in the source language, translations +are in the target language, genders are shown, no same-sense synonyms +appear as distractors, no duplicate options. + +--- + +## Phase 4: Production Deploy — Detailed + +### 4.1 Pre-deploy checklist + +- [ ] All Phase 3 acceptance criteria pass +- [ ] `git status` is clean, all changes committed +- [ ] The Drizzle migration file is committed to the repo +- [ ] You know your prod database connection string +- [ ] You have a backup method for prod (pg_dump, hosting provider + snapshot, etc.) + +### 4.2 Deploy + +- Back up prod: + ``` + pg_dump -h -U -d > backup_$(date +%Y%m%d).sql + ``` +- Run migration on prod: + ``` + DATABASE_URL= npx drizzle-kit migrate + ``` +- Run import script against prod: + ``` + DATABASE_URL= npx tsx scripts/import-to-postgres.ts + ``` +- Verify counts on prod. + +### 4.3 Post-deploy verification + +- Open the live app. Play 2 full games with different settings. +- Check server logs for errors. +- Wait 24h. Check logs again. +- If everything is clean, drop old tables: + ```sql + DROP TABLE IF EXISTS entry_translations; + DROP TABLE IF EXISTS vocabulary_entries; + ``` + (Or keep them for another week if you want a safety net.) + +### 4.4 Rollback plan + +If something goes wrong: + +- Restore the backup: + ``` + psql -h -U -d < backup_20260722.sql + ``` +- Revert the code to the previous commit. +- Redeploy. + +--- + +## Phase 5: Extend POS — Detailed + +### 5.1 Per POS + +Repeat the Phase 1 pipeline for each new POS: + +- Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages. +- Adjust the Gemini prompt: + - Verbs: ask for transitivity, common prepositions, or other + verb-specific metadata if needed for future exercises. + - Adjectives: ask for the base form. Note that adjective metadata + differs from noun metadata (no gender on the adjective itself + in the same way — gender applies to the noun it modifies). + - Adverbs: typically simpler. May not need gender at all. +- Run pipeline → validate → SQLite → import → Postgres. +- Test in the app with the POS filter set to the new type. + +### 5.2 No schema changes + +The `pos` column already supports all four types. The `gender` +column is nullable and simply won't be populated for adverbs or +English words. No migration needed. + +--- + +# Risks & Mitigations + +| Risk | Likelihood | Impact | Mitigation | +| ------------------------------------------------------ | ----------------- | ------------------------- | ------------------------------------------------------------------------------------------------- | +| Gemini returns malformed JSON | Medium | Pipeline stalls | Strip markdown fences, wrap parsing in try/catch, log and skip bad batches | +| Gemini produces wrong genders or difficulties | Medium | Bad exercise data | Validation module rejects invalid entries. Spot-check 50+ entries per language | +| Too few words at a given difficulty/pos/language combo | Medium | Game can't fill 4 options | Graceful fallback: reduce options or show a "not enough words" message. Log which combos are thin | +| SQLite → Postgres import fails mid-way | Low | Partial data in Postgres | Transactions per batch. Re-run with ON CONFLICT DO NOTHING | +| ORDER BY RANDOM() gets slow at scale | Low (not at 500k) | Slow game load | Add a TODO comment. Optimize with random() pre-filter when needed | +| Prod migration breaks the live app | Low | Downtime | Backup before migrating. Rollback plan documented. Deploy during low-traffic hours | + +--- + +# File / Folder Structure (new files) + +``` +project/ +├── docs/ +│ ├── schema-design.md ← companion design doc +│ └── roadmap.md ← this document +├── data/ +│ ├── wordlists/ +│ │ ├── nouns_de.txt +│ │ ├── nouns_en.txt +│ │ ├── nouns_es.txt +│ │ ├── nouns_fr.txt +│ │ └── nouns_it.txt +│ ├── rejections/ ← invalid Gemini entries (for review) +│ └── staging.db ← SQLite staging database +├── pipeline/ +│ ├── run.ts ← main pipeline script +│ ├── validate.ts ← validation module +│ ├── prompt.ts ← Gemini prompt template +│ └── schema.sql ← SQLite schema definition +├── scripts/ +│ └── import-to-postgres.ts ← SQLite → Postgres import +├── src/ +│ └── db/ +│ └── schema.ts ← Drizzle schema (updated) +└── drizzle/ ← generated migration files +```