forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

updating documentation

Browse source
This commit is contained in:
lila 2026-07-23 16:57:07 +02:00
parent de3ebf66a9
commit e0b1f7f92f
2 changed files with 139 additions and 143 deletions
Download patch file Download diff file Expand all files Collapse all files

47
documentation/pipeline/design-doc.md
View file

@ -3,7 +3,7 @@
> **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline > **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline
> **Author:** [Your Name] > **Author:** [Your Name]
> **Date:** July 2026 > **Date:** July 2026
> **Status:** Approved — ready for implementation > **Status:** Approved — schema implemented
--- ---
@ -83,9 +83,8 @@ staging).
│ headword │ └───>│ word_id(FK) │ └───>│ sense_id (FK) │ │ headword │ └───>│ word_id(FK) │ └───>│ sense_id (FK) │
│ language_code│ │ sense_index │ │ target_lang_code │ │ language_code│ │ sense_index │ │ target_lang_code │
│ pos │ │ difficulty │ │ translation │ │ pos │ │ difficulty │ │ translation │
│ │ │ cefr_level │ │ gender │ │ │ │ definitions │ │ gender │
│ │ │ definitions │ │ difficulty │ │ │ │ examples │ │ difficulty │
│ │ │ examples │ │ │
└─────────────┘ └─────────────┘ └──────────────────┘ └─────────────┘ └─────────────┘ └──────────────────┘
Future (not yet implemented): Future (not yet implemented):
@ -119,7 +118,7 @@ One row per unique word in a specific language.
**Index:** **Index:**
- `idx_words_lang_pos ON (language_code, pos)` — accelerates the - `idx_language_code_pos ON (language_code, pos)` — accelerates the
primary game query filter. primary game query filter.
**Design note:** Each language gets its own headword entries. "Haus" **Design note:** Each language gets its own headword entries. "Haus"
@ -139,7 +138,6 @@ two senses of one word.
| word_id | UUID | FK → words.id, ON DELETE CASCADE | | | word_id | UUID | FK → words.id, ON DELETE CASCADE | |
| sense_index | SMALLINT | NOT NULL, default 0 | 0 = primary meaning, 1 = secondary, etc. | | sense_index | SMALLINT | NOT NULL, default 0 | 0 = primary meaning, 1 = secondary, etc. |
| difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | "easy", "medium", "hard" | | difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | "easy", "medium", "hard" |
| cefr_level | VARCHAR(2) | nullable, CHECK in allowed list | "A1","A2","B1","B2","C1","C2" |
| definitions | TEXT[] | NOT NULL, default '{}' | 1–3 definitions in the word's language | | definitions | TEXT[] | NOT NULL, default '{}' | 1–3 definitions in the word's language |
| examples | TEXT[] | NOT NULL, default '{}' | 1–3 example sentences in the word's language | | examples | TEXT[] | NOT NULL, default '{}' | 1–3 example sentences in the word's language |
| created_at | TIMESTAMPTZ | NOT NULL, default now() | | | created_at | TIMESTAMPTZ | NOT NULL, default now() | |
@ -148,18 +146,11 @@ two senses of one word.
- `UNIQUE (word_id, sense_index)` — one sense per index per word. - `UNIQUE (word_id, sense_index)` — one sense per index per word.
- `CHECK (difficulty IN ('easy','medium','hard'))` - `CHECK (difficulty IN ('easy','medium','hard'))`
- `CHECK (cefr_level IS NULL OR cefr_level IN ('A1','A2','B1','B2','C1','C2'))`
**CEFR → difficulty mapping:**
- A1, A2 → easy
- B1, B2 → medium
- C1, C2 → hard
**Index:** **Index:**
- `idx_senses_word_diff ON (word_id, difficulty)` — accelerates the - `idx_word_sense_difficulty ON (word_id, difficulty)` — accelerates
join from words and the difficulty filter. the join from words and the difficulty filter.
**Design note — definitions and examples as arrays:** **Design note — definitions and examples as arrays:**
Definitions and examples are stored as `TEXT[]` arrays on the sense Definitions and examples are stored as `TEXT[]` arrays on the sense
@ -201,7 +192,7 @@ different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium).
**Index:** **Index:**
- `idx_translations_sense_lang_diff ON (sense_id, target_language_code, difficulty)` - `idx_translations_sense_language_difficulty ON (sense_id, target_language_code, difficulty)`
— accelerates the join from senses and the language/difficulty filter. — accelerates the join from senses and the language/difficulty filter.
**Design note — gender as a real column:** **Design note — gender as a real column:**
@ -409,7 +400,6 @@ The API is prompted to return an array of objects. Expected shape:
"senses": [ "senses": [
{ {
"sense_index": 0, "sense_index": 0,
"cefr_level": "A1",
"difficulty": "easy", "difficulty": "easy",
"definitions": ["Ein Gebäude zum Wohnen."], "definitions": ["Ein Gebäude zum Wohnen."],
"examples": ["Sie kauften ein Haus in der Stadt."], "examples": ["Sie kauften ein Haus in der Stadt."],
@ -442,7 +432,6 @@ The API is prompted to return an array of objects. Expected shape:
}, },
{ {
"sense_index": 1, "sense_index": 1,
"cefr_level": "C1",
"difficulty": "hard", "difficulty": "hard",
"definitions": ["Ein Adelsgeschlecht, eine Dynastie."], "definitions": ["Ein Adelsgeschlecht, eine Dynastie."],
"examples": ["Das Haus der Merowinger herrschte über Franken."], "examples": ["Das Haus der Merowinger herrschte über Franken."],
@ -474,8 +463,6 @@ Before writing to SQLite, every entry is checked:
- At least one sense per word. - At least one sense per word.
- Each sense has at least one definition and one example. - Each sense has at least one definition and one example.
- `difficulty` is one of: `easy`, `medium`, `hard`. - `difficulty` is one of: `easy`, `medium`, `hard`.
- `cefr_level` is one of: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`.
- CEFR → difficulty mapping is consistent.
- `gender` is valid for the target language: - `gender` is valid for the target language:
- German: masculine, feminine, neuter - German: masculine, feminine, neuter
- French, Spanish, Italian: masculine, feminine - French, Spanish, Italian: masculine, feminine
@ -502,13 +489,13 @@ used only as a pipeline staging file.
Three indexes cover the game and distractor queries: Three indexes cover the game and distractor queries:
```sql ```sql
CREATE INDEX idx_words_lang_pos CREATE INDEX idx_language_code_pos
ON words (language_code, pos); ON words (language_code, pos);
CREATE INDEX idx_senses_word_diff CREATE INDEX idx_word_sense_difficulty
ON senses (word_id, difficulty); ON senses (word_id, difficulty);
CREATE INDEX idx_translations_sense_lang_diff CREATE INDEX idx_translations_sense_language_difficulty
ON translations (sense_id, target_language_code, difficulty); ON translations (sense_id, target_language_code, difficulty);
``` ```
@ -563,13 +550,13 @@ one-time or occasional operation.
## 9. Implementation Plan ## 9. Implementation Plan
``` ```
1. ✅ Design doc (this document) 1. ✅ Schema design (this document)
2. Acquire word frequency lists (5 languages, nouns first) 2. ✅ Drizzle schema: words, senses, translations, relations, indexes
3. Build + test Gemini prompt (5 words → 20 words → full batch) 3. Acquire word frequency lists (5 languages, nouns first)
4. Validation script (Gemini output → clean JSON) 4. Build + test Gemini prompt (5 words → 20 words → full batch)
5. SQLite staging schema + pipeline write 5. Validation script (Gemini output → clean JSON)
6. Import script (SQLite → local Postgres) 6. SQLite staging schema + pipeline write
7. Drizzle schema: words, senses, translations + indexes 7. Import script (SQLite → local Postgres)
8. Drizzle migration on local Postgres 8. Drizzle migration on local Postgres
9. Dev branch: new queries (game + distractor), full game flow test 9. Dev branch: new queries (game + distractor), full game flow test
10. Drizzle migration on prod Postgres + data import + verify 10. Drizzle migration on prod Postgres + data import + verify

231
documentation/pipeline/roadmap.md
View file

@ -14,42 +14,48 @@
This roadmap has three zoom levels: This roadmap has three zoom levels:
- **Part 1 — Overview:** The phases at a glance. Read this to understand the full scope in 30 seconds. - **Part 1 — Overview:** The phases at a glance. Read this to
- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and acceptance criteria per phase. Read this to plan your week. understand the full scope in 30 seconds.
- **Part 3 — Detailed Tasks:** Step-by-step instructions within each phase. Read this when you sit down to code. - **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and
acceptance criteria per phase. Read this to plan your week.
- **Part 3 — Detailed Tasks:** Step-by-step instructions within each
phase. Read this when you sit down to code.
--- ---
# Part 1 — High-Level Overview # Part 1 — High-Level Overview
``` ```
Phase 0 Preparation Phase 1 Schema ✅
New Drizzle schema (words, senses, translations) with
relations, constraints, and indexes. Committed.
Phase 2 Preparation
Get wordlists, set up tooling, finalize the Gemini prompt. Get wordlists, set up tooling, finalize the Gemini prompt.
Phase 1 Data Pipeline Phase 3 Data Pipeline
Build the Gemini → validate → SQLite pipeline. Build the Gemini → validate → SQLite pipeline.
Produce a clean dataset of ~1000 nouns × 5 languages. Produce a clean dataset of ~1000 nouns × 5 languages.
Phase 2 Schema & Migration Phase 4 Migration & Import
Create the new Drizzle schema (words, senses, translations). Generate and apply the Drizzle migration.
Write the SQLite → Postgres import script. Write the SQLite → Postgres import script.
Migrate the local dev database.
Phase 3 App Integration Phase 5 App Integration
Rewrite the game and distractor queries against the new schema. Rewrite the game and distractor queries against the new schema.
Update the exercise-generation logic. Update the exercise-generation logic.
Test the full game flow in dev. Test the full game flow in dev.
Phase 4 Production Deploy Phase 6 Production Deploy
Run the Drizzle migration on prod. Run the Drizzle migration on prod.
Import the dataset. Import the dataset.
Verify the live app works end-to-end. Verify the live app works end-to-end.
Phase 5 Extend POS Phase 7 Extend POS
Run the pipeline for verbs, adjectives, adverbs. Run the pipeline for verbs, adjectives, adverbs.
No schema changes needed — new wordlists + adjusted prompts. No schema changes needed — new wordlists + adjusted prompts.
Phase 6 Future Features (out of scope for now) Phase 8 Future Features (out of scope for now)
Inflection tables, conjugation/declension exercises, Inflection tables, conjugation/declension exercises,
gender exercises, spaced-repetition scheduling. gender exercises, spaced-repetition scheduling.
``` ```
@ -57,22 +63,41 @@ Phase 6 Future Features (out of scope for now)
**Dependency chain:** **Dependency chain:**
``` ```
Phase 0 → Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5 Phase 1 ✅ → Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6 → Phase 7
│ │
▼ ▼
Phase 6 Phase 8
``` ```
Each phase depends on the previous one being complete. Do not skip
ahead.
--- ---
# Part 2 — Phase Breakdown # Part 2 — Phase Breakdown
--- ---
## Phase 0: Preparation ## Phase 1: Schema ✅ COMPLETE
**Goal:** New normalized schema exists in Drizzle with all tables,
constraints, indexes, and relations.
**Completed tasks:**
- [x] `words` table: headword, language_code, pos, UNIQUE, CHECKs, index
- [x] `senses` table: word_id FK (cascade), sense_index, difficulty,
definitions TEXT[], examples TEXT[], UNIQUE, CHECK, index
- [x] `translations` table: sense_id FK (cascade), target_language_code,
translation, gender (nullable), difficulty, UNIQUE, CHECKs, index
- [x] Relations: words→senses (many), senses→word (one) + translations
(many), translations→sense (one)
- [x] `NOUN_GENDERS` constant added to `@lila/shared`
- [x] `DIFFICULTY_LEVELS` updated: "intermediate" → "medium"
- [x] Old tables (`vocabulary_entries`, `entry_translations`) untouched
- [x] Auth and lobby tables untouched
- [x] Build passes, committed
---
## Phase 2: Preparation
**Goal:** Have everything you need before writing pipeline code. **Goal:** Have everything you need before writing pipeline code.
@ -86,11 +111,12 @@ ahead.
- [ ] Refine the prompt until the JSON output matches the contract - [ ] Refine the prompt until the JSON output matches the contract
defined in `docs/schema-design.md` §6.3 defined in `docs/schema-design.md` §6.3
**Dependencies:** None. **Dependencies:** Phase 1 complete.
**Acceptance criteria:** **Acceptance criteria:**
- You have 5 wordlist files in `data/wordlists/` (one per language). - You have 5 wordlist files in `data-pipeline/source-data/` (one per
language).
- A Gemini call with 5 German nouns returns valid JSON matching the - A Gemini call with 5 German nouns returns valid JSON matching the
contract, including definitions, examples, translations with gender, contract, including definitions, examples, translations with gender,
and difficulty levels. and difficulty levels.
@ -98,7 +124,7 @@ ahead.
--- ---
## Phase 1: Data Pipeline ## Phase 3: Data Pipeline
**Goal:** A repeatable script that takes a wordlist, calls Gemini in **Goal:** A repeatable script that takes a wordlist, calls Gemini in
batches of 20, validates the output, and writes clean rows to SQLite. batches of 20, validates the output, and writes clean rows to SQLite.
@ -112,7 +138,7 @@ batches of 20, validates the output, and writes clean rows to SQLite.
- [ ] Review the rejection log, fix prompt issues, re-run failed batches - [ ] Review the rejection log, fix prompt issues, re-run failed batches
- [ ] Spot-check 50 random entries for correctness - [ ] Spot-check 50 random entries for correctness
**Dependencies:** Phase 0 complete. **Dependencies:** Phase 2 complete.
**Acceptance criteria:** **Acceptance criteria:**
@ -125,16 +151,13 @@ batches of 20, validates the output, and writes clean rows to SQLite.
--- ---
## Phase 2: Schema & Migration ## Phase 4: Migration & Import
**Goal:** The new normalized schema exists in Drizzle, the local **Goal:** The new schema exists in Postgres and the SQLite data is
Postgres database is migrated, and the SQLite data is imported. imported.
**Tasks:** **Tasks:**
- [ ] Write the new Drizzle schema file: - `words` table (headword, language_code, pos) - `senses` table (word_id FK, sense_index, difficulty,
cefr_level, definitions TEXT[], examples TEXT[]) - `translations` table (sense_id FK, target_language_code,
translation, gender, difficulty) - All CHECK constraints, UNIQUE constraints, indexes
- [ ] Generate the Drizzle migration (`npx drizzle-kit generate`) - [ ] Generate the Drizzle migration (`npx drizzle-kit generate`)
- [ ] Inspect the generated SQL file — verify it creates the right - [ ] Inspect the generated SQL file — verify it creates the right
tables, constraints, and indexes tables, constraints, and indexes
@ -146,7 +169,7 @@ Postgres database is migrated, and the SQLite data is imported.
- [ ] Run 3–5 manual SQL queries against Postgres to sanity-check - [ ] Run 3–5 manual SQL queries against Postgres to sanity-check
the data the data
**Dependencies:** Phase 1 complete (SQLite has data). **Dependencies:** Phase 3 complete (SQLite has data).
**Acceptance criteria:** **Acceptance criteria:**
@ -158,7 +181,7 @@ Postgres database is migrated, and the SQLite data is imported.
--- ---
## Phase 3: App Integration ## Phase 5: App Integration
**Goal:** The running app uses the new schema. Game rounds and **Goal:** The running app uses the new schema. Game rounds and
distractors work correctly for all language pairs. distractors work correctly for all language pairs.
@ -177,7 +200,7 @@ distractors work correctly for all language pairs.
- [ ] Test edge cases: - A difficulty/pos/language combo with very few words - [ ] Test edge cases: - A difficulty/pos/language combo with very few words
(does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?) (does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?)
**Dependencies:** Phase 2 complete (Postgres has data, schema exists). **Dependencies:** Phase 4 complete (Postgres has data, schema exists).
**Acceptance criteria:** **Acceptance criteria:**
@ -189,7 +212,7 @@ distractors work correctly for all language pairs.
--- ---
## Phase 4: Production Deploy ## Phase 6: Production Deploy
**Goal:** The live app runs on the new schema with the new data. **Goal:** The live app runs on the new schema with the new data.
@ -203,7 +226,7 @@ distractors work correctly for all language pairs.
- [ ] Monitor for errors (server logs, browser console) for 24h - [ ] Monitor for errors (server logs, browser console) for 24h
- [ ] Remove old tables from prod (after confirming everything works) - [ ] Remove old tables from prod (after confirming everything works)
**Dependencies:** Phase 3 complete (dev is fully working). **Dependencies:** Phase 5 complete (dev is fully working).
**Acceptance criteria:** **Acceptance criteria:**
@ -214,7 +237,7 @@ distractors work correctly for all language pairs.
--- ---
## Phase 5: Extend POS ## Phase 7: Extend POS
**Goal:** Verbs, adjectives, and adverbs are in the database and **Goal:** Verbs, adjectives, and adverbs are in the database and
usable in the app. usable in the app.
@ -229,7 +252,7 @@ usable in the app.
- [ ] Test game flow with verbs, adjectives, adverbs - [ ] Test game flow with verbs, adjectives, adverbs
- [ ] Verify the POS filter in the app UI works for all types - [ ] Verify the POS filter in the app UI works for all types
**Dependencies:** Phase 4 complete. **Dependencies:** Phase 6 complete.
**Acceptance criteria:** **Acceptance criteria:**
@ -238,7 +261,7 @@ usable in the app.
--- ---
## Phase 6: Future Features (out of scope) ## Phase 8: Future Features (out of scope)
Listed here for visibility. Not planned, not estimated. Listed here for visibility. Not planned, not estimated.
@ -255,9 +278,9 @@ Listed here for visibility. Not planned, not estimated.
--- ---
## Phase 0: Preparation — Detailed ## Phase 2: Preparation — Detailed
### 0.1 Acquire wordlists ### 2.1 Acquire wordlists
- Search for frequency lists. Good starting points: - Search for frequency lists. Good starting points:
- "Leipzig Corpora Collection" (academic, per-language) - "Leipzig Corpora Collection" (academic, per-language)
@ -267,11 +290,11 @@ Listed here for visibility. Not planned, not estimated.
- Tatoeba sentence counts as a proxy for word frequency - Tatoeba sentence counts as a proxy for word frequency
- Target: ~1000 nouns per language, sorted by frequency. - Target: ~1000 nouns per language, sorted by frequency.
- Format: plain text, one word per line, UTF-8, no headers. - Format: plain text, one word per line, UTF-8, no headers.
- Save to `data/wordlists/nouns_de.txt`, `nouns_es.txt`, etc. - Save to `data-pipeline/source-data/{language}/nouns/`.
- Clean the lists: remove duplicates, remove words with spaces - Clean the lists: remove duplicates, remove words with spaces
(multi-word expressions), remove proper nouns if desired. (multi-word expressions), remove proper nouns if desired.
### 0.2 Set up local Postgres ### 2.2 Set up local Postgres
- Option A (Docker): - Option A (Docker):
``` ```
@ -288,7 +311,7 @@ Listed here for visibility. Not planned, not estimated.
- Verify: connect with `psql` or a GUI client (TablePlus, DBeaver, - Verify: connect with `psql` or a GUI client (TablePlus, DBeaver,
pgAdmin). Run `SELECT 1;`. pgAdmin). Run `SELECT 1;`.
### 0.3 Write and test the Gemini prompt ### 2.3 Write and test the Gemini prompt
- Start with 5 German nouns. - Start with 5 German nouns.
- The prompt should specify: - The prompt should specify:
@ -297,7 +320,6 @@ Listed here for visibility. Not planned, not estimated.
- That translations are needed for all 4 other supported languages - That translations are needed for all 4 other supported languages
- That gender must be provided for de/fr/es/it, null for en - That gender must be provided for de/fr/es/it, null for en
- That difficulty must be one of: easy, medium, hard - That difficulty must be one of: easy, medium, hard
- That CEFR level must be one of: A1, A2, B1, B2, C1, C2
- That multiple senses should be included for polysemous words - That multiple senses should be included for polysemous words
- Call the API. Inspect the JSON. Common issues to fix: - Call the API. Inspect the JSON. Common issues to fix:
- Gemini wraps the JSON in markdown code fences → strip them - Gemini wraps the JSON in markdown code fences → strip them
@ -308,12 +330,12 @@ Listed here for visibility. Not planned, not estimated.
--- ---
## Phase 1: Data Pipeline — Detailed ## Phase 3: Data Pipeline — Detailed
### 1.1 Create the SQLite schema ### 3.1 Create the SQLite schema
- Create a file `pipeline/schema.sql` or define it in your pipeline - Create a file `data-pipeline/schema.sql` or define it in your
script. pipeline script.
- Tables mirror the Postgres schema: - Tables mirror the Postgres schema:
```sql ```sql
@ -331,7 +353,6 @@ Listed here for visibility. Not planned, not estimated.
word_id TEXT NOT NULL REFERENCES words(id), word_id TEXT NOT NULL REFERENCES words(id),
sense_index INTEGER NOT NULL DEFAULT 0, sense_index INTEGER NOT NULL DEFAULT 0,
difficulty TEXT NOT NULL, difficulty TEXT NOT NULL,
cefr_level TEXT,
definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text
examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text
created_at TEXT DEFAULT (datetime('now')), created_at TEXT DEFAULT (datetime('now')),
@ -353,9 +374,9 @@ Listed here for visibility. Not planned, not estimated.
- Note: SQLite has no native TEXT[]. Store arrays as JSON text. - Note: SQLite has no native TEXT[]. Store arrays as JSON text.
The import script will parse them into Postgres TEXT[] on import. The import script will parse them into Postgres TEXT[] on import.
### 1.2 Write the validation module ### 3.2 Write the validation module
- Create `pipeline/validate.ts` (or .js). - Create `data-pipeline/validate.ts`.
- Input: one parsed Gemini entry (the JSON object for one word). - Input: one parsed Gemini entry (the JSON object for one word).
- Checks (return a list of errors, empty = valid): - Checks (return a list of errors, empty = valid):
- `headword` is a non-empty string - `headword` is a non-empty string
@ -365,8 +386,6 @@ Listed here for visibility. Not planned, not estimated.
- Each sense has: - Each sense has:
- `sense_index` is a non-negative integer - `sense_index` is a non-negative integer
- `difficulty` is in ['easy','medium','hard'] - `difficulty` is in ['easy','medium','hard']
- `cefr_level` is in ['A1','A2','B1','B2','C1','C2'] or null
- CEFR ↔ difficulty mapping is consistent
- `definitions` is a non-empty array of non-empty strings - `definitions` is a non-empty array of non-empty strings
- `examples` is a non-empty array of non-empty strings - `examples` is a non-empty array of non-empty strings
- `translations` is a non-empty array - `translations` is a non-empty array
@ -381,9 +400,9 @@ Listed here for visibility. Not planned, not estimated.
- `difficulty` is in ['easy','medium','hard'] - `difficulty` is in ['easy','medium','hard']
- Output: `{ valid: boolean, errors: string[] }` - Output: `{ valid: boolean, errors: string[] }`
### 1.3 Write the pipeline script ### 3.3 Write the pipeline script
- Create `pipeline/run.ts`. - Create `data-pipeline/run.ts`.
- Pseudocode: - Pseudocode:
``` ```
for each language in [de, en, es, fr, it]: for each language in [de, en, es, fr, it]:
@ -398,7 +417,7 @@ Listed here for visibility. Not planned, not estimated.
generate UUIDs for word, senses, translations generate UUIDs for word, senses, translations
INSERT into SQLite (words, senses, translations) INSERT into SQLite (words, senses, translations)
else: else:
append to rejection log (data/rejections/{lang}_{pos}.jsonl) append to rejection log
log progress: "Batch 12/50 done. 238 words imported, 2 rejected." log progress: "Batch 12/50 done. 238 words imported, 2 rejected."
sleep 1s (rate limiting) sleep 1s (rate limiting)
``` ```
@ -407,7 +426,7 @@ Listed here for visibility. Not planned, not estimated.
- Store definitions/examples as JSON strings in SQLite - Store definitions/examples as JSON strings in SQLite
(`JSON.stringify(arr)`). (`JSON.stringify(arr)`).
### 1.4 Run and review ### 3.4 Run and review
- Run the pipeline for all 5 languages. - Run the pipeline for all 5 languages.
- Check the rejection log. If rejection rate > 10%, fix the prompt - Check the rejection log. If rejection rate > 10%, fix the prompt
@ -427,31 +446,21 @@ Listed here for visibility. Not planned, not estimated.
--- ---
## Phase 2: Schema & Migration — Detailed ## Phase 4: Migration & Import — Detailed
### 2.1 Write the Drizzle schema ### 4.1 Generate and inspect the migration
- Create or update `src/db/schema.ts`.
- Define the three tables using Drizzle's `pgTable` builder.
- Include all columns, types, constraints, and indexes as specified
in `docs/schema-design.md` §3.
- Keep the old tables in the file for now (commented out or in a
separate file) so the app doesn't break during development.
### 2.2 Generate and inspect the migration
- Run: `npx drizzle-kit generate` - Run: `npx drizzle-kit generate`
- Open the generated SQL file in `drizzle/` (or wherever your - Open the generated SQL file in `packages/db/drizzle/`.
config puts it).
- Read it. Verify: - Read it. Verify:
- Three CREATE TABLE statements - Three CREATE TABLE statements (words, senses, translations)
- CHECK constraints match your schema - CHECK constraints match your schema
- UNIQUE constraints are present - UNIQUE constraints are present
- Three CREATE INDEX statements - Three CREATE INDEX statements
- Foreign keys reference the correct tables with ON DELETE CASCADE - Foreign keys reference the correct tables with ON DELETE CASCADE
- If something looks wrong, fix the schema file and regenerate. - If something looks wrong, fix the schema file and regenerate.
### 2.3 Apply the migration locally ### 4.2 Apply the migration locally
- Run: `npx drizzle-kit migrate` - Run: `npx drizzle-kit migrate`
- Connect to local Postgres and verify: - Connect to local Postgres and verify:
@ -462,9 +471,9 @@ Listed here for visibility. Not planned, not estimated.
\d translations -- describe translations table \d translations -- describe translations table
``` ```
### 2.4 Write the import script ### 4.3 Write the import script
- Create `scripts/import-to-postgres.ts`. - Create `data-pipeline/import-to-postgres.ts`.
- Pseudocode: - Pseudocode:
``` ```
@ -492,7 +501,7 @@ Listed here for visibility. Not planned, not estimated.
into actual arrays (Postgres TEXT[]). into actual arrays (Postgres TEXT[]).
- Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully. - Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully.
### 2.5 Run and verify ### 4.4 Run and verify
- Run the import script. - Run the import script.
- Compare counts: - Compare counts:
@ -529,11 +538,11 @@ Listed here for visibility. Not planned, not estimated.
--- ---
## Phase 3: App Integration — Detailed ## Phase 5: App Integration — Detailed
### 3.1 Rewrite getGameTerms ### 5.1 Rewrite getGameTerms
- Open the file containing `getGameTerms`. - Open `packages/db/src/models/termModel.ts`.
- Replace the old query (2-table join on `vocabulary_entries` + - Replace the old query (2-table join on `vocabulary_entries` +
`entry_translations`) with the new 3-table join `entry_translations`) with the new 3-table join
(words → senses → translations). (words → senses → translations).
@ -545,7 +554,7 @@ Listed here for visibility. Not planned, not estimated.
- Return: word_id, headword, sense_id, definitions, examples, - Return: word_id, headword, sense_id, definitions, examples,
translation, gender. translation, gender.
### 3.2 Rewrite getDistractors ### 5.2 Rewrite getDistractors
- Same 3-table join. - Same 3-table join.
- Additional filters: - Additional filters:
@ -555,7 +564,7 @@ Listed here for visibility. Not planned, not estimated.
- If fewer than 3 distractors are found (small data pool), handle - If fewer than 3 distractors are found (small data pool), handle
gracefully: reduce the number of options or log a warning. gracefully: reduce the number of options or log a warning.
### 3.3 Update exercise generation ### 5.3 Update exercise generation
- In the function that assembles a game round: - In the function that assembles a game round:
- Pick one random definition: - Pick one random definition:
@ -566,7 +575,7 @@ Listed here for visibility. Not planned, not estimated.
- Shuffle the 4 options (Fisher-Yates or similar). - Shuffle the 4 options (Fisher-Yates or similar).
- Attach gender to each option for display. - Attach gender to each option for display.
### 3.4 Test matrix ### 5.4 Test matrix
Run through this matrix manually in the dev app: Run through this matrix manually in the dev app:
@ -584,18 +593,18 @@ appear as distractors, no duplicate options.
--- ---
## Phase 4: Production Deploy — Detailed ## Phase 6: Production Deploy — Detailed
### 4.1 Pre-deploy checklist ### 6.1 Pre-deploy checklist
- [ ] All Phase 3 acceptance criteria pass - [ ] All Phase 5 acceptance criteria pass
- [ ] `git status` is clean, all changes committed - [ ] `git status` is clean, all changes committed
- [ ] The Drizzle migration file is committed to the repo - [ ] The Drizzle migration file is committed to the repo
- [ ] You know your prod database connection string - [ ] You know your prod database connection string
- [ ] You have a backup method for prod (pg_dump, hosting provider - [ ] You have a backup method for prod (pg_dump, hosting provider
snapshot, etc.) snapshot, etc.)
### 4.2 Deploy ### 6.2 Deploy
- Back up prod: - Back up prod:
``` ```
@ -607,11 +616,11 @@ appear as distractors, no duplicate options.
``` ```
- Run import script against prod: - Run import script against prod:
``` ```
DATABASE_URL=<prod-url> npx tsx scripts/import-to-postgres.ts DATABASE_URL=<prod-url> npx tsx data-pipeline/import-to-postgres.ts
``` ```
- Verify counts on prod. - Verify counts on prod.
### 4.3 Post-deploy verification ### 6.3 Post-deploy verification
- Open the live app. Play 2 full games with different settings. - Open the live app. Play 2 full games with different settings.
- Check server logs for errors. - Check server logs for errors.
@ -623,24 +632,24 @@ appear as distractors, no duplicate options.
``` ```
(Or keep them for another week if you want a safety net.) (Or keep them for another week if you want a safety net.)
### 4.4 Rollback plan ### 6.4 Rollback plan
If something goes wrong: If something goes wrong:
- Restore the backup: - Restore the backup:
``` ```
psql -h <host> -U <user> -d <db> < backup_20260722.sql psql -h <host> -U <user> -d <db> < backup_YYYYMMDD.sql
``` ```
- Revert the code to the previous commit. - Revert the code to the previous commit.
- Redeploy. - Redeploy.
--- ---
## Phase 5: Extend POS — Detailed ## Phase 7: Extend POS — Detailed
### 5.1 Per POS ### 7.1 Per POS
Repeat the Phase 1 pipeline for each new POS: Repeat the Phase 3 pipeline for each new POS:
- Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages. - Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages.
- Adjust the Gemini prompt: - Adjust the Gemini prompt:
@ -653,7 +662,7 @@ Repeat the Phase 1 pipeline for each new POS:
- Run pipeline → validate → SQLite → import → Postgres. - Run pipeline → validate → SQLite → import → Postgres.
- Test in the app with the POS filter set to the new type. - Test in the app with the POS filter set to the new type.
### 5.2 No schema changes ### 7.2 No schema changes
The `pos` column already supports all four types. The `gender` The `pos` column already supports all four types. The `gender`
column is nullable and simply won't be populated for adverbs or column is nullable and simply won't be populated for adverbs or
@ -678,27 +687,27 @@ English words. No migration needed.
``` ```
project/ project/
├── docs/ ├── data-pipeline/
│ ├── schema-design.md ← companion design doc │ ├── source-data/
│ └── roadmap.md ← this document │ │ ├── german/nouns/ ← wordlist files
├── data/ │ │ ├── english/nouns/
│ ├── wordlists/ │ │ ├── spanish/nouns/
│ │ ├── nouns_de.txt │ │ ├── french/nouns/
│ │ ├── nouns_en.txt │ │ └── italian/nouns/
│ │ ├── nouns_es.txt
│ │ ├── nouns_fr.txt
│ │ └── nouns_it.txt
│ ├── rejections/ ← invalid Gemini entries (for review) │ ├── rejections/ ← invalid Gemini entries (for review)
│ └── staging.db ← SQLite staging database │ ├── staging.db ← SQLite staging database
├── pipeline/ │ ├── schema.sql ← SQLite schema definition
│ ├── run.ts ← main pipeline script
│ ├── validate.ts ← validation module │ ├── validate.ts ← validation module
│ ├── prompt.ts ← Gemini prompt template │ ├── run.ts ← main pipeline script
│ └── schema.sql ← SQLite schema definition
├── scripts/
│ └── import-to-postgres.ts ← SQLite → Postgres import │ └── import-to-postgres.ts ← SQLite → Postgres import
├── src/ ├── packages/
│ └── db/ │ ├── db/
│ └── schema.ts ← Drizzle schema (updated) │ │ └── src/db/schema.ts ← Drizzle schema (updated ✅)
└── drizzle/ ← generated migration files │ └── shared/
│ └── src/constants.ts ← NOUN_GENDERS added ✅, "medium" ✅
├── documentation/
│ └── pipeline/
│ ├── design-doc.md ← schema design doc (updated ✅)
│ └── roadmap.md ← this document (updated ✅)
└── ...
``` ```