forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

updating documentation

Browse source
This commit is contained in:
lila 2026-07-23 16:57:07 +02:00
parent de3ebf66a9
commit e0b1f7f92f
2 changed files with 139 additions and 143 deletions
Download patch file Download diff file Expand all files Collapse all files

235
documentation/pipeline/roadmap.md
View file

@ -14,42 +14,48 @@
This roadmap has three zoom levels:
- **Part 1 — Overview:** The phases at a glance. Read this to understand the full scope in 30 seconds.
- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and acceptance criteria per phase. Read this to plan your week.
- **Part 3 — Detailed Tasks:** Step-by-step instructions within each phase. Read this when you sit down to code.
- **Part 1 — Overview:** The phases at a glance. Read this to
understand the full scope in 30 seconds.
- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and
acceptance criteria per phase. Read this to plan your week.
- **Part 3 — Detailed Tasks:** Step-by-step instructions within each
phase. Read this when you sit down to code.
---
# Part 1 — High-Level Overview
```
Phase 0 Preparation
Phase 1 Schema ✅
New Drizzle schema (words, senses, translations) with
relations, constraints, and indexes. Committed.
Phase 2 Preparation
Get wordlists, set up tooling, finalize the Gemini prompt.
Phase 1 Data Pipeline
Phase 3 Data Pipeline
Build the Gemini → validate → SQLite pipeline.
Produce a clean dataset of ~1000 nouns × 5 languages.
Phase 2 Schema & Migration
Create the new Drizzle schema (words, senses, translations).
Phase 4 Migration & Import
Generate and apply the Drizzle migration.
Write the SQLite → Postgres import script.
Migrate the local dev database.
Phase 3 App Integration
Phase 5 App Integration
Rewrite the game and distractor queries against the new schema.
Update the exercise-generation logic.
Test the full game flow in dev.
Phase 4 Production Deploy
Phase 6 Production Deploy
Run the Drizzle migration on prod.
Import the dataset.
Verify the live app works end-to-end.
Phase 5 Extend POS
Phase 7 Extend POS
Run the pipeline for verbs, adjectives, adverbs.
No schema changes needed — new wordlists + adjusted prompts.
Phase 6 Future Features (out of scope for now)
Phase 8 Future Features (out of scope for now)
Inflection tables, conjugation/declension exercises,
gender exercises, spaced-repetition scheduling.
```
@ -57,22 +63,41 @@ Phase 6 Future Features (out of scope for now)
**Dependency chain:**
```
Phase 0 → Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5
│
▼
Phase 6
Phase 1 ✅ → Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6 → Phase 7
│
▼
Phase 8
```
Each phase depends on the previous one being complete. Do not skip
ahead.
---
# Part 2 — Phase Breakdown
---
## Phase 0: Preparation
## Phase 1: Schema ✅ COMPLETE
**Goal:** New normalized schema exists in Drizzle with all tables,
constraints, indexes, and relations.
**Completed tasks:**
- [x] `words` table: headword, language_code, pos, UNIQUE, CHECKs, index
- [x] `senses` table: word_id FK (cascade), sense_index, difficulty,
definitions TEXT[], examples TEXT[], UNIQUE, CHECK, index
- [x] `translations` table: sense_id FK (cascade), target_language_code,
translation, gender (nullable), difficulty, UNIQUE, CHECKs, index
- [x] Relations: words→senses (many), senses→word (one) + translations
(many), translations→sense (one)
- [x] `NOUN_GENDERS` constant added to `@lila/shared`
- [x] `DIFFICULTY_LEVELS` updated: "intermediate" → "medium"
- [x] Old tables (`vocabulary_entries`, `entry_translations`) untouched
- [x] Auth and lobby tables untouched
- [x] Build passes, committed
---
## Phase 2: Preparation
**Goal:** Have everything you need before writing pipeline code.
@ -86,11 +111,12 @@ ahead.
- [ ] Refine the prompt until the JSON output matches the contract
defined in `docs/schema-design.md` §6.3
**Dependencies:** None.
**Dependencies:** Phase 1 complete.
**Acceptance criteria:**
- You have 5 wordlist files in `data/wordlists/` (one per language).
- You have 5 wordlist files in `data-pipeline/source-data/` (one per
language).
- A Gemini call with 5 German nouns returns valid JSON matching the
contract, including definitions, examples, translations with gender,
and difficulty levels.
@ -98,7 +124,7 @@ ahead.
---
## Phase 1: Data Pipeline
## Phase 3: Data Pipeline
**Goal:** A repeatable script that takes a wordlist, calls Gemini in
batches of 20, validates the output, and writes clean rows to SQLite.
@ -112,7 +138,7 @@ batches of 20, validates the output, and writes clean rows to SQLite.
- [ ] Review the rejection log, fix prompt issues, re-run failed batches
- [ ] Spot-check 50 random entries for correctness
**Dependencies:** Phase 0 complete.
**Dependencies:** Phase 2 complete.
**Acceptance criteria:**
@ -125,16 +151,13 @@ batches of 20, validates the output, and writes clean rows to SQLite.
---
## Phase 2: Schema & Migration
## Phase 4: Migration & Import
**Goal:** The new normalized schema exists in Drizzle, the local
Postgres database is migrated, and the SQLite data is imported.
**Goal:** The new schema exists in Postgres and the SQLite data is
imported.
**Tasks:**
- [ ] Write the new Drizzle schema file: - `words` table (headword, language_code, pos) - `senses` table (word_id FK, sense_index, difficulty,
cefr_level, definitions TEXT[], examples TEXT[]) - `translations` table (sense_id FK, target_language_code,
translation, gender, difficulty) - All CHECK constraints, UNIQUE constraints, indexes
- [ ] Generate the Drizzle migration (`npx drizzle-kit generate`)
- [ ] Inspect the generated SQL file — verify it creates the right
tables, constraints, and indexes
@ -146,7 +169,7 @@ Postgres database is migrated, and the SQLite data is imported.
- [ ] Run 3–5 manual SQL queries against Postgres to sanity-check
the data
**Dependencies:** Phase 1 complete (SQLite has data).
**Dependencies:** Phase 3 complete (SQLite has data).
**Acceptance criteria:**
@ -158,7 +181,7 @@ Postgres database is migrated, and the SQLite data is imported.
---
## Phase 3: App Integration
## Phase 5: App Integration
**Goal:** The running app uses the new schema. Game rounds and
distractors work correctly for all language pairs.
@ -177,7 +200,7 @@ distractors work correctly for all language pairs.
- [ ] Test edge cases: - A difficulty/pos/language combo with very few words
(does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?)
**Dependencies:** Phase 2 complete (Postgres has data, schema exists).
**Dependencies:** Phase 4 complete (Postgres has data, schema exists).
**Acceptance criteria:**
@ -189,7 +212,7 @@ distractors work correctly for all language pairs.
---
## Phase 4: Production Deploy
## Phase 6: Production Deploy
**Goal:** The live app runs on the new schema with the new data.
@ -203,7 +226,7 @@ distractors work correctly for all language pairs.
- [ ] Monitor for errors (server logs, browser console) for 24h
- [ ] Remove old tables from prod (after confirming everything works)
**Dependencies:** Phase 3 complete (dev is fully working).
**Dependencies:** Phase 5 complete (dev is fully working).
**Acceptance criteria:**
@ -214,7 +237,7 @@ distractors work correctly for all language pairs.
---
## Phase 5: Extend POS
## Phase 7: Extend POS
**Goal:** Verbs, adjectives, and adverbs are in the database and
usable in the app.
@ -229,7 +252,7 @@ usable in the app.
- [ ] Test game flow with verbs, adjectives, adverbs
- [ ] Verify the POS filter in the app UI works for all types
**Dependencies:** Phase 4 complete.
**Dependencies:** Phase 6 complete.
**Acceptance criteria:**
@ -238,7 +261,7 @@ usable in the app.
---
## Phase 6: Future Features (out of scope)
## Phase 8: Future Features (out of scope)
Listed here for visibility. Not planned, not estimated.
@ -255,9 +278,9 @@ Listed here for visibility. Not planned, not estimated.
---
## Phase 0: Preparation — Detailed
## Phase 2: Preparation — Detailed
### 0.1 Acquire wordlists
### 2.1 Acquire wordlists
- Search for frequency lists. Good starting points:
- "Leipzig Corpora Collection" (academic, per-language)
@ -267,11 +290,11 @@ Listed here for visibility. Not planned, not estimated.
- Tatoeba sentence counts as a proxy for word frequency
- Target: ~1000 nouns per language, sorted by frequency.
- Format: plain text, one word per line, UTF-8, no headers.
- Save to `data/wordlists/nouns_de.txt`, `nouns_es.txt`, etc.
- Save to `data-pipeline/source-data/{language}/nouns/`.
- Clean the lists: remove duplicates, remove words with spaces
(multi-word expressions), remove proper nouns if desired.
### 0.2 Set up local Postgres
### 2.2 Set up local Postgres
- Option A (Docker):
```
@ -288,7 +311,7 @@ Listed here for visibility. Not planned, not estimated.
- Verify: connect with `psql` or a GUI client (TablePlus, DBeaver,
pgAdmin). Run `SELECT 1;`.
### 0.3 Write and test the Gemini prompt
### 2.3 Write and test the Gemini prompt
- Start with 5 German nouns.
- The prompt should specify:
@ -297,7 +320,6 @@ Listed here for visibility. Not planned, not estimated.
- That translations are needed for all 4 other supported languages
- That gender must be provided for de/fr/es/it, null for en
- That difficulty must be one of: easy, medium, hard
- That CEFR level must be one of: A1, A2, B1, B2, C1, C2
- That multiple senses should be included for polysemous words
- Call the API. Inspect the JSON. Common issues to fix:
- Gemini wraps the JSON in markdown code fences → strip them
@ -308,12 +330,12 @@ Listed here for visibility. Not planned, not estimated.
---
## Phase 1: Data Pipeline — Detailed
## Phase 3: Data Pipeline — Detailed
### 1.1 Create the SQLite schema
### 3.1 Create the SQLite schema
- Create a file `pipeline/schema.sql` or define it in your pipeline
script.
- Create a file `data-pipeline/schema.sql` or define it in your
pipeline script.
- Tables mirror the Postgres schema:
```sql
@ -331,7 +353,6 @@ Listed here for visibility. Not planned, not estimated.
word_id TEXT NOT NULL REFERENCES words(id),
sense_index INTEGER NOT NULL DEFAULT 0,
difficulty TEXT NOT NULL,
cefr_level TEXT,
definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text
examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text
created_at TEXT DEFAULT (datetime('now')),
@ -353,9 +374,9 @@ Listed here for visibility. Not planned, not estimated.
- Note: SQLite has no native TEXT[]. Store arrays as JSON text.
The import script will parse them into Postgres TEXT[] on import.
### 1.2 Write the validation module
### 3.2 Write the validation module
- Create `pipeline/validate.ts` (or .js).
- Create `data-pipeline/validate.ts`.
- Input: one parsed Gemini entry (the JSON object for one word).
- Checks (return a list of errors, empty = valid):
- `headword` is a non-empty string
@ -365,8 +386,6 @@ Listed here for visibility. Not planned, not estimated.
- Each sense has:
- `sense_index` is a non-negative integer
- `difficulty` is in ['easy','medium','hard']
- `cefr_level` is in ['A1','A2','B1','B2','C1','C2'] or null
- CEFR ↔ difficulty mapping is consistent
- `definitions` is a non-empty array of non-empty strings
- `examples` is a non-empty array of non-empty strings
- `translations` is a non-empty array
@ -381,9 +400,9 @@ Listed here for visibility. Not planned, not estimated.
- `difficulty` is in ['easy','medium','hard']
- Output: `{ valid: boolean, errors: string[] }`
### 1.3 Write the pipeline script
### 3.3 Write the pipeline script
- Create `pipeline/run.ts`.
- Create `data-pipeline/run.ts`.
- Pseudocode:
```
for each language in [de, en, es, fr, it]:
@ -398,7 +417,7 @@ Listed here for visibility. Not planned, not estimated.
generate UUIDs for word, senses, translations
INSERT into SQLite (words, senses, translations)
else:
append to rejection log (data/rejections/{lang}_{pos}.jsonl)
append to rejection log
log progress: "Batch 12/50 done. 238 words imported, 2 rejected."
sleep 1s (rate limiting)
```
@ -407,7 +426,7 @@ Listed here for visibility. Not planned, not estimated.
- Store definitions/examples as JSON strings in SQLite
(`JSON.stringify(arr)`).
### 1.4 Run and review
### 3.4 Run and review
- Run the pipeline for all 5 languages.
- Check the rejection log. If rejection rate > 10%, fix the prompt
@ -427,31 +446,21 @@ Listed here for visibility. Not planned, not estimated.
---
## Phase 2: Schema & Migration — Detailed
## Phase 4: Migration & Import — Detailed
### 2.1 Write the Drizzle schema
- Create or update `src/db/schema.ts`.
- Define the three tables using Drizzle's `pgTable` builder.
- Include all columns, types, constraints, and indexes as specified
in `docs/schema-design.md` §3.
- Keep the old tables in the file for now (commented out or in a
separate file) so the app doesn't break during development.
### 2.2 Generate and inspect the migration
### 4.1 Generate and inspect the migration
- Run: `npx drizzle-kit generate`
- Open the generated SQL file in `drizzle/` (or wherever your
config puts it).
- Open the generated SQL file in `packages/db/drizzle/`.
- Read it. Verify:
- Three CREATE TABLE statements
- Three CREATE TABLE statements (words, senses, translations)
- CHECK constraints match your schema
- UNIQUE constraints are present
- Three CREATE INDEX statements
- Foreign keys reference the correct tables with ON DELETE CASCADE
- If something looks wrong, fix the schema file and regenerate.
### 2.3 Apply the migration locally
### 4.2 Apply the migration locally
- Run: `npx drizzle-kit migrate`
- Connect to local Postgres and verify:
@ -462,9 +471,9 @@ Listed here for visibility. Not planned, not estimated.
\d translations -- describe translations table
```
### 2.4 Write the import script
### 4.3 Write the import script
- Create `scripts/import-to-postgres.ts`.
- Create `data-pipeline/import-to-postgres.ts`.
- Pseudocode:
```
@ -492,7 +501,7 @@ Listed here for visibility. Not planned, not estimated.
into actual arrays (Postgres TEXT[]).
- Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully.
### 2.5 Run and verify
### 4.4 Run and verify
- Run the import script.
- Compare counts:
@ -529,11 +538,11 @@ Listed here for visibility. Not planned, not estimated.
---
## Phase 3: App Integration — Detailed
## Phase 5: App Integration — Detailed
### 3.1 Rewrite getGameTerms
### 5.1 Rewrite getGameTerms
- Open the file containing `getGameTerms`.
- Open `packages/db/src/models/termModel.ts`.
- Replace the old query (2-table join on `vocabulary_entries` +
`entry_translations`) with the new 3-table join
(words → senses → translations).
@ -545,7 +554,7 @@ Listed here for visibility. Not planned, not estimated.
- Return: word_id, headword, sense_id, definitions, examples,
translation, gender.
### 3.2 Rewrite getDistractors
### 5.2 Rewrite getDistractors
- Same 3-table join.
- Additional filters:
@ -555,7 +564,7 @@ Listed here for visibility. Not planned, not estimated.
- If fewer than 3 distractors are found (small data pool), handle
gracefully: reduce the number of options or log a warning.
### 3.3 Update exercise generation
### 5.3 Update exercise generation
- In the function that assembles a game round:
- Pick one random definition:
@ -566,7 +575,7 @@ Listed here for visibility. Not planned, not estimated.
- Shuffle the 4 options (Fisher-Yates or similar).
- Attach gender to each option for display.
### 3.4 Test matrix
### 5.4 Test matrix
Run through this matrix manually in the dev app:
@ -584,18 +593,18 @@ appear as distractors, no duplicate options.
---
## Phase 4: Production Deploy — Detailed
## Phase 6: Production Deploy — Detailed
### 4.1 Pre-deploy checklist
### 6.1 Pre-deploy checklist
- [ ] All Phase 3 acceptance criteria pass
- [ ] All Phase 5 acceptance criteria pass
- [ ] `git status` is clean, all changes committed
- [ ] The Drizzle migration file is committed to the repo
- [ ] You know your prod database connection string
- [ ] You have a backup method for prod (pg_dump, hosting provider
snapshot, etc.)
### 4.2 Deploy
### 6.2 Deploy
- Back up prod:
```
@ -607,11 +616,11 @@ appear as distractors, no duplicate options.
```
- Run import script against prod:
```
DATABASE_URL=<prod-url> npx tsx scripts/import-to-postgres.ts
DATABASE_URL=<prod-url> npx tsx data-pipeline/import-to-postgres.ts
```
- Verify counts on prod.
### 4.3 Post-deploy verification
### 6.3 Post-deploy verification
- Open the live app. Play 2 full games with different settings.
- Check server logs for errors.
@ -623,24 +632,24 @@ appear as distractors, no duplicate options.
```
(Or keep them for another week if you want a safety net.)
### 4.4 Rollback plan
### 6.4 Rollback plan
If something goes wrong:
- Restore the backup:
```
psql -h <host> -U <user> -d <db> < backup_20260722.sql
psql -h <host> -U <user> -d <db> < backup_YYYYMMDD.sql
```
- Revert the code to the previous commit.
- Redeploy.
---
## Phase 5: Extend POS — Detailed
## Phase 7: Extend POS — Detailed
### 5.1 Per POS
### 7.1 Per POS
Repeat the Phase 1 pipeline for each new POS:
Repeat the Phase 3 pipeline for each new POS:
- Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages.
- Adjust the Gemini prompt:
@ -653,7 +662,7 @@ Repeat the Phase 1 pipeline for each new POS:
- Run pipeline → validate → SQLite → import → Postgres.
- Test in the app with the POS filter set to the new type.
### 5.2 No schema changes
### 7.2 No schema changes
The `pos` column already supports all four types. The `gender`
column is nullable and simply won't be populated for adverbs or
@ -678,27 +687,27 @@ English words. No migration needed.
```
project/
├── docs/
│ ├── schema-design.md ← companion design doc
│ └── roadmap.md ← this document
├── data/
│ ├── wordlists/
│ │ ├── nouns_de.txt
│ │ ├── nouns_en.txt
│ │ ├── nouns_es.txt
│ │ ├── nouns_fr.txt
│ │ └── nouns_it.txt
├── data-pipeline/
│ ├── source-data/
│ │ ├── german/nouns/ ← wordlist files
│ │ ├── english/nouns/
│ │ ├── spanish/nouns/
│ │ ├── french/nouns/
│ │ └── italian/nouns/
│ ├── rejections/ ← invalid Gemini entries (for review)
│ └── staging.db ← SQLite staging database
├── pipeline/
│ ├── run.ts ← main pipeline script
│ ├── staging.db ← SQLite staging database
│ ├── schema.sql ← SQLite schema definition
│ ├── validate.ts ← validation module
│ ├── prompt.ts ← Gemini prompt template
│ └── schema.sql ← SQLite schema definition
├── scripts/
│ ├── run.ts ← main pipeline script
│ └── import-to-postgres.ts ← SQLite → Postgres import
├── src/
│ └── db/
│ └── schema.ts ← Drizzle schema (updated)
└── drizzle/ ← generated migration files
├── packages/
│ ├── db/
│ │ └── src/db/schema.ts ← Drizzle schema (updated ✅)
│ └── shared/
│ └── src/constants.ts ← NOUN_GENDERS added ✅, "medium" ✅
├── documentation/
│ └── pipeline/
│ ├── design-doc.md ← schema design doc (updated ✅)
│ └── roadmap.md ← this document (updated ✅)
└── ...
```