forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

updating docs

Browse source
This commit is contained in:
lila 2026-07-23 14:15:33 +02:00
parent 88b16a1ed7
commit 61fa032d6c
2 changed files with 1321 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

617
documentation/pipeline/design-doc.md Normal file
View file

@ -0,0 +1,617 @@
# Vocabulary Trainer — Data Schema Design Document
> **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline
> **Author:** [Your Name]
> **Date:** July 2026
> **Status:** Approved — ready for implementation
---
## 1. Overview
This document describes the database schema, data pipeline, and query
patterns for the vocabulary trainer application. The app supports 5
languages (English, German, Italian, French, Spanish) and tests learners
by showing a word with a definition and example sentence, then asking
them to pick the correct translation from 4 options (1 correct, 3
distractors).
The previous data source (OpenWordNet / kaikki.org) produced
low-quality and inaccurate entries. The new approach uses a batch
pipeline: word frequency lists are fed to the Gemini API in groups of
20, which generates structured metadata (definitions, examples,
translations, difficulty levels). The output is validated, stored in a
local SQLite staging database, and mirrored into the production
Postgres database.
---
## 2. Database Choice
**Postgres** (production and development) with **SQLite** (pipeline
staging).
### Why Postgres
- The data is inherently relational: words → senses → translations.
Foreign keys enforce referential integrity at the database level.
- The query pattern (filter → join → random → limit) is exactly what
SQL is designed for.
- Postgres provides JSONB and native arrays for semi-structured fields
(definitions, examples, inflection tags) without sacrificing
relational structure.
- ACID transactions ensure batch imports are atomic.
- Already part of the PERN stack. No additional infrastructure.
### Why not MongoDB
- The data has real, meaningful relationships (not arbitrary nested
documents). The distractor query ("exclude translations from the
same sense") is a single `WHERE sense_id != X` in SQL but requires
a complex aggregation pipeline in MongoDB.
- No foreign key enforcement. Data integrity would depend entirely on
application code — risky with LLM-generated data.
- Future inflection tables (one word → 30–50 forms) are a natural
relational fit, not a document-store fit.
### Why not DynamoDB
- Designed for simple key-value lookups at massive scale (billions of
rows). Cannot do ad-hoc filtering, joins, or `ORDER BY RANDOM()`.
- The query pattern (filter by language + pos + difficulty, then
randomize) would require pre-building indexes for every combination.
- Massive overkill for 500k words.
### Why SQLite for staging
- Zero-config, file-based. Ideal for the single-writer batch pipeline.
- The pipeline writes to SQLite, a separate import script mirrors the
data into Postgres. The application (dev and prod) always reads
from Postgres to avoid SQLite/Postgres dialect differences.
---
## 3. Schema
### 3.1 Entity Relationship
```
┌─────────────┐ ┌─────────────┐ ┌──────────────────┐
│ words │ │ senses │ │ translations │
├─────────────┤ ├─────────────┤ ├──────────────────┤
│ id (PK) │──┐ │ id (PK) │──┐ │ id (PK) │
│ headword │ └───>│ word_id(FK) │ └───>│ sense_id (FK) │
│ language_code│ │ sense_index │ │ target_lang_code │
│ pos │ │ difficulty │ │ translation │
│ │ │ cefr_level │ │ gender │
│ │ │ definitions │ │ difficulty │
│ │ │ examples │ │ │
└─────────────┘ └─────────────┘ └──────────────────┘
Future (not yet implemented):
┌──────────────────┐
│ inflection_forms │
├──────────────────┤
│ id (PK) │
│ word_id (FK) ────────> words.id
│ form │
│ tags (JSONB) │
└──────────────────┘
```
### 3.2 Table: `words`
One row per unique word in a specific language.
| Column | Type | Constraints | Notes |
| ------------- | ----------- | --------------------------------- | ------------------------------------- |
| id | UUID | PK, default random | |
| headword | TEXT | NOT NULL | "Haus", "casa", "house" |
| language_code | VARCHAR(10) | NOT NULL, CHECK in supported list | "de", "es", "en", "fr", "it" |
| pos | VARCHAR(20) | NOT NULL, CHECK in supported list | "noun", "verb", "adjective", "adverb" |
| created_at | TIMESTAMPTZ | NOT NULL, default now() | |
**Constraints:**
- `UNIQUE (headword, language_code, pos)` — prevents duplicate entries.
- `CHECK (language_code IN ('en','de','it','fr','es'))`
- `CHECK (pos IN ('noun','verb','adjective','adverb'))`
**Index:**
- `idx_words_lang_pos ON (language_code, pos)` — accelerates the
primary game query filter.
**Design note:** Each language gets its own headword entries. "Haus"
is a German word row. "casa" is a Spanish word row. They are separate
entries, linked through the translations table. This is what enables
any language pair as source/target.
### 3.3 Table: `senses`
One row per distinct meaning of a word. This is where polysemy is
handled: "bank" (financial institution) and "bank" (river edge) are
two senses of one word.
| Column | Type | Constraints | Notes |
| ----------- | ----------- | -------------------------------- | -------------------------------------------- |
| id | UUID | PK, default random | |
| word_id | UUID | FK → words.id, ON DELETE CASCADE | |
| sense_index | SMALLINT | NOT NULL, default 0 | 0 = primary meaning, 1 = secondary, etc. |
| difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | "easy", "medium", "hard" |
| cefr_level | VARCHAR(2) | nullable, CHECK in allowed list | "A1","A2","B1","B2","C1","C2" |
| definitions | TEXT[] | NOT NULL, default '{}' | 1–3 definitions in the word's language |
| examples | TEXT[] | NOT NULL, default '{}' | 1–3 example sentences in the word's language |
| created_at | TIMESTAMPTZ | NOT NULL, default now() | |
**Constraints:**
- `UNIQUE (word_id, sense_index)` — one sense per index per word.
- `CHECK (difficulty IN ('easy','medium','hard'))`
- `CHECK (cefr_level IS NULL OR cefr_level IN ('A1','A2','B1','B2','C1','C2'))`
**CEFR → difficulty mapping:**
- A1, A2 → easy
- B1, B2 → medium
- C1, C2 → hard
**Index:**
- `idx_senses_word_diff ON (word_id, difficulty)` — accelerates the
join from words and the difficulty filter.
**Design note — definitions and examples as arrays:**
Definitions and examples are stored as `TEXT[]` arrays on the sense
row rather than in separate tables. Rationale:
- Each sense has at most 2–3 definitions/examples (1-to-few).
- They are always fetched together with the sense (no independent
querying needed).
- Separate tables would add 2 JOINs to the hottest query for no
practical benefit.
- The exercise generator picks one definition and one example
randomly in application code:
`arr[Math.floor(Math.random() * arr.length)]`.
### 3.4 Table: `translations`
One row per translation of a sense into another language. A single
sense can have multiple translations into the same language at
different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium).
| Column | Type | Constraints | Notes |
| -------------------- | ----------- | --------------------------------- | ------------------------------------- |
| id | UUID | PK, default random | |
| sense_id | UUID | FK → senses.id, ON DELETE CASCADE | |
| target_language_code | VARCHAR(10) | NOT NULL, CHECK in supported list | Language of the translation |
| translation | TEXT | NOT NULL | "casa", "Haus", "maison" |
| gender | VARCHAR(20) | nullable | "masculine","feminine","neuter", NULL |
| difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | Can differ from sense difficulty |
| created_at | TIMESTAMPTZ | NOT NULL, default now() | |
**Constraints:**
- `UNIQUE (sense_id, target_language_code, translation)` — allows
multiple translations per language (synonyms) but no exact
duplicates.
- `CHECK (target_language_code IN ('en','de','it','fr','es'))`
- `CHECK (difficulty IN ('easy','medium','hard'))`
- `CHECK (gender IS NULL OR gender IN ('masculine','feminine','neuter'))`
**Index:**
- `idx_translations_sense_lang_diff ON (sense_id, target_language_code, difficulty)`
— accelerates the join from senses and the language/difficulty filter.
**Design note — gender as a real column:**
Gender is stored as a column, not embedded in a JSON blob, because
future gender exercises will need to filter and group by gender.
English nouns have `gender = NULL`.
### 3.5 Future Table: `inflection_forms` (not yet implemented)
Will be added when verb conjugation and adjective declension exercises
are built.
| Column | Type | Constraints | Notes |
| ------- | ----- | -------------------------------- | ------------------------------------- |
| id | UUID | PK, default random | |
| word_id | UUID | FK → words.id, ON DELETE CASCADE | |
| form | TEXT | NOT NULL | "Häuser", "ginge", "grüner" |
| tags | JSONB | NOT NULL, default '{}' | {"case":"genitive","number":"plural"} |
This table is the primary reason the schema is normalized rather than
using JSONB documents. A German verb has 30–50 inflected forms. An
adjective has 12+. These are one-to-many relationships that require
their own table with a foreign key.
---
## 4. Difficulty Model
There are two difficulty columns. They answer different questions.
### `senses.difficulty` — "Is this meaning appropriate for the level?"
Controls which _meanings_ of a word are shown. A beginner should not
be tested on "Haus = noble dynasty" because the concept itself is
advanced, regardless of how hard the translation word is.
### `translations.difficulty` — "Is this word an appropriate answer?"
Controls which _translation word_ is the correct answer. The concept
of "bank" is easy, but "Geldinstitut" is a harder word than "Bank"
for that same concept.
### Query filter logic: sense as ceiling, translation as target
```sql
WHERE s.difficulty IN ('easy', 'medium') -- sense at or below level
AND t.difficulty = 'medium' -- translation exactly at level
```
- The sense difficulty acts as a **ceiling**: don't show meanings
harder than the selected level.
- The translation difficulty acts as the **target**: test the learner
on a word at exactly this level.
This ensures that easy concepts with medium-level synonyms (e.g.,
"bank" → "Geldinstitut") are reachable at the medium level, while
advanced concepts (e.g., "Haus" → "dynasty") remain gated.
### Example data
| Word | Sense | Sense Diff. | Translation | Trans. Diff. |
| ---- | --------------- | ----------- | ------------------- | ------------ |
| Haus | building | easy | casa (es) | easy |
| Haus | noble dynasty | hard | dinastía (es) | hard |
| bank | financial inst. | easy | Bank (de) | easy |
| bank | financial inst. | easy | Geldinstitut (de) | medium |
| bank | financial inst. | easy | Kreditinstitut (de) | hard |
| bank | river edge | medium | Ufer (de) | medium |
At **medium** level, the query returns: Geldinstitut, Ufer.
At **hard** level: Kreditinstitut, dinastía.
"Bank" (easy translation) never appears at medium/hard as a correct
answer. "dinastía" (hard sense) never appears at easy/medium.
---
## 5. Query Patterns
### 5.1 Game query — get N random words
Scenario: user picks German → Spanish, nouns, medium, 20 rounds.
```sql
SELECT
w.id AS word_id,
w.headword,
s.id AS sense_id,
s.definitions,
s.examples,
t.translation,
t.gender
FROM words w
INNER JOIN senses s
ON s.word_id = w.id
INNER JOIN translations t
ON t.sense_id = s.id
WHERE w.language_code = 'de'
AND w.pos = 'noun'
AND s.difficulty IN ('easy', 'medium')
AND t.target_language_code = 'es'
AND t.difficulty = 'medium'
ORDER BY RANDOM()
LIMIT 20;
```
The application then picks one random definition and one random
example from the arrays for each word.
### 5.2 Distractor query — get 3 wrong answers
For a given correct answer, fetch 3 distractors from the same
language, pos, and difficulty pool.
```sql
SELECT t.translation, t.gender
FROM translations t
INNER JOIN senses s
ON t.sense_id = s.id
INNER JOIN words w
ON s.word_id = w.id
WHERE w.language_code = 'de'
AND w.pos = 'noun'
AND s.difficulty IN ('easy', 'medium')
AND t.target_language_code = 'es'
AND t.difficulty = 'medium'
AND t.sense_id != :current_sense_id
AND t.translation != :correct_answer
ORDER BY RANDOM()
LIMIT 3;
```
### 5.3 Distractor exclusion rule
**Distractors must come from a different sense than the correct
answer.** Not just a different word — a different sense.
Rationale: multiple translations of the same sense are all valid
answers. "Bank" and "Geldinstitut" are both correct translations of
the financial-institution sense. Showing one as a distractor for the
other would confuse the learner and break trust.
The `sense_id != :current_sense_id` filter excludes all synonyms of
the same sense in one condition. No synonym table needed.
Translations from a _different_ sense of the same word are valid
distractors (e.g., "Ufer" from the river-bank sense is a fine
distractor for the financial-institution sense — the definition makes
it clearly wrong).
### 5.4 Edge case: identical translation text across senses
Two different senses of different words may share the same translation
text (e.g., "Schloss" = castle and "Schloss" = lock). The
`t.translation != :correct_answer` filter handles this by excluding
the exact text regardless of sense.
---
## 6. Data Pipeline
### 6.1 Flow
```
Word frequency lists (per language, per POS)
│
▼
Gemini API (batches of 20 words)
│
▼
Validation script (reject/flag bad entries)
│
▼
SQLite staging database (local file)
│
▼
Import script (SQLite → Postgres, batch inserts)
│
▼
Postgres (dev) → test full game flow
│
▼
Postgres (prod) via Drizzle migration
```
### 6.2 Wordlist source
Frequency-based word lists, one per language. Example: "1000 most
common German nouns." Sources: Leipzig Corpora, Wiktionary frequency
lists, or similar open-source frequency data.
Each language is processed independently. This ensures language-native
definitions and examples (a German word gets a German definition, not
a translated English one).
### 6.3 Gemini output JSON contract
The API is prompted to return an array of objects. Expected shape:
```json
[
{
"headword": "Haus",
"language": "de",
"pos": "noun",
"senses": [
{
"sense_index": 0,
"cefr_level": "A1",
"difficulty": "easy",
"definitions": ["Ein Gebäude zum Wohnen."],
"examples": ["Sie kauften ein Haus in der Stadt."],
"translations": [
{
"target_language": "en",
"word": "house",
"gender": null,
"difficulty": "easy"
},
{
"target_language": "es",
"word": "casa",
"gender": "feminine",
"difficulty": "easy"
},
{
"target_language": "fr",
"word": "maison",
"gender": "feminine",
"difficulty": "easy"
},
{
"target_language": "it",
"word": "casa",
"gender": "feminine",
"difficulty": "easy"
}
]
},
{
"sense_index": 1,
"cefr_level": "C1",
"difficulty": "hard",
"definitions": ["Ein Adelsgeschlecht, eine Dynastie."],
"examples": ["Das Haus der Merowinger herrschte über Franken."],
"translations": [
{
"target_language": "en",
"word": "house",
"gender": null,
"difficulty": "hard"
},
{
"target_language": "es",
"word": "dinastía",
"gender": "feminine",
"difficulty": "hard"
}
]
}
]
}
]
```
### 6.4 Validation rules
Before writing to SQLite, every entry is checked:
- `headword`, `language`, `pos` are present and valid.
- At least one sense per word.
- Each sense has at least one definition and one example.
- `difficulty` is one of: `easy`, `medium`, `hard`.
- `cefr_level` is one of: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`.
- CEFR → difficulty mapping is consistent.
- `gender` is valid for the target language:
- German: masculine, feminine, neuter
- French, Spanish, Italian: masculine, feminine
- English: null
- Translations exist for at least the 4 other supported languages.
- No duplicate entries (headword + language + pos + sense_index).
Invalid entries are logged and excluded. They can be reviewed and
re-processed manually.
### 6.5 Import: SQLite → Postgres
A Node.js script reads from SQLite (via `better-sqlite3`) and
batch-inserts into Postgres (via Drizzle). Inserts are wrapped in
transactions per batch (20 words) for atomicity.
The application (dev and prod) always reads from Postgres. SQLite is
used only as a pipeline staging file.
---
## 7. Indexes
Three indexes cover the game and distractor queries:
```sql
CREATE INDEX idx_words_lang_pos
ON words (language_code, pos);
CREATE INDEX idx_senses_word_diff
ON senses (word_id, difficulty);
CREATE INDEX idx_translations_sense_lang_diff
ON translations (sense_id, target_language_code, difficulty);
```
---
## 8. Performance
### Estimated data volume at target scale
| Table | Rows | Derivation |
| ------------ | ----- | -------------------------------- |
| words | 500k | ~100k per language × 5 languages |
| senses | ~750k | ~1.5 senses per word average |
| translations | ~3M | ~4 translations per sense |
### Query performance
| Scale | Game query (LIMIT 20) | Distractor query (LIMIT 3) |
| ---------- | --------------------- | -------------------------- |
| 10k words | < 10 ms | < 10 ms |
| 500k words | 20–80 ms | 15–60 ms |
| 5M words | 100–300 ms | 80–200 ms |
The bottleneck at scale is `ORDER BY RANDOM()`, which sorts the
entire filtered result set before applying LIMIT. At 500k words, the
filtered set per query is ~5k–15k rows — well within comfortable
range.
**Future optimization** (if filtered sets exceed ~100k rows):
```sql
WHERE ... AND random() < 0.05 -- pre-filter to ~5% of rows
ORDER BY RANDOM()
LIMIT 20;
```
Or use `TABLESAMPLE`. Not needed at current scale.
### Import performance
| Method | Time for 3.5M rows |
| ---------------------------- | ------------------ |
| Individual INSERT | ~30–60 min |
| Batch INSERT (1000 per stmt) | ~2–5 min |
| Postgres COPY (CSV) | ~10–30 sec |
The pipeline uses batch inserts via Drizzle. The full import is a
one-time or occasional operation.
---
## 9. Implementation Plan
```
1. ✅ Design doc (this document)
2. Acquire word frequency lists (5 languages, nouns first)
3. Build + test Gemini prompt (5 words → 20 words → full batch)
4. Validation script (Gemini output → clean JSON)
5. SQLite staging schema + pipeline write
6. Import script (SQLite → local Postgres)
7. Drizzle schema: words, senses, translations + indexes
8. Drizzle migration on local Postgres
9. Dev branch: new queries (game + distractor), full game flow test
10. Drizzle migration on prod Postgres + data import + verify
11. Extend pipeline: verbs, adjectives, adverbs (schema unchanged)
12. Later: inflection_forms table + conjugation/declension exercises
```
---
## 10. Future Extensions
### Verb conjugation / adjective declension exercises
The `inflection_forms` table (section 3.5) will store inflected forms
with grammatical tags as JSONB. The schema is normalized specifically
to support this: one word → many forms, each independently queryable.
### Gender exercises
The `gender` column on `translations` enables filtering and grouping
by grammatical gender for dedicated gender practice rounds.
### Additional POS
The `pos` column already supports noun, verb, adjective, adverb.
Adding a new POS requires no schema change — only a new wordlist and
an adjusted Gemini prompt.
---
## 11. Key Design Decisions — Summary
| Decision | Choice | Rationale |
| ----------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------------ |
| Database | Postgres + SQLite staging | Relational data, FK integrity, SQL query pattern, already in stack |
| Schema structure | 3 normalized tables | Matches query pattern, supports any language pair, extensible for inflections |
| Definitions / examples | `TEXT[]` arrays on `senses` | 1-to-few relationship, always fetched with sense, avoids 2 extra JOINs |
| Gender | Column on `translations` | Needed for filtering in future gender exercises |
| Difficulty | Two columns: `senses.difficulty` + `translations.difficulty` | Sense = concept gate, translation = word-level target |
| Difficulty filter logic | Sense as ceiling, translation as exact match | Ensures easy concepts with hard synonyms are reachable; advanced concepts stay gated |
| Distractor exclusion | `sense_id != current` | Prevents valid synonyms from appearing as wrong answers |
| Language direction | Any of 5 languages as source or target | Each language has its own headword entries; translations link them |
| Pipeline | Gemini → validate → SQLite → Postgres | Batch-generated data, LLM output needs validation, SQLite for staging simplicity |
| Table-per-language/pos | Rejected | Anti-pattern: 40+ tables, exponential maintenance |
| Single JSONB blob | Rejected | Cannot support inflection tables, cannot index gender, no FK integrity |

704
documentation/pipeline/roadmap.md Normal file
View file

@ -0,0 +1,704 @@
# Vocabulary Trainer — Roadmap: Data Pipeline & Schema Migration
> **Objective:** Replace the OpenWordNet/kaikki data source with a
> Gemini-powered pipeline that generates high-quality vocabulary data,
> backed by a normalized Postgres schema.
>
> **Author:** [Your Name]
> **Date:** July 2026
> **Companion doc:** `docs/schema-design.md`
---
## How to Read This Document
This roadmap has three zoom levels:
- **Part 1 — Overview:** The phases at a glance. Read this to understand the full scope in 30 seconds.
- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and acceptance criteria per phase. Read this to plan your week.
- **Part 3 — Detailed Tasks:** Step-by-step instructions within each phase. Read this when you sit down to code.
---
# Part 1 — High-Level Overview
```
Phase 0 Preparation
Get wordlists, set up tooling, finalize the Gemini prompt.
Phase 1 Data Pipeline
Build the Gemini → validate → SQLite pipeline.
Produce a clean dataset of ~1000 nouns × 5 languages.
Phase 2 Schema & Migration
Create the new Drizzle schema (words, senses, translations).
Write the SQLite → Postgres import script.
Migrate the local dev database.
Phase 3 App Integration
Rewrite the game and distractor queries against the new schema.
Update the exercise-generation logic.
Test the full game flow in dev.
Phase 4 Production Deploy
Run the Drizzle migration on prod.
Import the dataset.
Verify the live app works end-to-end.
Phase 5 Extend POS
Run the pipeline for verbs, adjectives, adverbs.
No schema changes needed — new wordlists + adjusted prompts.
Phase 6 Future Features (out of scope for now)
Inflection tables, conjugation/declension exercises,
gender exercises, spaced-repetition scheduling.
```
**Dependency chain:**
```
Phase 0 → Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5
│
▼
Phase 6
```
Each phase depends on the previous one being complete. Do not skip
ahead.
---
# Part 2 — Phase Breakdown
---
## Phase 0: Preparation
**Goal:** Have everything you need before writing pipeline code.
**Tasks:**
- [ ] Acquire frequency-based noun lists for all 5 languages
- [ ] Clean and format the lists (one word per line, UTF-8)
- [ ] Set up local Postgres (Docker or native)
- [ ] Set up a SQLite database file for staging
- [ ] Write and test the Gemini prompt with 5 sample words
- [ ] Refine the prompt until the JSON output matches the contract
defined in `docs/schema-design.md` §6.3
**Dependencies:** None.
**Acceptance criteria:**
- You have 5 wordlist files in `data/wordlists/` (one per language).
- A Gemini call with 5 German nouns returns valid JSON matching the
contract, including definitions, examples, translations with gender,
and difficulty levels.
- Local Postgres is running and reachable from your app.
---
## Phase 1: Data Pipeline
**Goal:** A repeatable script that takes a wordlist, calls Gemini in
batches of 20, validates the output, and writes clean rows to SQLite.
**Tasks:**
- [ ] Write the validation module (see schema-design §6.4)
- [ ] Write the pipeline script: - Read wordlist file - Split into batches of 20 - Call Gemini API per batch - Parse JSON response - Validate each entry - Write valid entries to SQLite - Log invalid entries to a rejection file
- [ ] Create the SQLite schema (mirrors the Postgres schema)
- [ ] Run the pipeline for all 5 languages (nouns only)
- [ ] Review the rejection log, fix prompt issues, re-run failed batches
- [ ] Spot-check 50 random entries for correctness
**Dependencies:** Phase 0 complete.
**Acceptance criteria:**
- SQLite database contains ~1000 nouns × 5 languages with senses and
translations.
- Rejection rate is below 10%.
- Spot-checked entries have correct definitions, plausible examples,
correct genders, and reasonable difficulty levels.
- The pipeline is re-runnable (idempotent or with duplicate handling).
---
## Phase 2: Schema & Migration
**Goal:** The new normalized schema exists in Drizzle, the local
Postgres database is migrated, and the SQLite data is imported.
**Tasks:**
- [ ] Write the new Drizzle schema file: - `words` table (headword, language_code, pos) - `senses` table (word_id FK, sense_index, difficulty,
cefr_level, definitions TEXT[], examples TEXT[]) - `translations` table (sense_id FK, target_language_code,
translation, gender, difficulty) - All CHECK constraints, UNIQUE constraints, indexes
- [ ] Generate the Drizzle migration (`npx drizzle-kit generate`)
- [ ] Inspect the generated SQL file — verify it creates the right
tables, constraints, and indexes
- [ ] Apply the migration to local Postgres (`npx drizzle-kit migrate`)
- [ ] Write the import script (SQLite → Postgres): - Read all rows from SQLite - Insert into Postgres in dependency order:
words → senses → translations - Use batch inserts (not row-by-row) - Wrap in transactions (per batch of 20 words) - Handle duplicates gracefully (skip or upsert)
- [ ] Run the import script
- [ ] Verify row counts match between SQLite and Postgres
- [ ] Run 3–5 manual SQL queries against Postgres to sanity-check
the data
**Dependencies:** Phase 1 complete (SQLite has data).
**Acceptance criteria:**
- `npx drizzle-kit migrate` runs without errors on local Postgres.
- Import script completes without errors.
- Row counts in Postgres match SQLite (±rejection count).
- Manual query: "Give me 5 random German nouns with Spanish
translations at easy difficulty" returns sensible results.
---
## Phase 3: App Integration
**Goal:** The running app uses the new schema. Game rounds and
distractors work correctly for all language pairs.
**Tasks:**
- [ ] Rewrite `getGameTerms` query: - JOIN words → senses → translations - Filter: source language, pos, sense difficulty (ceiling),
target language, translation difficulty (exact) - ORDER BY RANDOM(), LIMIT rounds
- [ ] Rewrite `getDistractors` query: - Same JOINs and filters - Exclude: `sense_id != current`, `translation != correct` - ORDER BY RANDOM(), LIMIT 3
- [ ] Update the exercise-generation logic: - Pick one random definition from the `definitions` array - Pick one random example from the `examples` array - Assemble the 4 answer options (1 correct + 3 distractors) - Shuffle the options
- [ ] Remove or deprecate old schema references
(old `vocabulary_entries`, `entry_translations` tables)
- [ ] Test manually: - German → Spanish, nouns, easy, 10 rounds - Spanish → German, nouns, medium, 10 rounds - English → French, nouns, hard, 10 rounds - Italian → German, nouns, easy, 10 rounds - Verify: no duplicate answers, no same-sense synonyms as
distractors, definitions and examples are in the source
language, genders display correctly
- [ ] Test edge cases: - A difficulty/pos/language combo with very few words
(does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?)
**Dependencies:** Phase 2 complete (Postgres has data, schema exists).
**Acceptance criteria:**
- Full game flow works in dev for at least 4 different language pairs.
- No same-sense synonyms appear as distractors.
- Definitions and examples are in the correct language.
- Gender is displayed where applicable.
- No console errors or unhandled query failures.
---
## Phase 4: Production Deploy
**Goal:** The live app runs on the new schema with the new data.
**Tasks:**
- [ ] Back up the production database
- [ ] Run the Drizzle migration on prod
- [ ] Run the import script against prod Postgres
- [ ] Verify row counts on prod
- [ ] Test the live app: - Play 2 full games on the deployed app - Check different language pairs and difficulties
- [ ] Monitor for errors (server logs, browser console) for 24h
- [ ] Remove old tables from prod (after confirming everything works)
**Dependencies:** Phase 3 complete (dev is fully working).
**Acceptance criteria:**
- Live app serves game rounds from the new schema.
- No errors in server logs for 24 hours.
- Old tables are dropped (or scheduled for removal).
- Rollback plan exists: the prod backup can be restored if needed.
---
## Phase 5: Extend POS
**Goal:** Verbs, adjectives, and adverbs are in the database and
usable in the app.
**Tasks:**
- [ ] Acquire frequency lists for verbs, adjectives, adverbs
(all 5 languages)
- [ ] Adjust the Gemini prompt per POS: - Verbs: may need different metadata (transitivity, etc.) - Adjectives: may need base form info - Adverbs: typically simpler metadata
- [ ] Run the pipeline for each POS
- [ ] Validate, import to Postgres (dev → prod)
- [ ] Test game flow with verbs, adjectives, adverbs
- [ ] Verify the POS filter in the app UI works for all types
**Dependencies:** Phase 4 complete.
**Acceptance criteria:**
- All 4 POS types are playable in the app.
- Pipeline is repeatable for future data additions.
---
## Phase 6: Future Features (out of scope)
Listed here for visibility. Not planned, not estimated.
- [ ] `inflection_forms` table + conjugation exercises (verbs)
- [ ] Adjective declension exercises (der grüne Mann, grüner Mann…)
- [ ] Gender exercises (pick the correct article)
- [ ] Spaced-repetition scheduling (track which words the user knows)
- [ ] User accounts and progress persistence
- [ ] Additional languages (if ever)
---
# Part 3 — Detailed Task Breakdown
---
## Phase 0: Preparation — Detailed
### 0.1 Acquire wordlists
- Search for frequency lists. Good starting points:
- "Leipzig Corpora Collection" (academic, per-language)
- Wiktionary frequency lists
- GitHub repos: search "german noun frequency list",
"spanish noun frequency list", etc.
- Tatoeba sentence counts as a proxy for word frequency
- Target: ~1000 nouns per language, sorted by frequency.
- Format: plain text, one word per line, UTF-8, no headers.
- Save to `data/wordlists/nouns_de.txt`, `nouns_es.txt`, etc.
- Clean the lists: remove duplicates, remove words with spaces
(multi-word expressions), remove proper nouns if desired.
### 0.2 Set up local Postgres
- Option A (Docker):
```
docker run --name vocab-dev \
-e POSTGRES_USER=dev \
-e POSTGRES_PASSWORD=dev \
-e POSTGRES_DB=vocab \
-p 5432:5432 \
-d postgres:16
```
- Option B (native install): install Postgres, create a `vocab`
database.
- Update your `.env` / `.env.local` with the connection string.
- Verify: connect with `psql` or a GUI client (TablePlus, DBeaver,
pgAdmin). Run `SELECT 1;`.
### 0.3 Write and test the Gemini prompt
- Start with 5 German nouns.
- The prompt should specify:
- The exact JSON structure expected (copy from schema-design §6.3)
- That definitions and examples must be in the word's language
- That translations are needed for all 4 other supported languages
- That gender must be provided for de/fr/es/it, null for en
- That difficulty must be one of: easy, medium, hard
- That CEFR level must be one of: A1, A2, B1, B2, C1, C2
- That multiple senses should be included for polysemous words
- Call the API. Inspect the JSON. Common issues to fix:
- Gemini wraps the JSON in markdown code fences → strip them
- Gemini returns "intermediate" instead of "medium" → normalize
- Gemini omits a language → re-prompt or reject
- Gemini returns gender "common" for German → reject
- Iterate until 3 consecutive batches of 20 return clean JSON.
---
## Phase 1: Data Pipeline — Detailed
### 1.1 Create the SQLite schema
- Create a file `pipeline/schema.sql` or define it in your pipeline
script.
- Tables mirror the Postgres schema:
```sql
CREATE TABLE words (
id TEXT PRIMARY KEY,
headword TEXT NOT NULL,
language_code TEXT NOT NULL,
pos TEXT NOT NULL,
created_at TEXT DEFAULT (datetime('now')),
UNIQUE(headword, language_code, pos)
);
CREATE TABLE senses (
id TEXT PRIMARY KEY,
word_id TEXT NOT NULL REFERENCES words(id),
sense_index INTEGER NOT NULL DEFAULT 0,
difficulty TEXT NOT NULL,
cefr_level TEXT,
definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text
examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text
created_at TEXT DEFAULT (datetime('now')),
UNIQUE(word_id, sense_index)
);
CREATE TABLE translations (
id TEXT PRIMARY KEY,
sense_id TEXT NOT NULL REFERENCES senses(id),
target_language_code TEXT NOT NULL,
translation TEXT NOT NULL,
gender TEXT,
difficulty TEXT NOT NULL,
created_at TEXT DEFAULT (datetime('now')),
UNIQUE(sense_id, target_language_code, translation)
);
```
- Note: SQLite has no native TEXT[]. Store arrays as JSON text.
The import script will parse them into Postgres TEXT[] on import.
### 1.2 Write the validation module
- Create `pipeline/validate.ts` (or .js).
- Input: one parsed Gemini entry (the JSON object for one word).
- Checks (return a list of errors, empty = valid):
- `headword` is a non-empty string
- `language` is in ['en','de','it','fr','es']
- `pos` is in ['noun','verb','adjective','adverb']
- `senses` is a non-empty array
- Each sense has:
- `sense_index` is a non-negative integer
- `difficulty` is in ['easy','medium','hard']
- `cefr_level` is in ['A1','A2','B1','B2','C1','C2'] or null
- CEFR ↔ difficulty mapping is consistent
- `definitions` is a non-empty array of non-empty strings
- `examples` is a non-empty array of non-empty strings
- `translations` is a non-empty array
- Each translation has:
- `target_language` is in the supported list
- `target_language` != the word's own language
- `word` is a non-empty string
- `gender` is valid for the target language
(de: masculine/feminine/neuter/null,
fr/es/it: masculine/feminine/null,
en: null)
- `difficulty` is in ['easy','medium','hard']
- Output: `{ valid: boolean, errors: string[] }`
### 1.3 Write the pipeline script
- Create `pipeline/run.ts`.
- Pseudocode:
```
for each language in [de, en, es, fr, it]:
read wordlist file → array of words
split into batches of 20
for each batch:
call Gemini API with the batch
parse JSON response (strip markdown fences if present)
for each entry in response:
validate(entry)
if valid:
generate UUIDs for word, senses, translations
INSERT into SQLite (words, senses, translations)
else:
append to rejection log (data/rejections/{lang}_{pos}.jsonl)
log progress: "Batch 12/50 done. 238 words imported, 2 rejected."
sleep 1s (rate limiting)
```
- Use `better-sqlite3` for SQLite access (synchronous, simple).
- Generate UUIDs with `crypto.randomUUID()`.
- Store definitions/examples as JSON strings in SQLite
(`JSON.stringify(arr)`).
### 1.4 Run and review
- Run the pipeline for all 5 languages.
- Check the rejection log. If rejection rate > 10%, fix the prompt
and re-run failed batches.
- Spot-check: open the SQLite DB, run:
```sql
SELECT w.headword, s.definitions, t.translation, t.gender
FROM words w
JOIN senses s ON s.word_id = w.id
JOIN translations t ON t.sense_id = s.id
WHERE w.language_code = 'de' AND w.pos = 'noun'
ORDER BY RANDOM()
LIMIT 20;
```
- Read the definitions. Are they in German? Do they make sense?
Are the genders correct? Are the difficulties reasonable?
---
## Phase 2: Schema & Migration — Detailed
### 2.1 Write the Drizzle schema
- Create or update `src/db/schema.ts`.
- Define the three tables using Drizzle's `pgTable` builder.
- Include all columns, types, constraints, and indexes as specified
in `docs/schema-design.md` §3.
- Keep the old tables in the file for now (commented out or in a
separate file) so the app doesn't break during development.
### 2.2 Generate and inspect the migration
- Run: `npx drizzle-kit generate`
- Open the generated SQL file in `drizzle/` (or wherever your
config puts it).
- Read it. Verify:
- Three CREATE TABLE statements
- CHECK constraints match your schema
- UNIQUE constraints are present
- Three CREATE INDEX statements
- Foreign keys reference the correct tables with ON DELETE CASCADE
- If something looks wrong, fix the schema file and regenerate.
### 2.3 Apply the migration locally
- Run: `npx drizzle-kit migrate`
- Connect to local Postgres and verify:
```sql
\dt -- list tables
\d words -- describe words table
\d senses -- describe senses table
\d translations -- describe translations table
```
### 2.4 Write the import script
- Create `scripts/import-to-postgres.ts`.
- Pseudocode:
```
open SQLite database (read-only)
connect to Postgres via Drizzle
read all words from SQLite
for each batch of 100 words:
begin transaction
INSERT words into Postgres
for each word:
read its senses from SQLite
INSERT senses into Postgres
for each sense:
read its translations from SQLite
parse definitions/examples from JSON string → TEXT[]
INSERT translations into Postgres
commit transaction
log progress
log final counts: words, senses, translations
```
- Parse `definitions` and `examples` from JSON strings (SQLite)
into actual arrays (Postgres TEXT[]).
- Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully.
### 2.5 Run and verify
- Run the import script.
- Compare counts:
```sql
-- In SQLite
SELECT COUNT(*) FROM words;
SELECT COUNT(*) FROM senses;
SELECT COUNT(*) FROM translations;
-- In Postgres
SELECT COUNT(*) FROM words;
SELECT COUNT(*) FROM senses;
SELECT COUNT(*) FROM translations;
```
- Counts should match (minus any rows that failed validation).
- Run the game query manually in Postgres:
```sql
SELECT w.headword, s.definitions, s.examples,
t.translation, t.gender
FROM words w
JOIN senses s ON s.word_id = w.id
JOIN translations t ON t.sense_id = s.id
WHERE w.language_code = 'de'
AND w.pos = 'noun'
AND s.difficulty IN ('easy', 'medium')
AND t.target_language_code = 'es'
AND t.difficulty = 'medium'
ORDER BY RANDOM()
LIMIT 5;
```
- Verify the results make sense.
---
## Phase 3: App Integration — Detailed
### 3.1 Rewrite getGameTerms
- Open the file containing `getGameTerms`.
- Replace the old query (2-table join on `vocabulary_entries` +
`entry_translations`) with the new 3-table join
(words → senses → translations).
- Parameters: sourceLanguage, targetLanguage, pos, difficulty, rounds.
- Difficulty filter:
- `senses.difficulty IN (all levels up to and including selected)`
— ceiling logic
- `translations.difficulty = selected` — exact match
- Return: word_id, headword, sense_id, definitions, examples,
translation, gender.
### 3.2 Rewrite getDistractors
- Same 3-table join.
- Additional filters:
- `t.sense_id != :currentSenseId`
- `t.translation != :correctAnswer`
- LIMIT 3.
- If fewer than 3 distractors are found (small data pool), handle
gracefully: reduce the number of options or log a warning.
### 3.3 Update exercise generation
- In the function that assembles a game round:
- Pick one random definition:
`definitions[Math.floor(Math.random() * definitions.length)]`
- Pick one random example:
`examples[Math.floor(Math.random() * examples.length)]`
- Combine 1 correct translation + 3 distractors.
- Shuffle the 4 options (Fisher-Yates or similar).
- Attach gender to each option for display.
### 3.4 Test matrix
Run through this matrix manually in the dev app:
| Source | Target | POS | Difficulty | Rounds | Pass? |
| ------ | ------ | ---- | ---------- | ------ | ----- |
| de | es | noun | easy | 10 | |
| es | de | noun | medium | 10 | |
| en | fr | noun | hard | 10 | |
| it | de | noun | easy | 10 | |
| fr | en | noun | medium | 10 | |
For each: verify definitions are in the source language, translations
are in the target language, genders are shown, no same-sense synonyms
appear as distractors, no duplicate options.
---
## Phase 4: Production Deploy — Detailed
### 4.1 Pre-deploy checklist
- [ ] All Phase 3 acceptance criteria pass
- [ ] `git status` is clean, all changes committed
- [ ] The Drizzle migration file is committed to the repo
- [ ] You know your prod database connection string
- [ ] You have a backup method for prod (pg_dump, hosting provider
snapshot, etc.)
### 4.2 Deploy
- Back up prod:
```
pg_dump -h <host> -U <user> -d <db> > backup_$(date +%Y%m%d).sql
```
- Run migration on prod:
```
DATABASE_URL=<prod-url> npx drizzle-kit migrate
```
- Run import script against prod:
```
DATABASE_URL=<prod-url> npx tsx scripts/import-to-postgres.ts
```
- Verify counts on prod.
### 4.3 Post-deploy verification
- Open the live app. Play 2 full games with different settings.
- Check server logs for errors.
- Wait 24h. Check logs again.
- If everything is clean, drop old tables:
```sql
DROP TABLE IF EXISTS entry_translations;
DROP TABLE IF EXISTS vocabulary_entries;
```
(Or keep them for another week if you want a safety net.)
### 4.4 Rollback plan
If something goes wrong:
- Restore the backup:
```
psql -h <host> -U <user> -d <db> < backup_20260722.sql
```
- Revert the code to the previous commit.
- Redeploy.
---
## Phase 5: Extend POS — Detailed
### 5.1 Per POS
Repeat the Phase 1 pipeline for each new POS:
- Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages.
- Adjust the Gemini prompt:
- Verbs: ask for transitivity, common prepositions, or other
verb-specific metadata if needed for future exercises.
- Adjectives: ask for the base form. Note that adjective metadata
differs from noun metadata (no gender on the adjective itself
in the same way — gender applies to the noun it modifies).
- Adverbs: typically simpler. May not need gender at all.
- Run pipeline → validate → SQLite → import → Postgres.
- Test in the app with the POS filter set to the new type.
### 5.2 No schema changes
The `pos` column already supports all four types. The `gender`
column is nullable and simply won't be populated for adverbs or
English words. No migration needed.
---
# Risks & Mitigations
| Risk | Likelihood | Impact | Mitigation |
| ------------------------------------------------------ | ----------------- | ------------------------- | ------------------------------------------------------------------------------------------------- |
| Gemini returns malformed JSON | Medium | Pipeline stalls | Strip markdown fences, wrap parsing in try/catch, log and skip bad batches |
| Gemini produces wrong genders or difficulties | Medium | Bad exercise data | Validation module rejects invalid entries. Spot-check 50+ entries per language |
| Too few words at a given difficulty/pos/language combo | Medium | Game can't fill 4 options | Graceful fallback: reduce options or show a "not enough words" message. Log which combos are thin |
| SQLite → Postgres import fails mid-way | Low | Partial data in Postgres | Transactions per batch. Re-run with ON CONFLICT DO NOTHING |
| ORDER BY RANDOM() gets slow at scale | Low (not at 500k) | Slow game load | Add a TODO comment. Optimize with random() pre-filter when needed |
| Prod migration breaks the live app | Low | Downtime | Backup before migrating. Rollback plan documented. Deploy during low-traffic hours |
---
# File / Folder Structure (new files)
```
project/
├── docs/
│ ├── schema-design.md ← companion design doc
│ └── roadmap.md ← this document
├── data/
│ ├── wordlists/
│ │ ├── nouns_de.txt
│ │ ├── nouns_en.txt
│ │ ├── nouns_es.txt
│ │ ├── nouns_fr.txt
│ │ └── nouns_it.txt
│ ├── rejections/ ← invalid Gemini entries (for review)
│ └── staging.db ← SQLite staging database
├── pipeline/
│ ├── run.ts ← main pipeline script
│ ├── validate.ts ← validation module
│ ├── prompt.ts ← Gemini prompt template
│ └── schema.sql ← SQLite schema definition
├── scripts/
│ └── import-to-postgres.ts ← SQLite → Postgres import
├── src/
│ └── db/
│ └── schema.ts ← Drizzle schema (updated)
└── drizzle/ ← generated migration files
```