updating documentation

documentation/pipeline/design-doc.md

@@ -3,7 +3,7 @@
 > **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline
 > **Author:** [Your Name]
 > **Date:** July 2026
-> **Status:** Approved — ready for implementation
+> **Status:** Approved — schema implemented
 
 ---
 
@@ -83,9 +83,8 @@ staging).
 │ headword    │  └───>│ word_id(FK) │  └───>│ sense_id (FK)    │
 │ language_code│      │ sense_index │       │ target_lang_code │
 │ pos         │       │ difficulty  │       │ translation      │
-│             │       │ cefr_level  │       │ gender           │
-│             │       │ definitions │       │ difficulty       │
-│             │       │ examples    │       │                  │
+│             │       │ definitions │       │ gender           │
+│             │       │ examples    │       │ difficulty       │
 └─────────────┘       └─────────────┘       └──────────────────┘
 
 Future (not yet implemented):
@@ -119,7 +118,7 @@ One row per unique word in a specific language.
 
 **Index:**
 
-- `idx_words_lang_pos ON (language_code, pos)` — accelerates the
+- `idx_language_code_pos ON (language_code, pos)` — accelerates the
   primary game query filter.
 
 **Design note:** Each language gets its own headword entries. "Haus"
@@ -139,7 +138,6 @@ two senses of one word.
 | word_id     | UUID        | FK → words.id, ON DELETE CASCADE |                                              |
 | sense_index | SMALLINT    | NOT NULL, default 0              | 0 = primary meaning, 1 = secondary, etc.     |
 | difficulty  | VARCHAR(20) | NOT NULL, CHECK in allowed list  | "easy", "medium", "hard"                     |
-| cefr_level  | VARCHAR(2)  | nullable, CHECK in allowed list  | "A1","A2","B1","B2","C1","C2"                |
 | definitions | TEXT[]      | NOT NULL, default '{}'           | 1–3 definitions in the word's language       |
 | examples    | TEXT[]      | NOT NULL, default '{}'           | 1–3 example sentences in the word's language |
 | created_at  | TIMESTAMPTZ | NOT NULL, default now()          |                                              |
@@ -148,18 +146,11 @@ two senses of one word.
 
 - `UNIQUE (word_id, sense_index)` — one sense per index per word.
 - `CHECK (difficulty IN ('easy','medium','hard'))`
-- `CHECK (cefr_level IS NULL OR cefr_level IN ('A1','A2','B1','B2','C1','C2'))`
-
-**CEFR → difficulty mapping:**
-
-- A1, A2 → easy
-- B1, B2 → medium
-- C1, C2 → hard
 
 **Index:**
 
-- `idx_senses_word_diff ON (word_id, difficulty)` — accelerates the
-  join from words and the difficulty filter.
+- `idx_word_sense_difficulty ON (word_id, difficulty)` — accelerates
+  the join from words and the difficulty filter.
 
 **Design note — definitions and examples as arrays:**
 Definitions and examples are stored as `TEXT[]` arrays on the sense
@@ -201,7 +192,7 @@ different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium).
 
 **Index:**
 
-- `idx_translations_sense_lang_diff ON (sense_id, target_language_code, difficulty)`
+- `idx_translations_sense_language_difficulty ON (sense_id, target_language_code, difficulty)`
   — accelerates the join from senses and the language/difficulty filter.
 
 **Design note — gender as a real column:**
@@ -409,7 +400,6 @@ The API is prompted to return an array of objects. Expected shape:
     "senses": [
       {
         "sense_index": 0,
-        "cefr_level": "A1",
         "difficulty": "easy",
         "definitions": ["Ein Gebäude zum Wohnen."],
         "examples": ["Sie kauften ein Haus in der Stadt."],
@@ -442,7 +432,6 @@ The API is prompted to return an array of objects. Expected shape:
       },
       {
         "sense_index": 1,
-        "cefr_level": "C1",
         "difficulty": "hard",
         "definitions": ["Ein Adelsgeschlecht, eine Dynastie."],
         "examples": ["Das Haus der Merowinger herrschte über Franken."],
@@ -474,8 +463,6 @@ Before writing to SQLite, every entry is checked:
 - At least one sense per word.
 - Each sense has at least one definition and one example.
 - `difficulty` is one of: `easy`, `medium`, `hard`.
-- `cefr_level` is one of: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`.
-- CEFR → difficulty mapping is consistent.
 - `gender` is valid for the target language:
   - German: masculine, feminine, neuter
   - French, Spanish, Italian: masculine, feminine
@@ -502,13 +489,13 @@ used only as a pipeline staging file.
 Three indexes cover the game and distractor queries:
 
 ```sql
-CREATE INDEX idx_words_lang_pos
+CREATE INDEX idx_language_code_pos
   ON words (language_code, pos);
 
-CREATE INDEX idx_senses_word_diff
+CREATE INDEX idx_word_sense_difficulty
   ON senses (word_id, difficulty);
 
-CREATE INDEX idx_translations_sense_lang_diff
+CREATE INDEX idx_translations_sense_language_difficulty
   ON translations (sense_id, target_language_code, difficulty);
 ```
 
@@ -563,13 +550,13 @@ one-time or occasional operation.
 ## 9. Implementation Plan
 
 ```
- 1. ✅ Design doc (this document)
- 2.    Acquire word frequency lists (5 languages, nouns first)
- 3.    Build + test Gemini prompt (5 words → 20 words → full batch)
- 4.    Validation script (Gemini output → clean JSON)
- 5.    SQLite staging schema + pipeline write
- 6.    Import script (SQLite → local Postgres)
- 7.    Drizzle schema: words, senses, translations + indexes
+ 1. ✅ Schema design (this document)
+ 2. ✅ Drizzle schema: words, senses, translations, relations, indexes
+ 3.    Acquire word frequency lists (5 languages, nouns first)
+ 4.    Build + test Gemini prompt (5 words → 20 words → full batch)
+ 5.    Validation script (Gemini output → clean JSON)
+ 6.    SQLite staging schema + pipeline write
+ 7.    Import script (SQLite → local Postgres)
  8.    Drizzle migration on local Postgres
  9.    Dev branch: new queries (game + distractor), full game flow test
 10.    Drizzle migration on prod Postgres + data import + verify