diff --git a/documentation/pipeline/design-doc.md b/documentation/pipeline/design-doc.md index a1d7666..255c04e 100644 --- a/documentation/pipeline/design-doc.md +++ b/documentation/pipeline/design-doc.md @@ -3,7 +3,7 @@ > **Project:** PERN-stack vocabulary trainer with Gemini-powered data pipeline > **Author:** [Your Name] > **Date:** July 2026 -> **Status:** Approved — ready for implementation +> **Status:** Approved — schema implemented --- @@ -83,9 +83,8 @@ staging). │ headword │ └───>│ word_id(FK) │ └───>│ sense_id (FK) │ │ language_code│ │ sense_index │ │ target_lang_code │ │ pos │ │ difficulty │ │ translation │ -│ │ │ cefr_level │ │ gender │ -│ │ │ definitions │ │ difficulty │ -│ │ │ examples │ │ │ +│ │ │ definitions │ │ gender │ +│ │ │ examples │ │ difficulty │ └─────────────┘ └─────────────┘ └──────────────────┘ Future (not yet implemented): @@ -119,7 +118,7 @@ One row per unique word in a specific language. **Index:** -- `idx_words_lang_pos ON (language_code, pos)` — accelerates the +- `idx_language_code_pos ON (language_code, pos)` — accelerates the primary game query filter. **Design note:** Each language gets its own headword entries. "Haus" @@ -139,7 +138,6 @@ two senses of one word. | word_id | UUID | FK → words.id, ON DELETE CASCADE | | | sense_index | SMALLINT | NOT NULL, default 0 | 0 = primary meaning, 1 = secondary, etc. | | difficulty | VARCHAR(20) | NOT NULL, CHECK in allowed list | "easy", "medium", "hard" | -| cefr_level | VARCHAR(2) | nullable, CHECK in allowed list | "A1","A2","B1","B2","C1","C2" | | definitions | TEXT[] | NOT NULL, default '{}' | 1–3 definitions in the word's language | | examples | TEXT[] | NOT NULL, default '{}' | 1–3 example sentences in the word's language | | created_at | TIMESTAMPTZ | NOT NULL, default now() | | @@ -148,18 +146,11 @@ two senses of one word. - `UNIQUE (word_id, sense_index)` — one sense per index per word. - `CHECK (difficulty IN ('easy','medium','hard'))` -- `CHECK (cefr_level IS NULL OR cefr_level IN ('A1','A2','B1','B2','C1','C2'))` - -**CEFR → difficulty mapping:** - -- A1, A2 → easy -- B1, B2 → medium -- C1, C2 → hard **Index:** -- `idx_senses_word_diff ON (word_id, difficulty)` — accelerates the - join from words and the difficulty filter. +- `idx_word_sense_difficulty ON (word_id, difficulty)` — accelerates + the join from words and the difficulty filter. **Design note — definitions and examples as arrays:** Definitions and examples are stored as `TEXT[]` arrays on the sense @@ -201,7 +192,7 @@ different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium). **Index:** -- `idx_translations_sense_lang_diff ON (sense_id, target_language_code, difficulty)` +- `idx_translations_sense_language_difficulty ON (sense_id, target_language_code, difficulty)` — accelerates the join from senses and the language/difficulty filter. **Design note — gender as a real column:** @@ -409,7 +400,6 @@ The API is prompted to return an array of objects. Expected shape: "senses": [ { "sense_index": 0, - "cefr_level": "A1", "difficulty": "easy", "definitions": ["Ein Gebäude zum Wohnen."], "examples": ["Sie kauften ein Haus in der Stadt."], @@ -442,7 +432,6 @@ The API is prompted to return an array of objects. Expected shape: }, { "sense_index": 1, - "cefr_level": "C1", "difficulty": "hard", "definitions": ["Ein Adelsgeschlecht, eine Dynastie."], "examples": ["Das Haus der Merowinger herrschte über Franken."], @@ -474,8 +463,6 @@ Before writing to SQLite, every entry is checked: - At least one sense per word. - Each sense has at least one definition and one example. - `difficulty` is one of: `easy`, `medium`, `hard`. -- `cefr_level` is one of: `A1`, `A2`, `B1`, `B2`, `C1`, `C2`. -- CEFR → difficulty mapping is consistent. - `gender` is valid for the target language: - German: masculine, feminine, neuter - French, Spanish, Italian: masculine, feminine @@ -502,13 +489,13 @@ used only as a pipeline staging file. Three indexes cover the game and distractor queries: ```sql -CREATE INDEX idx_words_lang_pos +CREATE INDEX idx_language_code_pos ON words (language_code, pos); -CREATE INDEX idx_senses_word_diff +CREATE INDEX idx_word_sense_difficulty ON senses (word_id, difficulty); -CREATE INDEX idx_translations_sense_lang_diff +CREATE INDEX idx_translations_sense_language_difficulty ON translations (sense_id, target_language_code, difficulty); ``` @@ -563,13 +550,13 @@ one-time or occasional operation. ## 9. Implementation Plan ``` - 1. ✅ Design doc (this document) - 2. Acquire word frequency lists (5 languages, nouns first) - 3. Build + test Gemini prompt (5 words → 20 words → full batch) - 4. Validation script (Gemini output → clean JSON) - 5. SQLite staging schema + pipeline write - 6. Import script (SQLite → local Postgres) - 7. Drizzle schema: words, senses, translations + indexes + 1. ✅ Schema design (this document) + 2. ✅ Drizzle schema: words, senses, translations, relations, indexes + 3. Acquire word frequency lists (5 languages, nouns first) + 4. Build + test Gemini prompt (5 words → 20 words → full batch) + 5. Validation script (Gemini output → clean JSON) + 6. SQLite staging schema + pipeline write + 7. Import script (SQLite → local Postgres) 8. Drizzle migration on local Postgres 9. Dev branch: new queries (game + distractor), full game flow test 10. Drizzle migration on prod Postgres + data import + verify diff --git a/documentation/pipeline/roadmap.md b/documentation/pipeline/roadmap.md index c6baef7..f01bcb5 100644 --- a/documentation/pipeline/roadmap.md +++ b/documentation/pipeline/roadmap.md @@ -14,42 +14,48 @@ This roadmap has three zoom levels: -- **Part 1 — Overview:** The phases at a glance. Read this to understand the full scope in 30 seconds. -- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and acceptance criteria per phase. Read this to plan your week. -- **Part 3 — Detailed Tasks:** Step-by-step instructions within each phase. Read this when you sit down to code. +- **Part 1 — Overview:** The phases at a glance. Read this to + understand the full scope in 30 seconds. +- **Part 2 — Phase Breakdown:** Goals, tasks, dependencies, and + acceptance criteria per phase. Read this to plan your week. +- **Part 3 — Detailed Tasks:** Step-by-step instructions within each + phase. Read this when you sit down to code. --- # Part 1 — High-Level Overview ``` -Phase 0 Preparation +Phase 1 Schema ✅ + New Drizzle schema (words, senses, translations) with + relations, constraints, and indexes. Committed. + +Phase 2 Preparation Get wordlists, set up tooling, finalize the Gemini prompt. -Phase 1 Data Pipeline +Phase 3 Data Pipeline Build the Gemini → validate → SQLite pipeline. Produce a clean dataset of ~1000 nouns × 5 languages. -Phase 2 Schema & Migration - Create the new Drizzle schema (words, senses, translations). +Phase 4 Migration & Import + Generate and apply the Drizzle migration. Write the SQLite → Postgres import script. - Migrate the local dev database. -Phase 3 App Integration +Phase 5 App Integration Rewrite the game and distractor queries against the new schema. Update the exercise-generation logic. Test the full game flow in dev. -Phase 4 Production Deploy +Phase 6 Production Deploy Run the Drizzle migration on prod. Import the dataset. Verify the live app works end-to-end. -Phase 5 Extend POS +Phase 7 Extend POS Run the pipeline for verbs, adjectives, adverbs. No schema changes needed — new wordlists + adjusted prompts. -Phase 6 Future Features (out of scope for now) +Phase 8 Future Features (out of scope for now) Inflection tables, conjugation/declension exercises, gender exercises, spaced-repetition scheduling. ``` @@ -57,22 +63,41 @@ Phase 6 Future Features (out of scope for now) **Dependency chain:** ``` -Phase 0 → Phase 1 → Phase 2 → Phase 3 → Phase 4 → Phase 5 - │ - ▼ - Phase 6 +Phase 1 ✅ → Phase 2 → Phase 3 → Phase 4 → Phase 5 → Phase 6 → Phase 7 + │ + ▼ + Phase 8 ``` -Each phase depends on the previous one being complete. Do not skip -ahead. - --- # Part 2 — Phase Breakdown --- -## Phase 0: Preparation +## Phase 1: Schema ✅ COMPLETE + +**Goal:** New normalized schema exists in Drizzle with all tables, +constraints, indexes, and relations. + +**Completed tasks:** + +- [x] `words` table: headword, language_code, pos, UNIQUE, CHECKs, index +- [x] `senses` table: word_id FK (cascade), sense_index, difficulty, + definitions TEXT[], examples TEXT[], UNIQUE, CHECK, index +- [x] `translations` table: sense_id FK (cascade), target_language_code, + translation, gender (nullable), difficulty, UNIQUE, CHECKs, index +- [x] Relations: words→senses (many), senses→word (one) + translations + (many), translations→sense (one) +- [x] `NOUN_GENDERS` constant added to `@lila/shared` +- [x] `DIFFICULTY_LEVELS` updated: "intermediate" → "medium" +- [x] Old tables (`vocabulary_entries`, `entry_translations`) untouched +- [x] Auth and lobby tables untouched +- [x] Build passes, committed + +--- + +## Phase 2: Preparation **Goal:** Have everything you need before writing pipeline code. @@ -86,11 +111,12 @@ ahead. - [ ] Refine the prompt until the JSON output matches the contract defined in `docs/schema-design.md` §6.3 -**Dependencies:** None. +**Dependencies:** Phase 1 complete. **Acceptance criteria:** -- You have 5 wordlist files in `data/wordlists/` (one per language). +- You have 5 wordlist files in `data-pipeline/source-data/` (one per + language). - A Gemini call with 5 German nouns returns valid JSON matching the contract, including definitions, examples, translations with gender, and difficulty levels. @@ -98,7 +124,7 @@ ahead. --- -## Phase 1: Data Pipeline +## Phase 3: Data Pipeline **Goal:** A repeatable script that takes a wordlist, calls Gemini in batches of 20, validates the output, and writes clean rows to SQLite. @@ -112,7 +138,7 @@ batches of 20, validates the output, and writes clean rows to SQLite. - [ ] Review the rejection log, fix prompt issues, re-run failed batches - [ ] Spot-check 50 random entries for correctness -**Dependencies:** Phase 0 complete. +**Dependencies:** Phase 2 complete. **Acceptance criteria:** @@ -125,16 +151,13 @@ batches of 20, validates the output, and writes clean rows to SQLite. --- -## Phase 2: Schema & Migration +## Phase 4: Migration & Import -**Goal:** The new normalized schema exists in Drizzle, the local -Postgres database is migrated, and the SQLite data is imported. +**Goal:** The new schema exists in Postgres and the SQLite data is +imported. **Tasks:** -- [ ] Write the new Drizzle schema file: - `words` table (headword, language_code, pos) - `senses` table (word_id FK, sense_index, difficulty, - cefr_level, definitions TEXT[], examples TEXT[]) - `translations` table (sense_id FK, target_language_code, - translation, gender, difficulty) - All CHECK constraints, UNIQUE constraints, indexes - [ ] Generate the Drizzle migration (`npx drizzle-kit generate`) - [ ] Inspect the generated SQL file — verify it creates the right tables, constraints, and indexes @@ -146,7 +169,7 @@ Postgres database is migrated, and the SQLite data is imported. - [ ] Run 3–5 manual SQL queries against Postgres to sanity-check the data -**Dependencies:** Phase 1 complete (SQLite has data). +**Dependencies:** Phase 3 complete (SQLite has data). **Acceptance criteria:** @@ -158,7 +181,7 @@ Postgres database is migrated, and the SQLite data is imported. --- -## Phase 3: App Integration +## Phase 5: App Integration **Goal:** The running app uses the new schema. Game rounds and distractors work correctly for all language pairs. @@ -177,7 +200,7 @@ distractors work correctly for all language pairs. - [ ] Test edge cases: - A difficulty/pos/language combo with very few words (does the app handle < 4 available words gracefully?) - A word with multiple senses (does the correct sense appear?) -**Dependencies:** Phase 2 complete (Postgres has data, schema exists). +**Dependencies:** Phase 4 complete (Postgres has data, schema exists). **Acceptance criteria:** @@ -189,7 +212,7 @@ distractors work correctly for all language pairs. --- -## Phase 4: Production Deploy +## Phase 6: Production Deploy **Goal:** The live app runs on the new schema with the new data. @@ -203,7 +226,7 @@ distractors work correctly for all language pairs. - [ ] Monitor for errors (server logs, browser console) for 24h - [ ] Remove old tables from prod (after confirming everything works) -**Dependencies:** Phase 3 complete (dev is fully working). +**Dependencies:** Phase 5 complete (dev is fully working). **Acceptance criteria:** @@ -214,7 +237,7 @@ distractors work correctly for all language pairs. --- -## Phase 5: Extend POS +## Phase 7: Extend POS **Goal:** Verbs, adjectives, and adverbs are in the database and usable in the app. @@ -229,7 +252,7 @@ usable in the app. - [ ] Test game flow with verbs, adjectives, adverbs - [ ] Verify the POS filter in the app UI works for all types -**Dependencies:** Phase 4 complete. +**Dependencies:** Phase 6 complete. **Acceptance criteria:** @@ -238,7 +261,7 @@ usable in the app. --- -## Phase 6: Future Features (out of scope) +## Phase 8: Future Features (out of scope) Listed here for visibility. Not planned, not estimated. @@ -255,9 +278,9 @@ Listed here for visibility. Not planned, not estimated. --- -## Phase 0: Preparation — Detailed +## Phase 2: Preparation — Detailed -### 0.1 Acquire wordlists +### 2.1 Acquire wordlists - Search for frequency lists. Good starting points: - "Leipzig Corpora Collection" (academic, per-language) @@ -267,11 +290,11 @@ Listed here for visibility. Not planned, not estimated. - Tatoeba sentence counts as a proxy for word frequency - Target: ~1000 nouns per language, sorted by frequency. - Format: plain text, one word per line, UTF-8, no headers. -- Save to `data/wordlists/nouns_de.txt`, `nouns_es.txt`, etc. +- Save to `data-pipeline/source-data/{language}/nouns/`. - Clean the lists: remove duplicates, remove words with spaces (multi-word expressions), remove proper nouns if desired. -### 0.2 Set up local Postgres +### 2.2 Set up local Postgres - Option A (Docker): ``` @@ -288,7 +311,7 @@ Listed here for visibility. Not planned, not estimated. - Verify: connect with `psql` or a GUI client (TablePlus, DBeaver, pgAdmin). Run `SELECT 1;`. -### 0.3 Write and test the Gemini prompt +### 2.3 Write and test the Gemini prompt - Start with 5 German nouns. - The prompt should specify: @@ -297,7 +320,6 @@ Listed here for visibility. Not planned, not estimated. - That translations are needed for all 4 other supported languages - That gender must be provided for de/fr/es/it, null for en - That difficulty must be one of: easy, medium, hard - - That CEFR level must be one of: A1, A2, B1, B2, C1, C2 - That multiple senses should be included for polysemous words - Call the API. Inspect the JSON. Common issues to fix: - Gemini wraps the JSON in markdown code fences → strip them @@ -308,12 +330,12 @@ Listed here for visibility. Not planned, not estimated. --- -## Phase 1: Data Pipeline — Detailed +## Phase 3: Data Pipeline — Detailed -### 1.1 Create the SQLite schema +### 3.1 Create the SQLite schema -- Create a file `pipeline/schema.sql` or define it in your pipeline - script. +- Create a file `data-pipeline/schema.sql` or define it in your + pipeline script. - Tables mirror the Postgres schema: ```sql @@ -331,7 +353,6 @@ Listed here for visibility. Not planned, not estimated. word_id TEXT NOT NULL REFERENCES words(id), sense_index INTEGER NOT NULL DEFAULT 0, difficulty TEXT NOT NULL, - cefr_level TEXT, definitions TEXT NOT NULL DEFAULT '[]', -- JSON array as text examples TEXT NOT NULL DEFAULT '[]', -- JSON array as text created_at TEXT DEFAULT (datetime('now')), @@ -353,9 +374,9 @@ Listed here for visibility. Not planned, not estimated. - Note: SQLite has no native TEXT[]. Store arrays as JSON text. The import script will parse them into Postgres TEXT[] on import. -### 1.2 Write the validation module +### 3.2 Write the validation module -- Create `pipeline/validate.ts` (or .js). +- Create `data-pipeline/validate.ts`. - Input: one parsed Gemini entry (the JSON object for one word). - Checks (return a list of errors, empty = valid): - `headword` is a non-empty string @@ -365,8 +386,6 @@ Listed here for visibility. Not planned, not estimated. - Each sense has: - `sense_index` is a non-negative integer - `difficulty` is in ['easy','medium','hard'] - - `cefr_level` is in ['A1','A2','B1','B2','C1','C2'] or null - - CEFR ↔ difficulty mapping is consistent - `definitions` is a non-empty array of non-empty strings - `examples` is a non-empty array of non-empty strings - `translations` is a non-empty array @@ -381,9 +400,9 @@ Listed here for visibility. Not planned, not estimated. - `difficulty` is in ['easy','medium','hard'] - Output: `{ valid: boolean, errors: string[] }` -### 1.3 Write the pipeline script +### 3.3 Write the pipeline script -- Create `pipeline/run.ts`. +- Create `data-pipeline/run.ts`. - Pseudocode: ``` for each language in [de, en, es, fr, it]: @@ -398,7 +417,7 @@ Listed here for visibility. Not planned, not estimated. generate UUIDs for word, senses, translations INSERT into SQLite (words, senses, translations) else: - append to rejection log (data/rejections/{lang}_{pos}.jsonl) + append to rejection log log progress: "Batch 12/50 done. 238 words imported, 2 rejected." sleep 1s (rate limiting) ``` @@ -407,7 +426,7 @@ Listed here for visibility. Not planned, not estimated. - Store definitions/examples as JSON strings in SQLite (`JSON.stringify(arr)`). -### 1.4 Run and review +### 3.4 Run and review - Run the pipeline for all 5 languages. - Check the rejection log. If rejection rate > 10%, fix the prompt @@ -427,31 +446,21 @@ Listed here for visibility. Not planned, not estimated. --- -## Phase 2: Schema & Migration — Detailed +## Phase 4: Migration & Import — Detailed -### 2.1 Write the Drizzle schema - -- Create or update `src/db/schema.ts`. -- Define the three tables using Drizzle's `pgTable` builder. -- Include all columns, types, constraints, and indexes as specified - in `docs/schema-design.md` §3. -- Keep the old tables in the file for now (commented out or in a - separate file) so the app doesn't break during development. - -### 2.2 Generate and inspect the migration +### 4.1 Generate and inspect the migration - Run: `npx drizzle-kit generate` -- Open the generated SQL file in `drizzle/` (or wherever your - config puts it). +- Open the generated SQL file in `packages/db/drizzle/`. - Read it. Verify: - - Three CREATE TABLE statements + - Three CREATE TABLE statements (words, senses, translations) - CHECK constraints match your schema - UNIQUE constraints are present - Three CREATE INDEX statements - Foreign keys reference the correct tables with ON DELETE CASCADE - If something looks wrong, fix the schema file and regenerate. -### 2.3 Apply the migration locally +### 4.2 Apply the migration locally - Run: `npx drizzle-kit migrate` - Connect to local Postgres and verify: @@ -462,9 +471,9 @@ Listed here for visibility. Not planned, not estimated. \d translations -- describe translations table ``` -### 2.4 Write the import script +### 4.3 Write the import script -- Create `scripts/import-to-postgres.ts`. +- Create `data-pipeline/import-to-postgres.ts`. - Pseudocode: ``` @@ -492,7 +501,7 @@ Listed here for visibility. Not planned, not estimated. into actual arrays (Postgres TEXT[]). - Use `ON CONFLICT DO NOTHING` to handle re-runs gracefully. -### 2.5 Run and verify +### 4.4 Run and verify - Run the import script. - Compare counts: @@ -529,11 +538,11 @@ Listed here for visibility. Not planned, not estimated. --- -## Phase 3: App Integration — Detailed +## Phase 5: App Integration — Detailed -### 3.1 Rewrite getGameTerms +### 5.1 Rewrite getGameTerms -- Open the file containing `getGameTerms`. +- Open `packages/db/src/models/termModel.ts`. - Replace the old query (2-table join on `vocabulary_entries` + `entry_translations`) with the new 3-table join (words → senses → translations). @@ -545,7 +554,7 @@ Listed here for visibility. Not planned, not estimated. - Return: word_id, headword, sense_id, definitions, examples, translation, gender. -### 3.2 Rewrite getDistractors +### 5.2 Rewrite getDistractors - Same 3-table join. - Additional filters: @@ -555,7 +564,7 @@ Listed here for visibility. Not planned, not estimated. - If fewer than 3 distractors are found (small data pool), handle gracefully: reduce the number of options or log a warning. -### 3.3 Update exercise generation +### 5.3 Update exercise generation - In the function that assembles a game round: - Pick one random definition: @@ -566,7 +575,7 @@ Listed here for visibility. Not planned, not estimated. - Shuffle the 4 options (Fisher-Yates or similar). - Attach gender to each option for display. -### 3.4 Test matrix +### 5.4 Test matrix Run through this matrix manually in the dev app: @@ -584,18 +593,18 @@ appear as distractors, no duplicate options. --- -## Phase 4: Production Deploy — Detailed +## Phase 6: Production Deploy — Detailed -### 4.1 Pre-deploy checklist +### 6.1 Pre-deploy checklist -- [ ] All Phase 3 acceptance criteria pass +- [ ] All Phase 5 acceptance criteria pass - [ ] `git status` is clean, all changes committed - [ ] The Drizzle migration file is committed to the repo - [ ] You know your prod database connection string - [ ] You have a backup method for prod (pg_dump, hosting provider snapshot, etc.) -### 4.2 Deploy +### 6.2 Deploy - Back up prod: ``` @@ -607,11 +616,11 @@ appear as distractors, no duplicate options. ``` - Run import script against prod: ``` - DATABASE_URL= npx tsx scripts/import-to-postgres.ts + DATABASE_URL= npx tsx data-pipeline/import-to-postgres.ts ``` - Verify counts on prod. -### 4.3 Post-deploy verification +### 6.3 Post-deploy verification - Open the live app. Play 2 full games with different settings. - Check server logs for errors. @@ -623,24 +632,24 @@ appear as distractors, no duplicate options. ``` (Or keep them for another week if you want a safety net.) -### 4.4 Rollback plan +### 6.4 Rollback plan If something goes wrong: - Restore the backup: ``` - psql -h -U -d < backup_20260722.sql + psql -h -U -d < backup_YYYYMMDD.sql ``` - Revert the code to the previous commit. - Redeploy. --- -## Phase 5: Extend POS — Detailed +## Phase 7: Extend POS — Detailed -### 5.1 Per POS +### 7.1 Per POS -Repeat the Phase 1 pipeline for each new POS: +Repeat the Phase 3 pipeline for each new POS: - Acquire wordlists (verbs, adjectives, adverbs) for all 5 languages. - Adjust the Gemini prompt: @@ -653,7 +662,7 @@ Repeat the Phase 1 pipeline for each new POS: - Run pipeline → validate → SQLite → import → Postgres. - Test in the app with the POS filter set to the new type. -### 5.2 No schema changes +### 7.2 No schema changes The `pos` column already supports all four types. The `gender` column is nullable and simply won't be populated for adverbs or @@ -678,27 +687,27 @@ English words. No migration needed. ``` project/ -├── docs/ -│ ├── schema-design.md ← companion design doc -│ └── roadmap.md ← this document -├── data/ -│ ├── wordlists/ -│ │ ├── nouns_de.txt -│ │ ├── nouns_en.txt -│ │ ├── nouns_es.txt -│ │ ├── nouns_fr.txt -│ │ └── nouns_it.txt +├── data-pipeline/ +│ ├── source-data/ +│ │ ├── german/nouns/ ← wordlist files +│ │ ├── english/nouns/ +│ │ ├── spanish/nouns/ +│ │ ├── french/nouns/ +│ │ └── italian/nouns/ │ ├── rejections/ ← invalid Gemini entries (for review) -│ └── staging.db ← SQLite staging database -├── pipeline/ -│ ├── run.ts ← main pipeline script +│ ├── staging.db ← SQLite staging database +│ ├── schema.sql ← SQLite schema definition │ ├── validate.ts ← validation module -│ ├── prompt.ts ← Gemini prompt template -│ └── schema.sql ← SQLite schema definition -├── scripts/ +│ ├── run.ts ← main pipeline script │ └── import-to-postgres.ts ← SQLite → Postgres import -├── src/ -│ └── db/ -│ └── schema.ts ← Drizzle schema (updated) -└── drizzle/ ← generated migration files +├── packages/ +│ ├── db/ +│ │ └── src/db/schema.ts ← Drizzle schema (updated ✅) +│ └── shared/ +│ └── src/constants.ts ← NOUN_GENDERS added ✅, "medium" ✅ +├── documentation/ +│ └── pipeline/ +│ ├── design-doc.md ← schema design doc (updated ✅) +│ └── roadmap.md ← this document (updated ✅) +└── ... ```