lila e0b1f7f92f updating documentation

2026-07-23 16:57:07 +02:00

24 KiB

Raw Blame History

Vocabulary Trainer — Data Schema Design Document

Project: PERN-stack vocabulary trainer with Gemini-powered data pipeline Author: [Your Name] Date: July 2026 Status: Approved — schema implemented

1. Overview

This document describes the database schema, data pipeline, and query patterns for the vocabulary trainer application. The app supports 5 languages (English, German, Italian, French, Spanish) and tests learners by showing a word with a definition and example sentence, then asking them to pick the correct translation from 4 options (1 correct, 3 distractors).

The previous data source (OpenWordNet / kaikki.org) produced low-quality and inaccurate entries. The new approach uses a batch pipeline: word frequency lists are fed to the Gemini API in groups of 20, which generates structured metadata (definitions, examples, translations, difficulty levels). The output is validated, stored in a local SQLite staging database, and mirrored into the production Postgres database.

2. Database Choice

Postgres (production and development) with SQLite (pipeline staging).

Why Postgres

The data is inherently relational: words → senses → translations. Foreign keys enforce referential integrity at the database level.
The query pattern (filter → join → random → limit) is exactly what SQL is designed for.
Postgres provides JSONB and native arrays for semi-structured fields (definitions, examples, inflection tags) without sacrificing relational structure.
ACID transactions ensure batch imports are atomic.
Already part of the PERN stack. No additional infrastructure.

Why not MongoDB

The data has real, meaningful relationships (not arbitrary nested documents). The distractor query ("exclude translations from the same sense") is a single WHERE sense_id != X in SQL but requires a complex aggregation pipeline in MongoDB.
No foreign key enforcement. Data integrity would depend entirely on application code — risky with LLM-generated data.
Future inflection tables (one word → 30–50 forms) are a natural relational fit, not a document-store fit.

Why not DynamoDB

Designed for simple key-value lookups at massive scale (billions of rows). Cannot do ad-hoc filtering, joins, or ORDER BY RANDOM().
The query pattern (filter by language + pos + difficulty, then randomize) would require pre-building indexes for every combination.
Massive overkill for 500k words.

Why SQLite for staging

Zero-config, file-based. Ideal for the single-writer batch pipeline.
The pipeline writes to SQLite, a separate import script mirrors the data into Postgres. The application (dev and prod) always reads from Postgres to avoid SQLite/Postgres dialect differences.

3. Schema

3.1 Entity Relationship

┌─────────────┐       ┌─────────────┐       ┌──────────────────┐
│    words    │       │   senses    │       │  translations    │
├─────────────┤       ├─────────────┤       ├──────────────────┤
│ id (PK)     │──┐    │ id (PK)     │──┐    │ id (PK)          │
│ headword    │  └───>│ word_id(FK) │  └───>│ sense_id (FK)    │
│ language_code│      │ sense_index │       │ target_lang_code │
│ pos         │       │ difficulty  │       │ translation      │
│             │       │ definitions │       │ gender           │
│             │       │ examples    │       │ difficulty       │
└─────────────┘       └─────────────┘       └──────────────────┘

Future (not yet implemented):
┌──────────────────┐
│ inflection_forms │
├──────────────────┤
│ id (PK)          │
│ word_id (FK) ────────> words.id
│ form             │
│ tags (JSONB)     │
└──────────────────┘

3.2 Table: `words`

One row per unique word in a specific language.

Column	Type	Constraints	Notes
id	UUID	PK, default random
headword	TEXT	NOT NULL	"Haus", "casa", "house"
language_code	VARCHAR(10)	NOT NULL, CHECK in supported list	"de", "es", "en", "fr", "it"
pos	VARCHAR(20)	NOT NULL, CHECK in supported list	"noun", "verb", "adjective", "adverb"
created_at	TIMESTAMPTZ	NOT NULL, default now()

Constraints:

UNIQUE (headword, language_code, pos) — prevents duplicate entries.
CHECK (language_code IN ('en','de','it','fr','es'))
CHECK (pos IN ('noun','verb','adjective','adverb'))

Index:

idx_language_code_pos ON (language_code, pos) — accelerates the primary game query filter.

Design note: Each language gets its own headword entries. "Haus" is a German word row. "casa" is a Spanish word row. They are separate entries, linked through the translations table. This is what enables any language pair as source/target.

3.3 Table: `senses`

One row per distinct meaning of a word. This is where polysemy is handled: "bank" (financial institution) and "bank" (river edge) are two senses of one word.

Column	Type	Constraints	Notes
id	UUID	PK, default random
word_id	UUID	FK → words.id, ON DELETE CASCADE
sense_index	SMALLINT	NOT NULL, default 0	0 = primary meaning, 1 = secondary, etc.
difficulty	VARCHAR(20)	NOT NULL, CHECK in allowed list	"easy", "medium", "hard"
definitions	TEXT[]	NOT NULL, default '{}'	1–3 definitions in the word's language
examples	TEXT[]	NOT NULL, default '{}'	1–3 example sentences in the word's language
created_at	TIMESTAMPTZ	NOT NULL, default now()

Constraints:

UNIQUE (word_id, sense_index) — one sense per index per word.
CHECK (difficulty IN ('easy','medium','hard'))

Index:

idx_word_sense_difficulty ON (word_id, difficulty) — accelerates the join from words and the difficulty filter.

Design note — definitions and examples as arrays: Definitions and examples are stored as TEXT[] arrays on the sense row rather than in separate tables. Rationale:

Each sense has at most 2–3 definitions/examples (1-to-few).
They are always fetched together with the sense (no independent querying needed).
Separate tables would add 2 JOINs to the hottest query for no practical benefit.
The exercise generator picks one definition and one example randomly in application code: arr[Math.floor(Math.random() * arr.length)].

3.4 Table: `translations`

One row per translation of a sense into another language. A single sense can have multiple translations into the same language at different difficulty levels (e.g., "Bank" easy, "Geldinstitut" medium).

Column	Type	Constraints	Notes
id	UUID	PK, default random
sense_id	UUID	FK → senses.id, ON DELETE CASCADE
target_language_code	VARCHAR(10)	NOT NULL, CHECK in supported list	Language of the translation
translation	TEXT	NOT NULL	"casa", "Haus", "maison"
gender	VARCHAR(20)	nullable	"masculine","feminine","neuter", NULL
difficulty	VARCHAR(20)	NOT NULL, CHECK in allowed list	Can differ from sense difficulty
created_at	TIMESTAMPTZ	NOT NULL, default now()

Constraints:

UNIQUE (sense_id, target_language_code, translation) — allows multiple translations per language (synonyms) but no exact duplicates.
CHECK (target_language_code IN ('en','de','it','fr','es'))
CHECK (difficulty IN ('easy','medium','hard'))
CHECK (gender IS NULL OR gender IN ('masculine','feminine','neuter'))

Index:

idx_translations_sense_language_difficulty ON (sense_id, target_language_code, difficulty) — accelerates the join from senses and the language/difficulty filter.

Design note — gender as a real column: Gender is stored as a column, not embedded in a JSON blob, because future gender exercises will need to filter and group by gender. English nouns have gender = NULL.

3.5 Future Table: `inflection_forms` (not yet implemented)

Will be added when verb conjugation and adjective declension exercises are built.

Column	Type	Constraints	Notes
id	UUID	PK, default random
word_id	UUID	FK → words.id, ON DELETE CASCADE
form	TEXT	NOT NULL	"Häuser", "ginge", "grüner"
tags	JSONB	NOT NULL, default '{}'	{"case":"genitive","number":"plural"}

This table is the primary reason the schema is normalized rather than using JSONB documents. A German verb has 30–50 inflected forms. An adjective has 12+. These are one-to-many relationships that require their own table with a foreign key.

4. Difficulty Model

There are two difficulty columns. They answer different questions.

`senses.difficulty` — "Is this meaning appropriate for the level?"

Controls which meanings of a word are shown. A beginner should not be tested on "Haus = noble dynasty" because the concept itself is advanced, regardless of how hard the translation word is.

`translations.difficulty` — "Is this word an appropriate answer?"

Controls which translation word is the correct answer. The concept of "bank" is easy, but "Geldinstitut" is a harder word than "Bank" for that same concept.

Query filter logic: sense as ceiling, translation as target

WHERE s.difficulty IN ('easy', 'medium')   -- sense at or below level
  AND t.difficulty = 'medium'              -- translation exactly at level

The sense difficulty acts as a ceiling: don't show meanings harder than the selected level.
The translation difficulty acts as the target: test the learner on a word at exactly this level.

This ensures that easy concepts with medium-level synonyms (e.g., "bank" → "Geldinstitut") are reachable at the medium level, while advanced concepts (e.g., "Haus" → "dynasty") remain gated.

Example data

Word	Sense	Sense Diff.	Translation	Trans. Diff.
Haus	building	easy	casa (es)	easy
Haus	noble dynasty	hard	dinastía (es)	hard
bank	financial inst.	easy	Bank (de)	easy
bank	financial inst.	easy	Geldinstitut (de)	medium
bank	financial inst.	easy	Kreditinstitut (de)	hard
bank	river edge	medium	Ufer (de)	medium

At medium level, the query returns: Geldinstitut, Ufer. At hard level: Kreditinstitut, dinastía. "Bank" (easy translation) never appears at medium/hard as a correct answer. "dinastía" (hard sense) never appears at easy/medium.

5. Query Patterns

5.1 Game query — get N random words

Scenario: user picks German → Spanish, nouns, medium, 20 rounds.

SELECT
  w.id          AS word_id,
  w.headword,
  s.id          AS sense_id,
  s.definitions,
  s.examples,
  t.translation,
  t.gender
FROM words w
INNER JOIN senses s
  ON s.word_id = w.id
INNER JOIN translations t
  ON t.sense_id = s.id
WHERE w.language_code = 'de'
  AND w.pos = 'noun'
  AND s.difficulty IN ('easy', 'medium')
  AND t.target_language_code = 'es'
  AND t.difficulty = 'medium'
ORDER BY RANDOM()
LIMIT 20;

The application then picks one random definition and one random example from the arrays for each word.

5.2 Distractor query — get 3 wrong answers

For a given correct answer, fetch 3 distractors from the same language, pos, and difficulty pool.

SELECT t.translation, t.gender
FROM translations t
INNER JOIN senses s
  ON t.sense_id = s.id
INNER JOIN words w
  ON s.word_id = w.id
WHERE w.language_code = 'de'
  AND w.pos = 'noun'
  AND s.difficulty IN ('easy', 'medium')
  AND t.target_language_code = 'es'
  AND t.difficulty = 'medium'
  AND t.sense_id != :current_sense_id
  AND t.translation != :correct_answer
ORDER BY RANDOM()
LIMIT 3;

5.3 Distractor exclusion rule

Distractors must come from a different sense than the correct answer. Not just a different word — a different sense.

Rationale: multiple translations of the same sense are all valid answers. "Bank" and "Geldinstitut" are both correct translations of the financial-institution sense. Showing one as a distractor for the other would confuse the learner and break trust.

The sense_id != :current_sense_id filter excludes all synonyms of the same sense in one condition. No synonym table needed.

Translations from a different sense of the same word are valid distractors (e.g., "Ufer" from the river-bank sense is a fine distractor for the financial-institution sense — the definition makes it clearly wrong).

5.4 Edge case: identical translation text across senses

Two different senses of different words may share the same translation text (e.g., "Schloss" = castle and "Schloss" = lock). The t.translation != :correct_answer filter handles this by excluding the exact text regardless of sense.

6. Data Pipeline

6.1 Flow

Word frequency lists (per language, per POS)
        │
        ▼
Gemini API (batches of 20 words)
        │
        ▼
Validation script (reject/flag bad entries)
        │
        ▼
SQLite staging database (local file)
        │
        ▼
Import script (SQLite → Postgres, batch inserts)
        │
        ▼
Postgres (dev)  →  test full game flow
        │
        ▼
Postgres (prod) via Drizzle migration

6.2 Wordlist source

Frequency-based word lists, one per language. Example: "1000 most common German nouns." Sources: Leipzig Corpora, Wiktionary frequency lists, or similar open-source frequency data.

Each language is processed independently. This ensures language-native definitions and examples (a German word gets a German definition, not a translated English one).

6.3 Gemini output JSON contract

The API is prompted to return an array of objects. Expected shape:

[
  {
    "headword": "Haus",
    "language": "de",
    "pos": "noun",
    "senses": [
      {
        "sense_index": 0,
        "difficulty": "easy",
        "definitions": ["Ein Gebäude zum Wohnen."],
        "examples": ["Sie kauften ein Haus in der Stadt."],
        "translations": [
          {
            "target_language": "en",
            "word": "house",
            "gender": null,
            "difficulty": "easy"
          },
          {
            "target_language": "es",
            "word": "casa",
            "gender": "feminine",
            "difficulty": "easy"
          },
          {
            "target_language": "fr",
            "word": "maison",
            "gender": "feminine",
            "difficulty": "easy"
          },
          {
            "target_language": "it",
            "word": "casa",
            "gender": "feminine",
            "difficulty": "easy"
          }
        ]
      },
      {
        "sense_index": 1,
        "difficulty": "hard",
        "definitions": ["Ein Adelsgeschlecht, eine Dynastie."],
        "examples": ["Das Haus der Merowinger herrschte über Franken."],
        "translations": [
          {
            "target_language": "en",
            "word": "house",
            "gender": null,
            "difficulty": "hard"
          },
          {
            "target_language": "es",
            "word": "dinastía",
            "gender": "feminine",
            "difficulty": "hard"
          }
        ]
      }
    ]
  }
]

6.4 Validation rules

Before writing to SQLite, every entry is checked:

headword, language, pos are present and valid.
At least one sense per word.
Each sense has at least one definition and one example.
difficulty is one of: easy, medium, hard.
gender is valid for the target language:
- German: masculine, feminine, neuter
- French, Spanish, Italian: masculine, feminine
- English: null
Translations exist for at least the 4 other supported languages.
No duplicate entries (headword + language + pos + sense_index).

Invalid entries are logged and excluded. They can be reviewed and re-processed manually.

6.5 Import: SQLite → Postgres

A Node.js script reads from SQLite (via better-sqlite3) and batch-inserts into Postgres (via Drizzle). Inserts are wrapped in transactions per batch (20 words) for atomicity.

The application (dev and prod) always reads from Postgres. SQLite is used only as a pipeline staging file.

7. Indexes

Three indexes cover the game and distractor queries:

CREATE INDEX idx_language_code_pos
  ON words (language_code, pos);

CREATE INDEX idx_word_sense_difficulty
  ON senses (word_id, difficulty);

CREATE INDEX idx_translations_sense_language_difficulty
  ON translations (sense_id, target_language_code, difficulty);

8. Performance

Estimated data volume at target scale

Table	Rows	Derivation
words	500k	~100k per language × 5 languages
senses	~750k	~1.5 senses per word average
translations	~3M	~4 translations per sense

Query performance

Scale	Game query (LIMIT 20)	Distractor query (LIMIT 3)
10k words	< 10 ms	< 10 ms
500k words	20–80 ms	15–60 ms
5M words	100–300 ms	80–200 ms

The bottleneck at scale is ORDER BY RANDOM(), which sorts the entire filtered result set before applying LIMIT. At 500k words, the filtered set per query is ~5k–15k rows — well within comfortable range.

Future optimization (if filtered sets exceed ~100k rows):

WHERE ... AND random() < 0.05   -- pre-filter to ~5% of rows
ORDER BY RANDOM()
LIMIT 20;

Or use TABLESAMPLE. Not needed at current scale.

Import performance

Method	Time for 3.5M rows
Individual INSERT	~30–60 min
Batch INSERT (1000 per stmt)	~2–5 min
Postgres COPY (CSV)	~10–30 sec

The pipeline uses batch inserts via Drizzle. The full import is a one-time or occasional operation.

9. Implementation Plan

 1. ✅ Schema design (this document)
 2. ✅ Drizzle schema: words, senses, translations, relations, indexes
 3.    Acquire word frequency lists (5 languages, nouns first)
 4.    Build + test Gemini prompt (5 words → 20 words → full batch)
 5.    Validation script (Gemini output → clean JSON)
 6.    SQLite staging schema + pipeline write
 7.    Import script (SQLite → local Postgres)
 8.    Drizzle migration on local Postgres
 9.    Dev branch: new queries (game + distractor), full game flow test
10.    Drizzle migration on prod Postgres + data import + verify
11.    Extend pipeline: verbs, adjectives, adverbs (schema unchanged)
12.    Later: inflection_forms table + conjugation/declension exercises

10. Future Extensions

Verb conjugation / adjective declension exercises

The inflection_forms table (section 3.5) will store inflected forms with grammatical tags as JSONB. The schema is normalized specifically to support this: one word → many forms, each independently queryable.

Gender exercises

The gender column on translations enables filtering and grouping by grammatical gender for dedicated gender practice rounds.

Additional POS

The pos column already supports noun, verb, adjective, adverb. Adding a new POS requires no schema change — only a new wordlist and an adjusted Gemini prompt.

11. Key Design Decisions — Summary

Decision	Choice	Rationale
Database	Postgres + SQLite staging	Relational data, FK integrity, SQL query pattern, already in stack
Schema structure	3 normalized tables	Matches query pattern, supports any language pair, extensible for inflections
Definitions / examples	`TEXT[]` arrays on `senses`	1-to-few relationship, always fetched with sense, avoids 2 extra JOINs
Gender	Column on `translations`	Needed for filtering in future gender exercises
Difficulty	Two columns: `senses.difficulty` + `translations.difficulty`	Sense = concept gate, translation = word-level target
Difficulty filter logic	Sense as ceiling, translation as exact match	Ensures easy concepts with hard synonyms are reachable; advanced concepts stay gated
Distractor exclusion	`sense_id != current`	Prevents valid synonyms from appearing as wrong answers
Language direction	Any of 5 languages as source or target	Each language has its own headword entries; translations link them
Pipeline	Gemini → validate → SQLite → Postgres	Batch-generated data, LLM output needs validation, SQLite for staging simplicity
Table-per-language/pos	Rejected	Anti-pattern: 40+ tables, exponential maintenance
Single JSONB blob	Rejected	Cannot support inflection tables, cannot index gender, no FK integrity

24 KiB Raw Blame History Unescape Escape

Vocabulary Trainer — Data Schema Design Document

1. Overview

2. Database Choice

Why Postgres

Why not MongoDB

Why not DynamoDB

Why SQLite for staging

3. Schema

3.1 Entity Relationship

3.2 Table: words

3.3 Table: senses

3.4 Table: translations

3.5 Future Table: inflection_forms (not yet implemented)

4. Difficulty Model

senses.difficulty — "Is this meaning appropriate for the level?"

translations.difficulty — "Is this word an appropriate answer?"

Query filter logic: sense as ceiling, translation as target

Example data

5. Query Patterns

5.1 Game query — get N random words

5.2 Distractor query — get 3 wrong answers

5.3 Distractor exclusion rule

5.4 Edge case: identical translation text across senses

6. Data Pipeline

6.1 Flow

6.2 Wordlist source

6.3 Gemini output JSON contract

6.4 Validation rules

6.5 Import: SQLite → Postgres

7. Indexes

8. Performance

Estimated data volume at target scale

Query performance

Import performance

9. Implementation Plan

10. Future Extensions

Verb conjugation / adjective declension exercises

Gender exercises

Additional POS

11. Key Design Decisions — Summary

24 KiB

Raw Blame History

3.2 Table: `words`

3.3 Table: `senses`

3.4 Table: `translations`

3.5 Future Table: `inflection_forms` (not yet implemented)

`senses.difficulty` — "Is this meaning appropriate for the level?"

`translations.difficulty` — "Is this word an appropriate answer?"