# Decisions Log

A record of non-obvious technical decisions made during development, with reasoning. Intended to preserve context across sessions.

---

## Tooling

### Monorepo: pnpm workspaces (not Turborepo)

Turborepo adds parallel task running and build caching on top of pnpm workspaces. For a two-app monorepo of this size, plain pnpm workspace commands are sufficient and there is one less tool to configure and maintain.

### TypeScript runner: `tsx` (not `ts-node`)

`tsx` is faster, requires no configuration, and uses esbuild under the hood. `ts-node` is older and more complex to configure. `tsx` does not do type checking — that is handled separately by `tsc` and the editor. Installed as a dev dependency in `apps/api` only.

### ORM: Drizzle (not Prisma)

Drizzle is lighter — no binary, no engine. Queries map closely to SQL. Migrations are plain SQL files. Works naturally with Zod for type inference. Prisma would add Docker complexity (engine binary in containers) and abstraction that is not needed for this schema.

### WebSocket: `ws` library (not Socket.io)

For rooms of 2–4 players, Socket.io's room management, transport fallbacks, and reconnection abstractions are unnecessary overhead. The WS protocol is defined explicitly as a Zod discriminated union in `packages/shared`, giving the same type safety guarantees. Reconnection logic is deferred to Phase 7.

### Auth: OpenAuth (not rolling own JWT)

All auth delegated to OpenAuth service at `auth.yourdomain.com`. Providers: Google, GitHub. The API validates the JWT on every protected request. User rows are created or updated on first login via the `sub` claim as the primary key.

---

## Docker

### Multi-stage builds for monorepo context

Both `apps/web` and `apps/api` use multi-stage Dockerfiles (`deps`, `dev`, `builder`, `runner`) because:

- The monorepo structure requires copying `pnpm-workspace.yaml`, root `package.json`, and cross-dependencies (`packages/shared`, `packages/db`) before installing
- `node_modules` paths differ between host and container due to workspace hoisting
- Stages allow caching `pnpm install` separately from source code changes

### Vite as dev server (not Nginx)

In development, `apps/web` uses `vite dev` directly, not Nginx. Reasons:

- Hot Module Replacement (HMR) requires Vite's WebSocket dev server
- Source maps and error overlay need direct Vite integration
- Nginx would add unnecessary proxy complexity for local dev

Production will use Nginx to serve static Vite build output.

---

## Architecture

### Express app structure: factory function pattern

`app.ts` exports a `createApp()` factory function. `server.ts` imports it and calls `.listen()`. This allows tests to import the app directly without starting a server, keeping tests isolated and fast.

### Data model: `decks` separate from `terms` (not frequency_rank filtering)

**Original approach:** Store `frequency_rank` on `terms` table and filter by rank range for difficulty.

**Problem discovered:** WordNet/OMW frequency data is unreliable for language learning. Extraction produced results like:

- Rank 1: "In" → "indio" (chemical symbol: Indium)
- Rank 2: "Be" → "berillio" (chemical symbol: Beryllium)
- Rank 7: "He" → "elio" (chemical symbol: Helium)

These are technically "common" in WordNet (every element is a noun) but useless for vocabulary learning.

**Decision:**

- `terms` table stores ALL available OMW synsets (raw data, no frequency filtering)
- `decks` table stores curated learning lists (A1, A2, B1, "Most Common 1000", etc.)
- `deck_terms` junction table links terms to decks with position ordering
- `rooms.deck_id` specifies which vocabulary deck a game uses

**Benefits:**

- Curricula can come from external sources (CEFR lists, Oxford 3000, SUBTLEX)
- Bad data (chemical symbols, obscure words) excluded at deck level, not schema level
- Users can create custom decks later
- Multiple difficulty levels without schema changes

### Multiplayer mechanic: simultaneous answers (not buzz-first)

All players see the same question at the same time and submit independently. The server waits for all answers or a 15-second timeout, then broadcasts the result. This keeps the experience Duolingo-like and symmetric. A buzz-first mechanic was considered and rejected.

### Room model: room codes (not matchmaking queue)

Players create rooms and share a human-readable code (e.g. `WOLF-42`) to invite friends. Auto-matchmaking via a queue is out of scope for MVP. Valkey is included in the stack and can support a queue in a future phase.

---

## TypeScript Configuration

### Base config: no `lib`, `module`, or `moduleResolution`

These are intentionally omitted from `tsconfig.base.json` because different packages need different values — `apps/api` uses `NodeNext`, `apps/web` uses `ESNext`/`bundler` (Vite), and mixing them in the base caused errors. Each package declares its own.

### `outDir: "./dist"` per package

The base config originally had `outDir: "dist"` which resolved relative to the base file location, pointing to the root `dist` folder. Overridden in each package with `"./dist"` to ensure compiled output stays inside the package.

### `apps/web` tsconfig: deferred to Vite scaffold

The web tsconfig was left as a placeholder and filled in after `pnpm create vite` generated `tsconfig.json`, `tsconfig.app.json`, and `tsconfig.node.json`. The generated files were then trimmed to remove options already covered by the base.

### `rootDir: "."` on `apps/api`

Set explicitly to allow `vitest.config.ts` (which lives outside `src/`) to be included in the TypeScript program. Without it, TypeScript infers `rootDir` as `src/` and rejects any file outside that directory.

---

## ESLint

### Two-config approach for `apps/web`

The root `eslint.config.mjs` handles TypeScript linting across all packages. `apps/web/eslint.config.js` is kept as a local addition for React-specific plugins only: `eslint-plugin-react-hooks` and `eslint-plugin-react-refresh`. ESLint flat config merges them automatically by directory proximity — no explicit import between them needed.

### Coverage config at root only

Vitest coverage configuration lives in the root `vitest.config.ts` only. Individual package configs omit it to produce a single aggregated report rather than separate per-package reports.

### `globals: true` with `"types": ["vitest/globals"]`

Using Vitest globals (`describe`, `it`, `expect` without imports) requires `"types": ["vitest/globals"]` in each package's tsconfig `compilerOptions`. Added to `apps/api`, `packages/shared`, and `packages/db`. Added to `apps/web/tsconfig.app.json`.

---

## Known Issues / Dev Notes

### glossa-web has no healthcheck

The `web` service in `docker-compose.yml` has no `healthcheck` defined. Reason: Vite's dev server (`vite dev`) has no built-in health endpoint. Unlike the API's `/api/health`, there's no URL to poll.

Workaround: `depends_on` uses `api` healthcheck as proxy. For production (Nginx), add a health endpoint or use TCP port check.

### Valkey memory overcommit warning

Valkey logs this on start in development:

```text
WARNING Memory overcommit must be enabled for proper functionality
```

This is **harmless in dev** but should be fixed before production. The warning appears because Docker containers don't inherit host sysctl settings by default.

Fix: Add to host `/etc/sysctl.conf`:

```conf
vm.overcommit_memory = 1
```

Then `sudo sysctl -p` or restart Docker.

---

## Data Model

### Users: internal UUID + openauth_sub (not sub as PK)

**Original approach:** Use OpenAuth `sub` claim directly as `users.id` (text primary key).

**Problem:** Embeds auth provider in the primary key (e.g. `"google|12345"`). If OpenAuth changes format or a second provider is added, the PK cascades through all FKs (`rooms.host_id`, `room_players.user_id`).

**Decision:**

- `users.id` = internal UUID (stable FK target)
- `users.openauth_sub` = text UNIQUE (auth provider claim)
- Allows adding multiple auth providers per user later without FK changes

### Rooms: `updated_at` for stale recovery only

Most tables omit `updated_at` (unnecessary for MVP). `rooms.updated_at` is kept specifically for stale room recovery—identifying rooms stuck in `in_progress` status after server crashes.

### Translations: UNIQUE (term_id, language_code, text)

Allows multiple synonyms per language per term (e.g. "dog", "hound" for same synset). Prevents exact duplicate rows. Homonyms (e.g. "Lead" metal vs. "Lead" guide) are handled by different `term_id` values (different synsets), so no constraint conflict.

### Decks: `source_language` + `validated_languages` (not `pair_id`)

**Original approach:** `decks.pair_id` references `language_pairs`, tying each deck to a single language pair.

**Problem:** One deck can serve multiple target languages as long as translations exist for all its terms. A `pair_id` FK would require duplicating the deck for each target language.

**Decision:**

- `decks.source_language` — the language the wordlist was curated from (e.g. `"en"`). A deck sourced from an English frequency list is fundamentally different from one sourced from an Italian list.
- `decks.validated_languages` — array of language codes (excluding `source_language`) for which full translation coverage exists across all terms in the deck. Recalculated and updated on every run of the generation script.
- The language pair used for a quiz session is determined at session start, not at deck creation time.

**Benefits:**

- One deck serves multiple target languages (e.g. en→it and en→fr) without duplication
- `validated_languages` stays accurate as translation data grows
- DB enforces via CHECK constraint that `source_language` is never included in `validated_languages`


### Decks: wordlist tiers as scope (not POS-split decks)

**Rejected approach:** one deck per POS (e.g. `en-nouns`, `en-verbs`).

**Problem:** POS is already a filterable column on `terms`, so a POS-scoped deck duplicates logic the query already handles for free. A word like "run" (noun and verb, different synsets) would also appear in two decks, requiring deduplication in the generation script.

**Decision:** one deck per frequency tier per source language (e.g. `en-core-1000`, `en-core-2000`). POS, difficulty, and category are query filters applied inside that boundary at query time. The user never sees or picks a deck — they pick a direction, POS, and difficulty, and the app resolves those to the right deck + filters.

Progression works by expanding the deck set as the user advances:

```sql
WHERE dt.deck_id IN ('en-core-1000', 'en-core-2000')
  AND t.pos = 'noun'
  AND t.cefr_level = 'B1'
```

Decks must not overlap — each term appears in exactly one tier. The generation script already deduplicates, so this is enforced at import time.

### Decks: SUBTLEX as wordlist source (not manual curation)

**Problem:** the most common 1000 nouns in English are not the same 1000 nouns that are most common in Italian — not just in translation, but conceptually. Building decks from English frequency data alone gives Italian learners a distorted picture of what is actually common in Italian.

**Decision:** use SUBTLEX, which exists in per-language editions (SUBTLEX-EN, SUBTLEX-IT, etc.) derived from subtitle corpora using the same methodology, making them comparable across languages.

This is why `decks.source_language` is not just a technical detail — it is the reason the data model is correct:

- `en-core-1000` built from SUBTLEX-EN → used when source language is English (en→it)
- `it-core-1000` built from SUBTLEX-IT → used when source language is Italian (it→en)

Same translation data underneath, correctly frequency-grounded per direction. Two wordlist files, two generation script runs.

### Terms: `synset_id` nullable (not NOT NULL)

**Problem:** non-WordNet terms (custom words, Wiktionary-sourced entries added later) won't have a synset ID. `NOT NULL` is too strict.

**Decision:** make `synset_id` nullable. `synset_id` remains the WordNet idempotency key — it prevents duplicate imports on re-runs and allows cross-referencing back to WordNet. It is not removed.

Postgres `UNIQUE` on a nullable column allows multiple `NULL` values (nulls are not considered equal), so no additional constraint logic is needed beyond dropping `notNull()`. For extra defensiveness a partial unique index can be added later:

```sql
CREATE UNIQUE INDEX idx_terms_synset_id ON terms (synset_id) WHERE synset_id IS NOT NULL;
```

### Terms: `source` + `source_id` columns

Once multiple import pipelines exist (OMW today, Wiktionary later), `synset_id` alone is insufficient as an idempotency key — Wiktionary terms won't have a synset ID.

**Decision:** add `source` (varchar, e.g. `'omw'`, `'wiktionary'`, null for manual) and `source_id` (text, the pipeline's internal identifier) with a unique constraint on the pair:

```ts
unique("unique_source_id").on(table.source, table.source_id)
```

Postgres allows multiple `NULL` pairs under a unique constraint, so manual entries don't conflict. For existing OMW terms, backfill `source = 'omw'` and `source_id = synset_id`. `synset_id` remains for now to avoid pipeline churn — deprecate it during a future pipeline refactor.

No CHECK constraint on `source` — it is only written by controlled import scripts, not user input. A free varchar is sufficient.

### Translations: `cefr_level` column (deferred population, not on `terms`)

CEFR difficulty is language-relative, not concept-relative. "House" in English is A1, "domicile" is also English but B2 — same concept, different words, different difficulty. Moving `cefr_level` to `translations` allows each language's word to have its own level independently.

Added as nullable `varchar(2)` with CHECK constraint against `CEFR_LEVELS` (`A1`–`C2`) on the `translations` table. Left null for MVP; populated later via SUBTLEX or an external CEFR wordlist. Also included in the `translations` index since the quiz query filters on it:

```ts
index("idx_translations_lang").on(table.language_code, table.cefr_level, table.term_id)
```

### `language_pairs` table: dropped

Valid language pairs are already implicitly defined by `decks.source_language` + `decks.validated_languages`. The table was redundant — the same information can be derived directly from decks:

```sql
SELECT DISTINCT source_language, unnest(validated_languages) AS target_language
FROM decks
WHERE validated_languages != '{}'
```

The only thing `language_pairs` added was an `active` flag to manually disable a direction. This is an edge case not needed for MVP. Dropped to remove a maintenance surface that required staying in sync with deck data.

### Schema: `categories` + `term_categories` (empty for MVP)

Added to schema now, left empty for MVP. Grammar and Media work without them — Grammar maps to POS (already on `terms`), Media maps to deck membership. Thematic categories (animals, kitchen, etc.) require a metadata source that is still under research.

```sql
categories:      id, slug, label, created_at
term_categories: term_id → terms.id, category_id → categories.id, PK(term_id, category_id)
```

See Open Research section for source options.

### Schema constraints: CHECK over pgEnum for extensible value sets

**Question:** use `pgEnum` for columns like `pos`, `cefr_level`, and `source` since the values are driven by TypeScript constants anyway?

**Decision:** no. Use CHECK constraints for any value set that will grow over time.

**Reason:** `ALTER TYPE enum_name ADD VALUE` in Postgres is non-transactional — it cannot be rolled back if a migration fails partway through, leaving the DB in a dirty state that requires manual intervention. CHECK constraints are fully transactional — if the migration fails it rolls back cleanly.

**Rule of thumb:** pgEnum is appropriate for truly static value sets that will never grow (e.g. `('pending', 'active', 'cancelled')` on an orders table). Any value set tied to a growing constant in the codebase (`SUPPORTED_POS`, `CEFR_LEVELS`, `SUPPORTED_LANGUAGE_CODES`) stays as a CHECK constraint.

### Schema constraints: `language_code` always CHECK-constrained

`language_code` columns on `translations` and `term_glosses` are constrained via CHECK against `SUPPORTED_LANGUAGE_CODES`, the same pattern used for `pos` and `cefr_level`.

**Reason:** unlike `source`, which is only written by controlled import scripts and failing silently is recoverable, `language_code` is a query-critical filter column. A typo (`'ita'` instead of `'it'`, `'en '` with a trailing space) would silently produce missing data in the UI — terms with no translation shown, glosses not displayed — which is harder to debug than a DB constraint violation.

**Rule:** any column that game queries filter on should be CHECK-constrained. Columns only used for internal bookkeeping (like `source`) can be left as free varchars.

### Schema: unique constraints make explicit FK indexes redundant

Postgres automatically creates an index to enforce a unique constraint. An explicit index on a column that is already the leading column of a unique constraint is redundant.

Example: `unique("unique_term_gloss").on(term_id, language_code, text)` already indexes `term_id` as the leading column. A separate `index("idx_term_glosses_term").on(term_id)` adds no value and was dropped.

**Rule:** before adding an explicit index, check whether an existing unique constraint already covers it.

### Future extensions: morphology and pronunciation (deferred, additive)

The following features are explicitly deferred post-MVP. All are purely additive — new tables referencing existing `terms` rows via FK. No existing schema changes required when implemented:

- `noun_forms` — gender, singular, plural, articles per language (source: Wiktionary)
- `verb_forms` — conjugation tables per language (source: Wiktionary)
- `term_pronunciations` — IPA and audio URLs per language (source: Wiktionary / Forvo)

Exercise types split naturally into Type A (translation, current model) and Type B (morphology, future). The data layer is independent — the same `terms` anchor both.

---

### Term glosses: Italian coverage is sparse (expected)

OMW gloss data is primarily in English. After full import:

- English glosses: 95,882 (~100% of terms)
- Italian glosses: 1,964 (~2% of terms)

This is not a data pipeline problem — it reflects the actual state of OMW. Italian
glosses simply don't exist for most synsets in the dataset.

**Handling in the UI:** fall back to the English gloss when no gloss exists for the
user's language. This is acceptable UX — a definition in the wrong language is better
than no definition at all.

If Italian gloss coverage needs to improve in the future, Wiktionary is the most
likely source — it has broader multilingual definition coverage than OMW.

---

## Open Research

### Semantic category metadata source

Categories (`animals`, `kitchen`, etc.) are in the schema but empty for MVP.
Grammar and Media work without them (Grammar = POS filter, Media = deck membership).
Needs research before populating `term_categories`. Options:

**Option 1: WordNet domain labels**
Already in OMW, extractable in the existing pipeline. Free, no extra dependency.
Problem: coarse and patchy — many terms untagged, vocabulary is academic ("fauna" not "animals").

**Option 2: Princeton WordNet Domains**
Separate project built on WordNet. ~200 hierarchical domains mapped to synsets. More structured
and consistent than basic WordNet labels. Freely available. Meaningfully better than Option 1.

**Option 3: Kelly Project**
Frequency lists with CEFR levels AND semantic field tags, explicitly designed for language learning,
multiple languages. Could solve frequency tiers (`cefr_level`) and semantic categories in one shot.
Investigate coverage for your languages and POS range first.

**Option 4: BabelNet / WikiData**
Rich, multilingual, community-maintained. Maps WordNet synsets to Wikipedia categories.
Problem: complex integration, BabelNet has commercial licensing restrictions, WikiData category
trees are deep and noisy.

**Option 5: LLM-assisted categorization**
Run terms through Claude/GPT-4 with a fixed category list, spot-check output, import.
Fast and cheap at current term counts (3171 terms ≈ negligible cost). Not reproducible
without saving output. Good fallback if structured sources have insufficient coverage.

**Option 6: Hybrid — WordNet Domains as baseline, LLM gap-fill**
Use Option 2 for automated coverage, LLM for terms with no domain tag, manual spot-check pass.
Combines automation with control. Likely the most practical approach.

**Option 7: Manual curation**
Flat file mapping synset IDs to your own category slugs. Full control, matches UI exactly.
Too expensive at scale — only viable for small curated additions on top of an automated baseline.

**Current recommendation:** research Kelly Project first. If coverage is insufficient, go with Option 6.

---

## Current State

Phase 0 complete. Phase 1 data pipeline complete. Phase 2 data model finalized and migrated.

### Completed (Phase 1 — data pipeline)

- [x] Run `extract-en-it-nouns.py` locally → generates `datafiles/en-it-nouns.json`
- [x] Write Drizzle schema: `terms`, `translations`, `language_pairs`, `term_glosses`, `decks`, `deck_terms`
- [x] Write and run migration (includes CHECK constraints for `pos`, `gloss_type`)
- [x] Write `packages/db/src/seed.ts` (imports ALL terms + translations, NO decks)
- [x] Write `packages/db/src/generating-decks.ts` — idempotent deck generation script
  - reads and deduplicates source wordlist
  - matches words to DB terms (homonyms included)
  - writes unmatched words to `-missing` file
  - determines `validated_languages` by checking full translation coverage per language
  - creates deck if it doesn't exist, adds only missing terms on subsequent runs
  - recalculates and persists `validated_languages` on every run

### Completed (Phase 2 — data model)

- [x] `synset_id` removed, replaced by `source` + `source_id` on `terms`
- [x] `cefr_level` added to `translations` (not `terms` — difficulty is language-relative)
- [x] `language_code` CHECK constraint added to `translations` and `term_glosses`
- [x] `language_pairs` table dropped — pairs derived from decks at query time
- [x] `is_public` and `added_at` dropped from `decks` and `deck_terms`
- [x] `type` added to `decks` with CHECK against `SUPPORTED_DECK_TYPES`
- [x] `topics` and `term_topics` tables added (empty for MVP)
- [x] Migration generated and run against fresh database

### Known data facts (pre-wipe, for reference)

- Wordlist: 999 unique words after deduplication (1000 lines, 1 duplicate)
- Term IDs resolved: 3171 (higher than word count due to homonyms)
- Words not found in DB: 34
- Italian (`it`) coverage: 3171 / 3171 — full coverage, included in `validated_languages`

### Next (Phase 3 — data pipeline + API)

3. **Expand data pipeline** — import all OMW languages and POS, not just English nouns with Italian translations
4. **Decide SUBTLEX → `cefr_level` mapping strategy** — raw frequency ranks need a mapping to A1–C2 bands before tiered decks are meaningful
5. **Generate decks** — run generation script with SUBTLEX-grounded wordlists per source language
6. **Finalize game selection flow** — direction → category → POS → difficulty → round count
7. **Define Zod schemas in `packages/shared`** — based on finalized game flow and API shape
8. **Implement API**