# Lila Data Pipeline — Technical Documentation

Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer.
Last updated: 2026-07-18

## Table of Contents

- [1. Executive Summary](#1-executive-summary)
- [2. Problem & Context](#2-problem--context)
- [3. Architecture Overview](#3-architecture-overview)
- [4. Current Implementation](#4-current-implementation)
- [5. The LLM Layer](#5-the-llm-layer)
  - 5.1 [Local Model Evaluation](#51-local-model-evaluation)
  - 5.2 [Online API Options](#52-online-api-options)
  - 5.3 [Model Selection Criteria](#53-model-selection-criteria)
- [6. The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration)
- [7. Batching Strategy](#7-batching-strategy)
- [8. Hardware Constraints](#8-hardware-constraints)
- [9. Testing & Quality Assurance](#9-testing--quality-assurance)
- [10. Interactive CLI](#10-interactive-cli)
- [11. Future Extensions & Roadmap](#11-future-extensions--roadmap)
- [12. Decisions Log](#12-decisions-log)
- [13. Known Issues & Dev Notes](#13-known-issues--dev-notes)
- [14. How to Run](#14-how-to-run)
- [15. Roadmap](#15-roadmap)

## Quick Reference

| What                | Where                                      |
| ------------------- | ------------------------------------------ |
| Entry point         | `pipeline.ts`                              |
| Interactive CLI     | `utils/cli.ts`                             |
| LLM config schema   | `config/llm.ts`                            |
| System prompt       | `config/prompt.ts` — `buildSystemPrompt()` |
| Batch config schema | `config/batch.ts`                          |
| Shared constants    | `config/constants.ts`                      |
| Output schema       | `utils/merge-enriched-data.ts`             |
| LLM adapters        | `utils/llm-adapters/`                      |
| **Current model**   | **`gemma-4-E2B_q4_0-it.gguf`**             |
| Target scale        | 100,000+ words                             |

---

## 1. Executive Summary

The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:

- One or more senses (definitions)
- A natural example sentence per sense
- A CEFR-based difficulty level (`easy` / `medium` / `hard`)
- Translations into all target languages except the source (as raw strings)

The pipeline is designed to scale to 100,000+ words across multiple languages and parts of speech. It supports both local inference (`llama.cpp`) and cloud providers via a pluggable adapter pattern.

### Key Design Principles

| Principle                       | Rationale                                                                                              |
| ------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **Quality first**               | Definitions, examples, and translations must be accurate. Speed and cost are secondary.                |
| **Local-first, cloud-fallback** | Local LLMs are the default for cost control and data privacy.                                          |
| **Deterministic Grammar**       | Grammatical gender is decoupled from the LLM and resolved via Kaikki Wiktionary dumps.                 |
| **Resumable & idempotent**      | Each word writes to its own JSON file. The pipeline skips already-processed words on restart.          |
| **Configurable batching**       | Batch size is selected interactively at runtime. The pipeline adapts without code changes.             |
| **Honest metrics**              | Local models report detailed prompt/completion timing. Cloud providers report total request time only. |

### Resolved: Gender Accuracy

Grammatical gender is **no longer generated by the LLM**. Comprehensive testing across 10 models (2026-07-18) confirmed that small local models systematically hallucinate or default to `neuter` for Romance languages.
**Decision:** The LLM only outputs translation strings. A deterministic post-processing step looks up the exact grammatical gender from Kaikki Wiktionary dumps. This guarantees 100% gender accuracy and allows us to use smaller, faster, and highly nuanced local models.

### Current Status (2026-07-18)

- **Core pipeline:** Complete (scanning, enrichment, merging, verification, writing).
- **Local LLM:** Gemma 4 E2B selected as production model after exhaustive 10-model evaluation. `llama.cpp` server optimized with KV-cache quantization.
- **Batching:** 20-word batches validated for local hardware.
- **Gender:** Decoupled from LLM; Kaikki lookup architecture confirmed.
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) as speed fallback.

### One-Line Architecture

```text
source wordlists -> Interactive CLI -> LLM adapter -> merge senses -> Kaikki Gender Lookup -> verify schema -> write .json
```

### Files at a Glance

| File                                      | Purpose                                                                                               |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `pipeline.ts`                             | Orchestrator. Runs CLI, scans sources, loops words, coordinates all stages                            |
| `utils/cli.ts`                            | Interactive CLI. Provider/model/batch selection, config persistence                                   |
| `config/llm.ts`                           | LLM config schema (provider, url, model)                                                              |
| `config/prompt.ts`                        | `buildSystemPrompt()` — dynamic prompt with auto-target languages                                     |
| `config/batch.ts`                         | Batch size and max retry count schema                                                                 |
| `config/constants.ts`                     | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES`                                                         |
| `utils/enrich-word.ts`                    | Calls LLM via adapter, parses response, validates senses, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts`            | Merges skeleton + enriched senses into final JSON                                                     |
| `utils/verify-enriched-file.ts`           | Schema validation (required fields, types)                                                            |
| `utils/check-llm-server.ts`               | Health check for local server; skipped for cloud providers                                            |
| `utils/scanning-source-files.ts`          | Discovers wordlists from `source-data/` directory                                                     |
| `utils/create-base-json.ts`               | Writes skeleton `{word, language, pos}` files                                                         |
| `utils/write-json-file.ts`                | Atomic `.tmp` → rename writes                                                                         |
| `utils/check-if-json-exists.ts`           | Resumability: checks if word already has enriched senses                                              |
| `utils/create-line-reader.ts`             | Streaming line reader for large wordlists                                                             |
| `utils/create-output-dirs.ts`             | Creates `worddata/{language}/{pos}/` folders                                                          |
| `utils/delete-file.ts`                    | Cleanup helper for failed batches                                                                     |
| `utils/get-word-file-path.ts`             | Path construction helper                                                                              |
| `utils/progress-tracker.ts`               | `[current/total]` formatting for console output                                                       |
| `utils/pipeline-timer.ts`                 | Timing + token metrics; unified throughput for all providers                                          |
| `utils/llm-adapters/factory.ts`           | Creates the right adapter based on runtime config                                                     |
| `utils/llm-adapters/types.ts`             | `LlmAdapter` interface                                                                                |
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek                                                                 |
| `utils/llm-adapters/gemini.ts`            | Google Gemini native API                                                                              |

### Scale Target

| Metric          | Target                                                       |
| --------------- | ------------------------------------------------------------ |
| Words           | 100,000+                                                     |
| Languages       | English (source), German, Italian, Spanish, French (targets) |
| Parts of speech | Nouns, verbs, adjectives, adverbs                            |
| Output          | One `.json` file per word, ~2-5KB each                       |

---

## 2. Problem & Context

### Why Build This?

Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:

| What Exists              | What is Missing                    | Why It Matters                                                |
| ------------------------ | ---------------------------------- | ------------------------------------------------------------- |
| Word + translation       | Student-friendly definition        | Learners need explanations, not just equivalents              |
| Static difficulty labels | CEFR-calibrated difficulty         | "Easy" vs "hard" must map to learner proficiency levels       |
| Isolated translations    | Natural example sentences          | Context is how vocabulary is actually acquired                |
| Raw gender data          | Gender integrated with translation | Flashcards must show gender immediately (resolved via Kaikki) |
| Bulk word lists          | Per-word structured JSON           | The trainer consumes one file per word for fast random access |

### The Target User

A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender — all calibrated to their CEFR level (A1-C2).

### Why Not Use Existing Dictionaries?

- **Wiktionary:** Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
- **Kaikki (Wiktionary dump):** Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
- **Google Translate / DeepL:** No definitions, no examples, no difficulty, no structured output
- **Existing language learning apps:** Closed data, no export, no control over content

The LLM fills the gap: it generates pedagogical content (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.

### Language Direction

The pipeline is direction-agnostic. A wordlist is defined by:

- **Source language:** the language of the input words
- **Target languages:** all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source)

Current focus: English -> German/Italian/Spanish/French. Planned directions include German -> French, Italian -> Spanish, etc.

### Why 100,000+ Words?

- **Coverage:** A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels.
- **Languages:** English (source) -> German, Italian, Spanish, French (targets).
- **Parts of speech:** Nouns (current), verbs, adjectives, adverbs.

### The Quality Challenge

| Risk                            | Mitigation                                                     |
| ------------------------------- | -------------------------------------------------------------- |
| Hallucinated definitions        | Low temperature (0.1), strict system prompt, schema validation |
| Incorrect grammatical gender    | **Resolved:** Decoupled from LLM; Kaikki Wiktionary lookup     |
| POS bleed (verb defs for nouns) | Explicit negative constraint in system prompt                  |
| Inconsistent difficulty levels  | Explicit CEFR mapping in prompt, spot-checking                 |
| JSON parse failures             | Retry + split logic, schema validation, cleanup on failure     |
| Model drift (online APIs)       | Version pinning, local fallback                                |
| Provider downtime               | Adapter pattern allows hot-swapping providers                  |

### Why TypeScript + Node?

- **Familiarity:** Existing project uses TypeScript (frontend in TanStack Router + React)
- **Ecosystem:** `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP
- **Portability:** Runs on the same Debian laptop as the llama.cpp server
- **No build complexity:** `tsx` for direct execution, no bundler needed

### Why llama.cpp?

- **GGUF format:** Single-file models, easy to swap, quantize, and version
- **OpenAI-compatible API:** `/v1/chat/completions` means the same adapter code works for local and online models
- **No dependencies:** Self-contained binary, runs on old hardware (tested on GTX 950M)
- **Privacy:** Local inference means no data leaves the machine

---

## 3. Architecture Overview

### Pipeline Flow

```text
Run CLI -> Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
    -> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
    -> Retry/split on failure -> Merge -> Kaikki Gender Lookup -> Write atomically -> Verify schema -> Log metrics
```

### Resumability

- **Skip existing:** `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses`
- **Atomic writes:** `.tmp` -> rename in `write-json-file.ts`, no partial files on crash
- **Cleanup on failure:** Deletes partially-written files for failed batches, continues to next batch

### Directory Structure

```text
data-pipeline/
 |-- pipeline.ts              # Entry point / orchestrator
 |-- utils/
 |   |-- cli.ts               # Interactive CLI module
 |   |-- enrich-word.ts       # LLM call, parse, retry/split
 |   |-- merge-enriched-data.ts  # Schema types + merge logic
 |   |-- verify-enriched-file.ts # Schema validation
 |   |-- check-llm-server.ts  # Health check (local only)
 |   |-- scanning-source-files.ts # Source discovery
 |   |-- create-base-json.ts  # Skeleton writer
 |   |-- write-json-file.ts   # Atomic JSON writer
 |   |-- check-if-json-exists.ts # Resumability check
 |   |-- create-line-reader.ts # Streaming file reader
 |   |-- create-output-dirs.ts # Directory creation
 |   |-- delete-file.ts       # Cleanup helper
 |   |-- get-word-file-path.ts # Path helper
 |   |-- progress-tracker.ts  # Console progress formatting
 |   |-- pipeline-timer.ts    # Timing + token metrics
 |   |-- llm-adapters/
 |       |-- factory.ts       # Adapter selection
 |       |-- types.ts         # LlmAdapter interface
 |       |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
 |       |-- gemini.ts        # Google Gemini
 |-- config/
 |   |-- llm.ts               # LLM config schema
 |   |-- prompt.ts            # buildSystemPrompt()
 |   |-- batch.ts             # Batch config schema
 |   |-- constants.ts         # LANG_MAP, POS_MAP, ALL_LANGUAGES
 |-- source-data/
 |   |-- {language}/
 |       |-- {pos}            # One word per line, no extension
 |-- worddata/
 |   |-- {language}/
 |       |-- {pos}/
 |           |-- {word}.json  # One self-contained file per word
 |-- kaikki-source-files/     # Wiktionary dumps for gender lookup
 |-- .pipeline-config.json    # Saved CLI configuration
```

### Output Schema

Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `id`, `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`.
_Note: The `translations` object contains arrays of strings (e.g., `{"de": ["Haus", "Gebäude"]}`). Grammatical gender is appended later via the Kaikki integration step._

### Error Handling

| Failure                        | Behavior                                                 |
| ------------------------------ | -------------------------------------------------------- |
| LLM server offline             | Hard fail at startup (`check-llm-server.ts`, local only) |
| LLM returns bad JSON           | Retry up to 3 times, then split batch. Log and continue  |
| LLM returns malformed senses   | `validateSense()` catches it before file write           |
| Schema validation fails        | Log warnings, keep file                                  |
| Individual batch fails         | Does not stop pipeline; cleans up partial files          |
| Individual word fails (size 1) | Log and continue to next word                            |

### Metrics

Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`.

- **Unified throughput (all providers):** total tokens / total request time
- **Detailed breakdown (local only):** prompt speed vs completion speed

---

## 4. Current Implementation

### Tech Stack

| Layer       | Choice                        | Why                                        |
| ----------- | ----------------------------- | ------------------------------------------ |
| Runtime     | Node.js + `tsx`               | Direct TypeScript execution, no build step |
| HTTP client | Native `fetch`                | Works for local llama.cpp and online APIs  |
| File I/O    | `fs` + `readline`             | Streaming line reader for large wordlists  |
| JSON        | Native `JSON.parse/stringify` | Simple, no schema library needed           |
| CLI         | Native `readline`             | No external dependencies                   |

### Current Model

| Property         | Value                                                |
| ---------------- | ---------------------------------------------------- |
| **Model**        | **`gemma-4-E2B_q4_0-it.gguf`**                       |
| Size             | ~3.2GB (file) / ~2.06GB (VRAM weights)               |
| Quantization     | Q4_0                                                 |
| Server           | `llama.cpp` (`llama-server`)                         |
| API              | OpenAI-compatible `/v1/chat/completions`             |
| VRAM Usage       | ~2.65GB total (weights + KV cache + compute buffers) |
| Generation Speed | ~10.9 tok/s (20-word batch)                          |
| Est. 100k Time   | ~7 days (20-word batches, 24/7)                      |

### `llama-server` Flags: History & Rationale

The server flags evolved through rigorous empirical testing on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM) across 10 different models on 2026-07-18.

#### Flag Evolution

| Flag              | Value Tried                                | Result                            | Why                                                                                                                                              |
| ----------------- | ------------------------------------------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-m`              | `qwen3.5-4b-q4_k_m.gguf`                   | Works, ~6.3 tok/s                 | Quality baseline. Correct translations. 2.6GB, tight on VRAM.                                                                                    |
| `-m`              | `qwen2.5-1.5b-instruct-q4_k_m.gguf`        | Works, ~18.5 tok/s                | Fast but poor instruction following, gender wrong.                                                                                               |
| `-m`              | `Qwen3.5-2B-Q4_K_M.gguf`                   | Works, ~13.2 tok/s                | Good translations, but failed polysemy (2 identical senses).                                                                                     |
| `-m`              | `Llama-3.2-3B-Instruct-Q4_K_M.gguf`        | Works, ~9.1 tok/s                 | Dangerous false friend trap (cognates).                                                                                                          |
| `-m`              | `Llama-3.2-3B-Instruct-UD-Q6_K_XL.gguf`    | Works, ~6.7 tok/s, 3.44GB VRAM    | Higher quant did NOT fix Llama's translation issues. Slower.                                                                                     |
| `-m`              | `Ministral-3-3B-Instruct-2512-Q4_K_M.gguf` | Works, ~8.9 tok/s                 | Fixed tokenizer bug (older version was broken). But messy translations, markdown violations.                                                     |
| `-m`              | `gemma-4-E2B_q4_0-it.gguf`                 | **Works, ~13.1 tok/s, 2.06GB**    | **Winner.** Perfect polysemy, false friends, nuance. Half VRAM of Qwen 4B.                                                                       |
| `-m`              | `gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf`       | Works, ~7.7 tok/s, 3.23GB         | QAT compression is incredible. Same quality as E2B but slower.                                                                                   |
| `-m`              | `qwen3.5-9b-q3_k_s.gguf`                   | Works, ~3.3 tok/s (split CPU/GPU) | Brilliant quality but bottlenecked by CPU/Swap. `-ngl 28` max.                                                                                   |
| `-m`              | `qwen3.5-9b-q4_k_m.gguf`                   | Works, ~3.3 tok/s (split CPU/GPU) | Same quality as 9B Q3. Not worth 2x the file size.                                                                                               |
| `-ngl`            | 999                                        | Keeps                             | Offload all layers to GPU. Required for any speed.                                                                                               |
| `-ngl`            | 28                                         | 9B models only                    | Max GPU layers for 9B models before OOM. Found via binary search.                                                                                |
| `-ngl`            | 30                                         | OOM crash (9B)                    | Pushed 2 layers too far into compute buffers.                                                                                                    |
| `-c`              | 4096                                       | Wasteful                          | 4K context for 300-token dictionary entries wastes VRAM.                                                                                         |
| `-c`              | 2048                                       | Good for small batches            | Sufficient for 4-word batches.                                                                                                                   |
| `-c`              | **8192**                                   | **Current**                       | Required for 20-word batches. Combined with KV cache quantization.                                                                               |
| `-b` / `-ub`      | 512                                        | **Current**                       | Sweet spot for Maxwell memory bandwidth.                                                                                                         |
| `-b` / `-ub`      | 1024                                       | Tested                            | Slightly faster prompt processing, but no generation speedup.                                                                                    |
| `-b` / `-ub`      | 2048                                       | Slower on 950M                    | Memory pressure on bandwidth-starved GPU.                                                                                                        |
| `-t`              | 4                                          | Slower                            | Hyperthreading cores hurt llama.cpp performance.                                                                                                 |
| `-t`              | **2**                                      | **Current**                       | Matches 2 physical cores.                                                                                                                        |
| `--threads-batch` | **2**                                      | **Current**                       | Explicit match to `-t`.                                                                                                                          |
| `--flash-attn`    | (omitted)                                  | Correct                           | On Maxwell (compute 5.0), Flash Attention adds overhead.                                                                                         |
| `--mlock`         | Tested                                     | Omitted for large models          | Pins model in RAM. Causes OOM on models >2.5GB with large KV cache.                                                                              |
| `--prio`          | **2**                                      | **Current**                       | Raises process priority. Marginal, harmless.                                                                                                     |
| `--reasoning`     | **off**                                    | **Critical**                      | **Mandatory for Qwen 3.5 and Gemma 4.** Without this, models "think" silently, consume all `max_tokens`, and crash with `finish_reason: length`. |
| `--cache-type-k`  | **q4_0**                                   | **Current**                       | Compresses KV cache keys to 4-bit. Cuts KV VRAM by ~75%. Enables 8192 context on 4GB GPU.                                                        |
| `--cache-type-v`  | **q4_0**                                   | **Current**                       | Compresses KV cache values to 4-bit. Paradoxically improves translation variety (reduces "lazy duplication" bug).                                |

#### Current Production Command

```bash
cd ~/Downloads/llama.cpp
./build/bin/llama-server \
  -m models/gemma-4-E2B_q4_0-it.gguf \
  -ngl 999 \
  -c 8192 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --prio 2 \
  --reasoning off \
  --cache-type-k q4_0 \
  --cache-type-v q4_0
```

#### VRAM Budget (Production Config)

| Component                            | VRAM Usage                |
| ------------------------------------ | ------------------------- |
| Model Weights (Gemma 4 E2B Q4_0)     | ~1.50 GB                  |
| KV Cache (8192 ctx, q4_0 compressed) | ~0.35 GB                  |
| Compute Buffers (batch 512)          | ~0.20 GB                  |
| **Total**                            | **~2.05 GB (51% of 4GB)** |

#### Why Not Use the Remaining 2GB VRAM?

Generation speed is bottlenecked by **memory bandwidth** (~32 GB/s on GTX 950M), not VRAM capacity. To generate one token, the GPU must read the entire ~1.5GB model from VRAM. The theoretical maximum is ~21 tok/s. At 10.9 tok/s, the GPU is already operating at ~50% of its physical limit. Empty VRAM cannot be converted into faster generation.

Testing uncompressed `f16` KV cache (2.65GB total VRAM) yielded 12.6 tok/s but caused a severe "lazy duplication" regression (model copy-pasted the same translation twice instead of providing distinct synonyms). The `q4_0` compressed KV cache is the correct choice for translation quality.

### Performance Baseline (20-Word Batch)

| Metric                | Gemma 4 E2B | Qwen 3.5 4B  |
| --------------------- | ----------- | ------------ |
| Time/batch (20 words) | ~5 min      | ~9 min       |
| Completion tok/s      | ~10.9       | ~6.1         |
| Prompt tok/s          | ~132        | ~78          |
| VRAM Usage            | 2.06 GB     | 3.95 GB      |
| Lazy Duplication Bug  | No          | Yes (severe) |

---

## 5. The LLM Layer

### 5.1 Local Model Evaluation (Complete — 2026-07-18)

All 10 downloaded models were evaluated on the target hardware. Testing progressed from 4-word smoke tests to a 20-word "nightmare" torture suite covering extreme polysemy, false friends, abstract concepts, and legal/financial terminology.

#### Final Leaderboard

| Rank   | Model                    | Size | VRAM          | Speed (tok/s) | Polysemy                    | False Friends             | Verdict                                                     |
| ------ | ------------------------ | ---- | ------------- | ------------- | --------------------------- | ------------------------- | ----------------------------------------------------------- |
| **🥇** | **Gemma 4 E2B Q4_0**     | 3.2G | **2.06 GB**   | **~13.1**     | **Perfect**                 | **Perfect**               | **Production model.** Best quality/speed/VRAM ratio.        |
| 🥈     | Qwen 3.5 4B Q4_K_M       | 2.6G | ~3.6 GB       | ~6.3          | Perfect                     | Perfect                   | Quality King, but 2x slower and maxes VRAM.                 |
| 🥉     | Gemma 4 E4B Q4_K_XL      | 4.0G | 3.23 GB       | ~7.7          | Perfect                     | Perfect                   | Incredible QAT compression. Same quality as E2B but slower. |
| 4      | Qwen 3.5 2B Q4_K_M       | 1.2G | ~1.6G         | ~13.2         | Failed (2 identical senses) | Passed                    | Good translations, lacks conceptual branching.              |
| 5      | Qwen 2.5 1.5B Q4_K_M     | 1.1G | ~1.5G         | ~18.5         | Ignored instruction         | Failed                    | Fast but easily confused.                                   |
| 6      | Ministral 3B 2512 Q4_K_M | 2.0G | ~2.7G         | ~8.9          | Messy translations          | Failed                    | Fixed tokenizer, but outclassed. Markdown violations.       |
| 7      | Llama 3.2 3B Q4_K_M      | 1.9G | ~2.6G         | ~9.1          | Good structure, bad IT/ES   | **Failed (cognate trap)** | Dangerous for language learners.                            |
| 8      | Llama 3.2 3B Q6_K_XL     | 2.8G | 3.44G         | ~6.7          | Good structure, bad IT/ES   | **Failed (cognate trap)** | Higher quant did NOT fix translation issues.                |
| 9      | Qwen 3.5 9B Q3_K_S       | 4.1G | Split CPU/GPU | ~3.3          | Perfect                     | Perfect                   | Brilliant but 35-40 days for 100k words.                    |
| 10     | Qwen 3.5 9B Q4_K_M       | 5.3G | Split CPU/GPU | ~3.3          | Perfect                     | Perfect                   | Same quality as 9B Q3. Not worth the size.                  |

#### Key Findings

1. **The "Thinking" Trap:** Both Qwen 3.5 and Gemma 4 have built-in Chain-of-Thought reasoning. Without `--reasoning off`, they silently "think" in a hidden JSON field, consume all `max_tokens`, and crash with `finish_reason: length`. This flag is **mandatory**.
2. **The 2B vs 4B Quality Cliff:** 2B models struggle with polysemy (e.g., cannot distinguish "bank" = financial vs river). 4B+ models act like professional lexicographers.
3. **Llama 3.2 is Unsafe for Language Learners:** Consistently fell for false friend traps (e.g., translating "actual" = _real_ to _aktuell/attuale/actual/actuel_ = _current_).
4. **KV Cache Quantization Improves Translation Variety:** Compressing the KV cache to `q4_0` introduces microscopic noise that prevents the "lazy duplication" bug (model copy-pasting the same synonym twice).
5. **POS Bleed is Universal:** All models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_ instead of _a jogging session_). Fix: explicit negative constraint in the system prompt.

### 5.2 Online API Options

Evaluated as fallbacks if local models fail quality or speed targets.

| Provider        | Model                | Input $/1M | Output $/1M | Free Tier     | Rate Limit | Est. Cost (100k words) | Est. Time           |
| --------------- | -------------------- | ---------- | ----------- | ------------- | ---------- | ---------------------- | ------------------- |
| DeepSeek        | V4 Flash             | $0.14      | $0.28       | 5M tokens     | None       | $0-15                  | ~1-2 days           |
| Gemini          | 2.5 Flash-Lite       | $0.10      | $0.40       | 1,500 req/day | 1M TPM     | $0 (free tier)         | ~1.5 days (batched) |
| Qwen/Alibaba    | Qwen-Turbo           | $0.05      | $0.20       | Unknown       | 600 RPM    | ~$11                   | ~1-2 days           |
| Groq            | Llama 3.1 8B Instant | $0.05      | $0.08       | Yes           | High       | ~$7                    | ~3-4 hours          |
| OpenRouter free | Various              | $0         | $0          | 200 req/day   | 20 RPM     | $0                     | ~10 days            |

### 5.3 Model Selection Criteria

Decision flow for 100,000 words:

```text
Start
   |
   v
 Gemma 4 E2B (local) — SELECTED
   |
   |-- Speed acceptable? (~7 days) -----> Use Gemma 4 E2B locally, $0
   |
   |-- Need faster? -------> Test Gemini 2.5 Flash-Lite (free)
                               |
                               |-- Quality good? --> Batch 50, free tier
                               |                      ~1.5 days, $0
                               |
                               |-- Quality meh? ---> Test Groq or DeepSeek paid
```

Quality gates:

- 100% JSON parse rate
- No hallucinated definitions on polysemous words
- Natural, contextually appropriate example sentences
- Sensible difficulty classification (CEFR mapping)
- Gender accuracy is no longer an LLM criterion (handled by Kaikki)

---

## 6. The Gender Problem & Kaikki Integration

### The Problem (Resolved)

Grammatical gender was originally embedded in the LLM's `translations` object. Testing across all 10 models confirmed that small local models systematically hallucinate gender, defaulting to `neuter` for Romance languages (Italian, Spanish, French) which do not have a neuter grammatical gender.

### The Solution: Decoupled Architecture

**Decision (2026-07-18):** Grammatical gender is no longer generated by the LLM. The pipeline now uses a two-stage approach:

| Stage | Component         | Responsibility                                                       |
| ----- | ----------------- | -------------------------------------------------------------------- |
| 1     | LLM (Gemma 4 E2B) | Generates translation **strings only** (e.g., `["Haus", "Gebäude"]`) |
| 2     | Kaikki Lookup     | Deterministically resolves grammatical gender from Wiktionary dumps  |

### Benefits

- **100% deterministic gender accuracy** — no hallucination possible
- **Faster LLM generation** — ~15-20% fewer output tokens per word
- **Simpler JSON schema** — translations are string arrays, not object arrays
- **Model-agnostic** — works with any LLM regardless of multilingual training quality

### Kaikki Data

| Language | Kaikki File                           | Coverage |
| -------- | ------------------------------------- | -------- |
| German   | `kaikki.org-dictionary-German.jsonl`  | High     |
| Italian  | `kaikki.org-dictionary-Italian.jsonl` | High     |
| Spanish  | `kaikki.org-dictionary-Spanish.jsonl` | High     |
| French   | `kaikki.org-dictionary-French.jsonl`  | High     |

### Lookup Logic

Match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`.

---

## 7. Batching Strategy

### Why Batching is Necessary

At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.

### Optimal Batch Size (Local)

**20 words** is the validated sweet spot for the GTX 950M with Gemma 4 E2B.

| Batch Size | VRAM            | Speed            | Quality                  | Verdict                                            |
| ---------- | --------------- | ---------------- | ------------------------ | -------------------------------------------------- |
| 1-4        | ~2.1 GB         | ~13 tok/s        | Perfect                  | Safe but slow (amortization waste)                 |
| **20**     | **~2.6 GB**     | **~10.9 tok/s**  | **Perfect**              | **Sweet spot**                                     |
| 30-40      | ~3.0 GB (est.)  | ~10 tok/s (est.) | Likely good              | Worth testing                                      |
| 50+        | ~3.5 GB+ (est.) | Unknown          | Risk of JSON degradation | **Not recommended for 2B models**                  |
| 100        | OOM risk        | N/A              | Attention degradation    | Small models lose JSON structure past ~6000 tokens |

### Retry & Split Strategy

If a batch fails (bad JSON, missing key, etc.):

```text
Batch of 20 fails (3 retries exhausted)
    |
    v
Split into 2 batches of 10
    |
    v
If a batch of 10 fails (3 retries), split into 2 batches of 5
    |
    v
If a batch of 5 fails, split into batches of 1
    |
    v
If a single word fails (3 retries), log and skip
```

---

## 8. Hardware Constraints

### Current Machine

| Component     | Spec                                                                |
| ------------- | ------------------------------------------------------------------- |
| OS            | Debian GNU/Linux 13 (trixie) x86_64                                 |
| CPU           | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz |
| GPU           | NVIDIA GeForce GTX 950M (Maxwell, 2015)                             |
| GPU VRAM      | 4GB (4037 MiB reported by CUDA)                                     |
| GPU Bandwidth | ~32 GB/s (DDR3)                                                     |
| RAM           | 8GB (~3.67GB usable at idle)                                        |
| Swap          | 5.62 GB                                                             |
| Disk          | 102GB ext4 (~74GB used)                                             |

### What Fits in 4GB VRAM (Empirically Verified)

| Model                    | File Size | Total VRAM  | Fits?      | Notes                                  |
| ------------------------ | --------- | ----------- | ---------- | -------------------------------------- |
| Qwen 2.5 1.5B Q4_K_M     | 1.1G      | ~1.5 GB     | ✅ Easy    |                                        |
| Qwen 3.5 2B Q4_K_M       | 1.2G      | ~1.6 GB     | ✅ Easy    |                                        |
| Ministral 3B 2512 Q4_K_M | 2.0G      | ~2.7 GB     | ✅ Yes     |                                        |
| Llama 3.2 3B Q4_K_M      | 1.9G      | ~2.6 GB     | ✅ Yes     |                                        |
| Llama 3.2 3B Q6_K_XL     | 2.8G      | 3.44 GB     | ✅ Tight   |                                        |
| **Gemma 4 E2B Q4_0**     | **3.2G**  | **2.06 GB** | **✅ Yes** | **QAT compression. Production model.** |
| Gemma 4 E4B Q4_K_XL      | 4.0G      | 3.23 GB     | ✅ Yes     | QAT compression is incredible.         |
| Qwen 3.5 4B Q4_K_M       | 2.6G      | 3.95 GB     | ⚠️ Barely  | 50MB headroom with 8192 ctx.           |
| Qwen 3.5 9B Q3_K_S       | 4.1G      | Split       | ⚠️ Partial | `-ngl 28` max. Rest on CPU/Swap.       |
| Qwen 3.5 9B Q4_K_M       | 5.3G      | Split       | ⚠️ Partial | `-ngl 20` max. Heavy swap usage.       |

### GPU Rental Alternatives

If local hardware becomes the bottleneck:
| Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost |
| --- | --- | --- | --- | --- | --- |
| Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | ~$2-5 |
| RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | ~$4-6 |

---

## 9. Testing & Quality Assurance

### 20-Word Torture Suite (Completed 2026-07-18)

Tested on Gemma 4 E2B and Qwen 3.5 4B with 20 challenging nouns:

| Category          | Words                                                   | The Trap                                                       |
| ----------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
| Extreme Polysemy  | `match`, `date`, `right`, `set`, `well`                 | Does it translate "match" as fire, sports, or dating?          |
| False Friends     | `sense`, `fabric`, `sympathy`, `eventuality`, `billion` | "Fabric" = material (_tissu_) not factory (_fabrique_)         |
| Abstract/Cultural | `serendipity`, `accountability`                         | Concepts lacking 1:1 dictionary equivalents                    |
| Action-Nouns      | `run`, `drive`, `play`                                  | "Run" as jog vs tear vs campaign                               |
| Legal/Financial   | `mortgage`, `lease`, `court`, `board`, `draft`          | Requires specific legal vocabulary (_Hypothek/mutuo/hipoteca_) |

### Results Summary

| Criterion                            | Gemma 4 E2B                                | Qwen 3.5 4B                                             |
| ------------------------------------ | ------------------------------------------ | ------------------------------------------------------- |
| JSON Reliability                     | 10/10 (raw JSON)                           | 10/10 (raw JSON, but added `sense_index` hallucination) |
| Polysemy (bank, match)               | 10/10 (Ufer/riva/orilla/rive)              | 10/10                                                   |
| False Friends (fabric)               | 10/10 (Stoff/tessuto/tela/tissu)           | 10/10                                                   |
| Legal Nuance (mortgage)              | 10/10 (Hypothek/mutuo/hipoteca/hypothèque) | 10/10                                                   |
| Lazy Duplication Bug                 | None                                       | **Severe** (copy-pasted same word 20+ times)            |
| POS Bleed                            | Minor (verb defs for run/match)            | Minor (verb defs for run/match)                         |
| Attention Degradation (word 20 vs 1) | None                                       | None                                                    |

### Verification

`verify-enriched-file.ts` checks:

- Required top-level fields: `word`, `language`, `pos`, `senses`
- Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard}
- Each translation: array of strings (gender appended later by Kaikki)

---

## 10. Interactive CLI

### Batch Size Recommendations (Updated)

| Provider             | Recommended | Rationale                                       |
| -------------------- | ----------- | ----------------------------------------------- |
| **local (GTX 950M)** | **20**      | Validated sweet spot. 8192 ctx + q4_0 KV cache. |
| local (RTX 4090)     | 50          | Fast, more VRAM                                 |
| gemini               | 50          | Free tier: 1,500 req/day                        |
| deepseek             | 20          | 5M free tokens                                  |
| groq                 | 50          | Very fast                                       |

---

## 11. Future Extensions & Roadmap

### Near-Term (Next 2-4 Weeks)

| Item                                | Status       | Notes                                               |
| ----------------------------------- | ------------ | --------------------------------------------------- |
| 10-model evaluation                 | **Complete** | Gemma 4 E2B selected                                |
| KV cache quantization               | **Complete** | `--cache-type-k/v q4_0` enables 8192 ctx on 4GB GPU |
| Gender decoupling                   | **Complete** | Kaikki lookup replaces LLM gender                   |
| 20-word torture suite               | **Complete** | Validated on Gemma E2B and Qwen 4B                  |
| POS bleed fix                       | **Pending**  | Add negative constraint to system prompt            |
| Kaikki gender lookup implementation | **Pending**  | Post-processing step after LLM enrichment           |
| Online API testing                  | **Pending**  | Gemini free tier, DeepSeek, Groq                    |

### Medium-Term (1-3 Months)

| Item                         | Notes                                                  |
| ---------------------------- | ------------------------------------------------------ |
| Multi-POS support            | Verbs, adjectives, adverbs need prompt variants        |
| Multi-language source        | German -> French, Italian -> Spanish, etc.             |
| Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously |
| Incremental enrichment       | Only process new/changed words in a wordlist           |

### Long-Term (3-6 Months)

| Item                     | Notes                                                            |
| ------------------------ | ---------------------------------------------------------------- |
| GPU rental integration   | Script to spin up Vast.ai/RunPod, run pipeline, download results |
| Quality regression tests | Run torture suite on every model change                          |
| Community contributions  | Open-source the pipeline for other language learners             |

---

## 12. Decisions Log

| Date           | Decision                            | Context                                                         | Rationale                                                                                         |
| -------------- | ----------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| 2026-01-04     | TanStack Router for frontend        | Previous project used React Router                              | Simpler, type-safe routing                                                                        |
| 2026-06-16     | llama.cpp for local LLM             | Need local inference on old laptop                              | GGUF format, OpenAI-compatible API                                                                |
| 2026-06-16     | Q4_K_M quantization                 | Balance size vs quality                                         | Community standard for 4-bit                                                                      |
| 2026-06-16     | `-t 2` physical cores               | Default was 4 (HT threads)                                      | Hyperthreading hurts llama.cpp                                                                    |
| 2026-06-17     | Qwen2.5-1.5B as initial model       | Qwen3.5-4B too slow (47s/word)                                  | 6x speedup, quality under evaluation                                                              |
| 2026-07-06     | Adapter pattern for LLM providers   | Need to evaluate local vs cloud                                 | `utils/llm-adapters/` with factory                                                                |
| 2026-07-06     | Retry + split batching              | LLM JSON parse failures                                         | 3 retries, then halve batch                                                                       |
| 2026-07-06     | Interactive CLI                     | Editing config files is error-prone                             | `utils/cli.ts` with native readline                                                               |
| **2026-07-18** | **Gemma 4 E2B as production model** | **10-model evaluation completed**                               | **2x faster than Qwen 4B, half VRAM, perfect translation quality**                                |
| **2026-07-18** | **Gender decoupled from LLM**       | **All 10 models failed gender for Romance languages**           | **Kaikki Wiktionary lookup is deterministic and 100% accurate**                                   |
| **2026-07-18** | **`--reasoning off` is mandatory**  | **Qwen 3.5 and Gemma 4 "think" silently, consuming all tokens** | **Without this flag, output crashes with `finish_reason: length`**                                |
| **2026-07-18** | **KV cache quantization (`q4_0`)**  | **8192 context needed for 20-word batches**                     | **Cuts KV VRAM by 75%, enables large batches on 4GB GPU, improves translation variety**           |
| **2026-07-18** | **20-word batch size for local**    | **Tested 4, 20 words**                                          | **Sweet spot: no attention degradation, no JSON breakage, 10.9 tok/s**                            |
| **2026-07-18** | **Llama 3.2 discarded**             | **Failed false friend tests**                                   | **Translates "actual" (real) to cognates (aktuell/attuale) = "current". Dangerous for learners.** |
| **2026-07-18** | **`-c 8192` replaces `-c 2048`**    | **20-word batches need more context**                           | **Combined with q4_0 KV cache, fits in 2.65GB VRAM**                                              |

---

## 13. Known Issues & Dev Notes

### Data Pipeline

| Issue                          | Details                                                                                | Severity                                        |
| ------------------------------ | -------------------------------------------------------------------------------------- | ----------------------------------------------- |
| POS bleed (all models)         | Models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_) | Medium — fix with negative constraint in prompt |
| Lazy duplication (Qwen 4B)     | Qwen 3.5 4B copy-pastes the same translation twice to fill arrays                      | Medium — use Gemma 4 E2B instead                |
| Schema hallucination (Qwen 4B) | Adds `sense_index` field not in schema                                                 | Low — ignorable                                 |
| Pre-scanning wordlists         | Entire file read into memory before processing                                         | Medium — streaming refactor planned             |
| Single POS tested              | Only nouns validated. Verbs/adjectives need prompt changes                             | Known limitation                                |

### Hardware

| Issue                 | Details                                                                      |
| --------------------- | ---------------------------------------------------------------------------- |
| GTX 950M VRAM ceiling | 4GB hard limit. Models >3.5GB need KV cache quantization or CPU offloading.  |
| Maxwell GPU aging     | No Flash Attention, bandwidth-starved (~32 GB/s). Theoretical max ~21 tok/s. |
| Laptop thermals       | Cannot run 24/7 for weeks unattended. Monitor temps.                         |

---

## 14. How to Run

### Prerequisites

- Node.js + npm
- `tsx` installed globally: `npm install -g tsx`
- llama.cpp built from source (for local mode)
- GGUF model downloaded to `~/Downloads/llama.cpp/models/`
- API keys set as environment variables (for cloud mode)

### Start the LLM Server (Local Mode)

```bash
cd ~/Downloads/llama.cpp
./build/bin/llama-server \
  -m models/gemma-4-E2B_q4_0-it.gguf \
  -ngl 999 \
  -c 8192 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --prio 2 \
  --reasoning off \
  --cache-type-k q4_0 \
  --cache-type-v q4_0
```

### Run the Pipeline

```bash
cd /path/to/data-pipeline
npx tsx pipeline.ts
```

Follow the interactive prompts to select provider, model, and batch size.

---

## 15. Roadmap

### Phase 1: Batching (Complete)

| Task                                    | Status       |
| --------------------------------------- | ------------ |
| Implement configurable batch size       | Complete     |
| Implement retry + split logic           | Complete     |
| Honest timing metrics                   | Complete     |
| Validate LLM responses                  | Complete     |
| Verify batching quality (20-word suite) | **Complete** |

### Phase 2: Interactive CLI (Complete)

| Task                  | Status   |
| --------------------- | -------- |
| Design prompt flow    | Complete |
| Implement CLI module  | Complete |
| Save/load config      | Complete |
| Wire into pipeline.ts | Complete |

### Phase 3: Model Selection (Complete)

| Task                    | Status                     |
| ----------------------- | -------------------------- |
| 10-model evaluation     | **Complete**               |
| 20-word torture suite   | **Complete**               |
| Select production model | **Complete (Gemma 4 E2B)** |
| Test online APIs        | Pending                    |

### Phase 4: Scale

| Task                           | Status  | Notes                                      |
| ------------------------------ | ------- | ------------------------------------------ |
| Implement Kaikki gender lookup | Pending | Post-processing step                       |
| Fix POS bleed in prompt        | Pending | Add negative constraint                    |
| Run 100k word pipeline         | Pending | ~7 days local (Gemma E2B, 20-word batches) |
| Spot-check output quality      | Pending | Random sample of 100 entries               |

### Phase 5: Extend

| Task                         | Status  |
| ---------------------------- | ------- |
| Multi-POS support            | Pending |
| Multi-language source        | Pending |
| Parallel wordlist processing | Pending |
| GPU rental integration       | Pending |
| Quality regression tests     | Pending |