# Lila Data Pipeline — Technical Documentation

> Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer.
> Last updated: 2026-07-06

---

## Table of Contents

1. [Executive Summary](#1-executive-summary)
2. [Problem & Context](#2-problem--context)
3. [Architecture Overview](#3-architecture-overview)
4. [Current Implementation](#4-current-implementation)
5. [The LLM Layer](#5-the-llm-layer)
   - 5.1 [Local Model Evaluation](#51-local-model-evaluation)
   - 5.2 [Online API Options](#52-online-api-options)
   - 5.3 [Model Selection Criteria](#53-model-selection-criteria)
6. [The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration)
7. [Batching Strategy](#7-batching-strategy)
8. [Hardware Constraints](#8-hardware-constraints)
9. [Testing & Quality Assurance](#9-testing--quality-assurance)
10. [Future Extensions & Roadmap](#10-future-extensions--roadmap)
11. [Decisions Log](#11-decisions-log)
12. [Known Issues & Dev Notes](#12-known-issues--dev-notes)
13. [How to Run](#13-how-to-run)
14. [Roadmap](#14-roadmap)

---

## Quick Reference

| What          | Where                               |
| ------------- | ----------------------------------- |
| Entry point   | `pipeline.ts`                       |
| LLM config    | `config/llm.ts`                     |
| System prompt | `config/prompt.ts`                  |
| Batch config  | `config/batch.ts`                   |
| Output schema | `utils/merge-enriched-data.ts`      |
| LLM adapters  | `utils/llm-adapters/`               |
| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
| Target scale  | 100,000+ words                      |

---

## 1. Executive Summary

The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:

- One or more **senses** (definitions)
- A **natural example sentence** per sense
- A **CEFR-based difficulty level** (`easy` / `medium` / `hard`)
- **Translations** into German, Italian, Spanish, and French, each with grammatical **gender**

The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It is currently in active development: the core architecture is stable, the LLM integration layer supports both local and cloud providers via a pluggable adapter pattern, and a configurable batching system with retry/split logic is fully implemented.

### Key Design Principles

| Principle                       | Rationale                                                                                                                      |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| **Quality first**               | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary.                                |
| **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. |
| **Resumable & idempotent**      | Each word writes to its own JSON file. The pipeline skips already-processed words on restart.                                  |
| **Configurable batching**       | Batch size is a single config value (`config/batch.ts`). The pipeline adapts without code changes.                             |
| **Provider-agnostic**           | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface.                                       |

### Open Question: Gender Accuracy

Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that **Qwen2.5-1.5B systematically defaults to `neuter`** for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a **model size issue** (fixable by moving to 3B+) or a **training data issue** (requiring an external lookup) is unresolved.

**Options under evaluation:**

- Larger local models (Qwen2.5-3B, Qwen3.5-1.7B)
- Online models with stronger multilingual training (Gemini, DeepSeek)
- Post-processing lookup via **Kaikki Wiktionary dumps** as a fallback or replacement

No decision made. Gender handling will be determined by the 20-word quality torture suite.

### Current Status (2026-07-06)

- Core pipeline: scanning, enrichment, merging, verification, writing
- Local LLM integration via llama.cpp server (OpenAI-compatible API)
- **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/`
- **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1)
- Schema validation for generated JSON
- Progress tracking and timing metrics
- **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
- **Pending:** 20-word quality torture suite (will decide gender approach)
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq)

### One-Line Architecture

```
source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> write .json
                                    |
                         [gender: LLM-generated, accuracy TBD]
```

### Files at a Glance

| File                                      | Purpose                                                                             |
| ----------------------------------------- | ----------------------------------------------------------------------------------- |
| `pipeline.ts`                             | Orchestrator. Scans sources, loops words, coordinates all stages                    |
| `config/llm.ts`                           | Provider selection, API URL, model name                                             |
| `config/prompt.ts`                        | System prompt sent to the LLM                                                       |
| `config/batch.ts`                         | Batch size and max retry count                                                      |
| `utils/enrich-word.ts`                    | Calls LLM via adapter, parses response, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts`            | Merges skeleton + enriched senses into final JSON; defines TypeScript schema        |
| `utils/verify-enriched-file.ts`           | Schema validation (required fields, types, gender enum)                             |
| `utils/check-llm-server.ts`               | Health check before pipeline starts                                                 |
| `utils/scanning-source-files.ts`          | Discovers wordlists from `source-data/` directory                                   |
| `utils/create-base-json.ts`               | Writes skeleton `{word, language, pos}` files                                       |
| `utils/write-json-file.ts`                | Atomic `.tmp` → rename writes                                                       |
| `utils/check-if-json-exists.ts`           | Resumability: checks if word already has enriched senses                            |
| `utils/create-line-reader.ts`             | Streaming line reader for large wordlists                                           |
| `utils/create-output-dirs.ts`             | Creates `worddata/{language}/{pos}/` folders                                        |
| `utils/delete-file.ts`                    | Cleanup helper for failed batches                                                   |
| `utils/get-word-file-path.ts`             | Path construction helper                                                            |
| `utils/progress-tracker.ts`               | `[current/total]` formatting for console output                                     |
| `utils/pipeline-timer.ts`                 | Per-word and global timing + LLM token metrics                                      |
| `utils/llm-adapters/factory.ts`           | Creates the right adapter based on `LLM_CONFIG.provider`                            |
| `utils/llm-adapters/types.ts`             | `LlmAdapter` interface                                                              |
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek                                               |
| `utils/llm-adapters/gemini.ts`            | Google Gemini native API                                                            |

### Scale Target

| Metric          | Target                                                       |
| --------------- | ------------------------------------------------------------ |
| Words           | 100,000+                                                     |
| Languages       | English (source), German, Italian, Spanish, French (targets) |
| Parts of speech | Nouns, verbs, adjectives, adverbs                            |
| Output          | One `.json` file per word, ~2-5KB each                       |

---

## 2. Problem & Context

### Why Build This?

Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:

| What Exists              | What is Missing                        | Why It Matters                                                    |
| ------------------------ | -------------------------------------- | ----------------------------------------------------------------- |
| Word + translation       | **Student-friendly definition**        | Learners need explanations, not just equivalents                  |
| Static difficulty labels | **CEFR-calibrated difficulty**         | "Easy" vs "hard" must map to learner proficiency levels           |
| Isolated translations    | **Natural example sentences**          | Context is how vocabulary is actually acquired                    |
| Raw gender data          | **Gender integrated with translation** | Flashcards must show gender immediately, not as a separate lookup |
| Bulk word lists          | **Per-word structured JSON**           | The trainer consumes one file per word for fast random access     |

### The Target User

A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender — all calibrated to their CEFR level (A1-C2).

### Why Not Use Existing Dictionaries?

- **Wiktionary**: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
- **Kaikki (Wiktionary dump)**: Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
- **Google Translate / DeepL**: No definitions, no examples, no difficulty, no structured output
- **Existing language learning apps**: Closed data, no export, no control over content

The LLM fills the gap: it generates **pedagogical content** (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.

### Language Direction

The pipeline is **direction-agnostic**. A wordlist is defined by:

- **Source language**: the language of the input words
- **Target languages**: the languages to translate into

Current focus: **English -> German/Italian/Spanish/French**

Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist and the target languages specified in the prompt.

### Why 100,000+ Words?

- **Coverage**: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation.
- **Languages**: English (source) -> German, Italian, Spanish, French (targets).
- **Parts of speech**: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.).

### The Data Flow

```
Source files          LLM enrichment          Final JSON
(one word per line)   (definitions,           (one per word,
                      examples,               self-contained)
english/nouns         difficulty,
english/verbs         translations)            time.json
italian/nouns                                year.json
...                                          people.json
```

### The Quality Challenge

Generating 100,000 entries with an LLM introduces risks:

| Risk                           | Mitigation                                                     |
| ------------------------------ | -------------------------------------------------------------- |
| Hallucinated definitions       | Low temperature (0.1), strict system prompt, schema validation |
| Incorrect grammatical gender   | Under evaluation: larger models or external lookup             |
| Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking                 |
| JSON parse failures            | Retry + split logic, schema validation, cleanup on failure     |
| Model drift (online APIs)      | Version pinning, local fallback                                |
| Provider downtime              | Adapter pattern allows hot-swapping providers                  |

### Why TypeScript + Node?

- **Familiarity**: Existing project uses TypeScript (frontend in TanStack Router + React)
- **Ecosystem**: `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP
- **Portability**: Runs on the same Debian laptop as the llama.cpp server
- **No build complexity**: `tsx` for direct execution, no bundler needed

### Why llama.cpp?

- **GGUF format**: Single-file models, easy to swap, quantize, and version
- **OpenAI-compatible API**: `/v1/chat/completions` means the same adapter code works for local and online models
- **No dependencies**: Self-contained binary, runs on old hardware (tested on GTX 950M)
- **Privacy**: Local inference means no data leaves the machine

---

## 3. Architecture Overview

### Pipeline Flow

```
Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
    -> Create skeletons (batch) -> Call LLM -> Parse JSON -> Retry/split on failure
    -> Merge -> Write atomically -> Verify schema -> Log metrics
```

### Resumability

- **Skip existing**: `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses`
- **Atomic writes**: `.tmp` -> rename in `write-json-file.ts`, no partial files on crash
- **Cleanup on failure**: Deletes skeleton files for failed batches, continues to next batch

### Directory Structure

```
data-pipeline/
|-- pipeline.ts              # Entry point / orchestrator
|-- config/
|   |-- llm.ts               # Provider, API URL, model name
|   |-- prompt.ts            # System prompt
|   |-- batch.ts             # Batch size and retry config
|-- utils/
|   |-- enrich-word.ts          # LLM call, parse, retry/split
|   |-- merge-enriched-data.ts  # Schema types + merge logic
|   |-- verify-enriched-file.ts # Schema validation
|   |-- check-llm-server.ts     # Health check
|   |-- scanning-source-files.ts # Source discovery
|   |-- create-base-json.ts     # Skeleton writer
|   |-- write-json-file.ts      # Atomic JSON writer
|   |-- check-if-json-exists.ts # Resumability check
|   |-- create-line-reader.ts   # Streaming file reader
|   |-- create-output-dirs.ts   # Directory creation
|   |-- delete-file.ts          # Cleanup helper
|   |-- get-word-file-path.ts   # Path helper
|   |-- progress-tracker.ts     # Console progress formatting
|   |-- pipeline-timer.ts       # Timing + token metrics
|   |-- llm-adapters/
|       |-- factory.ts          # Adapter selection
|       |-- types.ts            # LlmAdapter interface
|       |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
|       |-- gemini.ts           # Google Gemini
|-- source-data/
|   |-- {language}/
|       |-- {pos}               # One word per line, no extension
|-- worddata/
    |-- {language}/
        |-- {pos}/
            |-- {word}.json     # One self-contained file per word
|-- kaikki-source-files/        # Wiktionary dumps for gender lookup (planned)
```

### Output Schema

Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `id`, `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`.

Full TypeScript interfaces: `utils/merge-enriched-data.ts`.

### Error Handling

| Failure                        | Behavior                                                |
| ------------------------------ | ------------------------------------------------------- |
| LLM server offline             | Hard fail at startup (`check-llm-server.ts`)            |
| LLM returns bad JSON           | Retry up to 3 times, then split batch. Log and continue |
| Schema validation fails        | Log warnings, keep file                                 |
| Individual batch fails         | Does not stop pipeline; cleans up skeletons             |
| Individual word fails (size 1) | Log and continue to next word                           |

### Metrics

Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`.

---

## 4. Current Implementation

### Tech Stack

| Layer       | Choice                        | Why                                        |
| ----------- | ----------------------------- | ------------------------------------------ |
| Runtime     | Node.js + `tsx`               | Direct TypeScript execution, no build step |
| HTTP client | Native `fetch`                | Works for local llama.cpp and online APIs  |
| File I/O    | `fs` + `readline`             | Streaming line reader for large wordlists  |
| JSON        | Native `JSON.parse/stringify` | Simple, no schema library needed           |

### Configuration

| File               | Purpose                                                          |
| ------------------ | ---------------------------------------------------------------- | ------------ | ---------- | ------------------------- |
| `config/llm.ts`    | `provider` (`local`                                              | `openrouter` | `deepseek` | `gemini`), `url`, `model` |
| `config/prompt.ts` | System prompt with CEFR mapping, required fields, example output |
| `config/batch.ts`  | `BATCH_CONFIG.size` (words per call), `maxRetries`               |

### Key Modules

| File                                      | Responsibility                                                                                                            |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `utils/enrich-word.ts`                    | Calls LLM via adapter, strips markdown, parses JSON array, builds `EnrichedSense[]` with composite IDs, retry/split logic |
| `utils/merge-enriched-data.ts`            | Merges skeleton `{word, language, pos}` with LLM senses, adds `enrichedAt` and `model`                                    |
| `utils/verify-enriched-file.ts`           | Schema validation: required fields, array lengths, gender enum, translation structure                                     |
| `utils/pipeline-timer.ts`                 | Tracks per-word and global metrics (duration, tokens, throughput)                                                         |
| `utils/llm-adapters/factory.ts`           | Creates adapter based on `LLM_CONFIG.provider`                                                                            |
| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek                                                     |
| `utils/llm-adapters/gemini.ts`            | Google Gemini `generateContent` API                                                                                       |

### Current Model

| Property     | Value                                    |
| ------------ | ---------------------------------------- |
| Model        | `qwen2.5-1.5b-instruct-q4_k_m.gguf`      |
| Size         | ~1.0GB                                   |
| Quantization | Q4_K_M                                   |
| Server       | llama.cpp (`llama-server`)               |
| API          | OpenAI-compatible `/v1/chat/completions` |

### llama-server Flags: History & Rationale

The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen.

#### Hardware Constraints

| Component | Spec                                      | Implication                                                                 |
| --------- | ----------------------------------------- | --------------------------------------------------------------------------- |
| CPU       | i7-6500U (2 physical cores, 4 threads HT) | `-t 2` matches physical cores; HT hurts more than helps                     |
| GPU       | GTX 950M (Maxwell, 2015)                  | 32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved |
| RAM       | 8GB (3.95GB usable)                       | `--mlock` pins model in RAM; system must not swap                           |

#### Flag Evolution

| Flag              | Value Tried                         | Result                                                              | Why                                                                                                                     |
| ----------------- | ----------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `-m`              | `qwen3.5-4b-q4_k_m.gguf`            | Works, ~47s/word                                                    | Baseline. Correct genders. 2.6GB file, tight on VRAM.                                                                   |
| `-m`              | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken**                                                          | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#12-known-issues--dev-notes). |
| `-m`              | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~8s/word                                                     | Current. Fast but gender accuracy degraded.                                                                             |
| `-ngl`            | `999`                               | Keeps                                                               | Offload all layers to GPU. Required for any speed.                                                                      |
| `-c`              | `4096`                              | Wasteful                                                            | 4K context for 300-token dictionary entries wastes ~400MB VRAM.                                                         |
| `-c`              | `2048`                              | Current                                                             | Sufficient headroom. Frees VRAM for KV cache.                                                                           |
| `-b` / `-ub`      | `512`                               | Keeps                                                               | Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs.                                                |
| `-b` / `-ub`      | `2048`                              | Slower on 950M                                                      | Tested briefly. No improvement, possibly worse due to memory pressure.                                                  |
| `-t`              | `4`                                 | Slower                                                              | Hyperthreading cores hurt llama.cpp performance.                                                                        |
| `-t`              | `2`                                 | Current                                                             | Matches 2 physical cores.                                                                                               |
| `--threads-batch` | (default)                           | Risky                                                               | Defaults to same as `-t`, but explicit is safer.                                                                        |
| `--threads-batch` | `2`                                 | Current                                                             | Explicit match to `-t`.                                                                                                 |
| `--flash-attn`    | (omitted)                           | Correct                                                             | On Maxwell (compute 5.0), Flash Attention adds overhead. Not used.                                                      |
| `--flash-attn`    | (tested)                            | No gain                                                             | Briefly tried with Qwen3.5-4B. No speedup, possibly regression.                                                         |
| `--mlock`         | Keeps                               | Pins model weights in RAM. Prevents OS swapping on memory pressure. |
| `--prio`          | `2`                                 | Keeps                                                               | Raises process priority. Marginal on this hardware, harmless.                                                           |
| `--reasoning`     | `off`                               | (Qwen3.5 only)                                                      | Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5.                         |

#### Current Command

```bash
./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2
```

#### What Was Not Tried (And Why)

| Flag                        | Reason Skipped                                                                                             |
| --------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `-fa` / `--flash-attn`      | Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs. |
| `--no-mmap`                 | `--mlock` achieves the same (pin in RAM) without the I/O overhead of disabling mmap.                       |
| `-np` / `--parallel`        | Not needed. Single sequential pipeline, no concurrent requests.                                            |
| `--cont-batching`           | Default in recent llama.cpp. No need to toggle.                                                            |
| `--defrag-thold`            | KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx.    |
| `-ot` / `--override-tensor` | Expert-level. No tensor-specific issues observed.                                                          |

#### Future Flag Experiments

| Experiment       | Expected Outcome                                                           |
| ---------------- | -------------------------------------------------------------------------- |
| `-c 1024`        | Further VRAM savings. Risk: insufficient for batching larger prompt sizes. |
| `-b 256 -ub 256` | Test if smaller batches reduce latency on bandwidth-starved Maxwell.       |
| `--metrics`      | Enable Prometheus endpoint for automated performance tracking.             |

### Performance Baseline

| Metric                | Qwen3.5-4B | Qwen2.5-1.5B |
| --------------------- | ---------- | ------------ |
| Time/word             | ~47s       | ~8s          |
| Completion tok/s      | ~6.4       | ~18.8        |
| Prompt tok/s          | ~81        | ~549         |
| Avg completion tokens | ~274       | ~132         |
| Avg prompt tokens     | ~327       | ~308         |

### Known Limitations (Current)

- **Gender accuracy**: Qwen2.5-1.5B systematically defaults to `neuter` for Romance languages. Under evaluation whether larger models fix this.
- **Single POS**: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments.
- **Hardcoded model name**: `merge-enriched-data.ts` hardcodes `"qwen3.5-4b-q4_k_m"` regardless of actual model used.
- **Duplicated mappings**: `LANG_MAP`/`POS_MAP` exist in three separate files.

---

## 5. The LLM Layer

### 5.1 Local Model Evaluation

Models are evaluated on three criteria in order of priority: **quality** (definitions, examples, translations, gender accuracy), **speed** (completion tokens/sec), **VRAM fit** (must run on GTX 950M 4GB).

| Model                   | Size   | VRAM  | Speed               | Quality                     | Status                           |
| ----------------------- | ------ | ----- | ------------------- | --------------------------- | -------------------------------- |
| **Qwen3.5-4B Q4_K_M**   | 2.6GB  | Tight | ~6.4 tok/s          | Baseline (assumed good)     | Baseline - too slow              |
| **Qwen2.5-1.5B Q4_K_M** | 1.0GB  | Easy  | ~18.8 tok/s         | Gender systematically wrong | Current - fast, needs validation |
| **Ministral-3B Q4_K_M** | 1.9GB  | Fits  | -                   | Broken (tokenizer)          | Abandoned                        |
| **Qwen2.5-3B Q4_K_M**   | 1.9GB  | Fits  | ~12-15 tok/s (est.) | Unknown                     | Pending download                 |
| **Gemma 4 E2B Q4_K_M**  | 3.46GB | No    | -                   | -                           | Too large for 4GB VRAM           |
| **Gemma 4 E2B IQ2_M**   | 2.62GB | Fits  | ~8-12 tok/s (est.)  | "Low quality" per Google    | Not worth it                     |

#### Qwen2.5-1.5B Test Results (3 words)

| Word   | Definition                                   | Example                            | Gender (de/it/es/fr)              | Verdict   |
| ------ | -------------------------------------------- | ---------------------------------- | --------------------------------- | --------- |
| time   | "A period of duration..."                    | "The meeting was scheduled..."     | neuter/neuter/neuter/neuter       | All wrong |
| year   | "A period of time consisting of 365 days..." | "The year 2023 is a leap year."    | neuter/neuter/neuter/neuter       | All wrong |
| people | "Individuals who are part of a group."       | "The people gathered at the park." | neuter/feminine/feminine/feminine | Mixed     |

**Pattern:** Defaults to `neuter` when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident.

#### Pending Tests

- **Qwen2.5-3B**: Same architecture, 2x params. If gender fixes, it was a size problem.
- **20-word torture suite**: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models.

### 5.2 Online API Options

Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API (except Gemini, which has a native adapter).

| Provider            | Model                | Input $/1M | Output $/1M | Free Tier     | Rate Limit       | Est. Cost (100k words) | Est. Time           |
| ------------------- | -------------------- | ---------- | ----------- | ------------- | ---------------- | ---------------------- | ------------------- |
| **DeepSeek**        | V4 Flash             | $0.14      | $0.28       | 5M tokens     | None             | **$0-15**              | ~1-2 days           |
| **Gemini**          | 2.5 Flash-Lite       | $0.10      | $0.40       | 1,500 req/day | 1M TPM           | **$0** (free tier)     | ~1.5 days (batched) |
| **Qwen/Alibaba**    | Qwen-Turbo           | $0.05      | $0.20       | Unknown       | 600 RPM          | **~$11**               | ~1-2 days           |
| **Groq**            | Llama 3.1 8B Instant | $0.05      | $0.08       | Yes           | High             | **~$7**                | **~3-4 hours**      |
| **OpenRouter free** | Various              | $0         | $0          | 200 req/day   | 20 RPM           | **$0**                 | ~10 days            |
| **OpenRouter paid** | DeepSeek V4 Flash    | $0.14      | $0.28       | -             | Same as provider | **~$16** (+5.5% fee)   | ~2-3 days           |

**Notes:**

- Costs assume ~550 tokens/word (300 prompt + 250 completion).
- Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day.
- Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable.
- DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million.

### 5.3 Model Selection Criteria

Decision flow for 100,000 words:

```
Start
  |
  v
Run 20-word torture suite
on Qwen2.5-3B (local)
  |
  |-- Quality good? -----> Use Qwen2.5-3B locally
  |   (gender correct)       ~20 days, $0
  |
  |-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free)
                              |
                              |-- Quality good? --> Batch 50, free tier
                              |                      ~1.5 days, $0
                              |
                              |-- Quality meh? ---> Test Groq or DeepSeek paid
                                                        |
                                                        |-- Speed priority? --> Groq
                                                        |                       ~$7, 3-4 hours
                                                        |
                                                        |-- Cost priority? ---> DeepSeek
                                                                                  ~$15, 1-2 days
```

**Quality gates:**

- > = 90% gender accuracy (de/it/es/fr)
- 100% JSON parse rate
- No hallucinated definitions on polysemous words
- Natural, contextually appropriate example sentences
- Sensible difficulty classification (CEFR mapping)

---

## 6. The Gender Problem & Kaikki Integration

### The Problem

Grammatical gender is embedded in the `translations` object of each sense:

```json
"translations": {
  "de": [{"word": "Haus", "gender": "neuter"}],
  "it": [{"word": "casa", "gender": "feminine"}],
  "es": [{"word": "casa", "gender": "feminine"}],
  "fr": [{"word": "maison", "gender": "feminine"}]
}
```

Early testing with **Qwen2.5-1.5B** showed systematic failure: the model defaults to `neuter` for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all — only masculine and feminine.

Whether this is a **model size issue** (1.5B too small to retain gender facts) or a **training data gap** (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation.

### Two Approaches Under Consideration

| Approach                   | How It Works                                                                                         | Pros                                                    | Cons                                                                                            |
| -------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| **A. LLM-only**            | Trust the model to generate correct gender                                                           | Zero additional complexity                              | Unreliable at small sizes; hallucination risk                                                   |
| **B. LLM + Kaikki lookup** | LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump | 100% deterministic; decouples gender from model quality | Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries |

### Kaikki Data

Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender.

| Language | Kaikki File                           | Coverage |
| -------- | ------------------------------------- | -------- |
| German   | `kaikki.org-dictionary-German.jsonl`  | High     |
| Italian  | `kaikki.org-dictionary-Italian.jsonl` | High     |
| Spanish  | `kaikki.org-dictionary-Spanish.jsonl` | High     |
| French   | `kaikki.org-dictionary-French.jsonl`  | High     |

Lookup logic: match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`.

### Decision Pending

- If **Qwen2.5-3B** or an online model produces >=90% accurate gender: **Approach A**, no Kaikki needed.
- If all tested models fail gender: **Approach B**, implement Kaikki lookup as a post-processing step after LLM enrichment.

No implementation work started until the 20-word torture suite resolves this.

---

## 7. Batching Strategy

### Why Batching is Necessary

At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.

### Configurable Batch Size

Single config point controls batch size everywhere:

```typescript
// config/batch.ts
export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const;
```

Change `size` to 1, 2, 5, 10, 20, etc. The pipeline adapts without code changes.

### Prompt Structure

**Single word:**

```
["house"]
```

**Batch of 4:**

```
["house", "car", "tree", "water"]
```

LLM returns a JSON object with word keys:

```json
{
  "house": [ { "sense": "...", "example": "...", ... } ],
  "car": [ { ... } ],
  "tree": [ { ... } ],
  "water": [ { ... } ]
}
```

### Retry & Split Strategy

If a batch fails (bad JSON, missing key, etc.):

```
Batch of 4 fails (3 retries exhausted)
    |
    v
Split into 2 batches of 2
    |
    v
If a batch of 2 fails (3 retries), split into 2 batches of 1
    |
    v
If a single word fails (3 retries), log and skip
```

This gives resilience without losing the speed benefit of large batches. The `maxRetries` config controls how many attempts are made before splitting.

### Expected Impact by Environment

| Environment        | Batch Size | Expected Speedup | Notes                                                    |
| ------------------ | ---------- | ---------------- | -------------------------------------------------------- |
| Local GTX 950M     | 3          | ~1.05-1.15x      | Memory bandwidth limited. KV cache pressure on 4GB VRAM. |
| Local GTX 950M     | 5          | ~1.10-1.20x      | Sweet spot for this hardware.                            |
| Local GTX 950M     | 15         | Risky            | May OOM. Test carefully.                                 |
| Local GTX 950M     | 50         | Unlikely         | VRAM insufficient.                                       |
| Cloud API (Gemini) | 50         | 5x fewer calls   | Unlocks free tier viability.                             |
| Cloud API (Groq)   | 50         | 5x fewer calls   | Minimal gain - already fast.                             |

### Batching is Non-Negotiable

Regardless of local vs cloud, batching is required for 100k words:

- **Local**: Better GPU utilization, amortized prompt processing
- **Cloud**: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts

---

## 8. Hardware Constraints

### Current Machine

| Component | Spec                                                                |
| --------- | ------------------------------------------------------------------- |
| OS        | Debian GNU/Linux 13 (trixie) x86_64                                 |
| CPU       | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz |
| GPU       | NVIDIA GeForce GTX 950M (Maxwell, 2015)                             |
| GPU VRAM  | 4GB                                                                 |
| RAM       | 8GB (3.95GB usable at idle)                                         |
| Disk      | 102GB ext4 (~63GB used)                                             |

### What Fits in 4GB VRAM

| Model               | File Size | KV Cache (2048 ctx) | Total VRAM | Fits?              |
| ------------------- | --------- | ------------------- | ---------- | ------------------ |
| Qwen2.5-1.5B Q4_K_M | ~1.0GB    | ~0.5GB              | ~1.5GB     | Yes                |
| Qwen2.5-3B Q4_K_M   | ~1.9GB    | ~0.8GB              | ~2.7GB     | Yes                |
| Ministral-3B Q4_K_M | ~1.9GB    | ~0.8GB              | ~2.7GB     | Yes (but broken)   |
| Qwen3.5-4B Q4_K_M   | 2.6GB     | ~1.0GB              | ~3.6GB     | Tight              |
| Gemma 4 E2B Q4_K_M  | 3.46GB    | ~1.2GB              | ~4.7GB     | No                 |
| Gemma 4 E2B IQ2_M   | 2.62GB    | ~1.0GB              | ~3.6GB     | Maybe, low quality |

### GPU Rental Alternatives

If local hardware becomes the bottleneck:

| Provider | GPU      | VRAM | Price/Hour  | Time for 100k Words | Total Cost |
| -------- | -------- | ---- | ----------- | ------------------- | ---------- |
| Vast.ai  | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours          | **~$2-5**  |
| RunPod   | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours          | **~$4-6**  |
| Vast.ai  | RTX 3090 | 24GB | ~$0.20-0.40 | ~8-10 hours         | **~$2-4**  |

With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day.

---

## 9. Testing & Quality Assurance

### 20-Word Torture Suite

Planned test set covering edge cases:

| Category       | Words                                                  | Why                                 |
| -------------- | ------------------------------------------------------ | ----------------------------------- |
| Easy concrete  | `house`, `water`, `book`                               | Baseline                            |
| Easy abstract  | `time`, `love`, `hope`                                 | Abstract nouns harder to define     |
| Polysemous     | `bank`, `run`, `light`                                 | Multiple senses test disambiguation |
| Hard/technical | `democracy`, `photosynthesis`, `entropy`               | Complex definitions                 |
| False friends  | `actual` (en/es), `sensible` (en/fr), `fabric` (en/de) | Cross-lingual traps                 |

### Evaluation Criteria

For each word and each candidate model:

| Criterion                     | Pass Threshold                                    |
| ----------------------------- | ------------------------------------------------- |
| Definition accuracy           | Factually correct, max 15 words, student-friendly |
| Example quality               | Natural sentence, word used correctly in context  |
| Translation correctness       | Correct word sense match                          |
| Gender accuracy (de/it/es/fr) | >=90% correct                                     |
| Difficulty classification     | Sensible per CEFR mapping                         |
| JSON reliability              | 100% parse rate, valid schema                     |

### Verification

`verify-enriched-file.ts` checks:

- Required top-level fields: `word`, `language`, `pos`, `senses`
- Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard}
- Each translation: `word` (string), `gender` in {masculine, feminine, neuter, null}

---

## 10. Future Extensions & Roadmap

### Near-Term (Next 2-4 Weeks)

| Item                     | Status       | Notes                                                           |
| ------------------------ | ------------ | --------------------------------------------------------------- |
| Configurable batching    | **Complete** | Single `BATCH_CONFIG.size` value, retry/split logic implemented |
| 20-word torture suite    | Pending      | Decides gender approach and model selection                     |
| Qwen2.5-3B evaluation    | Pending      | Download and test                                               |
| Online API testing       | Pending      | Gemini free tier, DeepSeek, Groq                                |
| Fix hardcoded model name | Pending      | `merge-enriched-data.ts` hardcodes `"qwen3.5-4b"`               |
| Extract shared constants | Pending      | `LANG_MAP`/`POS_MAP` duplicated in 3 files                      |

### Medium-Term (1-3 Months)

| Item                         | Notes                                                  |
| ---------------------------- | ------------------------------------------------------ |
| Multi-POS support            | Verbs, adjectives, adverbs need prompt variants        |
| Multi-language source        | German -> French, Italian -> Spanish, etc.             |
| Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously |
| Incremental enrichment       | Only process new/changed words in a wordlist           |
| Model auto-switching         | Fallback to online API if local server fails mid-run   |

### Long-Term (3-6 Months)

| Item                     | Notes                                                            |
| ------------------------ | ---------------------------------------------------------------- |
| Batch API discounts      | Gemini, Qwen, Azure offer 50% off for 24h SLA                    |
| GPU rental integration   | Script to spin up Vast.ai/RunPod, run pipeline, download results |
| Quality regression tests | Run torture suite on every model change                          |
| Community contributions  | Open-source the pipeline for other language learners             |

---

## 11. Decisions Log

| Date       | Decision                          | Context                                              | Rationale                                                                   |
| ---------- | --------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------- |
| 2026-01-04 | TanStack Router for frontend      | Previous project used React Router                   | Simpler, type-safe routing for the trainer app                              |
| 2026-01-04 | Vite dev server (no Nginx)        | Docker setup for glossa-web                          | Nginx unnecessary for dev; Vite handles HMR and proxying                    |
| 2026-01-17 | Backend answer verification       | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas                           |
| 2026-03-26 | Multi-stage Docker builds         | glossa-api and glossa-web containers                 | Smaller images, faster deploys                                              |
| 2026-06-16 | llama.cpp for local LLM           | Need local inference on old laptop                   | GGUF format, OpenAI-compatible API, no dependencies                         |
| 2026-06-16 | Q4_K_M quantization               | Balance size vs quality                              | Q4_K_M is the community standard for 4-bit inference                        |
| 2026-06-16 | `-c 2048` context                 | Default was 4096                                     | Dictionary entries need ~500 tokens max; frees VRAM                         |
| 2026-06-16 | `-t 2` physical cores             | Default was 4 (HT threads)                           | Hyperthreading hurts llama.cpp performance                                  |
| 2026-06-17 | Qwen2.5-1.5B as current model     | Qwen3.5-4B too slow (47s/word)                       | 6x speedup (8s/word), quality under evaluation                              |
| 2026-06-17 | Skip Gemma 4                      | E2B Q4_K_M is 3.46GB                                 | Does not fit in 4GB VRAM; lower quants sacrifice quality                    |
| 2026-06-17 | Skip Ministral-3B                 | Tokenizer mismatch (Tekken)                          | Outputs gibberish regardless of template; not fixable without re-conversion |
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud                      | `utils/llm-adapters/` with factory + types + per-provider implementations   |
| 2026-07-06 | Retry + split batching            | LLM JSON parse failures on larger batches            | `enrichWordWithRetry` retries 3 times, then halves batch until size 1       |

---

## 12. Known Issues & Dev Notes

### glossa-web (Frontend)

| Issue                    | Details                                                                                   |
| ------------------------ | ----------------------------------------------------------------------------------------- |
| No healthcheck           | Vite dev server has no health endpoint. Docker `HEALTHCHECK` cannot verify running state. |
| Valkey memory overcommit | Harmless warning in dev: `vm.overcommit_memory = 1` recommended before production.        |

### Data Pipeline

| Issue                             | Details                                                                                         | Severity                         |
| --------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------- |
| Ministral-3B tokenizer mismatch   | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish.                       | Blocker - abandoned              |
| Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages.                                              | Under evaluation                 |
| Hardcoded model name              | `merge-enriched-data.ts` always writes `"qwen3.5-4b-q4_k_m"` regardless of actual model.        | Minor - fix before production    |
| Duplicated LANG_MAP/POS_MAP       | Identical mapping objects in `create-base-json.ts`, `merge-enriched-data.ts`, `enrich-word.ts`. | Minor - refactor risk            |
| OpenAI-compatible timings         | `json.timings` is llama.cpp-specific. Will break for OpenRouter/DeepSeek.                       | Medium - needs graceful fallback |
| Single POS tested                 | Only nouns validated. Verbs/adjectives need prompt changes.                                     | Known limitation                 |
| Batch metrics averaging           | Split-and-merge averages tokens/sec instead of weighting by token count.                        | Minor - summary stats only       |

### Hardware

| Issue                 | Details                                               |
| --------------------- | ----------------------------------------------------- |
| GTX 950M VRAM ceiling | 4GB hard limit. Models >3GB risk OOM.                 |
| Maxwell GPU aging     | No Flash Attention support, bandwidth-starved.        |
| Laptop thermals       | Cannot run 24/7 for weeks. Batch processing required. |

---

## 13. How to Run

### Prerequisites

- Node.js + npm
- `tsx` installed globally: `npm install -g tsx`
- llama.cpp built from source (for local mode)
- GGUF model downloaded to `~/Downloads/llama.cpp/models/` (for local mode)
- API keys set as environment variables (for cloud mode):
  ```bash
  export DEEPSEEK_API_KEY="sk-..."
  export GEMINI_API_KEY="..."
  export OPENROUTER_API_KEY="..."
  ```

### Start the LLM Server (Local Mode)

```bash
cd ~/Downloads/llama.cpp

./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2
```

### Configure Provider

Edit `config/llm.ts`:

```typescript
// Local
export const LLM_CONFIG = {
  provider: "local" as const,
  url: "http://127.0.0.1:8080/v1/chat/completions",
  model: undefined,
};

// Gemini
export const LLM_CONFIG = {
  provider: "gemini" as const,
  url: "",
  model: "gemini-2.5-flash-lite",
};
```

### Run the Pipeline

```bash
cd /path/to/data-pipeline
npx tsx pipeline.ts
```

### Expected Output

```
Starting data pipeline...

 step 1: scanning the source files...
✅ Scan complete! Found 1 wordlist(s):
   • ENGLISH (nouns)

 step 2: creating necessary output folders...
✅ All required output directories have been verified and created successfully.

 step 3: verifying local AI engine status...
🟢 Local AI engine is connected and ready for inference!

 step 4: looping through the wordlists...

Reading list: [ENGLISH] -> [NOUNS]

  Batch 1/1: [house, car, tree, water]
  [1/4] (0 failed) Enriched and saved: house.json
  [2/4] (0 failed) Enriched and saved: car.json
  [3/4] (0 failed) Enriched and saved: tree.json
  [4/4] (0 failed) Enriched and saved: water.json
  ⏱️  Word took 8.2s

⏱️  Pipeline Summary
   Duration: 32.8s
   Processed: 4
   Skipped: 0
   Failed: 0
   Total: 4
   Throughput: 0.12 words/sec

🤖 LLM Metrics
   Calls: 1
   Avg prompt tokens: 312
   Avg completion tokens: 524
   Avg prompt speed: 548.2 tok/s
   Avg completion speed: 18.8 tok/s

Global data pipeline run completed successfully.
```

---

## 14. Roadmap

### Phase 1: Batching (Complete)

| Task                                  | Status       | Notes                                                                                |
| ------------------------------------- | ------------ | ------------------------------------------------------------------------------------ |
| Implement configurable batch size     | **Complete** | `config/batch.ts` with `size` and `maxRetries`                                       |
| Implement retry + split logic         | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch                                   |
| Verify batching doesn't break quality | Pending      | Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output. |
| Measure speedup vs batch size         | Pending      | Track throughput at 1, 5, 15 on local hardware.                                      |

**Goal:** Unlock 5-15x speedup on local, unlock free online API tiers.

---

### Phase 2: Model Selection

| Task                                             | Status  | Notes                                                                                    |
| ------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------- |
| Download Qwen2.5-3B Q4_K_M                       | Pending | ~1.9GB, fits in 4GB VRAM.                                                                |
| Run torture suite on Qwen2.5-3B (batched)        | Pending | Check if gender accuracy improves with 2x params.                                        |
| Test Gemini 2.5 Flash-Lite free tier (batched)   | Pending | 1,500 req/day x 50 words = 75k words/day. Zero cost.                                     |
| Test DeepSeek V4 Flash free tier (batched)       | Pending | 5M tokens free. ~9k words.                                                               |
| Test Groq Llama 3.1 8B (batched, paid if needed) | Pending | Fastest option. ~$7 for 100k words.                                                      |
| Decide: local vs online, which model             | Pending | Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost. |

**Goal:** Pick the model and provider for the 100k word run.

---

### Phase 3: Scale

| Task                      | Status  | Notes                                                                                                                      |
| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| Run 100k word pipeline    | Pending | Estimated time depends on Phase 2 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). |
| Spot-check output quality | Pending | Random sample of 100 entries.                                                                                              |
| Fix gender if needed      | Pending | Kaikki lookup post-processing if LLM gender remains unreliable.                                                            |
| Handle failures & retries | Pending | Exponential backoff, split-and-retry for batch failures.                                                                   |

**Goal:** Complete 100k word dataset.

---

### Phase 4: Extend

| Task                         | Status  | Notes                                                                                       |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------- |
| Multi-POS support            | Pending | Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.).     |
| Multi-language source        | Pending | German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo. |
| Parallel wordlist processing | Pending | Run `english/nouns` and `english/verbs` simultaneously.                                     |
| Incremental enrichment       | Pending | Only process new/changed words in a wordlist.                                               |
| GPU rental integration       | Pending | Script to spin up Vast.ai/RunPod, run pipeline, download results.                           |
| Quality regression tests     | Pending | Run torture suite on every model change.                                                    |

**Goal:** Generalize pipeline for any language direction and POS.

---

### Backlog (Unscheduled)

| Task                                        | Context                                                                                     |
| ------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Batch API discounts                         | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. |
| Model auto-switching                        | Fallback to online API if local server fails mid-run.                                       |
| Community open-source                       | Clean up, document, publish for other language learners.                                    |
| Prometheus metrics                          | `--metrics` flag on llama-server for automated performance tracking.                        |
| `-c 1024` / `-b 256` experiments            | Further VRAM optimization on GTX 950M. Low priority if moving to cloud.                     |
| Extract shared LANG_MAP/POS_MAP             | Single source of truth for language/pos mappings.                                           |
| Fix hardcoded model name                    | Pass actual model name through enrichment chain.                                            |
| Graceful timing fallback for cloud adapters | Handle missing `timings` field in OpenRouter/DeepSeek responses.                            |