# Lila Data Pipeline — Technical Documentation > Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer. > Last updated: 2026-06-17 --- ## Table of Contents 1. [Executive Summary](#1-executive-summary) 2. [Problem & Context](#2-problem--context) 3. [Architecture Overview](#3-architecture-overview) 4. [Current Implementation](#4-current-implementation) 5. [The LLM Layer](#5-the-llm-layer) - 5.1 [Local Model Evaluation](#51-local-model-evaluation) - 5.2 [Online API Options](#52-online-api-options) - 5.3 [Model Selection Criteria](#53-model-selection-criteria) 6. [The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration) 7. [Batching Strategy](#7-batching-strategy) 8. [Hardware Constraints](#8-hardware-constraints) 9. [Testing & Quality Assurance](#9-testing--quality-assurance) 10. [Future Extensions & Roadmap](#10-future-extensions--roadmap) 11. [Decisions Log](#11-decisions-log) 12. [Known Issues & Dev Notes](#12-known-issues--dev-notes) 13. [How to Run](#13-how-to-run) 14. [Roadmap](#14-roadmap) --- ## Quick Reference | What | Where | | ----------------- | ----------------------------------- | | Entry point | `pipeline.ts` | | LLM config | `config/llm.ts` | | System prompt | `config/prompt.ts` | | Output schema | `utils/merge-enriched-data.ts` | | Batch size config | `config/batch.ts` _(planned)_ | | Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | | Target scale | 100,000+ words | --- ## 1. Executive Summary The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates: - One or more **senses** (definitions) - A **natural example sentence** per sense - A **CEFR-based difficulty level** (`easy` / `medium` / `hard`) - **Translations** into German, Italian, Spanish, and French, each with grammatical **gender** The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It is currently in active development: the core architecture is stable, the LLM integration layer is being evaluated across local and online providers, and a configurable batching system is planned to unlock throughput at scale. ### Key Design Principles | Principle | Rationale | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | **Quality first** | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary. | | **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. | | **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. | | **Configurable batching** | Batch size (1, 5, 15, 50, etc.) is a single config value. The pipeline adapts without code changes. | ### Open Question: Gender Accuracy Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that **Qwen2.5-1.5B systematically defaults to `neuter`** for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a **model size issue** (fixable by moving to 3B+) or a **training data issue** (requiring an external lookup) is unresolved. **Options under evaluation:** - Larger local models (Qwen2.5-3B, Qwen3.5-1.7B) - Online models with stronger multilingual training (Gemini, DeepSeek) - Post-processing lookup via **Kaikki Wiktionary dumps** as a fallback or replacement No decision made. Gender handling will be determined by the 20-word quality torture suite. ### Current Status (2026-06-17) - Core pipeline: scanning, enrichment, merging, verification, writing - Local LLM integration via llama.cpp server (OpenAI-compatible API) - Schema validation for generated JSON - Progress tracking and timing metrics - **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending) - **In progress:** Designing configurable batching system - **Pending:** 20-word quality torture suite (will decide gender approach) - **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) ### One-Line Architecture ``` source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> write .json | [gender: LLM-generated, accuracy TBD] ``` ### Files at a Glance | File | Purpose | | ------------------------------- | ---------------------------------------------------------------- | | `pipeline.ts` | Orchestrator. Scans sources, loops words, coordinates all stages | | `config/llm.ts` | API URL, default parameters (`temperature`, `max_tokens`, etc.) | | `config/prompt.ts` | System prompt sent to the LLM | | `utils/enrich-word.ts` | Calls LLM, parses response, builds `EnrichedSense[]` | | `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON | | `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) | | `utils/check-llm-server.ts` | Health check before pipeline starts | | `utils/progress-tracker.ts` | `[current/total]` formatting for console output | | `utils/pipeline-timer.ts` | Per-word and global timing + LLM token metrics | ### Scale Target | Metric | Target | | --------------- | ------------------------------------------------------------ | | Words | 100,000+ | | Languages | English (source), German, Italian, Spanish, French (targets) | | Parts of speech | Nouns, verbs, adjectives, adverbs | | Output | One `.json` file per word, ~2-5KB each | --- ## 2. Problem & Context ### Why Build This? Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer: | What Exists | What is Missing | Why It Matters | | ------------------------ | -------------------------------------- | ----------------------------------------------------------------- | | Word + translation | **Student-friendly definition** | Learners need explanations, not just equivalents | | Static difficulty labels | **CEFR-calibrated difficulty** | "Easy" vs "hard" must map to learner proficiency levels | | Isolated translations | **Natural example sentences** | Context is how vocabulary is actually acquired | | Raw gender data | **Gender integrated with translation** | Flashcards must show gender immediately, not as a separate lookup | | Bulk word lists | **Per-word structured JSON** | The trainer consumes one file per word for fast random access | ### The Target User A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender - all calibrated to their CEFR level (A1-C2). ### Why Not Use Existing Dictionaries? - **Wiktionary**: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions - **Kaikki (Wiktionary dump)**: Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation - **Google Translate / DeepL**: No definitions, no examples, no difficulty, no structured output - **Existing language learning apps**: Closed data, no export, no control over content The LLM fills the gap: it generates **pedagogical content** (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale. ### Language Direction The pipeline is **direction-agnostic**. A wordlist is defined by: - **Source language**: the language of the input words - **Target languages**: the languages to translate into Current focus: **English -> German/Italian/Spanish/French** Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination - the only change is the source wordlist and the target languages specified in the prompt. ### Why 100,000+ Words? - **Coverage**: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation. - **Languages**: English (source) -> German, Italian, Spanish, French (targets). - **Parts of speech**: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.). ### The Data Flow ``` Source files LLM enrichment Final JSON (one word per line) (definitions, (one per word, examples, self-contained) english/nouns difficulty, english/verbs translations) time.json italian/nouns year.json ... people.json ``` ### The Quality Challenge Generating 100,000 entries with an LLM introduces risks: | Risk | Mitigation | | ------------------------------ | -------------------------------------------------------------- | | Hallucinated definitions | Low temperature (0.1), strict system prompt, schema validation | | Incorrect grammatical gender | Under evaluation: larger models or external lookup | | Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking | | JSON parse failures | Retry logic, schema validation, cleanup on failure | | Model drift (online APIs) | Version pinning, local fallback | ### Why TypeScript + Node? - **Familiarity**: Existing project uses TypeScript (frontend in TanStack Router + React) - **Ecosystem**: `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP - **Portability**: Runs on the same Debian laptop as the llama.cpp server - **No build complexity**: `tsx` for direct execution, no bundler needed ### Why llama.cpp? - **GGUF format**: Single-file models, easy to swap, quantize, and version - **OpenAI-compatible API**: `/v1/chat/completions` means the same code works for local and online models - **No dependencies**: Self-contained binary, runs on old hardware (tested on GTX 950M) - **Privacy**: Local inference means no data leaves the machine --- ## 3. Architecture Overview ### Pipeline Flow ``` Scan sources -> Check LLM -> Loop words -> Skip if exists -> Create skeleton -> Call LLM -> Parse JSON -> Merge -> Write atomically -> Verify schema ``` ### Resumability - **Skip existing**: Checks if `{word}.json` exists with non-empty `senses` - **Atomic writes**: `.tmp` -> rename, no partial files on crash - **Cleanup on failure**: Deletes incomplete file, continues to next word ### Directory Structure ``` data-pipeline/ |-- pipeline.ts # Entry point / orchestrator |-- config/ | |-- llm.ts # API URL, model params | |-- prompt.ts # System prompt |-- utils/ # See source files (provided separately) |-- source-data/ | |-- {language}/ | |-- {pos} # One word per line, no extension |-- worddata/ |-- {language}/ |-- {pos}/ |-- {word}.json # One self-contained file per word ``` ### Output Schema Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`. Full TypeScript interfaces: `utils/merge-enriched-data.ts`. ### Error Handling | Failure | Behavior | | ----------------------- | ------------------------------ | | LLM server offline | Hard fail at startup | | LLM returns bad JSON | Log, delete skeleton, continue | | Schema validation fails | Log warnings, keep file | | Individual word fails | Does not stop pipeline | ### Metrics Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`. --- ## 4. Current Implementation ### Tech Stack | Layer | Choice | Why | | ----------- | ----------------------------- | ------------------------------------------ | | Runtime | Node.js + `tsx` | Direct TypeScript execution, no build step | | HTTP client | Native `fetch` | Works for local llama.cpp and online APIs | | File I/O | `fs` + `readline` | Streaming line reader for large wordlists | | JSON | Native `JSON.parse/stringify` | Simple, no schema library needed | ### Configuration | File | Purpose | | ------------------ | -------------------------------------------------------------------------- | | `config/llm.ts` | `LLM_API_URL`, `LLM_DEFAULT_PARAMS` (`temperature`, `top_p`, `max_tokens`) | | `config/prompt.ts` | System prompt with CEFR mapping, required fields, example output | ### Key Modules | File | Responsibility | | ------------------------------- | ------------------------------------------------------------------------------------------ | | `utils/enrich-word.ts` | Calls LLM, strips markdown, parses JSON array, builds `EnrichedSense[]` with composite IDs | | `utils/merge-enriched-data.ts` | Merges skeleton `{word, language, pos}` with LLM senses, adds `enrichedAt` and `model` | | `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure | | `utils/pipeline-timer.ts` | Tracks per-word and global metrics (duration, tokens, throughput) | ### Current Model | Property | Value | | ------------ | ---------------------------------------- | | Model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | | Size | ~1.0GB | | Quantization | Q4_K_M | | Server | llama.cpp (`llama-server`) | | API | OpenAI-compatible `/v1/chat/completions` | ### llama-server Flags: History & Rationale The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen. #### Hardware Constraints | Component | Spec | Implication | | --------- | ----------------------------------------- | --------------------------------------------------------------------------- | | CPU | i7-6500U (2 physical cores, 4 threads HT) | `-t 2` matches physical cores; HT hurts more than helps | | GPU | GTX 950M (Maxwell, 2015) | 32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved | | RAM | 8GB (3.95GB usable) | `--mlock` pins model in RAM; system must not swap | #### Flag Evolution | Flag | Value Tried | Result | Why | | ----------------- | ----------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~47s/word | Baseline. Correct genders. 2.6GB file, tight on VRAM. | | `-m` | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken** | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#12-known-issues--dev-notes). | | `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~8s/word | Current. Fast but gender accuracy degraded. | | `-ngl` | `999` | Keeps | Offload all layers to GPU. Required for any speed. | | `-c` | `4096` | Wasteful | 4K context for 300-token dictionary entries wastes ~400MB VRAM. | | `-c` | `2048` | Current | Sufficient headroom. Frees VRAM for KV cache. | | `-b` / `-ub` | `512` | Keeps | Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs. | | `-b` / `-ub` | `2048` | Slower on 950M | Tested briefly. No improvement, possibly worse due to memory pressure. | | `-t` | `4` | Slower | Hyperthreading cores hurt llama.cpp performance. | | `-t` | `2` | Current | Matches 2 physical cores. | | `--threads-batch` | (default) | Risky | Defaults to same as `-t`, but explicit is safer. | | `--threads-batch` | `2` | Current | Explicit match to `-t`. | | `--flash-attn` | (omitted) | Correct | On Maxwell (compute 5.0), Flash Attention adds overhead. Not used. | | `--flash-attn` | (tested) | No gain | Briefly tried with Qwen3.5-4B. No speedup, possibly regression. | | `--mlock` | Keeps | Pins model weights in RAM. Prevents OS swapping on memory pressure. | | `--prio` | `2` | Keeps | Raises process priority. Marginal on this hardware, harmless. | | `--reasoning` | `off` | (Qwen3.5 only) | Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5. | #### Current Command ```bash ./build/bin/llama-server \ -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \ -ngl 999 \ -c 2048 \ -b 512 \ -ub 512 \ -t 2 \ --threads-batch 2 \ --host 127.0.0.1 \ --port 8080 \ --mlock \ --prio 2 ``` #### What Was Not Tried (And Why) | Flag | Reason Skipped | | --------------------------- | ---------------------------------------------------------------------------------------------------------- | | `-fa` / `--flash-attn` | Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs. | | `--no-mmap` | `--mlock` achieves the same (pin in RAM) without the I/O overhead of disabling mmap. | | `-np` / `--parallel` | Not needed. Single sequential pipeline, no concurrent requests. | | `--cont-batching` | Default in recent llama.cpp. No need to toggle. | | `--defrag-thold` | KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx. | | `-ot` / `--override-tensor` | Expert-level. No tensor-specific issues observed. | #### Future Flag Experiments | Experiment | Expected Outcome | | ---------------- | -------------------------------------------------------------------------- | | `-c 1024` | Further VRAM savings. Risk: insufficient for batching larger prompt sizes. | | `-b 256 -ub 256` | Test if smaller batches reduce latency on bandwidth-starved Maxwell. | | `--metrics` | Enable Prometheus endpoint for automated performance tracking. | ### Performance Baseline | Metric | Qwen3.5-4B | Qwen2.5-1.5B | | --------------------- | ---------- | ------------ | | Time/word | ~47s | ~8s | | Completion tok/s | ~6.4 | ~18.8 | | Prompt tok/s | ~81 | ~549 | | Avg completion tokens | ~274 | ~132 | | Avg prompt tokens | ~327 | ~308 | ### Known Limitations (Current) - **Gender accuracy**: Qwen2.5-1.5B systematically defaults to `neuter` for Romance languages. Under evaluation whether larger models fix this. - **No batching**: One word = one LLM call. System prompt re-processed every time. - **No retry logic**: LLM parse failures are logged and skipped, not retried. - **Single POS**: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments. --- ## 5. The LLM Layer ### 5.1 Local Model Evaluation Models are evaluated on three criteria in order of priority: **quality** (definitions, examples, translations, gender accuracy), **speed** (completion tokens/sec), **VRAM fit** (must run on GTX 950M 4GB). | Model | Size | VRAM | Speed | Quality | Status | | ----------------------- | ------ | ----- | ------------------- | --------------------------- | -------------------------------- | | **Qwen3.5-4B Q4_K_M** | 2.6GB | Tight | ~6.4 tok/s | Baseline (assumed good) | Baseline - too slow | | **Qwen2.5-1.5B Q4_K_M** | 1.0GB | Easy | ~18.8 tok/s | Gender systematically wrong | Current - fast, needs validation | | **Ministral-3B Q4_K_M** | 1.9GB | Fits | - | Broken (tokenizer) | Abandoned | | **Qwen2.5-3B Q4_K_M** | 1.9GB | Fits | ~12-15 tok/s (est.) | Unknown | Pending download | | **Gemma 4 E2B Q4_K_M** | 3.46GB | No | - | - | Too large for 4GB VRAM | | **Gemma 4 E2B IQ2_M** | 2.62GB | Fits | ~8-12 tok/s (est.) | "Low quality" per Google | Not worth it | #### Qwen2.5-1.5B Test Results (3 words) | Word | Definition | Example | Gender (de/it/es/fr) | Verdict | | ------ | -------------------------------------------- | ---------------------------------- | --------------------------------- | --------- | | time | "A period of duration..." | "The meeting was scheduled..." | neuter/neuter/neuter/neuter | All wrong | | year | "A period of time consisting of 365 days..." | "The year 2023 is a leap year." | neuter/neuter/neuter/neuter | All wrong | | people | "Individuals who are part of a group." | "The people gathered at the park." | neuter/feminine/feminine/feminine | Mixed | **Pattern:** Defaults to `neuter` when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident. #### Pending Tests - **Qwen2.5-3B**: Same architecture, 2x params. If gender fixes, it was a size problem. - **20-word torture suite**: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models. ### 5.2 Online API Options Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API. | Provider | Model | Input $/1M | Output $/1M | Free Tier | Rate Limit | Est. Cost (100k words) | Est. Time | | ------------------- | -------------------- | ---------- | ----------- | ------------- | ---------------- | ---------------------- | ------------------- | | **DeepSeek** | V4 Flash | $0.14 | $0.28 | 5M tokens | None | **$0-15** | ~1-2 days | | **Gemini** | 2.5 Flash-Lite | $0.10 | $0.40 | 1,500 req/day | 1M TPM | **$0** (free tier) | ~1.5 days (batched) | | **Qwen/Alibaba** | Qwen-Turbo | $0.05 | $0.20 | Unknown | 600 RPM | **~$11** | ~1-2 days | | **Groq** | Llama 3.1 8B Instant | $0.05 | $0.08 | Yes | High | **~$7** | **~3-4 hours** | | **OpenRouter free** | Various | $0 | $0 | 200 req/day | 20 RPM | **$0** | ~10 days | | **OpenRouter paid** | DeepSeek V4 Flash | $0.14 | $0.28 | - | Same as provider | **~$16** (+5.5% fee) | ~2-3 days | **Notes:** - Costs assume ~550 tokens/word (300 prompt + 250 completion). - Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day. - Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable. - DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million. ### 5.3 Model Selection Criteria Decision flow for 100,000 words: ``` Start | v Run 20-word torture suite on Qwen2.5-3B (local) | |-- Quality good? -----> Use Qwen2.5-3B locally | (gender correct) ~20 days, $0 | |-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free) | |-- Quality good? --> Batch 50, free tier | ~1.5 days, $0 | |-- Quality meh? ---> Test Groq or DeepSeek paid | |-- Speed priority? --> Groq | ~$7, 3-4 hours | |-- Cost priority? ---> DeepSeek ~$15, 1-2 days ``` **Quality gates:** - > =90% gender accuracy (de/it/es/fr) - 100% JSON parse rate - No hallucinated definitions on polysemous words - Natural, contextually appropriate example sentences - Sensible difficulty classification (CEFR mapping) --- ## 6. The Gender Problem & Kaikki Integration ### The Problem Grammatical gender is embedded in the `translations` object of each sense: ```json "translations": { "de": [{"word": "Haus", "gender": "neuter"}], "it": [{"word": "casa", "gender": "feminine"}], "es": [{"word": "casa", "gender": "feminine"}], "fr": [{"word": "maison", "gender": "feminine"}] } ``` Early testing with **Qwen2.5-1.5B** showed systematic failure: the model defaults to `neuter` for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all - only masculine and feminine. Whether this is a **model size issue** (1.5B too small to retain gender facts) or a **training data gap** (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation. ### Two Approaches Under Consideration | Approach | How It Works | Pros | Cons | | -------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | **A. LLM-only** | Trust the model to generate correct gender | Zero additional complexity | Unreliable at small sizes; hallucination risk | | **B. LLM + Kaikki lookup** | LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump | 100% deterministic; decouples gender from model quality | Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries | ### Kaikki Data Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender. | Language | Kaikki File | Coverage | | -------- | ------------------------------------- | -------- | | German | `kaikki.org-dictionary-German.jsonl` | High | | Italian | `kaikki.org-dictionary-Italian.jsonl` | High | | Spanish | `kaikki.org-dictionary-Spanish.jsonl` | High | | French | `kaikki.org-dictionary-French.jsonl` | High | Lookup logic: match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`. ### Decision Pending - If **Qwen2.5-3B** or an online model produces >=90% accurate gender: **Approach A**, no Kaikki needed. - If all tested models fail gender: **Approach B**, implement Kaikki lookup as a post-processing step after LLM enrichment. No implementation work started until the 20-word torture suite resolves this. --- ## 7. Batching Strategy ### Why Batching is Necessary At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost. ### Configurable Batch Size Single config point controls batch size everywhere: ```typescript // config/batch.ts (planned) export const BATCH_CONFIG = { size: 5, // Change to 15 or 50 to test get maxTokens() { return Math.ceil(this.size * 250 * 1.2); // 250 tok/word + 20% buffer }, } as const; ``` ### Prompt Structure **Single word:** ``` Word: house ``` **Batch of 5:** ``` Words: ["house", "car", "tree", "water", "book"] ``` LLM returns a JSON object with word keys: ```json { "house": [ { "sense": "...", "example": "...", ... } ], "car": [ { ... } ], "tree": [ { ... } ], "water": [ { ... } ], "book": [ { ... } ] } ``` ### Retry Strategy If a batch fails (bad JSON, missing key, etc.): ``` Batch of 50 fails | v Retry as 2 batches of 25 | v If a 25 fails, retry as 5 batches of 5 | v If a 5 fails, retry as individual words (fallback) ``` This gives resilience without losing the speed benefit of large batches. ### Expected Impact by Environment | Environment | Batch Size | Expected Speedup | Notes | | ------------------ | ---------- | ---------------- | -------------------------------------------------------- | | Local GTX 950M | 3 | ~1.05-1.15x | Memory bandwidth limited. KV cache pressure on 4GB VRAM. | | Local GTX 950M | 5 | ~1.10-1.20x | Sweet spot for this hardware. | | Local GTX 950M | 15 | Risky | May OOM. Test carefully. | | Local GTX 950M | 50 | Unlikely | VRAM insufficient. | | Cloud API (Gemini) | 50 | 5x fewer calls | Unlocks free tier viability. | | Cloud API (Groq) | 50 | 5x fewer calls | Minimal gain - already fast. | ### Batching is Non-Negotiable Regardless of local vs cloud, batching is required for 100k words: - **Local**: Better GPU utilization, amortized prompt processing - **Cloud**: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts --- ## 8. Hardware Constraints ### Current Machine | Component | Spec | | --------- | ------------------------------------------------------------------- | | OS | Debian GNU/Linux 13 (trixie) x86_64 | | CPU | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz | | GPU | NVIDIA GeForce GTX 950M (Maxwell, 2015) | | GPU VRAM | 4GB | | RAM | 8GB (3.95GB usable at idle) | | Disk | 102GB ext4 (~63GB used) | ### What Fits in 4GB VRAM | Model | File Size | KV Cache (2048 ctx) | Total VRAM | Fits? | | ------------------- | --------- | ------------------- | ---------- | ------------------ | | Qwen2.5-1.5B Q4_K_M | ~1.0GB | ~0.5GB | ~1.5GB | Yes | | Qwen2.5-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes | | Ministral-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes (but broken) | | Qwen3.5-4B Q4_K_M | 2.6GB | ~1.0GB | ~3.6GB | Tight | | Gemma 4 E2B Q4_K_M | 3.46GB | ~1.2GB | ~4.7GB | No | | Gemma 4 E2B IQ2_M | 2.62GB | ~1.0GB | ~3.6GB | Maybe, low quality | ### GPU Rental Alternatives If local hardware becomes the bottleneck: | Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost | | -------- | -------- | ---- | ----------- | ------------------- | ---------- | | Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | **~$2-5** | | RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | **~$4-6** | | Vast.ai | RTX 3090 | 24GB | ~$0.20-0.40 | ~8-10 hours | **~$2-4** | With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day. --- ## 9. Testing & Quality Assurance ### 20-Word Torture Suite Planned test set covering edge cases: | Category | Words | Why | | -------------- | ------------------------------------------------------ | ----------------------------------- | | Easy concrete | `house`, `water`, `book` | Baseline | | Easy abstract | `time`, `love`, `hope` | Abstract nouns harder to define | | Polysemous | `bank`, `run`, `light` | Multiple senses test disambiguation | | Hard/technical | `democracy`, `photosynthesis`, `entropy` | Complex definitions | | False friends | `actual` (en/es), `sensible` (en/fr), `fabric` (en/de) | Cross-lingual traps | ### Evaluation Criteria For each word and each candidate model: | Criterion | Pass Threshold | | ----------------------------- | ------------------------------------------------- | | Definition accuracy | Factually correct, max 15 words, student-friendly | | Example quality | Natural sentence, word used correctly in context | | Translation correctness | Correct word sense match | | Gender accuracy (de/it/es/fr) | >=90% correct | | Difficulty classification | Sensible per CEFR mapping | | JSON reliability | 100% parse rate, valid schema | ### Verification `verify-enriched-file.ts` checks: - Required top-level fields: `word`, `language`, `pos`, `senses` - Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard} - Each translation: `word` (string), `gender` in {masculine, feminine, neuter, null} --- ## 10. Future Extensions & Roadmap ### Near-Term (Next 2-4 Weeks) | Item | Status | Notes | | --------------------- | ----------- | ------------------------------------------- | | Configurable batching | In progress | Single `BATCH_CONFIG.size` value | | 20-word torture suite | Pending | Decides gender approach and model selection | | Qwen2.5-3B evaluation | Pending | Download and test | | Online API testing | Pending | Gemini free tier, DeepSeek, Groq | | Retry logic | Pending | Exponential backoff on LLM failures | ### Medium-Term (1-3 Months) | Item | Notes | | ---------------------------- | ------------------------------------------------------ | | Multi-POS support | Verbs, adjectives, adverbs need prompt variants | | Multi-language source | German -> French, Italian -> Spanish, etc. | | Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously | | Incremental enrichment | Only process new/changed words in a wordlist | | Model auto-switching | Fallback to online API if local server fails | ### Long-Term (3-6 Months) | Item | Notes | | ------------------------ | ---------------------------------------------------------------- | | Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA | | GPU rental integration | Script to spin up Vast.ai/RunPod, run pipeline, download results | | Quality regression tests | Run torture suite on every model change | | Community contributions | Open-source the pipeline for other language learners | --- ## 11. Decisions Log | Date | Decision | Context | Rationale | | ---------- | ----------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------- | | 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app | | 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying | | 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas | | 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys | | 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies | | 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference | | 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM | | 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance | | 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation | | 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality | | 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion | --- ## 12. Known Issues & Dev Notes ### glossa-web (Frontend) | Issue | Details | | ------------------------ | ----------------------------------------------------------------------------------------- | | No healthcheck | Vite dev server has no health endpoint. Docker `HEALTHCHECK` cannot verify running state. | | Valkey memory overcommit | Harmless warning in dev: `vm.overcommit_memory = 1` recommended before production. | ### Data Pipeline | Issue | Details | Severity | | --------------------------------- | ------------------------------------------------------------------------- | ------------------- | | Ministral-3B tokenizer mismatch | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. | Blocker - abandoned | | Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages. | Under evaluation | | No batching | 1 word = 1 call. System prompt re-processed every time. | In progress | | No retry logic | LLM parse failures are logged and skipped. | Planned | | Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes. | Known limitation | ### Hardware | Issue | Details | | --------------------- | ----------------------------------------------------- | | GTX 950M VRAM ceiling | 4GB hard limit. Models >3GB risk OOM. | | Maxwell GPU aging | No Flash Attention support, bandwidth-starved. | | Laptop thermals | Cannot run 24/7 for weeks. Batch processing required. | --- ## 13. How to Run ### Prerequisites - Node.js + npm - `tsx` installed globally: `npm install -g tsx` - llama.cpp built from source - GGUF model downloaded to `~/Downloads/llama.cpp/models/` ### Start the LLM Server ```bash cd ~/Downloads/llama.cpp ./build/bin/llama-server \ -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \ -ngl 999 \ -c 2048 \ -b 512 \ -ub 512 \ -t 2 \ --threads-batch 2 \ --host 127.0.0.1 \ --port 8080 \ --mlock \ --prio 2 ``` ### Run the Pipeline ```bash cd /path/to/data-pipeline npx tsx pipeline.ts ``` ### Expected Output ``` Starting data pipeline... step 1: scanning the source files... Scan complete! Found 1 wordlist(s): - ENGLISH (nouns) ... Pipeline Summary Duration: 23.0s Processed: 3 Skipped: 0 Failed: 0 Total: 3 Throughput: 0.13 words/sec LLM Metrics Calls: 3 Avg prompt tokens: 308 Avg completion tokens: 132 Avg prompt speed: 549.4 tok/s Avg completion speed: 18.8 tok/s Global data pipeline run completed successfully. ``` ### Environment Variables (Online Mode) ```bash export DEEPSEEK_API_KEY="sk-..." export GEMINI_API_KEY="..." export GROQ_API_KEY="..." ``` Then update `config/llm.ts` to point to the online API URL. --- ## 14. Roadmap ### Phase 1: Batching (Current) | Task | Status | Notes | | ------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Implement configurable batch size | In progress | Single `BATCH_CONFIG.size` value. Prompt formatting: single word -> word array. Response parsing: keyed JSON object. Retry: 50 -> 25 -> 5 -> 1. | | Verify batching doesn't break quality | Pending | Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output. | | Measure speedup vs batch size | Pending | Track throughput at 1, 5, 15 on local hardware. | **Goal:** Unlock 5-15x speedup on local, unlock free online API tiers. --- ### Phase 2: Model Selection | Task | Status | Notes | | ------------------------------------------------ | ------- | ------------------------------------------------------------------------------------------- | | Download Qwen2.5-3B Q4_K_M | Pending | ~1.9GB, fits in 4GB VRAM. | | Run torture suite on Qwen2.5-3B (batched) | Pending | Check if gender accuracy improves with 2x params. | | Test Gemini 2.5 Flash-Lite free tier (batched) | Pending | 1,500 req/day x 50 words = 75k words/day. Zero cost. | | Test DeepSeek V4 Flash free tier (batched) | Pending | 5M tokens free. ~9k words. | | Test Groq Llama 3.1 8B (batched, paid if needed) | Pending | Fastest option. ~$7 for 100k words. | | Decide: local vs online, which model | Pending | Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost. | **Goal:** Pick the model and provider for the 100k word run. --- ### Phase 3: Scale | Task | Status | Notes | | ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | | Run 100k word pipeline | Pending | Estimated time depends on Phase 2 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). | | Spot-check output quality | Pending | Random sample of 100 entries. | | Fix gender if needed | Pending | Kaikki lookup post-processing if LLM gender remains unreliable. | | Handle failures & retries | Pending | Exponential backoff, split-and-retry for batch failures. | **Goal:** Complete 100k word dataset. --- ### Phase 4: Extend | Task | Status | Notes | | ---------------------------- | ------- | ------------------------------------------------------------------------------------------------- | | Multi-POS support | Pending | Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.). | | Multi-language source | Pending | German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo. | | Parallel wordlist processing | Pending | Run `english/nouns` and `english/verbs` simultaneously. | | Incremental enrichment | Pending | Only process new/changed words in a wordlist. | | GPU rental integration | Pending | Script to spin up Vast.ai/RunPod, run pipeline, download results. | | Quality regression tests | Pending | Run torture suite on every model change. | **Goal:** Generalize pipeline for any language direction and POS. --- ### Backlog (Unscheduled) | Task | Context | | -------------------------------- | ------------------------------------------------------------------------------------------- | | Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. | | Model auto-switching | Fallback to online API if local server fails mid-run. | | Community open-source | Clean up, document, publish for other language learners. | | Prometheus metrics | `--metrics` flag on llama-server for automated performance tracking. | | `-c 1024` / `-b 256` experiments | Further VRAM optimization on GTX 950M. Low priority if moving to cloud. |