forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

updating documentation, prettier format

Browse source
This commit is contained in:
lila 2026-08-02 01:01:33 +02:00
parent 039ed50567
commit e534b98bc5
16 changed files with 1271 additions and 1070 deletions
Download patch file Download diff file Expand all files Collapse all files

790
documentation/archive/data-pipeline-local-llm.md Normal file
View file

@ -0,0 +1,790 @@
# Lila Data Pipeline — Technical Documentation
Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer.
Last updated: 2026-07-18
## Table of Contents
- [1. Executive Summary](#1-executive-summary)
- [2. Problem & Context](#2-problem--context)
- [3. Architecture Overview](#3-architecture-overview)
- [4. Current Implementation](#4-current-implementation)
- [5. The LLM Layer](#5-the-llm-layer)
- 5.1 [Local Model Evaluation](#51-local-model-evaluation)
- 5.2 [Online API Options](#52-online-api-options)
- 5.3 [Model Selection Criteria](#53-model-selection-criteria)
- [6. The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration)
- [7. Batching Strategy](#7-batching-strategy)
- [8. Hardware Constraints](#8-hardware-constraints)
- [9. Testing & Quality Assurance](#9-testing--quality-assurance)
- [10. Interactive CLI](#10-interactive-cli)
- [11. Future Extensions & Roadmap](#11-future-extensions--roadmap)
- [12. Decisions Log](#12-decisions-log)
- [13. Known Issues & Dev Notes](#13-known-issues--dev-notes)
- [14. How to Run](#14-how-to-run)
- [15. Roadmap](#15-roadmap)
## Quick Reference
| What | Where |
| ------------------- | ------------------------------------------ |
| Entry point | `pipeline.ts` |
| Interactive CLI | `utils/cli.ts` |
| LLM config schema | `config/llm.ts` |
| System prompt | `config/prompt.ts` — `buildSystemPrompt()` |
| Batch config schema | `config/batch.ts` |
| Shared constants | `config/constants.ts` |
| Output schema | `utils/merge-enriched-data.ts` |
| LLM adapters | `utils/llm-adapters/` |
| **Current model** | **`gemma-4-E2B_q4_0-it.gguf`** |
| Target scale | 100,000+ words |
---
## 1. Executive Summary
The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:
- One or more senses (definitions)
- A natural example sentence per sense
- A CEFR-based difficulty level (`easy` / `medium` / `hard`)
- Translations into all target languages except the source (as raw strings)
The pipeline is designed to scale to 100,000+ words across multiple languages and parts of speech. It supports both local inference (`llama.cpp`) and cloud providers via a pluggable adapter pattern.
### Key Design Principles
| Principle | Rationale |
| ------------------------------- | ------------------------------------------------------------------------------------------------------ |
| **Quality first** | Definitions, examples, and translations must be accurate. Speed and cost are secondary. |
| **Local-first, cloud-fallback** | Local LLMs are the default for cost control and data privacy. |
| **Deterministic Grammar** | Grammatical gender is decoupled from the LLM and resolved via Kaikki Wiktionary dumps. |
| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. |
| **Configurable batching** | Batch size is selected interactively at runtime. The pipeline adapts without code changes. |
| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only. |
### Resolved: Gender Accuracy
Grammatical gender is **no longer generated by the LLM**. Comprehensive testing across 10 models (2026-07-18) confirmed that small local models systematically hallucinate or default to `neuter` for Romance languages.
**Decision:** The LLM only outputs translation strings. A deterministic post-processing step looks up the exact grammatical gender from Kaikki Wiktionary dumps. This guarantees 100% gender accuracy and allows us to use smaller, faster, and highly nuanced local models.
### Current Status (2026-07-18)
- **Core pipeline:** Complete (scanning, enrichment, merging, verification, writing).
- **Local LLM:** Gemma 4 E2B selected as production model after exhaustive 10-model evaluation. `llama.cpp` server optimized with KV-cache quantization.
- **Batching:** 20-word batches validated for local hardware.
- **Gender:** Decoupled from LLM; Kaikki lookup architecture confirmed.
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) as speed fallback.
### One-Line Architecture
```text
source wordlists -> Interactive CLI -> LLM adapter -> merge senses -> Kaikki Gender Lookup -> verify schema -> write .json
```
### Files at a Glance
| File | Purpose |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `pipeline.ts` | Orchestrator. Runs CLI, scans sources, loops words, coordinates all stages |
| `utils/cli.ts` | Interactive CLI. Provider/model/batch selection, config persistence |
| `config/llm.ts` | LLM config schema (provider, url, model) |
| `config/prompt.ts` | `buildSystemPrompt()` — dynamic prompt with auto-target languages |
| `config/batch.ts` | Batch size and max retry count schema |
| `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
| `utils/enrich-word.ts` | Calls LLM via adapter, parses response, validates senses, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON |
| `utils/verify-enriched-file.ts` | Schema validation (required fields, types) |
| `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers |
| `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory |
| `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files |
| `utils/write-json-file.ts` | Atomic `.tmp` → rename writes |
| `utils/check-if-json-exists.ts` | Resumability: checks if word already has enriched senses |
| `utils/create-line-reader.ts` | Streaming line reader for large wordlists |
| `utils/create-output-dirs.ts` | Creates `worddata/{language}/{pos}/` folders |
| `utils/delete-file.ts` | Cleanup helper for failed batches |
| `utils/get-word-file-path.ts` | Path construction helper |
| `utils/progress-tracker.ts` | `[current/total]` formatting for console output |
| `utils/pipeline-timer.ts` | Timing + token metrics; unified throughput for all providers |
| `utils/llm-adapters/factory.ts` | Creates the right adapter based on runtime config |
| `utils/llm-adapters/types.ts` | `LlmAdapter` interface |
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek |
| `utils/llm-adapters/gemini.ts` | Google Gemini native API |
### Scale Target
| Metric | Target |
| --------------- | ------------------------------------------------------------ |
| Words | 100,000+ |
| Languages | English (source), German, Italian, Spanish, French (targets) |
| Parts of speech | Nouns, verbs, adjectives, adverbs |
| Output | One `.json` file per word, ~2-5KB each |
---
## 2. Problem & Context
### Why Build This?
Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:
| What Exists | What is Missing | Why It Matters |
| ------------------------ | ---------------------------------- | ------------------------------------------------------------- |
| Word + translation | Student-friendly definition | Learners need explanations, not just equivalents |
| Static difficulty labels | CEFR-calibrated difficulty | "Easy" vs "hard" must map to learner proficiency levels |
| Isolated translations | Natural example sentences | Context is how vocabulary is actually acquired |
| Raw gender data | Gender integrated with translation | Flashcards must show gender immediately (resolved via Kaikki) |
| Bulk word lists | Per-word structured JSON | The trainer consumes one file per word for fast random access |
### The Target User
A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender — all calibrated to their CEFR level (A1-C2).
### Why Not Use Existing Dictionaries?
- **Wiktionary:** Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
- **Kaikki (Wiktionary dump):** Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
- **Google Translate / DeepL:** No definitions, no examples, no difficulty, no structured output
- **Existing language learning apps:** Closed data, no export, no control over content
The LLM fills the gap: it generates pedagogical content (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.
### Language Direction
The pipeline is direction-agnostic. A wordlist is defined by:
- **Source language:** the language of the input words
- **Target languages:** all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source)
Current focus: English -> German/Italian/Spanish/French. Planned directions include German -> French, Italian -> Spanish, etc.
### Why 100,000+ Words?
- **Coverage:** A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels.
- **Languages:** English (source) -> German, Italian, Spanish, French (targets).
- **Parts of speech:** Nouns (current), verbs, adjectives, adverbs.
### The Quality Challenge
| Risk | Mitigation |
| ------------------------------- | -------------------------------------------------------------- |
| Hallucinated definitions | Low temperature (0.1), strict system prompt, schema validation |
| Incorrect grammatical gender | **Resolved:** Decoupled from LLM; Kaikki Wiktionary lookup |
| POS bleed (verb defs for nouns) | Explicit negative constraint in system prompt |
| Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking |
| JSON parse failures | Retry + split logic, schema validation, cleanup on failure |
| Model drift (online APIs) | Version pinning, local fallback |
| Provider downtime | Adapter pattern allows hot-swapping providers |
### Why TypeScript + Node?
- **Familiarity:** Existing project uses TypeScript (frontend in TanStack Router + React)
- **Ecosystem:** `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP
- **Portability:** Runs on the same Debian laptop as the llama.cpp server
- **No build complexity:** `tsx` for direct execution, no bundler needed
### Why llama.cpp?
- **GGUF format:** Single-file models, easy to swap, quantize, and version
- **OpenAI-compatible API:** `/v1/chat/completions` means the same adapter code works for local and online models
- **No dependencies:** Self-contained binary, runs on old hardware (tested on GTX 950M)
- **Privacy:** Local inference means no data leaves the machine
---
## 3. Architecture Overview
### Pipeline Flow
```text
Run CLI -> Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
-> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
-> Retry/split on failure -> Merge -> Kaikki Gender Lookup -> Write atomically -> Verify schema -> Log metrics
```
### Resumability
- **Skip existing:** `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses`
- **Atomic writes:** `.tmp` -> rename in `write-json-file.ts`, no partial files on crash
- **Cleanup on failure:** Deletes partially-written files for failed batches, continues to next batch
### Directory Structure
```text
data-pipeline/
|-- pipeline.ts # Entry point / orchestrator
|-- utils/
| |-- cli.ts # Interactive CLI module
| |-- enrich-word.ts # LLM call, parse, retry/split
| |-- merge-enriched-data.ts # Schema types + merge logic
| |-- verify-enriched-file.ts # Schema validation
| |-- check-llm-server.ts # Health check (local only)
| |-- scanning-source-files.ts # Source discovery
| |-- create-base-json.ts # Skeleton writer
| |-- write-json-file.ts # Atomic JSON writer
| |-- check-if-json-exists.ts # Resumability check
| |-- create-line-reader.ts # Streaming file reader
| |-- create-output-dirs.ts # Directory creation
| |-- delete-file.ts # Cleanup helper
| |-- get-word-file-path.ts # Path helper
| |-- progress-tracker.ts # Console progress formatting
| |-- pipeline-timer.ts # Timing + token metrics
| |-- llm-adapters/
| |-- factory.ts # Adapter selection
| |-- types.ts # LlmAdapter interface
| |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
| |-- gemini.ts # Google Gemini
|-- config/
| |-- llm.ts # LLM config schema
| |-- prompt.ts # buildSystemPrompt()
| |-- batch.ts # Batch config schema
| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES
|-- source-data/
| |-- {language}/
| |-- {pos} # One word per line, no extension
|-- worddata/
| |-- {language}/
| |-- {pos}/
| |-- {word}.json # One self-contained file per word
|-- kaikki-source-files/ # Wiktionary dumps for gender lookup
|-- .pipeline-config.json # Saved CLI configuration
```
### Output Schema
Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `id`, `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`.
_Note: The `translations` object contains arrays of strings (e.g., `{"de": ["Haus", "Gebäude"]}`). Grammatical gender is appended later via the Kaikki integration step._
### Error Handling
| Failure | Behavior |
| ------------------------------ | -------------------------------------------------------- |
| LLM server offline | Hard fail at startup (`check-llm-server.ts`, local only) |
| LLM returns bad JSON | Retry up to 3 times, then split batch. Log and continue |
| LLM returns malformed senses | `validateSense()` catches it before file write |
| Schema validation fails | Log warnings, keep file |
| Individual batch fails | Does not stop pipeline; cleans up partial files |
| Individual word fails (size 1) | Log and continue to next word |
### Metrics
Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`.
- **Unified throughput (all providers):** total tokens / total request time
- **Detailed breakdown (local only):** prompt speed vs completion speed
---
## 4. Current Implementation
### Tech Stack
| Layer | Choice | Why |
| ----------- | ----------------------------- | ------------------------------------------ |
| Runtime | Node.js + `tsx` | Direct TypeScript execution, no build step |
| HTTP client | Native `fetch` | Works for local llama.cpp and online APIs |
| File I/O | `fs` + `readline` | Streaming line reader for large wordlists |
| JSON | Native `JSON.parse/stringify` | Simple, no schema library needed |
| CLI | Native `readline` | No external dependencies |
### Current Model
| Property | Value |
| ---------------- | ---------------------------------------------------- |
| **Model** | **`gemma-4-E2B_q4_0-it.gguf`** |
| Size | ~3.2GB (file) / ~2.06GB (VRAM weights) |
| Quantization | Q4_0 |
| Server | `llama.cpp` (`llama-server`) |
| API | OpenAI-compatible `/v1/chat/completions` |
| VRAM Usage | ~2.65GB total (weights + KV cache + compute buffers) |
| Generation Speed | ~10.9 tok/s (20-word batch) |
| Est. 100k Time | ~7 days (20-word batches, 24/7) |
### `llama-server` Flags: History & Rationale
The server flags evolved through rigorous empirical testing on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM) across 10 different models on 2026-07-18.
#### Flag Evolution
| Flag | Value Tried | Result | Why |
| ----------------- | ------------------------------------------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~6.3 tok/s | Quality baseline. Correct translations. 2.6GB, tight on VRAM. |
| `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~18.5 tok/s | Fast but poor instruction following, gender wrong. |
| `-m` | `Qwen3.5-2B-Q4_K_M.gguf` | Works, ~13.2 tok/s | Good translations, but failed polysemy (2 identical senses). |
| `-m` | `Llama-3.2-3B-Instruct-Q4_K_M.gguf` | Works, ~9.1 tok/s | Dangerous false friend trap (cognates). |
| `-m` | `Llama-3.2-3B-Instruct-UD-Q6_K_XL.gguf` | Works, ~6.7 tok/s, 3.44GB VRAM | Higher quant did NOT fix Llama's translation issues. Slower. |
| `-m` | `Ministral-3-3B-Instruct-2512-Q4_K_M.gguf` | Works, ~8.9 tok/s | Fixed tokenizer bug (older version was broken). But messy translations, markdown violations. |
| `-m` | `gemma-4-E2B_q4_0-it.gguf` | **Works, ~13.1 tok/s, 2.06GB** | **Winner.** Perfect polysemy, false friends, nuance. Half VRAM of Qwen 4B. |
| `-m` | `gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf` | Works, ~7.7 tok/s, 3.23GB | QAT compression is incredible. Same quality as E2B but slower. |
| `-m` | `qwen3.5-9b-q3_k_s.gguf` | Works, ~3.3 tok/s (split CPU/GPU) | Brilliant quality but bottlenecked by CPU/Swap. `-ngl 28` max. |
| `-m` | `qwen3.5-9b-q4_k_m.gguf` | Works, ~3.3 tok/s (split CPU/GPU) | Same quality as 9B Q3. Not worth 2x the file size. |
| `-ngl` | 999 | Keeps | Offload all layers to GPU. Required for any speed. |
| `-ngl` | 28 | 9B models only | Max GPU layers for 9B models before OOM. Found via binary search. |
| `-ngl` | 30 | OOM crash (9B) | Pushed 2 layers too far into compute buffers. |
| `-c` | 4096 | Wasteful | 4K context for 300-token dictionary entries wastes VRAM. |
| `-c` | 2048 | Good for small batches | Sufficient for 4-word batches. |
| `-c` | **8192** | **Current** | Required for 20-word batches. Combined with KV cache quantization. |
| `-b` / `-ub` | 512 | **Current** | Sweet spot for Maxwell memory bandwidth. |
| `-b` / `-ub` | 1024 | Tested | Slightly faster prompt processing, but no generation speedup. |
| `-b` / `-ub` | 2048 | Slower on 950M | Memory pressure on bandwidth-starved GPU. |
| `-t` | 4 | Slower | Hyperthreading cores hurt llama.cpp performance. |
| `-t` | **2** | **Current** | Matches 2 physical cores. |
| `--threads-batch` | **2** | **Current** | Explicit match to `-t`. |
| `--flash-attn` | (omitted) | Correct | On Maxwell (compute 5.0), Flash Attention adds overhead. |
| `--mlock` | Tested | Omitted for large models | Pins model in RAM. Causes OOM on models >2.5GB with large KV cache. |
| `--prio` | **2** | **Current** | Raises process priority. Marginal, harmless. |
| `--reasoning` | **off** | **Critical** | **Mandatory for Qwen 3.5 and Gemma 4.** Without this, models "think" silently, consume all `max_tokens`, and crash with `finish_reason: length`. |
| `--cache-type-k` | **q4_0** | **Current** | Compresses KV cache keys to 4-bit. Cuts KV VRAM by ~75%. Enables 8192 context on 4GB GPU. |
| `--cache-type-v` | **q4_0** | **Current** | Compresses KV cache values to 4-bit. Paradoxically improves translation variety (reduces "lazy duplication" bug). |
#### Current Production Command
```bash
cd ~/Downloads/llama.cpp
./build/bin/llama-server \
-m models/gemma-4-E2B_q4_0-it.gguf \
-ngl 999 \
-c 8192 \
-b 512 \
-ub 512 \
-t 2 \
--threads-batch 2 \
--host 127.0.0.1 \
--port 8080 \
--prio 2 \
--reasoning off \
--cache-type-k q4_0 \
--cache-type-v q4_0
```
#### VRAM Budget (Production Config)
| Component | VRAM Usage |
| ------------------------------------ | ------------------------- |
| Model Weights (Gemma 4 E2B Q4_0) | ~1.50 GB |
| KV Cache (8192 ctx, q4_0 compressed) | ~0.35 GB |
| Compute Buffers (batch 512) | ~0.20 GB |
| **Total** | **~2.05 GB (51% of 4GB)** |
#### Why Not Use the Remaining 2GB VRAM?
Generation speed is bottlenecked by **memory bandwidth** (~32 GB/s on GTX 950M), not VRAM capacity. To generate one token, the GPU must read the entire ~1.5GB model from VRAM. The theoretical maximum is ~21 tok/s. At 10.9 tok/s, the GPU is already operating at ~50% of its physical limit. Empty VRAM cannot be converted into faster generation.
Testing uncompressed `f16` KV cache (2.65GB total VRAM) yielded 12.6 tok/s but caused a severe "lazy duplication" regression (model copy-pasted the same translation twice instead of providing distinct synonyms). The `q4_0` compressed KV cache is the correct choice for translation quality.
### Performance Baseline (20-Word Batch)
| Metric | Gemma 4 E2B | Qwen 3.5 4B |
| --------------------- | ----------- | ------------ |
| Time/batch (20 words) | ~5 min | ~9 min |
| Completion tok/s | ~10.9 | ~6.1 |
| Prompt tok/s | ~132 | ~78 |
| VRAM Usage | 2.06 GB | 3.95 GB |
| Lazy Duplication Bug | No | Yes (severe) |
---
## 5. The LLM Layer
### 5.1 Local Model Evaluation (Complete — 2026-07-18)
All 10 downloaded models were evaluated on the target hardware. Testing progressed from 4-word smoke tests to a 20-word "nightmare" torture suite covering extreme polysemy, false friends, abstract concepts, and legal/financial terminology.
#### Final Leaderboard
| Rank | Model | Size | VRAM | Speed (tok/s) | Polysemy | False Friends | Verdict |
| ------ | ------------------------ | ---- | ------------- | ------------- | --------------------------- | ------------------------- | ----------------------------------------------------------- |
| **🥇** | **Gemma 4 E2B Q4_0** | 3.2G | **2.06 GB** | **~13.1** | **Perfect** | **Perfect** | **Production model.** Best quality/speed/VRAM ratio. |
| 🥈 | Qwen 3.5 4B Q4_K_M | 2.6G | ~3.6 GB | ~6.3 | Perfect | Perfect | Quality King, but 2x slower and maxes VRAM. |
| 🥉 | Gemma 4 E4B Q4_K_XL | 4.0G | 3.23 GB | ~7.7 | Perfect | Perfect | Incredible QAT compression. Same quality as E2B but slower. |
| 4 | Qwen 3.5 2B Q4_K_M | 1.2G | ~1.6G | ~13.2 | Failed (2 identical senses) | Passed | Good translations, lacks conceptual branching. |
| 5 | Qwen 2.5 1.5B Q4_K_M | 1.1G | ~1.5G | ~18.5 | Ignored instruction | Failed | Fast but easily confused. |
| 6 | Ministral 3B 2512 Q4_K_M | 2.0G | ~2.7G | ~8.9 | Messy translations | Failed | Fixed tokenizer, but outclassed. Markdown violations. |
| 7 | Llama 3.2 3B Q4_K_M | 1.9G | ~2.6G | ~9.1 | Good structure, bad IT/ES | **Failed (cognate trap)** | Dangerous for language learners. |
| 8 | Llama 3.2 3B Q6_K_XL | 2.8G | 3.44G | ~6.7 | Good structure, bad IT/ES | **Failed (cognate trap)** | Higher quant did NOT fix translation issues. |
| 9 | Qwen 3.5 9B Q3_K_S | 4.1G | Split CPU/GPU | ~3.3 | Perfect | Perfect | Brilliant but 35-40 days for 100k words. |
| 10 | Qwen 3.5 9B Q4_K_M | 5.3G | Split CPU/GPU | ~3.3 | Perfect | Perfect | Same quality as 9B Q3. Not worth the size. |
#### Key Findings
1. **The "Thinking" Trap:** Both Qwen 3.5 and Gemma 4 have built-in Chain-of-Thought reasoning. Without `--reasoning off`, they silently "think" in a hidden JSON field, consume all `max_tokens`, and crash with `finish_reason: length`. This flag is **mandatory**.
2. **The 2B vs 4B Quality Cliff:** 2B models struggle with polysemy (e.g., cannot distinguish "bank" = financial vs river). 4B+ models act like professional lexicographers.
3. **Llama 3.2 is Unsafe for Language Learners:** Consistently fell for false friend traps (e.g., translating "actual" = _real_ to _aktuell/attuale/actual/actuel_ = _current_).
4. **KV Cache Quantization Improves Translation Variety:** Compressing the KV cache to `q4_0` introduces microscopic noise that prevents the "lazy duplication" bug (model copy-pasting the same synonym twice).
5. **POS Bleed is Universal:** All models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_ instead of _a jogging session_). Fix: explicit negative constraint in the system prompt.
### 5.2 Online API Options
Evaluated as fallbacks if local models fail quality or speed targets.
| Provider | Model | Input $/1M | Output $/1M | Free Tier | Rate Limit | Est. Cost (100k words) | Est. Time |
| --------------- | -------------------- | ---------- | ----------- | ------------- | ---------- | ---------------------- | ------------------- |
| DeepSeek | V4 Flash | $0.14 | $0.28 | 5M tokens | None | $0-15 | ~1-2 days |
| Gemini | 2.5 Flash-Lite | $0.10 | $0.40 | 1,500 req/day | 1M TPM | $0 (free tier) | ~1.5 days (batched) |
| Qwen/Alibaba | Qwen-Turbo | $0.05 | $0.20 | Unknown | 600 RPM | ~$11 | ~1-2 days |
| Groq | Llama 3.1 8B Instant | $0.05 | $0.08 | Yes | High | ~$7 | ~3-4 hours |
| OpenRouter free | Various | $0 | $0 | 200 req/day | 20 RPM | $0 | ~10 days |
### 5.3 Model Selection Criteria
Decision flow for 100,000 words:
```text
Start
|
v
Gemma 4 E2B (local) — SELECTED
|
|-- Speed acceptable? (~7 days) -----> Use Gemma 4 E2B locally, $0
|
|-- Need faster? -------> Test Gemini 2.5 Flash-Lite (free)
|
|-- Quality good? --> Batch 50, free tier
| ~1.5 days, $0
|
|-- Quality meh? ---> Test Groq or DeepSeek paid
```
Quality gates:
- 100% JSON parse rate
- No hallucinated definitions on polysemous words
- Natural, contextually appropriate example sentences
- Sensible difficulty classification (CEFR mapping)
- Gender accuracy is no longer an LLM criterion (handled by Kaikki)
---
## 6. The Gender Problem & Kaikki Integration
### The Problem (Resolved)
Grammatical gender was originally embedded in the LLM's `translations` object. Testing across all 10 models confirmed that small local models systematically hallucinate gender, defaulting to `neuter` for Romance languages (Italian, Spanish, French) which do not have a neuter grammatical gender.
### The Solution: Decoupled Architecture
**Decision (2026-07-18):** Grammatical gender is no longer generated by the LLM. The pipeline now uses a two-stage approach:
| Stage | Component | Responsibility |
| ----- | ----------------- | -------------------------------------------------------------------- |
| 1 | LLM (Gemma 4 E2B) | Generates translation **strings only** (e.g., `["Haus", "Gebäude"]`) |
| 2 | Kaikki Lookup | Deterministically resolves grammatical gender from Wiktionary dumps |
### Benefits
- **100% deterministic gender accuracy** — no hallucination possible
- **Faster LLM generation** — ~15-20% fewer output tokens per word
- **Simpler JSON schema** — translations are string arrays, not object arrays
- **Model-agnostic** — works with any LLM regardless of multilingual training quality
### Kaikki Data
| Language | Kaikki File | Coverage |
| -------- | ------------------------------------- | -------- |
| German | `kaikki.org-dictionary-German.jsonl` | High |
| Italian | `kaikki.org-dictionary-Italian.jsonl` | High |
| Spanish | `kaikki.org-dictionary-Spanish.jsonl` | High |
| French | `kaikki.org-dictionary-French.jsonl` | High |
### Lookup Logic
Match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`.
---
## 7. Batching Strategy
### Why Batching is Necessary
At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.
### Optimal Batch Size (Local)
**20 words** is the validated sweet spot for the GTX 950M with Gemma 4 E2B.
| Batch Size | VRAM | Speed | Quality | Verdict |
| ---------- | --------------- | ---------------- | ------------------------ | -------------------------------------------------- |
| 1-4 | ~2.1 GB | ~13 tok/s | Perfect | Safe but slow (amortization waste) |
| **20** | **~2.6 GB** | **~10.9 tok/s** | **Perfect** | **Sweet spot** |
| 30-40 | ~3.0 GB (est.) | ~10 tok/s (est.) | Likely good | Worth testing |
| 50+ | ~3.5 GB+ (est.) | Unknown | Risk of JSON degradation | **Not recommended for 2B models** |
| 100 | OOM risk | N/A | Attention degradation | Small models lose JSON structure past ~6000 tokens |
### Retry & Split Strategy
If a batch fails (bad JSON, missing key, etc.):
```text
Batch of 20 fails (3 retries exhausted)
|
v
Split into 2 batches of 10
|
v
If a batch of 10 fails (3 retries), split into 2 batches of 5
|
v
If a batch of 5 fails, split into batches of 1
|
v
If a single word fails (3 retries), log and skip
```
---
## 8. Hardware Constraints
### Current Machine
| Component | Spec |
| ------------- | ------------------------------------------------------------------- |
| OS | Debian GNU/Linux 13 (trixie) x86_64 |
| CPU | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz |
| GPU | NVIDIA GeForce GTX 950M (Maxwell, 2015) |
| GPU VRAM | 4GB (4037 MiB reported by CUDA) |
| GPU Bandwidth | ~32 GB/s (DDR3) |
| RAM | 8GB (~3.67GB usable at idle) |
| Swap | 5.62 GB |
| Disk | 102GB ext4 (~74GB used) |
### What Fits in 4GB VRAM (Empirically Verified)
| Model | File Size | Total VRAM | Fits? | Notes |
| ------------------------ | --------- | ----------- | ---------- | -------------------------------------- |
| Qwen 2.5 1.5B Q4_K_M | 1.1G | ~1.5 GB | ✅ Easy | |
| Qwen 3.5 2B Q4_K_M | 1.2G | ~1.6 GB | ✅ Easy | |
| Ministral 3B 2512 Q4_K_M | 2.0G | ~2.7 GB | ✅ Yes | |
| Llama 3.2 3B Q4_K_M | 1.9G | ~2.6 GB | ✅ Yes | |
| Llama 3.2 3B Q6_K_XL | 2.8G | 3.44 GB | ✅ Tight | |
| **Gemma 4 E2B Q4_0** | **3.2G** | **2.06 GB** | **✅ Yes** | **QAT compression. Production model.** |
| Gemma 4 E4B Q4_K_XL | 4.0G | 3.23 GB | ✅ Yes | QAT compression is incredible. |
| Qwen 3.5 4B Q4_K_M | 2.6G | 3.95 GB | ⚠️ Barely | 50MB headroom with 8192 ctx. |
| Qwen 3.5 9B Q3_K_S | 4.1G | Split | ⚠️ Partial | `-ngl 28` max. Rest on CPU/Swap. |
| Qwen 3.5 9B Q4_K_M | 5.3G | Split | ⚠️ Partial | `-ngl 20` max. Heavy swap usage. |
### GPU Rental Alternatives
If local hardware becomes the bottleneck:
| Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost |
| --- | --- | --- | --- | --- | --- |
| Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | ~$2-5 |
| RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | ~$4-6 |
---
## 9. Testing & Quality Assurance
### 20-Word Torture Suite (Completed 2026-07-18)
Tested on Gemma 4 E2B and Qwen 3.5 4B with 20 challenging nouns:
| Category | Words | The Trap |
| ----------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
| Extreme Polysemy | `match`, `date`, `right`, `set`, `well` | Does it translate "match" as fire, sports, or dating? |
| False Friends | `sense`, `fabric`, `sympathy`, `eventuality`, `billion` | "Fabric" = material (_tissu_) not factory (_fabrique_) |
| Abstract/Cultural | `serendipity`, `accountability` | Concepts lacking 1:1 dictionary equivalents |
| Action-Nouns | `run`, `drive`, `play` | "Run" as jog vs tear vs campaign |
| Legal/Financial | `mortgage`, `lease`, `court`, `board`, `draft` | Requires specific legal vocabulary (_Hypothek/mutuo/hipoteca_) |
### Results Summary
| Criterion | Gemma 4 E2B | Qwen 3.5 4B |
| ------------------------------------ | ------------------------------------------ | ------------------------------------------------------- |
| JSON Reliability | 10/10 (raw JSON) | 10/10 (raw JSON, but added `sense_index` hallucination) |
| Polysemy (bank, match) | 10/10 (Ufer/riva/orilla/rive) | 10/10 |
| False Friends (fabric) | 10/10 (Stoff/tessuto/tela/tissu) | 10/10 |
| Legal Nuance (mortgage) | 10/10 (Hypothek/mutuo/hipoteca/hypothèque) | 10/10 |
| Lazy Duplication Bug | None | **Severe** (copy-pasted same word 20+ times) |
| POS Bleed | Minor (verb defs for run/match) | Minor (verb defs for run/match) |
| Attention Degradation (word 20 vs 1) | None | None |
### Verification
`verify-enriched-file.ts` checks:
- Required top-level fields: `word`, `language`, `pos`, `senses`
- Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard}
- Each translation: array of strings (gender appended later by Kaikki)
---
## 10. Interactive CLI
### Batch Size Recommendations (Updated)
| Provider | Recommended | Rationale |
| -------------------- | ----------- | ----------------------------------------------- |
| **local (GTX 950M)** | **20** | Validated sweet spot. 8192 ctx + q4_0 KV cache. |
| local (RTX 4090) | 50 | Fast, more VRAM |
| gemini | 50 | Free tier: 1,500 req/day |
| deepseek | 20 | 5M free tokens |
| groq | 50 | Very fast |
---
## 11. Future Extensions & Roadmap
### Near-Term (Next 2-4 Weeks)
| Item | Status | Notes |
| ----------------------------------- | ------------ | --------------------------------------------------- |
| 10-model evaluation | **Complete** | Gemma 4 E2B selected |
| KV cache quantization | **Complete** | `--cache-type-k/v q4_0` enables 8192 ctx on 4GB GPU |
| Gender decoupling | **Complete** | Kaikki lookup replaces LLM gender |
| 20-word torture suite | **Complete** | Validated on Gemma E2B and Qwen 4B |
| POS bleed fix | **Pending** | Add negative constraint to system prompt |
| Kaikki gender lookup implementation | **Pending** | Post-processing step after LLM enrichment |
| Online API testing | **Pending** | Gemini free tier, DeepSeek, Groq |
### Medium-Term (1-3 Months)
| Item | Notes |
| ---------------------------- | ------------------------------------------------------ |
| Multi-POS support | Verbs, adjectives, adverbs need prompt variants |
| Multi-language source | German -> French, Italian -> Spanish, etc. |
| Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously |
| Incremental enrichment | Only process new/changed words in a wordlist |
### Long-Term (3-6 Months)
| Item | Notes |
| ------------------------ | ---------------------------------------------------------------- |
| GPU rental integration | Script to spin up Vast.ai/RunPod, run pipeline, download results |
| Quality regression tests | Run torture suite on every model change |
| Community contributions | Open-source the pipeline for other language learners |
---
## 12. Decisions Log
| Date | Decision | Context | Rationale |
| -------------- | ----------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing |
| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API |
| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Community standard for 4-bit |
| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp |
| 2026-06-17 | Qwen2.5-1.5B as initial model | Qwen3.5-4B too slow (47s/word) | 6x speedup, quality under evaluation |
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory |
| 2026-07-06 | Retry + split batching | LLM JSON parse failures | 3 retries, then halve batch |
| 2026-07-06 | Interactive CLI | Editing config files is error-prone | `utils/cli.ts` with native readline |
| **2026-07-18** | **Gemma 4 E2B as production model** | **10-model evaluation completed** | **2x faster than Qwen 4B, half VRAM, perfect translation quality** |
| **2026-07-18** | **Gender decoupled from LLM** | **All 10 models failed gender for Romance languages** | **Kaikki Wiktionary lookup is deterministic and 100% accurate** |
| **2026-07-18** | **`--reasoning off` is mandatory** | **Qwen 3.5 and Gemma 4 "think" silently, consuming all tokens** | **Without this flag, output crashes with `finish_reason: length`** |
| **2026-07-18** | **KV cache quantization (`q4_0`)** | **8192 context needed for 20-word batches** | **Cuts KV VRAM by 75%, enables large batches on 4GB GPU, improves translation variety** |
| **2026-07-18** | **20-word batch size for local** | **Tested 4, 20 words** | **Sweet spot: no attention degradation, no JSON breakage, 10.9 tok/s** |
| **2026-07-18** | **Llama 3.2 discarded** | **Failed false friend tests** | **Translates "actual" (real) to cognates (aktuell/attuale) = "current". Dangerous for learners.** |
| **2026-07-18** | **`-c 8192` replaces `-c 2048`** | **20-word batches need more context** | **Combined with q4_0 KV cache, fits in 2.65GB VRAM** |
---
## 13. Known Issues & Dev Notes
### Data Pipeline
| Issue | Details | Severity |
| ------------------------------ | -------------------------------------------------------------------------------------- | ----------------------------------------------- |
| POS bleed (all models) | Models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_) | Medium — fix with negative constraint in prompt |
| Lazy duplication (Qwen 4B) | Qwen 3.5 4B copy-pastes the same translation twice to fill arrays | Medium — use Gemma 4 E2B instead |
| Schema hallucination (Qwen 4B) | Adds `sense_index` field not in schema | Low — ignorable |
| Pre-scanning wordlists | Entire file read into memory before processing | Medium — streaming refactor planned |
| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes | Known limitation |
### Hardware
| Issue | Details |
| --------------------- | ---------------------------------------------------------------------------- |
| GTX 950M VRAM ceiling | 4GB hard limit. Models >3.5GB need KV cache quantization or CPU offloading. |
| Maxwell GPU aging | No Flash Attention, bandwidth-starved (~32 GB/s). Theoretical max ~21 tok/s. |
| Laptop thermals | Cannot run 24/7 for weeks unattended. Monitor temps. |
---
## 14. How to Run
### Prerequisites
- Node.js + npm
- `tsx` installed globally: `npm install -g tsx`
- llama.cpp built from source (for local mode)
- GGUF model downloaded to `~/Downloads/llama.cpp/models/`
- API keys set as environment variables (for cloud mode)
### Start the LLM Server (Local Mode)
```bash
cd ~/Downloads/llama.cpp
./build/bin/llama-server \
-m models/gemma-4-E2B_q4_0-it.gguf \
-ngl 999 \
-c 8192 \
-b 512 \
-ub 512 \
-t 2 \
--threads-batch 2 \
--host 127.0.0.1 \
--port 8080 \
--prio 2 \
--reasoning off \
--cache-type-k q4_0 \
--cache-type-v q4_0
```
### Run the Pipeline
```bash
cd /path/to/data-pipeline
npx tsx pipeline.ts
```
Follow the interactive prompts to select provider, model, and batch size.
---
## 15. Roadmap
### Phase 1: Batching (Complete)
| Task | Status |
| --------------------------------------- | ------------ |
| Implement configurable batch size | Complete |
| Implement retry + split logic | Complete |
| Honest timing metrics | Complete |
| Validate LLM responses | Complete |
| Verify batching quality (20-word suite) | **Complete** |
### Phase 2: Interactive CLI (Complete)
| Task | Status |
| --------------------- | -------- |
| Design prompt flow | Complete |
| Implement CLI module | Complete |
| Save/load config | Complete |
| Wire into pipeline.ts | Complete |
### Phase 3: Model Selection (Complete)
| Task | Status |
| ----------------------- | -------------------------- |
| 10-model evaluation | **Complete** |
| 20-word torture suite | **Complete** |
| Select production model | **Complete (Gemma 4 E2B)** |
| Test online APIs | Pending |
### Phase 4: Scale
| Task | Status | Notes |
| ------------------------------ | ------- | ------------------------------------------ |
| Implement Kaikki gender lookup | Pending | Post-processing step |
| Fix POS bleed in prompt | Pending | Add negative constraint |
| Run 100k word pipeline | Pending | ~7 days local (Gemma E2B, 20-word batches) |
| Spot-check output quality | Pending | Random sample of 100 entries |
### Phase 5: Extend
| Task | Status |
| ---------------------------- | ------- |
| Multi-POS support | Pending |
| Multi-language source | Pending |
| Parallel wordlist processing | Pending |
| GPU rental integration | Pending |
| Quality regression tests | Pending |

285
documentation/archive/llm-setup-local.md Normal file
View file

@ -0,0 +1,285 @@
# LLM Setup — lila pipeline
This document covers the LLM infrastructure for stage 3 (enrich) of the lila data pipeline. It documents the hardware constraints, supported providers, model recommendations, and how to configure and swap providers in the test and production scripts.
---
## Provider model
Each provider + model combination counts as one vote in the final majority. Running the same model twice is not supported — one model, one vote. To increase vote confidence, add more models rather than re-running existing ones.
---
## Hardware (dev machine)
| Component | Spec |
| --------- | --------------------------------------------------------------- |
| CPU | Intel Core i7-6500U (2 cores / 4 threads @ 3.10 GHz) |
| RAM | 8 GB |
| GPU | NVIDIA GeForce GTX 950M — 4 GB VRAM (Maxwell, CUDA compute 5.0) |
| OS | Debian GNU/Linux 13 (trixie) x86_64 |
**Local inference verdict:** viable for small/quantized models, not for production runs. See the [Local inference](#local-inference-llamacpp) section for details.
---
## Provider overview
The enrich script uses a single, swappable provider config. All providers except Anthropic expose an OpenAI-compatible API, so the same client code works across all of them — only `baseURL`, `apiKey`, and `model` change.
| Provider | Use case | Cost | Rate limits |
| ---------------------- | --------------------------------------------- | ------------------ | ---------------------- |
| llama.cpp (local) | Quality testing, overnight dev runs | Free (electricity) | None |
| OpenRouter (free tier) | Quality comparison, multi-model evaluation | Free | 50 req/day, 20 req/min |
| OpenRouter (paid) | Production runs if local quality insufficient | Pay-per-token | None |
| Anthropic API | Quality baseline / reference | Pay-per-token | Standard |
---
## Local inference (llama.cpp)
### Why local inference is worth testing
Time is not a constraint — the pipeline scripts are fully resumable. The laptop can run overnight for multiple nights. The only question is output quality, which the test script evaluates empirically.
### Hardware constraints
The GTX 950M has 4 GB VRAM and Maxwell architecture (CUDA compute 5.0). llama.cpp supports Maxwell via CUDA backend but newer builds may require the `--cuda-no-kv-offload` flag depending on the version.
llama.cpp splits model layers between GPU and CPU automatically via `--n-gpu-layers`. You set how many layers go on the GPU; the rest run on CPU/RAM. This means a model larger than VRAM is not a dead end — it runs in hybrid mode, slower than full-GPU but much faster than pure CPU.
Practical estimates for this hardware (~3.5 GB VRAM usable after drivers):
| Model size | Q4 VRAM | Mode | Est. speed |
| ---------- | ------- | ----------------------------- | ------------ |
| 3B | ~2.0 GB | Full GPU | ~15–20 tok/s |
| 4B | ~2.5 GB | Full GPU | ~12–18 tok/s |
| 7B | ~4.5 GB | Hybrid (~26/32 layers on GPU) | ~8–12 tok/s |
| 13B+ | ~8 GB+ | CPU-heavy hybrid | too slow |
### Recommended local models
Two candidates worth testing, covering different points on the size/quality tradeoff:
**Gemma 4 E4B Instruct (Q4 / UD-Q4_K_XL)**
- GGUF file: `gemma-4-E4B-it-UD-Q4_K_XL.gguf` (~2.5 GB)
- Source: https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF
- Runs fully on GPU. Brand new (April 2025), built for edge hardware, 140+ language support including all five pipeline languages. First candidate to test.
**Qwen2.5 7B Instruct (Q4_K_M)**
- GGUF file: `Qwen2.5-7B-Instruct-Q4_K_M.gguf` (~4.5 GB)
- Source: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF
- Runs in hybrid mode (~26 of 32 layers on GPU, rest on CPU), ~8–12 tok/s. Stronger multilingual generation than any 3–4B model. Second candidate, for comparison against the smaller Gemma 4 E4B.
### Installation
```bash
# Install build dependencies
sudo apt install build-essential cmake git
# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
# Build with CUDA support (GTX 950M — compute 5.0)
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=50
cmake --build build --config Release -j$(nproc)
# Download model (example — adjust path as needed)
mkdir -p models
wget -O models/qwen2.5-3b-instruct-q4_k_m.gguf \
https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf
```
### Starting the server
**Gemma 4 E4B** (full GPU):
```bash
./build/bin/llama-server \
--model models/gemma-4-e4b-it-ud-q4_k_xl.gguf \
--port 8080 \
--ctx-size 4096 \
--n-gpu-layers 999 \
--host 127.0.0.1
```
**Qwen2.5 7B** (hybrid — tune `--n-gpu-layers` to fit your VRAM):
```bash
./build/bin/llama-server \
--model models/qwen2.5-7b-instruct-q4_k_m.gguf \
--port 8080 \
--ctx-size 4096 \
--n-gpu-layers 28 \
--host 127.0.0.1
```
`--n-gpu-layers 999` means "put everything on GPU" — llama.cpp caps at the
actual layer count automatically, so 999 is safe as a "full offload" value.
For the 7B hybrid, start with `28` and reduce by 2 if the server reports
out-of-memory at startup.
### Verify the server is running
```bash
curl http://127.0.0.1:8080/health
# Expected: {"status":"ok"}
```
---
## OpenRouter (free tier)
OpenRouter exposes all models via an OpenAI-compatible API. No code changes
are needed to switch from local llama.cpp to OpenRouter — only the config
object changes.
### Rate limits (free tier)
- **50 requests per day** (account total, not per model)
- 20 requests per minute
> **Implication for testing:** with a 10-record test set you have headroom
> to test 4–5 models per day. With a 100-record test set, plan one model per
> day.
> **Implication for production:** the free tier is not viable for 117k
> records. If local quality is insufficient, use paid OpenRouter credits or
> a dedicated provider.
### Free models recommended for this pipeline
Ranked by expected multilingual generation quality for en/it/de/fr/es:
| Model ID | Params | Notes |
| ---------------------------------------- | --------------------- | ------------------------------------------------------------------------------------ |
| `qwen/qwen3-coder:free` | 480B MoE (35B active) | Best free option. Strong multilingual despite "coder" label. Use as quality ceiling. |
| `qwen/qwen3-next-80b-a3b-instruct:free` | 80B MoE (3B active) | Smaller Qwen, useful comparison point. |
| `nvidia/nemotron-3-super-120b-a12b:free` | 120B MoE (12B active) | 262K context, supports structured output. |
| `google/gemma-4-31b-it:free` | 31B | 140+ language support, good European language coverage. |
| `zhipuai/glm-4.5-air:free` | MoE | Multilingual-focused. |
**Skip for this pipeline:**
- Llama models — weaker European language generation than Qwen/Gemma
- Mistral free tier — requests may be used for model training
### API endpoint
```
https://openrouter.ai/api/v1/chat/completions
```
Set `Authorization: Bearer <OPENROUTER_API_KEY>` in the request headers.
---
## Provider configuration in the enrich script
The enrich script reads a single config object. To switch providers,
change this object and re-run. The `name` field is used as the model
identifier in `pipeline.db` — it must be unique across all runs.
```typescript
// config.ts
export type ProviderConfig = {
name: string; // used as model identifier in pipeline.db — must be unique
baseURL: string;
apiKey: string;
model: string;
maxTokens: number;
};
// Local llama.cpp
export const LOCAL_QWEN3B: ProviderConfig = {
name: "local-qwen2.5-3b",
baseURL: "http://127.0.0.1:8080/v1",
apiKey: "none", // llama.cpp ignores this
model: "qwen2.5-3b", // llama.cpp ignores model name, uses loaded model
maxTokens: 512,
};
// OpenRouter — Qwen3 480B (free)
export const OR_QWEN3_480B: ProviderConfig = {
name: "or-qwen3-480b",
baseURL: "https://openrouter.ai/api/v1",
apiKey: process.env.OPENROUTER_API_KEY!,
model: "qwen/qwen3-coder:free",
maxTokens: 512,
};
// OpenRouter — Gemma 4 31B (free)
export const OR_GEMMA4_31B: ProviderConfig = {
name: "or-gemma4-31b",
baseURL: "https://openrouter.ai/api/v1",
apiKey: process.env.OPENROUTER_API_KEY!,
model: "google/gemma-4-31b-it:free",
maxTokens: 512,
};
// Anthropic (reference baseline — different adapter required)
export const ANTHROPIC_SONNET: ProviderConfig = {
name: "anthropic-sonnet",
baseURL: "https://api.anthropic.com/v1", // adapter handles format difference
apiKey: process.env.ANTHROPIC_API_KEY!,
model: "claude-sonnet-4-6",
maxTokens: 512,
};
```
All output is written to `pipeline.db`. Each record is stored with the
model name as identifier so results from different providers can be
compared and compiled into votes.
---
## Evaluation metrics
The test script measures the following per provider run:
| Metric | What it measures |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| **JSON parse rate** | % of responses that are valid, schema-compliant JSON. Critical — a failed parse is a wasted call. Target: >97% |
| **Field coverage** | % of records where all required fields are present (cefr votes for all translations, descriptions for all languages, glosses/examples for fr/es) |
| **CEFR agreement** | For records that have a `cefr_source` vote, % where the model agrees. Measures calibration. |
| **Language correctness** | Manual spot-check only — automated detection not reliable enough |
| **Tokens/second** | Local only. Indicates overnight run feasibility |
### Decision thresholds
| Metric | Threshold | Action if below |
| --------------- | --------- | ---------------------------------------------- |
| JSON parse rate | < 97% | Do not use this model for production |
| Field coverage | < 95% | Prompt needs revision before production |
| CEFR agreement | < 70% | Model lacks vocabulary knowledge for this task |
---
## Recommended test sequence
1. **Start local, minimal dataset (5–10 records)**
Install llama.cpp, run Qwen2.5 3B against 5–10 hand-picked records.
Verify the server works, the output parses, and the model produces
something reasonable. This is purely a smoke test.
2. **Expand local to full 100-record sample**
Once the pipeline is confirmed working, run all 100 records locally.
Collect metrics. This is your local quality baseline.
3. **Run the same 100 records through OpenRouter free models**
One model per day (50 req/day limit). Start with `qwen/qwen3-coder:free`
as the quality ceiling.
4. **Compare metrics side by side**
If local 3B is within acceptable range of the cloud models on CEFR
agreement and field coverage, proceed with local overnight runs for
production. If not, use the cloud model that passed.
5. **Production run**
Full 117k records. Resume-safe — each record is written to `pipeline.db`
atomically as it is processed. Overnight runs can be stopped and
continued at any time without losing work.

181
documentation/archive/model-strategy-cefr-voters.md Normal file
View file

@ -0,0 +1,181 @@
# Model Strategy
## The problem
The pipeline requires LLMs to perform four tasks per vocabulary entry:
1. **Gloss review** — confirm or improve the existing gloss
2. **Example review** — confirm or improve existing examples
3. **Translation validation** — confirm valid translations, reject bad data, generate missing ones
4. **CEFR assignment** — assign A1-C2 to the headword and each translation
The core challenge is that vocabulary entries have **multiple senses**. The word "cat" appears five times in the database — as an animal, as slang for "guy", as a nautical term, as a verb meaning "to vomit", and as a verb meaning "to hoist an anchor". Each sense requires a different CEFR level and different translations. A model that only knows "cat" is A1 gets four out of five wrong.
This makes CEFR assignment fundamentally a **sense-disambiguation problem**, not just a vocabulary lookup. Specialized CEFR classifiers (like `cefrpy` or `dksysd/cefr-classifier`) operate at the word or sentence level and cannot distinguish between senses of the same word. General LLMs handle sense disambiguation well but introduce quality and reliability problems that depend heavily on model size.
The secondary challenge is **hardware constraints**. The available local hardware (GTX 950M, 4GB VRAM) can only run models up to approximately 4B parameters fully in GPU memory. Larger models run in hybrid CPU/GPU mode which is significantly slower. Free cloud API tiers are generous enough for the sample dataset but have daily limits that make processing 100k+ entries across multiple sub-stages a multi-day or multi-week operation.
## What we tried and why it failed or worked
### Single-prompt design (abandoned)
The first enrich script sent one large prompt per entry covering all four tasks at once — CEFR voting, gloss improvement, example improvement, translation validation, and missing translation generation. This produced the following problems:
- The model skipped translations it considered invalid rather than explicitly rejecting them, causing validation failures
- Bad data in the translation table (`it:free`, `de:-frei`, `es:de fai`) caused consistent validation failures because the model refused to vote on them even when explicitly instructed
- The combined prompt was large enough to trigger reasoning mode on Gemma 4 E4B, consuming all available tokens on thinking before producing output
- 20% of entries required manual review
### Sub-stage design (current)
Splitting into four ordered sub-stages fixed the reasoning and validation problems:
1. `round1_gloss` — LLM reviews the gloss in isolation
2. `round1_example` — LLM reviews examples with verified gloss as context
3. `round1_translations` — LLM validates translations with verified gloss as context
4. `round1_cefr` — LLM assigns CEFR levels only to validated translations
This ordering ensures the CEFR sub-stage never sees bad data. The smaller, focused prompts eliminated reasoning mode triggering and reduced per-entry time from ~120 seconds to ~25 seconds.
### Gloss quality (ongoing)
Testing on 50 entries with Qwen3.5-4B showed ~80% good quality. The 20% failures fall into three categories:
- **Category header glosses** — Kaikki occasionally uses "Terms relating to people." or "Terms relating to things." as a gloss instead of a real definition. No model handles these correctly because there is no real meaning to improve.
- **Rare/obscure senses** — slang, archaic, and theological senses that a 4B model does not have enough knowledge to handle (e.g. "cat" meaning "to vomit", "word" meaning "Logos, Christ").
- **Short ambiguous glosses** — one or two word glosses with no example context cause hallucination.
### Gemma 4 E4B (rejected)
Gemma 4 E4B is a hybrid reasoning model. Disabling thinking via `--reasoning-budget 0` or `--chat-template-kwargs '{"enable_thinking":false}'` does not work reliably in llama.cpp for the E4B variant — the model either puts reasoning into the content field as plain text or returns empty content with reasoning in `reasoning_content`. Per-entry time exceeded 100 seconds making it impractical.
### Qwen3.5-4B (current local model)
Non-thinking by default for the small series. Runs fully in 4GB VRAM at ~5 seconds per sub-stage. Acceptable quality for common vocabulary (A1-B2) but struggles with rare and specialized senses. Used as the primary local voter.
### Specialized CEFR classifiers (rejected for primary use)
HuggingFace hosts several CEFR text classifiers (`dksysd/cefr-classifier`, `AbdulSami/bert-base-cased-cefr`) and the `cefrpy` Python library maps individual words to CEFR levels. These operate at the word or sentence level and cannot distinguish between senses. "cat" would always be assigned A1 regardless of whether the sense is the animal or obscure nautical slang. Useful only as a sanity check signal, not as a primary voter.
## Available free resources
| Resource | Type | Requests/day | Quality | Notes |
| ---------------------------- | ------------------ | ----------------- | --------- | ---------------------------------------------------------------------- |
| Local Qwen3.5-4B Q4_K_M | Local model | Unlimited | Decent | Non-thinking by default, fits in 4GB VRAM, ~5s per sub-stage |
| Local Qwen3.5-9B Q4_K_M | Local model | Unlimited | Good | Hybrid CPU/GPU mode on 4GB VRAM, slower but better quality |
| Local Llama 3.1 8B Q4_K_M | Local model | Unlimited | Decent | ~4.3GB, fits in VRAM or light hybrid, different architecture from Qwen |
| Groq — Llama 3.3 70B | Cloud API | 1,000 | Excellent | Best free quality available, 5-10x with batching |
| Groq — Llama 3.1 8B | Cloud API | 14,400 | Decent | High volume, similar quality to local 4B |
| Google Gemini AI Studio | Cloud API | 1,500 | Very good | Google account required, 5-10x with batching |
| OpenRouter free rotation | Cloud API | 50–1,000 | Varies | Rotates between free models automatically via `openrouter/free` |
| Wiktionary API | Context enrichment | Unlimited | N/A | Structured vocabulary data, directly related to Kaikki source |
| `cefrpy` Python library | Word lookup | Unlimited | Limited | Deterministic English word CEFR lookup, no sense disambiguation |
| HuggingFace CEFR classifiers | Text classifier | Unlimited (local) | Limited | Sentence-level difficulty, not sense-aware |
### Batching
All cloud APIs support sending multiple entries in a single request. Sending 5 entries per request multiplies effective daily capacity by 5x:
- Groq Llama 3.3 70B: 1,000 requests → ~5,000 entries/day
- Gemini: 1,500 requests → ~7,500 entries/day
### Multiple accounts
Prohibited by the terms of service of all providers listed above.
## Final approach per sub-stage
The pipeline runs multiple models as independent voters. Each model processes every entry once and writes its votes to `pipeline.db`. The merge stage resolves disagreements by majority vote. A tiebreaker runs additional models on flagged entries where no majority was reached.
### round1_gloss and round1_example
These sub-stages require a model that understands sense context from examples. Specialized classifiers cannot help here — only general LLMs can evaluate whether a gloss correctly describes a specific sense.
**Primary voter:** Local Qwen3.5-9B Q4_K_M — runs overnight, unlimited, handles common vocabulary well.
**Secondary voter:** Groq Llama 3.3 70B with 5-entry batching — higher quality, catches errors the local model makes on rare or specialized senses.
**Tertiary voter:** Gemini AI Studio with 5-entry batching — third independent opinion, different training data from both Groq and local model.
**Context enrichment via Wiktionary API:** Before calling any model for the gloss or example sub-stage, the pipeline queries the Wiktionary API for the headword. The API returns the full Wiktionary entry including all senses, usage notes, and examples. This structured data is added to the prompt as additional context, giving the model a much clearer picture of which specific sense it is working with.
This directly fixes the two hardest failure cases:
- **Category header glosses** ("Terms relating to people.") — the Wiktionary entry contains the real definition which the model can use to generate a proper gloss
- **Short ambiguous glosses** — the additional sense context prevents the model from guessing the wrong meaning
The Wiktionary API is free, has no rate limits for reasonable use, and is directly related to the Kaikki data source since Kaikki extracts from Wiktionary.
### round1_translations
Same voter stack as gloss/example. The few-shot examples in the prompt (showing that `it:free` → reject and `de:-frei` → reject) handle the bad data cases that caused validation failures in the single-prompt design.
### round1_cefr
This sub-stage only receives translations that survived the validation step. All bad data is already excluded.
**Primary voter:** Local Qwen3.5-9B Q4_K_M.
**Secondary voter:** Groq Llama 3.3 70B with 5-entry batching.
**Tertiary voter:** Gemini AI Studio with 5-entry batching.
**Sanity check:** `cefrpy` provides a deterministic English word CEFR level as a reference signal. If the majority LLM vote disagrees significantly (e.g. LLMs vote C2 for "cat" the animal), the entry is flagged for human review. `cefrpy` does not vote — it only triggers review flags.
### Voter summary
| Sub-stage | Voter 1 | Voter 2 | Voter 3 |
| ------------------- | ------------------ | ------------------ | ------- |
| round1_gloss | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
| round1_example | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
| round1_translations | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
| round1_cefr | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
Three voters means a correct majority requires at least two models to agree. Even if the local model gets a difficult sense wrong, the two cloud models will likely agree on the correct answer and outvote it.
## Open questions
### Wiktionary API context extraction
The Wiktionary API returns the full entry for a word including all senses. For a word like "free" with 8+ senses, dumping the entire entry into the prompt wastes tokens and may confuse the model. The open question is how to extract only the relevant sense — options include matching by sense_index, fuzzy-matching the Kaikki gloss against Wiktionary glosses, or letting the model see all senses and identify the correct one itself.
### Batching prompt design
Batching 5-10 entries per API call multiplies effective daily capacity significantly. The prompt and validation logic for batched requests is more complex — the model must return a structured JSON object keyed by entry ID, and partial failures (one entry in a batch fails validation) need careful handling. Not yet designed or tested.
### Groq and Gemini API integration
Neither Groq nor Gemini is integrated into the pipeline yet. Both use OpenAI-compatible APIs so integration is straightforward — add provider configs to `stage-3-enrich/config.ts` and set API keys in `.env`. The batching prompt design needs to be finalised first.
### OpenRouter free model rotation
OpenRouter's `openrouter/free` router selects a model at random from available free models. This means output style and quality vary between requests, which complicates round 2 voting where models review each other's candidates. May need to pin specific free models rather than using the router.
### Qwen3.5-9B performance on hard cases
The 9B model has not yet been tested. It is expected to handle rare and specialized senses better than the 4B model but this has not been verified. Needs a test run against the same 50 entries used to evaluate the 4B model.
### Llama.cpp Gemma 4 bug
The llama.cpp chat template bug preventing reliable JSON output from Gemma 4 E4B may be fixed in a future release. The model fits in 4GB VRAM and would be a useful additional local voter if the bug is resolved. Worth checking periodically.
### Full dataset scale
The current pipeline runs on a 500-entry sample per language. The full Kaikki English file contains approximately 1.3 million entries, of which a fraction will pass the POS and translation filters. The exact count and the time required to run all sub-stages across all models at full scale is not yet known.
### Category header glosses
Kaikki occasionally uses category headers ("Terms relating to people.", "Terms relating to things.") as glosses. These are not real definitions and no model produces useful output for them. Options include pre-filtering them before the gloss sub-stage and generating a gloss purely from examples, or flagging them as a special case for human review.
wget -O models/llama-3.1-8b-instruct-q4_k_m.gguf \
"https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
# Q4_K_M (5.68GB — hybrid mode, better quality)
wget -O models/qwen3.5-9b-q4_k_m.gguf \
"https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf"
# Q3_K_S (4.32GB — might fit fully in VRAM)
wget -O models/qwen3.5-9b-q3_k_s.gguf \
"https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q3_K_S.gguf"