lila/documentation/DATA_PIPELINE.md

Lila Data Pipeline — Technical Documentation

Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer. Last updated: 2026-07-06

---

Table of Contents

1. Executive Summary
2. Problem & Context
3. Architecture Overview
4. Current Implementation
5. The LLM Layer
   • 5.1 Local Model Evaluation
   • 5.2 Online API Options
   • 5.3 Model Selection Criteria
6. The Gender Problem & Kaikki Integration
7. Batching Strategy
8. Hardware Constraints
9. Testing & Quality Assurance
10. Interactive CLI
11. Future Extensions & Roadmap
12. Decisions Log
13. Known Issues & Dev Notes
14. How to Run
15. Roadmap

---

Quick Reference

What	Where
Entry point	pipeline.ts
LLM config	config/llm.ts
System prompt	config/prompt.ts — buildSystemPrompt()
Batch config	config/batch.ts
Shared constants	config/constants.ts
Output schema	utils/merge-enriched-data.ts
LLM adapters	utils/llm-adapters/
Current model	qwen2.5-1.5b-instruct-q4_k_m.gguf
Target scale	100,000+ words

---

1. Executive Summary

The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:

• One or more senses (definitions)
• A natural example sentence per sense
• A CEFR-based difficulty level (easy / medium / hard)
• Translations into all target languages except the source, each with grammatical gender

The pipeline is designed to scale to 100,000+ words across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an interactive CLI for provider selection.

Key Design Principles

Principle	Rationale
Quality first	Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary.
Local-first, cloud-fallback	Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed.
Resumable & idempotent	Each word writes to its own JSON file. The pipeline skips already-processed words on restart.
Configurable batching	Batch size is a single config value (config/batch.ts). The pipeline adapts without code changes.
Provider-agnostic	LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface.
Honest metrics	Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns.

Open Question: Gender Accuracy

Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that Qwen2.5-1.5B systematically defaults to neuter for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a model size issue (fixable by moving to 3B+) or a training data issue (requiring an external lookup) is unresolved.

Options under evaluation:

• Larger local models (Qwen2.5-3B, Qwen3.5-1.7B)
• Online models with stronger multilingual training (Gemini, DeepSeek)
• Post-processing lookup via Kaikki Wiktionary dumps as a fallback or replacement

No decision made. Gender handling will be determined by the 20-word quality torture suite.

Current Status (2026-07-06)

• Core pipeline: scanning, enrichment, merging, verification, writing
• Local LLM integration via llama.cpp server (OpenAI-compatible API)
• Cloud provider adapters: Gemini, DeepSeek, OpenRouter via utils/llm-adapters/
• Batching with retry/split: configurable batch size, exponential split-on-failure (4 → 2 → 1)
• Honest timing: unified throughput for all providers, detailed breakdown only for local
• Auto-target languages: prompt dynamically excludes source language from targets
• Schema validation: validates LLM response structure before file writes
• Interactive CLI: planned — provider/model/batch selection with saved config
• In progress: Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
• Pending: 20-word quality torture suite (will decide gender approach)
• Pending: Online API evaluation (Gemini free tier, DeepSeek, Groq)

One-Line Architecture

source wordlists -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json
                                    |
                         [gender: LLM-generated, accuracy TBD]

Files at a Glance

File	Purpose
pipeline.ts	Orchestrator. Scans sources, loops words, coordinates all stages
config/llm.ts	Provider selection, API URL, model name
config/prompt.ts	buildSystemPrompt() — dynamic prompt with auto-target languages
config/batch.ts	Batch size and max retry count
config/constants.ts	Shared LANG_MAP, POS_MAP, ALL_LANGUAGES
utils/enrich-word.ts	Calls LLM via adapter, parses response, builds EnrichedSense[], retry/split logic
utils/merge-enriched-data.ts	Merges skeleton + enriched senses into final JSON
utils/verify-enriched-file.ts	Schema validation (required fields, types, gender enum)
utils/check-llm-server.ts	Health check for local server; skipped for cloud providers
utils/scanning-source-files.ts	Discovers wordlists from source-data/ directory
utils/create-base-json.ts	Writes skeleton {word, language, pos} files
utils/write-json-file.ts	Atomic .tmp → rename writes
utils/check-if-json-exists.ts	Resumability: checks if word already has enriched senses
utils/create-line-reader.ts	Streaming line reader for large wordlists
utils/create-output-dirs.ts	Creates worddata/{language}/{pos}/ folders
utils/delete-file.ts	Cleanup helper for failed batches
utils/get-word-file-path.ts	Path construction helper
utils/progress-tracker.ts	[current/total] formatting for console output
utils/pipeline-timer.ts	Timing + token metrics; unified throughput for all providers
utils/llm-adapters/factory.ts	Creates the right adapter based on LLM_CONFIG.provider
utils/llm-adapters/types.ts	LlmAdapter interface
utils/llm-adapters/openai-compatible.ts	Local llama.cpp, OpenRouter, DeepSeek
utils/llm-adapters/gemini.ts	Google Gemini native API

Scale Target

Metric	Target
Words	100,000+
Languages	English (source), German, Italian, Spanish, French (targets)
Parts of speech	Nouns, verbs, adjectives, adverbs
Output	One .json file per word, ~2-5KB each

---

2. Problem & Context

Why Build This?

Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:

What Exists	What is Missing	Why It Matters
Word + translation	Student-friendly definition	Learners need explanations, not just equivalents
Static difficulty labels	CEFR-calibrated difficulty	"Easy" vs "hard" must map to learner proficiency levels
Isolated translations	Natural example sentences	Context is how vocabulary is actually acquired
Raw gender data	Gender integrated with translation	Flashcards must show gender immediately, not as a separate lookup
Bulk word lists	Per-word structured JSON	The trainer consumes one file per word for fast random access

The Target User

A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender — all calibrated to their CEFR level (A1-C2).

Why Not Use Existing Dictionaries?

• Wiktionary: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
• Kaikki (Wiktionary dump): Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
• Google Translate / DeepL: No definitions, no examples, no difficulty, no structured output
• Existing language learning apps: Closed data, no export, no control over content

The LLM fills the gap: it generates pedagogical content (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.

Language Direction

The pipeline is direction-agnostic. A wordlist is defined by:

• Source language: the language of the input words
• Target languages: all other languages in the system (auto-derived from ALL_LANGUAGES minus source)

Current focus: English -> German/Italian/Spanish/French

Planned directions include German -> French, Italian -> Spanish, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist. Target languages are computed automatically.

Why 100,000+ Words?

• Coverage: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation.
• Languages: English (source) -> German, Italian, Spanish, French (targets).
• Parts of speech: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.).

The Data Flow

Source files          LLM enrichment          Final JSON
(one word per line)   (definitions,           (one per word,
                      examples,               self-contained)
english/nouns         difficulty,
english/verbs         translations)            time.json
italian/nouns                                year.json
...                                          people.json

The Quality Challenge

Generating 100,000 entries with an LLM introduces risks:

Risk	Mitigation
Hallucinated definitions	Low temperature (0.1), strict system prompt, schema validation
Incorrect grammatical gender	Under evaluation: larger models or external lookup
Inconsistent difficulty levels	Explicit CEFR mapping in prompt, spot-checking
JSON parse failures	Retry + split logic, schema validation, cleanup on failure
Model drift (online APIs)	Version pinning, local fallback
Provider downtime	Adapter pattern allows hot-swapping providers
Malformed LLM responses	validateSense() catches bad data before file writes

Why TypeScript + Node?

• Familiarity: Existing project uses TypeScript (frontend in TanStack Router + React)
• Ecosystem: readline for streaming files, fs for JSON I/O, native fetch for HTTP
• Portability: Runs on the same Debian laptop as the llama.cpp server
• No build complexity: tsx for direct execution, no bundler needed

Why llama.cpp?

• GGUF format: Single-file models, easy to swap, quantize, and version
• OpenAI-compatible API: /v1/chat/completions means the same adapter code works for local and online models
• No dependencies: Self-contained binary, runs on old hardware (tested on GTX 950M)
• Privacy: Local inference means no data leaves the machine

---

3. Architecture Overview

Pipeline Flow

Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
    -> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
    -> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics

Resumability

• Skip existing: check-if-json-exists.ts checks if {word}.json exists with non-empty senses
• Atomic writes: .tmp -> rename in write-json-file.ts, no partial files on crash
• Cleanup on failure: Deletes partially-written files for failed batches, continues to next batch

Directory Structure

data-pipeline/
|-- pipeline.ts              # Entry point / orchestrator
|-- config/
|   |-- llm.ts               # Provider, API URL, model name
|   |-- prompt.ts            # buildSystemPrompt() — dynamic prompt
|   |-- batch.ts             # Batch size and retry config
|   |-- constants.ts         # LANG_MAP, POS_MAP, ALL_LANGUAGES
|-- utils/
|   |-- enrich-word.ts          # LLM call, parse, retry/split
|   |-- merge-enriched-data.ts  # Schema types + merge logic
|   |-- verify-enriched-file.ts # Schema validation
|   |-- check-llm-server.ts     # Health check (local only)
|   |-- scanning-source-files.ts # Source discovery
|   |-- create-base-json.ts     # Skeleton writer
|   |-- write-json-file.ts      # Atomic JSON writer
|   |-- check-if-json-exists.ts # Resumability check
|   |-- create-line-reader.ts   # Streaming file reader
|   |-- create-output-dirs.ts   # Directory creation
|   |-- delete-file.ts          # Cleanup helper
|   |-- get-word-file-path.ts   # Path helper
|   |-- progress-tracker.ts     # Console progress formatting
|   |-- pipeline-timer.ts       # Timing + token metrics
|   |-- llm-adapters/
|       |-- factory.ts          # Adapter selection
|       |-- types.ts            # LlmAdapter interface
|       |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
|       |-- gemini.ts           # Google Gemini
|-- source-data/
|   |-- {language}/
|       |-- {pos}               # One word per line, no extension
|-- worddata/
    |-- {language}/
        |-- {pos}/
            |-- {word}.json     # One self-contained file per word
|-- kaikki-source-files/        # Wiktionary dumps for gender lookup (planned)

Output Schema

Each .json file contains: word, language, pos, senses[] (each with id, sense, example, difficulty_level, translations per target language), enrichedAt, model.

Full TypeScript interfaces: utils/merge-enriched-data.ts.

Error Handling

Failure	Behavior
LLM server offline	Hard fail at startup (check-llm-server.ts, local only)
LLM returns bad JSON	Retry up to 3 times, then split batch. Log and continue
LLM returns malformed senses	validateSense() catches it before file write
Schema validation fails	Log warnings, keep file
Individual batch fails	Does not stop pipeline; cleans up partial files
Individual word fails (size 1)	Log and continue to next word

Metrics

Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See utils/pipeline-timer.ts.

Unified throughput (all providers): total tokens / total request time Detailed breakdown (local only): prompt speed vs completion speed

---

4. Current Implementation

Tech Stack

Layer	Choice	Why
Runtime	Node.js + tsx	Direct TypeScript execution, no build step
HTTP client	Native fetch	Works for local llama.cpp and online APIs
File I/O	fs + readline	Streaming line reader for large wordlists
JSON	Native JSON.parse/stringify	Simple, no schema library needed

Configuration

File	Purpose
config/llm.ts	provider (local	openrouter	deepseek	gemini), url, model
config/prompt.ts	buildSystemPrompt(sourceLanguage, pos, targetLanguages)
config/batch.ts	BATCH_CONFIG.size (words per call), maxRetries
config/constants.ts	LANG_MAP, POS_MAP, ALL_LANGUAGES

Key Modules

File	Responsibility
utils/enrich-word.ts	Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds EnrichedSense[], retry/split logic
utils/merge-enriched-data.ts	Merges skeleton with LLM senses, adds enrichedAt and model
utils/verify-enriched-file.ts	Schema validation: required fields, array lengths, gender enum, translation structure
utils/pipeline-timer.ts	Tracks per-word and global metrics; unified throughput for all providers
utils/llm-adapters/factory.ts	Creates adapter based on LLM_CONFIG.provider
utils/llm-adapters/openai-compatible.ts	OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek
utils/llm-adapters/gemini.ts	Google Gemini generateContent API with systemInstruction

Current Model

Property	Value
Model	qwen2.5-1.5b-instruct-q4_k_m.gguf
Size	~1.0GB
Quantization	Q4_K_M
Server	llama.cpp (llama-server)
API	OpenAI-compatible /v1/chat/completions

llama-server Flags: History & Rationale

The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen.

Hardware Constraints

Component	Spec	Implication
CPU	i7-6500U (2 physical cores, 4 threads HT)	-t 2 matches physical cores; HT hurts more than helps
GPU	GTX 950M (Maxwell, 2015)	32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved
RAM	8GB (3.95GB usable)	--mlock pins model in RAM; system must not swap

Flag Evolution

Flag	Value Tried	Result	Why
-m	qwen3.5-4b-q4_k_m.gguf	Works, ~47s/word	Baseline. Correct genders. 2.6GB file, tight on VRAM.
-m	Ministral-3b-instruct.Q4_K_M.gguf	Broken	Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See Known Issues.
-m	qwen2.5-1.5b-instruct-q4_k_m.gguf	Works, ~8s/word	Current. Fast but gender accuracy degraded.
-ngl	999	Keeps	Offload all layers to GPU. Required for any speed.
-c	4096	Wasteful	4K context for 300-token dictionary entries wastes ~400MB VRAM.
-c	2048	Current	Sufficient headroom. Frees VRAM for KV cache.
-b / -ub	512	Keeps	Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs.
-b / -ub	2048	Slower on 950M	Tested briefly. No improvement, possibly worse due to memory pressure.
-t	4	Slower	Hyperthreading cores hurt llama.cpp performance.
-t	2	Current	Matches 2 physical cores.
--threads-batch	(default)	Risky	Defaults to same as -t, but explicit is safer.
--threads-batch	2	Current	Explicit match to -t.
--flash-attn	(omitted)	Correct	On Maxwell (compute 5.0), Flash Attention adds overhead. Not used.
--flash-attn	(tested)	No gain	Briefly tried with Qwen3.5-4B. No speedup, possibly regression.
--mlock	Keeps	Pins model weights in RAM. Prevents OS swapping on memory pressure.
--prio	2	Keeps	Raises process priority. Marginal on this hardware, harmless.
--reasoning	off	(Qwen3.5 only)	Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5.

Current Command

./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2

What Was Not Tried (And Why)

Flag	Reason Skipped
-fa / --flash-attn	Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs.
--no-mmap	--mlock achieves the same (pin in RAM) without the I/O overhead of disabling mmap.
-np / --parallel	Not needed. Single sequential pipeline, no concurrent requests.
--cont-batching	Default in recent llama.cpp. No need to toggle.
--defrag-thold	KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx.
-ot / --override-tensor	Expert-level. No tensor-specific issues observed.

Future Flag Experiments

Experiment	Expected Outcome
-c 1024	Further VRAM savings. Risk: insufficient for batching larger prompt sizes.
-b 256 -ub 256	Test if smaller batches reduce latency on bandwidth-starved Maxwell.
--metrics	Enable Prometheus endpoint for automated performance tracking.

Performance Baseline

Metric	Qwen3.5-4B	Qwen2.5-1.5B
Time/word	~47s	~8s
Completion tok/s	~6.4	~18.8
Prompt tok/s	~81	~549
Avg completion tokens	~274	~132
Avg prompt tokens	~327	~308

Known Limitations (Current)

• Gender accuracy: Qwen2.5-1.5B systematically defaults to neuter for Romance languages. Under evaluation whether larger models fix this.
• Single POS: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments.
• Pre-scanning wordlists: Entire file read into memory before processing. Inefficient for 100k words. See Refactor Notes.

---

5. The LLM Layer

5.1 Local Model Evaluation

Models are evaluated on three criteria in order of priority: quality (definitions, examples, translations, gender accuracy), speed (completion tokens/sec), VRAM fit (must run on GTX 950M 4GB).

Model	Size	VRAM	Speed	Quality	Status
Qwen3.5-4B Q4_K_M	2.6GB	Tight	~6.4 tok/s	Baseline (assumed good)	Baseline - too slow
Qwen2.5-1.5B Q4_K_M	1.0GB	Easy	~18.8 tok/s	Gender systematically wrong	Current - fast, needs validation
Ministral-3B Q4_K_M	1.9GB	Fits	-	Broken (tokenizer)	Abandoned
Qwen2.5-3B Q4_K_M	1.9GB	Fits	~12-15 tok/s (est.)	Unknown	Pending download
Gemma 4 E2B Q4_K_M	3.46GB	No	-	-	Too large for 4GB VRAM
Gemma 4 E2B IQ2_M	2.62GB	Fits	~8-12 tok/s (est.)	"Low quality" per Google	Not worth it

Qwen2.5-1.5B Test Results (3 words)

Word	Definition	Example	Gender (de/it/es/fr)	Verdict
time	"A period of duration..."	"The meeting was scheduled..."	neuter/neuter/neuter/neuter	All wrong
year	"A period of time consisting of 365 days..."	"The year 2023 is a leap year."	neuter/neuter/neuter/neuter	All wrong
people	"Individuals who are part of a group."	"The people gathered at the park."	neuter/feminine/feminine/feminine	Mixed

Pattern: Defaults to neuter when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident.

Pending Tests

• Qwen2.5-3B: Same architecture, 2x params. If gender fixes, it was a size problem.
• 20-word torture suite: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models.

5.2 Online API Options

Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API (except Gemini, which has a native adapter).

Provider	Model	Input $/1M	Output $/1M	Free Tier	Rate Limit	Est. Cost (100k words)	Est. Time
DeepSeek	V4 Flash	$0.14	$0.28	5M tokens	None	$0-15	~1-2 days
Gemini	2.5 Flash-Lite	$0.10	$0.40	1,500 req/day	1M TPM	$0 (free tier)	~1.5 days (batched)
Qwen/Alibaba	Qwen-Turbo	$0.05	$0.20	Unknown	600 RPM	~$11	~1-2 days
Groq	Llama 3.1 8B Instant	$0.05	$0.08	Yes	High	~$7	~3-4 hours
OpenRouter free	Various	$0	$0	200 req/day	20 RPM	$0	~10 days
OpenRouter paid	DeepSeek V4 Flash	$0.14	$0.28	-	Same as provider	~$16 (+5.5% fee)	~2-3 days

Notes:

• Costs assume ~550 tokens/word (300 prompt + 250 completion).
• Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day.
• Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable.
• DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million.

5.3 Model Selection Criteria

Decision flow for 100,000 words:

Start
  |
  v
Run 20-word torture suite
on Qwen2.5-3B (local)
  |
  |-- Quality good? -----> Use Qwen2.5-3B locally
  |   (gender correct)       ~20 days, $0
  |
  |-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free)
                              |
                              |-- Quality good? --> Batch 50, free tier
                              |                      ~1.5 days, $0
                              |
                              |-- Quality meh? ---> Test Groq or DeepSeek paid
                                                        |
                                                        |-- Speed priority? --> Groq
                                                        |                       ~$7, 3-4 hours
                                                        |
                                                        |-- Cost priority? ---> DeepSeek
                                                                                  ~$15, 1-2 days

Quality gates:

• = 90% gender accuracy (de/it/es/fr)

• 100% JSON parse rate
• No hallucinated definitions on polysemous words
• Natural, contextually appropriate example sentences
• Sensible difficulty classification (CEFR mapping)

---

6. The Gender Problem & Kaikki Integration

The Problem

Grammatical gender is embedded in the translations object of each sense:

"translations": {
  "de": [{"word": "Haus", "gender": "neuter"}],
  "it": [{"word": "casa", "gender": "feminine"}],
  "es": [{"word": "casa", "gender": "feminine"}],
  "fr": [{"word": "maison", "gender": "feminine"}]
}

Early testing with Qwen2.5-1.5B showed systematic failure: the model defaults to neuter for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all — only masculine and feminine.

Whether this is a model size issue (1.5B too small to retain gender facts) or a training data gap (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation.

Two Approaches Under Consideration

Approach	How It Works	Pros	Cons
A. LLM-only	Trust the model to generate correct gender	Zero additional complexity	Unreliable at small sizes; hallucination risk
B. LLM + Kaikki lookup	LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump	100% deterministic; decouples gender from model quality	Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries

Kaikki Data

Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender.

Language	Kaikki File	Coverage
German	kaikki.org-dictionary-German.jsonl	High
Italian	kaikki.org-dictionary-Italian.jsonl	High
Spanish	kaikki.org-dictionary-Spanish.jsonl	High
French	kaikki.org-dictionary-French.jsonl	High

Lookup logic: match on word (the translated string) -> extract gender field -> map to "masculine" | "feminine" | "neuter" | null.

Decision Pending

• If Qwen2.5-3B or an online model produces >=90% accurate gender: Approach A, no Kaikki needed.
• If all tested models fail gender: Approach B, implement Kaikki lookup as a post-processing step after LLM enrichment.

No implementation work started until the 20-word torture suite resolves this.

---

7. Batching Strategy

Why Batching is Necessary

At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.

Configurable Batch Size

Single config point controls batch size everywhere:

// config/batch.ts
export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const;

Change size to 1, 2, 5, 10, 20, etc. The pipeline adapts without code changes.

Prompt Structure

Single word:

["house"]

Batch of 4:

["house", "car", "tree", "water"]

LLM returns a JSON object with word keys:

{
  "house": [ { "sense": "...", "example": "...", ... } ],
  "car": [ { ... } ],
  "tree": [ { ... } ],
  "water": [ { ... } ]
}

Retry & Split Strategy

If a batch fails (bad JSON, missing key, etc.):

Batch of 4 fails (3 retries exhausted)
    |
    v
Split into 2 batches of 2
    |
    v
If a batch of 2 fails (3 retries), split into 2 batches of 1
    |
    v
If a single word fails (3 retries), log and skip

This gives resilience without losing the speed benefit of large batches. The maxRetries config controls how many attempts are made before splitting.

Expected Impact by Environment

Environment	Batch Size	Expected Speedup	Notes
Local GTX 950M	3	~1.05-1.15x	Memory bandwidth limited. KV cache pressure on 4GB VRAM.
Local GTX 950M	5	~1.10-1.20x	Sweet spot for this hardware.
Local GTX 950M	15	Risky	May OOM. Test carefully.
Local GTX 950M	50	Unlikely	VRAM insufficient.
Cloud API (Gemini)	50	5x fewer calls	Unlocks free tier viability.
Cloud API (Groq)	50	5x fewer calls	Minimal gain - already fast.

Batching is Non-Negotiable

Regardless of local vs cloud, batching is required for 100k words:

• Local: Better GPU utilization, amortized prompt processing
• Cloud: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts

---

8. Hardware Constraints

Current Machine

Component	Spec
OS	Debian GNU/Linux 13 (trixie) x86_64
CPU	Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz
GPU	NVIDIA GeForce GTX 950M (Maxwell, 2015)
GPU VRAM	4GB
RAM	8GB (3.95GB usable at idle)
Disk	102GB ext4 (~63GB used)

What Fits in 4GB VRAM

Model	File Size	KV Cache (2048 ctx)	Total VRAM	Fits?
Qwen2.5-1.5B Q4_K_M	~1.0GB	~0.5GB	~1.5GB	Yes
Qwen2.5-3B Q4_K_M	~1.9GB	~0.8GB	~2.7GB	Yes
Ministral-3B Q4_K_M	~1.9GB	~0.8GB	~2.7GB	Yes (but broken)
Qwen3.5-4B Q4_K_M	2.6GB	~1.0GB	~3.6GB	Tight
Gemma 4 E2B Q4_K_M	3.46GB	~1.2GB	~4.7GB	No
Gemma 4 E2B IQ2_M	2.62GB	~1.0GB	~3.6GB	Maybe, low quality

GPU Rental Alternatives

If local hardware becomes the bottleneck:

Provider	GPU	VRAM	Price/Hour	Time for 100k Words	Total Cost
Vast.ai	RTX 4090	24GB	~$0.30-0.60	~6-8 hours	~$2-5
RunPod	RTX 4090	24GB	~$0.50-0.80	~6-8 hours	~$4-6
Vast.ai	RTX 3090	24GB	~$0.20-0.40	~8-10 hours	~$2-4

With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day.

---

9. Testing & Quality Assurance

20-Word Torture Suite

Planned test set covering edge cases:

Category	Words	Why
Easy concrete	house, water, book	Baseline
Easy abstract	time, love, hope	Abstract nouns harder to define
Polysemous	bank, run, light	Multiple senses test disambiguation
Hard/technical	democracy, photosynthesis, entropy	Complex definitions
False friends	actual (en/es), sensible (en/fr), fabric (en/de)	Cross-lingual traps

Evaluation Criteria

For each word and each candidate model:

Criterion	Pass Threshold
Definition accuracy	Factually correct, max 15 words, student-friendly
Example quality	Natural sentence, word used correctly in context
Translation correctness	Correct word sense match
Gender accuracy (de/it/es/fr)	>=90% correct
Difficulty classification	Sensible per CEFR mapping
JSON reliability	100% parse rate, valid schema

Verification

verify-enriched-file.ts checks:

• Required top-level fields: word, language, pos, senses
• Each sense: sense (string), example (string), difficulty_level in {easy, medium, hard}
• Each translation: word (string), gender in {masculine, feminine, neuter, null}

Additionally, validateSense() in enrich-word.ts catches malformed senses before file writes, triggering retry/split instead of writing bad data.

---

10. Interactive CLI

Overview

The pipeline includes an interactive CLI that asks the user to select provider, model, and batch size on each run. No editing of config/llm.ts required.

Flow

$ npx tsx pipeline.ts

🌐 Lila Data Pipeline
─────────────────────

[1] Use last config: online → gemini → gemini-2.5-flash-lite → batch 50
[2] Configure new run

> 2

Provider type:
  [1] Local (llama.cpp)
  [2] Online API

> 2

Online provider:
  [1] Gemini
  [2] DeepSeek
  [3] OpenRouter
  [4] Groq

> 1

Model:
  [1] gemini-2.5-flash-lite  (recommended for cost)
  [2] gemini-2.5-pro         (recommended for quality)

> 1

Batch size:
  [1] 1   (safest, slowest)
  [2] 5   (recommended for local)
  [3] 10
  [4] 20
  [5] 50  (recommended for Gemini)
  [6] Custom

> 5

✅ Configuration:
   Provider: gemini
   Model: gemini-2.5-flash-lite
   API key: GEMINI_API_KEY found in environment ✓
   Batch size: 50

Start pipeline with these settings? [Y/n]
> Y

Saved Config

On first run, after confirming, write to .pipeline-config.json:

{
  "provider": "gemini",
  "model": "gemini-2.5-flash-lite",
  "batchSize": 50,
  "lastRun": "2026-07-06T13:54:00Z"
}

Next run shows [1] Use last config at the top.

API Key Rules

• Never prompt for keys. Check process.env for the provider's key.
• If missing: Print which env var is needed, then exit.
• Supported env vars: GEMINI_API_KEY, DEEPSEEK_API_KEY, OPENROUTER_API_KEY

Batch Size Recommendations

Provider	Recommended	Rationale
local (GTX 950M)	5	VRAM-limited, KV cache pressure
local (RTX 4090)	20	Fast, more VRAM
gemini	50	Free tier: 1,500 req/day
deepseek	20	5M free tokens, balance speed/cost
groq	50	Very fast, rate limits generous
openrouter	10	200 req/day free tier

---

11. Future Extensions & Roadmap

Near-Term (Next 2-4 Weeks)

Item	Status	Notes
Configurable batching	Complete	config/batch.ts with size and maxRetries
Retry + split logic	Complete	enrichWordWithRetry: 3 retries, then halve batch
Honest timing metrics	Complete	Unified throughput for all providers, detailed only for local
Auto-target languages	Complete	buildSystemPrompt() excludes source from targets
Validate LLM responses	Complete	validateSense() catches bad data before writes
Interactive CLI	Planned	Provider/model/batch selection with saved config
20-word torture suite	Pending	Decides gender approach and model selection
Qwen2.5-3B evaluation	Pending	Download and test
Online API testing	Pending	Gemini free tier, DeepSeek, Groq

Medium-Term (1-3 Months)

Item	Notes
Multi-POS support	Verbs, adjectives, adverbs need prompt variants
Multi-language source	German -> French, Italian -> Spanish, etc.
Parallel wordlist processing	Run english/nouns and english/verbs simultaneously
Incremental enrichment	Only process new/changed words in a wordlist
Model auto-switching	Fallback to online API if local server fails mid-run

Long-Term (3-6 Months)

Item	Notes
Batch API discounts	Gemini, Qwen, Azure offer 50% off for 24h SLA
GPU rental integration	Script to spin up Vast.ai/RunPod, run pipeline, download results
Quality regression tests	Run torture suite on every model change
Community contributions	Open-source the pipeline for other language learners

---

12. Decisions Log

Date	Decision	Context	Rationale
2026-01-04	TanStack Router for frontend	Previous project used React Router	Simpler, type-safe routing for the trainer app
2026-01-04	Vite dev server (no Nginx)	Docker setup for glossa-web	Nginx unnecessary for dev; Vite handles HMR and proxying
2026-01-17	Backend answer verification	Security vulnerability: correctAnswer exposed in API	Moved verification to server-side, shared schemas
2026-03-26	Multi-stage Docker builds	glossa-api and glossa-web containers	Smaller images, faster deploys
2026-06-16	llama.cpp for local LLM	Need local inference on old laptop	GGUF format, OpenAI-compatible API, no dependencies
2026-06-16	Q4_K_M quantization	Balance size vs quality	Q4_K_M is the community standard for 4-bit inference
2026-06-16	-c 2048 context	Default was 4096	Dictionary entries need ~500 tokens max; frees VRAM
2026-06-16	-t 2 physical cores	Default was 4 (HT threads)	Hyperthreading hurts llama.cpp performance
2026-06-17	Qwen2.5-1.5B as current model	Qwen3.5-4B too slow (47s/word)	6x speedup (8s/word), quality under evaluation
2026-06-17	Skip Gemma 4	E2B Q4_K_M is 3.46GB	Does not fit in 4GB VRAM; lower quants sacrifice quality
2026-06-17	Skip Ministral-3B	Tokenizer mismatch (Tekken)	Outputs gibberish regardless of template; not fixable without re-conversion
2026-07-06	Adapter pattern for LLM providers	Need to evaluate local vs cloud	utils/llm-adapters/ with factory + types + per-provider implementations
2026-07-06	Retry + split batching	LLM JSON parse failures on larger batches	enrichWordWithRetry retries 3 times, then halves batch until size 1
2026-07-06	Honest timing metrics	Cloud providers don't expose prompt/completion breakdown	Unified totalTimeMs for all; detailed breakdown only when available
2026-07-06	Auto-target languages	Prompt hardcoded English -> de/it/es/fr	ALL_LANGUAGES minus source = targets; works for any source language
2026-07-06	Validate LLM responses before write	Bad data was written then warned about	validateSense() catches malformed responses early, triggers retry

---

13. Known Issues & Dev Notes

glossa-web (Frontend)

Issue	Details
No healthcheck	Vite dev server has no health endpoint. Docker HEALTHCHECK cannot verify running state.
Valkey memory overcommit	Harmless warning in dev: vm.overcommit_memory = 1 recommended before production.

Data Pipeline

Issue	Details	Severity
Ministral-3B tokenizer mismatch	Tekken tokenizer not properly converted to GGUF. Model outputs gibberish.	Blocker - abandoned
Qwen2.5-1.5B gender hallucination	Systematic neuter default for Romance languages.	Under evaluation
Pre-scanning wordlists	Entire file read into memory before processing. Inefficient for 100k words.	Medium - streaming refactor planned
Single POS tested	Only nouns validated. Verbs/adjectives need prompt changes.	Known limitation

Hardware

Issue	Details
GTX 950M VRAM ceiling	4GB hard limit. Models >3GB risk OOM.
Maxwell GPU aging	No Flash Attention support, bandwidth-starved.
Laptop thermals	Cannot run 24/7 for weeks. Batch processing required.

Refactor Notes

Note	File	Context
Streaming vs pre-scan	pipeline.ts	For 100k words, stream and batch on-the-fly instead of reading entire file

---

14. How to Run

Prerequisites

• Node.js + npm
• tsx installed globally: npm install -g tsx
• llama.cpp built from source (for local mode)
• GGUF model downloaded to ~/Downloads/llama.cpp/models/ (for local mode)
• API keys set as environment variables (for cloud mode):

  export DEEPSEEK_API_KEY="sk-..."
  export GEMINI_API_KEY="..."
  export OPENROUTER_API_KEY="..."
  

Start the LLM Server (Local Mode)

cd ~/Downloads/llama.cpp

./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2

Run the Pipeline (Interactive CLI)

cd /path/to/data-pipeline
npx tsx pipeline.ts

Follow the prompts to select provider, model, and batch size.

Run with Last Config

cd /path/to/data-pipeline
npx tsx pipeline.ts
# Select [1] Use last config

Expected Output

🌐 Lila Data Pipeline
─────────────────────

[1] Use last config: local → qwen2.5-1.5b → batch 5
[2] Configure new run

> 1

🟢 Local AI engine is connected and ready for inference!

 step 1: scanning the source files...
✅ Scan complete! Found 1 wordlist(s):
   • ENGLISH (nouns)

 step 2: creating necessary output folders...
✅ All required output directories have been verified and created successfully.

 step 3: verifying local AI engine status...
🟢 Local AI engine is connected and ready for inference!

 step 4: looping through the wordlists...

Reading list: [ENGLISH] -> [NOUNS]

  Batch 1/1: [house, car, tree, water]
  [1/4] (0 failed) Enriched and saved: house.json
  [2/4] (0 failed) Enriched and saved: car.json
  [3/4] (0 failed) Enriched and saved: tree.json
  [4/4] (0 failed) Enriched and saved: water.json
  ⏱️  Word took 8.2s

⏱️  Pipeline Summary
   Duration: 32.8s
   Processed: 4
   Skipped: 0
   Failed: 0
   Total: 4
   Throughput: 0.12 words/sec

🤖 LLM Metrics
   Calls: 1
   Avg prompt tokens: 312
   Avg completion tokens: 524
   Avg total tokens: 836
   Avg total request time: 32800ms
   Avg throughput: 25.5 tok/s

   [Local breakdown]
   Avg prompt speed: 548.2 tok/s
   Avg completion speed: 18.8 tok/s

Global data pipeline run completed successfully.

---

15. Roadmap

Phase 1: Batching (Complete)

Task	Status	Notes
Implement configurable batch size	Complete	config/batch.ts with size and maxRetries
Implement retry + split logic	Complete	enrichWordWithRetry: 3 retries, then halve batch
Honest timing metrics	Complete	Unified throughput for all providers
Validate LLM responses	Complete	validateSense() catches bad data before writes
Verify batching doesn't break quality	Pending	Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output.
Measure speedup vs batch size	Pending	Track throughput at 1, 5, 15 on local hardware.

Goal: Unlock 5-15x speedup on local, unlock free online API tiers.

---

Phase 2: Interactive CLI

Task	Status	Notes
Design prompt flow	Planned	Provider → model → batch size → confirm
Implement CLI module	Planned	Use readline or inquirer for prompts
Save/load config	Planned	.pipeline-config.json
Wire into pipeline.ts	Planned	Replace static config with runtime config

Goal: No editing of TypeScript files to switch providers.

---

Phase 3: Model Selection

Task	Status	Notes
Download Qwen2.5-3B Q4_K_M	Pending	~1.9GB, fits in 4GB VRAM.
Run torture suite on Qwen2.5-3B (batched)	Pending	Check if gender accuracy improves with 2x params.
Test Gemini 2.5 Flash-Lite free tier (batched)	Pending	1,500 req/day x 50 words = 75k words/day. Zero cost.
Test DeepSeek V4 Flash free tier (batched)	Pending	5M tokens free. ~9k words.
Test Groq Llama 3.1 8B (batched, paid if needed)	Pending	Fastest option. ~$7 for 100k words.
Decide: local vs online, which model	Pending	Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost.

Goal: Pick the model and provider for the 100k word run.

---

Phase 4: Scale

Task	Status	Notes
Run 100k word pipeline	Pending	Estimated time depends on Phase 3 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq).
Spot-check output quality	Pending	Random sample of 100 entries.
Fix gender if needed	Pending	Kaikki lookup post-processing if LLM gender remains unreliable.
Handle failures & retries	Pending	Exponential backoff, split-and-retry for batch failures.

Goal: Complete 100k word dataset.

---

Phase 5: Extend

Task	Status	Notes
Multi-POS support	Pending	Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.).
Multi-language source	Pending	German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo.
Parallel wordlist processing	Pending	Run english/nouns and english/verbs simultaneously.
Incremental enrichment	Pending	Only process new/changed words in a wordlist.
GPU rental integration	Pending	Script to spin up Vast.ai/RunPod, run pipeline, download results.
Quality regression tests	Pending	Run torture suite on every model change.

Goal: Generalize pipeline for any language direction and POS.

---

Backlog (Unscheduled)

Task	Context
Batch API discounts	Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches.
Model auto-switching	Fallback to online API if local server fails mid-run.
Community open-source	Clean up, document, publish for other language learners.
Prometheus metrics	--metrics flag on llama-server for automated performance tracking.
-c 1024 / -b 256 experiments	Further VRAM optimization on GTX 950M. Low priority if moving to cloud.
Streaming wordlist processing	Read file line-by-line and batch on-the-fly. Eliminates pre-scan memory usage.