forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions
lila/documentation/DATA_PIPELINE.md
lila afd28d934e wip
2026-07-06 13:09:30 +02:00

51 KiB
Raw Blame History

Lila Data Pipeline — Technical Documentation

Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer. Last updated: 2026-06-17

Table of Contents

  1. Executive Summary
  2. Problem & Context
  3. Architecture Overview
  4. Current Implementation
  5. The LLM Layer
  6. The Gender Problem & Kaikki Integration
  7. Batching Strategy
  8. Hardware Constraints
  9. Testing & Quality Assurance
  10. Future Extensions & Roadmap
  11. Decisions Log
  12. Known Issues & Dev Notes
  13. How to Run
  14. Roadmap

Quick Reference

What Where
Entry point pipeline.ts
LLM config config/llm.ts
System prompt config/prompt.ts
Output schema utils/merge-enriched-data.ts
Batch size config config/batch.ts (planned)
Current model qwen2.5-1.5b-instruct-q4_k_m.gguf
Target scale 100,000+ words

1. Executive Summary

The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:

  • One or more senses (definitions)
  • A natural example sentence per sense
  • A CEFR-based difficulty level (easy / medium / hard)
  • Translations into German, Italian, Spanish, and French, each with grammatical gender

The pipeline is designed to scale to 100,000+ words across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It is currently in active development: the core architecture is stable, the LLM integration layer is being evaluated across local and online providers, and a configurable batching system is planned to unlock throughput at scale.

Key Design Principles

Principle Rationale
Quality first Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary.
Local-first, cloud-fallback Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed.
Resumable & idempotent Each word writes to its own JSON file. The pipeline skips already-processed words on restart.
Configurable batching Batch size (1, 5, 15, 50, etc.) is a single config value. The pipeline adapts without code changes.

Open Question: Gender Accuracy

Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that Qwen2.5-1.5B systematically defaults to neuter for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a model size issue (fixable by moving to 3B+) or a training data issue (requiring an external lookup) is unresolved.

Options under evaluation:

  • Larger local models (Qwen2.5-3B, Qwen3.5-1.7B)
  • Online models with stronger multilingual training (Gemini, DeepSeek)
  • Post-processing lookup via Kaikki Wiktionary dumps as a fallback or replacement

No decision made. Gender handling will be determined by the 20-word quality torture suite.

Current Status (2026-06-17)

  • Core pipeline: scanning, enrichment, merging, verification, writing
  • Local LLM integration via llama.cpp server (OpenAI-compatible API)
  • Schema validation for generated JSON
  • Progress tracking and timing metrics
  • In progress: Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
  • In progress: Designing configurable batching system
  • Pending: 20-word quality torture suite (will decide gender approach)
  • Pending: Online API evaluation (Gemini free tier, DeepSeek, Groq)

One-Line Architecture

source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> write .json
                                    |
                         [gender: LLM-generated, accuracy TBD]

Files at a Glance

File Purpose
pipeline.ts Orchestrator. Scans sources, loops words, coordinates all stages
config/llm.ts API URL, default parameters (temperature, max_tokens, etc.)
config/prompt.ts System prompt sent to the LLM
utils/enrich-word.ts Calls LLM, parses response, builds EnrichedSense[]
utils/merge-enriched-data.ts Merges skeleton + enriched senses into final JSON
utils/verify-enriched-file.ts Schema validation (required fields, types, gender enum)
utils/check-llm-server.ts Health check before pipeline starts
utils/progress-tracker.ts [current/total] formatting for console output
utils/pipeline-timer.ts Per-word and global timing + LLM token metrics

Scale Target

Metric Target
Words 100,000+
Languages English (source), German, Italian, Spanish, French (targets)
Parts of speech Nouns, verbs, adjectives, adverbs
Output One .json file per word, ~2-5KB each

2. Problem & Context

Why Build This?

Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:

What Exists What is Missing Why It Matters
Word + translation Student-friendly definition Learners need explanations, not just equivalents
Static difficulty labels CEFR-calibrated difficulty "Easy" vs "hard" must map to learner proficiency levels
Isolated translations Natural example sentences Context is how vocabulary is actually acquired
Raw gender data Gender integrated with translation Flashcards must show gender immediately, not as a separate lookup
Bulk word lists Per-word structured JSON The trainer consumes one file per word for fast random access

The Target User

A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender - all calibrated to their CEFR level (A1-C2).

Why Not Use Existing Dictionaries?

  • Wiktionary: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
  • Kaikki (Wiktionary dump): Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
  • Google Translate / DeepL: No definitions, no examples, no difficulty, no structured output
  • Existing language learning apps: Closed data, no export, no control over content

The LLM fills the gap: it generates pedagogical content (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.

Language Direction

The pipeline is direction-agnostic. A wordlist is defined by:

  • Source language: the language of the input words
  • Target languages: the languages to translate into

Current focus: English -> German/Italian/Spanish/French

Planned directions include German -> French, Italian -> Spanish, etc. The LLM prompt and output schema support any combination - the only change is the source wordlist and the target languages specified in the prompt.

Why 100,000+ Words?

  • Coverage: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation.
  • Languages: English (source) -> German, Italian, Spanish, French (targets).
  • Parts of speech: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.).

The Data Flow

Source files          LLM enrichment          Final JSON
(one word per line)   (definitions,           (one per word,
                      examples,               self-contained)
english/nouns         difficulty,
english/verbs         translations)            time.json
italian/nouns                                year.json
...                                          people.json

The Quality Challenge

Generating 100,000 entries with an LLM introduces risks:

Risk Mitigation
Hallucinated definitions Low temperature (0.1), strict system prompt, schema validation
Incorrect grammatical gender Under evaluation: larger models or external lookup
Inconsistent difficulty levels Explicit CEFR mapping in prompt, spot-checking
JSON parse failures Retry logic, schema validation, cleanup on failure
Model drift (online APIs) Version pinning, local fallback

Why TypeScript + Node?

  • Familiarity: Existing project uses TypeScript (frontend in TanStack Router + React)
  • Ecosystem: readline for streaming files, fs for JSON I/O, native fetch for HTTP
  • Portability: Runs on the same Debian laptop as the llama.cpp server
  • No build complexity: tsx for direct execution, no bundler needed

Why llama.cpp?

  • GGUF format: Single-file models, easy to swap, quantize, and version
  • OpenAI-compatible API: /v1/chat/completions means the same code works for local and online models
  • No dependencies: Self-contained binary, runs on old hardware (tested on GTX 950M)
  • Privacy: Local inference means no data leaves the machine

3. Architecture Overview

Pipeline Flow

Scan sources -> Check LLM -> Loop words -> Skip if exists -> Create skeleton
    -> Call LLM -> Parse JSON -> Merge -> Write atomically -> Verify schema

Resumability

  • Skip existing: Checks if {word}.json exists with non-empty senses
  • Atomic writes: .tmp -> rename, no partial files on crash
  • Cleanup on failure: Deletes incomplete file, continues to next word

Directory Structure

data-pipeline/
|-- pipeline.ts              # Entry point / orchestrator
|-- config/
|   |-- llm.ts               # API URL, model params
|   |-- prompt.ts            # System prompt
|-- utils/                   # See source files (provided separately)
|-- source-data/
|   |-- {language}/
|       |-- {pos}            # One word per line, no extension
|-- worddata/
    |-- {language}/
        |-- {pos}/
            |-- {word}.json  # One self-contained file per word

Output Schema

Each .json file contains: word, language, pos, senses[] (each with sense, example, difficulty_level, translations per target language), enrichedAt, model.

Full TypeScript interfaces: utils/merge-enriched-data.ts.

Error Handling

Failure Behavior
LLM server offline Hard fail at startup
LLM returns bad JSON Log, delete skeleton, continue
Schema validation fails Log warnings, keep file
Individual word fails Does not stop pipeline

Metrics

Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See utils/pipeline-timer.ts.

4. Current Implementation

Tech Stack

Layer Choice Why
Runtime Node.js + tsx Direct TypeScript execution, no build step
HTTP client Native fetch Works for local llama.cpp and online APIs
File I/O fs + readline Streaming line reader for large wordlists
JSON Native JSON.parse/stringify Simple, no schema library needed

Configuration

File Purpose
config/llm.ts LLM_API_URL, LLM_DEFAULT_PARAMS (temperature, top_p, max_tokens)
config/prompt.ts System prompt with CEFR mapping, required fields, example output

Key Modules

File Responsibility
utils/enrich-word.ts Calls LLM, strips markdown, parses JSON array, builds EnrichedSense[] with composite IDs
utils/merge-enriched-data.ts Merges skeleton {word, language, pos} with LLM senses, adds enrichedAt and model
utils/verify-enriched-file.ts Schema validation: required fields, array lengths, gender enum, translation structure
utils/pipeline-timer.ts Tracks per-word and global metrics (duration, tokens, throughput)

Current Model

Property Value
Model qwen2.5-1.5b-instruct-q4_k_m.gguf
Size ~1.0GB
Quantization Q4_K_M
Server llama.cpp (llama-server)
API OpenAI-compatible /v1/chat/completions

llama-server Flags: History & Rationale

The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen.

Hardware Constraints

Component Spec Implication
CPU i7-6500U (2 physical cores, 4 threads HT) -t 2 matches physical cores; HT hurts more than helps
GPU GTX 950M (Maxwell, 2015) 32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved
RAM 8GB (3.95GB usable) --mlock pins model in RAM; system must not swap

Flag Evolution

Flag Value Tried Result Why
-m qwen3.5-4b-q4_k_m.gguf Works, ~47s/word Baseline. Correct genders. 2.6GB file, tight on VRAM.
-m Ministral-3b-instruct.Q4_K_M.gguf Broken Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See Known Issues.
-m qwen2.5-1.5b-instruct-q4_k_m.gguf Works, ~8s/word Current. Fast but gender accuracy degraded.
-ngl 999 Keeps Offload all layers to GPU. Required for any speed.
-c 4096 Wasteful 4K context for 300-token dictionary entries wastes ~400MB VRAM.
-c 2048 Current Sufficient headroom. Frees VRAM for KV cache.
-b / -ub 512 Keeps Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs.
-b / -ub 2048 Slower on 950M Tested briefly. No improvement, possibly worse due to memory pressure.
-t 4 Slower Hyperthreading cores hurt llama.cpp performance.
-t 2 Current Matches 2 physical cores.
--threads-batch (default) Risky Defaults to same as -t, but explicit is safer.
--threads-batch 2 Current Explicit match to -t.
--flash-attn (omitted) Correct On Maxwell (compute 5.0), Flash Attention adds overhead. Not used.
--flash-attn (tested) No gain Briefly tried with Qwen3.5-4B. No speedup, possibly regression.
--mlock Keeps Pins model weights in RAM. Prevents OS swapping on memory pressure.
--prio 2 Keeps Raises process priority. Marginal on this hardware, harmless.
--reasoning off (Qwen3.5 only) Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5.

Current Command

./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2

What Was Not Tried (And Why)

Flag Reason Skipped
-fa / --flash-attn Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs.
--no-mmap --mlock achieves the same (pin in RAM) without the I/O overhead of disabling mmap.
-np / --parallel Not needed. Single sequential pipeline, no concurrent requests.
--cont-batching Default in recent llama.cpp. No need to toggle.
--defrag-thold KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx.
-ot / --override-tensor Expert-level. No tensor-specific issues observed.

Future Flag Experiments

Experiment Expected Outcome
-c 1024 Further VRAM savings. Risk: insufficient for batching larger prompt sizes.
-b 256 -ub 256 Test if smaller batches reduce latency on bandwidth-starved Maxwell.
--metrics Enable Prometheus endpoint for automated performance tracking.

Performance Baseline

Metric Qwen3.5-4B Qwen2.5-1.5B
Time/word ~47s ~8s
Completion tok/s ~6.4 ~18.8
Prompt tok/s ~81 ~549
Avg completion tokens ~274 ~132
Avg prompt tokens ~327 ~308

Known Limitations (Current)

  • Gender accuracy: Qwen2.5-1.5B systematically defaults to neuter for Romance languages. Under evaluation whether larger models fix this.
  • No batching: One word = one LLM call. System prompt re-processed every time.
  • No retry logic: LLM parse failures are logged and skipped, not retried.
  • Single POS: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments.

5. The LLM Layer

5.1 Local Model Evaluation

Models are evaluated on three criteria in order of priority: quality (definitions, examples, translations, gender accuracy), speed (completion tokens/sec), VRAM fit (must run on GTX 950M 4GB).

Model Size VRAM Speed Quality Status
Qwen3.5-4B Q4_K_M 2.6GB Tight ~6.4 tok/s Baseline (assumed good) Baseline - too slow
Qwen2.5-1.5B Q4_K_M 1.0GB Easy ~18.8 tok/s Gender systematically wrong Current - fast, needs validation
Ministral-3B Q4_K_M 1.9GB Fits - Broken (tokenizer) Abandoned
Qwen2.5-3B Q4_K_M 1.9GB Fits ~12-15 tok/s (est.) Unknown Pending download
Gemma 4 E2B Q4_K_M 3.46GB No - - Too large for 4GB VRAM
Gemma 4 E2B IQ2_M 2.62GB Fits ~8-12 tok/s (est.) "Low quality" per Google Not worth it

Qwen2.5-1.5B Test Results (3 words)

Word Definition Example Gender (de/it/es/fr) Verdict
time "A period of duration..." "The meeting was scheduled..." neuter/neuter/neuter/neuter All wrong
year "A period of time consisting of 365 days..." "The year 2023 is a leap year." neuter/neuter/neuter/neuter All wrong
people "Individuals who are part of a group." "The people gathered at the park." neuter/feminine/feminine/feminine Mixed

Pattern: Defaults to neuter when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident.

Pending Tests

  • Qwen2.5-3B: Same architecture, 2x params. If gender fixes, it was a size problem.
  • 20-word torture suite: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models.

5.2 Online API Options

Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API.

Provider Model Input $/1M Output $/1M Free Tier Rate Limit Est. Cost (100k words) Est. Time
DeepSeek V4 Flash $0.14 $0.28 5M tokens None $0-15 ~1-2 days
Gemini 2.5 Flash-Lite $0.10 $0.40 1,500 req/day 1M TPM $0 (free tier) ~1.5 days (batched)
Qwen/Alibaba Qwen-Turbo $0.05 $0.20 Unknown 600 RPM ~$11 ~1-2 days
Groq Llama 3.1 8B Instant $0.05 $0.08 Yes High ~$7 ~3-4 hours
OpenRouter free Various $0 $0 200 req/day 20 RPM $0 ~10 days
OpenRouter paid DeepSeek V4 Flash $0.14 $0.28 - Same as provider ~$16 (+5.5% fee) ~2-3 days

Notes:

  • Costs assume ~550 tokens/word (300 prompt + 250 completion).
  • Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day.
  • Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable.
  • DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million.

5.3 Model Selection Criteria

Decision flow for 100,000 words:

Start
  |
  v
Run 20-word torture suite
on Qwen2.5-3B (local)
  |
  |-- Quality good? -----> Use Qwen2.5-3B locally
  |   (gender correct)       ~20 days, $0
  |
  |-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free)
                              |
                              |-- Quality good? --> Batch 50, free tier
                              |                      ~1.5 days, $0
                              |
                              |-- Quality meh? ---> Test Groq or DeepSeek paid
                                                        |
                                                        |-- Speed priority? --> Groq
                                                        |                       ~$7, 3-4 hours
                                                        |
                                                        |-- Cost priority? ---> DeepSeek
                                                                                  ~$15, 1-2 days

Quality gates:

  • =90% gender accuracy (de/it/es/fr)

  • 100% JSON parse rate
  • No hallucinated definitions on polysemous words
  • Natural, contextually appropriate example sentences
  • Sensible difficulty classification (CEFR mapping)

6. The Gender Problem & Kaikki Integration

The Problem

Grammatical gender is embedded in the translations object of each sense:

"translations": {
  "de": [{"word": "Haus", "gender": "neuter"}],
  "it": [{"word": "casa", "gender": "feminine"}],
  "es": [{"word": "casa", "gender": "feminine"}],
  "fr": [{"word": "maison", "gender": "feminine"}]
}

Early testing with Qwen2.5-1.5B showed systematic failure: the model defaults to neuter for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all - only masculine and feminine.

Whether this is a model size issue (1.5B too small to retain gender facts) or a training data gap (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation.

Two Approaches Under Consideration

Approach How It Works Pros Cons
A. LLM-only Trust the model to generate correct gender Zero additional complexity Unreliable at small sizes; hallucination risk
B. LLM + Kaikki lookup LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump 100% deterministic; decouples gender from model quality Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries

Kaikki Data

Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender.

Language Kaikki File Coverage
German kaikki.org-dictionary-German.jsonl High
Italian kaikki.org-dictionary-Italian.jsonl High
Spanish kaikki.org-dictionary-Spanish.jsonl High
French kaikki.org-dictionary-French.jsonl High

Lookup logic: match on word (the translated string) -> extract gender field -> map to "masculine" | "feminine" | "neuter" | null.

Decision Pending

  • If Qwen2.5-3B or an online model produces >=90% accurate gender: Approach A, no Kaikki needed.
  • If all tested models fail gender: Approach B, implement Kaikki lookup as a post-processing step after LLM enrichment.

No implementation work started until the 20-word torture suite resolves this.

7. Batching Strategy

Why Batching is Necessary

At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.

Configurable Batch Size

Single config point controls batch size everywhere:

// config/batch.ts (planned)
export const BATCH_CONFIG = {
  size: 5, // Change to 15 or 50 to test
  get maxTokens() {
    return Math.ceil(this.size * 250 * 1.2); // 250 tok/word + 20% buffer
  },
} as const;

Prompt Structure

Single word:

Word: house

Batch of 5:

Words: ["house", "car", "tree", "water", "book"]

LLM returns a JSON object with word keys:

{
  "house": [ { "sense": "...", "example": "...", ... } ],
  "car": [ { ... } ],
  "tree": [ { ... } ],
  "water": [ { ... } ],
  "book": [ { ... } ]
}

Retry Strategy

If a batch fails (bad JSON, missing key, etc.):

Batch of 50 fails
    |
    v
Retry as 2 batches of 25
    |
    v
If a 25 fails, retry as 5 batches of 5
    |
    v
If a 5 fails, retry as individual words (fallback)

This gives resilience without losing the speed benefit of large batches.

Expected Impact by Environment

Environment Batch Size Expected Speedup Notes
Local GTX 950M 3 ~1.05-1.15x Memory bandwidth limited. KV cache pressure on 4GB VRAM.
Local GTX 950M 5 ~1.10-1.20x Sweet spot for this hardware.
Local GTX 950M 15 Risky May OOM. Test carefully.
Local GTX 950M 50 Unlikely VRAM insufficient.
Cloud API (Gemini) 50 5x fewer calls Unlocks free tier viability.
Cloud API (Groq) 50 5x fewer calls Minimal gain - already fast.

Batching is Non-Negotiable

Regardless of local vs cloud, batching is required for 100k words:

  • Local: Better GPU utilization, amortized prompt processing
  • Cloud: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts

8. Hardware Constraints

Current Machine

Component Spec
OS Debian GNU/Linux 13 (trixie) x86_64
CPU Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz
GPU NVIDIA GeForce GTX 950M (Maxwell, 2015)
GPU VRAM 4GB
RAM 8GB (3.95GB usable at idle)
Disk 102GB ext4 (~63GB used)

What Fits in 4GB VRAM

Model File Size KV Cache (2048 ctx) Total VRAM Fits?
Qwen2.5-1.5B Q4_K_M ~1.0GB ~0.5GB ~1.5GB Yes
Qwen2.5-3B Q4_K_M ~1.9GB ~0.8GB ~2.7GB Yes
Ministral-3B Q4_K_M ~1.9GB ~0.8GB ~2.7GB Yes (but broken)
Qwen3.5-4B Q4_K_M 2.6GB ~1.0GB ~3.6GB Tight
Gemma 4 E2B Q4_K_M 3.46GB ~1.2GB ~4.7GB No
Gemma 4 E2B IQ2_M 2.62GB ~1.0GB ~3.6GB Maybe, low quality

GPU Rental Alternatives

If local hardware becomes the bottleneck:

Provider GPU VRAM Price/Hour Time for 100k Words Total Cost
Vast.ai RTX 4090 24GB ~$0.30-0.60 ~6-8 hours ~$2-5
RunPod RTX 4090 24GB ~$0.50-0.80 ~6-8 hours ~$4-6
Vast.ai RTX 3090 24GB ~$0.20-0.40 ~8-10 hours ~$2-4

With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day.

9. Testing & Quality Assurance

20-Word Torture Suite

Planned test set covering edge cases:

Category Words Why
Easy concrete house, water, book Baseline
Easy abstract time, love, hope Abstract nouns harder to define
Polysemous bank, run, light Multiple senses test disambiguation
Hard/technical democracy, photosynthesis, entropy Complex definitions
False friends actual (en/es), sensible (en/fr), fabric (en/de) Cross-lingual traps

Evaluation Criteria

For each word and each candidate model:

Criterion Pass Threshold
Definition accuracy Factually correct, max 15 words, student-friendly
Example quality Natural sentence, word used correctly in context
Translation correctness Correct word sense match
Gender accuracy (de/it/es/fr) >=90% correct
Difficulty classification Sensible per CEFR mapping
JSON reliability 100% parse rate, valid schema

Verification

verify-enriched-file.ts checks:

  • Required top-level fields: word, language, pos, senses
  • Each sense: sense (string), example (string), difficulty_level in {easy, medium, hard}
  • Each translation: word (string), gender in {masculine, feminine, neuter, null}

10. Future Extensions & Roadmap

Near-Term (Next 2-4 Weeks)

Item Status Notes
Configurable batching In progress Single BATCH_CONFIG.size value
20-word torture suite Pending Decides gender approach and model selection
Qwen2.5-3B evaluation Pending Download and test
Online API testing Pending Gemini free tier, DeepSeek, Groq
Retry logic Pending Exponential backoff on LLM failures

Medium-Term (1-3 Months)

Item Notes
Multi-POS support Verbs, adjectives, adverbs need prompt variants
Multi-language source German -> French, Italian -> Spanish, etc.
Parallel wordlist processing Run english/nouns and english/verbs simultaneously
Incremental enrichment Only process new/changed words in a wordlist
Model auto-switching Fallback to online API if local server fails

Long-Term (3-6 Months)

Item Notes
Batch API discounts Gemini, Qwen, Azure offer 50% off for 24h SLA
GPU rental integration Script to spin up Vast.ai/RunPod, run pipeline, download results
Quality regression tests Run torture suite on every model change
Community contributions Open-source the pipeline for other language learners

11. Decisions Log

Date Decision Context Rationale
2026-01-04 TanStack Router for frontend Previous project used React Router Simpler, type-safe routing for the trainer app
2026-01-04 Vite dev server (no Nginx) Docker setup for glossa-web Nginx unnecessary for dev; Vite handles HMR and proxying
2026-01-17 Backend answer verification Security vulnerability: correctAnswer exposed in API Moved verification to server-side, shared schemas
2026-03-26 Multi-stage Docker builds glossa-api and glossa-web containers Smaller images, faster deploys
2026-06-16 llama.cpp for local LLM Need local inference on old laptop GGUF format, OpenAI-compatible API, no dependencies
2026-06-16 Q4_K_M quantization Balance size vs quality Q4_K_M is the community standard for 4-bit inference
2026-06-16 -c 2048 context Default was 4096 Dictionary entries need ~500 tokens max; frees VRAM
2026-06-16 -t 2 physical cores Default was 4 (HT threads) Hyperthreading hurts llama.cpp performance
2026-06-17 Qwen2.5-1.5B as current model Qwen3.5-4B too slow (47s/word) 6x speedup (8s/word), quality under evaluation
2026-06-17 Skip Gemma 4 E2B Q4_K_M is 3.46GB Does not fit in 4GB VRAM; lower quants sacrifice quality
2026-06-17 Skip Ministral-3B Tokenizer mismatch (Tekken) Outputs gibberish regardless of template; not fixable without re-conversion

12. Known Issues & Dev Notes

glossa-web (Frontend)

Issue Details
No healthcheck Vite dev server has no health endpoint. Docker HEALTHCHECK cannot verify running state.
Valkey memory overcommit Harmless warning in dev: vm.overcommit_memory = 1 recommended before production.

Data Pipeline

Issue Details Severity
Ministral-3B tokenizer mismatch Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. Blocker - abandoned
Qwen2.5-1.5B gender hallucination Systematic neuter default for Romance languages. Under evaluation
No batching 1 word = 1 call. System prompt re-processed every time. In progress
No retry logic LLM parse failures are logged and skipped. Planned
Single POS tested Only nouns validated. Verbs/adjectives need prompt changes. Known limitation

Hardware

Issue Details
GTX 950M VRAM ceiling 4GB hard limit. Models >3GB risk OOM.
Maxwell GPU aging No Flash Attention support, bandwidth-starved.
Laptop thermals Cannot run 24/7 for weeks. Batch processing required.

13. How to Run

Prerequisites

  • Node.js + npm
  • tsx installed globally: npm install -g tsx
  • llama.cpp built from source
  • GGUF model downloaded to ~/Downloads/llama.cpp/models/

Start the LLM Server

cd ~/Downloads/llama.cpp

./build/bin/llama-server \
  -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
  -ngl 999 \
  -c 2048 \
  -b 512 \
  -ub 512 \
  -t 2 \
  --threads-batch 2 \
  --host 127.0.0.1 \
  --port 8080 \
  --mlock \
  --prio 2

Run the Pipeline

cd /path/to/data-pipeline
npx tsx pipeline.ts

Expected Output

Starting data pipeline...

 step 1: scanning the source files...
Scan complete! Found 1 wordlist(s):
   - ENGLISH (nouns)

...

Pipeline Summary
   Duration: 23.0s
   Processed: 3
   Skipped: 0
   Failed: 0
   Total: 3
   Throughput: 0.13 words/sec

LLM Metrics
   Calls: 3
   Avg prompt tokens: 308
   Avg completion tokens: 132
   Avg prompt speed: 549.4 tok/s
   Avg completion speed: 18.8 tok/s

Global data pipeline run completed successfully.

Environment Variables (Online Mode)

export DEEPSEEK_API_KEY="sk-..."
export GEMINI_API_KEY="..."
export GROQ_API_KEY="..."

Then update config/llm.ts to point to the online API URL.

14. Roadmap

Phase 1: Batching (Current)

Task Status Notes
Implement configurable batch size In progress Single BATCH_CONFIG.size value. Prompt formatting: single word -> word array. Response parsing: keyed JSON object. Retry: 50 -> 25 -> 5 -> 1.
Verify batching doesn't break quality Pending Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output.
Measure speedup vs batch size Pending Track throughput at 1, 5, 15 on local hardware.

Goal: Unlock 5-15x speedup on local, unlock free online API tiers.

Phase 2: Model Selection

Task Status Notes
Download Qwen2.5-3B Q4_K_M Pending ~1.9GB, fits in 4GB VRAM.
Run torture suite on Qwen2.5-3B (batched) Pending Check if gender accuracy improves with 2x params.
Test Gemini 2.5 Flash-Lite free tier (batched) Pending 1,500 req/day x 50 words = 75k words/day. Zero cost.
Test DeepSeek V4 Flash free tier (batched) Pending 5M tokens free. ~9k words.
Test Groq Llama 3.1 8B (batched, paid if needed) Pending Fastest option. ~$7 for 100k words.
Decide: local vs online, which model Pending Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost.

Goal: Pick the model and provider for the 100k word run.

Phase 3: Scale

Task Status Notes
Run 100k word pipeline Pending Estimated time depends on Phase 2 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq).
Spot-check output quality Pending Random sample of 100 entries.
Fix gender if needed Pending Kaikki lookup post-processing if LLM gender remains unreliable.
Handle failures & retries Pending Exponential backoff, split-and-retry for batch failures.

Goal: Complete 100k word dataset.

Phase 4: Extend

Task Status Notes
Multi-POS support Pending Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.).
Multi-language source Pending German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo.
Parallel wordlist processing Pending Run english/nouns and english/verbs simultaneously.
Incremental enrichment Pending Only process new/changed words in a wordlist.
GPU rental integration Pending Script to spin up Vast.ai/RunPod, run pipeline, download results.
Quality regression tests Pending Run torture suite on every model change.

Goal: Generalize pipeline for any language direction and POS.

Backlog (Unscheduled)

Task Context
Batch API discounts Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches.
Model auto-switching Fallback to online API if local server fails mid-run.
Community open-source Clean up, document, publish for other language learners.
Prometheus metrics --metrics flag on llama-server for automated performance tracking.
-c 1024 / -b 256 experiments Further VRAM optimization on GTX 950M. Low priority if moving to cloud.