1149 lines
65 KiB
Markdown
1149 lines
65 KiB
Markdown
# Lila Data Pipeline — Technical Documentation
|
|
|
|
> Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer.
|
|
> Last updated: 2026-07-06
|
|
|
|
---
|
|
|
|
## Table of Contents
|
|
|
|
1. [Executive Summary](#1-executive-summary)
|
|
2. [Problem & Context](#2-problem--context)
|
|
3. [Architecture Overview](#3-architecture-overview)
|
|
4. [Current Implementation](#4-current-implementation)
|
|
5. [The LLM Layer](#5-the-llm-layer)
|
|
- 5.1 [Local Model Evaluation](#51-local-model-evaluation)
|
|
- 5.2 [Online API Options](#52-online-api-options)
|
|
- 5.3 [Model Selection Criteria](#53-model-selection-criteria)
|
|
6. [The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration)
|
|
7. [Batching Strategy](#7-batching-strategy)
|
|
8. [Hardware Constraints](#8-hardware-constraints)
|
|
9. [Testing & Quality Assurance](#9-testing--quality-assurance)
|
|
10. [Interactive CLI](#10-interactive-cli)
|
|
11. [Future Extensions & Roadmap](#11-future-extensions--roadmap)
|
|
12. [Decisions Log](#12-decisions-log)
|
|
13. [Known Issues & Dev Notes](#13-known-issues--dev-notes)
|
|
14. [How to Run](#14-how-to-run)
|
|
15. [Roadmap](#15-roadmap)
|
|
|
|
---
|
|
|
|
## Quick Reference
|
|
|
|
| What | Where |
|
|
| ------------------- | ------------------------------------------ |
|
|
| Entry point | `pipeline.ts` |
|
|
| Interactive CLI | `utils/cli.ts` |
|
|
| LLM config schema | `config/llm.ts` |
|
|
| System prompt | `config/prompt.ts` — `buildSystemPrompt()` |
|
|
| Batch config schema | `config/batch.ts` |
|
|
| Shared constants | `config/constants.ts` |
|
|
| Output schema | `utils/merge-enriched-data.ts` |
|
|
| LLM adapters | `utils/llm-adapters/` |
|
|
| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
|
|
| Target scale | 100,000+ words |
|
|
|
|
---
|
|
|
|
## 1. Executive Summary
|
|
|
|
The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates:
|
|
|
|
- One or more **senses** (definitions)
|
|
- A **natural example sentence** per sense
|
|
- A **CEFR-based difficulty level** (`easy` / `medium` / `hard`)
|
|
- **Translations** into all target languages except the source, each with grammatical **gender**
|
|
|
|
The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an **interactive CLI** for provider selection.
|
|
|
|
### Key Design Principles
|
|
|
|
| Principle | Rationale |
|
|
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
|
|
| **Quality first** | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary. |
|
|
| **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. |
|
|
| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. |
|
|
| **Configurable batching** | Batch size is selected interactively at runtime. The pipeline adapts without code changes. |
|
|
| **Provider-agnostic** | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface. |
|
|
| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns. |
|
|
|
|
### Open Question: Gender Accuracy
|
|
|
|
Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that **Qwen2.5-1.5B systematically defaults to `neuter`** for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a **model size issue** (fixable by moving to 3B+) or a **training data issue** (requiring an external lookup) is unresolved.
|
|
|
|
**Options under evaluation:**
|
|
|
|
- Larger local models (Qwen2.5-3B, Qwen3.5-1.7B)
|
|
- Online models with stronger multilingual training (Gemini, DeepSeek)
|
|
- Post-processing lookup via **Kaikki Wiktionary dumps** as a fallback or replacement
|
|
|
|
No decision made. Gender handling will be determined by the 20-word quality torture suite.
|
|
|
|
### Current Status (2026-07-06)
|
|
|
|
- Core pipeline: scanning, enrichment, merging, verification, writing
|
|
- Local LLM integration via llama.cpp server (OpenAI-compatible API)
|
|
- **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/`
|
|
- **Interactive CLI**: provider/model/batch selection with saved config
|
|
- **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1)
|
|
- **Honest timing**: unified throughput for all providers, detailed breakdown only for local
|
|
- **Auto-target languages**: prompt dynamically excludes source language from targets
|
|
- **Schema validation**: validates LLM response structure before file writes
|
|
- **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
|
|
- **Pending:** 20-word quality torture suite (will decide gender approach)
|
|
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq)
|
|
|
|
### One-Line Architecture
|
|
|
|
```
|
|
source wordlists -> Interactive CLI -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json
|
|
|
|
|
[gender: LLM-generated, accuracy TBD]
|
|
```
|
|
|
|
### Files at a Glance
|
|
|
|
| File | Purpose |
|
|
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------- |
|
|
| `pipeline.ts` | Orchestrator. Runs CLI, scans sources, loops words, coordinates all stages |
|
|
| `utils/cli.ts` | Interactive CLI. Provider/model/batch selection, config persistence |
|
|
| `config/llm.ts` | LLM config schema (provider, url, model) |
|
|
| `config/prompt.ts` | `buildSystemPrompt()` — dynamic prompt with auto-target languages |
|
|
| `config/batch.ts` | Batch size and max retry count schema |
|
|
| `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
|
|
| `utils/enrich-word.ts` | Calls LLM via adapter, parses response, validates senses, builds `EnrichedSense[]`, retry/split logic |
|
|
| `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON |
|
|
| `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) |
|
|
| `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers |
|
|
| `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory |
|
|
| `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files |
|
|
| `utils/write-json-file.ts` | Atomic `.tmp` → rename writes |
|
|
| `utils/check-if-json-exists.ts` | Resumability: checks if word already has enriched senses |
|
|
| `utils/create-line-reader.ts` | Streaming line reader for large wordlists |
|
|
| `utils/create-output-dirs.ts` | Creates `worddata/{language}/{pos}/` folders |
|
|
| `utils/delete-file.ts` | Cleanup helper for failed batches |
|
|
| `utils/get-word-file-path.ts` | Path construction helper |
|
|
| `utils/progress-tracker.ts` | `[current/total]` formatting for console output |
|
|
| `utils/pipeline-timer.ts` | Timing + token metrics; unified throughput for all providers |
|
|
| `utils/llm-adapters/factory.ts` | Creates the right adapter based on runtime config |
|
|
| `utils/llm-adapters/types.ts` | `LlmAdapter` interface |
|
|
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek |
|
|
| `utils/llm-adapters/gemini.ts` | Google Gemini native API |
|
|
|
|
### Scale Target
|
|
|
|
| Metric | Target |
|
|
| --------------- | ------------------------------------------------------------ |
|
|
| Words | 100,000+ |
|
|
| Languages | English (source), German, Italian, Spanish, French (targets) |
|
|
| Parts of speech | Nouns, verbs, adjectives, adverbs |
|
|
| Output | One `.json` file per word, ~2-5KB each |
|
|
|
|
---
|
|
|
|
## 2. Problem & Context
|
|
|
|
### Why Build This?
|
|
|
|
Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer:
|
|
|
|
| What Exists | What is Missing | Why It Matters |
|
|
| ------------------------ | -------------------------------------- | ----------------------------------------------------------------- |
|
|
| Word + translation | **Student-friendly definition** | Learners need explanations, not just equivalents |
|
|
| Static difficulty labels | **CEFR-calibrated difficulty** | "Easy" vs "hard" must map to learner proficiency levels |
|
|
| Isolated translations | **Natural example sentences** | Context is how vocabulary is actually acquired |
|
|
| Raw gender data | **Gender integrated with translation** | Flashcards must show gender immediately, not as a separate lookup |
|
|
| Bulk word lists | **Per-word structured JSON** | The trainer consumes one file per word for fast random access |
|
|
|
|
### The Target User
|
|
|
|
A language learner using the Lila vocabulary trainer. They see a word, its definition, an example sentence, and translations with gender — all calibrated to their CEFR level (A1-C2).
|
|
|
|
### Why Not Use Existing Dictionaries?
|
|
|
|
- **Wiktionary**: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions
|
|
- **Kaikki (Wiktionary dump)**: Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation
|
|
- **Google Translate / DeepL**: No definitions, no examples, no difficulty, no structured output
|
|
- **Existing language learning apps**: Closed data, no export, no control over content
|
|
|
|
The LLM fills the gap: it generates **pedagogical content** (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale.
|
|
|
|
### Language Direction
|
|
|
|
The pipeline is **direction-agnostic**. A wordlist is defined by:
|
|
|
|
- **Source language**: the language of the input words
|
|
- **Target languages**: all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source)
|
|
|
|
Current focus: **English -> German/Italian/Spanish/French**
|
|
|
|
Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist. Target languages are computed automatically.
|
|
|
|
### Why 100,000+ Words?
|
|
|
|
- **Coverage**: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation.
|
|
- **Languages**: English (source) -> German, Italian, Spanish, French (targets).
|
|
- **Parts of speech**: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.).
|
|
|
|
### The Data Flow
|
|
|
|
```
|
|
Source files LLM enrichment Final JSON
|
|
(one word per line) (definitions, (one per word,
|
|
examples, self-contained)
|
|
english/nouns difficulty,
|
|
english/verbs translations) time.json
|
|
italian/nouns year.json
|
|
... people.json
|
|
```
|
|
|
|
### The Quality Challenge
|
|
|
|
Generating 100,000 entries with an LLM introduces risks:
|
|
|
|
| Risk | Mitigation |
|
|
| ------------------------------ | -------------------------------------------------------------- |
|
|
| Hallucinated definitions | Low temperature (0.1), strict system prompt, schema validation |
|
|
| Incorrect grammatical gender | Under evaluation: larger models or external lookup |
|
|
| Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking |
|
|
| JSON parse failures | Retry + split logic, schema validation, cleanup on failure |
|
|
| Model drift (online APIs) | Version pinning, local fallback |
|
|
| Provider downtime | Adapter pattern allows hot-swapping providers |
|
|
| Malformed LLM responses | `validateSense()` catches bad data before file writes |
|
|
|
|
### Why TypeScript + Node?
|
|
|
|
- **Familiarity**: Existing project uses TypeScript (frontend in TanStack Router + React)
|
|
- **Ecosystem**: `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP
|
|
- **Portability**: Runs on the same Debian laptop as the llama.cpp server
|
|
- **No build complexity**: `tsx` for direct execution, no bundler needed
|
|
|
|
### Why llama.cpp?
|
|
|
|
- **GGUF format**: Single-file models, easy to swap, quantize, and version
|
|
- **OpenAI-compatible API**: `/v1/chat/completions` means the same adapter code works for local and online models
|
|
- **No dependencies**: Self-contained binary, runs on old hardware (tested on GTX 950M)
|
|
- **Privacy**: Local inference means no data leaves the machine
|
|
|
|
---
|
|
|
|
## 3. Architecture Overview
|
|
|
|
### Pipeline Flow
|
|
|
|
```
|
|
Run CLI -> Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
|
|
-> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
|
|
-> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics
|
|
```
|
|
|
|
### Resumability
|
|
|
|
- **Skip existing**: `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses`
|
|
- **Atomic writes**: `.tmp` -> rename in `write-json-file.ts`, no partial files on crash
|
|
- **Cleanup on failure**: Deletes partially-written files for failed batches, continues to next batch
|
|
|
|
### Directory Structure
|
|
|
|
```
|
|
data-pipeline/
|
|
|-- pipeline.ts # Entry point / orchestrator
|
|
|-- utils/
|
|
| |-- cli.ts # Interactive CLI module
|
|
| |-- enrich-word.ts # LLM call, parse, retry/split
|
|
| |-- merge-enriched-data.ts # Schema types + merge logic
|
|
| |-- verify-enriched-file.ts # Schema validation
|
|
| |-- check-llm-server.ts # Health check (local only)
|
|
| |-- scanning-source-files.ts # Source discovery
|
|
| |-- create-base-json.ts # Skeleton writer
|
|
| |-- write-json-file.ts # Atomic JSON writer
|
|
| |-- check-if-json-exists.ts # Resumability check
|
|
| |-- create-line-reader.ts # Streaming file reader
|
|
| |-- create-output-dirs.ts # Directory creation
|
|
| |-- delete-file.ts # Cleanup helper
|
|
| |-- get-word-file-path.ts # Path helper
|
|
| |-- progress-tracker.ts # Console progress formatting
|
|
| |-- pipeline-timer.ts # Timing + token metrics
|
|
| |-- llm-adapters/
|
|
| |-- factory.ts # Adapter selection (uses runtime config)
|
|
| |-- types.ts # LlmAdapter interface
|
|
| |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
|
|
| |-- gemini.ts # Google Gemini
|
|
|-- config/
|
|
| |-- llm.ts # LLM config schema
|
|
| |-- prompt.ts # buildSystemPrompt() — dynamic prompt
|
|
| |-- batch.ts # Batch config schema
|
|
| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES
|
|
|-- source-data/
|
|
| |-- {language}/
|
|
| |-- {pos} # One word per line, no extension
|
|
|-- worddata/
|
|
|-- {language}/
|
|
|-- {pos}/
|
|
|-- {word}.json # One self-contained file per word
|
|
|-- kaikki-source-files/ # Wiktionary dumps for gender lookup (planned)
|
|
|-- .pipeline-config.json # Saved CLI configuration
|
|
```
|
|
|
|
### Output Schema
|
|
|
|
Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `id`, `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`.
|
|
|
|
Full TypeScript interfaces: `utils/merge-enriched-data.ts`.
|
|
|
|
### Error Handling
|
|
|
|
| Failure | Behavior |
|
|
| ------------------------------ | -------------------------------------------------------- |
|
|
| LLM server offline | Hard fail at startup (`check-llm-server.ts`, local only) |
|
|
| LLM returns bad JSON | Retry up to 3 times, then split batch. Log and continue |
|
|
| LLM returns malformed senses | `validateSense()` catches it before file write |
|
|
| Schema validation fails | Log warnings, keep file |
|
|
| Individual batch fails | Does not stop pipeline; cleans up partial files |
|
|
| Individual word fails (size 1) | Log and continue to next word |
|
|
|
|
### Metrics
|
|
|
|
Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`.
|
|
|
|
**Unified throughput** (all providers): total tokens / total request time
|
|
**Detailed breakdown** (local only): prompt speed vs completion speed
|
|
|
|
---
|
|
|
|
## 4. Current Implementation
|
|
|
|
### Tech Stack
|
|
|
|
| Layer | Choice | Why |
|
|
| ----------- | ----------------------------- | ------------------------------------------ |
|
|
| Runtime | Node.js + `tsx` | Direct TypeScript execution, no build step |
|
|
| HTTP client | Native `fetch` | Works for local llama.cpp and online APIs |
|
|
| File I/O | `fs` + `readline` | Streaming line reader for large wordlists |
|
|
| JSON | Native `JSON.parse/stringify` | Simple, no schema library needed |
|
|
| CLI | Native `readline` | No external dependencies |
|
|
|
|
### Configuration
|
|
|
|
| File | Purpose |
|
|
| ----------------------- | --------------------------------------------------------- |
|
|
| `config/llm.ts` | Config schema: `provider`, `url`, `model` |
|
|
| `config/prompt.ts` | `buildSystemPrompt(sourceLanguage, pos, targetLanguages)` |
|
|
| `config/batch.ts` | Config schema: `size`, `maxRetries` |
|
|
| `config/constants.ts` | `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
|
|
| `.pipeline-config.json` | Saved runtime config (auto-generated by CLI) |
|
|
|
|
### Key Modules
|
|
|
|
| File | Responsibility |
|
|
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
|
|
| `utils/cli.ts` | Interactive prompts: provider → model → batch size. Saves/loads `.pipeline-config.json`. Returns runtime config. |
|
|
| `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds `EnrichedSense[]`, retry/split logic |
|
|
| `utils/merge-enriched-data.ts` | Merges skeleton with LLM senses, adds `enrichedAt` and `model` |
|
|
| `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure |
|
|
| `utils/pipeline-timer.ts` | Tracks per-word and global metrics; unified throughput for all providers |
|
|
| `utils/llm-adapters/factory.ts` | Creates adapter based on runtime config from CLI |
|
|
| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek |
|
|
| `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API with `systemInstruction` |
|
|
|
|
### Current Model
|
|
|
|
| Property | Value |
|
|
| ------------ | ---------------------------------------- |
|
|
| Model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
|
|
| Size | ~1.0GB |
|
|
| Quantization | Q4_K_M |
|
|
| Server | llama.cpp (`llama-server`) |
|
|
| API | OpenAI-compatible `/v1/chat/completions` |
|
|
|
|
### llama-server Flags: History & Rationale
|
|
|
|
The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen.
|
|
|
|
#### Hardware Constraints
|
|
|
|
| Component | Spec | Implication |
|
|
| --------- | ----------------------------------------- | --------------------------------------------------------------------------- |
|
|
| CPU | i7-6500U (2 physical cores, 4 threads HT) | `-t 2` matches physical cores; HT hurts more than helps |
|
|
| GPU | GTX 950M (Maxwell, 2015) | 32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved |
|
|
| RAM | 8GB (3.95GB usable) | `--mlock` pins model in RAM; system must not swap |
|
|
|
|
#### Flag Evolution
|
|
|
|
| Flag | Value Tried | Result | Why |
|
|
| ----------------- | ----------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
|
|
| `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~47s/word | Baseline. Correct genders. 2.6GB file, tight on VRAM. |
|
|
| `-m` | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken** | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#13-known-issues--dev-notes). |
|
|
| `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~8s/word | Current. Fast but gender accuracy degraded. |
|
|
| `-ngl` | `999` | Keeps | Offload all layers to GPU. Required for any speed. |
|
|
| `-c` | `4096` | Wasteful | 4K context for 300-token dictionary entries wastes ~400MB VRAM. |
|
|
| `-c` | `2048` | Current | Sufficient headroom. Frees VRAM for KV cache. |
|
|
| `-b` / `-ub` | `512` | Keeps | Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs. |
|
|
| `-b` / `-ub` | `2048` | Slower on 950M | Tested briefly. No improvement, possibly worse due to memory pressure. |
|
|
| `-t` | `4` | Slower | Hyperthreading cores hurt llama.cpp performance. |
|
|
| `-t` | `2` | Current | Matches 2 physical cores. |
|
|
| `--threads-batch` | (default) | Risky | Defaults to same as `-t`, but explicit is safer. |
|
|
| `--threads-batch` | `2` | Current | Explicit match to `-t`. |
|
|
| `--flash-attn` | (omitted) | Correct | On Maxwell (compute 5.0), Flash Attention adds overhead. Not used. |
|
|
| `--flash-attn` | (tested) | No gain | Briefly tried with Qwen3.5-4B. No speedup, possibly regression. |
|
|
| `--mlock` | Keeps | Pins model weights in RAM. Prevents OS swapping on memory pressure. |
|
|
| `--prio` | `2` | Keeps | Raises process priority. Marginal on this hardware, harmless. |
|
|
| `--reasoning` | `off` | (Qwen3.5 only) | Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5. |
|
|
|
|
#### Current Command
|
|
|
|
```bash
|
|
./build/bin/llama-server \
|
|
-m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
|
|
-ngl 999 \
|
|
-c 2048 \
|
|
-b 512 \
|
|
-ub 512 \
|
|
-t 2 \
|
|
--threads-batch 2 \
|
|
--host 127.0.0.1 \
|
|
--port 8080 \
|
|
--mlock \
|
|
--prio 2
|
|
```
|
|
|
|
#### What Was Not Tried (And Why)
|
|
|
|
| Flag | Reason Skipped |
|
|
| --------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
|
| `-fa` / `--flash-attn` | Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs. |
|
|
| `--no-mmap` | `--mlock` achieves the same (pin in RAM) without the I/O overhead of disabling mmap. |
|
|
| `-np` / `--parallel` | Not needed. Single sequential pipeline, no concurrent requests. |
|
|
| `--cont-batching` | Default in recent llama.cpp. No need to toggle. |
|
|
| `--defrag-thold` | KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx. |
|
|
| `-ot` / `--override-tensor` | Expert-level. No tensor-specific issues observed. |
|
|
|
|
#### Future Flag Experiments
|
|
|
|
| Experiment | Expected Outcome |
|
|
| ---------------- | -------------------------------------------------------------------------- |
|
|
| `-c 1024` | Further VRAM savings. Risk: insufficient for batching larger prompt sizes. |
|
|
| `-b 256 -ub 256` | Test if smaller batches reduce latency on bandwidth-starved Maxwell. |
|
|
| `--metrics` | Enable Prometheus endpoint for automated performance tracking. |
|
|
|
|
### Performance Baseline
|
|
|
|
| Metric | Qwen3.5-4B | Qwen2.5-1.5B |
|
|
| --------------------- | ---------- | ------------ |
|
|
| Time/word | ~47s | ~8s |
|
|
| Completion tok/s | ~6.4 | ~18.8 |
|
|
| Prompt tok/s | ~81 | ~549 |
|
|
| Avg completion tokens | ~274 | ~132 |
|
|
| Avg prompt tokens | ~327 | ~308 |
|
|
|
|
### Known Limitations (Current)
|
|
|
|
- **Gender accuracy**: Qwen2.5-1.5B systematically defaults to `neuter` for Romance languages. Under evaluation whether larger models fix this.
|
|
- **Single POS**: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments.
|
|
- **Pre-scanning wordlists**: Entire file read into memory before processing. Inefficient for 100k words. See [Refactor Notes](#refactor-notes).
|
|
|
|
---
|
|
|
|
## 5. The LLM Layer
|
|
|
|
### 5.1 Local Model Evaluation
|
|
|
|
Models are evaluated on three criteria in order of priority: **quality** (definitions, examples, translations, gender accuracy), **speed** (completion tokens/sec), **VRAM fit** (must run on GTX 950M 4GB).
|
|
|
|
| Model | Size | VRAM | Speed | Quality | Status |
|
|
| ----------------------- | ------ | ----- | ------------------- | --------------------------- | -------------------------------- |
|
|
| **Qwen3.5-4B Q4_K_M** | 2.6GB | Tight | ~6.4 tok/s | Baseline (assumed good) | Baseline - too slow |
|
|
| **Qwen2.5-1.5B Q4_K_M** | 1.0GB | Easy | ~18.8 tok/s | Gender systematically wrong | Current - fast, needs validation |
|
|
| **Ministral-3B Q4_K_M** | 1.9GB | Fits | - | Broken (tokenizer) | Abandoned |
|
|
| **Qwen2.5-3B Q4_K_M** | 1.9GB | Fits | ~12-15 tok/s (est.) | Unknown | Pending download |
|
|
| **Gemma 4 E2B Q4_K_M** | 3.46GB | No | - | - | Too large for 4GB VRAM |
|
|
| **Gemma 4 E2B IQ2_M** | 2.62GB | Fits | ~8-12 tok/s (est.) | "Low quality" per Google | Not worth it |
|
|
|
|
#### Qwen2.5-1.5B Test Results (3 words)
|
|
|
|
| Word | Definition | Example | Gender (de/it/es/fr) | Verdict |
|
|
| ------ | -------------------------------------------- | ---------------------------------- | --------------------------------- | --------- |
|
|
| time | "A period of duration..." | "The meeting was scheduled..." | neuter/neuter/neuter/neuter | All wrong |
|
|
| year | "A period of time consisting of 365 days..." | "The year 2023 is a leap year." | neuter/neuter/neuter/neuter | All wrong |
|
|
| people | "Individuals who are part of a group." | "The people gathered at the park." | neuter/feminine/feminine/feminine | Mixed |
|
|
|
|
**Pattern:** Defaults to `neuter` when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident.
|
|
|
|
#### Pending Tests
|
|
|
|
- **Qwen2.5-3B**: Same architecture, 2x params. If gender fixes, it was a size problem.
|
|
- **20-word torture suite**: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models.
|
|
|
|
### 5.2 Online API Options
|
|
|
|
Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API (except Gemini, which has a native adapter).
|
|
|
|
| Provider | Model | Input $/1M | Output $/1M | Free Tier | Rate Limit | Est. Cost (100k words) | Est. Time |
|
|
| ------------------- | -------------------- | ---------- | ----------- | ------------- | ---------------- | ---------------------- | ------------------- |
|
|
| **DeepSeek** | V4 Flash | $0.14 | $0.28 | 5M tokens | None | **$0-15** | ~1-2 days |
|
|
| **Gemini** | 2.5 Flash-Lite | $0.10 | $0.40 | 1,500 req/day | 1M TPM | **$0** (free tier) | ~1.5 days (batched) |
|
|
| **Qwen/Alibaba** | Qwen-Turbo | $0.05 | $0.20 | Unknown | 600 RPM | **~$11** | ~1-2 days |
|
|
| **Groq** | Llama 3.1 8B Instant | $0.05 | $0.08 | Yes | High | **~$7** | **~3-4 hours** |
|
|
| **OpenRouter free** | Various | $0 | $0 | 200 req/day | 20 RPM | **$0** | ~10 days |
|
|
| **OpenRouter paid** | DeepSeek V4 Flash | $0.14 | $0.28 | - | Same as provider | **~$16** (+5.5% fee) | ~2-3 days |
|
|
|
|
**Notes:**
|
|
|
|
- Costs assume ~550 tokens/word (300 prompt + 250 completion).
|
|
- Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day.
|
|
- Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable.
|
|
- DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million.
|
|
|
|
### 5.3 Model Selection Criteria
|
|
|
|
Decision flow for 100,000 words:
|
|
|
|
```
|
|
Start
|
|
|
|
|
v
|
|
Run 20-word torture suite
|
|
on Qwen2.5-3B (local)
|
|
|
|
|
|-- Quality good? -----> Use Qwen2.5-3B locally
|
|
| (gender correct) ~20 days, $0
|
|
|
|
|
|-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free)
|
|
|
|
|
|-- Quality good? --> Batch 50, free tier
|
|
| ~1.5 days, $0
|
|
|
|
|
|-- Quality meh? ---> Test Groq or DeepSeek paid
|
|
|
|
|
|-- Speed priority? --> Groq
|
|
| ~$7, 3-4 hours
|
|
|
|
|
|-- Cost priority? ---> DeepSeek
|
|
~$15, 1-2 days
|
|
```
|
|
|
|
**Quality gates:**
|
|
|
|
- > = 90% gender accuracy (de/it/es/fr)
|
|
- 100% JSON parse rate
|
|
- No hallucinated definitions on polysemous words
|
|
- Natural, contextually appropriate example sentences
|
|
- Sensible difficulty classification (CEFR mapping)
|
|
|
|
---
|
|
|
|
## 6. The Gender Problem & Kaikki Integration
|
|
|
|
### The Problem
|
|
|
|
Grammatical gender is embedded in the `translations` object of each sense:
|
|
|
|
```json
|
|
"translations": {
|
|
"de": [{"word": "Haus", "gender": "neuter"}],
|
|
"it": [{"word": "casa", "gender": "feminine"}],
|
|
"es": [{"word": "casa", "gender": "feminine"}],
|
|
"fr": [{"word": "maison", "gender": "feminine"}]
|
|
}
|
|
```
|
|
|
|
Early testing with **Qwen2.5-1.5B** showed systematic failure: the model defaults to `neuter` for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all — only masculine and feminine.
|
|
|
|
Whether this is a **model size issue** (1.5B too small to retain gender facts) or a **training data gap** (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation.
|
|
|
|
### Two Approaches Under Consideration
|
|
|
|
| Approach | How It Works | Pros | Cons |
|
|
| -------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
|
|
| **A. LLM-only** | Trust the model to generate correct gender | Zero additional complexity | Unreliable at small sizes; hallucination risk |
|
|
| **B. LLM + Kaikki lookup** | LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump | 100% deterministic; decouples gender from model quality | Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries |
|
|
|
|
### Kaikki Data
|
|
|
|
Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender.
|
|
|
|
| Language | Kaikki File | Coverage |
|
|
| -------- | ------------------------------------- | -------- |
|
|
| German | `kaikki.org-dictionary-German.jsonl` | High |
|
|
| Italian | `kaikki.org-dictionary-Italian.jsonl` | High |
|
|
| Spanish | `kaikki.org-dictionary-Spanish.jsonl` | High |
|
|
| French | `kaikki.org-dictionary-French.jsonl` | High |
|
|
|
|
Lookup logic: match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`.
|
|
|
|
### Decision Pending
|
|
|
|
- If **Qwen2.5-3B** or an online model produces >=90% accurate gender: **Approach A**, no Kaikki needed.
|
|
- If all tested models fail gender: **Approach B**, implement Kaikki lookup as a post-processing step after LLM enrichment.
|
|
|
|
No implementation work started until the 20-word torture suite resolves this.
|
|
|
|
---
|
|
|
|
## 7. Batching Strategy
|
|
|
|
### Why Batching is Necessary
|
|
|
|
At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost.
|
|
|
|
### Configurable Batch Size
|
|
|
|
Batch size is selected interactively at runtime via the CLI. The schema lives in `config/batch.ts`:
|
|
|
|
```typescript
|
|
export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const;
|
|
```
|
|
|
|
The CLI presents preset options (1, 5, 10, 20, 50) with provider-specific recommendations.
|
|
|
|
### Prompt Structure
|
|
|
|
**Single word:**
|
|
|
|
```
|
|
["house"]
|
|
```
|
|
|
|
**Batch of 4:**
|
|
|
|
```
|
|
["house", "car", "tree", "water"]
|
|
```
|
|
|
|
LLM returns a JSON object with word keys:
|
|
|
|
```json
|
|
{
|
|
"house": [ { "sense": "...", "example": "...", ... } ],
|
|
"car": [ { ... } ],
|
|
"tree": [ { ... } ],
|
|
"water": [ { ... } ]
|
|
}
|
|
```
|
|
|
|
### Retry & Split Strategy
|
|
|
|
If a batch fails (bad JSON, missing key, etc.):
|
|
|
|
```
|
|
Batch of 4 fails (3 retries exhausted)
|
|
|
|
|
v
|
|
Split into 2 batches of 2
|
|
|
|
|
v
|
|
If a batch of 2 fails (3 retries), split into 2 batches of 1
|
|
|
|
|
v
|
|
If a single word fails (3 retries), log and skip
|
|
```
|
|
|
|
This gives resilience without losing the speed benefit of large batches. The `maxRetries` config controls how many attempts are made before splitting.
|
|
|
|
### Expected Impact by Environment
|
|
|
|
| Environment | Batch Size | Expected Speedup | Notes |
|
|
| ------------------ | ---------- | ---------------- | -------------------------------------------------------- |
|
|
| Local GTX 950M | 3 | ~1.05-1.15x | Memory bandwidth limited. KV cache pressure on 4GB VRAM. |
|
|
| Local GTX 950M | 5 | ~1.10-1.20x | Sweet spot for this hardware. |
|
|
| Local GTX 950M | 15 | Risky | May OOM. Test carefully. |
|
|
| Local GTX 950M | 50 | Unlikely | VRAM insufficient. |
|
|
| Cloud API (Gemini) | 50 | 5x fewer calls | Unlocks free tier viability. |
|
|
| Cloud API (Groq) | 50 | 5x fewer calls | Minimal gain - already fast. |
|
|
|
|
### Batching is Non-Negotiable
|
|
|
|
Regardless of local vs cloud, batching is required for 100k words:
|
|
|
|
- **Local**: Better GPU utilization, amortized prompt processing
|
|
- **Cloud**: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts
|
|
|
|
---
|
|
|
|
## 8. Hardware Constraints
|
|
|
|
### Current Machine
|
|
|
|
| Component | Spec |
|
|
| --------- | ------------------------------------------------------------------- |
|
|
| OS | Debian GNU/Linux 13 (trixie) x86_64 |
|
|
| CPU | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz |
|
|
| GPU | NVIDIA GeForce GTX 950M (Maxwell, 2015) |
|
|
| GPU VRAM | 4GB |
|
|
| RAM | 8GB (3.95GB usable at idle) |
|
|
| Disk | 102GB ext4 (~63GB used) |
|
|
|
|
### What Fits in 4GB VRAM
|
|
|
|
| Model | File Size | KV Cache (2048 ctx) | Total VRAM | Fits? |
|
|
| ------------------- | --------- | ------------------- | ---------- | ------------------ |
|
|
| Qwen2.5-1.5B Q4_K_M | ~1.0GB | ~0.5GB | ~1.5GB | Yes |
|
|
| Qwen2.5-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes |
|
|
| Ministral-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes (but broken) |
|
|
| Qwen3.5-4B Q4_K_M | 2.6GB | ~1.0GB | ~3.6GB | Tight |
|
|
| Gemma 4 E2B Q4_K_M | 3.46GB | ~1.2GB | ~4.7GB | No |
|
|
| Gemma 4 E2B IQ2_M | 2.62GB | ~1.0GB | ~3.6GB | Maybe, low quality |
|
|
|
|
### GPU Rental Alternatives
|
|
|
|
If local hardware becomes the bottleneck:
|
|
|
|
| Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost |
|
|
| -------- | -------- | ---- | ----------- | ------------------- | ---------- |
|
|
| Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | **~$2-5** |
|
|
| RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | **~$4-6** |
|
|
| Vast.ai | RTX 3090 | 24GB | ~$0.20-0.40 | ~8-10 hours | **~$2-4** |
|
|
|
|
With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day.
|
|
|
|
---
|
|
|
|
## 9. Testing & Quality Assurance
|
|
|
|
### 20-Word Torture Suite
|
|
|
|
Planned test set covering edge cases:
|
|
|
|
| Category | Words | Why |
|
|
| -------------- | ------------------------------------------------------ | ----------------------------------- |
|
|
| Easy concrete | `house`, `water`, `book` | Baseline |
|
|
| Easy abstract | `time`, `love`, `hope` | Abstract nouns harder to define |
|
|
| Polysemous | `bank`, `run`, `light` | Multiple senses test disambiguation |
|
|
| Hard/technical | `democracy`, `photosynthesis`, `entropy` | Complex definitions |
|
|
| False friends | `actual` (en/es), `sensible` (en/fr), `fabric` (en/de) | Cross-lingual traps |
|
|
|
|
### Evaluation Criteria
|
|
|
|
For each word and each candidate model:
|
|
|
|
| Criterion | Pass Threshold |
|
|
| ----------------------------- | ------------------------------------------------- |
|
|
| Definition accuracy | Factually correct, max 15 words, student-friendly |
|
|
| Example quality | Natural sentence, word used correctly in context |
|
|
| Translation correctness | Correct word sense match |
|
|
| Gender accuracy (de/it/es/fr) | >=90% correct |
|
|
| Difficulty classification | Sensible per CEFR mapping |
|
|
| JSON reliability | 100% parse rate, valid schema |
|
|
|
|
### Verification
|
|
|
|
`verify-enriched-file.ts` checks:
|
|
|
|
- Required top-level fields: `word`, `language`, `pos`, `senses`
|
|
- Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard}
|
|
- Each translation: `word` (string), `gender` in {masculine, feminine, neuter, null}
|
|
|
|
Additionally, `validateSense()` in `enrich-word.ts` catches malformed senses **before** file writes, triggering retry/split instead of writing bad data.
|
|
|
|
---
|
|
|
|
## 10. Interactive CLI
|
|
|
|
### Overview
|
|
|
|
The pipeline includes an interactive CLI (`utils/cli.ts`) that asks the user to select provider, model, and batch size on each run. No editing of `config/llm.ts` or `config/batch.ts` required.
|
|
|
|
### Flow
|
|
|
|
```
|
|
$ npx tsx pipeline.ts
|
|
|
|
🌐 Lila Data Pipeline
|
|
─────────────────────
|
|
|
|
[1] Use last config: online → gemini → gemini-2.5-flash-lite → batch 50
|
|
[2] Configure new run
|
|
|
|
> 2
|
|
|
|
Provider type:
|
|
[1] Local (llama.cpp)
|
|
[2] Online API
|
|
|
|
> 2
|
|
|
|
Online provider:
|
|
[1] Gemini
|
|
[2] DeepSeek
|
|
[3] OpenRouter
|
|
[4] Groq
|
|
|
|
> 1
|
|
|
|
Model:
|
|
[1] gemini-2.5-flash-lite (recommended for cost)
|
|
[2] gemini-2.5-pro (recommended for quality)
|
|
|
|
> 1
|
|
|
|
Batch size:
|
|
[1] 1 (safest, slowest)
|
|
[2] 5 (recommended for local)
|
|
[3] 10
|
|
[4] 20
|
|
[5] 50 (recommended for Gemini)
|
|
[6] Custom
|
|
|
|
> 5
|
|
|
|
✅ Configuration:
|
|
Provider: gemini
|
|
Model: gemini-2.5-flash-lite
|
|
API key: GEMINI_API_KEY found in environment ✓
|
|
Batch size: 50
|
|
|
|
Start pipeline with these settings? [Y/n]
|
|
> Y
|
|
```
|
|
|
|
### Saved Config
|
|
|
|
On first run, after confirming, write to `.pipeline-config.json`:
|
|
|
|
```json
|
|
{
|
|
"provider": "gemini",
|
|
"model": "gemini-2.5-flash-lite",
|
|
"batchSize": 50,
|
|
"lastRun": "2026-07-06T13:54:00Z"
|
|
}
|
|
```
|
|
|
|
Next run shows `[1] Use last config` at the top.
|
|
|
|
### API Key Rules
|
|
|
|
- **Never prompt for keys.** Check `process.env` for the provider's key.
|
|
- **If missing:** Print which env var is needed, then exit.
|
|
- **Supported env vars:** `GEMINI_API_KEY`, `DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`
|
|
|
|
### Batch Size Recommendations
|
|
|
|
| Provider | Recommended | Rationale |
|
|
| ------------------ | ----------- | ---------------------------------- |
|
|
| `local` (GTX 950M) | 5 | VRAM-limited, KV cache pressure |
|
|
| `local` (RTX 4090) | 20 | Fast, more VRAM |
|
|
| `gemini` | 50 | Free tier: 1,500 req/day |
|
|
| `deepseek` | 20 | 5M free tokens, balance speed/cost |
|
|
| `groq` | 50 | Very fast, rate limits generous |
|
|
| `openrouter` | 10 | 200 req/day free tier |
|
|
|
|
### Implementation
|
|
|
|
The CLI is implemented in `utils/cli.ts` using Node.js native `readline` module. No external dependencies.
|
|
|
|
```typescript
|
|
// utils/cli.ts
|
|
import readline from "readline";
|
|
|
|
export async function runCli(): Promise<{
|
|
provider: string;
|
|
model: string;
|
|
url: string;
|
|
batchSize: number;
|
|
}> {
|
|
// ... interactive prompts ...
|
|
}
|
|
```
|
|
|
|
`pipeline.ts` imports and calls `runCli()` at startup:
|
|
|
|
```typescript
|
|
import { runCli } from "./utils/cli.js";
|
|
|
|
async function main() {
|
|
const config = await runCli();
|
|
// Use config.provider, config.model, etc.
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 11. Future Extensions & Roadmap
|
|
|
|
### Near-Term (Next 2-4 Weeks)
|
|
|
|
| Item | Status | Notes |
|
|
| ---------------------- | ------------ | ----------------------------------------------------------------- |
|
|
| Configurable batching | **Complete** | `config/batch.ts` with `size` and `maxRetries` |
|
|
| Retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch |
|
|
| Honest timing metrics | **Complete** | Unified throughput for all providers, detailed only for local |
|
|
| Auto-target languages | **Complete** | `buildSystemPrompt()` excludes source from targets |
|
|
| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes |
|
|
| Interactive CLI | **Complete** | `utils/cli.ts` — provider/model/batch selection with saved config |
|
|
| 20-word torture suite | Pending | Decides gender approach and model selection |
|
|
| Qwen2.5-3B evaluation | Pending | Download and test |
|
|
| Online API testing | Pending | Gemini free tier, DeepSeek, Groq |
|
|
|
|
### Medium-Term (1-3 Months)
|
|
|
|
| Item | Notes |
|
|
| ---------------------------- | ------------------------------------------------------ |
|
|
| Multi-POS support | Verbs, adjectives, adverbs need prompt variants |
|
|
| Multi-language source | German -> French, Italian -> Spanish, etc. |
|
|
| Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously |
|
|
| Incremental enrichment | Only process new/changed words in a wordlist |
|
|
| Model auto-switching | Fallback to online API if local server fails mid-run |
|
|
|
|
### Long-Term (3-6 Months)
|
|
|
|
| Item | Notes |
|
|
| ------------------------ | ---------------------------------------------------------------- |
|
|
| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA |
|
|
| GPU rental integration | Script to spin up Vast.ai/RunPod, run pipeline, download results |
|
|
| Quality regression tests | Run torture suite on every model change |
|
|
| Community contributions | Open-source the pipeline for other language learners |
|
|
|
|
---
|
|
|
|
## 12. Decisions Log
|
|
|
|
| Date | Decision | Context | Rationale |
|
|
| ---------- | ----------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
|
| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app |
|
|
| 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying |
|
|
| 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas |
|
|
| 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys |
|
|
| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies |
|
|
| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference |
|
|
| 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM |
|
|
| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance |
|
|
| 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation |
|
|
| 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality |
|
|
| 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion |
|
|
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations |
|
|
| 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 |
|
|
| 2026-07-06 | Honest timing metrics | Cloud providers don't expose prompt/completion breakdown | Unified `totalTimeMs` for all; detailed breakdown only when available |
|
|
| 2026-07-06 | Auto-target languages | Prompt hardcoded English -> de/it/es/fr | `ALL_LANGUAGES` minus source = targets; works for any source language |
|
|
| 2026-07-06 | Validate LLM responses before write | Bad data was written then warned about | `validateSense()` catches malformed responses early, triggers retry |
|
|
| 2026-07-06 | Interactive CLI | Editing `config/llm.ts` to switch providers is error-prone | `utils/cli.ts` with native `readline`; no external dependencies; saves config |
|
|
|
|
---
|
|
|
|
## 13. Known Issues & Dev Notes
|
|
|
|
### glossa-web (Frontend)
|
|
|
|
| Issue | Details |
|
|
| ------------------------ | ----------------------------------------------------------------------------------------- |
|
|
| No healthcheck | Vite dev server has no health endpoint. Docker `HEALTHCHECK` cannot verify running state. |
|
|
| Valkey memory overcommit | Harmless warning in dev: `vm.overcommit_memory = 1` recommended before production. |
|
|
|
|
### Data Pipeline
|
|
|
|
| Issue | Details | Severity |
|
|
| --------------------------------- | --------------------------------------------------------------------------- | ----------------------------------- |
|
|
| Ministral-3B tokenizer mismatch | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. | Blocker - abandoned |
|
|
| Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages. | Under evaluation |
|
|
| Pre-scanning wordlists | Entire file read into memory before processing. Inefficient for 100k words. | Medium - streaming refactor planned |
|
|
| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes. | Known limitation |
|
|
|
|
### Hardware
|
|
|
|
| Issue | Details |
|
|
| --------------------- | ----------------------------------------------------- |
|
|
| GTX 950M VRAM ceiling | 4GB hard limit. Models >3GB risk OOM. |
|
|
| Maxwell GPU aging | No Flash Attention support, bandwidth-starved. |
|
|
| Laptop thermals | Cannot run 24/7 for weeks. Batch processing required. |
|
|
|
|
### Refactor Notes
|
|
|
|
| Note | File | Context |
|
|
| --------------------- | ------------- | -------------------------------------------------------------------------- |
|
|
| Streaming vs pre-scan | `pipeline.ts` | For 100k words, stream and batch on-the-fly instead of reading entire file |
|
|
|
|
---
|
|
|
|
## 14. How to Run
|
|
|
|
### Prerequisites
|
|
|
|
- Node.js + npm
|
|
- `tsx` installed globally: `npm install -g tsx`
|
|
- llama.cpp built from source (for local mode)
|
|
- GGUF model downloaded to `~/Downloads/llama.cpp/models/` (for local mode)
|
|
- API keys set as environment variables (for cloud mode):
|
|
```bash
|
|
export DEEPSEEK_API_KEY="sk-..."
|
|
export GEMINI_API_KEY="..."
|
|
export OPENROUTER_API_KEY="..."
|
|
```
|
|
|
|
### Start the LLM Server (Local Mode)
|
|
|
|
```bash
|
|
cd ~/Downloads/llama.cpp
|
|
|
|
./build/bin/llama-server \
|
|
-m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
|
|
-ngl 999 \
|
|
-c 2048 \
|
|
-b 512 \
|
|
-ub 512 \
|
|
-t 2 \
|
|
--threads-batch 2 \
|
|
--host 127.0.0.1 \
|
|
--port 8080 \
|
|
--mlock \
|
|
--prio 2
|
|
```
|
|
|
|
### Run the Pipeline
|
|
|
|
```bash
|
|
cd /path/to/data-pipeline
|
|
npx tsx pipeline.ts
|
|
```
|
|
|
|
Follow the interactive prompts to select provider, model, and batch size. Or select "Use last config" to reuse previous settings.
|
|
|
|
### Run with Last Config
|
|
|
|
```bash
|
|
cd /path/to/data-pipeline
|
|
npx tsx pipeline.ts
|
|
# Select [1] Use last config
|
|
```
|
|
|
|
### Expected Output
|
|
|
|
```
|
|
🌐 Lila Data Pipeline
|
|
─────────────────────
|
|
|
|
[1] Use last config: local → qwen2.5-1.5b → batch 5
|
|
[2] Configure new run
|
|
|
|
> 1
|
|
|
|
🟢 Local AI engine is connected and ready for inference!
|
|
|
|
step 1: scanning the source files...
|
|
✅ Scan complete! Found 1 wordlist(s):
|
|
• ENGLISH (nouns)
|
|
|
|
step 2: creating necessary output folders...
|
|
✅ All required output directories have been verified and created successfully.
|
|
|
|
step 3: verifying local AI engine status...
|
|
🟢 Local AI engine is connected and ready for inference!
|
|
|
|
step 4: looping through the wordlists...
|
|
|
|
Reading list: [ENGLISH] -> [NOUNS]
|
|
|
|
Batch 1/1: [house, car, tree, water]
|
|
[1/4] (0 failed) Enriched and saved: house.json
|
|
[2/4] (0 failed) Enriched and saved: car.json
|
|
[3/4] (0 failed) Enriched and saved: tree.json
|
|
[4/4] (0 failed) Enriched and saved: water.json
|
|
⏱️ Word took 8.2s
|
|
|
|
⏱️ Pipeline Summary
|
|
Duration: 32.8s
|
|
Processed: 4
|
|
Skipped: 0
|
|
Failed: 0
|
|
Total: 4
|
|
Throughput: 0.12 words/sec
|
|
|
|
🤖 LLM Metrics
|
|
Calls: 1
|
|
Avg prompt tokens: 312
|
|
Avg completion tokens: 524
|
|
Avg total tokens: 836
|
|
Avg total request time: 32800ms
|
|
Avg throughput: 25.5 tok/s
|
|
|
|
[Local breakdown]
|
|
Avg prompt speed: 548.2 tok/s
|
|
Avg completion speed: 18.8 tok/s
|
|
|
|
Global data pipeline run completed successfully.
|
|
```
|
|
|
|
---
|
|
|
|
## 15. Roadmap
|
|
|
|
### Phase 1: Batching (Complete)
|
|
|
|
| Task | Status | Notes |
|
|
| ------------------------------------- | ------------ | ------------------------------------------------------------------------------------ |
|
|
| Implement configurable batch size | **Complete** | `config/batch.ts` with `size` and `maxRetries` |
|
|
| Implement retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch |
|
|
| Honest timing metrics | **Complete** | Unified throughput for all providers |
|
|
| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes |
|
|
| Verify batching doesn't break quality | Pending | Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output. |
|
|
| Measure speedup vs batch size | Pending | Track throughput at 1, 5, 15 on local hardware. |
|
|
|
|
**Goal:** Unlock 5-15x speedup on local, unlock free online API tiers.
|
|
|
|
---
|
|
|
|
### Phase 2: Interactive CLI (Complete)
|
|
|
|
| Task | Status | Notes |
|
|
| --------------------- | ------------ | ------------------------------------------ |
|
|
| Design prompt flow | **Complete** | Provider → model → batch size → confirm |
|
|
| Implement CLI module | **Complete** | `utils/cli.ts` with native `readline` |
|
|
| Save/load config | **Complete** | `.pipeline-config.json` |
|
|
| Wire into pipeline.ts | **Complete** | Replaces static config with runtime config |
|
|
|
|
**Goal:** No editing of TypeScript files to switch providers.
|
|
|
|
---
|
|
|
|
### Phase 3: Model Selection
|
|
|
|
| Task | Status | Notes |
|
|
| ------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------- |
|
|
| Download Qwen2.5-3B Q4_K_M | Pending | ~1.9GB, fits in 4GB VRAM. |
|
|
| Run torture suite on Qwen2.5-3B (batched) | Pending | Check if gender accuracy improves with 2x params. |
|
|
| Test Gemini 2.5 Flash-Lite free tier (batched) | Pending | 1,500 req/day x 50 words = 75k words/day. Zero cost. |
|
|
| Test DeepSeek V4 Flash free tier (batched) | Pending | 5M tokens free. ~9k words. |
|
|
| Test Groq Llama 3.1 8B (batched, paid if needed) | Pending | Fastest option. ~$7 for 100k words. |
|
|
| Decide: local vs online, which model | Pending | Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost. |
|
|
|
|
**Goal:** Pick the model and provider for the 100k word run.
|
|
|
|
---
|
|
|
|
### Phase 4: Scale
|
|
|
|
| Task | Status | Notes |
|
|
| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
|
|
| Run 100k word pipeline | Pending | Estimated time depends on Phase 3 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). |
|
|
| Spot-check output quality | Pending | Random sample of 100 entries. |
|
|
| Fix gender if needed | Pending | Kaikki lookup post-processing if LLM gender remains unreliable. |
|
|
| Handle failures & retries | Pending | Exponential backoff, split-and-retry for batch failures. |
|
|
|
|
**Goal:** Complete 100k word dataset.
|
|
|
|
---
|
|
|
|
### Phase 5: Extend
|
|
|
|
| Task | Status | Notes |
|
|
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------- |
|
|
| Multi-POS support | Pending | Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.). |
|
|
| Multi-language source | Pending | German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo. |
|
|
| Parallel wordlist processing | Pending | Run `english/nouns` and `english/verbs` simultaneously. |
|
|
| Incremental enrichment | Pending | Only process new/changed words in a wordlist. |
|
|
| GPU rental integration | Pending | Script to spin up Vast.ai/RunPod, run pipeline, download results. |
|
|
| Quality regression tests | Pending | Run torture suite on every model change. |
|
|
|
|
**Goal:** Generalize pipeline for any language direction and POS.
|
|
|
|
---
|
|
|
|
### Backlog (Unscheduled)
|
|
|
|
| Task | Context |
|
|
| -------------------------------- | ------------------------------------------------------------------------------------------- |
|
|
| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. |
|
|
| Model auto-switching | Fallback to online API if local server fails mid-run. |
|
|
| Community open-source | Clean up, document, publish for other language learners. |
|
|
| Prometheus metrics | `--metrics` flag on llama-server for automated performance tracking. |
|
|
| `-c 1024` / `-b 256` experiments | Further VRAM optimization on GTX 950M. Low priority if moving to cloud. |
|
|
| Streaming wordlist processing | Read file line-by-line and batch on-the-fly. Eliminates pre-scan memory usage. |
|