diff --git a/documentation/DATA_PIPELINE.md b/documentation/DATA_PIPELINE.md index 66a9dbc..b7a1bf6 100644 --- a/documentation/DATA_PIPELINE.md +++ b/documentation/DATA_PIPELINE.md @@ -1,32 +1,28 @@ # Lila Data Pipeline — Technical Documentation -> Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer. -> Last updated: 2026-07-06 - ---- +Multilingual dictionary enrichment pipeline for the Lila vocabulary trainer. +Last updated: 2026-07-18 ## Table of Contents -1. [Executive Summary](#1-executive-summary) -2. [Problem & Context](#2-problem--context) -3. [Architecture Overview](#3-architecture-overview) -4. [Current Implementation](#4-current-implementation) -5. [The LLM Layer](#5-the-llm-layer) - - 5.1 [Local Model Evaluation](#51-local-model-evaluation) - - 5.2 [Online API Options](#52-online-api-options) - - 5.3 [Model Selection Criteria](#53-model-selection-criteria) -6. [The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration) -7. [Batching Strategy](#7-batching-strategy) -8. [Hardware Constraints](#8-hardware-constraints) -9. [Testing & Quality Assurance](#9-testing--quality-assurance) -10. [Interactive CLI](#10-interactive-cli) -11. [Future Extensions & Roadmap](#11-future-extensions--roadmap) -12. [Decisions Log](#12-decisions-log) -13. [Known Issues & Dev Notes](#13-known-issues--dev-notes) -14. [How to Run](#14-how-to-run) -15. [Roadmap](#15-roadmap) - ---- +- [1. Executive Summary](#1-executive-summary) +- [2. Problem & Context](#2-problem--context) +- [3. Architecture Overview](#3-architecture-overview) +- [4. Current Implementation](#4-current-implementation) +- [5. The LLM Layer](#5-the-llm-layer) + - 5.1 [Local Model Evaluation](#51-local-model-evaluation) + - 5.2 [Online API Options](#52-online-api-options) + - 5.3 [Model Selection Criteria](#53-model-selection-criteria) +- [6. The Gender Problem & Kaikki Integration](#6-the-gender-problem--kaikki-integration) +- [7. Batching Strategy](#7-batching-strategy) +- [8. Hardware Constraints](#8-hardware-constraints) +- [9. Testing & Quality Assurance](#9-testing--quality-assurance) +- [10. Interactive CLI](#10-interactive-cli) +- [11. Future Extensions & Roadmap](#11-future-extensions--roadmap) +- [12. Decisions Log](#12-decisions-log) +- [13. Known Issues & Dev Notes](#13-known-issues--dev-notes) +- [14. How to Run](#14-how-to-run) +- [15. Roadmap](#15-roadmap) ## Quick Reference @@ -40,7 +36,7 @@ | Shared constants | `config/constants.ts` | | Output schema | `utils/merge-enriched-data.ts` | | LLM adapters | `utils/llm-adapters/` | -| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | +| **Current model** | **`gemma-4-E2B_q4_0-it.gguf`** | | Target scale | 100,000+ words | --- @@ -49,56 +45,41 @@ The Lila Data Pipeline is a TypeScript-based batch processing system that enriches raw word lists into structured multilingual dictionary entries for the Lila vocabulary trainer. It takes a source wordlist (e.g., English nouns) and, for each word, generates: -- One or more **senses** (definitions) -- A **natural example sentence** per sense -- A **CEFR-based difficulty level** (`easy` / `medium` / `hard`) -- **Translations** into all target languages except the source, each with grammatical **gender** +- One or more senses (definitions) +- A natural example sentence per sense +- A CEFR-based difficulty level (`easy` / `medium` / `hard`) +- Translations into all target languages except the source (as raw strings) -The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an **interactive CLI** for provider selection. +The pipeline is designed to scale to 100,000+ words across multiple languages and parts of speech. It supports both local inference (`llama.cpp`) and cloud providers via a pluggable adapter pattern. ### Key Design Principles -| Principle | Rationale | -| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | -| **Quality first** | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary. | -| **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. | -| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. | -| **Configurable batching** | Batch size is selected interactively at runtime. The pipeline adapts without code changes. | -| **Provider-agnostic** | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface. | -| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns. | +| Principle | Rationale | +| ------------------------------- | ------------------------------------------------------------------------------------------------------ | +| **Quality first** | Definitions, examples, and translations must be accurate. Speed and cost are secondary. | +| **Local-first, cloud-fallback** | Local LLMs are the default for cost control and data privacy. | +| **Deterministic Grammar** | Grammatical gender is decoupled from the LLM and resolved via Kaikki Wiktionary dumps. | +| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. | +| **Configurable batching** | Batch size is selected interactively at runtime. The pipeline adapts without code changes. | +| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only. | -### Open Question: Gender Accuracy +### Resolved: Gender Accuracy -Grammatical gender is currently generated by the LLM as part of the translation object. Early testing showed that **Qwen2.5-1.5B systematically defaults to `neuter`** for languages that do not have neuter grammatical gender (Italian, Spanish, French). Whether this is a **model size issue** (fixable by moving to 3B+) or a **training data issue** (requiring an external lookup) is unresolved. +Grammatical gender is **no longer generated by the LLM**. Comprehensive testing across 10 models (2026-07-18) confirmed that small local models systematically hallucinate or default to `neuter` for Romance languages. +**Decision:** The LLM only outputs translation strings. A deterministic post-processing step looks up the exact grammatical gender from Kaikki Wiktionary dumps. This guarantees 100% gender accuracy and allows us to use smaller, faster, and highly nuanced local models. -**Options under evaluation:** +### Current Status (2026-07-18) -- Larger local models (Qwen2.5-3B, Qwen3.5-1.7B) -- Online models with stronger multilingual training (Gemini, DeepSeek) -- Post-processing lookup via **Kaikki Wiktionary dumps** as a fallback or replacement - -No decision made. Gender handling will be determined by the 20-word quality torture suite. - -### Current Status (2026-07-06) - -- Core pipeline: scanning, enrichment, merging, verification, writing -- Local LLM integration via llama.cpp server (OpenAI-compatible API) -- **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/` -- **Interactive CLI**: provider/model/batch selection with saved config -- **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1) -- **Honest timing**: unified throughput for all providers, detailed breakdown only for local -- **Auto-target languages**: prompt dynamically excludes source language from targets -- **Schema validation**: validates LLM response structure before file writes -- **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending) -- **Pending:** 20-word quality torture suite (will decide gender approach) -- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) +- **Core pipeline:** Complete (scanning, enrichment, merging, verification, writing). +- **Local LLM:** Gemma 4 E2B selected as production model after exhaustive 10-model evaluation. `llama.cpp` server optimized with KV-cache quantization. +- **Batching:** 20-word batches validated for local hardware. +- **Gender:** Decoupled from LLM; Kaikki lookup architecture confirmed. +- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) as speed fallback. ### One-Line Architecture -``` -source wordlists -> Interactive CLI -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json - | - [gender: LLM-generated, accuracy TBD] +```text +source wordlists -> Interactive CLI -> LLM adapter -> merge senses -> Kaikki Gender Lookup -> verify schema -> write .json ``` ### Files at a Glance @@ -113,7 +94,7 @@ source wordlists -> Interactive CLI -> LLM adapter (local or cloud) -> merge sen | `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` | | `utils/enrich-word.ts` | Calls LLM via adapter, parses response, validates senses, builds `EnrichedSense[]`, retry/split logic | | `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON | -| `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) | +| `utils/verify-enriched-file.ts` | Schema validation (required fields, types) | | `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers | | `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory | | `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files | @@ -147,13 +128,13 @@ source wordlists -> Interactive CLI -> LLM adapter (local or cloud) -> merge sen Existing multilingual dictionaries and translation APIs provide raw word-to-word mappings. They do not provide the structured, pedagogical data needed for a vocabulary trainer: -| What Exists | What is Missing | Why It Matters | -| ------------------------ | -------------------------------------- | ----------------------------------------------------------------- | -| Word + translation | **Student-friendly definition** | Learners need explanations, not just equivalents | -| Static difficulty labels | **CEFR-calibrated difficulty** | "Easy" vs "hard" must map to learner proficiency levels | -| Isolated translations | **Natural example sentences** | Context is how vocabulary is actually acquired | -| Raw gender data | **Gender integrated with translation** | Flashcards must show gender immediately, not as a separate lookup | -| Bulk word lists | **Per-word structured JSON** | The trainer consumes one file per word for fast random access | +| What Exists | What is Missing | Why It Matters | +| ------------------------ | ---------------------------------- | ------------------------------------------------------------- | +| Word + translation | Student-friendly definition | Learners need explanations, not just equivalents | +| Static difficulty labels | CEFR-calibrated difficulty | "Easy" vs "hard" must map to learner proficiency levels | +| Isolated translations | Natural example sentences | Context is how vocabulary is actually acquired | +| Raw gender data | Gender integrated with translation | Flashcards must show gender immediately (resolved via Kaikki) | +| Bulk word lists | Per-word structured JSON | The trainer consumes one file per word for fast random access | ### The Target User @@ -161,69 +142,53 @@ A language learner using the Lila vocabulary trainer. They see a word, its defin ### Why Not Use Existing Dictionaries? -- **Wiktionary**: Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions -- **Kaikki (Wiktionary dump)**: Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation -- **Google Translate / DeepL**: No definitions, no examples, no difficulty, no structured output -- **Existing language learning apps**: Closed data, no export, no control over content +- **Wiktionary:** Rich data, but unstructured, inconsistent formatting, no CEFR levels, no student-friendly definitions +- **Kaikki (Wiktionary dump):** Structured JSON, excellent for gender/translation lookup, but definitions are often technical, no difficulty classification, no example curation +- **Google Translate / DeepL:** No definitions, no examples, no difficulty, no structured output +- **Existing language learning apps:** Closed data, no export, no control over content -The LLM fills the gap: it generates **pedagogical content** (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale. +The LLM fills the gap: it generates pedagogical content (student-friendly definitions, natural examples, difficulty classification) that no existing database provides at scale. ### Language Direction -The pipeline is **direction-agnostic**. A wordlist is defined by: +The pipeline is direction-agnostic. A wordlist is defined by: -- **Source language**: the language of the input words -- **Target languages**: all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source) +- **Source language:** the language of the input words +- **Target languages:** all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source) -Current focus: **English -> German/Italian/Spanish/French** - -Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist. Target languages are computed automatically. +Current focus: English -> German/Italian/Spanish/French. Planned directions include German -> French, Italian -> Spanish, etc. ### Why 100,000+ Words? -- **Coverage**: A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels with room for curation. -- **Languages**: English (source) -> German, Italian, Spanish, French (targets). -- **Parts of speech**: Nouns (current), verbs, adjectives, adverbs. Each POS has different enrichment needs (verb conjugations, adjective agreement, etc.). - -### The Data Flow - -``` -Source files LLM enrichment Final JSON -(one word per line) (definitions, (one per word, - examples, self-contained) -english/nouns difficulty, -english/verbs translations) time.json -italian/nouns year.json -... people.json -``` +- **Coverage:** A learner needs ~10,000 words for B2 proficiency. The pipeline targets 100k to cover multiple languages, POS categories, and difficulty levels. +- **Languages:** English (source) -> German, Italian, Spanish, French (targets). +- **Parts of speech:** Nouns (current), verbs, adjectives, adverbs. ### The Quality Challenge -Generating 100,000 entries with an LLM introduces risks: - -| Risk | Mitigation | -| ------------------------------ | -------------------------------------------------------------- | -| Hallucinated definitions | Low temperature (0.1), strict system prompt, schema validation | -| Incorrect grammatical gender | Under evaluation: larger models or external lookup | -| Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking | -| JSON parse failures | Retry + split logic, schema validation, cleanup on failure | -| Model drift (online APIs) | Version pinning, local fallback | -| Provider downtime | Adapter pattern allows hot-swapping providers | -| Malformed LLM responses | `validateSense()` catches bad data before file writes | +| Risk | Mitigation | +| ------------------------------- | -------------------------------------------------------------- | +| Hallucinated definitions | Low temperature (0.1), strict system prompt, schema validation | +| Incorrect grammatical gender | **Resolved:** Decoupled from LLM; Kaikki Wiktionary lookup | +| POS bleed (verb defs for nouns) | Explicit negative constraint in system prompt | +| Inconsistent difficulty levels | Explicit CEFR mapping in prompt, spot-checking | +| JSON parse failures | Retry + split logic, schema validation, cleanup on failure | +| Model drift (online APIs) | Version pinning, local fallback | +| Provider downtime | Adapter pattern allows hot-swapping providers | ### Why TypeScript + Node? -- **Familiarity**: Existing project uses TypeScript (frontend in TanStack Router + React) -- **Ecosystem**: `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP -- **Portability**: Runs on the same Debian laptop as the llama.cpp server -- **No build complexity**: `tsx` for direct execution, no bundler needed +- **Familiarity:** Existing project uses TypeScript (frontend in TanStack Router + React) +- **Ecosystem:** `readline` for streaming files, `fs` for JSON I/O, native `fetch` for HTTP +- **Portability:** Runs on the same Debian laptop as the llama.cpp server +- **No build complexity:** `tsx` for direct execution, no bundler needed ### Why llama.cpp? -- **GGUF format**: Single-file models, easy to swap, quantize, and version -- **OpenAI-compatible API**: `/v1/chat/completions` means the same adapter code works for local and online models -- **No dependencies**: Self-contained binary, runs on old hardware (tested on GTX 950M) -- **Privacy**: Local inference means no data leaves the machine +- **GGUF format:** Single-file models, easy to swap, quantize, and version +- **OpenAI-compatible API:** `/v1/chat/completions` means the same adapter code works for local and online models +- **No dependencies:** Self-contained binary, runs on old hardware (tested on GTX 950M) +- **Privacy:** Local inference means no data leaves the machine --- @@ -231,65 +196,64 @@ Generating 100,000 entries with an LLM introduces risks: ### Pipeline Flow -``` +```text Run CLI -> Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed -> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses - -> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics + -> Retry/split on failure -> Merge -> Kaikki Gender Lookup -> Write atomically -> Verify schema -> Log metrics ``` ### Resumability -- **Skip existing**: `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses` -- **Atomic writes**: `.tmp` -> rename in `write-json-file.ts`, no partial files on crash -- **Cleanup on failure**: Deletes partially-written files for failed batches, continues to next batch +- **Skip existing:** `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses` +- **Atomic writes:** `.tmp` -> rename in `write-json-file.ts`, no partial files on crash +- **Cleanup on failure:** Deletes partially-written files for failed batches, continues to next batch ### Directory Structure -``` +```text data-pipeline/ -|-- pipeline.ts # Entry point / orchestrator -|-- utils/ -| |-- cli.ts # Interactive CLI module -| |-- enrich-word.ts # LLM call, parse, retry/split -| |-- merge-enriched-data.ts # Schema types + merge logic -| |-- verify-enriched-file.ts # Schema validation -| |-- check-llm-server.ts # Health check (local only) -| |-- scanning-source-files.ts # Source discovery -| |-- create-base-json.ts # Skeleton writer -| |-- write-json-file.ts # Atomic JSON writer -| |-- check-if-json-exists.ts # Resumability check -| |-- create-line-reader.ts # Streaming file reader -| |-- create-output-dirs.ts # Directory creation -| |-- delete-file.ts # Cleanup helper -| |-- get-word-file-path.ts # Path helper -| |-- progress-tracker.ts # Console progress formatting -| |-- pipeline-timer.ts # Timing + token metrics -| |-- llm-adapters/ -| |-- factory.ts # Adapter selection (uses runtime config) -| |-- types.ts # LlmAdapter interface -| |-- openai-compatible.ts # Local, OpenRouter, DeepSeek -| |-- gemini.ts # Google Gemini -|-- config/ -| |-- llm.ts # LLM config schema -| |-- prompt.ts # buildSystemPrompt() — dynamic prompt -| |-- batch.ts # Batch config schema -| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES -|-- source-data/ -| |-- {language}/ -| |-- {pos} # One word per line, no extension -|-- worddata/ - |-- {language}/ - |-- {pos}/ - |-- {word}.json # One self-contained file per word -|-- kaikki-source-files/ # Wiktionary dumps for gender lookup (planned) -|-- .pipeline-config.json # Saved CLI configuration + |-- pipeline.ts # Entry point / orchestrator + |-- utils/ + | |-- cli.ts # Interactive CLI module + | |-- enrich-word.ts # LLM call, parse, retry/split + | |-- merge-enriched-data.ts # Schema types + merge logic + | |-- verify-enriched-file.ts # Schema validation + | |-- check-llm-server.ts # Health check (local only) + | |-- scanning-source-files.ts # Source discovery + | |-- create-base-json.ts # Skeleton writer + | |-- write-json-file.ts # Atomic JSON writer + | |-- check-if-json-exists.ts # Resumability check + | |-- create-line-reader.ts # Streaming file reader + | |-- create-output-dirs.ts # Directory creation + | |-- delete-file.ts # Cleanup helper + | |-- get-word-file-path.ts # Path helper + | |-- progress-tracker.ts # Console progress formatting + | |-- pipeline-timer.ts # Timing + token metrics + | |-- llm-adapters/ + | |-- factory.ts # Adapter selection + | |-- types.ts # LlmAdapter interface + | |-- openai-compatible.ts # Local, OpenRouter, DeepSeek + | |-- gemini.ts # Google Gemini + |-- config/ + | |-- llm.ts # LLM config schema + | |-- prompt.ts # buildSystemPrompt() + | |-- batch.ts # Batch config schema + | |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES + |-- source-data/ + | |-- {language}/ + | |-- {pos} # One word per line, no extension + |-- worddata/ + | |-- {language}/ + | |-- {pos}/ + | |-- {word}.json # One self-contained file per word + |-- kaikki-source-files/ # Wiktionary dumps for gender lookup + |-- .pipeline-config.json # Saved CLI configuration ``` ### Output Schema Each `.json` file contains: `word`, `language`, `pos`, `senses[]` (each with `id`, `sense`, `example`, `difficulty_level`, `translations` per target language), `enrichedAt`, `model`. - -Full TypeScript interfaces: `utils/merge-enriched-data.ts`. +_Note: The `translations` object contains arrays of strings (e.g., `{"de": ["Haus", "Gebäude"]}`). Grammatical gender is appended later via the Kaikki integration step._ ### Error Handling @@ -306,8 +270,8 @@ Full TypeScript interfaces: `utils/merge-enriched-data.ts`. Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`. -**Unified throughput** (all providers): total tokens / total request time -**Detailed breakdown** (local only): prompt speed vs completion speed +- **Unified throughput (all providers):** total tokens / total request time +- **Detailed breakdown (local only):** prompt speed vs completion speed --- @@ -323,245 +287,198 @@ Per-run: words processed/skipped/failed, duration, throughput, LLM token counts | JSON | Native `JSON.parse/stringify` | Simple, no schema library needed | | CLI | Native `readline` | No external dependencies | -### Configuration - -| File | Purpose | -| ----------------------- | --------------------------------------------------------- | -| `config/llm.ts` | Config schema: `provider`, `url`, `model` | -| `config/prompt.ts` | `buildSystemPrompt(sourceLanguage, pos, targetLanguages)` | -| `config/batch.ts` | Config schema: `size`, `maxRetries` | -| `config/constants.ts` | `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` | -| `.pipeline-config.json` | Saved runtime config (auto-generated by CLI) | - -### Key Modules - -| File | Responsibility | -| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | -| `utils/cli.ts` | Interactive prompts: provider → model → batch size. Saves/loads `.pipeline-config.json`. Returns runtime config. | -| `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds `EnrichedSense[]`, retry/split logic | -| `utils/merge-enriched-data.ts` | Merges skeleton with LLM senses, adds `enrichedAt` and `model` | -| `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure | -| `utils/pipeline-timer.ts` | Tracks per-word and global metrics; unified throughput for all providers | -| `utils/llm-adapters/factory.ts` | Creates adapter based on runtime config from CLI | -| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek | -| `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API with `systemInstruction` | - ### Current Model -| Property | Value | -| ------------ | ---------------------------------------- | -| Model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | -| Size | ~1.0GB | -| Quantization | Q4_K_M | -| Server | llama.cpp (`llama-server`) | -| API | OpenAI-compatible `/v1/chat/completions` | +| Property | Value | +| ---------------- | ---------------------------------------------------- | +| **Model** | **`gemma-4-E2B_q4_0-it.gguf`** | +| Size | ~3.2GB (file) / ~2.06GB (VRAM weights) | +| Quantization | Q4_0 | +| Server | `llama.cpp` (`llama-server`) | +| API | OpenAI-compatible `/v1/chat/completions` | +| VRAM Usage | ~2.65GB total (weights + KV cache + compute buffers) | +| Generation Speed | ~10.9 tok/s (20-word batch) | +| Est. 100k Time | ~7 days (20-word batches, 24/7) | -### llama-server Flags: History & Rationale +### `llama-server` Flags: History & Rationale -The server flags evolved through trial and error on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM). Below is what was tried, what failed, and why the current flags were chosen. - -#### Hardware Constraints - -| Component | Spec | Implication | -| --------- | ----------------------------------------- | --------------------------------------------------------------------------- | -| CPU | i7-6500U (2 physical cores, 4 threads HT) | `-t 2` matches physical cores; HT hurts more than helps | -| GPU | GTX 950M (Maxwell, 2015) | 32 GB/s memory bandwidth, 4GB VRAM - bandwidth-starved, not compute-starved | -| RAM | 8GB (3.95GB usable) | `--mlock` pins model in RAM; system must not swap | +The server flags evolved through rigorous empirical testing on the target hardware (Intel i7-6500U, GTX 950M 4GB, 8GB RAM) across 10 different models on 2026-07-18. #### Flag Evolution -| Flag | Value Tried | Result | Why | -| ----------------- | ----------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -| `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~47s/word | Baseline. Correct genders. 2.6GB file, tight on VRAM. | -| `-m` | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken** | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#13-known-issues--dev-notes). | -| `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~8s/word | Current. Fast but gender accuracy degraded. | -| `-ngl` | `999` | Keeps | Offload all layers to GPU. Required for any speed. | -| `-c` | `4096` | Wasteful | 4K context for 300-token dictionary entries wastes ~400MB VRAM. | -| `-c` | `2048` | Current | Sufficient headroom. Frees VRAM for KV cache. | -| `-b` / `-ub` | `512` | Keeps | Sweet spot for Maxwell. Larger batches (1024+) add overhead on old GPUs. | -| `-b` / `-ub` | `2048` | Slower on 950M | Tested briefly. No improvement, possibly worse due to memory pressure. | -| `-t` | `4` | Slower | Hyperthreading cores hurt llama.cpp performance. | -| `-t` | `2` | Current | Matches 2 physical cores. | -| `--threads-batch` | (default) | Risky | Defaults to same as `-t`, but explicit is safer. | -| `--threads-batch` | `2` | Current | Explicit match to `-t`. | -| `--flash-attn` | (omitted) | Correct | On Maxwell (compute 5.0), Flash Attention adds overhead. Not used. | -| `--flash-attn` | (tested) | No gain | Briefly tried with Qwen3.5-4B. No speedup, possibly regression. | -| `--mlock` | Keeps | Pins model weights in RAM. Prevents OS swapping on memory pressure. | -| `--prio` | `2` | Keeps | Raises process priority. Marginal on this hardware, harmless. | -| `--reasoning` | `off` | (Qwen3.5 only) | Qwen3.5 has reasoning mode. Disabling it speeds up non-reasoning tasks. Irrelevant for Qwen2.5. | +| Flag | Value Tried | Result | Why | +| ----------------- | ------------------------------------------ | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | +| `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~6.3 tok/s | Quality baseline. Correct translations. 2.6GB, tight on VRAM. | +| `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~18.5 tok/s | Fast but poor instruction following, gender wrong. | +| `-m` | `Qwen3.5-2B-Q4_K_M.gguf` | Works, ~13.2 tok/s | Good translations, but failed polysemy (2 identical senses). | +| `-m` | `Llama-3.2-3B-Instruct-Q4_K_M.gguf` | Works, ~9.1 tok/s | Dangerous false friend trap (cognates). | +| `-m` | `Llama-3.2-3B-Instruct-UD-Q6_K_XL.gguf` | Works, ~6.7 tok/s, 3.44GB VRAM | Higher quant did NOT fix Llama's translation issues. Slower. | +| `-m` | `Ministral-3-3B-Instruct-2512-Q4_K_M.gguf` | Works, ~8.9 tok/s | Fixed tokenizer bug (older version was broken). But messy translations, markdown violations. | +| `-m` | `gemma-4-E2B_q4_0-it.gguf` | **Works, ~13.1 tok/s, 2.06GB** | **Winner.** Perfect polysemy, false friends, nuance. Half VRAM of Qwen 4B. | +| `-m` | `gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf` | Works, ~7.7 tok/s, 3.23GB | QAT compression is incredible. Same quality as E2B but slower. | +| `-m` | `qwen3.5-9b-q3_k_s.gguf` | Works, ~3.3 tok/s (split CPU/GPU) | Brilliant quality but bottlenecked by CPU/Swap. `-ngl 28` max. | +| `-m` | `qwen3.5-9b-q4_k_m.gguf` | Works, ~3.3 tok/s (split CPU/GPU) | Same quality as 9B Q3. Not worth 2x the file size. | +| `-ngl` | 999 | Keeps | Offload all layers to GPU. Required for any speed. | +| `-ngl` | 28 | 9B models only | Max GPU layers for 9B models before OOM. Found via binary search. | +| `-ngl` | 30 | OOM crash (9B) | Pushed 2 layers too far into compute buffers. | +| `-c` | 4096 | Wasteful | 4K context for 300-token dictionary entries wastes VRAM. | +| `-c` | 2048 | Good for small batches | Sufficient for 4-word batches. | +| `-c` | **8192** | **Current** | Required for 20-word batches. Combined with KV cache quantization. | +| `-b` / `-ub` | 512 | **Current** | Sweet spot for Maxwell memory bandwidth. | +| `-b` / `-ub` | 1024 | Tested | Slightly faster prompt processing, but no generation speedup. | +| `-b` / `-ub` | 2048 | Slower on 950M | Memory pressure on bandwidth-starved GPU. | +| `-t` | 4 | Slower | Hyperthreading cores hurt llama.cpp performance. | +| `-t` | **2** | **Current** | Matches 2 physical cores. | +| `--threads-batch` | **2** | **Current** | Explicit match to `-t`. | +| `--flash-attn` | (omitted) | Correct | On Maxwell (compute 5.0), Flash Attention adds overhead. | +| `--mlock` | Tested | Omitted for large models | Pins model in RAM. Causes OOM on models >2.5GB with large KV cache. | +| `--prio` | **2** | **Current** | Raises process priority. Marginal, harmless. | +| `--reasoning` | **off** | **Critical** | **Mandatory for Qwen 3.5 and Gemma 4.** Without this, models "think" silently, consume all `max_tokens`, and crash with `finish_reason: length`. | +| `--cache-type-k` | **q4_0** | **Current** | Compresses KV cache keys to 4-bit. Cuts KV VRAM by ~75%. Enables 8192 context on 4GB GPU. | +| `--cache-type-v` | **q4_0** | **Current** | Compresses KV cache values to 4-bit. Paradoxically improves translation variety (reduces "lazy duplication" bug). | -#### Current Command +#### Current Production Command ```bash +cd ~/Downloads/llama.cpp ./build/bin/llama-server \ - -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \ + -m models/gemma-4-E2B_q4_0-it.gguf \ -ngl 999 \ - -c 2048 \ + -c 8192 \ -b 512 \ -ub 512 \ -t 2 \ --threads-batch 2 \ --host 127.0.0.1 \ --port 8080 \ - --mlock \ - --prio 2 + --prio 2 \ + --reasoning off \ + --cache-type-k q4_0 \ + --cache-type-v q4_0 ``` -#### What Was Not Tried (And Why) +#### VRAM Budget (Production Config) -| Flag | Reason Skipped | -| --------------------------- | ---------------------------------------------------------------------------------------------------------- | -| `-fa` / `--flash-attn` | Maxwell architecture lacks efficient FA kernels. Benchmarks show regression or no gain on pre-Ampere GPUs. | -| `--no-mmap` | `--mlock` achieves the same (pin in RAM) without the I/O overhead of disabling mmap. | -| `-np` / `--parallel` | Not needed. Single sequential pipeline, no concurrent requests. | -| `--cont-batching` | Default in recent llama.cpp. No need to toggle. | -| `--defrag-thold` | KV cache defragmentation. Only relevant for very long contexts or heavy reuse. Not needed for 2048 ctx. | -| `-ot` / `--override-tensor` | Expert-level. No tensor-specific issues observed. | +| Component | VRAM Usage | +| ------------------------------------ | ------------------------- | +| Model Weights (Gemma 4 E2B Q4_0) | ~1.50 GB | +| KV Cache (8192 ctx, q4_0 compressed) | ~0.35 GB | +| Compute Buffers (batch 512) | ~0.20 GB | +| **Total** | **~2.05 GB (51% of 4GB)** | -#### Future Flag Experiments +#### Why Not Use the Remaining 2GB VRAM? -| Experiment | Expected Outcome | -| ---------------- | -------------------------------------------------------------------------- | -| `-c 1024` | Further VRAM savings. Risk: insufficient for batching larger prompt sizes. | -| `-b 256 -ub 256` | Test if smaller batches reduce latency on bandwidth-starved Maxwell. | -| `--metrics` | Enable Prometheus endpoint for automated performance tracking. | +Generation speed is bottlenecked by **memory bandwidth** (~32 GB/s on GTX 950M), not VRAM capacity. To generate one token, the GPU must read the entire ~1.5GB model from VRAM. The theoretical maximum is ~21 tok/s. At 10.9 tok/s, the GPU is already operating at ~50% of its physical limit. Empty VRAM cannot be converted into faster generation. -### Performance Baseline +Testing uncompressed `f16` KV cache (2.65GB total VRAM) yielded 12.6 tok/s but caused a severe "lazy duplication" regression (model copy-pasted the same translation twice instead of providing distinct synonyms). The `q4_0` compressed KV cache is the correct choice for translation quality. -| Metric | Qwen3.5-4B | Qwen2.5-1.5B | -| --------------------- | ---------- | ------------ | -| Time/word | ~47s | ~8s | -| Completion tok/s | ~6.4 | ~18.8 | -| Prompt tok/s | ~81 | ~549 | -| Avg completion tokens | ~274 | ~132 | -| Avg prompt tokens | ~327 | ~308 | +### Performance Baseline (20-Word Batch) -### Known Limitations (Current) - -- **Gender accuracy**: Qwen2.5-1.5B systematically defaults to `neuter` for Romance languages. Under evaluation whether larger models fix this. -- **Single POS**: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments. -- **Pre-scanning wordlists**: Entire file read into memory before processing. Inefficient for 100k words. See [Refactor Notes](#refactor-notes). +| Metric | Gemma 4 E2B | Qwen 3.5 4B | +| --------------------- | ----------- | ------------ | +| Time/batch (20 words) | ~5 min | ~9 min | +| Completion tok/s | ~10.9 | ~6.1 | +| Prompt tok/s | ~132 | ~78 | +| VRAM Usage | 2.06 GB | 3.95 GB | +| Lazy Duplication Bug | No | Yes (severe) | --- ## 5. The LLM Layer -### 5.1 Local Model Evaluation +### 5.1 Local Model Evaluation (Complete — 2026-07-18) -Models are evaluated on three criteria in order of priority: **quality** (definitions, examples, translations, gender accuracy), **speed** (completion tokens/sec), **VRAM fit** (must run on GTX 950M 4GB). +All 10 downloaded models were evaluated on the target hardware. Testing progressed from 4-word smoke tests to a 20-word "nightmare" torture suite covering extreme polysemy, false friends, abstract concepts, and legal/financial terminology. -| Model | Size | VRAM | Speed | Quality | Status | -| ----------------------- | ------ | ----- | ------------------- | --------------------------- | -------------------------------- | -| **Qwen3.5-4B Q4_K_M** | 2.6GB | Tight | ~6.4 tok/s | Baseline (assumed good) | Baseline - too slow | -| **Qwen2.5-1.5B Q4_K_M** | 1.0GB | Easy | ~18.8 tok/s | Gender systematically wrong | Current - fast, needs validation | -| **Ministral-3B Q4_K_M** | 1.9GB | Fits | - | Broken (tokenizer) | Abandoned | -| **Qwen2.5-3B Q4_K_M** | 1.9GB | Fits | ~12-15 tok/s (est.) | Unknown | Pending download | -| **Gemma 4 E2B Q4_K_M** | 3.46GB | No | - | - | Too large for 4GB VRAM | -| **Gemma 4 E2B IQ2_M** | 2.62GB | Fits | ~8-12 tok/s (est.) | "Low quality" per Google | Not worth it | +#### Final Leaderboard -#### Qwen2.5-1.5B Test Results (3 words) +| Rank | Model | Size | VRAM | Speed (tok/s) | Polysemy | False Friends | Verdict | +| ------ | ------------------------ | ---- | ------------- | ------------- | --------------------------- | ------------------------- | ----------------------------------------------------------- | +| **🥇** | **Gemma 4 E2B Q4_0** | 3.2G | **2.06 GB** | **~13.1** | **Perfect** | **Perfect** | **Production model.** Best quality/speed/VRAM ratio. | +| 🥈 | Qwen 3.5 4B Q4_K_M | 2.6G | ~3.6 GB | ~6.3 | Perfect | Perfect | Quality King, but 2x slower and maxes VRAM. | +| 🥉 | Gemma 4 E4B Q4_K_XL | 4.0G | 3.23 GB | ~7.7 | Perfect | Perfect | Incredible QAT compression. Same quality as E2B but slower. | +| 4 | Qwen 3.5 2B Q4_K_M | 1.2G | ~1.6G | ~13.2 | Failed (2 identical senses) | Passed | Good translations, lacks conceptual branching. | +| 5 | Qwen 2.5 1.5B Q4_K_M | 1.1G | ~1.5G | ~18.5 | Ignored instruction | Failed | Fast but easily confused. | +| 6 | Ministral 3B 2512 Q4_K_M | 2.0G | ~2.7G | ~8.9 | Messy translations | Failed | Fixed tokenizer, but outclassed. Markdown violations. | +| 7 | Llama 3.2 3B Q4_K_M | 1.9G | ~2.6G | ~9.1 | Good structure, bad IT/ES | **Failed (cognate trap)** | Dangerous for language learners. | +| 8 | Llama 3.2 3B Q6_K_XL | 2.8G | 3.44G | ~6.7 | Good structure, bad IT/ES | **Failed (cognate trap)** | Higher quant did NOT fix translation issues. | +| 9 | Qwen 3.5 9B Q3_K_S | 4.1G | Split CPU/GPU | ~3.3 | Perfect | Perfect | Brilliant but 35-40 days for 100k words. | +| 10 | Qwen 3.5 9B Q4_K_M | 5.3G | Split CPU/GPU | ~3.3 | Perfect | Perfect | Same quality as 9B Q3. Not worth the size. | -| Word | Definition | Example | Gender (de/it/es/fr) | Verdict | -| ------ | -------------------------------------------- | ---------------------------------- | --------------------------------- | --------- | -| time | "A period of duration..." | "The meeting was scheduled..." | neuter/neuter/neuter/neuter | All wrong | -| year | "A period of time consisting of 365 days..." | "The year 2023 is a leap year." | neuter/neuter/neuter/neuter | All wrong | -| people | "Individuals who are part of a group." | "The people gathered at the park." | neuter/feminine/feminine/feminine | Mixed | +#### Key Findings -**Pattern:** Defaults to `neuter` when uncertain. Only correct when obvious (feminine endings in Romance languages). German "Jahr" is genuinely neuter - only correct by accident. - -#### Pending Tests - -- **Qwen2.5-3B**: Same architecture, 2x params. If gender fixes, it was a size problem. -- **20-word torture suite**: concrete, abstract, polysemous, technical, false friends. Will run on all candidate models. +1. **The "Thinking" Trap:** Both Qwen 3.5 and Gemma 4 have built-in Chain-of-Thought reasoning. Without `--reasoning off`, they silently "think" in a hidden JSON field, consume all `max_tokens`, and crash with `finish_reason: length`. This flag is **mandatory**. +2. **The 2B vs 4B Quality Cliff:** 2B models struggle with polysemy (e.g., cannot distinguish "bank" = financial vs river). 4B+ models act like professional lexicographers. +3. **Llama 3.2 is Unsafe for Language Learners:** Consistently fell for false friend traps (e.g., translating "actual" = _real_ to _aktuell/attuale/actual/actuel_ = _current_). +4. **KV Cache Quantization Improves Translation Variety:** Compressing the KV cache to `q4_0` introduces microscopic noise that prevents the "lazy duplication" bug (model copy-pasting the same synonym twice). +5. **POS Bleed is Universal:** All models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_ instead of _a jogging session_). Fix: explicit negative constraint in the system prompt. ### 5.2 Online API Options -Evaluated as fallbacks if local models fail quality or speed targets. All support OpenAI-compatible API (except Gemini, which has a native adapter). +Evaluated as fallbacks if local models fail quality or speed targets. -| Provider | Model | Input $/1M | Output $/1M | Free Tier | Rate Limit | Est. Cost (100k words) | Est. Time | -| ------------------- | -------------------- | ---------- | ----------- | ------------- | ---------------- | ---------------------- | ------------------- | -| **DeepSeek** | V4 Flash | $0.14 | $0.28 | 5M tokens | None | **$0-15** | ~1-2 days | -| **Gemini** | 2.5 Flash-Lite | $0.10 | $0.40 | 1,500 req/day | 1M TPM | **$0** (free tier) | ~1.5 days (batched) | -| **Qwen/Alibaba** | Qwen-Turbo | $0.05 | $0.20 | Unknown | 600 RPM | **~$11** | ~1-2 days | -| **Groq** | Llama 3.1 8B Instant | $0.05 | $0.08 | Yes | High | **~$7** | **~3-4 hours** | -| **OpenRouter free** | Various | $0 | $0 | 200 req/day | 20 RPM | **$0** | ~10 days | -| **OpenRouter paid** | DeepSeek V4 Flash | $0.14 | $0.28 | - | Same as provider | **~$16** (+5.5% fee) | ~2-3 days | - -**Notes:** - -- Costs assume ~550 tokens/word (300 prompt + 250 completion). -- Gemini free tier: 1,500 requests/day x 50 words/batch = 75k words/day. -- Groq: 500+ tok/s inference speed. Fastest option if cost is acceptable. -- DeepSeek: 5M free tokens ~ 9,000 words. Remainder at $0.14/$0.28 per million. +| Provider | Model | Input $/1M | Output $/1M | Free Tier | Rate Limit | Est. Cost (100k words) | Est. Time | +| --------------- | -------------------- | ---------- | ----------- | ------------- | ---------- | ---------------------- | ------------------- | +| DeepSeek | V4 Flash | $0.14 | $0.28 | 5M tokens | None | $0-15 | ~1-2 days | +| Gemini | 2.5 Flash-Lite | $0.10 | $0.40 | 1,500 req/day | 1M TPM | $0 (free tier) | ~1.5 days (batched) | +| Qwen/Alibaba | Qwen-Turbo | $0.05 | $0.20 | Unknown | 600 RPM | ~$11 | ~1-2 days | +| Groq | Llama 3.1 8B Instant | $0.05 | $0.08 | Yes | High | ~$7 | ~3-4 hours | +| OpenRouter free | Various | $0 | $0 | 200 req/day | 20 RPM | $0 | ~10 days | ### 5.3 Model Selection Criteria Decision flow for 100,000 words: -``` +```text Start - | - v -Run 20-word torture suite -on Qwen2.5-3B (local) - | - |-- Quality good? -----> Use Qwen2.5-3B locally - | (gender correct) ~20 days, $0 - | - |-- Quality meh? -------> Test Gemini 2.5 Flash-Lite (free) - | - |-- Quality good? --> Batch 50, free tier - | ~1.5 days, $0 - | - |-- Quality meh? ---> Test Groq or DeepSeek paid - | - |-- Speed priority? --> Groq - | ~$7, 3-4 hours - | - |-- Cost priority? ---> DeepSeek - ~$15, 1-2 days + | + v + Gemma 4 E2B (local) — SELECTED + | + |-- Speed acceptable? (~7 days) -----> Use Gemma 4 E2B locally, $0 + | + |-- Need faster? -------> Test Gemini 2.5 Flash-Lite (free) + | + |-- Quality good? --> Batch 50, free tier + | ~1.5 days, $0 + | + |-- Quality meh? ---> Test Groq or DeepSeek paid ``` -**Quality gates:** +Quality gates: -- > = 90% gender accuracy (de/it/es/fr) - 100% JSON parse rate - No hallucinated definitions on polysemous words - Natural, contextually appropriate example sentences - Sensible difficulty classification (CEFR mapping) +- Gender accuracy is no longer an LLM criterion (handled by Kaikki) --- ## 6. The Gender Problem & Kaikki Integration -### The Problem +### The Problem (Resolved) -Grammatical gender is embedded in the `translations` object of each sense: +Grammatical gender was originally embedded in the LLM's `translations` object. Testing across all 10 models confirmed that small local models systematically hallucinate gender, defaulting to `neuter` for Romance languages (Italian, Spanish, French) which do not have a neuter grammatical gender. -```json -"translations": { - "de": [{"word": "Haus", "gender": "neuter"}], - "it": [{"word": "casa", "gender": "feminine"}], - "es": [{"word": "casa", "gender": "feminine"}], - "fr": [{"word": "maison", "gender": "feminine"}] -} -``` +### The Solution: Decoupled Architecture -Early testing with **Qwen2.5-1.5B** showed systematic failure: the model defaults to `neuter` for any translation where it is uncertain. This is particularly broken for Romance languages (Italian, Spanish, French), which do not have a neuter grammatical gender at all — only masculine and feminine. +**Decision (2026-07-18):** Grammatical gender is no longer generated by the LLM. The pipeline now uses a two-stage approach: -Whether this is a **model size issue** (1.5B too small to retain gender facts) or a **training data gap** (Qwen2.5 family lacks gender-annotated multilingual data) is unresolved. Pending the Qwen2.5-3B evaluation. +| Stage | Component | Responsibility | +| ----- | ----------------- | -------------------------------------------------------------------- | +| 1 | LLM (Gemma 4 E2B) | Generates translation **strings only** (e.g., `["Haus", "Gebäude"]`) | +| 2 | Kaikki Lookup | Deterministically resolves grammatical gender from Wiktionary dumps | -### Two Approaches Under Consideration +### Benefits -| Approach | How It Works | Pros | Cons | -| -------------------------- | ---------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | -| **A. LLM-only** | Trust the model to generate correct gender | Zero additional complexity | Unreliable at small sizes; hallucination risk | -| **B. LLM + Kaikki lookup** | LLM generates word + translation string; post-processing step looks up gender from Kaikki JSONL dump | 100% deterministic; decouples gender from model quality | Adds pipeline stage; requires Kaikki data for each target language; must handle missing entries | +- **100% deterministic gender accuracy** — no hallucination possible +- **Faster LLM generation** — ~15-20% fewer output tokens per word +- **Simpler JSON schema** — translations are string arrays, not object arrays +- **Model-agnostic** — works with any LLM regardless of multilingual training quality ### Kaikki Data -Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Each line is a lexical entry with structured data including gender. - | Language | Kaikki File | Coverage | | -------- | ------------------------------------- | -------- | | German | `kaikki.org-dictionary-German.jsonl` | High | @@ -569,14 +486,9 @@ Kaikki provides processed Wiktionary dumps as JSONL files, one per language. Eac | Spanish | `kaikki.org-dictionary-Spanish.jsonl` | High | | French | `kaikki.org-dictionary-French.jsonl` | High | -Lookup logic: match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`. +### Lookup Logic -### Decision Pending - -- If **Qwen2.5-3B** or an online model produces >=90% accurate gender: **Approach A**, no Kaikki needed. -- If all tested models fail gender: **Approach B**, implement Kaikki lookup as a post-processing step after LLM enrichment. - -No implementation work started until the 20-word torture suite resolves this. +Match on `word` (the translated string) -> extract `gender` field -> map to `"masculine" | "feminine" | "neuter" | null`. --- @@ -586,144 +498,105 @@ No implementation work started until the 20-word torture suite resolves this. At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes the ~300-token system prompt. Batching amortizes this cost. -### Configurable Batch Size +### Optimal Batch Size (Local) -Batch size is selected interactively at runtime via the CLI. The schema lives in `config/batch.ts`: +**20 words** is the validated sweet spot for the GTX 950M with Gemma 4 E2B. -```typescript -export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const; -``` - -The CLI presents preset options (1, 5, 10, 20, 50) with provider-specific recommendations. - -### Prompt Structure - -**Single word:** - -``` -["house"] -``` - -**Batch of 4:** - -``` -["house", "car", "tree", "water"] -``` - -LLM returns a JSON object with word keys: - -```json -{ - "house": [ { "sense": "...", "example": "...", ... } ], - "car": [ { ... } ], - "tree": [ { ... } ], - "water": [ { ... } ] -} -``` +| Batch Size | VRAM | Speed | Quality | Verdict | +| ---------- | --------------- | ---------------- | ------------------------ | -------------------------------------------------- | +| 1-4 | ~2.1 GB | ~13 tok/s | Perfect | Safe but slow (amortization waste) | +| **20** | **~2.6 GB** | **~10.9 tok/s** | **Perfect** | **Sweet spot** | +| 30-40 | ~3.0 GB (est.) | ~10 tok/s (est.) | Likely good | Worth testing | +| 50+ | ~3.5 GB+ (est.) | Unknown | Risk of JSON degradation | **Not recommended for 2B models** | +| 100 | OOM risk | N/A | Attention degradation | Small models lose JSON structure past ~6000 tokens | ### Retry & Split Strategy If a batch fails (bad JSON, missing key, etc.): -``` -Batch of 4 fails (3 retries exhausted) +```text +Batch of 20 fails (3 retries exhausted) | v -Split into 2 batches of 2 +Split into 2 batches of 10 | v -If a batch of 2 fails (3 retries), split into 2 batches of 1 +If a batch of 10 fails (3 retries), split into 2 batches of 5 + | + v +If a batch of 5 fails, split into batches of 1 | v If a single word fails (3 retries), log and skip ``` -This gives resilience without losing the speed benefit of large batches. The `maxRetries` config controls how many attempts are made before splitting. - -### Expected Impact by Environment - -| Environment | Batch Size | Expected Speedup | Notes | -| ------------------ | ---------- | ---------------- | -------------------------------------------------------- | -| Local GTX 950M | 3 | ~1.05-1.15x | Memory bandwidth limited. KV cache pressure on 4GB VRAM. | -| Local GTX 950M | 5 | ~1.10-1.20x | Sweet spot for this hardware. | -| Local GTX 950M | 15 | Risky | May OOM. Test carefully. | -| Local GTX 950M | 50 | Unlikely | VRAM insufficient. | -| Cloud API (Gemini) | 50 | 5x fewer calls | Unlocks free tier viability. | -| Cloud API (Groq) | 50 | 5x fewer calls | Minimal gain - already fast. | - -### Batching is Non-Negotiable - -Regardless of local vs cloud, batching is required for 100k words: - -- **Local**: Better GPU utilization, amortized prompt processing -- **Cloud**: Slams into rate limits slower, unlocks free tiers, some APIs offer 50% batch discounts - --- ## 8. Hardware Constraints ### Current Machine -| Component | Spec | -| --------- | ------------------------------------------------------------------- | -| OS | Debian GNU/Linux 13 (trixie) x86_64 | -| CPU | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz | -| GPU | NVIDIA GeForce GTX 950M (Maxwell, 2015) | -| GPU VRAM | 4GB | -| RAM | 8GB (3.95GB usable at idle) | -| Disk | 102GB ext4 (~63GB used) | +| Component | Spec | +| ------------- | ------------------------------------------------------------------- | +| OS | Debian GNU/Linux 13 (trixie) x86_64 | +| CPU | Intel Core i7-6500U (2 physical cores, 4 threads via HT) @ 3.10 GHz | +| GPU | NVIDIA GeForce GTX 950M (Maxwell, 2015) | +| GPU VRAM | 4GB (4037 MiB reported by CUDA) | +| GPU Bandwidth | ~32 GB/s (DDR3) | +| RAM | 8GB (~3.67GB usable at idle) | +| Swap | 5.62 GB | +| Disk | 102GB ext4 (~74GB used) | -### What Fits in 4GB VRAM +### What Fits in 4GB VRAM (Empirically Verified) -| Model | File Size | KV Cache (2048 ctx) | Total VRAM | Fits? | -| ------------------- | --------- | ------------------- | ---------- | ------------------ | -| Qwen2.5-1.5B Q4_K_M | ~1.0GB | ~0.5GB | ~1.5GB | Yes | -| Qwen2.5-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes | -| Ministral-3B Q4_K_M | ~1.9GB | ~0.8GB | ~2.7GB | Yes (but broken) | -| Qwen3.5-4B Q4_K_M | 2.6GB | ~1.0GB | ~3.6GB | Tight | -| Gemma 4 E2B Q4_K_M | 3.46GB | ~1.2GB | ~4.7GB | No | -| Gemma 4 E2B IQ2_M | 2.62GB | ~1.0GB | ~3.6GB | Maybe, low quality | +| Model | File Size | Total VRAM | Fits? | Notes | +| ------------------------ | --------- | ----------- | ---------- | -------------------------------------- | +| Qwen 2.5 1.5B Q4_K_M | 1.1G | ~1.5 GB | ✅ Easy | | +| Qwen 3.5 2B Q4_K_M | 1.2G | ~1.6 GB | ✅ Easy | | +| Ministral 3B 2512 Q4_K_M | 2.0G | ~2.7 GB | ✅ Yes | | +| Llama 3.2 3B Q4_K_M | 1.9G | ~2.6 GB | ✅ Yes | | +| Llama 3.2 3B Q6_K_XL | 2.8G | 3.44 GB | ✅ Tight | | +| **Gemma 4 E2B Q4_0** | **3.2G** | **2.06 GB** | **✅ Yes** | **QAT compression. Production model.** | +| Gemma 4 E4B Q4_K_XL | 4.0G | 3.23 GB | ✅ Yes | QAT compression is incredible. | +| Qwen 3.5 4B Q4_K_M | 2.6G | 3.95 GB | ⚠️ Barely | 50MB headroom with 8192 ctx. | +| Qwen 3.5 9B Q3_K_S | 4.1G | Split | ⚠️ Partial | `-ngl 28` max. Rest on CPU/Swap. | +| Qwen 3.5 9B Q4_K_M | 5.3G | Split | ⚠️ Partial | `-ngl 20` max. Heavy swap usage. | ### GPU Rental Alternatives If local hardware becomes the bottleneck: - -| Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost | -| -------- | -------- | ---- | ----------- | ------------------- | ---------- | -| Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | **~$2-5** | -| RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | **~$4-6** | -| Vast.ai | RTX 3090 | 24GB | ~$0.20-0.40 | ~8-10 hours | **~$2-4** | - -With an RTX 4090, Qwen2.5-1.5B runs at ~100-150 tok/s. 100k words in under a day. +| Provider | GPU | VRAM | Price/Hour | Time for 100k Words | Total Cost | +| --- | --- | --- | --- | --- | --- | +| Vast.ai | RTX 4090 | 24GB | ~$0.30-0.60 | ~6-8 hours | ~$2-5 | +| RunPod | RTX 4090 | 24GB | ~$0.50-0.80 | ~6-8 hours | ~$4-6 | --- ## 9. Testing & Quality Assurance -### 20-Word Torture Suite +### 20-Word Torture Suite (Completed 2026-07-18) -Planned test set covering edge cases: +Tested on Gemma 4 E2B and Qwen 3.5 4B with 20 challenging nouns: -| Category | Words | Why | -| -------------- | ------------------------------------------------------ | ----------------------------------- | -| Easy concrete | `house`, `water`, `book` | Baseline | -| Easy abstract | `time`, `love`, `hope` | Abstract nouns harder to define | -| Polysemous | `bank`, `run`, `light` | Multiple senses test disambiguation | -| Hard/technical | `democracy`, `photosynthesis`, `entropy` | Complex definitions | -| False friends | `actual` (en/es), `sensible` (en/fr), `fabric` (en/de) | Cross-lingual traps | +| Category | Words | The Trap | +| ----------------- | ------------------------------------------------------- | -------------------------------------------------------------- | +| Extreme Polysemy | `match`, `date`, `right`, `set`, `well` | Does it translate "match" as fire, sports, or dating? | +| False Friends | `sense`, `fabric`, `sympathy`, `eventuality`, `billion` | "Fabric" = material (_tissu_) not factory (_fabrique_) | +| Abstract/Cultural | `serendipity`, `accountability` | Concepts lacking 1:1 dictionary equivalents | +| Action-Nouns | `run`, `drive`, `play` | "Run" as jog vs tear vs campaign | +| Legal/Financial | `mortgage`, `lease`, `court`, `board`, `draft` | Requires specific legal vocabulary (_Hypothek/mutuo/hipoteca_) | -### Evaluation Criteria +### Results Summary -For each word and each candidate model: - -| Criterion | Pass Threshold | -| ----------------------------- | ------------------------------------------------- | -| Definition accuracy | Factually correct, max 15 words, student-friendly | -| Example quality | Natural sentence, word used correctly in context | -| Translation correctness | Correct word sense match | -| Gender accuracy (de/it/es/fr) | >=90% correct | -| Difficulty classification | Sensible per CEFR mapping | -| JSON reliability | 100% parse rate, valid schema | +| Criterion | Gemma 4 E2B | Qwen 3.5 4B | +| ------------------------------------ | ------------------------------------------ | ------------------------------------------------------- | +| JSON Reliability | 10/10 (raw JSON) | 10/10 (raw JSON, but added `sense_index` hallucination) | +| Polysemy (bank, match) | 10/10 (Ufer/riva/orilla/rive) | 10/10 | +| False Friends (fabric) | 10/10 (Stoff/tessuto/tela/tissu) | 10/10 | +| Legal Nuance (mortgage) | 10/10 (Hypothek/mutuo/hipoteca/hypothèque) | 10/10 | +| Lazy Duplication Bug | None | **Severe** (copy-pasted same word 20+ times) | +| POS Bleed | Minor (verb defs for run/match) | Minor (verb defs for run/match) | +| Attention Degradation (word 20 vs 1) | None | None | ### Verification @@ -731,131 +604,21 @@ For each word and each candidate model: - Required top-level fields: `word`, `language`, `pos`, `senses` - Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard} -- Each translation: `word` (string), `gender` in {masculine, feminine, neuter, null} - -Additionally, `validateSense()` in `enrich-word.ts` catches malformed senses **before** file writes, triggering retry/split instead of writing bad data. +- Each translation: array of strings (gender appended later by Kaikki) --- ## 10. Interactive CLI -### Overview +### Batch Size Recommendations (Updated) -The pipeline includes an interactive CLI (`utils/cli.ts`) that asks the user to select provider, model, and batch size on each run. No editing of `config/llm.ts` or `config/batch.ts` required. - -### Flow - -``` -$ npx tsx pipeline.ts - -🌐 Lila Data Pipeline -───────────────────── - -[1] Use last config: online → gemini → gemini-2.5-flash-lite → batch 50 -[2] Configure new run - -> 2 - -Provider type: - [1] Local (llama.cpp) - [2] Online API - -> 2 - -Online provider: - [1] Gemini - [2] DeepSeek - [3] OpenRouter - [4] Groq - -> 1 - -Model: - [1] gemini-2.5-flash-lite (recommended for cost) - [2] gemini-2.5-pro (recommended for quality) - -> 1 - -Batch size: - [1] 1 (safest, slowest) - [2] 5 (recommended for local) - [3] 10 - [4] 20 - [5] 50 (recommended for Gemini) - [6] Custom - -> 5 - -✅ Configuration: - Provider: gemini - Model: gemini-2.5-flash-lite - API key: GEMINI_API_KEY found in environment ✓ - Batch size: 50 - -Start pipeline with these settings? [Y/n] -> Y -``` - -### Saved Config - -On first run, after confirming, write to `.pipeline-config.json`: - -```json -{ - "provider": "gemini", - "model": "gemini-2.5-flash-lite", - "batchSize": 50, - "lastRun": "2026-07-06T13:54:00Z" -} -``` - -Next run shows `[1] Use last config` at the top. - -### API Key Rules - -- **Never prompt for keys.** Check `process.env` for the provider's key. -- **If missing:** Print which env var is needed, then exit. -- **Supported env vars:** `GEMINI_API_KEY`, `DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY` - -### Batch Size Recommendations - -| Provider | Recommended | Rationale | -| ------------------ | ----------- | ---------------------------------- | -| `local` (GTX 950M) | 5 | VRAM-limited, KV cache pressure | -| `local` (RTX 4090) | 20 | Fast, more VRAM | -| `gemini` | 50 | Free tier: 1,500 req/day | -| `deepseek` | 20 | 5M free tokens, balance speed/cost | -| `groq` | 50 | Very fast, rate limits generous | -| `openrouter` | 10 | 200 req/day free tier | - -### Implementation - -The CLI is implemented in `utils/cli.ts` using Node.js native `readline` module. No external dependencies. - -```typescript -// utils/cli.ts -import readline from "readline"; - -export async function runCli(): Promise<{ - provider: string; - model: string; - url: string; - batchSize: number; -}> { - // ... interactive prompts ... -} -``` - -`pipeline.ts` imports and calls `runCli()` at startup: - -```typescript -import { runCli } from "./utils/cli.js"; - -async function main() { - const config = await runCli(); - // Use config.provider, config.model, etc. -} -``` +| Provider | Recommended | Rationale | +| -------------------- | ----------- | ----------------------------------------------- | +| **local (GTX 950M)** | **20** | Validated sweet spot. 8192 ctx + q4_0 KV cache. | +| local (RTX 4090) | 50 | Fast, more VRAM | +| gemini | 50 | Free tier: 1,500 req/day | +| deepseek | 20 | 5M free tokens | +| groq | 50 | Very fast | --- @@ -863,17 +626,15 @@ async function main() { ### Near-Term (Next 2-4 Weeks) -| Item | Status | Notes | -| ---------------------- | ------------ | ----------------------------------------------------------------- | -| Configurable batching | **Complete** | `config/batch.ts` with `size` and `maxRetries` | -| Retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch | -| Honest timing metrics | **Complete** | Unified throughput for all providers, detailed only for local | -| Auto-target languages | **Complete** | `buildSystemPrompt()` excludes source from targets | -| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes | -| Interactive CLI | **Complete** | `utils/cli.ts` — provider/model/batch selection with saved config | -| 20-word torture suite | Pending | Decides gender approach and model selection | -| Qwen2.5-3B evaluation | Pending | Download and test | -| Online API testing | Pending | Gemini free tier, DeepSeek, Groq | +| Item | Status | Notes | +| ----------------------------------- | ------------ | --------------------------------------------------- | +| 10-model evaluation | **Complete** | Gemma 4 E2B selected | +| KV cache quantization | **Complete** | `--cache-type-k/v q4_0` enables 8192 ctx on 4GB GPU | +| Gender decoupling | **Complete** | Kaikki lookup replaces LLM gender | +| 20-word torture suite | **Complete** | Validated on Gemma E2B and Qwen 4B | +| POS bleed fix | **Pending** | Add negative constraint to system prompt | +| Kaikki gender lookup implementation | **Pending** | Post-processing step after LLM enrichment | +| Online API testing | **Pending** | Gemini free tier, DeepSeek, Groq | ### Medium-Term (1-3 Months) @@ -883,13 +644,11 @@ async function main() { | Multi-language source | German -> French, Italian -> Spanish, etc. | | Parallel wordlist processing | Run `english/nouns` and `english/verbs` simultaneously | | Incremental enrichment | Only process new/changed words in a wordlist | -| Model auto-switching | Fallback to online API if local server fails mid-run | ### Long-Term (3-6 Months) | Item | Notes | | ------------------------ | ---------------------------------------------------------------- | -| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA | | GPU rental integration | Script to spin up Vast.ai/RunPod, run pipeline, download results | | Quality regression tests | Run torture suite on every model change | | Community contributions | Open-source the pipeline for other language learners | @@ -898,59 +657,45 @@ async function main() { ## 12. Decisions Log -| Date | Decision | Context | Rationale | -| ---------- | ----------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------- | -| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app | -| 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying | -| 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas | -| 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys | -| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies | -| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference | -| 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM | -| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance | -| 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation | -| 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality | -| 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion | -| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations | -| 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 | -| 2026-07-06 | Honest timing metrics | Cloud providers don't expose prompt/completion breakdown | Unified `totalTimeMs` for all; detailed breakdown only when available | -| 2026-07-06 | Auto-target languages | Prompt hardcoded English -> de/it/es/fr | `ALL_LANGUAGES` minus source = targets; works for any source language | -| 2026-07-06 | Validate LLM responses before write | Bad data was written then warned about | `validateSense()` catches malformed responses early, triggers retry | -| 2026-07-06 | Interactive CLI | Editing `config/llm.ts` to switch providers is error-prone | `utils/cli.ts` with native `readline`; no external dependencies; saves config | +| Date | Decision | Context | Rationale | +| -------------- | ----------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | +| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing | +| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API | +| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Community standard for 4-bit | +| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp | +| 2026-06-17 | Qwen2.5-1.5B as initial model | Qwen3.5-4B too slow (47s/word) | 6x speedup, quality under evaluation | +| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory | +| 2026-07-06 | Retry + split batching | LLM JSON parse failures | 3 retries, then halve batch | +| 2026-07-06 | Interactive CLI | Editing config files is error-prone | `utils/cli.ts` with native readline | +| **2026-07-18** | **Gemma 4 E2B as production model** | **10-model evaluation completed** | **2x faster than Qwen 4B, half VRAM, perfect translation quality** | +| **2026-07-18** | **Gender decoupled from LLM** | **All 10 models failed gender for Romance languages** | **Kaikki Wiktionary lookup is deterministic and 100% accurate** | +| **2026-07-18** | **`--reasoning off` is mandatory** | **Qwen 3.5 and Gemma 4 "think" silently, consuming all tokens** | **Without this flag, output crashes with `finish_reason: length`** | +| **2026-07-18** | **KV cache quantization (`q4_0`)** | **8192 context needed for 20-word batches** | **Cuts KV VRAM by 75%, enables large batches on 4GB GPU, improves translation variety** | +| **2026-07-18** | **20-word batch size for local** | **Tested 4, 20 words** | **Sweet spot: no attention degradation, no JSON breakage, 10.9 tok/s** | +| **2026-07-18** | **Llama 3.2 discarded** | **Failed false friend tests** | **Translates "actual" (real) to cognates (aktuell/attuale) = "current". Dangerous for learners.** | +| **2026-07-18** | **`-c 8192` replaces `-c 2048`** | **20-word batches need more context** | **Combined with q4_0 KV cache, fits in 2.65GB VRAM** | --- ## 13. Known Issues & Dev Notes -### glossa-web (Frontend) - -| Issue | Details | -| ------------------------ | ----------------------------------------------------------------------------------------- | -| No healthcheck | Vite dev server has no health endpoint. Docker `HEALTHCHECK` cannot verify running state. | -| Valkey memory overcommit | Harmless warning in dev: `vm.overcommit_memory = 1` recommended before production. | - ### Data Pipeline -| Issue | Details | Severity | -| --------------------------------- | --------------------------------------------------------------------------- | ----------------------------------- | -| Ministral-3B tokenizer mismatch | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. | Blocker - abandoned | -| Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages. | Under evaluation | -| Pre-scanning wordlists | Entire file read into memory before processing. Inefficient for 100k words. | Medium - streaming refactor planned | -| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes. | Known limitation | +| Issue | Details | Severity | +| ------------------------------ | -------------------------------------------------------------------------------------- | ----------------------------------------------- | +| POS bleed (all models) | Models occasionally generate verb definitions for nouns (e.g., "run" = _to move fast_) | Medium — fix with negative constraint in prompt | +| Lazy duplication (Qwen 4B) | Qwen 3.5 4B copy-pastes the same translation twice to fill arrays | Medium — use Gemma 4 E2B instead | +| Schema hallucination (Qwen 4B) | Adds `sense_index` field not in schema | Low — ignorable | +| Pre-scanning wordlists | Entire file read into memory before processing | Medium — streaming refactor planned | +| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes | Known limitation | ### Hardware -| Issue | Details | -| --------------------- | ----------------------------------------------------- | -| GTX 950M VRAM ceiling | 4GB hard limit. Models >3GB risk OOM. | -| Maxwell GPU aging | No Flash Attention support, bandwidth-starved. | -| Laptop thermals | Cannot run 24/7 for weeks. Batch processing required. | - -### Refactor Notes - -| Note | File | Context | -| --------------------- | ------------- | -------------------------------------------------------------------------- | -| Streaming vs pre-scan | `pipeline.ts` | For 100k words, stream and batch on-the-fly instead of reading entire file | +| Issue | Details | +| --------------------- | ---------------------------------------------------------------------------- | +| GTX 950M VRAM ceiling | 4GB hard limit. Models >3.5GB need KV cache quantization or CPU offloading. | +| Maxwell GPU aging | No Flash Attention, bandwidth-starved (~32 GB/s). Theoretical max ~21 tok/s. | +| Laptop thermals | Cannot run 24/7 for weeks unattended. Monitor temps. | --- @@ -961,31 +706,27 @@ async function main() { - Node.js + npm - `tsx` installed globally: `npm install -g tsx` - llama.cpp built from source (for local mode) -- GGUF model downloaded to `~/Downloads/llama.cpp/models/` (for local mode) -- API keys set as environment variables (for cloud mode): - ```bash - export DEEPSEEK_API_KEY="sk-..." - export GEMINI_API_KEY="..." - export OPENROUTER_API_KEY="..." - ``` +- GGUF model downloaded to `~/Downloads/llama.cpp/models/` +- API keys set as environment variables (for cloud mode) ### Start the LLM Server (Local Mode) ```bash cd ~/Downloads/llama.cpp - ./build/bin/llama-server \ - -m models/qwen2.5-1.5b-instruct-q4_k_m.gguf \ + -m models/gemma-4-E2B_q4_0-it.gguf \ -ngl 999 \ - -c 2048 \ + -c 8192 \ -b 512 \ -ub 512 \ -t 2 \ --threads-batch 2 \ --host 127.0.0.1 \ --port 8080 \ - --mlock \ - --prio 2 + --prio 2 \ + --reasoning off \ + --cache-type-k q4_0 \ + --cache-type-v q4_0 ``` ### Run the Pipeline @@ -995,72 +736,7 @@ cd /path/to/data-pipeline npx tsx pipeline.ts ``` -Follow the interactive prompts to select provider, model, and batch size. Or select "Use last config" to reuse previous settings. - -### Run with Last Config - -```bash -cd /path/to/data-pipeline -npx tsx pipeline.ts -# Select [1] Use last config -``` - -### Expected Output - -``` -🌐 Lila Data Pipeline -───────────────────── - -[1] Use last config: local → qwen2.5-1.5b → batch 5 -[2] Configure new run - -> 1 - -🟢 Local AI engine is connected and ready for inference! - - step 1: scanning the source files... -✅ Scan complete! Found 1 wordlist(s): - • ENGLISH (nouns) - - step 2: creating necessary output folders... -✅ All required output directories have been verified and created successfully. - - step 3: verifying local AI engine status... -🟢 Local AI engine is connected and ready for inference! - - step 4: looping through the wordlists... - -Reading list: [ENGLISH] -> [NOUNS] - - Batch 1/1: [house, car, tree, water] - [1/4] (0 failed) Enriched and saved: house.json - [2/4] (0 failed) Enriched and saved: car.json - [3/4] (0 failed) Enriched and saved: tree.json - [4/4] (0 failed) Enriched and saved: water.json - ⏱️ Word took 8.2s - -⏱️ Pipeline Summary - Duration: 32.8s - Processed: 4 - Skipped: 0 - Failed: 0 - Total: 4 - Throughput: 0.12 words/sec - -🤖 LLM Metrics - Calls: 1 - Avg prompt tokens: 312 - Avg completion tokens: 524 - Avg total tokens: 836 - Avg total request time: 32800ms - Avg throughput: 25.5 tok/s - - [Local breakdown] - Avg prompt speed: 548.2 tok/s - Avg completion speed: 18.8 tok/s - -Global data pipeline run completed successfully. -``` +Follow the interactive prompts to select provider, model, and batch size. --- @@ -1068,82 +744,47 @@ Global data pipeline run completed successfully. ### Phase 1: Batching (Complete) -| Task | Status | Notes | -| ------------------------------------- | ------------ | ------------------------------------------------------------------------------------ | -| Implement configurable batch size | **Complete** | `config/batch.ts` with `size` and `maxRetries` | -| Implement retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch | -| Honest timing metrics | **Complete** | Unified throughput for all providers | -| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes | -| Verify batching doesn't break quality | Pending | Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output. | -| Measure speedup vs batch size | Pending | Track throughput at 1, 5, 15 on local hardware. | - -**Goal:** Unlock 5-15x speedup on local, unlock free online API tiers. - ---- +| Task | Status | +| --------------------------------------- | ------------ | +| Implement configurable batch size | Complete | +| Implement retry + split logic | Complete | +| Honest timing metrics | Complete | +| Validate LLM responses | Complete | +| Verify batching quality (20-word suite) | **Complete** | ### Phase 2: Interactive CLI (Complete) -| Task | Status | Notes | -| --------------------- | ------------ | ------------------------------------------ | -| Design prompt flow | **Complete** | Provider → model → batch size → confirm | -| Implement CLI module | **Complete** | `utils/cli.ts` with native `readline` | -| Save/load config | **Complete** | `.pipeline-config.json` | -| Wire into pipeline.ts | **Complete** | Replaces static config with runtime config | +| Task | Status | +| --------------------- | -------- | +| Design prompt flow | Complete | +| Implement CLI module | Complete | +| Save/load config | Complete | +| Wire into pipeline.ts | Complete | -**Goal:** No editing of TypeScript files to switch providers. +### Phase 3: Model Selection (Complete) ---- - -### Phase 3: Model Selection - -| Task | Status | Notes | -| ------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------- | -| Download Qwen2.5-3B Q4_K_M | Pending | ~1.9GB, fits in 4GB VRAM. | -| Run torture suite on Qwen2.5-3B (batched) | Pending | Check if gender accuracy improves with 2x params. | -| Test Gemini 2.5 Flash-Lite free tier (batched) | Pending | 1,500 req/day x 50 words = 75k words/day. Zero cost. | -| Test DeepSeek V4 Flash free tier (batched) | Pending | 5M tokens free. ~9k words. | -| Test Groq Llama 3.1 8B (batched, paid if needed) | Pending | Fastest option. ~$7 for 100k words. | -| Decide: local vs online, which model | Pending | Criteria: quality >= 90% gender, 100% JSON, sensible definitions. Then speed, then cost. | - -**Goal:** Pick the model and provider for the 100k word run. - ---- +| Task | Status | +| ----------------------- | -------------------------- | +| 10-model evaluation | **Complete** | +| 20-word torture suite | **Complete** | +| Select production model | **Complete (Gemma 4 E2B)** | +| Test online APIs | Pending | ### Phase 4: Scale -| Task | Status | Notes | -| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- | -| Run 100k word pipeline | Pending | Estimated time depends on Phase 3 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). | -| Spot-check output quality | Pending | Random sample of 100 entries. | -| Fix gender if needed | Pending | Kaikki lookup post-processing if LLM gender remains unreliable. | -| Handle failures & retries | Pending | Exponential backoff, split-and-retry for batch failures. | - -**Goal:** Complete 100k word dataset. - ---- +| Task | Status | Notes | +| ------------------------------ | ------- | ------------------------------------------ | +| Implement Kaikki gender lookup | Pending | Post-processing step | +| Fix POS bleed in prompt | Pending | Add negative constraint | +| Run 100k word pipeline | Pending | ~7 days local (Gemma E2B, 20-word batches) | +| Spot-check output quality | Pending | Random sample of 100 entries | ### Phase 5: Extend -| Task | Status | Notes | -| ---------------------------- | ------- | ------------------------------------------------------------------------------------------- | -| Multi-POS support | Pending | Verbs, adjectives, adverbs. Each needs prompt variants (conjugations, agreement, etc.). | -| Multi-language source | Pending | German -> French, Italian -> Spanish, etc. Schema already supports any source/target combo. | -| Parallel wordlist processing | Pending | Run `english/nouns` and `english/verbs` simultaneously. | -| Incremental enrichment | Pending | Only process new/changed words in a wordlist. | -| GPU rental integration | Pending | Script to spin up Vast.ai/RunPod, run pipeline, download results. | -| Quality regression tests | Pending | Run torture suite on every model change. | - -**Goal:** Generalize pipeline for any language direction and POS. - ---- - -### Backlog (Unscheduled) - -| Task | Context | -| -------------------------------- | ------------------------------------------------------------------------------------------- | -| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. | -| Model auto-switching | Fallback to online API if local server fails mid-run. | -| Community open-source | Clean up, document, publish for other language learners. | -| Prometheus metrics | `--metrics` flag on llama-server for automated performance tracking. | -| `-c 1024` / `-b 256` experiments | Further VRAM optimization on GTX 950M. Low priority if moving to cloud. | -| Streaming wordlist processing | Read file line-by-line and batch on-the-fly. Eliminates pre-scan memory usage. | +| Task | Status | +| ---------------------------- | ------- | +| Multi-POS support | Pending | +| Multi-language source | Pending | +| Parallel wordlist processing | Pending | +| GPU rental integration | Pending | +| Quality regression tests | Pending | diff --git a/repomix-output.xml b/repomix-output.xml deleted file mode 100644 index 38daccf..0000000 --- a/repomix-output.xml +++ /dev/null @@ -1,1987 +0,0 @@ -This file is a merged representation of the entire codebase, combined into a single document by Repomix. - - -This section contains a summary of this file. - - -This file contains a packed representation of the entire repository's contents. -It is designed to be easily consumable by AI systems for analysis, code review, -or other automated processes. - - - -The content is organized as follows: -1. This summary section -2. Repository information -3. Directory structure -4. Repository files (if enabled) -5. Multiple file entries, each consisting of: - - File path as an attribute - - Full contents of the file - - - -- This file should be treated as read-only. Any changes should be made to the - original repository files, not this packed version. -- When processing this file, use the file path to distinguish - between different files in the repository. -- Be aware that this file may contain sensitive information. Handle it with - the same level of security as you would the original repository. - - - -- Some files may have been excluded based on .gitignore rules and Repomix's configuration -- Binary files are not included in this packed representation. Please refer to the Repository Structure section for a complete list of file paths, including binary files -- Files matching patterns in .gitignore are excluded -- Files matching default ignore patterns are excluded -- Files are sorted by Git change count (files with more changes are at the bottom) - - - - - -config/ - batch.ts - constants.ts - llm.ts - prompt.ts - providers.ts -source-data/ - english/ - nouns -utils/ - llm-adapters/ - factory.ts - gemini.ts - openai-compatible.ts - types.ts - check-if-json-exists.ts - check-llm-server.ts - cli.ts - create-base-json.ts - create-line-reader.ts - create-output-dirs.ts - delete-file.ts - enrich-word.ts - get-word-file-path.ts - merge-enriched-data.ts - pipeline-timer.ts - progress-tracker.ts - scanning-source-files.ts - verify-enriched-file.ts - write-json-file.ts -.env.example -.pipeline-config.json -package.json -pipeline.ts -tsconfig.json -vitest.config.ts - - - -This section contains the contents of the repository's files. - - -// Runtime-populated by pipeline.ts after CLI initialization -export const BATCH_CONFIG = { size: 4, maxRetries: 3 }; - - - -export const LANG_MAP: Record = { - english: "en", - italian: "it", - german: "de", - french: "fr", - spanish: "es", -}; - -export const POS_MAP: Record = { - nouns: "noun", - verbs: "verb", - adverbs: "adverb", - adjectives: "adjective", -}; - -export const ALL_LANGUAGES = ["en", "de", "it", "es", "fr"]; - - - -import type { OnlineProvider } from "./providers.js"; - -export type LlmProvider = "local" | OnlineProvider; - -// Runtime-populated by pipeline.ts after CLI initialization -export const LLM_CONFIG = { - provider: "local" as LlmProvider, - url: "http://127.0.0.1:8080/v1/chat/completions", - model: undefined as string | undefined, -}; - - - -export function buildSystemPrompt( - sourceLanguage: string, - pos: string, - targetLanguages: string[], -): string { - return `You are a multilingual dictionary engine. Output ONLY a JSON object. No markdown, no explanations. - -For each ${sourceLanguage} ${pos} provided, generate 1-2 distinct senses. - -CEFR difficulty mapping: -- A1/A2 → easy -- B1/B2 → medium -- C1/C2 → hard - -Each sense must have: -- sense: student-friendly definition, max 15 words -- example: natural sentence using the word -- difficulty_level: easy, medium, or hard -- translations: object with keys ${targetLanguages.join(", ")}; each value is an array of {word, gender} where gender MUST be masculine, feminine, or neuter. Use null ONLY if the language has no grammatical gender for that word. - -Output format: JSON object where keys are the input words, values are arrays of sense objects. - -Example for ["house"]: -{ - "house": [ - { - "sense": "A building for human habitation.", - "example": "They bought a house in the city.", - "difficulty_level": "easy", - "translations": { - "de": [{"word": "Haus", "gender": "neuter"}], - "it": [{"word": "casa", "gender": "feminine"}], - "es": [{"word": "casa", "gender": "feminine"}], - "fr": [{"word": "maison", "gender": "feminine"}] - } - } - ] -} -`; -} - - - -export type ProviderMeta = { - name: string; - envVar: string; - url: string; - requiresKey: boolean; - models: string[]; -}; - -export const ONLINE_PROVIDERS: Record = { - gemini: { - name: "Gemini", - envVar: "GEMINI_API_KEY", - url: "https://generativelanguage.googleapis.com/v1beta", - requiresKey: true, - models: ["gemini-2.5-flash", "gemini-2.5-pro"], - }, - deepseek: { - name: "DeepSeek", - envVar: "DEEPSEEK_API_KEY", - url: "https://api.deepseek.com/v1/chat/completions", - requiresKey: true, - models: ["deepseek-chat", "deepseek-reasoner"], - }, - openrouter: { - name: "OpenRouter", - envVar: "OPENROUTER_API_KEY", - url: "https://openrouter.ai/api/v1/chat/completions", - requiresKey: true, - models: [ - "openai/gpt-oss-120b:free", - "google/gemma-4-31b-it:free", - "qwen/qwen3-next-80b-a3b-instruct:free", - "meta-llama/llama-3.3-70b-instruct:free", - "anthropic/claude-sonnet-4", - "google/gemini-2.5-flash", - "deepseek/deepseek-chat-v3", - ], - }, - groq: { - name: "Groq", - envVar: "GROQ_API_KEY", - url: "https://api.groq.com/openai/v1/chat/completions", - requiresKey: true, - models: ["llama-3.3-70b-versatile", "gemma2-9b-it", "mixtral-8x7b-32768"], - }, -} as const; - -export const LOCAL_PROVIDER: ProviderMeta = { - name: "Local (llama.cpp / ollama / lm-studio)", - envVar: "", - url: "http://127.0.0.1:8080/v1/chat/completions", - requiresKey: false, - models: [], -}; - -export type OnlineProvider = keyof typeof ONLINE_PROVIDERS; - - - -house -time -water -year -people -day -way -man -woman -child -work -life -world -hand -eye -book -friend -school -city -family - - - -import { LLM_CONFIG } from "../../config/llm.js"; -import { OpenAiCompatibleAdapter } from "./openai-compatible.js"; -import { GeminiAdapter } from "./gemini.js"; -import type { LlmAdapter } from "./types.js"; - -export function createAdapter(): LlmAdapter { - switch (LLM_CONFIG.provider) { - case "local": - return new OpenAiCompatibleAdapter( - LLM_CONFIG.url, - undefined, - LLM_CONFIG.model, - ); - case "openrouter": - return new OpenAiCompatibleAdapter( - LLM_CONFIG.url, - process.env["OPENROUTER_API_KEY"], - LLM_CONFIG.model, - ); - case "deepseek": - return new OpenAiCompatibleAdapter( - LLM_CONFIG.url, - process.env["DEEPSEEK_API_KEY"], - LLM_CONFIG.model, - ); - case "groq": - return new OpenAiCompatibleAdapter( - LLM_CONFIG.url, - process.env["GROQ_API_KEY"], - LLM_CONFIG.model, - ); - case "gemini": { - const apiKey = process.env["GEMINI_API_KEY"]; - if (!apiKey) throw new Error("GEMINI_API_KEY env var not set"); - if (!LLM_CONFIG.model) - throw new Error("LLM_CONFIG.model required for gemini"); - return new GeminiAdapter(apiKey, LLM_CONFIG.model); - } - default: - throw new Error(`Unknown provider: ${LLM_CONFIG.provider}`); - } -} - - - -import type { LlmAdapter } from "./types.js"; - -interface GeminiResponse { - candidates: Array<{ content: { parts: Array<{ text: string }> } }>; - usageMetadata: { - promptTokenCount: number; - candidatesTokenCount: number; - totalTokenCount: number; - }; -} - -export class GeminiAdapter implements LlmAdapter { - private apiKey: string; - private model: string; - - constructor(apiKey: string, model: string) { - this.apiKey = apiKey; - this.model = model; - } - - async call( - words: string[], - systemPrompt: string, - ): Promise<{ - content: string; - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; - }> { - const url = `https://generativelanguage.googleapis.com/v1beta/models/${this.model}:generateContent?key=${this.apiKey}`; - - const payload = { - systemInstruction: { parts: [{ text: systemPrompt }] }, - contents: [ - { role: "user", parts: [{ text: "Words: " + JSON.stringify(words) }] }, - ], - generationConfig: { - temperature: 0.1, - topP: 0.9, - maxOutputTokens: Math.ceil(words.length * 250 * 1.2), - }, - }; - - const startTime = Date.now(); - - const response = await fetch(url, { - method: "POST", - headers: { "Content-Type": "application/json" }, - body: JSON.stringify(payload), - }); - - const totalTimeMs = Date.now() - startTime; - - if (!response.ok) { - throw new Error(`Gemini API responded with status: ${response.status}`); - } - - const json = (await response.json()) as GeminiResponse; - - const content = json.candidates[0]?.content?.parts[0]?.text; - if (!content) { - throw new Error("Gemini response content is empty"); - } - - const promptTokens = json.usageMetadata.promptTokenCount; - const completionTokens = json.usageMetadata.candidatesTokenCount; - - return { - content, - promptTokens, - completionTokens, - totalTokens: json.usageMetadata.totalTokenCount, - promptTimeMs: null, - completionTimeMs: null, - totalTimeMs, - }; - } -} - - - -import type { LlmAdapter } from "./types.js"; - -interface OpenAiResponse { - choices: Array<{ message: { content: string } }>; - usage: { - prompt_tokens: number; - completion_tokens: number; - total_tokens: number; - }; - timings?: { prompt_ms: number; predicted_ms: number }; -} - -export class OpenAiCompatibleAdapter implements LlmAdapter { - private url: string; - private apiKey: string | undefined; - private model: string | undefined; - - constructor(url: string, apiKey?: string, model?: string) { - this.url = url; - this.apiKey = apiKey; - this.model = model; - } - - async call( - words: string[], - systemPrompt: string, - ): Promise<{ - content: string; - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; - }> { - const payload: Record = { - messages: [ - { role: "system", content: systemPrompt }, - { role: "user", content: JSON.stringify(words) }, - ], - temperature: 0.1, - top_p: 0.9, - max_tokens: Math.ceil(words.length * 250 * 1.2), - }; - - if (this.model) { - payload["model"] = this.model; - } - - const headers: Record = { - "Content-Type": "application/json", - }; - - if (this.apiKey) { - headers["Authorization"] = `Bearer ${this.apiKey}`; - } - - const startTime = Date.now(); - - const response = await fetch(this.url, { - method: "POST", - headers, - body: JSON.stringify(payload), - }); - - const totalTimeMs = Date.now() - startTime; - - if (!response.ok) { - throw new Error(`LLM server responded with status: ${response.status}`); - } - - const json = (await response.json()) as OpenAiResponse; - - const content = json.choices[0]?.message?.content; - if (!content) { - throw new Error("LLM response content is empty"); - } - - const promptTokens = json.usage.prompt_tokens; - const completionTokens = json.usage.completion_tokens; - - return { - content, - promptTokens, - completionTokens, - totalTokens: json.usage.total_tokens, - promptTimeMs: json.timings?.prompt_ms ?? null, - completionTimeMs: json.timings?.predicted_ms ?? null, - totalTimeMs, - }; - } -} - - - -export interface LlmAdapter { - call( - words: string[], - systemPrompt: string, - ): Promise<{ - content: string; - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; - }>; -} - - - -import fs from "fs"; -import path from "path"; - -/** - * Checks if a JSON file for the given word exists AND contains enriched data. - * Returns false for skeleton files (missing senses array). - */ -export function isWordProcessed(word: string, outputDir: string): boolean { - const targetFilePath = path.join(outputDir, `${word}.json`); - - if (!fs.existsSync(targetFilePath)) { - return false; - } - - try { - const content = fs.readFileSync(targetFilePath, "utf-8"); - const data = JSON.parse(content) as Record; - return Array.isArray(data["senses"]) && data["senses"].length > 0; - } catch (_error: unknown) { - // Corrupted file => treat as not processed - return false; - } -} - - - -import { LLM_CONFIG } from "../config/llm.js"; - -/** - * Pings the local llama.cpp server to ensure it's up, running, and has a model loaded. - * If the server is offline or still loading, it terminates the pipeline gracefully. - * Skipped entirely when using a cloud provider. - */ -export async function checkLlmServer( - url = "http://127.0.0.1:8080/health", -): Promise { - if (LLM_CONFIG.provider !== "local") { - console.log("🌐 Using cloud provider — skipping local health check."); - return; - } - - try { - const response = await fetch(url); - - // llama.cpp returns a 503 status if the server is up but the model weights are still loading - if (response.status === 503) { - throw new Error( - "Local AI engine is starting up, but the model is still loading into memory. " + - "Please wait a minute for the weights to load, then run the pipeline again.", - ); - } - - // Parse the JSON health response (expected: { status: "ok" }) - const data = (await response.json()) as { status?: string }; - - if (response.ok && data.status === "ok") { - console.log("🟢 Local AI engine is connected and ready for inference!"); - return; - } - - // Catch-all for unexpected active server responses - throw new Error( - `Unknown response from local AI engine health check (Status: ${response.status}).`, - ); - } catch (error: unknown) { - if (error instanceof Error && error.message.includes("Local AI engine")) { - throw error; // Re-throw our own errors - } - throw new Error( - `Could not connect to the local AI engine at ${url}. ` + - "Make sure your './llama-server' command is actively running in another terminal tab.", - { cause: error }, - ); - } -} - - - -import { createInterface } from "node:readline"; -import { existsSync, readFileSync, writeFileSync } from "node:fs"; -import { join } from "node:path"; -import { - ONLINE_PROVIDERS, - LOCAL_PROVIDER, - type OnlineProvider, -} from "../config/providers.js"; - -// ── Types ────────────────────────────────────────────────────────────────── - -export interface PipelineConfig { - provider: "local" | OnlineProvider; - url: string; - model: string | undefined; - batchSize: number; - maxRetries: number; -} - -interface SavedConfig { - provider: PipelineConfig["provider"]; - model: string; - batchSize: number; - maxRetries: number; -} - -// ── Helpers ──────────────────────────────────────────────────────────────── - -function getConfigPath(): string { - return join(import.meta.dirname, "..", ".pipeline-config.json"); -} - -function loadLastConfig(): SavedConfig | null { - const path = getConfigPath(); - if (!existsSync(path)) return null; - try { - const raw = readFileSync(path, "utf-8"); - return JSON.parse(raw) as SavedConfig; - } catch { - return null; - } -} - -function saveConfig(config: SavedConfig): void { - writeFileSync(getConfigPath(), JSON.stringify(config, null, 2)); -} - -function ask( - rl: ReturnType, - prompt: string, -): Promise { - return new Promise((resolve) => { - rl.question(prompt, resolve); - }); -} - -function printLine(char = "─", length = 50): void { - console.log(char.repeat(length)); -} - -function formatProviderLabel(p: string): string { - const meta = p === "local" ? LOCAL_PROVIDER : ONLINE_PROVIDERS[p]; - return meta ? meta.name : p; -} - -// ── Validation ───────────────────────────────────────────────────────────── - -function validateBatchSize(input: string): number { - const n = parseInt(input.trim(), 10); - if (Number.isNaN(n) || n < 1 || n > 20) { - throw new Error("Batch size must be an integer between 1 and 20"); - } - return n; -} - -function checkApiKey(provider: string): void { - const meta = ONLINE_PROVIDERS[provider]; - if (!meta) return; - const key = process.env[meta.envVar]; - if (!key) { - console.error(`\n ❌ Missing API key: ${meta.envVar} is not set.`); - console.error(` Export it before running the pipeline:`); - console.error(` export ${meta.envVar}=your_key_here\n`); - process.exit(1); - } -} - -// ── Prompt flows ──────────────────────────────────────────────────────────── - -async function promptProviderType( - rl: ReturnType, -): Promise<"local" | "online"> { - console.log("\nSelect provider type:"); - console.log(" [1] Local (llama.cpp, ollama, lm-studio, etc.)"); - console.log(" [2] Online API (Gemini, DeepSeek, OpenRouter, Groq)"); - - while (true) { - const choice = (await ask(rl, "Choice [1/2]: ")).trim(); - if (choice === "1") return "local"; - if (choice === "2") return "online"; - console.log(" Invalid choice. Enter 1 or 2."); - } -} - -async function promptOnlineProvider( - rl: ReturnType, -): Promise { - console.log("\nSelect online provider:"); - const entries = Object.entries(ONLINE_PROVIDERS); - entries.forEach(([_key, meta], i) => { - const hasKey = process.env[meta.envVar] ? "✓" : "✗"; - console.log(` [${i + 1}] ${meta.name} (${hasKey} ${meta.envVar})`); - }); - - while (true) { - const choice = (await ask(rl, `Choice [1-${entries.length}]: `)).trim(); - const idx = parseInt(choice, 10) - 1; - if (idx >= 0 && idx < entries.length) { - const entry = entries[idx]!; - const provider = entry[0]; - checkApiKey(provider); - return provider; - } - console.log(` Invalid choice. Enter 1-${entries.length}.`); - } -} - -async function promptModel( - rl: ReturnType, - provider: string, -): Promise { - if (provider === "local") { - console.log("\nLocal provider selected."); - console.log(" Using: http://127.0.0.1:8080/v1/chat/completions"); - const model = ( - await ask(rl, "Model name (optional, press Enter to skip): ") - ).trim(); - return model || "local-model"; - } - - const meta = ONLINE_PROVIDERS[provider]; - if (!meta) { - throw new Error(`Unknown provider: ${provider}`); - } - - console.log(`\nSelect model for ${meta.name}:`); - meta.models.forEach((m, i) => console.log(` [${i + 1}] ${m}`)); - console.log(` [${meta.models.length + 1}] Other (type manually)`); - - while (true) { - const choice = ( - await ask(rl, `Choice [1-${meta.models.length + 1}]: `) - ).trim(); - const idx = parseInt(choice, 10) - 1; - if (idx >= 0 && idx < meta.models.length) { - return meta.models[idx]!; - } - if (idx === meta.models.length) { - const custom = (await ask(rl, "Enter model name: ")).trim(); - if (custom) return custom; - console.log(" Model name cannot be empty."); - continue; - } - console.log(` Invalid choice. Enter 1-${meta.models.length + 1}.`); - } -} - -async function promptBatchSize( - rl: ReturnType, -): Promise { - console.log("\nBatch size: how many words to enrich per LLM call."); - console.log(" Recommended: 2–6 for complex languages, 4–8 for simple."); - - while (true) { - const input = (await ask(rl, "Batch size [1-20, default 4]: ")).trim(); - if (!input) return 4; - try { - return validateBatchSize(input); - } catch (err) { - console.log(` ${(err as Error).message}`); - } - } -} - -async function promptConfirm( - rl: ReturnType, - config: PipelineConfig, -): Promise { - console.log("\n"); - printLine(); - console.log(" CONFIGURATION SUMMARY"); - printLine(); - console.log(` Provider: ${formatProviderLabel(config.provider)}`); - console.log(` URL: ${config.url}`); - console.log(` Model: ${config.model ?? "(none)"}`); - console.log(` Batch: ${config.batchSize} words/call`); - console.log(` Retries: ${config.maxRetries}`); - printLine(); - - const answer = (await ask(rl, "\nProceed with this configuration? [Y/n]: ")) - .trim() - .toLowerCase(); - return answer === "" || answer === "y" || answer === "yes"; -} - -// ── Main export ──────────────────────────────────────────────────────────── - -export async function runCli(): Promise { - const rl = createInterface({ input: process.stdin, output: process.stdout }); - - try { - const lastConfig = loadLastConfig(); - - // ── Startup menu ───────────────────────────────────────────────────────── - console.log("\n"); - printLine("═", 50); - console.log(" PIPELINE CONFIGURATION"); - printLine("═", 50); - - if (lastConfig) { - console.log("\nLast used configuration:"); - console.log(` Provider: ${formatProviderLabel(lastConfig.provider)}`); - console.log(` Model: ${lastConfig.model}`); - console.log(` Batch: ${lastConfig.batchSize}`); - } else { - console.log("\nNo previous configuration found."); - } - - console.log( - "\n[1] Use last config" + - (lastConfig ? "" : " (not available)") + - " [2] Configure new run", - ); - - let useLast = false; - if (lastConfig) { - while (true) { - const choice = (await ask(rl, "Choice [1/2]: ")).trim(); - if (choice === "1") { - useLast = true; - break; - } - if (choice === "2") break; - console.log(" Invalid choice. Enter 1 or 2."); - } - } else { - // No last config, auto-select new run - console.log("Auto-selecting: Configure new run"); - await ask(rl, "Press Enter to continue..."); - } - - // ── Build config ──────────────────────────────────────────────────────── - let config: PipelineConfig; - - if (useLast && lastConfig) { - // Re-validate API key before reusing - if (lastConfig.provider !== "local") { - checkApiKey(lastConfig.provider); - } - - const meta = - lastConfig.provider === "local" - ? LOCAL_PROVIDER - : ONLINE_PROVIDERS[lastConfig.provider]; - - config = { - provider: lastConfig.provider, - url: meta?.url ?? LOCAL_PROVIDER.url, - model: lastConfig.model, - batchSize: lastConfig.batchSize, - maxRetries: lastConfig.maxRetries, - }; - } else { - // New run flow - const providerType = await promptProviderType(rl); - - let provider: string; - let url: string; - - if (providerType === "local") { - provider = "local"; - url = LOCAL_PROVIDER.url; - } else { - provider = await promptOnlineProvider(rl); - url = ONLINE_PROVIDERS[provider]!.url; - } - - const model = await promptModel(rl, provider); - const batchSize = await promptBatchSize(rl); - - config = { - provider: provider, - url, - model: model || undefined, - batchSize, - maxRetries: 3, - }; - - // Confirm before saving - const confirmed = await promptConfirm(rl, config); - if (!confirmed) { - console.log("\n ❌ Configuration cancelled. Exiting.\n"); - process.exit(0); - } - - // Save for next time - saveConfig({ - provider: config.provider, - model: config.model ?? "", - batchSize: config.batchSize, - maxRetries: config.maxRetries, - }); - console.log("\n ✓ Configuration saved to .pipeline-config.json"); - } - - console.log("\n"); - return config; - } finally { - rl.close(); - } -} - - - -import fs from "fs"; -import path from "path"; -import { LANG_MAP, POS_MAP } from "../config/constants.js"; - -/** - * Creates the base JSON file with word, language, and pos. - * No logging — the orchestrator handles all console output. - */ -export function createBaseJson( - word: string, - outputDir: string, - rawLanguage: string, - rawPos: string, -): void { - const targetFilePath = path.join(outputDir, `${word}.json`); - - const dbLanguage = LANG_MAP[rawLanguage] || rawLanguage; - const dbPos = POS_MAP[rawPos] || rawPos; - - const initialData = { word, language: dbLanguage, pos: dbPos }; - - fs.writeFileSync( - targetFilePath, - JSON.stringify(initialData, null, 2), - "utf-8", - ); -} - - - -import fs from "fs"; -import readline from "readline"; - -/** - * Creates a line-by-line reader stream for a given file path. - */ -export function createLineReader(sourcePath: string): readline.Interface { - const fileStream = fs.createReadStream(sourcePath, "utf-8"); - - return readline.createInterface({ input: fileStream, crlfDelay: Infinity }); -} - - - -import fs from "fs"; -import type { Wordlist } from "./scanning-source-files.js"; - -/** - * Takes a list of scanned datasets and creates their output folders if missing. - */ -export function ensureOutputFolders(wordlists: Wordlist[]): void { - for (const wordlist of wordlists) { - if (!fs.existsSync(wordlist.outputDir)) { - fs.mkdirSync(wordlist.outputDir, { recursive: true }); - console.log( - `📁 Created target folder: worddata/${wordlist.language}/${wordlist.pos}`, - ); - } - } - - console.log( - "✅ All required output directories have been verified and created successfully.", - ); -} - - - -import fs from "fs"; - -/** - * Deletes a file if it exists. Silently ignores missing files. - */ -export function deleteFileIfExists(filePath: string): void { - if (fs.existsSync(filePath)) { - fs.unlinkSync(filePath); - } -} - - - -import { buildSystemPrompt } from "../config/prompt.js"; -import { createAdapter } from "./llm-adapters/factory.js"; -import { BATCH_CONFIG } from "../config/batch.js"; -import type { Language, Pos, EnrichedSense } from "./merge-enriched-data.js"; -import { LANG_MAP, POS_MAP, ALL_LANGUAGES } from "../config/constants.js"; - -interface LlmResponse { - content: string; - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; -} - -export interface EnrichmentResult { - results: Map; - metrics: { - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; - }; -} - -/** - * Calls the LLM with the enrichment prompt. - * Returns the response content and timing metrics. - */ -async function callLlm( - words: string[], - rawLanguage: string, - rawPos: string, -): Promise { - const adapter = createAdapter(); - const sourceCode = LANG_MAP[rawLanguage] || rawLanguage; - const targetLanguages = ALL_LANGUAGES.filter((lang) => lang !== sourceCode); - const prompt = buildSystemPrompt(rawLanguage, rawPos, targetLanguages); - return adapter.call(words, prompt); -} - -/** - * Strips markdown code blocks and extracts the JSON object from raw LLM output. - * Throws if no valid JSON object braces are found. - */ -function sanitizeLlmOutput(raw: string): string { - const cleaned = raw.replace(/```json\s*/g, "").replace(/```\s*$/g, ""); - const start = cleaned.indexOf("{"); - const end = cleaned.lastIndexOf("}"); - if (start === -1 || end === -1) { - throw new Error("No JSON object found in LLM output"); - } - return cleaned.slice(start, end + 1); -} - -function validateSense(item: unknown, word: string, index: number): void { - if (typeof item !== "object" || item === null || Array.isArray(item)) { - throw new Error(`Sense ${index} for "${word}" is not an object`); - } - - const sense = item as Record; - - if (typeof sense["sense"] !== "string" || !sense["sense"]) { - throw new Error(`Sense ${index} for "${word}": missing or invalid "sense"`); - } - if (typeof sense["example"] !== "string" || !sense["example"]) { - throw new Error( - `Sense ${index} for "${word}": missing or invalid "example"`, - ); - } - if ( - !["easy", "medium", "hard"].includes(sense["difficulty_level"] as string) - ) { - throw new Error(`Sense ${index} for "${word}": invalid "difficulty_level"`); - } - if ( - typeof sense["translations"] !== "object" || - sense["translations"] === null - ) { - throw new Error(`Sense ${index} for "${word}": missing "translations"`); - } - - const trans = sense["translations"] as Record; - for (const lang of ["de", "it", "es", "fr"]) { - if (!Array.isArray(trans[lang])) { - throw new Error( - `Sense ${index} for "${word}": missing or invalid "${lang}" translations`, - ); - } - for (let j = 0; j < (trans[lang] as unknown[]).length; j++) { - const t = (trans[lang] as unknown[])[j] as Record; - if (typeof t["word"] !== "string" || !t["word"]) { - throw new Error( - `Sense ${index} for "${word}": ${lang}[${j}] missing "word"`, - ); - } - if ( - !["masculine", "feminine", "neuter", null].includes( - t["gender"] as string | null, - ) - ) { - throw new Error( - `Sense ${index} for "${word}": ${lang}[${j}] invalid "gender"`, - ); - } - } - } -} - -/** - * Parses the LLM response string into a JavaScript object. - * Throws if the response is not valid JSON or not an object with expected keys. - */ -export function parseLlmResponse( - rawJson: string, - expectedWords: string[], -): Record { - let parsed: unknown; - - try { - const sanitized = sanitizeLlmOutput(rawJson); - parsed = JSON.parse(sanitized); - } catch (error: unknown) { - throw new Error(`Failed to parse LLM output as JSON: ${rawJson}`, { - cause: error, - }); - } - - if (typeof parsed !== "object" || parsed === null || Array.isArray(parsed)) { - throw new Error("LLM output is not a JSON object"); - } - - const obj = parsed as Record; - - for (const word of expectedWords) { - if (!(word in obj)) { - throw new Error(`Missing key in LLM output: "${word}"`); - } - if (!Array.isArray(obj[word]) || (obj[word] as unknown[]).length === 0) { - throw new Error(`LLM output for "${word}" is not a non-empty array`); - } - - // Validate each sense in the array - const senses = obj[word] as unknown[]; - for (let i = 0; i < senses.length; i++) { - validateSense(senses[i], word, i); - } - } - - return obj; -} - -/** - * Takes parsed LLM output and builds final enriched objects with composite IDs. - */ -export function buildEnrichedData( - parsed: Record, - rawLanguage: string, - rawPos: string, -): Map { - const language = (LANG_MAP[rawLanguage] || rawLanguage) as Language; - const pos = (POS_MAP[rawPos] || rawPos) as Pos; - const results = new Map(); - - for (const [word, sensesArray] of Object.entries(parsed)) { - const senses = (sensesArray as unknown[]).map((item, index) => { - const sense = item as Omit< - EnrichedSense, - "id" | "word" | "language" | "pos" - >; - - return { - id: `${word}:${language}:${pos}:${index}`, - word, - language, - pos, - ...sense, - } as EnrichedSense; - }); - - results.set(word, senses); - } - - return results; -} - -/** - * Enriches a batch of words by calling the LLM, parsing the response, and building final data. - */ -export async function enrichWord( - words: string[], - rawLanguage: string, - rawPos: string, -): Promise { - const llmResponse = await callLlm(words, rawLanguage, rawPos); - const parsed = parseLlmResponse(llmResponse.content, words); - const results = buildEnrichedData(parsed, rawLanguage, rawPos); - - return { - results, - metrics: { - promptTokens: llmResponse.promptTokens, - completionTokens: llmResponse.completionTokens, - totalTokens: llmResponse.totalTokens, - promptTimeMs: llmResponse.promptTimeMs, - completionTimeMs: llmResponse.completionTimeMs, - totalTimeMs: llmResponse.totalTimeMs, - }, - }; -} - -/** - * Enriches a batch of words with retry and split-on-failure logic. - * Retries up to BATCH_CONFIG.maxRetries times, then splits batch in half and retries each half. - * Continues splitting until batch size is 1, then throws if still failing. - */ -export async function enrichWordWithRetry( - words: string[], - rawLanguage: string, - rawPos: string, - attempt: number = 1, -): Promise { - try { - return await enrichWord(words, rawLanguage, rawPos); - } catch (error: unknown) { - const errorMessage = error instanceof Error ? error.message : String(error); - - if (words.length === 1) { - throw new Error( - `Failed to enrich word "${words[0]}" after ${attempt} attempts: ${errorMessage}`, - { cause: error }, - ); - } - - if (attempt < BATCH_CONFIG.maxRetries) { - console.log( - ` Retry ${attempt}/${BATCH_CONFIG.maxRetries} for batch [${words.join(", ")}]: ${errorMessage}`, - ); - return enrichWordWithRetry(words, rawLanguage, rawPos, attempt + 1); - } - - // Max retries reached — split and retry - console.log( - ` Splitting batch [${words.join(", ")}] after ${BATCH_CONFIG.maxRetries} failed attempts`, - ); - - const half = Math.ceil(words.length / 2); - const left = words.slice(0, half); - const right = words.slice(half); - - const leftResult = await enrichWordWithRetry(left, rawLanguage, rawPos, 1); - const rightResult = await enrichWordWithRetry( - right, - rawLanguage, - rawPos, - 1, - ); - - // Merge results - const merged = new Map([...leftResult.results, ...rightResult.results]); - const mergedMetrics = { - promptTokens: - leftResult.metrics.promptTokens + rightResult.metrics.promptTokens, - completionTokens: - leftResult.metrics.completionTokens + - rightResult.metrics.completionTokens, - totalTokens: - leftResult.metrics.totalTokens + rightResult.metrics.totalTokens, - promptTimeMs: - (leftResult.metrics.promptTimeMs ?? 0) + - (rightResult.metrics.promptTimeMs ?? 0), - completionTimeMs: - (leftResult.metrics.completionTimeMs ?? 0) + - (rightResult.metrics.completionTimeMs ?? 0), - totalTimeMs: - leftResult.metrics.totalTimeMs + rightResult.metrics.totalTimeMs, - }; - - return { results: merged, metrics: mergedMetrics }; - } -} - - - -import path from "path"; - -export function getWordFilePath(word: string, outputDir: string): string { - return path.join(outputDir, `${word}.json`); -} - - - -import { LLM_CONFIG } from "../config/llm.js"; - -export type Language = "en" | "de" | "it" | "es" | "fr"; -export type Pos = "noun" | "verb" | "adjective" | "adverb"; -export type Gender = "masculine" | "feminine" | "neuter" | null; -export type Difficulty = "easy" | "medium" | "hard"; - -export interface Translation { - word: string; - gender: Gender; -} - -export interface EnrichedSense { - id: string; - word: string; - language: Language; - pos: Pos; - sense: string; - example: string; - difficulty_level: Difficulty; - translations: { - de: Translation[]; - it: Translation[]; - es: Translation[]; - fr: Translation[]; - }; -} - -/** - * Merges skeleton data with enriched LLM senses into the final pipeline output. - */ -export function mergeEnrichedData( - word: string, - senses: EnrichedSense[], -): Record { - return { - word, - language: senses[0]?.language ?? "en", - pos: senses[0]?.pos ?? "noun", - senses, - enrichedAt: new Date().toISOString(), - model: LLM_CONFIG.model ?? "unknown", - }; -} - - - -interface LlmMetrics { - promptTokens: number; - completionTokens: number; - totalTokens: number; - promptTimeMs: number | null; - completionTimeMs: number | null; - totalTimeMs: number; -} - -interface PipelineMetrics { - startTime: Date; - endTime?: Date; - wordsProcessed: number; - wordsSkipped: number; - wordsFailed: number; - llmCalls: number; - totalPromptTokens: number; - totalCompletionTokens: number; - totalTokens: number; - totalPromptTimeMs: number; - totalCompletionTimeMs: number; - totalTimeMs: number; - currentWordStartTime?: Date; -} - -/** - * Simple timer and metrics tracker for the pipeline. - * Tracks both pipeline throughput and LLM performance. - */ -export class PipelineTimer { - private metrics: PipelineMetrics; - - constructor() { - this.metrics = { - startTime: new Date(), - wordsProcessed: 0, - wordsSkipped: 0, - wordsFailed: 0, - llmCalls: 0, - totalPromptTokens: 0, - totalCompletionTokens: 0, - totalTokens: 0, - totalPromptTimeMs: 0, - totalCompletionTimeMs: 0, - totalTimeMs: 0, - }; - } - - startWord(): void { - this.metrics.currentWordStartTime = new Date(); - } - - getWordDurationMs(): number { - if (!this.metrics.currentWordStartTime) return 0; - return new Date().getTime() - this.metrics.currentWordStartTime.getTime(); - } - - recordProcessed(llmMetrics?: LlmMetrics): void { - this.metrics.wordsProcessed++; - if (llmMetrics) { - this.metrics.llmCalls++; - this.metrics.totalPromptTokens += llmMetrics.promptTokens; - this.metrics.totalCompletionTokens += llmMetrics.completionTokens; - this.metrics.totalTokens += llmMetrics.totalTokens; - if (llmMetrics.promptTimeMs !== null) { - this.metrics.totalPromptTimeMs += llmMetrics.promptTimeMs; - } - if (llmMetrics.completionTimeMs !== null) { - this.metrics.totalCompletionTimeMs += llmMetrics.completionTimeMs; - } - this.metrics.totalTimeMs += llmMetrics.totalTimeMs; - } - } - - recordSkipped(): void { - this.metrics.wordsSkipped++; - } - - recordFailed(): void { - this.metrics.wordsFailed++; - } - - stop(): void { - this.metrics.endTime = new Date(); - } - - getWordTiming(): string { - const durationMs = this.getWordDurationMs(); - const durationSec = (durationMs / 1000).toFixed(1); - return `⏱️ Word took ${durationSec}s`; - } - - getEta(totalWords: number): string { - const processed = this.metrics.wordsProcessed; - const remaining = totalWords - processed - this.metrics.wordsSkipped; - - if (processed === 0 || remaining <= 0) return "ETA: calculating..."; - - const elapsedMs = new Date().getTime() - this.metrics.startTime.getTime(); - const avgMsPerWord = elapsedMs / processed; - const etaMs = avgMsPerWord * remaining; - - const etaMin = Math.round(etaMs / 60000); - const etaHour = (etaMs / 3600000).toFixed(1); - - if (etaMin < 60) { - return `ETA: ${etaMin} min`; - } - return `ETA: ${etaHour} hours`; - } - - getSummary(): string { - const end = this.metrics.endTime || new Date(); - const durationMs = end.getTime() - this.metrics.startTime.getTime(); - const durationSec = (durationMs / 1000).toFixed(1); - - const total = - this.metrics.wordsProcessed + - this.metrics.wordsSkipped + - this.metrics.wordsFailed; - const throughput = - this.metrics.wordsProcessed > 0 - ? (this.metrics.wordsProcessed / (durationMs / 1000)).toFixed(2) - : "0"; - - const avgPromptTokens = - this.metrics.llmCalls > 0 - ? (this.metrics.totalPromptTokens / this.metrics.llmCalls).toFixed(0) - : "0"; - const avgCompletionTokens = - this.metrics.llmCalls > 0 - ? (this.metrics.totalCompletionTokens / this.metrics.llmCalls).toFixed( - 0, - ) - : "0"; - const avgTotalTimeMs = - this.metrics.llmCalls > 0 - ? (this.metrics.totalTimeMs / this.metrics.llmCalls).toFixed(0) - : "0"; - - const unifiedThroughput = - this.metrics.totalTimeMs > 0 - ? ( - this.metrics.totalTokens / - (this.metrics.totalTimeMs / 1000) - ).toFixed(1) - : "N/A"; - - const hasDetailedTimings = - this.metrics.totalPromptTimeMs > 0 || - this.metrics.totalCompletionTimeMs > 0; - - const avgPromptSpeed = - this.metrics.totalPromptTimeMs > 0 - ? ( - this.metrics.totalPromptTokens / - (this.metrics.totalPromptTimeMs / 1000) - ).toFixed(1) - : "N/A"; - - const avgCompletionSpeed = - this.metrics.totalCompletionTimeMs > 0 - ? ( - this.metrics.totalCompletionTokens / - (this.metrics.totalCompletionTimeMs / 1000) - ).toFixed(1) - : "N/A"; - - const lines = [ - `⏱️ Pipeline Summary`, - ` Duration: ${durationSec}s`, - ` Processed: ${this.metrics.wordsProcessed}`, - ` Skipped: ${this.metrics.wordsSkipped}`, - ` Failed: ${this.metrics.wordsFailed}`, - ` Total: ${total}`, - ` Throughput: ${throughput} words/sec`, - ``, - `🤖 LLM Metrics`, - ` Calls: ${this.metrics.llmCalls}`, - ` Avg prompt tokens: ${avgPromptTokens}`, - ` Avg completion tokens: ${avgCompletionTokens}`, - ` Avg total tokens: ${avgPromptTokens + avgCompletionTokens}`, - ` Avg total request time: ${avgTotalTimeMs}ms`, - ` Avg throughput: ${unifiedThroughput} tok/s`, - ]; - - if (hasDetailedTimings) { - lines.push( - ``, - ` [Local breakdown]`, - ` Avg prompt speed: ${avgPromptSpeed} tok/s`, - ` Avg completion speed: ${avgCompletionSpeed} tok/s`, - ); - } - - return lines.join("\n"); - } -} - - - -/** - * Simple progress tracker for pipeline execution. - */ -export class ProgressTracker { - private current: number; - private failed: number; - private total: number; - - constructor(total: number) { - this.current = 0; - this.failed = 0; - this.total = total; - } - - next(): number { - this.current++; - return this.current; - } - - recordFailed(): void { - this.failed++; - } - - format(label: string): string { - return `[${this.current}/${this.total}] (${this.failed} failed) ${label}`; - } -} - - - -import fs from "fs"; -import path from "path"; - -// Define a simple shape for what a discovered dataset looks like -export interface Wordlist { - language: string; - pos: string; - sourcePath: string; - outputDir: string; -} - -/** - * Scans the source-data directory to find all available word lists. - */ -export function scanSourceData(baseDir: string): Wordlist[] { - const sourceBaseDir = path.join(baseDir, "source-data"); - const discoveredWordlists: Wordlist[] = []; - - // Safety check: if there's no source-data folder, return an empty array - if (!fs.existsSync(sourceBaseDir)) { - return discoveredWordlists; - } - - // 1. Read the language directories (e.g., ['english']) - const languages = fs.readdirSync(sourceBaseDir); - - for (const lang of languages) { - const langFolderPath = path.join(sourceBaseDir, lang); - - // Make sure it's a directory, not a stray file - if (!fs.statSync(langFolderPath).isDirectory()) continue; - - // 2. Read the files inside the language folder (e.g., ['nouns']) - const posFiles = fs.readdirSync(langFolderPath); - - for (const pos of posFiles) { - const fullSourcePath = path.join(langFolderPath, pos); - - // Make sure it's a file (like your extensionless "nouns" file) - if (!fs.statSync(fullSourcePath).isFile()) continue; - - // 3. Package everything into a flat item and add it to our array - discoveredWordlists.push({ - language: lang, - pos: pos, - sourcePath: fullSourcePath, - outputDir: path.join(baseDir, "worddata", lang, pos), - }); - } - } - - // show summary - console.log( - `✅ Scan complete! Found ${discoveredWordlists.length} wordlist(s):`, - ); - for (const list of discoveredWordlists) { - console.log(` • ${list.language.toUpperCase()} (${list.pos})`); - } - - return discoveredWordlists; -} - - - -import fs from "fs"; - -interface VerificationResult { - valid: boolean; - errors: string[]; -} - -/** - * Verifies that an enriched JSON file matches the expected schema. - * Returns detailed error messages for any violations. - */ -export function verifyEnrichedFile(filePath: string): VerificationResult { - const errors: string[] = []; - - if (!fs.existsSync(filePath)) { - return { valid: false, errors: ["File does not exist"] }; - } - - let data: unknown; - try { - data = JSON.parse(fs.readFileSync(filePath, "utf-8")); - } catch (_error: unknown) { - return { valid: false, errors: ["Invalid JSON syntax"] }; - } - - if (typeof data !== "object" || data === null || Array.isArray(data)) { - return { valid: false, errors: ["Root must be an object"] }; - } - - const obj = data as Record; - - // Required top-level fields - const requiredFields = ["word", "language", "pos", "senses"]; - for (const field of requiredFields) { - if (!(field in obj)) { - errors.push(`Missing required field: "${field}"`); - } - } - - // Validate senses array - if (!Array.isArray(obj["senses"])) { - errors.push('"senses" must be an array'); - } else if (obj["senses"].length === 0) { - errors.push('"senses" array cannot be empty'); - } else { - for (let i = 0; i < obj["senses"].length; i++) { - const sense = obj["senses"][i] as Record; - const sensePrefix = `senses[${i}]`; - - if (!sense["sense"] || typeof sense["sense"] !== "string") { - errors.push(`${sensePrefix}: missing or invalid "sense"`); - } - if (!sense["example"] || typeof sense["example"] !== "string") { - errors.push(`${sensePrefix}: missing or invalid "example"`); - } - if ( - !["easy", "medium", "hard"].includes( - sense["difficulty_level"] as string, - ) - ) { - errors.push(`${sensePrefix}: invalid "difficulty_level"`); - } - if (!sense["translations"] || typeof sense["translations"] !== "object") { - errors.push(`${sensePrefix}: missing "translations"`); - } else { - const trans = sense["translations"] as Record; - for (const lang of ["de", "it", "es", "fr"]) { - if (!Array.isArray(trans[lang])) { - errors.push( - `${sensePrefix}: missing or invalid "${lang}" translations`, - ); - } else { - for (let j = 0; j < (trans[lang] as unknown[]).length; j++) { - const t = (trans[lang] as unknown[])[j] as Record< - string, - unknown - >; - if (!t["word"] || typeof t["word"] !== "string") { - errors.push(`${sensePrefix}.${lang}[${j}]: missing "word"`); - } - if ( - !["masculine", "feminine", "neuter", null].includes( - t["gender"] as string | null, - ) - ) { - errors.push(`${sensePrefix}.${lang}[${j}]: invalid "gender"`); - } - } - } - } - } - } - } - - return { valid: errors.length === 0, errors }; -} - - - -import fs from "fs"; - -/** - * Writes data as formatted JSON to a file path. - * Safely catches and re-throws file system errors. - */ -export function writeJsonFile(filePath: string, data: unknown): void { - const tempPath = `${filePath}.tmp`; - fs.writeFileSync(tempPath, JSON.stringify(data, null, 2), "utf-8"); - fs.renameSync(tempPath, filePath); -} - - - -{ - "provider": "openrouter", - "model": "google/gemini-2.5-flash", - "batchSize": 5, - "maxRetries": 3 -} - - - -import { isWordProcessed } from "./utils/check-if-json-exists.js"; -import { createBaseJson } from "./utils/create-base-json.js"; -import { ensureOutputFolders } from "./utils/create-output-dirs.js"; -import { scanSourceData } from "./utils/scanning-source-files.js"; -import { createLineReader } from "./utils/create-line-reader.js"; -import { checkLlmServer } from "./utils/check-llm-server.js"; -import { getWordFilePath } from "./utils/get-word-file-path.js"; -import { mergeEnrichedData } from "./utils/merge-enriched-data.js"; -import { enrichWordWithRetry } from "./utils/enrich-word.js"; -import { writeJsonFile } from "./utils/write-json-file.js"; -import { deleteFileIfExists } from "./utils/delete-file.js"; -import { PipelineTimer } from "./utils/pipeline-timer.js"; -import { ProgressTracker } from "./utils/progress-tracker.js"; -import { verifyEnrichedFile } from "./utils/verify-enriched-file.js"; -import { runCli } from "./utils/cli.js"; -import type { PipelineConfig } from "./utils/cli.js"; -import { LLM_CONFIG } from "./config/llm.js"; -import { BATCH_CONFIG } from "./config/batch.js"; - -// Runtime config accessor for other modules -let RUNTIME_CONFIG: PipelineConfig; - -export function getRuntimeConfig(): PipelineConfig { - return RUNTIME_CONFIG; -} - -async function main() { - // ── Interactive CLI ────────────────────────────────────────────────────── - RUNTIME_CONFIG = await runCli(); - - // Populate shared config objects so existing imports keep working - LLM_CONFIG.provider = RUNTIME_CONFIG.provider; - LLM_CONFIG.url = RUNTIME_CONFIG.url; - LLM_CONFIG.model = RUNTIME_CONFIG.model; - BATCH_CONFIG.size = RUNTIME_CONFIG.batchSize; - BATCH_CONFIG.maxRetries = RUNTIME_CONFIG.maxRetries; - - console.log("Starting data pipeline...\n"); - console.log(`Provider: ${RUNTIME_CONFIG.provider}`); - console.log(`Model: ${RUNTIME_CONFIG.model ?? "(none)"}`); - console.log(`Batch: ${RUNTIME_CONFIG.batchSize} words/call\n`); - - const timer = new PipelineTimer(); - - // step 1: scanning for source files - console.log("\n step 1: scanning the source files..."); - const wordlists = scanSourceData(import.meta.dirname); - - // step 2: ensuring output folders exist - console.log("\n step 2: creating necessary output folders..."); - ensureOutputFolders(wordlists); - - // step 3: check to verify the local AI engine is ready before touching anything - console.log("\n step 3: verifying local AI engine status..."); - try { - await checkLlmServer(); - } catch (error: unknown) { - const message = error instanceof Error ? error.message : String(error); - console.error(`\n ❌ ${message}`); - process.exit(1); - } - - // Step 4: Loop through the wordlists array - console.log("\n step 4: looping through the wordlists..."); - - for (const wordlist of wordlists) { - console.log( - `\nReading list: [${wordlist.language.toUpperCase()}] -> [${wordlist.pos.toUpperCase()}]`, - ); - - const rl = createLineReader(wordlist.sourcePath); - - // Collect words and count them - const words: string[] = []; - for await (const line of rl) { - const word = line.trim().toLowerCase(); - if (word) words.push(word); - } - - // Filter out already-processed words - const unprocessedWords = words.filter( - (word) => !isWordProcessed(word, wordlist.outputDir), - ); - - const skippedCount = words.length - unprocessedWords.length; - if (skippedCount > 0) { - console.log(` Skipped ${skippedCount} already-processed words`); - } - - const progress = new ProgressTracker(unprocessedWords.length); - - // Step 5: Process in batches - for (let i = 0; i < unprocessedWords.length; i += BATCH_CONFIG.size) { - const batch = unprocessedWords.slice(i, i + BATCH_CONFIG.size); - const batchNum = Math.floor(i / BATCH_CONFIG.size) + 1; - const totalBatches = Math.ceil( - unprocessedWords.length / BATCH_CONFIG.size, - ); - const batchLabel = `Batch ${batchNum}/${totalBatches}`; - - console.log(`\n ${batchLabel}: [${batch.join(", ")}]`); - - // Create skeletons for all words in batch - for (const word of batch) { - createBaseJson( - word, - wordlist.outputDir, - wordlist.language, - wordlist.pos, - ); - } - - timer.startWord(); - - try { - // Step 6: enrich batch with senses (with retry/split) - const result = await enrichWordWithRetry( - batch, - wordlist.language, - wordlist.pos, - ); - - // Step 7: write each word's result - for (const [word, senses] of result.results) { - progress.next(); - console.log( - ` ${progress.format(`Enriched and saved: ${word}.json`)}`, - ); - - const targetFilePath = getWordFilePath(word, wordlist.outputDir); - const enrichedData = mergeEnrichedData(word, senses); - - writeJsonFile(targetFilePath, enrichedData); - - // Verify the generated file - const verification = verifyEnrichedFile(targetFilePath); - if (!verification.valid) { - console.error(` Warning: Schema violations in ${word}.json:`); - for (const error of verification.errors) { - console.error(` - ${error}`); - } - } - - timer.recordProcessed({ - promptTokens: result.metrics.promptTokens, - completionTokens: result.metrics.completionTokens, - totalTokens: result.metrics.totalTokens, - promptTimeMs: result.metrics.promptTimeMs, - completionTimeMs: result.metrics.completionTimeMs, - totalTimeMs: result.metrics.totalTimeMs, - }); - } - - console.log(` ${timer.getWordTiming()}`); - // Show ETA every 5 batches or on the last batch - if (batchNum % 5 === 0 || batchNum === totalBatches) { - console.log(` 📊 ${timer.getEta(unprocessedWords.length)}`); - } - } catch (error: unknown) { - const errorMessage = - error instanceof Error ? error.message : String(error); - console.error( - ` Failed to enrich batch [${batch.join(", ")}]: ${errorMessage}`, - ); - - // Cleanup: delete any partially-written files for the failed batch - for (const word of batch) { - const targetFilePath = getWordFilePath(word, wordlist.outputDir); - deleteFileIfExists(targetFilePath); - console.log(` Removed incomplete file: ${word}.json`); - progress.recordFailed(); - } - - timer.recordFailed(); - } - } - } - - timer.stop(); - console.log("\n" + timer.getSummary()); - console.log("\nGlobal data pipeline run completed successfully."); -} - -// Fire the orchestrator block -main().catch((err) => { - console.error("Critical unexpected pipeline failure:", err); -}); - - - -import { defineConfig } from "vitest/config"; - -export default defineConfig({ - test: { - environment: "node", - globals: true, - include: ["tests/**/*.test.ts"], - exclude: ["**/dist/**", "**/node_modules/**"], - testTimeout: 60_000, - }, -}); - - - -{ - "extends": "../tsconfig.base.json", - "compilerOptions": { - "module": "NodeNext", - "moduleResolution": "NodeNext", - "outDir": "dist", - "rootDir": ".", - "types": ["node"] - }, - "references": [{ "path": "../packages/shared" }], - "include": ["./**/*", "vitest.config.ts"] -} - - - -# OpenRouter API key — required for OpenRouter providers -# Get one at https://openrouter.ai/keys -OPENROUTER_API_KEY= - -# Anthropic API key — required for Anthropic provider (reference baseline only) -# Get one at https://console.anthropic.com/ -ANTHROPIC_API_KEY= - - - -{ - "name": "@lila/pipeline", - "version": "1.0.0", - "private": true, - "type": "module", - "scripts": { - "db:reset": "tsx db/reset.ts", - "extract": "tsx stage-1-extract/scripts/extract.ts", - "reverse-link": "tsx stage-2-reverse-link/scripts/reverse-link.ts", - "db:import": "tsx db/import.ts", - "db:init": "tsx db/init.ts", - "test": "vitest run", - "test:watch": "vitest", - "pipeline:run": "tsx --env-file .env pipeline.ts" - }, - "dependencies": { - "@lila/shared": "workspace:*", - "better-sqlite3": "^12.9.0" - }, - "devDependencies": { - "@types/better-sqlite3": "^7.6.13", - "@types/node": "^24.12.0", - "tsx": "^4.21.0", - "typescript": "^5.9.3", - "vitest": "^4.1.0" - } -} - - -