forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

wip

Browse source
This commit is contained in:
lila 2026-07-06 15:35:22 +02:00
parent e89e3b7a70
commit 8e8484f875
3 changed files with 281 additions and 145 deletions
Download patch file Download diff file Expand all files Collapse all files

388
documentation/DATA_PIPELINE.md
View file

@ -19,26 +19,28 @@
7. [Batching Strategy](#7-batching-strategy)
8. [Hardware Constraints](#8-hardware-constraints)
9. [Testing & Quality Assurance](#9-testing--quality-assurance)
10. [Future Extensions & Roadmap](#10-future-extensions--roadmap)
11. [Decisions Log](#11-decisions-log)
12. [Known Issues & Dev Notes](#12-known-issues--dev-notes)
13. [How to Run](#13-how-to-run)
14. [Roadmap](#14-roadmap)
10. [Interactive CLI](#10-interactive-cli)
11. [Future Extensions & Roadmap](#11-future-extensions--roadmap)
12. [Decisions Log](#12-decisions-log)
13. [Known Issues & Dev Notes](#13-known-issues--dev-notes)
14. [How to Run](#14-how-to-run)
15. [Roadmap](#15-roadmap)
---
## Quick Reference
| What | Where |
| ------------- | ----------------------------------- |
| Entry point | `pipeline.ts` |
| LLM config | `config/llm.ts` |
| System prompt | `config/prompt.ts` |
| Batch config | `config/batch.ts` |
| Output schema | `utils/merge-enriched-data.ts` |
| LLM adapters | `utils/llm-adapters/` |
| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
| Target scale | 100,000+ words |
| What | Where |
| ---------------- | ------------------------------------------ |
| Entry point | `pipeline.ts` |
| LLM config | `config/llm.ts` |
| System prompt | `config/prompt.ts` — `buildSystemPrompt()` |
| Batch config | `config/batch.ts` |
| Shared constants | `config/constants.ts` |
| Output schema | `utils/merge-enriched-data.ts` |
| LLM adapters | `utils/llm-adapters/` |
| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
| Target scale | 100,000+ words |
---
@ -49,9 +51,9 @@ The Lila Data Pipeline is a TypeScript-based batch processing system that enrich
- One or more **senses** (definitions)
- A **natural example sentence** per sense
- A **CEFR-based difficulty level** (`easy` / `medium` / `hard`)
- **Translations** into German, Italian, Spanish, and French, each with grammatical **gender**
- **Translations** into all target languages except the source, each with grammatical **gender**
The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It is currently in active development: the core architecture is stable, the LLM integration layer supports both local and cloud providers via a pluggable adapter pattern, and a configurable batching system with retry/split logic is fully implemented.
The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an interactive CLI for provider selection.
### Key Design Principles
@ -62,6 +64,7 @@ The pipeline is designed to scale to **100,000+ words** across multiple language
| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. |
| **Configurable batching** | Batch size is a single config value (`config/batch.ts`). The pipeline adapts without code changes. |
| **Provider-agnostic** | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface. |
| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns. |
### Open Question: Gender Accuracy
@ -81,8 +84,10 @@ No decision made. Gender handling will be determined by the 20-word quality tort
- Local LLM integration via llama.cpp server (OpenAI-compatible API)
- **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/`
- **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1)
- Schema validation for generated JSON
- Progress tracking and timing metrics
- **Honest timing**: unified throughput for all providers, detailed breakdown only for local
- **Auto-target languages**: prompt dynamically excludes source language from targets
- **Schema validation**: validates LLM response structure before file writes
- **Interactive CLI**: planned — provider/model/batch selection with saved config
- **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
- **Pending:** 20-word quality torture suite (will decide gender approach)
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq)
@ -90,7 +95,7 @@ No decision made. Gender handling will be determined by the 20-word quality tort
### One-Line Architecture
```
source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> write .json
source wordlists -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json
|
[gender: LLM-generated, accuracy TBD]
```
@ -101,12 +106,13 @@ source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> w
| ----------------------------------------- | ----------------------------------------------------------------------------------- |
| `pipeline.ts` | Orchestrator. Scans sources, loops words, coordinates all stages |
| `config/llm.ts` | Provider selection, API URL, model name |
| `config/prompt.ts` | System prompt sent to the LLM |
| `config/prompt.ts` | `buildSystemPrompt()` — dynamic prompt with auto-target languages |
| `config/batch.ts` | Batch size and max retry count |
| `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
| `utils/enrich-word.ts` | Calls LLM via adapter, parses response, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON; defines TypeScript schema |
| `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON |
| `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) |
| `utils/check-llm-server.ts` | Health check before pipeline starts |
| `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers |
| `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory |
| `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files |
| `utils/write-json-file.ts` | Atomic `.tmp` → rename writes |
@ -116,7 +122,7 @@ source wordlists -> llama.cpp server (LLM) -> merge senses -> verify schema -> w
| `utils/delete-file.ts` | Cleanup helper for failed batches |
| `utils/get-word-file-path.ts` | Path construction helper |
| `utils/progress-tracker.ts` | `[current/total]` formatting for console output |
| `utils/pipeline-timer.ts` | Per-word and global timing + LLM token metrics |
| `utils/pipeline-timer.ts` | Timing + token metrics; unified throughput for all providers |
| `utils/llm-adapters/factory.ts` | Creates the right adapter based on `LLM_CONFIG.provider` |
| `utils/llm-adapters/types.ts` | `LlmAdapter` interface |
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek |
@ -165,11 +171,11 @@ The LLM fills the gap: it generates **pedagogical content** (student-friendly de
The pipeline is **direction-agnostic**. A wordlist is defined by:
- **Source language**: the language of the input words
- **Target languages**: the languages to translate into
- **Target languages**: all other languages in the system (auto-derived from `ALL_LANGUAGES` minus source)
Current focus: **English -> German/Italian/Spanish/French**
Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist and the target languages specified in the prompt.
Planned directions include **German -> French**, **Italian -> Spanish**, etc. The LLM prompt and output schema support any combination — the only change is the source wordlist. Target languages are computed automatically.
### Why 100,000+ Words?
@ -201,6 +207,7 @@ Generating 100,000 entries with an LLM introduces risks:
| JSON parse failures | Retry + split logic, schema validation, cleanup on failure |
| Model drift (online APIs) | Version pinning, local fallback |
| Provider downtime | Adapter pattern allows hot-swapping providers |
| Malformed LLM responses | `validateSense()` catches bad data before file writes |
### Why TypeScript + Node?
@ -224,15 +231,15 @@ Generating 100,000 entries with an LLM introduces risks:
```
Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
-> Create skeletons (batch) -> Call LLM -> Parse JSON -> Retry/split on failure
-> Merge -> Write atomically -> Verify schema -> Log metrics
-> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
-> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics
```
### Resumability
- **Skip existing**: `check-if-json-exists.ts` checks if `{word}.json` exists with non-empty `senses`
- **Atomic writes**: `.tmp` -> rename in `write-json-file.ts`, no partial files on crash
- **Cleanup on failure**: Deletes skeleton files for failed batches, continues to next batch
- **Cleanup on failure**: Deletes partially-written files for failed batches, continues to next batch
### Directory Structure
@ -241,13 +248,14 @@ data-pipeline/
|-- pipeline.ts # Entry point / orchestrator
|-- config/
| |-- llm.ts # Provider, API URL, model name
| |-- prompt.ts # System prompt
| |-- prompt.ts # buildSystemPrompt() — dynamic prompt
| |-- batch.ts # Batch size and retry config
| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES
|-- utils/
| |-- enrich-word.ts # LLM call, parse, retry/split
| |-- merge-enriched-data.ts # Schema types + merge logic
| |-- verify-enriched-file.ts # Schema validation
| |-- check-llm-server.ts # Health check
| |-- check-llm-server.ts # Health check (local only)
| |-- scanning-source-files.ts # Source discovery
| |-- create-base-json.ts # Skeleton writer
| |-- write-json-file.ts # Atomic JSON writer
@ -281,18 +289,22 @@ Full TypeScript interfaces: `utils/merge-enriched-data.ts`.
### Error Handling
| Failure | Behavior |
| ------------------------------ | ------------------------------------------------------- |
| LLM server offline | Hard fail at startup (`check-llm-server.ts`) |
| LLM returns bad JSON | Retry up to 3 times, then split batch. Log and continue |
| Schema validation fails | Log warnings, keep file |
| Individual batch fails | Does not stop pipeline; cleans up skeletons |
| Individual word fails (size 1) | Log and continue to next word |
| Failure | Behavior |
| ------------------------------ | -------------------------------------------------------- |
| LLM server offline | Hard fail at startup (`check-llm-server.ts`, local only) |
| LLM returns bad JSON | Retry up to 3 times, then split batch. Log and continue |
| LLM returns malformed senses | `validateSense()` catches it before file write |
| Schema validation fails | Log warnings, keep file |
| Individual batch fails | Does not stop pipeline; cleans up partial files |
| Individual word fails (size 1) | Log and continue to next word |
### Metrics
Per-run: words processed/skipped/failed, duration, throughput, LLM token counts and speeds. See `utils/pipeline-timer.ts`.
**Unified throughput** (all providers): total tokens / total request time
**Detailed breakdown** (local only): prompt speed vs completion speed
---
## 4. Current Implementation
@ -308,23 +320,24 @@ Per-run: words processed/skipped/failed, duration, throughput, LLM token counts
### Configuration
| File | Purpose |
| ------------------ | ---------------------------------------------------------------- | ------------ | ---------- | ------------------------- |
| `config/llm.ts` | `provider` (`local` | `openrouter` | `deepseek` | `gemini`), `url`, `model` |
| `config/prompt.ts` | System prompt with CEFR mapping, required fields, example output |
| `config/batch.ts` | `BATCH_CONFIG.size` (words per call), `maxRetries` |
| File | Purpose |
| --------------------- | --------------------------------------------------------- | ------------ | ---------- | ------------------------- |
| `config/llm.ts` | `provider` (`local` | `openrouter` | `deepseek` | `gemini`), `url`, `model` |
| `config/prompt.ts` | `buildSystemPrompt(sourceLanguage, pos, targetLanguages)` |
| `config/batch.ts` | `BATCH_CONFIG.size` (words per call), `maxRetries` |
| `config/constants.ts` | `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
### Key Modules
| File | Responsibility |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON array, builds `EnrichedSense[]` with composite IDs, retry/split logic |
| `utils/merge-enriched-data.ts` | Merges skeleton `{word, language, pos}` with LLM senses, adds `enrichedAt` and `model` |
| `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure |
| `utils/pipeline-timer.ts` | Tracks per-word and global metrics (duration, tokens, throughput) |
| `utils/llm-adapters/factory.ts` | Creates adapter based on `LLM_CONFIG.provider` |
| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek |
| `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API |
| File | Responsibility |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts` | Merges skeleton with LLM senses, adds `enrichedAt` and `model` |
| `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure |
| `utils/pipeline-timer.ts` | Tracks per-word and global metrics; unified throughput for all providers |
| `utils/llm-adapters/factory.ts` | Creates adapter based on `LLM_CONFIG.provider` |
| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek |
| `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API with `systemInstruction` |
### Current Model
@ -353,7 +366,7 @@ The server flags evolved through trial and error on the target hardware (Intel i
| Flag | Value Tried | Result | Why |
| ----------------- | ----------------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `-m` | `qwen3.5-4b-q4_k_m.gguf` | Works, ~47s/word | Baseline. Correct genders. 2.6GB file, tight on VRAM. |
| `-m` | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken** | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#12-known-issues--dev-notes). |
| `-m` | `Ministral-3b-instruct.Q4_K_M.gguf` | **Broken** | Tokenizer mismatch (Tekken). Outputs gibberish regardless of template. See [Known Issues](#13-known-issues--dev-notes). |
| `-m` | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | Works, ~8s/word | Current. Fast but gender accuracy degraded. |
| `-ngl` | `999` | Keeps | Offload all layers to GPU. Required for any speed. |
| `-c` | `4096` | Wasteful | 4K context for 300-token dictionary entries wastes ~400MB VRAM. |
@ -420,8 +433,7 @@ The server flags evolved through trial and error on the target hardware (Intel i
- **Gender accuracy**: Qwen2.5-1.5B systematically defaults to `neuter` for Romance languages. Under evaluation whether larger models fix this.
- **Single POS**: Only nouns tested. Verbs/adjectives/adverbs need prompt adjustments.
- **Hardcoded model name**: `merge-enriched-data.ts` hardcodes `"qwen3.5-4b-q4_k_m"` regardless of actual model used.
- **Duplicated mappings**: `LANG_MAP`/`POS_MAP` exist in three separate files.
- **Pre-scanning wordlists**: Entire file read into memory before processing. Inefficient for 100k words. See [Refactor Notes](#refactor-notes).
---
@ -715,20 +727,118 @@ For each word and each candidate model:
- Each sense: `sense` (string), `example` (string), `difficulty_level` in {easy, medium, hard}
- Each translation: `word` (string), `gender` in {masculine, feminine, neuter, null}
Additionally, `validateSense()` in `enrich-word.ts` catches malformed senses **before** file writes, triggering retry/split instead of writing bad data.
---
## 10. Future Extensions & Roadmap
## 10. Interactive CLI
### Overview
The pipeline includes an interactive CLI that asks the user to select provider, model, and batch size on each run. No editing of `config/llm.ts` required.
### Flow
```
$ npx tsx pipeline.ts
🌐 Lila Data Pipeline
─────────────────────
[1] Use last config: online → gemini → gemini-2.5-flash-lite → batch 50
[2] Configure new run
> 2
Provider type:
[1] Local (llama.cpp)
[2] Online API
> 2
Online provider:
[1] Gemini
[2] DeepSeek
[3] OpenRouter
[4] Groq
> 1
Model:
[1] gemini-2.5-flash-lite (recommended for cost)
[2] gemini-2.5-pro (recommended for quality)
> 1
Batch size:
[1] 1 (safest, slowest)
[2] 5 (recommended for local)
[3] 10
[4] 20
[5] 50 (recommended for Gemini)
[6] Custom
> 5
✅ Configuration:
Provider: gemini
Model: gemini-2.5-flash-lite
API key: GEMINI_API_KEY found in environment ✓
Batch size: 50
Start pipeline with these settings? [Y/n]
> Y
```
### Saved Config
On first run, after confirming, write to `.pipeline-config.json`:
```json
{
"provider": "gemini",
"model": "gemini-2.5-flash-lite",
"batchSize": 50,
"lastRun": "2026-07-06T13:54:00Z"
}
```
Next run shows `[1] Use last config` at the top.
### API Key Rules
- **Never prompt for keys.** Check `process.env` for the provider's key.
- **If missing:** Print which env var is needed, then exit.
- **Supported env vars:** `GEMINI_API_KEY`, `DEEPSEEK_API_KEY`, `OPENROUTER_API_KEY`
### Batch Size Recommendations
| Provider | Recommended | Rationale |
| ------------------ | ----------- | ---------------------------------- |
| `local` (GTX 950M) | 5 | VRAM-limited, KV cache pressure |
| `local` (RTX 4090) | 20 | Fast, more VRAM |
| `gemini` | 50 | Free tier: 1,500 req/day |
| `deepseek` | 20 | 5M free tokens, balance speed/cost |
| `groq` | 50 | Very fast, rate limits generous |
| `openrouter` | 10 | 200 req/day free tier |
---
## 11. Future Extensions & Roadmap
### Near-Term (Next 2-4 Weeks)
| Item | Status | Notes |
| ------------------------ | ------------ | --------------------------------------------------------------- |
| Configurable batching | **Complete** | Single `BATCH_CONFIG.size` value, retry/split logic implemented |
| 20-word torture suite | Pending | Decides gender approach and model selection |
| Qwen2.5-3B evaluation | Pending | Download and test |
| Online API testing | Pending | Gemini free tier, DeepSeek, Groq |
| Fix hardcoded model name | Pending | `merge-enriched-data.ts` hardcodes `"qwen3.5-4b"` |
| Extract shared constants | Pending | `LANG_MAP`/`POS_MAP` duplicated in 3 files |
| Item | Status | Notes |
| ---------------------- | ------------ | ------------------------------------------------------------- |
| Configurable batching | **Complete** | `config/batch.ts` with `size` and `maxRetries` |
| Retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch |
| Honest timing metrics | **Complete** | Unified throughput for all providers, detailed only for local |
| Auto-target languages | **Complete** | `buildSystemPrompt()` excludes source from targets |
| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes |
| Interactive CLI | Planned | Provider/model/batch selection with saved config |
| 20-word torture suite | Pending | Decides gender approach and model selection |
| Qwen2.5-3B evaluation | Pending | Download and test |
| Online API testing | Pending | Gemini free tier, DeepSeek, Groq |
### Medium-Term (1-3 Months)
@ -751,27 +861,30 @@ For each word and each candidate model:
---
## 11. Decisions Log
## 12. Decisions Log
| Date | Decision | Context | Rationale |
| ---------- | --------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------- |
| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app |
| 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying |
| 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas |
| 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys |
| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies |
| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference |
| 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM |
| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance |
| 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation |
| 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality |
| 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion |
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations |
| 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 |
| Date | Decision | Context | Rationale |
| ---------- | ----------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------- |
| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app |
| 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying |
| 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas |
| 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys |
| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies |
| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference |
| 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM |
| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance |
| 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation |
| 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality |
| 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion |
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations |
| 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 |
| 2026-07-06 | Honest timing metrics | Cloud providers don't expose prompt/completion breakdown | Unified `totalTimeMs` for all; detailed breakdown only when available |
| 2026-07-06 | Auto-target languages | Prompt hardcoded English -> de/it/es/fr | `ALL_LANGUAGES` minus source = targets; works for any source language |
| 2026-07-06 | Validate LLM responses before write | Bad data was written then warned about | `validateSense()` catches malformed responses early, triggers retry |
---
## 12. Known Issues & Dev Notes
## 13. Known Issues & Dev Notes
### glossa-web (Frontend)
@ -782,15 +895,12 @@ For each word and each candidate model:
### Data Pipeline
| Issue | Details | Severity |
| --------------------------------- | ----------------------------------------------------------------------------------------------- | -------------------------------- |
| Ministral-3B tokenizer mismatch | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. | Blocker - abandoned |
| Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages. | Under evaluation |
| Hardcoded model name | `merge-enriched-data.ts` always writes `"qwen3.5-4b-q4_k_m"` regardless of actual model. | Minor - fix before production |
| Duplicated LANG_MAP/POS_MAP | Identical mapping objects in `create-base-json.ts`, `merge-enriched-data.ts`, `enrich-word.ts`. | Minor - refactor risk |
| OpenAI-compatible timings | `json.timings` is llama.cpp-specific. Will break for OpenRouter/DeepSeek. | Medium - needs graceful fallback |
| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes. | Known limitation |
| Batch metrics averaging | Split-and-merge averages tokens/sec instead of weighting by token count. | Minor - summary stats only |
| Issue | Details | Severity |
| --------------------------------- | --------------------------------------------------------------------------- | ----------------------------------- |
| Ministral-3B tokenizer mismatch | Tekken tokenizer not properly converted to GGUF. Model outputs gibberish. | Blocker - abandoned |
| Qwen2.5-1.5B gender hallucination | Systematic `neuter` default for Romance languages. | Under evaluation |
| Pre-scanning wordlists | Entire file read into memory before processing. Inefficient for 100k words. | Medium - streaming refactor planned |
| Single POS tested | Only nouns validated. Verbs/adjectives need prompt changes. | Known limitation |
### Hardware
@ -800,9 +910,15 @@ For each word and each candidate model:
| Maxwell GPU aging | No Flash Attention support, bandwidth-starved. |
| Laptop thermals | Cannot run 24/7 for weeks. Batch processing required. |
### Refactor Notes
| Note | File | Context |
| --------------------- | ------------- | -------------------------------------------------------------------------- |
| Streaming vs pre-scan | `pipeline.ts` | For 100k words, stream and batch on-the-fly instead of reading entire file |
---
## 13. How to Run
## 14. How to Run
### Prerequisites
@ -836,37 +952,35 @@ cd ~/Downloads/llama.cpp
--prio 2
```
### Configure Provider
Edit `config/llm.ts`:
```typescript
// Local
export const LLM_CONFIG = {
provider: "local" as const,
url: "http://127.0.0.1:8080/v1/chat/completions",
model: undefined,
};
// Gemini
export const LLM_CONFIG = {
provider: "gemini" as const,
url: "",
model: "gemini-2.5-flash-lite",
};
```
### Run the Pipeline
### Run the Pipeline (Interactive CLI)
```bash
cd /path/to/data-pipeline
npx tsx pipeline.ts
```
Follow the prompts to select provider, model, and batch size.
### Run with Last Config
```bash
cd /path/to/data-pipeline
npx tsx pipeline.ts
# Select [1] Use last config
```
### Expected Output
```
Starting data pipeline...
🌐 Lila Data Pipeline
─────────────────────
[1] Use last config: local → qwen2.5-1.5b → batch 5
[2] Configure new run
> 1
🟢 Local AI engine is connected and ready for inference!
step 1: scanning the source files...
✅ Scan complete! Found 1 wordlist(s):
@ -901,6 +1015,11 @@ Reading list: [ENGLISH] -> [NOUNS]
Calls: 1
Avg prompt tokens: 312
Avg completion tokens: 524
Avg total tokens: 836
Avg total request time: 32800ms
Avg throughput: 25.5 tok/s
[Local breakdown]
Avg prompt speed: 548.2 tok/s
Avg completion speed: 18.8 tok/s
@ -909,7 +1028,7 @@ Global data pipeline run completed successfully.
---
## 14. Roadmap
## 15. Roadmap
### Phase 1: Batching (Complete)
@ -917,6 +1036,8 @@ Global data pipeline run completed successfully.
| ------------------------------------- | ------------ | ------------------------------------------------------------------------------------ |
| Implement configurable batch size | **Complete** | `config/batch.ts` with `size` and `maxRetries` |
| Implement retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch |
| Honest timing metrics | **Complete** | Unified throughput for all providers |
| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes |
| Verify batching doesn't break quality | Pending | Run 20-word torture suite on Qwen2.5-1.5B with batch sizes 1, 5, 15. Compare output. |
| Measure speedup vs batch size | Pending | Track throughput at 1, 5, 15 on local hardware. |
@ -924,7 +1045,20 @@ Global data pipeline run completed successfully.
---
### Phase 2: Model Selection
### Phase 2: Interactive CLI
| Task | Status | Notes |
| --------------------- | ------- | ----------------------------------------- |
| Design prompt flow | Planned | Provider → model → batch size → confirm |
| Implement CLI module | Planned | Use `readline` or `inquirer` for prompts |
| Save/load config | Planned | `.pipeline-config.json` |
| Wire into pipeline.ts | Planned | Replace static config with runtime config |
**Goal:** No editing of TypeScript files to switch providers.
---
### Phase 3: Model Selection
| Task | Status | Notes |
| ------------------------------------------------ | ------- | ---------------------------------------------------------------------------------------- |
@ -939,11 +1073,11 @@ Global data pipeline run completed successfully.
---
### Phase 3: Scale
### Phase 4: Scale
| Task | Status | Notes |
| ------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------- |
| Run 100k word pipeline | Pending | Estimated time depends on Phase 2 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). |
| Run 100k word pipeline | Pending | Estimated time depends on Phase 3 decision: ~10 days (local 1.5B) to ~1.5 days (Gemini batched free) to ~3-4 hours (Groq). |
| Spot-check output quality | Pending | Random sample of 100 entries. |
| Fix gender if needed | Pending | Kaikki lookup post-processing if LLM gender remains unreliable. |
| Handle failures & retries | Pending | Exponential backoff, split-and-retry for batch failures. |
@ -952,7 +1086,7 @@ Global data pipeline run completed successfully.
---
### Phase 4: Extend
### Phase 5: Extend
| Task | Status | Notes |
| ---------------------------- | ------- | ------------------------------------------------------------------------------------------- |
@ -969,13 +1103,11 @@ Global data pipeline run completed successfully.
### Backlog (Unscheduled)
| Task | Context |
| ------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. |
| Model auto-switching | Fallback to online API if local server fails mid-run. |
| Community open-source | Clean up, document, publish for other language learners. |
| Prometheus metrics | `--metrics` flag on llama-server for automated performance tracking. |
| `-c 1024` / `-b 256` experiments | Further VRAM optimization on GTX 950M. Low priority if moving to cloud. |
| Extract shared LANG_MAP/POS_MAP | Single source of truth for language/pos mappings. |
| Fix hardcoded model name | Pass actual model name through enrichment chain. |
| Graceful timing fallback for cloud adapters | Handle missing `timings` field in OpenRouter/DeepSeek responses. |
| Task | Context |
| -------------------------------- | ------------------------------------------------------------------------------------------- |
| Batch API discounts | Gemini, Qwen, Azure offer 50% off for 24h SLA. Relevant if running recurring large batches. |
| Model auto-switching | Fallback to online API if local server fails mid-run. |
| Community open-source | Clean up, document, publish for other language learners. |
| Prometheus metrics | `--metrics` flag on llama-server for automated performance tracking. |
| `-c 1024` / `-b 256` experiments | Further VRAM optimization on GTX 950M. Low priority if moving to cloud. |
| Streaming wordlist processing | Read file line-by-line and batch on-the-fly. Eliminates pre-scan memory usage. |