forgejo-lila/lila
1
0
Fork 0
Code Issues 50 Pull requests Projects 1 Releases Packages Wiki Activity Actions

wip

Browse source
This commit is contained in:
lila 2026-07-06 18:07:32 +02:00
parent 4fb1550e7d
commit 56f479da45
3 changed files with 138 additions and 99 deletions
Download patch file Download diff file Expand all files Collapse all files

2
data-pipeline/.pipeline-config.json
View file

@ -1 +1 @@
{ "provider": "local", "model": "local-model", "batchSize": 4, "maxRetries": 3 } { "provider": "local", "model": "local-model", "batchSize": 1, "maxRetries": 3 }

3
data-pipeline/config/prompt.ts
View file

@ -36,5 +36,8 @@ Example for ["house"]:
} }
] ]
} }
/no-think
`; `;
} }

232
documentation/DATA_PIPELINE.md
View file

@ -30,17 +30,18 @@
## Quick Reference ## Quick Reference
| What | Where | | What | Where |
| ---------------- | ------------------------------------------ | | ------------------- | ------------------------------------------ |
| Entry point | `pipeline.ts` | | Entry point | `pipeline.ts` |
| LLM config | `config/llm.ts` | | Interactive CLI | `utils/cli.ts` |
| System prompt | `config/prompt.ts` — `buildSystemPrompt()` | | LLM config schema | `config/llm.ts` |
| Batch config | `config/batch.ts` | | System prompt | `config/prompt.ts` — `buildSystemPrompt()` |
| Shared constants | `config/constants.ts` | | Batch config schema | `config/batch.ts` |
| Output schema | `utils/merge-enriched-data.ts` | | Shared constants | `config/constants.ts` |
| LLM adapters | `utils/llm-adapters/` | | Output schema | `utils/merge-enriched-data.ts` |
| Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` | | LLM adapters | `utils/llm-adapters/` |
| Target scale | 100,000+ words | | Current model | `qwen2.5-1.5b-instruct-q4_k_m.gguf` |
| Target scale | 100,000+ words |
--- ---
@ -53,7 +54,7 @@ The Lila Data Pipeline is a TypeScript-based batch processing system that enrich
- A **CEFR-based difficulty level** (`easy` / `medium` / `hard`) - A **CEFR-based difficulty level** (`easy` / `medium` / `hard`)
- **Translations** into all target languages except the source, each with grammatical **gender** - **Translations** into all target languages except the source, each with grammatical **gender**
The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an interactive CLI for provider selection. The pipeline is designed to scale to **100,000+ words** across multiple languages and parts of speech (nouns, verbs, adjectives, adverbs). It supports both local inference (llama.cpp) and cloud providers (Gemini, DeepSeek, OpenRouter, Groq) via a pluggable adapter pattern, with an **interactive CLI** for provider selection.
### Key Design Principles ### Key Design Principles
@ -62,7 +63,7 @@ The pipeline is designed to scale to **100,000+ words** across multiple language
| **Quality first** | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary. | | **Quality first** | Definitions, examples, translations, and gender must be accurate. Speed and cost are secondary. |
| **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. | | **Local-first, cloud-fallback** | Local LLMs (llama.cpp) are the default for cost control and data privacy. Online APIs are evaluated as alternatives for speed. |
| **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. | | **Resumable & idempotent** | Each word writes to its own JSON file. The pipeline skips already-processed words on restart. |
| **Configurable batching** | Batch size is a single config value (`config/batch.ts`). The pipeline adapts without code changes. | | **Configurable batching** | Batch size is selected interactively at runtime. The pipeline adapts without code changes. |
| **Provider-agnostic** | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface. | | **Provider-agnostic** | LLM adapters abstract local, OpenRouter, DeepSeek, and Gemini behind a single interface. |
| **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns. | | **Honest metrics** | Local models report detailed prompt/completion timing. Cloud providers report total request time only — no fake breakdowns. |
@ -83,11 +84,11 @@ No decision made. Gender handling will be determined by the 20-word quality tort
- Core pipeline: scanning, enrichment, merging, verification, writing - Core pipeline: scanning, enrichment, merging, verification, writing
- Local LLM integration via llama.cpp server (OpenAI-compatible API) - Local LLM integration via llama.cpp server (OpenAI-compatible API)
- **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/` - **Cloud provider adapters**: Gemini, DeepSeek, OpenRouter via `utils/llm-adapters/`
- **Interactive CLI**: provider/model/batch selection with saved config
- **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1) - **Batching with retry/split**: configurable batch size, exponential split-on-failure (4 → 2 → 1)
- **Honest timing**: unified throughput for all providers, detailed breakdown only for local - **Honest timing**: unified throughput for all providers, detailed breakdown only for local
- **Auto-target languages**: prompt dynamically excludes source language from targets - **Auto-target languages**: prompt dynamically excludes source language from targets
- **Schema validation**: validates LLM response structure before file writes - **Schema validation**: validates LLM response structure before file writes
- **Interactive CLI**: planned — provider/model/batch selection with saved config
- **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending) - **In progress:** Evaluating local models (Qwen2.5-1.5B tested; Qwen2.5-3B download pending)
- **Pending:** 20-word quality torture suite (will decide gender approach) - **Pending:** 20-word quality torture suite (will decide gender approach)
- **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq) - **Pending:** Online API evaluation (Gemini free tier, DeepSeek, Groq)
@ -95,38 +96,39 @@ No decision made. Gender handling will be determined by the 20-word quality tort
### One-Line Architecture ### One-Line Architecture
``` ```
source wordlists -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json source wordlists -> Interactive CLI -> LLM adapter (local or cloud) -> merge senses -> verify schema -> write .json
| |
[gender: LLM-generated, accuracy TBD] [gender: LLM-generated, accuracy TBD]
``` ```
### Files at a Glance ### Files at a Glance
| File | Purpose | | File | Purpose |
| ----------------------------------------- | ----------------------------------------------------------------------------------- | | ----------------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `pipeline.ts` | Orchestrator. Scans sources, loops words, coordinates all stages | | `pipeline.ts` | Orchestrator. Runs CLI, scans sources, loops words, coordinates all stages |
| `config/llm.ts` | Provider selection, API URL, model name | | `utils/cli.ts` | Interactive CLI. Provider/model/batch selection, config persistence |
| `config/prompt.ts` | `buildSystemPrompt()` — dynamic prompt with auto-target languages | | `config/llm.ts` | LLM config schema (provider, url, model) |
| `config/batch.ts` | Batch size and max retry count | | `config/prompt.ts` | `buildSystemPrompt()` — dynamic prompt with auto-target languages |
| `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` | | `config/batch.ts` | Batch size and max retry count schema |
| `utils/enrich-word.ts` | Calls LLM via adapter, parses response, builds `EnrichedSense[]`, retry/split logic | | `config/constants.ts` | Shared `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
| `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON | | `utils/enrich-word.ts` | Calls LLM via adapter, parses response, validates senses, builds `EnrichedSense[]`, retry/split logic |
| `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) | | `utils/merge-enriched-data.ts` | Merges skeleton + enriched senses into final JSON |
| `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers | | `utils/verify-enriched-file.ts` | Schema validation (required fields, types, gender enum) |
| `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory | | `utils/check-llm-server.ts` | Health check for local server; skipped for cloud providers |
| `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files | | `utils/scanning-source-files.ts` | Discovers wordlists from `source-data/` directory |
| `utils/write-json-file.ts` | Atomic `.tmp` → rename writes | | `utils/create-base-json.ts` | Writes skeleton `{word, language, pos}` files |
| `utils/check-if-json-exists.ts` | Resumability: checks if word already has enriched senses | | `utils/write-json-file.ts` | Atomic `.tmp` → rename writes |
| `utils/create-line-reader.ts` | Streaming line reader for large wordlists | | `utils/check-if-json-exists.ts` | Resumability: checks if word already has enriched senses |
| `utils/create-output-dirs.ts` | Creates `worddata/{language}/{pos}/` folders | | `utils/create-line-reader.ts` | Streaming line reader for large wordlists |
| `utils/delete-file.ts` | Cleanup helper for failed batches | | `utils/create-output-dirs.ts` | Creates `worddata/{language}/{pos}/` folders |
| `utils/get-word-file-path.ts` | Path construction helper | | `utils/delete-file.ts` | Cleanup helper for failed batches |
| `utils/progress-tracker.ts` | `[current/total]` formatting for console output | | `utils/get-word-file-path.ts` | Path construction helper |
| `utils/pipeline-timer.ts` | Timing + token metrics; unified throughput for all providers | | `utils/progress-tracker.ts` | `[current/total]` formatting for console output |
| `utils/llm-adapters/factory.ts` | Creates the right adapter based on `LLM_CONFIG.provider` | | `utils/pipeline-timer.ts` | Timing + token metrics; unified throughput for all providers |
| `utils/llm-adapters/types.ts` | `LlmAdapter` interface | | `utils/llm-adapters/factory.ts` | Creates the right adapter based on runtime config |
| `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek | | `utils/llm-adapters/types.ts` | `LlmAdapter` interface |
| `utils/llm-adapters/gemini.ts` | Google Gemini native API | | `utils/llm-adapters/openai-compatible.ts` | Local llama.cpp, OpenRouter, DeepSeek |
| `utils/llm-adapters/gemini.ts` | Google Gemini native API |
### Scale Target ### Scale Target
@ -230,7 +232,7 @@ Generating 100,000 entries with an LLM introduces risks:
### Pipeline Flow ### Pipeline Flow
``` ```
Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed Run CLI -> Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
-> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses -> Create skeletons (batch) -> Call LLM -> Parse JSON -> Validate senses
-> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics -> Retry/split on failure -> Merge -> Write atomically -> Verify schema -> Log metrics
``` ```
@ -246,12 +248,8 @@ Scan sources -> Check LLM -> Loop wordlists -> Stream words -> Skip processed
``` ```
data-pipeline/ data-pipeline/
|-- pipeline.ts # Entry point / orchestrator |-- pipeline.ts # Entry point / orchestrator
|-- config/
| |-- llm.ts # Provider, API URL, model name
| |-- prompt.ts # buildSystemPrompt() — dynamic prompt
| |-- batch.ts # Batch size and retry config
| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES
|-- utils/ |-- utils/
| |-- cli.ts # Interactive CLI module
| |-- enrich-word.ts # LLM call, parse, retry/split | |-- enrich-word.ts # LLM call, parse, retry/split
| |-- merge-enriched-data.ts # Schema types + merge logic | |-- merge-enriched-data.ts # Schema types + merge logic
| |-- verify-enriched-file.ts # Schema validation | |-- verify-enriched-file.ts # Schema validation
@ -267,10 +265,15 @@ data-pipeline/
| |-- progress-tracker.ts # Console progress formatting | |-- progress-tracker.ts # Console progress formatting
| |-- pipeline-timer.ts # Timing + token metrics | |-- pipeline-timer.ts # Timing + token metrics
| |-- llm-adapters/ | |-- llm-adapters/
| |-- factory.ts # Adapter selection | |-- factory.ts # Adapter selection (uses runtime config)
| |-- types.ts # LlmAdapter interface | |-- types.ts # LlmAdapter interface
| |-- openai-compatible.ts # Local, OpenRouter, DeepSeek | |-- openai-compatible.ts # Local, OpenRouter, DeepSeek
| |-- gemini.ts # Google Gemini | |-- gemini.ts # Google Gemini
|-- config/
| |-- llm.ts # LLM config schema
| |-- prompt.ts # buildSystemPrompt() — dynamic prompt
| |-- batch.ts # Batch config schema
| |-- constants.ts # LANG_MAP, POS_MAP, ALL_LANGUAGES
|-- source-data/ |-- source-data/
| |-- {language}/ | |-- {language}/
| |-- {pos} # One word per line, no extension | |-- {pos} # One word per line, no extension
@ -279,6 +282,7 @@ data-pipeline/
|-- {pos}/ |-- {pos}/
|-- {word}.json # One self-contained file per word |-- {word}.json # One self-contained file per word
|-- kaikki-source-files/ # Wiktionary dumps for gender lookup (planned) |-- kaikki-source-files/ # Wiktionary dumps for gender lookup (planned)
|-- .pipeline-config.json # Saved CLI configuration
``` ```
### Output Schema ### Output Schema
@ -317,25 +321,28 @@ Per-run: words processed/skipped/failed, duration, throughput, LLM token counts
| HTTP client | Native `fetch` | Works for local llama.cpp and online APIs | | HTTP client | Native `fetch` | Works for local llama.cpp and online APIs |
| File I/O | `fs` + `readline` | Streaming line reader for large wordlists | | File I/O | `fs` + `readline` | Streaming line reader for large wordlists |
| JSON | Native `JSON.parse/stringify` | Simple, no schema library needed | | JSON | Native `JSON.parse/stringify` | Simple, no schema library needed |
| CLI | Native `readline` | No external dependencies |
### Configuration ### Configuration
| File | Purpose | | File | Purpose |
| --------------------- | --------------------------------------------------------- | ------------ | ---------- | ------------------------- | | ----------------------- | --------------------------------------------------------- |
| `config/llm.ts` | `provider` (`local` | `openrouter` | `deepseek` | `gemini`), `url`, `model` | | `config/llm.ts` | Config schema: `provider`, `url`, `model` |
| `config/prompt.ts` | `buildSystemPrompt(sourceLanguage, pos, targetLanguages)` | | `config/prompt.ts` | `buildSystemPrompt(sourceLanguage, pos, targetLanguages)` |
| `config/batch.ts` | `BATCH_CONFIG.size` (words per call), `maxRetries` | | `config/batch.ts` | Config schema: `size`, `maxRetries` |
| `config/constants.ts` | `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` | | `config/constants.ts` | `LANG_MAP`, `POS_MAP`, `ALL_LANGUAGES` |
| `.pipeline-config.json` | Saved runtime config (auto-generated by CLI) |
### Key Modules ### Key Modules
| File | Responsibility | | File | Responsibility |
| ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| `utils/cli.ts` | Interactive prompts: provider → model → batch size. Saves/loads `.pipeline-config.json`. Returns runtime config. |
| `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds `EnrichedSense[]`, retry/split logic | | `utils/enrich-word.ts` | Calls LLM via adapter, strips markdown, parses JSON, validates senses, builds `EnrichedSense[]`, retry/split logic |
| `utils/merge-enriched-data.ts` | Merges skeleton with LLM senses, adds `enrichedAt` and `model` | | `utils/merge-enriched-data.ts` | Merges skeleton with LLM senses, adds `enrichedAt` and `model` |
| `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure | | `utils/verify-enriched-file.ts` | Schema validation: required fields, array lengths, gender enum, translation structure |
| `utils/pipeline-timer.ts` | Tracks per-word and global metrics; unified throughput for all providers | | `utils/pipeline-timer.ts` | Tracks per-word and global metrics; unified throughput for all providers |
| `utils/llm-adapters/factory.ts` | Creates adapter based on `LLM_CONFIG.provider` | | `utils/llm-adapters/factory.ts` | Creates adapter based on runtime config from CLI |
| `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek | | `utils/llm-adapters/openai-compatible.ts` | OpenAI chat completions API for local llama.cpp, OpenRouter, DeepSeek |
| `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API with `systemInstruction` | | `utils/llm-adapters/gemini.ts` | Google Gemini `generateContent` API with `systemInstruction` |
@ -581,14 +588,13 @@ At 1 word per call, 100,000 words = 100,000 LLM requests. Each call re-processes
### Configurable Batch Size ### Configurable Batch Size
Single config point controls batch size everywhere: Batch size is selected interactively at runtime via the CLI. The schema lives in `config/batch.ts`:
```typescript ```typescript
// config/batch.ts
export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const; export const BATCH_CONFIG = { size: 4, maxRetries: 3 } as const;
``` ```
Change `size` to 1, 2, 5, 10, 20, etc. The pipeline adapts without code changes. The CLI presents preset options (1, 5, 10, 20, 50) with provider-specific recommendations.
### Prompt Structure ### Prompt Structure
@ -735,7 +741,7 @@ Additionally, `validateSense()` in `enrich-word.ts` catches malformed senses **b
### Overview ### Overview
The pipeline includes an interactive CLI that asks the user to select provider, model, and batch size on each run. No editing of `config/llm.ts` required. The pipeline includes an interactive CLI (`utils/cli.ts`) that asks the user to select provider, model, and batch size on each run. No editing of `config/llm.ts` or `config/batch.ts` required.
### Flow ### Flow
@ -822,23 +828,52 @@ Next run shows `[1] Use last config` at the top.
| `groq` | 50 | Very fast, rate limits generous | | `groq` | 50 | Very fast, rate limits generous |
| `openrouter` | 10 | 200 req/day free tier | | `openrouter` | 10 | 200 req/day free tier |
### Implementation
The CLI is implemented in `utils/cli.ts` using Node.js native `readline` module. No external dependencies.
```typescript
// utils/cli.ts
import readline from "readline";
export async function runCli(): Promise<{
provider: string;
model: string;
url: string;
batchSize: number;
}> {
// ... interactive prompts ...
}
```
`pipeline.ts` imports and calls `runCli()` at startup:
```typescript
import { runCli } from "./utils/cli.js";
async function main() {
const config = await runCli();
// Use config.provider, config.model, etc.
}
```
--- ---
## 11. Future Extensions & Roadmap ## 11. Future Extensions & Roadmap
### Near-Term (Next 2-4 Weeks) ### Near-Term (Next 2-4 Weeks)
| Item | Status | Notes | | Item | Status | Notes |
| ---------------------- | ------------ | ------------------------------------------------------------- | | ---------------------- | ------------ | ----------------------------------------------------------------- |
| Configurable batching | **Complete** | `config/batch.ts` with `size` and `maxRetries` | | Configurable batching | **Complete** | `config/batch.ts` with `size` and `maxRetries` |
| Retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch | | Retry + split logic | **Complete** | `enrichWordWithRetry`: 3 retries, then halve batch |
| Honest timing metrics | **Complete** | Unified throughput for all providers, detailed only for local | | Honest timing metrics | **Complete** | Unified throughput for all providers, detailed only for local |
| Auto-target languages | **Complete** | `buildSystemPrompt()` excludes source from targets | | Auto-target languages | **Complete** | `buildSystemPrompt()` excludes source from targets |
| Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes | | Validate LLM responses | **Complete** | `validateSense()` catches bad data before writes |
| Interactive CLI | Planned | Provider/model/batch selection with saved config | | Interactive CLI | **Complete** | `utils/cli.ts` — provider/model/batch selection with saved config |
| 20-word torture suite | Pending | Decides gender approach and model selection | | 20-word torture suite | Pending | Decides gender approach and model selection |
| Qwen2.5-3B evaluation | Pending | Download and test | | Qwen2.5-3B evaluation | Pending | Download and test |
| Online API testing | Pending | Gemini free tier, DeepSeek, Groq | | Online API testing | Pending | Gemini free tier, DeepSeek, Groq |
### Medium-Term (1-3 Months) ### Medium-Term (1-3 Months)
@ -863,24 +898,25 @@ Next run shows `[1] Use last config` at the top.
## 12. Decisions Log ## 12. Decisions Log
| Date | Decision | Context | Rationale | | Date | Decision | Context | Rationale |
| ---------- | ----------------------------------- | -------------------------------------------------------- | --------------------------------------------------------------------------- | | ---------- | ----------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------------- |
| 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app | | 2026-01-04 | TanStack Router for frontend | Previous project used React Router | Simpler, type-safe routing for the trainer app |
| 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying | | 2026-01-04 | Vite dev server (no Nginx) | Docker setup for glossa-web | Nginx unnecessary for dev; Vite handles HMR and proxying |
| 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas | | 2026-01-17 | Backend answer verification | Security vulnerability: correctAnswer exposed in API | Moved verification to server-side, shared schemas |
| 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys | | 2026-03-26 | Multi-stage Docker builds | glossa-api and glossa-web containers | Smaller images, faster deploys |
| 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies | | 2026-06-16 | llama.cpp for local LLM | Need local inference on old laptop | GGUF format, OpenAI-compatible API, no dependencies |
| 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference | | 2026-06-16 | Q4_K_M quantization | Balance size vs quality | Q4_K_M is the community standard for 4-bit inference |
| 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM | | 2026-06-16 | `-c 2048` context | Default was 4096 | Dictionary entries need ~500 tokens max; frees VRAM |
| 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance | | 2026-06-16 | `-t 2` physical cores | Default was 4 (HT threads) | Hyperthreading hurts llama.cpp performance |
| 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation | | 2026-06-17 | Qwen2.5-1.5B as current model | Qwen3.5-4B too slow (47s/word) | 6x speedup (8s/word), quality under evaluation |
| 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality | | 2026-06-17 | Skip Gemma 4 | E2B Q4_K_M is 3.46GB | Does not fit in 4GB VRAM; lower quants sacrifice quality |
| 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion | | 2026-06-17 | Skip Ministral-3B | Tokenizer mismatch (Tekken) | Outputs gibberish regardless of template; not fixable without re-conversion |
| 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations | | 2026-07-06 | Adapter pattern for LLM providers | Need to evaluate local vs cloud | `utils/llm-adapters/` with factory + types + per-provider implementations |
| 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 | | 2026-07-06 | Retry + split batching | LLM JSON parse failures on larger batches | `enrichWordWithRetry` retries 3 times, then halves batch until size 1 |
| 2026-07-06 | Honest timing metrics | Cloud providers don't expose prompt/completion breakdown | Unified `totalTimeMs` for all; detailed breakdown only when available | | 2026-07-06 | Honest timing metrics | Cloud providers don't expose prompt/completion breakdown | Unified `totalTimeMs` for all; detailed breakdown only when available |
| 2026-07-06 | Auto-target languages | Prompt hardcoded English -> de/it/es/fr | `ALL_LANGUAGES` minus source = targets; works for any source language | | 2026-07-06 | Auto-target languages | Prompt hardcoded English -> de/it/es/fr | `ALL_LANGUAGES` minus source = targets; works for any source language |
| 2026-07-06 | Validate LLM responses before write | Bad data was written then warned about | `validateSense()` catches malformed responses early, triggers retry | | 2026-07-06 | Validate LLM responses before write | Bad data was written then warned about | `validateSense()` catches malformed responses early, triggers retry |
| 2026-07-06 | Interactive CLI | Editing `config/llm.ts` to switch providers is error-prone | `utils/cli.ts` with native `readline`; no external dependencies; saves config |
--- ---
@ -952,14 +988,14 @@ cd ~/Downloads/llama.cpp
--prio 2 --prio 2
``` ```
### Run the Pipeline (Interactive CLI) ### Run the Pipeline
```bash ```bash
cd /path/to/data-pipeline cd /path/to/data-pipeline
npx tsx pipeline.ts npx tsx pipeline.ts
``` ```
Follow the prompts to select provider, model, and batch size. Follow the interactive prompts to select provider, model, and batch size. Or select "Use last config" to reuse previous settings.
### Run with Last Config ### Run with Last Config
@ -1045,14 +1081,14 @@ Global data pipeline run completed successfully.
--- ---
### Phase 2: Interactive CLI ### Phase 2: Interactive CLI (Complete)
| Task | Status | Notes | | Task | Status | Notes |
| --------------------- | ------- | ----------------------------------------- | | --------------------- | ------------ | ------------------------------------------ |
| Design prompt flow | Planned | Provider → model → batch size → confirm | | Design prompt flow | **Complete** | Provider → model → batch size → confirm |
| Implement CLI module | Planned | Use `readline` or `inquirer` for prompts | | Implement CLI module | **Complete** | `utils/cli.ts` with native `readline` |
| Save/load config | Planned | `.pipeline-config.json` | | Save/load config | **Complete** | `.pipeline-config.json` |
| Wire into pipeline.ts | Planned | Replace static config with runtime config | | Wire into pipeline.ts | **Complete** | Replaces static config with runtime config |
**Goal:** No editing of TypeScript files to switch providers. **Goal:** No editing of TypeScript files to switch providers.