173 lines
14 KiB
Markdown
173 lines
14 KiB
Markdown
# Model Strategy
|
||
|
||
## The problem
|
||
|
||
The pipeline requires LLMs to perform four tasks per vocabulary entry:
|
||
|
||
1. **Gloss review** — confirm or improve the existing gloss
|
||
2. **Example review** — confirm or improve existing examples
|
||
3. **Translation validation** — confirm valid translations, reject bad data, generate missing ones
|
||
4. **CEFR assignment** — assign A1-C2 to the headword and each translation
|
||
|
||
The core challenge is that vocabulary entries have **multiple senses**. The word "cat" appears five times in the database — as an animal, as slang for "guy", as a nautical term, as a verb meaning "to vomit", and as a verb meaning "to hoist an anchor". Each sense requires a different CEFR level and different translations. A model that only knows "cat" is A1 gets four out of five wrong.
|
||
|
||
This makes CEFR assignment fundamentally a **sense-disambiguation problem**, not just a vocabulary lookup. Specialized CEFR classifiers (like `cefrpy` or `dksysd/cefr-classifier`) operate at the word or sentence level and cannot distinguish between senses of the same word. General LLMs handle sense disambiguation well but introduce quality and reliability problems that depend heavily on model size.
|
||
|
||
The secondary challenge is **hardware constraints**. The available local hardware (GTX 950M, 4GB VRAM) can only run models up to approximately 4B parameters fully in GPU memory. Larger models run in hybrid CPU/GPU mode which is significantly slower. Free cloud API tiers are generous enough for the sample dataset but have daily limits that make processing 100k+ entries across multiple sub-stages a multi-day or multi-week operation.
|
||
|
||
## What we tried and why it failed or worked
|
||
|
||
### Single-prompt design (abandoned)
|
||
|
||
The first enrich script sent one large prompt per entry covering all four tasks at once — CEFR voting, gloss improvement, example improvement, translation validation, and missing translation generation. This produced the following problems:
|
||
|
||
- The model skipped translations it considered invalid rather than explicitly rejecting them, causing validation failures
|
||
- Bad data in the translation table (`it:free`, `de:-frei`, `es:de fai`) caused consistent validation failures because the model refused to vote on them even when explicitly instructed
|
||
- The combined prompt was large enough to trigger reasoning mode on Gemma 4 E4B, consuming all available tokens on thinking before producing output
|
||
- 20% of entries required manual review
|
||
|
||
### Sub-stage design (current)
|
||
|
||
Splitting into four ordered sub-stages fixed the reasoning and validation problems:
|
||
|
||
1. `round1_gloss` — LLM reviews the gloss in isolation
|
||
2. `round1_example` — LLM reviews examples with verified gloss as context
|
||
3. `round1_translations` — LLM validates translations with verified gloss as context
|
||
4. `round1_cefr` — LLM assigns CEFR levels only to validated translations
|
||
|
||
This ordering ensures the CEFR sub-stage never sees bad data. The smaller, focused prompts eliminated reasoning mode triggering and reduced per-entry time from ~120 seconds to ~25 seconds.
|
||
|
||
### Gloss quality (ongoing)
|
||
|
||
Testing on 50 entries with Qwen3.5-4B showed ~80% good quality. The 20% failures fall into three categories:
|
||
|
||
- **Category header glosses** — Kaikki occasionally uses "Terms relating to people." or "Terms relating to things." as a gloss instead of a real definition. No model handles these correctly because there is no real meaning to improve.
|
||
- **Rare/obscure senses** — slang, archaic, and theological senses that a 4B model does not have enough knowledge to handle (e.g. "cat" meaning "to vomit", "word" meaning "Logos, Christ").
|
||
- **Short ambiguous glosses** — one or two word glosses with no example context cause hallucination.
|
||
|
||
### Gemma 4 E4B (rejected)
|
||
|
||
Gemma 4 E4B is a hybrid reasoning model. Disabling thinking via `--reasoning-budget 0` or `--chat-template-kwargs '{"enable_thinking":false}'` does not work reliably in llama.cpp for the E4B variant — the model either puts reasoning into the content field as plain text or returns empty content with reasoning in `reasoning_content`. Per-entry time exceeded 100 seconds making it impractical.
|
||
|
||
### Qwen3.5-4B (current local model)
|
||
|
||
Non-thinking by default for the small series. Runs fully in 4GB VRAM at ~5 seconds per sub-stage. Acceptable quality for common vocabulary (A1-B2) but struggles with rare and specialized senses. Used as the primary local voter.
|
||
|
||
### Specialized CEFR classifiers (rejected for primary use)
|
||
|
||
HuggingFace hosts several CEFR text classifiers (`dksysd/cefr-classifier`, `AbdulSami/bert-base-cased-cefr`) and the `cefrpy` Python library maps individual words to CEFR levels. These operate at the word or sentence level and cannot distinguish between senses. "cat" would always be assigned A1 regardless of whether the sense is the animal or obscure nautical slang. Useful only as a sanity check signal, not as a primary voter.
|
||
|
||
## Available free resources
|
||
|
||
| Resource | Type | Requests/day | Quality | Notes |
|
||
| ---------------------------- | ------------------ | ----------------- | --------- | ---------------------------------------------------------------------- |
|
||
| Local Qwen3.5-4B Q4_K_M | Local model | Unlimited | Decent | Non-thinking by default, fits in 4GB VRAM, ~5s per sub-stage |
|
||
| Local Qwen3.5-9B Q4_K_M | Local model | Unlimited | Good | Hybrid CPU/GPU mode on 4GB VRAM, slower but better quality |
|
||
| Local Llama 3.1 8B Q4_K_M | Local model | Unlimited | Decent | ~4.3GB, fits in VRAM or light hybrid, different architecture from Qwen |
|
||
| Groq — Llama 3.3 70B | Cloud API | 1,000 | Excellent | Best free quality available, 5-10x with batching |
|
||
| Groq — Llama 3.1 8B | Cloud API | 14,400 | Decent | High volume, similar quality to local 4B |
|
||
| Google Gemini AI Studio | Cloud API | 1,500 | Very good | Google account required, 5-10x with batching |
|
||
| OpenRouter free rotation | Cloud API | 50–1,000 | Varies | Rotates between free models automatically via `openrouter/free` |
|
||
| Wiktionary API | Context enrichment | Unlimited | N/A | Structured vocabulary data, directly related to Kaikki source |
|
||
| `cefrpy` Python library | Word lookup | Unlimited | Limited | Deterministic English word CEFR lookup, no sense disambiguation |
|
||
| HuggingFace CEFR classifiers | Text classifier | Unlimited (local) | Limited | Sentence-level difficulty, not sense-aware |
|
||
|
||
### Batching
|
||
|
||
All cloud APIs support sending multiple entries in a single request. Sending 5 entries per request multiplies effective daily capacity by 5x:
|
||
|
||
- Groq Llama 3.3 70B: 1,000 requests → ~5,000 entries/day
|
||
- Gemini: 1,500 requests → ~7,500 entries/day
|
||
|
||
### Multiple accounts
|
||
|
||
Prohibited by the terms of service of all providers listed above.
|
||
|
||
## Final approach per sub-stage
|
||
|
||
The pipeline runs multiple models as independent voters. Each model processes every entry once and writes its votes to `pipeline.db`. The merge stage resolves disagreements by majority vote. A tiebreaker runs additional models on flagged entries where no majority was reached.
|
||
|
||
### round1_gloss and round1_example
|
||
|
||
These sub-stages require a model that understands sense context from examples. Specialized classifiers cannot help here — only general LLMs can evaluate whether a gloss correctly describes a specific sense.
|
||
|
||
**Primary voter:** Local Qwen3.5-9B Q4_K_M — runs overnight, unlimited, handles common vocabulary well.
|
||
|
||
**Secondary voter:** Groq Llama 3.3 70B with 5-entry batching — higher quality, catches errors the local model makes on rare or specialized senses.
|
||
|
||
**Tertiary voter:** Gemini AI Studio with 5-entry batching — third independent opinion, different training data from both Groq and local model.
|
||
|
||
**Context enrichment via Wiktionary API:** Before calling any model for the gloss or example sub-stage, the pipeline queries the Wiktionary API for the headword. The API returns the full Wiktionary entry including all senses, usage notes, and examples. This structured data is added to the prompt as additional context, giving the model a much clearer picture of which specific sense it is working with.
|
||
|
||
This directly fixes the two hardest failure cases:
|
||
- **Category header glosses** ("Terms relating to people.") — the Wiktionary entry contains the real definition which the model can use to generate a proper gloss
|
||
- **Short ambiguous glosses** — the additional sense context prevents the model from guessing the wrong meaning
|
||
|
||
The Wiktionary API is free, has no rate limits for reasonable use, and is directly related to the Kaikki data source since Kaikki extracts from Wiktionary.
|
||
|
||
### round1_translations
|
||
|
||
Same voter stack as gloss/example. The few-shot examples in the prompt (showing that `it:free` → reject and `de:-frei` → reject) handle the bad data cases that caused validation failures in the single-prompt design.
|
||
|
||
### round1_cefr
|
||
|
||
This sub-stage only receives translations that survived the validation step. All bad data is already excluded.
|
||
|
||
**Primary voter:** Local Qwen3.5-9B Q4_K_M.
|
||
|
||
**Secondary voter:** Groq Llama 3.3 70B with 5-entry batching.
|
||
|
||
**Tertiary voter:** Gemini AI Studio with 5-entry batching.
|
||
|
||
**Sanity check:** `cefrpy` provides a deterministic English word CEFR level as a reference signal. If the majority LLM vote disagrees significantly (e.g. LLMs vote C2 for "cat" the animal), the entry is flagged for human review. `cefrpy` does not vote — it only triggers review flags.
|
||
|
||
### Voter summary
|
||
|
||
| Sub-stage | Voter 1 | Voter 2 | Voter 3 |
|
||
| ------------------- | ------------------ | ------------------ | ------- |
|
||
| round1_gloss | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
|
||
| round1_example | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
|
||
| round1_translations | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
|
||
| round1_cefr | Qwen3.5-9B (local) | Groq Llama 3.3 70B | Gemini |
|
||
|
||
Three voters means a correct majority requires at least two models to agree. Even if the local model gets a difficult sense wrong, the two cloud models will likely agree on the correct answer and outvote it.
|
||
|
||
## Open questions
|
||
|
||
### Wiktionary API context extraction
|
||
The Wiktionary API returns the full entry for a word including all senses. For a word like "free" with 8+ senses, dumping the entire entry into the prompt wastes tokens and may confuse the model. The open question is how to extract only the relevant sense — options include matching by sense_index, fuzzy-matching the Kaikki gloss against Wiktionary glosses, or letting the model see all senses and identify the correct one itself.
|
||
|
||
### Batching prompt design
|
||
Batching 5-10 entries per API call multiplies effective daily capacity significantly. The prompt and validation logic for batched requests is more complex — the model must return a structured JSON object keyed by entry ID, and partial failures (one entry in a batch fails validation) need careful handling. Not yet designed or tested.
|
||
|
||
### Groq and Gemini API integration
|
||
Neither Groq nor Gemini is integrated into the pipeline yet. Both use OpenAI-compatible APIs so integration is straightforward — add provider configs to `stage-3-enrich/config.ts` and set API keys in `.env`. The batching prompt design needs to be finalised first.
|
||
|
||
### OpenRouter free model rotation
|
||
OpenRouter's `openrouter/free` router selects a model at random from available free models. This means output style and quality vary between requests, which complicates round 2 voting where models review each other's candidates. May need to pin specific free models rather than using the router.
|
||
|
||
### Qwen3.5-9B performance on hard cases
|
||
The 9B model has not yet been tested. It is expected to handle rare and specialized senses better than the 4B model but this has not been verified. Needs a test run against the same 50 entries used to evaluate the 4B model.
|
||
|
||
### Llama.cpp Gemma 4 bug
|
||
The llama.cpp chat template bug preventing reliable JSON output from Gemma 4 E4B may be fixed in a future release. The model fits in 4GB VRAM and would be a useful additional local voter if the bug is resolved. Worth checking periodically.
|
||
|
||
### Full dataset scale
|
||
The current pipeline runs on a 500-entry sample per language. The full Kaikki English file contains approximately 1.3 million entries, of which a fraction will pass the POS and translation filters. The exact count and the time required to run all sub-stages across all models at full scale is not yet known.
|
||
|
||
### Category header glosses
|
||
Kaikki occasionally uses category headers ("Terms relating to people.", "Terms relating to things.") as glosses. These are not real definitions and no model produces useful output for them. Options include pre-filtering them before the gloss sub-stage and generating a gloss purely from examples, or flagging them as a special case for human review.
|
||
|
||
|
||
|
||
|
||
wget -O models/llama-3.1-8b-instruct-q4_k_m.gguf \
|
||
"https://huggingface.co/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/resolve/main/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf"
|
||
|
||
# Q4_K_M (5.68GB — hybrid mode, better quality)
|
||
wget -O models/qwen3.5-9b-q4_k_m.gguf \
|
||
"https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf"
|
||
|
||
# Q3_K_S (4.32GB — might fit fully in VRAM)
|
||
wget -O models/qwen3.5-9b-q3_k_s.gguf \
|
||
"https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q3_K_S.gguf"
|