342 lines
9.8 KiB
Markdown
342 lines
9.8 KiB
Markdown
# english nouns
|
||
|
||
## step 1a
|
||
|
||
get freqency source list + verify lemmatization + confirm language/POS tagging
|
||
|
||
example:
|
||
|
||
house
|
||
bank
|
||
asdf
|
||
|
||
## step 1b
|
||
|
||
transform list into JSON (headword, language, POS)
|
||
|
||
example:
|
||
|
||
```json
|
||
{
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun"
|
||
},
|
||
{
|
||
"headword": "bank",
|
||
"language": "en",
|
||
"pos": "noun"
|
||
},
|
||
{
|
||
"headword": "asdf",
|
||
"language": "en",
|
||
"pos": "noun"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## step 2a
|
||
|
||
check existence in kaikki (eg. is there a english noun "asdf" in kaikki, if not put it separate list)
|
||
|
||
example output:
|
||
|
||
```json
|
||
{
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun"
|
||
},
|
||
{
|
||
"headword": "bank",
|
||
"language": "en",
|
||
"pos": "noun"
|
||
}
|
||
```
|
||
|
||
miss list (triage queue, not trash! contains junk, slipped inflections, and real Kaikki gaps, gets handled separately, never auto-drop!)
|
||
|
||
```json
|
||
{ "headword": "asdf", "language": "en", "pos": "noun" }
|
||
```
|
||
|
||
## step 2b
|
||
|
||
for each sense, extract: glosses, translations (with gender), examples
|
||
id will be generated via: `headword:lang:pos:sense_index`
|
||
|
||
words without glosses will be dropped! (put in special list)
|
||
|
||
example output:
|
||
|
||
```json
|
||
{
|
||
"id": "house:en:noun:0",
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"sense": ["A building for human habitation."],
|
||
"examples": ["They bought a house in the city."],
|
||
"translations": {
|
||
"de": [{ "word": "Haus", "gender": "neuter" }],
|
||
"it": [{ "word": "casa", "gender": "feminine" }],
|
||
"es": [{ "word": "casa", "gender": "feminine" }],
|
||
"fr": [{ "word": "maison", "gender": "feminine" }]
|
||
}
|
||
},
|
||
{
|
||
"id": "house:en:noun:1",
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"sense": ["A noble family or lineage."],
|
||
"examples": ["The House of Tudor ruled England."],
|
||
"translations": {
|
||
"de": [
|
||
{ "word": "Adelsgeschlecht", "gender": "neuter" },
|
||
{ "word": "Haus", "gender": "neuter" }
|
||
]
|
||
}
|
||
},
|
||
{
|
||
"id": "bank:en:noun:0",
|
||
"headword": "bank",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"sense": ["An institution where one can place and borrow money."],
|
||
"examples": ["She deposited her paycheck at the bank."],
|
||
"translations": {
|
||
"de": [{ "word": "Bank", "gender": "feminine" }],
|
||
"it": [{ "word": "banca", "gender": "feminine" }],
|
||
"es": [{ "word": "banco", "gender": "masculine" }],
|
||
"fr": [{ "word": "banque", "gender": "feminine" }]
|
||
}
|
||
},
|
||
{
|
||
"id": "bank:en:noun:1",
|
||
"headword": "bank",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"sense": ["The land alongside a river or lake."],
|
||
"examples": ["They picnicked on the bank of the river."],
|
||
"translations": {
|
||
"de": [{ "word": "Ufer", "gender": "neuter" }],
|
||
"it": [{ "word": "riva", "gender": "feminine" }],
|
||
"es": [{ "word": "orilla", "gender": "feminine" }],
|
||
"fr": [{ "word": "rive", "gender": "feminine" }]
|
||
}
|
||
},
|
||
{
|
||
"id": "bank:en:noun:2",
|
||
"headword": "bank",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"sense": ["A collection or store of something held in reserve."],
|
||
"examples": ["The hospital keeps a blood bank."],
|
||
"translations": {
|
||
"de": [{ "word": "Bank", "gender": "feminine" }]
|
||
}
|
||
}
|
||
```
|
||
|
||
## step 2c
|
||
|
||
fill gaps Kaikki left (LLM, 3 models, generate-then-vote)
|
||
|
||
takes 2b's partial cards. 3 different-family models (qwen, llama, gemma) generate, then vote.
|
||
|
||
separate focused sub-passes, one field at a time — never combined:
|
||
|
||
- **translations** → per missing language, generated with the gloss as context. generate → vote. no agreement → gap stays (cloud audit later)
|
||
- **examples** → for senses with no Kaikki example. generate → verify-vote ("is this a valid example of the gloss?"), since freeform sentences never exact-match. no agreement → no example (card still valid) no tiebreak: no agreement leaves the gap, never escalates to more models. vote records stored in pipeline.db for the cloud audit.
|
||
- **invariant**: the gloss is never LLM-generated — translations are filled, examples generated, difficulty graded, but the gloss must always be Kaikki's (no gloss → sense dropped in 2b)
|
||
|
||
example input:
|
||
|
||
```json
|
||
{
|
||
"id": "harbor:en:noun:0",
|
||
"headword": "harbor",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"glosses": ["A sheltered area of water where ships can dock safely."],
|
||
"examples": [],
|
||
"translations": { "de": [{ "word": "Hafen", "gender": "masculine" }] }
|
||
}
|
||
```
|
||
|
||
next substage is the translation generation:
|
||
|
||
```json
|
||
{
|
||
"id": "harbor:en:noun:0",
|
||
"headword": "harbor",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"glosses": ["A sheltered area of water where ships can dock safely."],
|
||
"examples": [],
|
||
"translations": {
|
||
"de": [{ "word": "Hafen", "gender": "masculine" }],
|
||
"it": [{ "word": "porto", "gender": null }],
|
||
"es": [{ "word": "puerto", "gender": null }],
|
||
"fr": [{ "word": "port", "gender": null }]
|
||
}
|
||
}
|
||
```
|
||
|
||
next substage is the gender modification (by looking up kaikki for it/es/fr or corresponding wiktionary, fill only nulls, not touching existing genders):
|
||
**runs only after all genders are present**
|
||
|
||
```json
|
||
{
|
||
"id": "harbor:en:noun:0",
|
||
"headword": "harbor",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"glosses": ["A sheltered area of water where ships can dock safely."],
|
||
"examples": [],
|
||
"translations": {
|
||
"de": [{ "word": "Hafen", "gender": "masculine" }],
|
||
"it": [{ "word": "porto", "gender": "masculine" }],
|
||
"es": [{ "word": "puerto", "gender": "masculine" }],
|
||
"fr": [{ "word": "port", "gender": "masculine" }]
|
||
}
|
||
}
|
||
```
|
||
|
||
next substage is the example generation:
|
||
|
||
```json
|
||
{
|
||
"id": "harbor:en:noun:0",
|
||
"headword": "harbor",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"glosses": ["A sheltered area of water where ships can dock safely."],
|
||
"examples": ["The fishing boats returned to the harbor at dusk."],
|
||
"translations": {
|
||
"de": [{ "word": "Hafen", "gender": "masculine" }],
|
||
"it": [{ "word": "porto", "gender": "masculine" }],
|
||
"es": [{ "word": "puerto", "gender": "masculine" }],
|
||
"fr": [{ "word": "port", "gender": "masculine" }]
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## step 3
|
||
|
||
adding difficulty level
|
||
|
||
CEFR is mapped to three buckets:
|
||
A1/A2 → easy
|
||
B1/B2 → intermediate
|
||
C1/C2 → hard
|
||
|
||
depending on number of senses per headword:
|
||
|
||
- One sense → derive difficulty from the CEFRLex distribution (first CEFR level crossing a frequency threshold, mapped to a bucket, not the peak), for the four covered languages (en/de/es/fr). Deterministic, no LLM
|
||
- Multiple senses → the three local LLMs grade each sense (easy/intermediate/hard) from the gloss. CEFRLex not involved.
|
||
- No CEFRLex entry at all (Italian, or single-sense word that's missing) → LLMs grade from gloss
|
||
|
||
to get coverage (not quality), 3 local llms are going to be used: gemma, qwen and llama
|
||
|
||
on the llm votes:
|
||
|
||
- **Majority agrees (3-0 or 2-1)** → ship the majority value.
|
||
- **3-way split (all three differ)** → no consensus → exclude the card to the review queue. Not shipped.
|
||
|
||
example output:
|
||
|
||
```json
|
||
{
|
||
"id": "house:en:noun:0",
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"difficulty_level": "easy",
|
||
"glosses": ["A building for human habitation."],
|
||
"examples": ["They bought a house in the city."],
|
||
"translations": {
|
||
"de": [{ "word": "Haus", "gender": "neuter" }],
|
||
"it": [{ "word": "casa", "gender": "feminine" }],
|
||
"es": [{ "word": "casa", "gender": "feminine" }],
|
||
"fr": [{ "word": "maison", "gender": "feminine" }]
|
||
}
|
||
},
|
||
{
|
||
"id": "house:en:noun:1",
|
||
"headword": "house",
|
||
"language": "en",
|
||
"pos": "noun",
|
||
"difficulty_level": "hard",
|
||
"glosses": ["A noble family or lineage."],
|
||
"examples": ["The House of Tudor ruled England."],
|
||
"translations": {
|
||
"de": [
|
||
{ "word": "Adelsgeschlecht", "gender": "neuter" },
|
||
{ "word": "Haus", "gender": "neuter" }
|
||
]
|
||
}
|
||
}
|
||
```
|
||
|
||
**important**: `difficulty_level` is mandatory on every shipped card (the user sets difficulty before a game)
|
||
|
||
This is coverage, not final quality: three local models make every shipped card's difficulty present and plausible, but not guaranteed correct
|
||
|
||
no cefr list: needs to be graded by online llms (see notes)
|
||
|
||
---
|
||
|
||
## coverage report (trial deliverable)
|
||
|
||
- not a data stage => reads the finished cards and summarises them
|
||
- the trial (english + italian nouns, local models) runs steps 1a→3 then emits a report
|
||
|
||
### pipeline health — did it run?
|
||
|
||
- input lemmas → output cards (ratio; multi-sense makes cards > lemmas)
|
||
- dropped per stage + why: 2a misses (junk/inflection/gap), 2b no-gloss drops, 3 no-consensus exclusions
|
||
- parse failures / errors
|
||
|
||
### coverage — how complete?
|
||
|
||
- translation completeness: all 4 langs vs gaps, per language
|
||
- examples: from Kaikki vs LLM-generated vs none
|
||
- gender: translations still null after Wiktionary fill
|
||
- difficulty distribution per bucket (lopsided = broken CEFRLex threshold or bad grading)
|
||
|
||
### local good enough? — model agreement
|
||
|
||
- difficulty votes: 3-0 / 2-1 / 3-way split rates (high 3-way = locals can't do it → need cloud)
|
||
- translation gap-fill: agreement rate, gaps left unfilled
|
||
- → go/no-go on local-for-launch
|
||
|
||
### rent vs API? — workload volume
|
||
|
||
- total LLM calls across 2c + 3 (× per-token rate = API cost)
|
||
- cards needing LLM vs handled deterministically (english: high deterministic; italian: ~0, no CEFRLex)
|
||
- per-language call counts → scale english (low) and italian (high) to estimate the middle three
|
||
|
||
### review backlog — what's deferred
|
||
|
||
- miss-list size + composition, per language
|
||
- no-consensus exclusions (cloud-audit queue size)
|
||
- unfilled translation gaps (also cloud-audit work)
|
||
|
||
**framing**: english = optimistic floor (rich Kaikki, CEFRLex exists, models strongest).
|
||
italian = pessimistic ceiling (no CEFRLex, thinner Kaikki). all five languages sit between.
|
||
read to decide, not to archive.
|
||
|
||
---
|
||
|
||
Notes:
|
||
|
||
- the miss list needs to get verified/re-worked later on
|
||
- the kaikki data contains ipas and links to audio files, add them later if needed, they are not needed now
|
||
- use online llms to set the difficulty(cefr) of the not shipped words
|
||
- add plurals from kaikki/wiktionary
|
||
- postgres sync to prod db
|