lila/scripts/README.md

CEFR Data Pipeline

This directory contains the source data files and extraction/merge pipeline for generating CEFR-enriched datasets. The final outputs (english-merged.json, italian-merged.json) are consumed by the database seeding process in packages/db.

Overview

The pipeline transforms raw vocabulary data from multiple sources into a standardized format, resolves conflicts between sources, and produces an authoritative CEFR dataset per language. This dataset is then used by the lila database package to update translation records.

Supported Languages

• ✅ English (en)
• ✅ Italian (it)

Pipeline Stages

Stage 1: Extraction

Each source file is processed by a dedicated extractor script. The extractor reads the source-specific format, normalizes the data, filters for supported parts of speech, and outputs a standardized JSON file.

Input: Raw source files (JSON, CSV, XLS) Output: {source}-extracted.json files (same directory as source)

Normalization rules:

• Words are lowercased and trimmed
• Part of speech is mapped to supported values (noun, verb)
• Entries with unsupported POS are skipped
• CEFR levels are validated against A1-C2
• Each record includes the source identifier for traceability

Extractor Scripts:

Language	Source	Script
English	cefrj.csv	extraction-scripts/english/extract-cefrj-csv.py
English	en_m3.xls	extraction-scripts/english/extract-en_m3.py
English	octanove.csv	extraction-scripts/english/extract-octanove.py
English	random.json	extraction-scripts/english/extract-random-json.py
Italian	it_m3.xls	extraction-scripts/italian/extract-it_m3.py
Italian	italian.json	extraction-scripts/italian/extract-italian-json.py

Stage 2: Comparison

Before merging, sources are compared to identify agreements and conflicts. This stage is read-only and serves as a quality gate.

Input: All {source}-extracted.json files for a language Output: Console report showing:

• Entry counts per source and CEFR level
• Overlap between sources (words appearing in multiple sources)
• Agreement rate (sources assigning the same CEFR level)
• Conflicts (same word/POS with different CEFR levels)

Comparison Scripts:

Language	Script
English	comparison-scripts/compare-english.py
Italian	comparison-scripts/compare-italian.py

Run from the scripts/ directory:

python comparison-scripts/compare-english.py
python comparison-scripts/compare-italian.py

Stage 3: Merge

Multiple extracted sources are merged into a single authoritative JSON file per language. When the same word/POS appears in multiple sources with different CEFR levels, the conflict is resolved using a predefined priority order.

Input: All {source}-extracted.json files for a language Output: {language}-merged.json in ../datafiles/

Merge rules:

• Single source: use that source's CEFR level
• Multiple sources agree: use the agreed CEFR level
• Multiple sources conflict: use the level from the highest-priority source

Difficulty derivation: Difficulty is not extracted from sources. It is derived from the final CEFR level:

• A1, A2 → easy
• B1, B2 → intermediate
• C1, C2 → hard

The merged file includes both CEFR level and derived difficulty, plus a list of sources that contributed to each entry.

Merge Scripts & Priorities:

Language	Script	Priority (lowest → highest)
English	merge-scripts/merge-english-json.py	random, octanove, cefrj, en_m3
Italian	merge-scripts/merge-italian-json.py	italian, it_m3

Run from the scripts/ directory:

python merge-scripts/merge-english-json.py
python merge-scripts/merge-italian-json.py

Stage 4: Enrichment

The authoritative merged file is consumed by the database package (packages/db) during the seeding or update process. This stage is implemented in TypeScript and is not part of the Python scripts in this directory.

File Organization

scripts/
├── comparison-scripts/
│ ├── compare-english.py
│ └── compare-italian.py        # Stage 2: compare extracted data
├── datafiles/
│   ├── english-merged.json # Stage 3 output (authoritative)
│   ├── italian-merged.json # Stage 3 output (authoritative)
│   ├── omw-noun.json
│   └── omw-verb.json
├── data-sources/
│   ├── english/
│   │   ├── cefrj.csv
│   │   ├── cefrj-extracted.json
│   │   ├── en_m3.xls
│   │   ├── en_m3-extracted.json
│   │   ├── octanove.csv
│   │   ├── octanove-extracted.json
│   │   ├── random.json
│   │   └── random-extracted.json
│   ├── french/                     # (future)
│   ├── german/                     # (future)
│   ├── italian/
│   │   ├── it_m3.xls
│   │   ├── it_m3-extracted.json
│   │   ├── italian.json
│   │   └── italian-extracted.json
│   └── spanish/                    # (future)
├── extraction-scripts/
│   └── english/
│       ├── extract-cefrj-csv.py
│       ├── extract-en_m3.py
│       ├── extract-octanove.py
│       └── extract-random-json.py
│   └── italian/
│       ├── extract-it_m3.py
│       └── extract-italian-json.py
├── merge-scripts/
│   └── merge-english-json.py       # Stage 3: merge into authority
├── extract-own-save-to-json.py # script to extract words from wordnet
├── requirements.txt
└── README.md                   # This file

Extracted files are co-located with their sources for easy traceability. Merged files live in ../datafiles/.

Source Priority by Language

Source priority determines which CEFR level wins when sources conflict:

English:

1. en_m3
2. cefrj
3. octanove
4. random

Italian:

1. it_m3
2. italian

Priority is defined in the merge configuration. Higher priority sources override lower priority sources when conflicts occur.

This is defined in merge-scripts/merge-english-json.py.

Data Flow Summary

Raw Source → Extracted JSON → Merged JSON → Database
    (1)           (2)            (3)           (4)

1. Extract: Transform source formats to normalized records
2. Compare: Validate source quality and surface conflicts
3. Merge: Resolve conflicts, derive difficulty, create authority
4. Enrich: Write to database (handled in packages/db)

Adding New Sources

To add a new source:

1. Place the raw file in the appropriate data-sources/{language}/ directory
2. Create an extractor script in ../extractors/{language}/
3. Run the extractor to generate {source}-extracted.json
4. Run comparison to assess coverage and conflicts
5. Update source priority in the merge configuration if needed
6. Run merge to regenerate the authoritative file
7. Run enrichment to update the database

Constants and Constraints

The pipeline respects these constraints from the lila shared constants:

• Supported languages: en, it
• Supported parts of speech: noun, verb
• CEFR levels: A1, A2, B1, B2, C1, C2
• Difficulty levels: easy, intermediate, hard

Entries violating these constraints are filtered out during extraction.