lila/data-pipeline
lila 37c978e230 implementing phase 3 pipeline: gemini structured output, validation, sqlite staging
Prompt is now a template (fixes the hardcoded en/es leftovers in rules
2, 3, 15, 16, 26, 31). pipeline.ts replaces the pseudocode: wordlist
normalization, skip-already-staged idempotency, batches of 20 against
gemini-3.6-flash with responseSchema, raw responses persisted per batch,
per-entry validation with rejection log, one transaction per word into
db/staging.db. Flags: --langs --pos --max-batches --delay-ms --dry-run.

Smoke run: 40/40 words staged (de+es, one batch each), 0 rejections.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 19:04:04 +02:00
..
db adding sql schema for sqlite db, adding sqlite db to gitignore 2026-07-24 00:33:28 +02:00
source-data renaming files and folders to convention so no mapping is needed 2026-07-24 14:12:12 +02:00
tests implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
.env.example feat: enrich stage foundation — provider config, env setup, schema fix 2026-05-03 22:44:14 +02:00
gemini.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
package.json removing not needed files 2026-07-21 17:45:10 +02:00
pipeline.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
prompt implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
promptTemplate.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
sourceLists.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
staging.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
tsconfig.json feat: add db schema, init, and vitest config 2026-05-03 17:56:29 +02:00
validate.ts implementing phase 3 pipeline: gemini structured output, validation, sqlite staging 2026-08-09 19:04:04 +02:00
vitest.config.ts feat: add stage 1 and 2 validation tests 2026-05-03 21:36:56 +02:00