Commit graph

229 commits

Author SHA1 Message Date
lila
7e0311683f updating documentation 2026-05-16 01:59:43 +02:00
lila
1ba57c7e9d adding tasks 2026-05-15 23:12:55 +02:00
lila
97b0f302d0 refactoring documentation 2026-05-15 23:09:54 +02:00
lila
04a581efe1 WIP: checkpoint before stage-3 sub-stage rewrite 2026-05-12 22:13:14 +02:00
lila
73fb12ac35 feat: enrich script working, redesigning to sub-stage architecture
- Enrich script functional with timeout, progress tracking, rejection mechanism
- Identified ordering issue: CEFR voting needs validated translations first
- Redesign: round1_gloss → round1_example → round1_translations → round1_cefr
- Update data-pipeline.md with new sub-stage design and roadmap
- Qwen3.5-4B confirmed working with thinking disabled
2026-05-07 13:09:43 +02:00
lila
7f10c35e03 docs: update roadmap — stage 3 enrich script written, llama.cpp next 2026-05-05 19:30:18 +02:00
lila
9642daf6dd feat: add stage 3 round 1 enrich script and wire into orchestrator 2026-05-05 19:28:38 +02:00
lila
76af2ab093 fix: update db import validation tests to account for reverse links
- Translation count test now adds reverse link count to expected total
- Non-English translations test now filters to kaikki source only
- Target language test now filters to kaikki source only — reverse links
  to English are valid and expected
2026-05-05 19:10:19 +02:00
lila
1c44ef989b feat: update pipeline orchestrator for Kaikki — wire up stages 1 and 2
- Replace checkOmwExists with checkExtractedFilesExist
- Wire up importKaikki and reverseLink as real stage implementations
- Track reverse link completion via sentinel row in run_status
- Update report to use resolved_entry_cefr and entry counts
- Stages 3 onwards remain as stubs
2026-05-05 19:04:28 +02:00
lila
6f9a42c707 feat: add stage 2 reverse link sync script 2026-05-05 18:57:55 +02:00
lila
b5a76ee178 docs: update roadmap — stage 1 in progress, sample extraction complete 2026-05-05 18:52:10 +02:00
lila
ba2635e3f7 feat: add stage 1 and db import validation tests for Kaikki schema 2026-05-05 18:51:11 +02:00
lila
0cc643e308 feat: update extractor for all 5 languages, update import for multi-language
- Extract.ts now processes all 5 language files, filters non-English
  entries by lang_code, skips translation extraction for non-English
  (no translations in source files)
- Import.ts now imports all 5 language output files, uses language
  field from ExtractedSense instead of hardcoding en
- Sample limit hardcoded to 500 entries per language for development
2026-05-05 18:46:32 +02:00
lila
209d52f54b feat: add Kaikki extraction and import scripts for stage 1
- Add stage-1-extract/scripts/extract.ts — streams Kaikki JSONL,
  filters to supported POS and languages, skips abbreviations and
  senses with no translations in supported languages
- Rewrite db/import.ts for Kaikki flat model — tracks sense_index
  offsets per headword+pos to handle duplicate JSONL entries
- Rewrite db/schema.sql for Kaikki model — entries, translations,
  LLM vote tables, resolved tables
- Add extract and db:import scripts to package.json
- Sample mode hardcoded to 500 entries for development
2026-05-05 18:11:53 +02:00
lila
963bff4eb8 feat: migrate production schema from OMW to Kaikki flat vocabulary model
- Replace terms/translations/term_glosses/term_examples with vocabulary_entries
  and entry_translations
- Remove decks, topics and related tables (deferred)
- Add cefr_level and difficulty to entry_translations for game query filtering
- Update termModel.ts for new schema — getDistractors now takes sourceLanguage
- Update gameService.ts and multiplayerGameService.ts for entryId rename
- Update all test fixtures from termId to entryId
- Generate and apply migration 0011
2026-05-05 17:39:25 +02:00
lila
38d8b85228 docs: rewrite data-pipeline.md for Kaikki migration 2026-05-05 17:14:48 +02:00
lila
87aeb072c5 feat: add pipeline orchestrator skeleton with startup checks, stage runners, shutdown handler, and report generation 2026-05-03 23:01:29 +02:00
lila
080fad1998 feat: enrich stage foundation — provider config, env setup, schema fix
- Remove foreign key on run_status.source_id to support sentinel rows
  for tracking one-time pipeline steps (compile_candidates, compile_votes,
  merge, compare)
- Add stage-3-enrich/config.ts with all provider configurations,
  ALL_PROVIDERS ordered local-first, and validateProviderKey() for
  startup key checks
- Add .env.example with required API keys for OpenRouter and Anthropic
- Add pipeline:run script to package.json using --env-file .env
- Add .env to root .gitignore coverage for data-pipeline/.env
2026-05-03 22:44:14 +02:00
lila
4d42fe4397 removing db from git tracking, adding it to gitignore, add db import validation tests 2026-05-03 22:16:43 +02:00
lila
f59399be02 feat: add db import script, fix duplicate translations in extract, add annotate script 2026-05-03 22:05:10 +02:00
lila
4a842140b9 feat: add stage 1 and 2 validation tests 2026-05-03 21:36:56 +02:00
lila
4fa3073412 feat: add db schema, init, and vitest config 2026-05-03 17:56:29 +02:00
lila
74cfc82bdd docs: finalise data-pipeline.md with tiebreak, pipeline.db, reports, sync 2026-05-03 17:21:02 +02:00
lila
6007fe1e38 docs: update data-pipeline.md and llm-setup.md to reflect sqlite architecture 2026-05-02 20:13:05 +02:00
lila
ccfd83d16c feat: email/password auth + email verification + password reset via Resend 2026-05-02 13:05:43 +02:00
lila
4ae2c568c6 fix: resolve ESLint config file ignores and project service coverage 2026-05-02 12:15:23 +02:00
lila
6ca6fc4e09 fix: correct dotenv path in packages/db/src/index.ts for compiled dist output 2026-05-02 11:23:10 +02:00
lila
e1c4fb5744 refactoring 2026-05-02 11:22:54 +02:00
lila
dc11213cb5 feat: replace login route with auth modal
- Add AuthModal to root layout driven by ?modal=auth search param
- Update multiplayer and play beforeLoad redirects to use modal
- Update NavAuth and Hero links to use modal
- Delete login route and NavLogin component
2026-04-30 19:46:45 +02:00
lila
32ee1edf80 feat: add AuthModal component with login, register and social tabs
- Add AuthModal with login/register tabs and social buttons
- Add forgot-password and reset-password routes
- Add Sonner toaster to root layout
- Add auth search schemas to @lila/shared
- Add ESLint overrides for TanStack Router generics
2026-04-30 19:38:43 +02:00
lila
6297dff399 feat: add email/password auth backend + forgot/reset password routes
- Configure Better Auth emailAndPassword plugin with Resend
- Add email verification and password reset email sending
- Create forgot-password and reset-password frontend routes
- Add auth schemas to @lila/shared
2026-04-30 18:30:20 +02:00
lila
35e54014b3 chore: add husky pre-commit and pre-push hooks
Some checks failed
Build and Deploy / quality (push) Failing after 1m53s
Build and Deploy / build-and-deploy (push) Failing after 1m3s
2026-04-30 01:15:14 +02:00
lila
4d64d50598 removing comment that tested pre-commit/pre-push hook 2026-04-30 01:14:53 +02:00
lila
1bfc0606c3 test: verify pre-commit hook 2026-04-30 01:13:01 +02:00
lila
8a121442a3 adding missing variables 2026-04-30 00:38:08 +02:00
lila
57d2190549 adding task to prompts 2026-04-30 00:36:28 +02:00
lila
fd9667c1fd updating documentation
Some checks failed
Build and Deploy / build-and-deploy (push) Failing after 1m27s
2026-04-28 17:26:01 +02:00
lila
98c59f33c5 formatting + adding issues 2026-04-28 16:39:36 +02:00
lila
648c5d2979 fix: improve error semantics, clarify answer key type 2026-04-28 16:07:19 +02:00
lila
6eaf282651 fix: sanitise Zod validation error messages in game controller 2026-04-28 15:51:57 +02:00
lila
c081e632cf fix: explicit store update in evaluateAnswer, remove mutation through reference 2026-04-28 15:47:53 +02:00
lila
a02d3b3335 fix: deduplicate distractors against each other, guard thin distractor pool 2026-04-28 15:44:29 +02:00
lila
a02f3b139d fix: deduplicate distractors, replace tautological test, add distractor failure test 2026-04-28 15:17:31 +02:00
lila
3d16ab0fff feat: guard against empty terms in createGameSession 2026-04-28 15:08:06 +02:00
lila
1e30f04e81 feat: add ownership check to evaluateAnswer, AuthenticatedRequest type 2026-04-28 14:39:13 +02:00
lila
fdeb769640 feat: add TTL to GameSessionStore, replay protection and session cleanup to evaluateAnswer 2026-04-28 14:03:15 +02:00
lila
54705943fa adding ticket for refactor: dependency injection for GameSessionStore via composition root 2026-04-28 13:50:56 +02:00
lila
a4a4bfff57 refactor: dependency injection for GameSessionStore via composition root 2026-04-28 13:48:50 +02:00
lila
4f59f3bc14 formatting 2026-04-28 13:18:18 +02:00
lila
2ff7d1759e refactor: extract shuffleArray to lib/utils, rename correctAnswers to terms 2026-04-28 13:17:24 +02:00