flooring sense difficulty instead of rejecting the entry

Every rejection in the German run was the same rule: a translation ranked
below its own sense. The model tags a sense "medium" while correctly
tagging some translations "easy" — the translations are right and the
derived sense label is wrong, but the whole entry was discarded.

The prompt defines sense difficulty as the easiest translation difficulty
in that sense, so it is a derived value rather than an independent
judgement. validate.ts now recomputes it via applySenseDifficultyFloor.

The floor only ever lowers. Raising a sense to match its translations
would gate a concept out of levels it belongs in and collapse the
concept-vs-word distinction the two difficulty columns exist to express
(design-doc section 4).

- validate.ts: drop the cross-field rejection, add the floor; the valid
  result now carries "normalizations" so repairs are reported, not silent
- pipeline.ts: count and print normalizations per batch and in the summary
- replay.ts: new, re-validates responses/ with the current rules and no
  API calls; --write stages recovered entries, --langs and --verbose
- tests: six cases covering the floor, replacing the old rejection test

Replaying all 38 saved responses took the reject rate from 20 entries to
zero. staging.db now holds 746 words / 774 senses / 3,436 translations
with no sense ranked above its easiest translation.

Docs also record the API quota ceiling found today: the free tier allows
about 20 requests/day, not the 1,000 previously assumed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

data-pipeline/validate.ts

@@ -39,9 +39,14 @@ export type ValidationContext = {
 /**
  * "empty" is the contract's way of saying "not a valid word of this POS"
  * (senses: []) — the word is skipped, not rejected.
+ *
+ * A "valid" result carries the *normalized* entry, which may differ from the
+ * input — see `applySenseDifficultyFloor`. `normalizations` is a human-readable
+ * record of any repair applied, so the pipeline can report how often the model
+ * needed correcting instead of silently papering over it.
  */
 export type ValidationResult =
-  | { status: "valid"; entry: GeminiWordEntry }
+  | { status: "valid"; entry: GeminiWordEntry; normalizations: string[] }
   | { status: "empty"; headword: string }
   | { status: "invalid"; errors: string[] };
 
@@ -82,7 +87,6 @@ const checkStringArray = (
 const checkTranslation = (
   raw: unknown,
   label: string,
-  senseDifficulty: DifficultyLevel | null,
   ctx: ValidationContext,
   errors: string[],
 ): void => {
@@ -99,18 +103,12 @@ const checkTranslation = (
   if (!isNonEmptyString(raw["word"])) {
     errors.push(`${label}: word must be a non-empty string`);
   }
-  const difficulty = raw["difficulty"];
-  if (!isDifficulty(difficulty)) {
+  // A translation ranked below its sense is not an error — the sense is the
+  // derived value and gets floored to match. See applySenseDifficultyFloor.
+  if (!isDifficulty(raw["difficulty"])) {
     errors.push(
       `${label}: difficulty must be one of ${DIFFICULTY_LEVELS.join(", ")}`,
     );
-  } else if (
-    senseDifficulty !== null &&
-    difficultyRank(difficulty) < difficultyRank(senseDifficulty)
-  ) {
-    errors.push(
-      `${label}: difficulty "${difficulty}" is lower than sense difficulty "${senseDifficulty}"`,
-    );
   }
   if (
     typeof target === "string" &&
@@ -157,13 +155,7 @@ const checkSense = (
     return;
   }
   translations.forEach((translation, i) => {
-    checkTranslation(
-      translation,
-      `${label}.translations[${i}]`,
-      senseDifficulty,
-      ctx,
-      errors,
-    );
+    checkTranslation(translation, `${label}.translations[${i}]`, ctx, errors);
   });
 
   const wordsPerTarget = new Map<string, string[]>();
@@ -196,6 +188,45 @@ const checkSense = (
   }
 };
 
+/**
+ * Lowers a sense's difficulty to that of its easiest translation when the model
+ * tagged the sense higher.
+ *
+ * Why this is a repair and not a rejection: design-doc §5.1 filters sense
+ * difficulty as a *ceiling* and translation difficulty as an *exact* target, so
+ * a translation ranked below its own sense can never be served — the sense is
+ * gated out at exactly the level where that translation would be the answer.
+ * The row is dead data. The prompt already defines sense difficulty as the
+ * easiest translation difficulty in the sense, which makes it a derived value
+ * rather than an independent judgement, so it is recomputed here instead of
+ * discarding an otherwise-good entry.
+ *
+ * The floor only ever lowers. Raising a sense to match its translations would
+ * gate a concept out of levels it belongs in, and would collapse the
+ * concept-vs-word distinction that the two difficulty columns exist to express
+ * (design-doc §4).
+ */
+const applySenseDifficultyFloor = (
+  entry: GeminiWordEntry,
+  normalizations: string[],
+): GeminiWordEntry => ({
+  ...entry,
+  senses: entry.senses.map((sense) => {
+    const easiest = sense.translations.reduce<DifficultyLevel>(
+      (lowest, translation) =>
+        difficultyRank(translation.difficulty) < difficultyRank(lowest)
+          ? translation.difficulty
+          : lowest,
+      sense.difficulty,
+    );
+    if (easiest === sense.difficulty) return sense;
+    normalizations.push(
+      `senses[${sense.sense_index}]: difficulty "${sense.difficulty}" → "${easiest}" (floored to easiest translation)`,
+    );
+    return { ...sense, difficulty: easiest };
+  }),
+});
+
 export const validateEntry = (
   raw: unknown,
   ctx: ValidationContext,
@@ -235,5 +266,10 @@ export const validateEntry = (
   if (errors.length > 0) {
     return { status: "invalid", errors };
   }
-  return { status: "valid", entry: raw as GeminiWordEntry };
+  const normalizations: string[] = [];
+  return {
+    status: "valid",
+    entry: applySenseDifficultyFloor(raw as GeminiWordEntry, normalizations),
+    normalizations,
+  };
 };