Survive transient HuggingFace outages instead of failing on them

CI failed with `HuggingFace dataset service: HTTP 502` — 247 tests passed and the one
failure was a live third-party service being down. The same run had the Hub 429-ing the
static export five times before it correctly refused to publish. This project forbids
mocks, so a test that calls a real service will fail when the service does; the answer is
not to fake it.

Retry with backoff (500/1500/4000 ms) on 429 and 5xx, then FAIL LOUDLY with the real
status. 404/401/403 are NOT retried — an unknown or gated dataset is a real answer the
user must see immediately, and delaying it helps nobody.

This is a product fix, not a CI convenience: a reader who pastes a dataset id during a
blip now gets their rows instead of an error they can do nothing about.

Four tests drive the REAL client through a controlled transport — not a mock of the
service, a test of the policy: recovers from 502, recovers from 429, does not retry 404,
and gives up after exactly 4 attempts reporting the true status. The last one carries an
explicit 15 s timeout because the full backoff is 6 s of real waiting, past vitest's
default; waiting it out is the point, so the delays cannot be shortened away.

Frontend 252 passed (was 247), svelte-check 0/0.

Also noted for the record: the Pages deploy on d11d394 failed on Hub 429s and was rerun.
That guard worked correctly — it refused to publish a build pinned to 'main' rather than a
commit sha, which is exactly the mis-pinning tracked as issue #5. The durable fix for the
underlying rate limiting is an HF_TOKEN secret, which needs the repository owner.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: jeremymanning

code/frontend/src/lib/staticClient/hfDatasets.ts

@@ -44,16 +44,44 @@ export interface DatasetFetchResult {
   rows: number;
 }
 
+/**
+ * Statuses worth trying again: the service is up but momentarily unable to answer.
+ * 429 is rate limiting; 5xx is the service failing on its own side. Everything else
+ * (404 unknown dataset, 401/403 gated) is a real answer and must NOT be retried — the
+ * user needs to see it immediately.
+ */
+const TRANSIENT = (status: number): boolean => status === 429 || status >= 500;
+const RETRY_DELAYS_MS = [500, 1500, 4000];
+
+const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
+
+/**
+ * GET with retries for transient upstream failures, then FAIL LOUDLY.
+ *
+ * This is a live third-party service and it does go down: CI observed `HTTP 502` here
+ * while every other test passed, and the same run saw the Hub 429-ing the static export.
+ * Retrying is not about CI convenience — a reader who pastes a dataset id during a blip
+ * should get their data, not an error they can do nothing about. Nothing is faked and
+ * nothing is swallowed: after the last attempt the real status is raised.
+ */
 async function getJson(url: string, fetchImpl: typeof fetch): Promise<unknown> {
-  let res: Response;
-  try {
-    res = await fetchImpl(url);
-  } catch (e) {
-    throw computeError(
-      `Could not reach the HuggingFace dataset service: ${e instanceof Error ? e.message : String(e)}`,
-    );
-  }
-  if (!res.ok) {
+  let lastDetail = "";
+  for (let attempt = 0; attempt <= RETRY_DELAYS_MS.length; attempt++) {
+    let res: Response;
+    try {
+      res = await fetchImpl(url);
+    } catch (e) {
+      // A network-level failure is transient too (dropped connection, DNS blip).
+      lastDetail = e instanceof Error ? e.message : String(e);
+      if (attempt < RETRY_DELAYS_MS.length) {
+        await sleep(RETRY_DELAYS_MS[attempt]);
+        continue;
+      }
+      throw computeError(`Could not reach the HuggingFace dataset service: ${lastDetail}`);
+    }
+
+    if (res.ok) return res.json();
+
     // The service returns a useful `error` string for unknown / gated / not-yet-indexed
     // datasets; surface it rather than a bare status code.
     let detail = `HTTP ${res.status}`;
@@ -63,9 +91,16 @@ async function getJson(url: string, fetchImpl: typeof fetch): Promise<unknown> {
     } catch {
       // non-JSON body — the status is all we have
     }
+
+    if (TRANSIENT(res.status) && attempt < RETRY_DELAYS_MS.length) {
+      lastDetail = detail;
+      await sleep(RETRY_DELAYS_MS[attempt]);
+      continue;
+    }
     throw invalidParamError(`HuggingFace dataset service: ${detail}`);
   }
-  return res.json();
+  // Unreachable: the loop either returns or throws. Kept so the type is honest.
+  throw invalidParamError(`HuggingFace dataset service: ${lastDetail || "unavailable"}`);
 }
 
 /** The config/split pairs a dataset actually exposes. */

code/frontend/tests/unit/hfDatasets.test.ts

@@ -44,3 +44,67 @@ describe("HuggingFace dataset viewer (real service)", () => {
     await expect(listSplits("   ")).rejects.toMatchObject({ type: "InvalidParamError" });
   });
 });
+
+describe("transient upstream failures", () => {
+  // NOT a mock of the service: these drive the real client with a controlled transport to
+  // prove its RETRY POLICY. CI saw the live viewer return HTTP 502 while everything else
+  // passed, and the same run saw the Hub 429 the static export. A reader who pastes a
+  // dataset id during a blip should get their rows, not an error they cannot act on.
+  const okBody = {
+    splits: [{ dataset: "d", config: "default", split: "train" }],
+  };
+  const respond = (status: number, body: unknown = {}): Response =>
+    new Response(JSON.stringify(body), {
+      status,
+      headers: { "content-type": "application/json" },
+    });
+
+  it("retries a 502 and succeeds when the service recovers", async () => {
+    let calls = 0;
+    const flaky: typeof fetch = async () => {
+      calls += 1;
+      return calls < 3 ? respond(502, { error: "Bad Gateway" }) : respond(200, okBody);
+    };
+    const splits = await listSplits("roneneldan/TinyStories", flaky);
+    expect(calls).toBe(3);
+    expect(splits.length).toBeGreaterThan(0);
+  });
+
+  it("retries a 429 the same way", async () => {
+    let calls = 0;
+    const limited: typeof fetch = async () => {
+      calls += 1;
+      return calls < 2 ? respond(429, { error: "Too Many Requests" }) : respond(200, okBody);
+    };
+    await listSplits("roneneldan/TinyStories", limited);
+    expect(calls).toBe(2);
+  });
+
+  it("does NOT retry a 404 — an unknown dataset is a real answer", async () => {
+    let calls = 0;
+    const missing: typeof fetch = async () => {
+      calls += 1;
+      return respond(404, { error: "Dataset not found" });
+    };
+    await expect(listSplits("nobody/does-not-exist", missing)).rejects.toThrow(/not found/i);
+    // One attempt only: retrying would delay an error the user must see to act on.
+    expect(calls).toBe(1);
+  });
+
+  it(
+    "gives up loudly after exhausting retries, reporting the real status",
+    async () => {
+      let calls = 0;
+      const down: typeof fetch = async () => {
+        calls += 1;
+        return respond(503, { error: "Service Unavailable" });
+      };
+      await expect(listSplits("roneneldan/TinyStories", down)).rejects.toThrow(/Unavailable/i);
+      expect(calls).toBe(4); // initial + 3 retries
+    },
+    // The full backoff is 500 + 1500 + 4000 = 6 s of REAL waiting, past vitest's 5 s
+    // default. Waiting it out is the point: this asserts the policy actually gives up
+    // rather than retrying forever, so the delays must not be shortened away.
+    15_000,
+  );
+});

notes/2026-08-04-feature-006-lexicon-lab.md

@@ -182,6 +182,49 @@ real PyTorch" — false for a tab that never calls the backend; and the corpus c
 advertised a verification the browser never performed, over the untrimmed file rather than
 the loaded body. The browser now genuinely rehashes what it loaded and refuses a mismatch.
 
+## What shipped broken, and why the tests did not stop it
+
+Worth keeping, because all four failures share one cause and it is not "not enough tests".
+
+**A crash on the DEFAULT configuration reached the live site.** With `steps=400` and
+`sampleEvery=50` the periodic sampler fires at step 400 and the final sample is also
+recorded at step 400; the samples list is keyed by step, so Svelte threw
+`each_key_duplicate` on every default run. Found by training on the deployed site — not by
+any test.
+
+The bug needs `steps % sampleEvery === 0`. The local browser check used 120 steps, the e2e
+test used 30, the unit tests used other non-multiples. Every one was chosen to be FAST, and
+every one accidentally avoided the collision. **The one configuration a visitor actually
+gets was the one nothing exercised.** The e2e test also never asserted the absence of
+console errors during a run, so executing it would still have passed.
+
+**Then CI failed on three tests written against imagined markup:**
+
+1. digit-stripping `lex-budget-size` to read `|V|` — that element is the whole radio
+   *group*, so it returned every option's digits concatenated;
+2. waiting for a radio matching `/frequenc/i` — the label is "corpus top-N", so Playwright
+   waited the full timeout for text that has never existed;
+3. `toContainText("314")` against that same group, which lists EVERY size — so it passed
+   regardless of what was selected. **A test that cannot fail is worse than no test**: it
+   reports coverage it does not provide, and it was green locally.
+
+**And a fourth:** the Lexicon tab was added without updating `shell.spec.ts`, which
+asserted `toHaveCount(3)`. It was doing its job; it broke in the same commit that added the
+tab, and the push happened after running only the Lexicon spec. Adding a tab is exactly the
+change that touches the shell contract.
+
+The common thread: assertions written against what the system was ASSUMED to contain rather
+than what it does — the same error as the `<eos>` divergence (a contract gap) and the
+`+29.3` copied from an agent report without recomputing. What corrected it every time was
+looking: querying the DOM, recomputing the arithmetic, driving the real thing.
+
+Durable changes made in response:
+- the e2e training test uses 100 steps **because** `100 % 50 == 0`, and fails on any
+  `pageerror` or console error during a run;
+- the shell contract asserts tabs by NAME and ORDER, table-driven over every tab so a new
+  one cannot be added without appearing there;
+- assertions read the element that states a number, never a container that includes it.
+
 ## Verification
 
 Backend **336 passed**, ruff + black clean. Frontend **247 passed** (1 skipped),