bug(session): ~2.9k/day anon sessions go dead — fresh-mint-then-401 dominates (~15% of recovery episodes end in a 15-min widget blackout); wm_session_dead (WORLDMONITOR-WG) quantifies it

[koala73]

Summary

The new wm_session_dead telemetry (#5245, deployed 2026-07-12 ~14:45 UTC) is the first honest counter of dead anonymous sessions, and it shows the problem is ~20× larger than the old per-endpoint 503 flood implied: ~120 episodes/hour, steady (~2.9k/day), 511 distinct users in the first ~4.7h. Each episode = anonymous API calls suppressed for 15 minutes (#5219 cooldown) = a dashboard of blank widgets.

Evidence (Sentry WORLDMONITOR-WG + Axiom, 2026-07-12)

• Hourly event counts since deploy: 128, 111, 120, 102, 119 — flat, not a spike; this is steady state.
• 609 events / 511 users in ~4.7h ⇒ ~1.2 events/user: a broad population, not a few pathological clients re-marking every cooldown expiry.
• Browser spread ∝ traffic mix (Chrome desktop 50%, Chrome Mobile, Edge, Firefox, Safari all present) — not a cookie-blocking-browser skew (and api.worldmonitor.app is same-site with www., so ITP/TCP shouldn't apply anyway).
• URL spread: all web dashboards (www + variant subdomains). No desktop app involvement.
• Mint endpoint is healthy: /api/wm-session over 07-10→07-12 = 53,085×200 vs 73×403 + 8×429 (~29 failures/day). Mint failure cannot explain ~2.9k dead episodes/day.
• Gateway 401 baseline: ~250–280k/day (11–12k/hr, flat through the day). At ~19k mints/day that's ≈13 endpoint-401s per recovery episode (parallel widget fan-out), i.e. ~19k recovery episodes/day of which ~2.9k (≈15%) end dead.

What this means

Since mint returns 200, the dominant dead path in wm-session.ts is fresh-mint-then-401: the retry with a just-minted cookie is still rejected (markWmSessionDead() at the retryResp.status === 401 branch), or the mint response's Set-Cookie never sticks. Candidate mechanisms, unverified:

1. Cookie never stored/sent — genuinely cookieless clients (strict privacy modes blocking all cookies, some in-app webviews/extensions). This is the bug(auth): cookieless anon browsers flood auth-required RPCs with 401s (~10k/3d on summarize alone) — client never detects a dead wms_ session #5207 population; its true size was never measured until now.
2. Verify-side rejection of a valid fresh cookie — if gateway wms_ verification reads state that can lag the mint write (e.g. Redis under the live perf(redis): Upstash bandwidth 30× (~300–500 GB/day vs 17 GB/day baseline) — #4499 removed the CDN shield from /api/bootstrap; every dashboard boot reads the full registry from Redis #5249 Upstash saturation, or cross-region execution drift per the perf(cloudflare): Smart Tiered Cache detours ~54% of US api.worldmonitor.app traffic through JNB→cpt1 — +300-500ms per call, 13x function time (found via #5229 attribution) #5230 cpt1 detour), a just-minted session can 401. Would affect all browsers proportionally — which matches the observed spread better than hypothesis 1 does.
3. Set-Cookie/req race — mint response and the immediate retry racing in the browser; would be browser-version-skewed, which we don't clearly see.

Ask

• Split the dead-marking telemetry so the two paths are distinguishable (mint-failed vs fresh-mint-401) — one extra tag on the existing captureMessage (reason: mint_failed | retry_401) turns this from hypothesis into measurement.
• Correlate episode timestamps against Upstash saturation windows (perf(redis): Upstash bandwidth 30× (~300–500 GB/day vs 17 GB/day baseline) — #4499 removed the CDN shield from /api/bootstrap; every dashboard boot reads the full registry from Redis #5249) once the CDN shield (PR perf(bootstrap): restore CDN cache shield for tier reads (#5249) #5250) lands — if the rate drops materially, it was (2).
• If (1) dominates: decide whether cookieless anon browsing deserves a header-token fallback instead of a 15-min blackout.

Related

• bug(auth): cookieless anon browsers flood auth-required RPCs with 401s (~10k/3d on summarize alone) — client never detects a dead wms_ session #5207 (client never detected dead sessions — closed by fix(auth): suppress dead anonymous session retry storms #5219's cooldown)
• fix(auth): suppress dead anonymous session retry storms #5219 (15-min dead-session cooldown + synthetic 503s)
• fix(observability): client-synthesized session-degraded 503s flood Sentry as per-endpoint "API 503" origin errors (#5219 side effect) #5245 (one-warning-per-episode telemetry; the counter this issue is based on)
• perf(redis): Upstash bandwidth 30× (~300–500 GB/day vs 17 GB/day baseline) — #4499 removed the CDN shield from /api/bootstrap; every dashboard boot reads the full registry from Redis #5249 / perf(cloudflare): Smart Tiered Cache detours ~54% of US api.worldmonitor.app traffic through JNB→cpt1 — +300-500ms per call, 13x function time (found via #5229 attribution) #5230 (candidate server-side mechanisms)