perf(bootstrap): R2 fast-tier read latency exceeds the mobile client budget from far regions — the incumbent-vs-R2 comparison is unmeasured (#5300)

[koala73]

Follow-up to the R2 origin rollout (#5325, #5326, #5328, #5331) and the plan at
docs/plans/2026-07-14-001-perf-bootstrap-r2-origin-plan.md (KTD7, R8). The U3a production-shadow
measurement has now collected a full steady-state day, and it gives clear — but forked — guidance.
Serving is still Redis; this is about whether the fast tier can cut over.

TL;DR

• R2 is reliable. Steady-state fallback rate 0.38% (2 unreadable + 6 timeout of 2,122 on 2026-07-15). The scary "16% unreadable" was a one-time 3-hour rollout window (2026-07-14 21:00 UTC, 407/407 = 100%, before creds/objects were wired), then 0% since. Not a steady-state problem.
• The fast tier fails the latency gate from far regions. Its p99 exceeds the fast-tier mobile client abort (1,200 ms) in 9 of 19 Vercel regions.
• But we're measuring the wrong thing. The gate compares R2 against an absolute 1,200 ms. The honest question is whether R2 is slower than the Redis read it replaces from the same region — and that number is not being logged, so the decision can't be made yet.

Evidence (Axiom wm_api_usage, event_type='bootstrap_r2_shadow', 2026-07-15 steady state)

R2 read latency, fast tier, by execution region vs the 1,200 ms mobile abort:

region	p50	p99
hkg1	643	2,331	❌
dub1	406	1,852	❌
cdg1	369	1,839	❌
bom1	768	1,732	❌
icn1	761	1,672	❌
sin1	682	1,536	❌
cpt1	656	1,419	❌
kix1	583	1,391	❌
syd1	594	1,338	❌
gru1	416	1,120	✅
…
fra1	433	957	✅
cle1	200	938	✅
sfo1	328	724	✅
iad1	188	408	✅

near-enam (iad1/cle1/yul1): p99 938 ms. far (Asia/EU/Africa/AU): p99 1,664 ms. The bucket is single-region (enam); the Vercel edge is global. This is a cross-region S3 hop, and its cost is geographic.

Per the plan's KTD7 selection rule, there is no single fast-tier T that keeps P(L>T) ≤ 0.2% while T ≤ C_happy(fast) ≈ 1,150 ms, because the fast-tier p99 (1,631 ms global) already exceeds C_happy. The plan correctly blocks its own fast-tier rollout — this was flagged in KTD7 as "a real possible outcome, not a formality."

The slow tier is fine: p99 2,111 ms against a 3,000 ms client budget.

The gap that blocks the decision

The 0.38% fallback rate is measured against U3a's uncensored 5,000 ms probe ceiling. A real serving timeout would be bounded by C_happy (~1,150 ms fast), which would convert the far-region slow tail into fallbacks — several percent in ~9 regions, failing R8's 1% worst-cohort gate.

BUT today, every fast-tier CDN miss from Singapore already reaches cross-region to Redis (us-east-1). If that Redis read is already ~1.5 s from Singapore, then far-region mobile users already abort on cold fast-tier misses today, and R2 is not a regression — just cheaper. We cannot tell, because:

• /api/bootstrap is not logged as a normal request event (no serving-duration telemetry).
• The shadow event's duration_ms is null (the probe fires in waitUntil, decoupled from the parent request).

So the decisive number — incumbent Redis-on-miss latency by region — does not exist.

Remediation

Step 0 (unblocks everything): capture the incumbent. Add the parent request's Redis-assembly duration to the shadow event (the plan's U3a specified this via Server-Timing; it was not wired). One field. Collect a steady-state day. Then the fork resolves:

Branch A — R2 ≈ Redis from far regions (plausible; both are us-east-1 cross-region hops):
the C_happy gate is comparing R2 to an absolute the incumbent also breaches → no regression → ship both tiers as built. Full savings, no new architecture.

Branch B — R2 materially slower than Redis:

1. Ship the slow tier now — it passes the gate on the current path (~1/3 of bootstrap egress, zero new work).
2. Fast tier → Cloudflare Worker + R2 binding (Deferred item Batch market and RSS fetching with progressive updates #1 in the plan). api.worldmonitor.app is already behind Cloudflare and R2 is a Cloudflare product; a Worker with a native R2 binding reads the object from the same POP the request hit — no cross-region hop. The publisher, tier-key extraction, envelope, health, and fallback logic all stay as built; only the ~30 lines of "how the edge fetches the object" move from Vercel-aws4fetch to Worker-binding. The fast tier is ~2/3 of the savings, so this is worth building if Branch B holds.

Not a blocker, but note

Once the incumbent field is added, the 3-hour startup burst (100% unreadable) should be excluded from any calibration window — U3a's own "reject incomplete windows" discipline. The steady-state numbers above already exclude it (window starts 2026-07-15T00:00Z).

Acceptance

• Shadow event carries the parent request's Redis-serving duration, per region.
• One steady-state day of incumbent-vs-R2 latency recorded in the U3a evidence artifact.
• Fork resolved with a one-line recommendation (Branch A ship-both, or Branch B slow-now + fast-via-Worker).
• If Branch B: slow tier cut over and its fallback rate confirmed < 1% worst-cohort before the fast-tier Worker work begins.

[koala73]

Storage-primitive comparison: Workers KV vs R2 (+ cache) for the bootstrap tiers

Prompted by a good observation while researching this: the R2 + Smart Tiered Cache + Cache Reserve stack we were assembling is, functionally, a hand-built version of Workers KV. KV is Cloudflare's managed primitive for exactly this shape — small, rarely-changed data read globally at high rates, served from the local POP. Worth deciding on evidence before we build more of the R2 cache stack. (Durable Objects were also considered and ruled out — single-location, serialized reads, a coordination primitive; it would re-centralize the global reads we're trying to distribute.)

The workload

• 2 keys (fast, slow), 0.77 MB / 1.62 MB
• Writes: fast every 2 min + slow every 10 min = 864/day (~26k/mo)
• Reads (origin misses behind CDN): ~41k/day (~1.25M/mo)
• Caller-invariant, public, rewritten on a fixed schedule, eventual-consistency-tolerant (CDN already serves ≤10-min-stale)

Cost — a tie, both effectively $0

	KV	R2
reads	$0.50/M (10M/mo incl.) → $0	$0.36/M Class B (10M/mo free) → $0
writes	$5.00/M (1M/mo incl.) → $0	$4.50/M Class A (1M/mo free) → $0
storage	2.4 MB ≪ 1 GB → $0	2.4 MB ≪ 10 GB → $0
egress	n/a	free
total	$0/mo	$0/mo

Even at 10× reads (~12.5M/mo, e.g. serving KV per-request with no CDN in front): KV $1.26/mo, R2 $0.91/mo. Cost is not a differentiator at any realistic scale here. (KV pricing, R2 pricing)

Where they actually differ

axis	KV	R2 + cache	winner
global-read latency, out of the box	push/pull replicated + edge-cached; hot reads ~0.5–10 ms from local POP, cold reads tiered (local→regional→central)	needs custom-domain cache + Smart Tiered Cache + Cache Reserve to get local reads — and we've already hit a Smart Tiered Cache detour (#5230)	KV
operational surface	1 primitive; seeder writes 2 keys, edge reads via 1 binding (no SigV4)	publisher + aws4fetch reader + Tiered Cache + Cache Reserve config	KV
consistency	eventual, ~60 s global propagation	strong per-object	tie (both fine; our cadence is 2–10 min)
size headroom	value cap 25 MiB (~16× the 1.62 MB slow tier)	object cap 5 TB	R2
already built	new write/read paths	publisher shipped (#5325)	R2 (but sunk cost ≠ argument, per this issue's own economic rule)

(How KV works, KV limits)

Read

• KV is the better-matched primitive for these payload sizes. It is the "globally-cached small blob" we were building R2 + Tiered Cache + Cache Reserve to approximate — without the assembly or the perf(cloudflare): Smart Tiered Cache detours ~54% of US api.worldmonitor.app traffic through JNB→cpt1 — +300-500ms per call, 13x function time (found via #5229 attribution) #5230 detour risk. Its 60 s propagation is a non-issue against a 2–10 min publish cadence.
• The one thing that keeps R2 in play is the 25 MiB cap, and this codebase has a demonstrated ability to bloat a bootstrap key (the 11.5 MB forecast:predictions-bootstrap:v1 incident, perf(forecasts): serve the dashboard the forecast LIST, not the dossiers (#5300) #5311 — that one fit under 25 MiB, but it shows the tail exists). Healthy tiers are 0.77/1.62 MB, so 16× headroom is comfortable — but if a tier could legitimately approach tens of MB, R2 is the safer store.

Recommended experiment — a three-way, one-window measurement

The publisher already assembles the exact tier payloads. To decide on evidence, not architecture aesthetics:

1. Have the publisher also write both tiers to KV (alongside R2 — a few lines).
2. In the same shadow path as perf(bootstrap): pair Redis latency with R2 shadow samples #5339, add a KV read probe emitting kv_duration_ms next to the existing r2_duration_ms and the new redis_duration_ms.
3. One steady-state 24 h window then gives a per-region, three-way comparison — Redis vs R2-via-Vercel vs KV — from real global traffic. Whichever wins the APAC p99 (hkg1/syd1/bom1/sin1, the cohorts that fail the gate today) wins the decision.

This folds cleanly into the existing #5339 instrumentation and resolves the fork in this issue with one measurement instead of committing to a storage primitive up front.

Sources: Choosing a storage product · KV pricing · KV limits · How KV works · R2 pricing