fix(taiko-client-rs): harden blob fetching and reconnect anchor resolution

[davidtaikocha]

Incident this fixes

On alethia-hoodi (2026-08-12 ~13:04 UTC), l2-node-reth-0's rust driver froze its L2 head permanently after a ~40s L1 RPC stall, while go-driver followers on the same L1 node rode it out. The chain of events:

1. The L1 EL stalled briefly → the event scanner died with terminal event scanner error.
2. The finalized reconnect-anchor lookup ran immediately, over the same dead RPC → failed → the driver fell back to the startup anchor captured 3 days earlier, forcing a ~22k-L1-block (~220k L2 blocks) re-derivation.
3. Scanner death closes preconf ingress until derivation catches back up — but re-derivation got stuck forever on a 2.5-day-old proposal whose blob could no longer be fetched:
   • beacon blob_sidecars exceeded the hardcoded 12s DEFAULT_HTTP_TIMEOUT;
   • the blobscan fallback returned HTTP 200 without an inline data field (blobscan serves dataStorageReferences + a separate /blobs/{hash}/data route), so decoding failed with Parse("error decoding response body").
4. Every preconf gossip block was then rejected with preconfirmation ingress loop is not ready → head frozen until manual pod restart.

Measurements behind the timeout choice

Against lighthouse v8.2.1 --semi-supernode (hoodi), blob_sidecars latency scales with blob count, not slot age (~1.5–5s per blob, no caching — repeat requests can be slower):

slot	approx. blobs	latency
near-head	~21	38.1s
6h old	~7	19.5s
1d old	~4	6–10s
2.7d old (the stuck one)	~19	36.6s / 46.4s

PeerDAS nodes reconstruct blobs from data columns per request, so a 12s timeout cannot succeed on busy slots at all. (The go client survives with its 60s DefaultRpcTimeout.) Meanwhile blobscan's /blobs/{hash}/data serves the same blob in ~2s.

Changes

1. Blob-server fallback understands blobscan responses (rpc/src/blob.rs): data is now optional in BlobServerResponse; when absent (or empty), the payload is fetched from /blobs/{hash}/data, accepting both a JSON-encoded string ("0x…", what blobscan serves) and a bare hex body. The computed-commitment/versioned-hash verification is unchanged, so a wrong payload is still rejected. Servers that inline data (Taiko blob storage service) behave exactly as before.

2. Dedicated, configurable blob fetch timeout (--blob.fetchTimeout, env BLOB_FETCH_TIMEOUT, default 120s): applied to the beacon client used for blob fetches and to the blob-server HTTP client. The global 12s DEFAULT_HTTP_TIMEOUT is intentionally left untouched for everything else (event-scanner reconnect, the whitelist runner's slot/epoch metadata beacon client) — those paths want fast failure detection.

3. Retry the finalized reconnect anchor before rewinding to the startup anchor (driver/src/sync/event.rs): up to 10 attempts with the existing scanner reconnect backoff (≈75s at the default 12s cap), which rides out the transient RPC outages that cause scanner interruptions in the first place. A fresh-chain "no finality yet" (Ok(None)) answer still short-circuits immediately — only transport errors are retried. This extends the resilience direction of feat(taiko-client-rs): harden event scanner against transient L1 stalls #21969 to the anchor-resolution path.

Tests

• 3 new tests in rpc::blob: blobscan-style metadata + /data fallback (JSON-string and raw-hex bodies), and rejection of a /data payload that doesn't match the requested versioned hash. The fallback test reproduces the exact production failure (it fails with Parse("error decoding response body") on the old code).
• 3 new paused-time tests in driver::sync::event: transient-failure retry, give-up after max attempts, and Ok(None) short-circuit (proved by queue-consumption order).
• cargo test -p rpc -p driver --lib: 14 + 129 passed.
• just clippy (both passes, -D warnings -D missing_docs -D clippy::missing_docs_in_private_items) and cargo +nightly fmt --check clean.

Notes for reviewers

• Draft: numbers above are from the live hoodi incident investigation; happy to split this into three PRs if that's preferred.
• The 120s default is sized for today's ~21-blob slots with contention margin; if BPO raises max blobs further, the serving side (full --supernode, or caching reconstructed blobs) needs attention regardless of any client timeout.

🤖 Generated with Claude Code