feat(indexer): log and count substate cache refusals, invalidations and evictions

[sdbondi]

Summary

A read refused because the shard's watermark was missing or stale came back as Ok(None), indistinguishable from an empty cache and invisible in the logs, so an indexer whose every read had started costing a committee round trip gave no hint why (raised twice on #2498).

• SqliteSubstateCache::read now logs a refusal at debug with the shard, how long ago it was last confirmed (or that it never was since startup) and the serve lag it exceeded. ShardWatermarks::age supplies the age.
• Three counters join the hit/miss pair, registered under substate_cache_:
  • invalidations — entries retired by a transition from the state sync stream
  • evictions — entries dropped to stay within substate_cache_max_entries
  • refused_stale — reads refused on the watermark gate
• substate_cache_invalidate and substate_cache_prune return the affected row counts to feed them; the metrics struct is shared between the cache and the sync worker.

Phase 3 items 2 and 3 from the indexer cache plan.

[sdbondi]

Read at cc8635b against development. No blocking findings.

Sound

• substate_cache_prune returns only the eviction delete's row count; the journal delete above it is not folded in, so evictions measures what its help text says (storage_sqlite/writer.rs:684-712).
• substate_cache_invalidate sums both the versioned and the nonexistence deletes, so invalidations counts every retired row rather than every invalidation record (storage_sqlite/writer.rs:641-660).
• read fetches the entry before consulting serve_watermark, so refused_stale only fires when the cache actually had something to serve. A plain miss stays a miss (substate_cache/mod.rs:213-257).
• The Option::inspect pattern for optional metrics matches cached_substate_manager.rs:212.
• SubstateCache::watermark still calls watermarks.get directly, which is right: that path decides a fetch watermark, it is not a refusal.

Non-blocking

• The description says the three counters "join the hit/miss pair, registered under substate_cache_". The hit/miss pair lives under substate_scanner_cache_hits/_misses (crates/indexer_lib/src/metrics.rs:14). The struct doc's "a miss rate that climbs alongside refused_stale" is correct advice, but someone looking for the miss rate under the same prefix won't find it. Worth a word in the description or the doc comment.
• serve_watermark takes the read lock twice: get then age (substate_cache/mod.rs:174-176). A confirm landing between them yields a log line like "last confirmed 0s ago, over the 60s serve lag". Harmless and rare, but a single ShardWatermarks call returning the (Option<StateVersion>, Option<Duration>) pair would make the message always consistent.

Nit

• One debug line per refused read means a shard that is unconfirmed at startup logs once per API request until its first sync round lands. Fine at debug; just noting it is not rate limited.

[sdbondi]

Addressed in the amended head: ShardWatermarks::confirmed returns the version and age under one read lock, replacing get + age in serve_watermark; the metrics doc now says the hit/miss pair lives under substate_scanner_. The per-read debug line stays unlimited.