feat(indexer): race committee members on a substate read

[sdbondi]

Summary

A single-substate read asked committee members one at a time, so whenever the shuffled first pick was unreachable the read waited out the full CONNECT_TIMEOUT (10s) before trying the next. On a small committee that is a 1/n chance per read of a multi-second stall.

Reads now keep up to READ_RACE_WIDTH = 3 members in flight and settle on the first response that decides the read. Requests still in flight when the read settles are dropped.

The decision rules are unchanged and are now in one place (CommitteeReadTally):

• a proven Up/Down (or any Up/Down while proofs are not required) answers on the spot
• an unproven Up/Down is held as a fallback (highest version wins) while the rest of the committee is tried for a proof
• DoesNotExist is believed once more than f members agree, and takes precedence over errors from unreachable members

One behavioural nit the rewrite removes: the old inline DoesNotExist check tested num > f before incrementing, so it needed f + 2 agreeing responses on the way through the loop but only f + 1 at the end. Both now settle on f + 1, which is what the comment always said.

Tests

committee_read::tests cover arrival-order cases against a stub committee: an unresponsive member does not delay a proven answer (fails with width 1), at most width requests in flight, a failed member frees its slot, unproven held until a proof arrives, highest unproven version served when nobody can prove, f + 1 agreement for nonexistence, single nonexistence outvoted by a proven version, and agreement outranking errors.

Phase 3 item 1 from the indexer cache plan.

[sdbondi]

Reviewed at c48c30b. No blocking findings.

Verified sound

• race_committee draws from the shuffled().map(...) iterator lazily, so request_substate_from_vn is only entered for members that get a slot; FuturesUnordered::extend(take(width)) plus one push per settled response bounds in-flight to width. no_more_than_the_window_is_in_flight pins this.
• Dropping in-flight requests on settle is safe: RpcMultiPool::get_or_connect dials without holding the pool lock, and the only write in the verify path (trusted_root_store.record) is best-effort. Nothing here is a with_write_tx on a deadpool thread, so the cooperative-teardown trap from the follow-mode work does not apply.
• Description claim checked: the old loop tested num > f before incrementing, so a mid-loop settle needed the (f + 2)th DoesNotExist while the post-loop check needed f + 1. The tally settles on the (f + 1)th in both places. nonexistence_needs_more_than_f_agreeing_members covers it (size 4, f = 1, two responses).
• Precedence is unchanged: proven Up/Down on the spot, f + 1 DoesNotExist on the spot, unproven fallback (highest version) then last error, then unverified DoesNotExist only when every member answered without settling and none errored. Empty committee is still rejected before the tally is built.
• .boxed() on a Send future to settle rust-lang/rust#102211 matches the workaround used elsewhere; SubstateRequirementRef is Copy, so passing it both into the map closure and as describe is fine.
• Cargo.lock delta is only the two new tari_indexer_lib deps (futures, dev tokio). No unrelated files.

Non-blocking

• crates/indexer_lib/src/committee_read.rs:104 conclude logs "Could not get substate for shard {describe}" but describe is a substate requirement, not a shard. Carried over from the old code; worth fixing while it is being moved.
• A dial dropped mid-connect keeps going inside the networking layer but its session never reaches the pool, so the next read to that member redials. Costs one extra dial per dropped in-flight member; fine for width 3, just noting it is not free.

Nit

• crates/indexer_lib/src/committee_read.rs:22 MemberResponse is Result<(SubstateResult, bool), _>; a named verified field would read better at the observe site, but the tuple mirrors get_substate_from_vn so leaving it is reasonable.

CI is still pending; will follow up once it is green.

[sdbondi]

Fixed the "for shard" wording in the conclude log line. MemberResponse left as the tuple, mirroring get_substate_from_vn.

[sdbondi]

Re-checked at 660344d: the only delta from c48c30b is the "for shard" log wording in CommitteeReadTally::conclude, which was the non-blocking note. CI is green on this head (clippy, all three test shards, integration test). Nothing outstanding from my side; approve is unavailable on my own PR so another maintainer needs to sign off.