Try in parallel commit-boost and MEV boost on Nimbus Hoodi fleet

[markoburcul]

We would like to try the commit-boost solution on the Hoodi fleet along with already used MEV boost over a period of time to compare its performance in terms of rewards obtained.

For this experiment to be more reliable I'm guessing we should go with the 2660 validator hosts and either use:

• one host with same EL but nimbus being stable and testing
• two different hosts(different EL) but nimbus being stable

Regarding the test duration, the longer the better and more reliable.

This issue is related to the one we have in Lido repo:

• https://github.com/Develp-GmbH/infra-lido/issues/401

@tersec I've added you to assignees because we need your opinion and approveal ultimately, so please do check this out.

Relevant links:

• https://commit-boost.github.io/commit-boost-client/overview/
• https://github.com/Commit-Boost/commit-boost-client
• Introducing commit-boost ObolNetwork/charon#3810
• https://x.com/Commit_Boost

[tersec]

Agree on trying mev-boost and commit-boost in parallel, for a long duration. They both work, they're both reasonable/useful software, et cetera.

If the question is between

• one host with same EL but nimbus being stable and testing
• two different hosts(different EL) but nimbus being stable

it's essentially a CL interaction, so largely, it's more useful to keep CL branch constant than EL constant for the purpose of a mev-boost vs commit-boost test. And since the relays aren't on the testnet setup anyway, but infra hosted elsewhere, that they're on different machines isn't critical.

[markoburcul]

I've deployed commit-boost on geth-05.ih-eu-mda1.nimbus.hoodi and replaced mev-boost with it. The metrics of commit-boost:

# HELP cb_pbs_beacon_node_status_code_total HTTP status code returned to beacon node
# TYPE cb_pbs_beacon_node_status_code_total counter
cb_pbs_beacon_node_status_code_total{endpoint="get_header",http_status_code="204"} 2
cb_pbs_beacon_node_status_code_total{endpoint="register_validator",http_status_code="200"} 180
cb_pbs_beacon_node_status_code_total{endpoint="status",http_status_code="200"} 584
# HELP cb_pbs_info Commit Boost info
# TYPE cb_pbs_info gauge
cb_pbs_info{commit="f51f5bd61831fde943057b29ffd6e26e7eb23765",network="Hoodi",version="0.8.0"} 1
# HELP cb_pbs_relay_latency HTTP latency by relay
# TYPE cb_pbs_relay_latency histogram
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.005"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.01"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.025"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.05"} 7
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.1"} 108
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.25"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="0.5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="1"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="2.5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="10"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="aestus",le="+Inf"} 180
cb_pbs_relay_latency_sum{endpoint="register_validator",relay_id="aestus"} 19.878807677000015
cb_pbs_relay_latency_count{endpoint="register_validator",relay_id="aestus"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.005"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.01"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.025"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.05"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.1"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.25"} 113
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="0.5"} 148
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="1"} 177
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="2.5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="10"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="blxrbdn",le="+Inf"} 180
cb_pbs_relay_latency_sum{endpoint="register_validator",relay_id="blxrbdn"} 54.669347859
cb_pbs_relay_latency_count{endpoint="register_validator",relay_id="blxrbdn"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.005"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.01"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.025"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.05"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.1"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.25"} 57
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="0.5"} 148
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="1"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="2.5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="10"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="flashbots",le="+Inf"} 180
cb_pbs_relay_latency_sum{endpoint="register_validator",relay_id="flashbots"} 61.82125052199998
cb_pbs_relay_latency_count{endpoint="register_validator",relay_id="flashbots"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.005"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.01"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.025"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.05"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.1"} 0
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.25"} 104
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="0.5"} 148
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="1"} 177
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="2.5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="5"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="10"} 180
cb_pbs_relay_latency_bucket{endpoint="register_validator",relay_id="titan",le="+Inf"} 180
cb_pbs_relay_latency_sum{endpoint="register_validator",relay_id="titan"} 56.563563579000025
cb_pbs_relay_latency_count{endpoint="register_validator",relay_id="titan"} 180
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.005"} 0
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.01"} 0
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.025"} 0
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.05"} 514
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.1"} 582
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.25"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="0.5"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="1"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="2.5"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="5"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="10"} 584
cb_pbs_relay_latency_bucket{endpoint="status",relay_id="aestus",le="+Inf"} 584
cb_pbs_relay_latency_sum{endpoint="status",relay_id="aestus"} 29.279003364999983
cb_pbs_relay_latency_count{endpoint="status",relay_id="aestus"} 584
# HELP cb_pbs_relay_status_code_total HTTP status code received by relay
# TYPE cb_pbs_relay_status_code_total counter
cb_pbs_relay_status_code_total{endpoint="register_validator",http_status_code="200",relay_id="aestus"} 180
cb_pbs_relay_status_code_total{endpoint="register_validator",http_status_code="200",relay_id="blxrbdn"} 180
cb_pbs_relay_status_code_total{endpoint="register_validator",http_status_code="200",relay_id="flashbots"} 180
cb_pbs_relay_status_code_total{endpoint="register_validator",http_status_code="200",relay_id="titan"} 180
cb_pbs_relay_status_code_total{endpoint="status",http_status_code="200",relay_id="aestus"} 584

We could create some tmp dashboard in grafana to track these values through time.

[markoburcul]

I've created the grafana dashboard.

[markoburcul]

The BN on geth-05 host has the validator client attached and enabled payload builder. I've struggled a bit to find where to check how are the proposals going on that BN, but I looked at Anton's finding in the Lido issue about missed proposals as well as the LIDO&ETH2 Discord chat and found his answer: I guess beacon_blocks_proposed_total without VC, beacon_blocks_sent_total with VC. We don't use this metric in our Nimbus dashboard on Infra Grafana instance.
Host comparison for now:

• The neth-05(mev boost) node had 50 proposals over the last 7 days.
• The geth-05(commit-boost) node had 63 proposals in the same period.

We are interested in the rewards these two BN's are obtaining through proposals so the next step would be to measure the actual rewards. I've tried exploring Dune Analytics but with no luck. Alexis pointed out he is creating configuration for eth validator watcher which is doing exactly that.

[markoburcul]

@leobago did the research to compare the two stable BN's proposal rewards and it turns out in the last 7 days none of the proposal rewards was MEV. We're investigating this currently..

[markoburcul]

Commit-Boost geth-05.ih-eu-mda1.nimbus.hoodi VC enabled, 2660 validators

• All the proposals missed mev reward.
• In the commit-boost logs the WARN : late in slot, skipping relay requests.. coincides with the timings of the proposals as seen on beaconcha.

I've analyzed all the late in slot logs and all of them go above 2 seconds and they are in the range 2019 ms (min) to 2176 ms (max). The commit boost has this setting late_in_slot_time_ms which is 2000ms by default and the explanation of it goes:

How late in milliseconds in the slot is "late". This impacts the `get_header` requests, by shortening timeouts for `get_header` calls to relays and make sure a header is returned within this deadline. If the request from the CL comes later in the slot, then fetching headers is skipped to force local building and miniminzing the risk of missed slots. See also the timing games section below

Since this node has Validator Client enabled, I've selected one proposal and extracted VC log just before it:

{"lvl":"INF","ts":"2025-11-03 08:24:01.951+00:00","msg":"Slot start","slot":1662070,"epoch":51939,"attestationIn":"4s","blockIn":"","validators":2660,"node_status":"synced","delay":"1s957ms472us880ns"}

Since the number of the validators on this node is 2660, the delay is always around 2s.

MEV-Boost neth-05.ih-eu-mda1.nimbus.hoodi VC disabled, 2660 validators

• we can observe that MEV rewards are obtained

When we look at MEV boost logs searching for the getHeaderand submitBlindedBlock the timings are much better(This analysis was done on the last 12 days.):

getHeader Requests (validator asking for best bid)
Sample size: 76 requests

Minimum: 110 ms
Maximum: 705 ms
Average: ~173 ms

submitBlindedBlock/getPayload Requests (validator submitting block)
Sample size: 72 requests

Minimum: 872 ms
Maximum: 3,087 ms
Average: ~1,290 ms

MEV-Boost neth-04.ih-eu-mda1.nimbus.hoodi VC enabled, 1500 validators

• we can observe that MEV rewards are obtained

getHeader Requests (validator asking for best bid)
Sample size: 48 requests

Minimum: 1,175 ms
Maximum: 1,328 ms
Average: ~1,263 ms

submitBlindedBlock/getPayload Requests (validator submitting block)
Sample size: 54 requests

Minimum: 2,094 ms
Maximum: 2,455 ms
Average: ~2,371 ms

Summary

• When using the VC client along with payload builder, the number of validators matters since the delay caused by VC will impact whether the payload builder will skip the relay request since it is too late into slot.

• We have observed that when having 1500 validators with VC enabled the getHeader request comes on average 1.2s into slot while with 2660 validators this is over 2s on average.

• An option would be to increase late_in_slot_time_ms for commit-boost to some reasonable value ~2.5s but that would push submitBlindedBlock request towards >3s which is a bit risky.