Nimbus VC v26.7.0 fails to start when Nimbus and Teku beacon nodes disagree on `PAYLOAD_DUE_BPS`

[suna4]

Summary

Nimbus VC v26.7.0 exits during startup when configured with two synced mainnet beacon nodes:

• Nimbus BN v26.7.0: PAYLOAD_DUE_BPS=7500
• Teku BN v26.7.1: PAYLOAD_DUE_BPS=5000

The log is:

WRN Received incompatible time configuration parameters node=http://<teku>:5052
    slot_duration_ms=12000 expected_slot_duration_ms=12000
ERR Validator client initialization failed
    err_msg="Could not obtain time configuration parameters"

Nimbus v26.7.0 uses consensus-specs v1.7.0-alpha.11, where this value is 7500. Teku v26.7.1 uses v1.7.0-alpha.12, where it was changed to 5000. This is a Gloas setting, and Gloas is not scheduled on mainnet.

Suspected cause

initTimeConfig() compares the TimeParams returned by all available beacon nodes. Any disagreement clears the result, so validator client initialization fails.

The runtime fallback check appears to compare slot duration but not these BPS fields, so startup and runtime handle the same configuration difference differently.

Expected behavior

A difference in an unscheduled fork setting should not prevent the validator client from using an otherwise compatible beacon node. Possible options might be:

• ignore settings belonging only to unscheduled forks; or
• reject only the disagreeing beacon node, using the same compatibility policy during startup and runtime.

It would also help if the warning logged the actual differing field and values instead of only the matching slot duration.

Reproduction

1. Run a synced Nimbus BN v26.7.0 and Teku BN v26.7.1 on mainnet.
2. Configure Nimbus VC v26.7.0 with both using --beacon-node.
3. Start the VC.

The VC exits with Could not obtain time configuration parameters. With a supervisor restart policy, this becomes a restart loop.

Note

AI assistance disclosure: I am a node operator, not a Nimbus developer. I used OpenAI Codex to help inspect the relevant source code and draft this report. I may not be able to answer implementation-level questions, and corrections are welcome. The reproduction and log excerpt are from my own nodes. I can provide the full keyless reproduction log, current /config/spec responses, or run suggested tests.

[tersec]

The issue here is that the next release of Nimbus, i.e. the next feasible opportunity to fix this, already uses alpha.12 (like Teku), so will automatically agree on PAYLOAD_DUE_BPS. So it's not clear that there's any specific further steps until then to be taken to fix it.

[suna4]

Thanks for clarifying. Sounds good — I’ll wait for the next release.

[b0a7]

I also ran into this issue when trying Lodestar v1.46.0-RC0.
Suggested improvement: don’t require cross-BN agreement on Gloas-only timing fields while GLOAS_FORK_EPOCH is far-future

[nflaig]

@tersec I think Nimbus VC should not assert these configs if GLOAS_FORK_EPOCH is not scheduled, ie. do not assert if GLOAS_FORK_EPOCH == FAR_FUTURE_EPOCH. otherwise I don't really see how you can prevent these issues from happening, see lodestar vc for reference

[b0a7]

one other possible gap to consider: the VC was throwing this timing params error on startup, but with a running VC I was able to swap one of the BNs from Lodestar v1.45 (PAYLOAD_DUE_BPS=7500) to v1.46-RC0 (PAYLOAD_DUE_BPS=5000) and the VC kept running without issue. So perhaps there is a missing BN param check upon reconnection (even though in this case we actually don't want these particular params to cause an error)

[tersec]

@tersec I think Nimbus VC should not assert these configs if GLOAS_FORK_EPOCH is not scheduled, ie. do not assert if GLOAS_FORK_EPOCH == FAR_FUTURE_EPOCH. otherwise I don't really see how you can prevent these issues from happening, see lodestar vc for reference

The question is how many more releases this is going to be the case, while GLOAS_FORK_EPOCH remains in fact unscheduled while the various parameters keep getting changed.

There's some cost in adding special cases which would almost immediately be removed and likely not turn out relevant (because at this point the various timing parameters have converged, and if they end up different by the time Gloas is scheduled, then it's in fact correct for the Nimbus VC to treat those as problematic).

So yeah, it would have been arguably better to avoid this issue to begin with. But it's also kind of too late to usefully add special-case-fixes for it.

[nflaig]

The question is how many more releases this is going to be the case, while GLOAS_FORK_EPOCH remains in fact unscheduled while the various parameters keep getting changed.

yeah not for gloas specifically, was thinking more of a general kind of machinery for this that would also work for heze, i*, etc.

for lodestar users, they can work around this by setting --params.PAYLOAD_DUE_BPS=7500 on the beaocn node side