Skip to content

Peer count continuously decays to zero in outbound-only enterprise environment while Geth and Erigon remain stable #363

Description

@atlasW

Environment

  • Client: reth-bsc
  • Version: v0.0.9-beta
  • Chain: BSC Mainnet
  • OS: Ubuntu 24.04 LTS &&。aws linux 2023
  • Deployment: Physical Server / VM
  • Network: Enterprise / Corporate Environment

Deployment Model

This node is deployed in a corporate environment where inbound connections are prohibited by security policy.

The deployment is intentionally outbound-only:

  • Outbound TCP connections allowed
  • Outbound UDP connections allowed
  • Inbound TCP connections blocked
  • Inbound UDP connections blocked

This restriction cannot be changed.

This deployment model is currently used successfully with other BSC clients.


Problem Description

The node starts normally and initially discovers/connects to several peers.

Typical startup peer count:

5-10 peers

However, the peer count continuously decreases over time and eventually reaches zero.

Once the peer count reaches zero, the node does not recover automatically and remains disconnected unless restarted.


Observed Behavior

Example:

Runtime Peer Count
0 min 7
30 min 5
60 min 3
90 min 1
120 min 0

The exact numbers vary between runs, but the pattern is always the same:

Initial peers
    ↓
Gradual peer loss
    ↓
No replacement peers discovered
    ↓
Eventually 0 peers

Comparison With Other Clients

The same server, firewall policy, network path, and outbound-only deployment model have been tested with:

  • BSC Geth
  • Erigon

Both clients maintain stable peer connectivity for extended periods of time.

Under identical network conditions:

Client Result
Geth Stable peers
Erigon Stable peers
reth-bsc Peer count continuously decays to 0

This suggests the issue may be specific to reth-bsc peer discovery, peer replacement, peer persistence, or reconnect behavior.


Current Configuration

/root/reth-bsc/reth-bsc node \
--datadir="/root/node" \
--chain=bsc \
--full \
--http \
--http.addr=0.0.0.0 \
--http.port=8545 \
--http.api=eth,debug,net,trace,web3,txpool \
--ws \
--ws.addr=0.0.0.0 \
--ws.port=8546 \
--ws.api=eth,net,web3,txpool \
--statedb.triedb \
--engine.parallel-sparse-trie \
--prune.accounthistory.distance=3000000 \
--prune.storagehistory.distance=3000000 \
--engine.cross-block-cache-size=16384 \
--rpc.max-blocking-io-requests=1024 \
--max-outbound-peers 100 \
--disable-nat \
--enable-discv5-discovery \
--peers-file=/root/node/peers.json \
--trusted-peers="enode://551c8009f1d5bbfb1d64983eeb4591e51ad488565b96cdde7e40a207cfd6c8efa5b5a7fa88ed4e71229c988979e4c720891287ddd7d00ba114408a3ceb972ccb@34.245.203.3:30311,enode://c637c90d6b9d1d0038788b163a749a7a86fed2e7d0d13e5dc920ab144bb432ed1e3e00b54c1a93cecba479037601ba9a5937a88fe0be949c651043473c0d1e5b@34.244.120.206:30311,enode://bac6a548c7884270d53c3694c93ea43fa87ac1c7219f9f25c9d57f6a2fec9d75441bc4bad1e81d78c049a1c4daf3b1404e2bbb5cd9bf60c0f3a723bbaea110bc@3.255.117.110:30311,enode://94e56c84a5a32e2ef744af500d0ddd769c317d3c3dd42d50f5ea95f5f3718a5f81bc5ce32a7a3ea127bc0f10d3f88f4526a67f5b06c1d85f9cdfc6eb46b2b375@3.255.231.219:30311,enode://5d54b9a5af87c3963cc619fe4ddd2ed7687e98363bfd1854f243b71a2225d33b9c9290e047d738e0c7795b4bc78073f0eb4d9f80f572764e970e23d02b3c2b1f@34.245.16.210:30311,enode://41d57b0f00d83016e1bb4eccff0f3034aa49345301b7be96c6bb23a0a852b9b87b9ed11827c188ad409019fb0e578917d722f318665f198340b8a15ae8beff36@34.245.72.231:30311,enode://1bb269476f62e99d17da561b1a6b0d0269b10afee029e1e9fdee9ac6a0e342ae562dfa8578d783109b80c0f100a19e03b057f37b2aff22d8a0aceb62020018fe@54.78.102.178:30311,enode://3c13113538f3ca7d898d99f9656e0939451558758fd9c9475cff29f020187a56e8140bd24bd57164b07c3d325fc53e1ef622f793851d2648ed93d9d5a7ce975c@34.254.238.155:30311,enode://d19fd92e4f061d82a92e32d377c568494edcc36883a02e9d527b69695b6ae9e857f1ace10399c2aee4f71f5885ca3fe6342af78c71ad43ec1ca890deb6aaf465@34.247.29.116:30311,enode://c014bbf48209cdf8ca6d3bf3ff5cf2fade45104283dcfc079df6c64e0f4b65e4afe28040fa1731a0732bd9cbb90786cf78f0174b5de7bd5b303088e80d8e6a83@54.74.101.143:30311"

None of the above prevented the gradual peer loss.


Expected Behavior

Even in an outbound-only environment:

  • Peer count may fluctuate
  • Disconnected peers should eventually be replaced
  • Discovery should continue finding new peers
  • Trusted peers should reconnect when possible

The node should not continuously decay to zero peers.


Actual Behavior

The node continuously loses peers over time.

Disconnected peers are not replaced fast enough (or at all), causing the network connection graph to shrink until zero peers remain.


Additional Notes

The issue does not appear to be caused solely by firewall or NAT restrictions because:

  • The same outbound-only deployment model works correctly with Geth
  • The same outbound-only deployment model works correctly with Erigon
  • Outbound TCP connectivity is available
  • Outbound UDP connectivity is available

This behavior currently blocks migration from Geth/Erigon to reth-bsc in enterprise environments where inbound connectivity is prohibited.


Questions

  1. Is outbound-only deployment officially supported by reth-bsc?

  2. Is peer discovery expected to function correctly when inbound connectivity is unavailable?

  3. Are trusted peers expected to reconnect automatically after disconnects?

  4. Is there any known issue related to peer discovery, peer replacement, peer persistence, or reconnect logic in outbound-only environments?

  5. Are there recommended settings for enterprise deployments where inbound connections are prohibited?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions