Skip to content

Commit

Permalink
Networking options for Ouroboros Genesis
Browse files Browse the repository at this point in the history
  • Loading branch information
@crocodile-dentist
crocodile-dentist committed Feb 19, 2025
1 parent 45b3135 commit 5621ed1
Showing 1 changed file with 138 additions and 52 deletions.
190 changes: 138 additions & 52 deletions docs/get-started/cardano-node/topology.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,47 @@ sidebar_label: Topology
sidebar_position: 5
description: OVerview and configiuration of P2P topology.
keywords: [Get-started, run the node, installation, networking, p2p, peer to peer, cardano-node, cardano node]
---
---

The P2P topology file specifies how to obtain the _root peers_.

* The term _local roots_ refers to a group of peer nodes with which a node will aim to maintain a specific number of active, or 'hot' connections.
These hot connections are those that play an active role in the consensus algorithm.
* The term _local roots_ refers to a group of peer nodes with which a node will aim to maintain a specific number of active or 'hot' connections.
These hot connections play an active role in the consensus algorithm.
Conversely, 'warm' connections refer to those not yet actively participating in the consensus algorithm.
Local roots should comprise local relays or a local block-producing node, and any other peers that the node needs to maintain a connection with.
Local roots should comprise local relays, a local block-producing node, and any other peers with which the node needs to maintain a connection.
These connections are typically kept private.


* _public roots_: publicly known nodes (e.g. IOG relays, or ledger nodes).
* _bootstrap peers_: trusted set of peers, only used for syncing. With
`cardano-node-10.2` one can use _Genesis_ instead.


* _public roots_: publicly known nodes (e.g. IOG relays or ledger nodes).
They are either read from the configuration file directly or from the chain.
The configured ones will be used to pass a recent snapshot of peers needed before the node caches up with the recent enough chain to construct root peers by itself.

The node does not guarantee a connection with every public root, unlike local ones.
Unlike local ones, the node does not guarantee a connection with every public root.
However, by being present in the set, it gets an opportunity to establish an outbound connection with that peer.

A minimal version of this file looks like this:

```json
{
"localRoots": [
{ "accessPoints": [
{
"address": "x.x.x.x",
"port": 3001
}
],
"advertise": false,
"valency": 1,
"warmValency": 2,
"trustable": true,
"diffusionMode": "InitiatorAndResponder"
}
],
"useLedgerAfterSlot": 128908821,
"bootstrapPeers": [
{
"address": "backbone.cardano.iog.io",
Expand All @@ -41,19 +60,6 @@ A minimal version of this file looks like this:
"port": 3001
}
],
"localRoots": [
{ "accessPoints": [
{
"address": "x.x.x.x",
"port": 3001
}
],
"advertise": false,
"valency": 1,
"warmValency": 2,
"trustable": true
}
],
"publicRoots": [
{ "accessPoints": [
{
Expand All @@ -63,46 +69,57 @@ A minimal version of this file looks like this:
],
"advertise": false
}
],
"useLedgerAfterSlot": 128908821,
]
"peerSnapshotFile": "path/to/big-ledger-peer-snapshot.json"
}
```

* The main difference between `LocalRoots` and `PublicRoots` is the ability to specify various groups with varying valencies in the former.
This can be valuable for informing your node about different targets within a group to achieve.
`LocalRoots` is designed for peers that the node should always keep as hot or warm, such as its own block producer.
On the other hand, `PublicRoots` serves as a source of fallback peers, to be used if we are before the configured `useLedgerAfterSlot` slot.
### Local Roots

* `LocalRoots` are designed for peers that the node **should always keep as hot or warm**, such as its own block producer or fullfill aranged agreements with other SPOs.

This means that your node will initiate contact with the node at IP `x.x.x.x` on `port 3001` and resolve the DNS domain `y.y.y.y` (provided it exists).
It will then try to connect with at least one of the resolved IPs.

* This means that your node will initiate contact with the node at IP `x.x.x.x` on `port 3001` and resolve the DNS domain `y.y.y.y` (provided it exists).
It will then make efforts to establish a connection with at least one of the resolved IPs.

* `hotValency` tells the node the number of connections it should attempt to select from the specified group.
When a DNS address is provided, valency determines the count of resolved IP addresses for which the node should maintain an active (hot) connection.
Note: one can also use the deprecated now `valency` field for `hotValency`.


- `warmValency` is an optional field, similar to `hotValency`, that informs the node about the number of peers it should maintain as warm.
* `warmValency` is an optional field, similar to `hotValency`, that informs the node about the number of peers it should maintain as warm.
This field is optional and defaults to the value set in the `valency` or `hotValency` field.
If a value is specified for `warmValency`, it should be greater than or equal to the one defined in `hotValency`; otherwise, or `hotValency` will be adjusted to match this value.
We recommend users set the `warmValency` value to `hotValency + 1` to ensure at least one backup peer is available to be promoted to a hot connection in case of unexpected events.
If a value is specified for `warmValency`, it should be greater than or equal to the one defined in `hotValency`; otherwise, `hotValency` will be adjusted to match this value.
We recommend that users set the `warmValency` value to `hotValency + 1` to ensure that at least one backup peer can be promoted to a hot connection in case of unexpected events.
Check [this issue](https://github.com/intersectmbo/ouroboros-network/issues/4565) for more context on this `WarmValency` option.


* Local root groups shall be non-overlapping.
* The `diffusionMode` is an optional field available since `cardano-node-10.2`.
It can either be `"InitiatorAndResponder"` (the default value) or `"InitiatorOnly"` (similar to `DiffusionMode` in the configuration file).
If `"InitiatorOnly"` is set, then all local roots in this group will negotiate initiator-only diffusion mode, e.g. the TCP connection will be used as a unidirectional connection.

The topology setting overwrites `DiffusionMode` from the configuration file for given local root peers.
It is meant to overwrite the diffusion mode when a node is running in `InitiatorAndResponder` mode (the default).
The other way is also possible, but note that when the option in the configuration file is set to `InitiatorOnly`, the node will not run the accept loop.


* The advertise parameter instructs a node about the acceptability of sharing its address through *peer sharing* (which we'll explain in more detail in a subsequent section).
In essence, if a node has activated peer sharing, it can receive requests from other nodes seeking peers.
* The `advertise` parameter instructs a node about the acceptability of sharing addresses through *peer sharing* (which we'll explain in more detail in a subsequent section).
If a node has activated peer sharing, it can receive requests from other nodes seeking peers.
However, it will only disclose those peers for which it has both local and remote permissions.

Local permission corresponds to the value of the advertise parameter.
On the other hand, 'remote permission' is tied to the `PeerSharing` value associated with the remote address, which is ascertained after the initial handshake between nodes.


* Local root groups shall be non-overlapping.


* Local roots should not be greater than the `TargetNumberOfKnownPeers`.
If they are, they will get clamped to the limit.

- For `trustablePeers` and `bootstrapPeers` read the next section.

* Read the next section for `trustablePeers` and `bootstrapPeers`.


Your __block-producing__ node must __ONLY__ talk to your __relay nodes__, and the relay node should talk to other relay nodes in the network.

Expand All @@ -115,47 +132,116 @@ Additionally, the SIGHUP signal will prompt the system to re-read the block forg
If this process fails, block forging will be disabled.
To re-enable block forging, ensure that the necessary files are present.

You can disable ledger peers by setting the `useLedgerAfterSlot` to a negative value.


### Ledger Peers / Public Roots & Big Ledger Peers

The option `useLedgerAfterSlot` configures from which slot the node should start to use peers registered on the ledger. Before the given slot, the node will use `PublicRoots`, unless `bootstrapPeers` are given (see below).
If a negative value is specified a node will not use ledger peers.
Ledger peers should be disabled for your block producing node.

Ledger peers are drawn from the ledger based on stake distribution.
Big ledger peers is a similar notion.
It is a subset of ledger peers which contains 90% of them with the largest stake.

`PublicRoots` serve as a source of fallback peers, which are used if we are before the configured `useLedgerAfterSlot` slot (please consider using `bootstrapPeers` instead or Genesis).

### Genesis lite a.k.a Bootstrap Peers

Bootstrap Peers is a pre-Genesis feature that means to provide a way for a node to connect to a trustable set of peers when the its chain falls behind.
Bootstrap Peers is a pre-Genesis feature that allows a node to connect to a trustable set of peers when its chain falls behind.

Bootstrap peers can be disabled by setting `bootstrapPeers: null`.
They are enabled by providing a list of addresses.
By default bootstrap peers are disabled.
By default, bootstrap peers are disabled.

Trustable peers are composed by the bootstrap peers (see `bootstrapPeers` option in the example topology file above) and the trustable local root peers (see `trustable` option in the example topology file above).
By default local root peers are not trustable.
Trustable peers comprise the bootstrap peers (see `bootstrapPeers` option in the example topology file above) and the trustable local root peers (see `trustable` option in the example topology file above).
By default, local root peers are not trustable.

In order for the node to be able to start and make progress, when bootstrap peers are enabled, the user _must_ provide a trustable source of peers via topology file.
For the node to be able to start and make progress when bootstrap peers are enabled, the user _must_ provide a trustable source of peers via a topology file.
This means that the node will only start if either the bootstrap peers list is non-empty or there's a local root group that is trustable.
Failing to configure the node with trustable peer sources will make it so that the node crashes with an exception.
*_Please note_* that if the only source of trustable peers is a DNS name, the node might not be able to make progress once in fallback state, if DNS is not providing any addresses.
Failing to configure the node with trustable peer sources will cause the node to crash with an exception.
*_Please note_* that if the only source of trustable peers is a DNS name, the node might be unable to make progress once in the fallback state if DNS is not providing any addresses.

With bootstrap peers enabled the node will trace:
With bootstrap peers enabled, the node will trace the following:

- `TraceLedgerStateJudgmentChanged {TooOld,YoungEnoug}`: If it has changed to any of these states.
- `TraceLedgerStateJudgmentChanged {TooOld,YoungEnough}`: If it has changed to any of these states.

- `TooOld` state means that the information the node is getting from its peers is outdated and behind at least 20 min.
This means there's something wrong and that the node should only connect to trusted peers (trusted peers are bootstrap peers and trustable local root peers) in order to sync.
This means there's something wrong, and the node should only connect to trusted peers (trusted peers are bootstrap peers and trustable local root peers) to sync.

- `YoungEnough` state means that the node is caught up and that it can now connect to non-trusted peers.
- The `YoungEnough` state means the node is caught up and connects to non-trusted peers.

- `TraceOnlyBootstrap`: Once the node transitions to `TooOld` the node will disconnect from all non-trusted peers and reconnect to only trusted ones in order to sync from trusted sources only.
- `TraceOnlyBootstrap`: Once the node transitions to `TooOld,` the node will disconnect from all non-trusted peers and reconnect only to trusted ones in order to sync from trusted sources.
This tracing message means that the node has successfully purged all non-trusted connections and is only going to connect to trusted peers.

### [Ouroboros Genesis](https://iohk.io/en/blog/posts/2024/05/08/ouroboros-genesis-design-update/)

Ouroboros Genesis is the upcoming mechanism of trustless syncing in P2P environment which is expected to supercede bootstrap peers described in the previous section. This feature is included starting with `cardano-node 10.2`, and at the time of this writing it is disabled by default - refer to config.json file section below on instructions how to enable this by toggling a feature flag. Once enabled, this mode is incompatibile with bootstrap peers, and will disable the latter by overriding the configuration, and emit a trace of such occurence to inform the operator to update topology file. From the perspective of the topology file a new entry must be added:

`"peerSnapshotFile": "path/to/snapshot.json"`

The file contains a snapshot of so-called big ledger peers which are the largest peers registered on the ledger which cumulatively hold 90% of stake at some arbitrary slot number. By virtue of the size of their stake in the system, they are postulated to be a proxy for honest ledger state. When syncing in this mode, these peers are sampled and connected with to bootstrap the process. Such a snapshot file can be created manually apriori with cardano-cli from a synced node, and may be distributed with a node release in the future. For reference,

### Configuring the node to use P2P

You can enable P2P from the configuration file, the field `EnableP2P` can be set to either `false` or `true`. When setting it to `true`, you will also need to configure the target number of _active_, _established_ and _known_ peers, together with the target number of _root_ peers, for example:
You can enable P2P from the configuration file; the field `EnableP2P` can be set to either `false` or `true`. When setting it to `true`, you will also need to configure the target number of _active_, _established_ and _known_ peers, together with the target number of _root_ peers. These values control the number of outbound connections the node will try to maintain in the appropriate mode. This is important for the node as blocks are downloadeded strictly from these upstream peers. Importantly, blocks that the node is itself serving on demand or the transactions it is requesting are determined by its downstream clients on which these values have no bearing on. The default configuration values are:

```json
{
...
"EnableP2P": true,
"TargetNumberOfActivePeers": 20,

"TargetNumberOfRootPeers": 60,

"TargetNumberOfActivePeers": 15,
"TargetNumberOfEstablishedPeers": 40,
"TargetNumberOfKnownPeers": 100,
"TargetNumberOfRootPeers": 10,
"TargetNumberOfKnownPeers": 85,

"TargetNumberOfActiveBigLedgerPeers": 5,
"TargetNumberOfEstablishedBigLedgerPeers": 10,
"TargetNumberOfKnownBigLedgerPeers": 15,
}
```
```

Collectively, these are known as the deadline targets. Prior to Ouroboros Genesis, this was the only set of static targets available to the node. When Genesis is enabled, these deadline targets
are in effect when the node deems itself caught up to its peers.

* `TargetNumberOfActivePeers` - the target for active ledger peers (aka hot peers); includes: local roots, ledger peers / public roots, peers from peer-sharing; excludes: big ledger peers. This ordinarily should be least the number of local root peers that are specified as hot in the topology file, otherwise the number of these connections will be clamped below the expected number. However, it is not strictly a misconfiguration and the node will run in such configuration.
* `TargetNumberOfEstablishedPeers` - the target for established connections (the sum of warm & hot peers); includes: local roots roots, ledger peers / public roots, peers from peer-sharing; excludes big ledger peers. Same note as for active peers above applied here as well.
* `TargetNumberOfKnownPeers` - the target for known peers (the sum of cold, warm & hot); includes: local roots, ledger peers / public roots, peers from peer-sharing; excludes big ledger peers.

* `TargetNumberOfActiveBigLedgerPeers` - the target for active big ledger peers (aka hot big ledger peers).
* `TargetNumberOfEstablishedBigLedgerPeers` - the target for established connections with big ledger peers (the sum of warm & hot big ledger peers).
* `TargetNumberOfKnownBigLedgerPeers` - the target for known big ledger peers (the sum of cold, warm & hot big ledger peers).

* `TargetNumberOfRootPeers` - a lower limit for known local roots, ledger peers / public root peers. Anything beyoned this target will be filled by peers from peer sharing, ledger / public root peers.

Note, when using bootstrap-peers, feature the targets have to be large enough to accomodate all bootstrap peers.

**Custom targets must satisfy the property that # of known >= # of established >= # of active >= 0 otherwise the node will fail to start as it is a serious misconfiguration.**

#### Genesis network specific configuration

Ouroboros Genesis is disabled by default at the time of this writing. To enable it, the node configuration file must contain
* `"ConsensusMode": "GenesisMode"`
or the option `--ConsensusMode GenesisMode` must be given when starting up the node process.

Detailed configuration settings for Genesis are beyond the scope of this article; however, below we document relevant networking options.
These options are available since `cardano-node-10.2` and by default their values are as shown below:

```json
{
"SyncTargetNumberOfActivePeers": 0,
"SyncTargetNumberOfActiveBigLedgerPeers": 30,
"SyncTargetNumberOfEstablishedBigLedgerPeers": 50,
"SyncTargetNumberOfKnownBigLedgerPeers": 100,
"MinBigLedgerPeersForTrustedState": 5
}
```

Collectively, these are known as the sync targets and they are in effect automatically when the consensus layer detects that the node is sufficiently behind its upstream peers. The node then proceeds
to download blocks from some of the active big ledger peers. Optionally, the `SyncTargetNumberOfActivePeers` can be set such that outbound connections are also
opened up to local root peers, if defined, as well as other public relays or nodes we learn about via peer sharing, if enabled. Care must be taken to ensure
that the targets specified satisfy the conditions given in the prior section, or the node will fail to start with an appropriate error message. Once sufficiently many blocks have been adopted and the node deems itself
caught up again, the number of connections will revert to the default ones as described in the previous section. If at any time during the syncing process the number of hot connections to big ledger peers drops below
`MinBigLedgerPeersForTrustedState` value, the node will pause and await until sufficiently many active outbound connections are online.

0 comments on commit 5621ed1

Please sign in to comment.