You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/architecture.md
+15-15
Original file line number
Diff line number
Diff line change
@@ -9,17 +9,17 @@ subgraph "Consensus Node"
9
9
engine[Engine API Client]
10
10
BAPI[Beacon API]
11
11
TICK[Slot processor]
12
-
blk_db[Block DB]
13
-
BS_db[Beacon State DB]
12
+
blk_db[Block DB]
13
+
BS_db[Beacon State DB]
14
14
brod[Broadway]
15
-
FCTree[Fork choice store - Genserver]
15
+
FCTree[Fork choice store - Genserver]
16
16
BAPI -->|Beacon state queries| BS_db
17
-
brod -->|Save blocks| blk_db
18
-
brod -->|Blocks and attestations| FCTree
19
-
TICK -->|New ticks| FCTree
17
+
brod -->|Save blocks| blk_db
18
+
brod -->|Blocks and attestations| FCTree
19
+
TICK -->|New ticks| FCTree
20
20
BAPI --> engine
21
-
BAPI --> |head/slot requests| FCTree
22
-
brod --> |Save new states|BS_db
21
+
BAPI --> |head/slot requests| FCTree
22
+
brod --> |Save new states|BS_db
23
23
end
24
24
GOS[Gossip Protocols]
25
25
exec[Execution Client]
@@ -31,20 +31,20 @@ VALIDATOR --> BAPI
31
31
32
32
## Networking
33
33
34
-
The main entry for new events is the gossip protocol, which is the way in which our consensus node communicates with other consensus nodes. This includes:
34
+
The main entry for new events is the gossip protocol, which is how our consensus node communicates with other consensus nodes. This includes:
35
35
36
36
1. Discovery: our node has a series of known `bootnodes` hardcoded. We request a list of the nodes they know about and add them to our list. We save them locally and now can use those too to request new nodes.
37
37
2. Message propagation. When a proposer sends a new block, or validators attest for a new block, they send those to other known nodes. Those, in turn, propagate the messages sent to other nodes. This process is repeated until, ideally, the whole network receives the messages.
38
38
39
-
We use the `go-libp2p` library for the networking primitives, which is an implementation of the `libp2p`networking stack.
39
+
We use the `go-libp2p` library for the networking primitives, which is an implementation of the `libp2p`networking stack.
40
40
41
-
We use ports to communicate with a go application and broadway to process notifications.
41
+
We use ports to communicate with a go application and Broadway to process notifications.
42
42
43
-
**TO DO**: We need to document the ports architecture.
43
+
**TO DO**: We need to document the port's architecture.
44
44
45
45
## Gossipsub
46
46
47
-
One of the main communication protocols is gossipsub. This allows us to tell peers which topics we're interested in and receive events for them. The main external events we react to are blocks and attestations.
47
+
One of the main communication protocols is GossipSub. This allows us to tell peers which topics we're interested in and receive events for them. The main external events we react to are blocks and attestations.
48
48
49
49
### Receiving an attestation
50
50
@@ -65,7 +65,7 @@ sequenceDiagram
65
65
66
66
When receiving an attestation, it's processed by the [on_attestation](https://eth2book.info/capella/annotated-spec/#on_attestation) callback. We just validate it and send it to the fork choice store to update its weights and target checkpoints. The attestation is only processed if this attestation is the latest message by that validator. If there's a newer one, it should be discarded.
67
67
68
-
The most relevant piece of the spec here is the [get_weight](https://eth2book.info/capella/annotated-spec/#get_weight) function, which is the core of the fork-choice algorithm. In the specs, this function is called on demand, when calling [get_head](https://eth2book.info/capella/annotated-spec/#get_head), works with the store's values and recalculates them each time. In our case, we cache the weights and the head root each time we add a block or attestation, so we don't need to do the same calculations again.
68
+
The most relevant piece of the spec here is the [get_weight](https://eth2book.info/capella/annotated-spec/#get_weight) function, which is the core of the fork-choice algorithm. In the specs, this function is called on demand, when calling [get_head](https://eth2book.info/capella/annotated-spec/#get_head), works with the store's values, and recalculates them each time. In our case, we cache the weights and the head root each time we add a block or attestation, so we don't need to do the same calculations again.
69
69
70
70
**To do**: we should probably save the latest messages in persistent storage as well so that if the node crashes we can recover the tree weights.
71
71
@@ -103,7 +103,7 @@ sequenceDiagram
103
103
Receiving a block is more complex:
104
104
105
105
- The block itself needs to be stored.
106
-
- The state transition needs to be applied, a new beacon state calculated and stored separately.
106
+
- The state transition needs to be applied, a new beacon state calculated, and stored separately.
107
107
- A new node needs to be added to the block tree aside from updating weights.
108
108
- on_attestation needs to be called for each attestation.
Copy file name to clipboardExpand all lines: docs/bitvectors.md
+13-13
Original file line number
Diff line number
Diff line change
@@ -2,13 +2,13 @@
2
2
3
3
## Representing integers
4
4
5
-
Computers use transistors to store data. These electrical components only have two possible states: `clear` or `set`. Numerically, we represent the clear state as a `0` and and the set state as `1`. Using 1s and 0s, we can represent any integer number using the binary system, the same way we use the decimal system in our daily lives.
5
+
Computers use transistors to store data. These electrical components only have two possible states: `clear` or `set`. Numerically, we represent the clear state as a `0` and the set state as `1`. Using 1s and 0s, we can represent any integer number using the binary system, the same way we use the decimal system in our daily lives.
6
6
7
-
As an example, let's take the number 259. For its decimal representation, we use the digits 2, 5 and 9, because each digit, or coefficient, represents a power of 10:
7
+
As an example, let's take the number 259. For its decimal representation, we use the digits 2, 5, and 9, because each digit, or coefficient, represents a power of 10:
@@ -44,7 +44,7 @@ Representing it as a byte array, we get `bytes = [3, 1]`. The lowest index, `byt
44
44
45
45
### "Little-endian bit order"
46
46
47
-
Why would we need a third representation? Let's first pose the problem. Imagine we have a fixed amount of validators, equal to 9, and we want to represent whether they attested in a block or not. If the validators 0, 1 and 8 attested, we may represent this with a boolean array, as follows:
47
+
Why would we need a third representation? Let's first pose the problem. Imagine we have a fixed amount of validators, equal to 9, and we want to represent whether they attested in a block or not. If the validators 0, 1, and 8 attested, we may represent this with a boolean array, as follows:
@@ -76,13 +76,13 @@ If we want to convert from each order to each other:
76
76
77
77
- Little-endian byte order to big-endian: reverse the bytes.
78
78
- Little-endian bit order to big-endian: reverse the bits of the whole number.
79
-
- Little-endian bit order to little-endian byte order: reverse the bits of each individual byte. This is equivalent to reversing all bits (converting to big-endian) and then reversing the bytes (big-endian to little-endian byte order) but in a single step.
79
+
- Little-endian bit order to little-endian byte order: reverse the bits of each byte. This is equivalent to reversing all bits (converting to big-endian) and then reversing the bytes (big-endian to little-endian byte order) but in a single step.
80
80
81
81
## Bit vectors
82
82
83
83
### Serialization (SSZ)
84
84
85
-
`bitvectors` are exactly that: a set of booleans with fixed size. SSZ represents bit vectors as follows:
85
+
`bitvectors` are exactly that: a set of booleans with a fixed size. SSZ represents bit vectors as follows:
86
86
87
87
- Conceptually, a set is represented in little-endian bit ordering, padded with 0s at the end to get full bytes.
88
88
- When serializing, we convert from little-endian bit ordering to little-endian byte ordering.
@@ -111,22 +111,22 @@ Moving it to little-endian byte order (we go byte by byte and reverse the bits):
111
111
00000011 00000001
112
112
```
113
113
114
-
This is how nodes send `bitvectors` over the network. We know that this array is of size 9 beforehand, so we know what bits are padding and should be ignored. For variable-sized bit arrays we'll use `bitlists`, which we'll talk about later.
114
+
This is how nodes send `bitvectors` over the network. We know that this array is of size 9 beforehand, so we know what bits are padding and should be ignored. For variable-sized bit arrays, we'll use `bitlists`, which we'll talk about later.
115
115
116
116
### Internal representation
117
117
118
-
There's a trick here: SSZ doesn't specify how to store a `bitvector` in memory after deserializing. We could, theoretically, read the serialized data, transform it from little-endian byte order to little-endian bit order, and use bit addressing (which elixir supports) to get individual values. This would imply, however, going through each byte and reversing the bits, which is a costly operation. If we stuck with little-endian byte order without transforming it, addressing individual bits would be more complicated, and shifting (moving every bit to the left or right) would be tricky.
118
+
There's a trick here: SSZ doesn't specify how to store a `bitvector` in memory after deserializing. We could, theoretically, read the serialized data, transform it from little-endian byte order to little-endian bit order, and use bit addressing (which Elixir supports) to get individual values. This would imply, however, going through each byte and reversing the bits, which is a costly operation. If we stuck with little-endian byte order without transforming it, addressing individual bits would be more complicated, and shifting (moving every bit to the left or right) would be tricky.
119
119
120
120
For this reason, we represent bitvectors in our node as big-endian binaries. That means that we reverse the bytes (a relatively cheap operation) and, for bit addressing, we just use the complementary index. An example:
121
121
122
-
If we are still representing the number 259 (validators with index 0, 1 and 8 attested) we'll have the two following representations (note, elixir has a `bitstring` type that lets you address bit by bit and store an amount of bits that's not a multiple of 8):
122
+
If we are still representing the number 259 (validators with index 0, 1, and 8 attested) we'll have the two following representations (note, elixir has a `bitstring` type that lets you address bit by bit and store several bits that's not a multiple of 8):
123
123
124
124
```
125
125
110000001 -> little-endian bit order
126
126
100000011 -> big-endian
127
127
```
128
128
129
-
If we watch closely, we confirm something we said before: these are bit-mirrored representations. That means that if I want to know if the validator i voted, in the little-endian bit order, we address `bitvector[i]`, and in the big-endian order, we just use `bitvector[N-i]`, where `N=9` as it's the size of the vector.
129
+
If we watch closely, we confirm something we said before: these are bit-mirrored representations. That means that if I want to know if the validator I voted, in the little-endian bit order, we address `bitvector[i]`, and in the big-endian order, we just use `bitvector[N-i]`, where `N=9` as it's the size of the vector.
130
130
131
131
This is the code that performs this conversion:
132
132
@@ -139,9 +139,9 @@ def new(bitstring, size) when is_bitstring(bitstring) do
139
139
end
140
140
```
141
141
142
-
It reads the input as a little-endian number, and then constructs a big-endian binary representation of it.
142
+
It reads the input as a little-endian number and then constructs a big-endian binary representation of it.
143
143
144
-
Instead of using Elixir's bitstrings, a possible optimization (we'd need to benchmark it) would be to represent the array as the number 259 directly, and use bitwise operations to address bits or shift.
144
+
Instead of using Elixir's bitstrings, a possible optimization (we'd need to benchmark it) would be to represent the array as the number 259 directly and use bitwise operations to address bits or shift.
145
145
146
146
## Bitlists
147
147
@@ -177,7 +177,7 @@ When deserializing, we'll look closely at the last byte, realize that there are
177
177
178
178
### Edge case: already a multiple of 8
179
179
180
-
It might be the case that we already have a multiple of 8 as the number of booleans we're representing. For instance, let's suppose that we have 8 validators and only the first and the second one attested. In little-endian bit order, that is:
180
+
It might be the case that we already have a multiple of 8 as the number of booleans we're representing. For instance, let's suppose that we have 8 validators and only the first and the second ones attested. In little-endian bit order, that is:
0 commit comments