Skip to content

Commit f8a6c04

Browse files
authored
docs: overhaul all (#585)
1 parent 39b04e8 commit f8a6c04

11 files changed

+106
-230
lines changed

CONTRIBUTING.md

Lines changed: 0 additions & 20 deletions
This file was deleted.

README.md

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -5,24 +5,22 @@ Monitor and analyze the emergent behaviors of Bitcoin networks.
55

66
## Major Features
77

8-
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology from a graphml file.
9-
* Scenarios can be run across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
10-
* Nodes can have traffic shaping parameters assigned to them via the graph using [tc-netem](https://manpages.ubuntu.com/manpages/trusty/man8/tc-netem.8.html) tool.
11-
* Data from nodes can be collected and searched including log files and p2p messages.
12-
* Performance data from containers can be monitored and visualized.
13-
* Lightning Network nodes can be deployed and operated.
14-
* Networks can be deployed using Kubernetes, e.g. via MiniKube (small network graphs) or a managed cluster for larger network graphs.
8+
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology.
9+
* Run scenarios of network behavior across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
10+
* Collect and search data from nodes including log files and p2p messages.
11+
* Monitor and visualize performance data from Bitcoin nodes.
12+
* Connect to a large network running in a remote cluster, or a smaller network running locally.
1513

1614
## Documentation
1715

1816
- [Installation](/docs/install.md)
1917
- [CLI Commands](/docs/warnet.md)
18+
- [Network configuration with yaml files](/docs/config.md)
2019
- [Scenarios](/docs/scenarios.md)
2120
- [Monitoring](/docs/logging_monitoring.md)
22-
- [Lightning Network](/docs/lightning.md)
21+
- [Snapshots](/docs/snapshots.md)
22+
- [Connecting to local nodes outside the cluster](/docs/connecting-local-nodes.md)
2323
- [Scaling](/docs/scaling.md)
24-
- [Connecting to local nodes](/docs/connecting-local-nodes.md)
25-
- [Understanding network configuration](/docs/config.md)
26-
- [Contributing](CONTRIBUTING.md)
24+
- [Contributing](/docs/developer-notes.md)
2725

2826
![warnet-art](https://raw.githubusercontent.com/bitcoin-dev-project/warnet/main/docs/machines.webp)

docs/connecting-local-nodes.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,7 @@
77
### Run Warnet network
88

99
```shell
10-
warnet cluster deploy
11-
warnet network start
10+
warnet deploy path/to/network/directory
1211
```
1312

1413
### Install Telepresence

docs/developer-notes.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,62 @@
11
# Developer notes
22

3+
## Contributing / Local Warnet Development
4+
5+
### Download the code repository
6+
7+
```bash
8+
git clone https://github.com/bitcoin-dev-project/warnet
9+
cd warnet
10+
```
11+
12+
### Recommended: use a virtual Python environment such as `venv`
13+
14+
```bash
15+
python3 -m venv .venv # Use alternative venv manager if desired
16+
source .venv/bin/activate
17+
```
18+
19+
```bash
20+
pip install --upgrade pip
21+
pip install -e .
22+
```
23+
24+
## Lint
25+
326
This project primarily uses the `uv` python packaging tool: https://docs.astral.sh/uv/ along with the sister formatter/linter `ruff` https://docs.astral.sh/ruff/
427

528
With `uv` installed you can add/remove dependencies using `uv add <dep>` or `uv remove <dep>.
629
This will update the [`uv.lock`](https://docs.astral.sh/uv/guides/projects/#uvlock) file automatically.
730

31+
832
`uv` can also run tools (like `ruff`) without external installation, simply run `uvx ruff check .` or `uvx ruff format .` to use a uv-managed format/lint on the project.
33+
34+
## Release process
35+
36+
Once a tag is pushed to GH this will start an image build using the tag
37+
38+
### Prerequisites
39+
40+
- [ ] Update version in pyproject.toml
41+
- [ ] Tag git with new version
42+
- [ ] Push tag to GitHub
43+
44+
### Manual Builds
45+
46+
```bash
47+
# Install build dependencies
48+
pip install -e .[build]
49+
50+
# Remove previous release metadata
51+
rm -i -Rf build/ dist/
52+
53+
# Build wheel
54+
python3 -m build
55+
```
56+
57+
#### Upload
58+
59+
```bash
60+
# Upload to Pypi
61+
python3 -m twine upload dist/*
62+
```

docs/lightning.md

Lines changed: 0 additions & 97 deletions
This file was deleted.

docs/logging_monitoring.md

Lines changed: 36 additions & 57 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,23 @@
11
# Logging and Monitoring
22

3-
Warnet allows different granularity of logging.
4-
53
## Logging
64

7-
### Warnet network level logging
8-
9-
Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network.
10-
11-
Examples of information provided:
5+
### Pod logs
126

13-
- how many tanks are running
14-
- what scenarios are running
15-
- warnet RPC requests
7+
The command `warnet logs` will bring up a menu of pods to print log output from,
8+
such as Bitcoin tanks, or scenario commanders. Follow the output with the `-f` option.
169

17-
Commands: `warnet network logs` or `warnet network logs --follow`.
18-
19-
See more details in [warnet](/docs/warnet.md#warnet-network-logs)
10+
See command [`warnet logs`](/docs/warnet.md#warnet-logs)
2011

2112
### Bitcoin Core logs
2213

23-
These are tank level or pod level log output from a Bitcoin Core node, useful for things like net logging and transaction propagation, retrieved by RPC `debug-log` using its network name and graph node index.
14+
Entire debug log files from a Bitcoin tank can be dumped by using the tank's
15+
pod name.
2416

2517
Example:
2618

2719
```sh
28-
$ warnet bitcoin debug-log 0
20+
$ warnet bitcoin debug-log tank-0000
2921

3022

3123
2023-10-11T17:54:39.616974Z Bitcoin Core version v25.0.0 (release build)
@@ -34,50 +26,46 @@ $ warnet bitcoin debug-log 0
3426
... (etc)
3527
```
3628

37-
For logs of lightning nodes, kubectl is required.
29+
See command [`warnet bitcoin debug-log`](/docs/warnet.md#warnet-bitcoin-debug-log)
3830

39-
### Aggregated logs from all nodes
31+
### Aggregated logs from all Bitcoin nodes
4032

4133
Aggregated logs can be searched using `warnet bitcoin grep-logs` with regex patterns.
4234

35+
See more details in [`warnet bitcoin grep-logs`](/docs/warnet.md#warnet-bitcoin-grep-logs)
36+
4337
Example:
4438

4539
```sh
4640
$ warnet bitcoin grep-logs 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
4741

48-
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
49-
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
50-
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
51-
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
52-
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
53-
warnet_test_uhynisdj_tank_000007: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
42+
tank-0001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
43+
tank-0001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
44+
tank-0001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
45+
tank-0001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
46+
tank-0001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
47+
tank-0002: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
5448
... (etc)
5549
```
5650

57-
See more details in [warnet](/docs/warnet.md#warnet-bitcoin-grep-logs)
5851

5952
## Monitoring and Metrics
6053

6154
## Install logging infrastructure
6255

63-
Ensure that [`helm`](https://helm.sh/docs/intro/install/) is installed, then simply run the following scripts:
56+
If any tank in a network is configured with `collectLogs: true` or `metricsExport: true`
57+
then the logging stack will be installed automatically when `warnet deploy` is executed.
6458

65-
```bash
66-
./resources/scripts/install_logging.sh
67-
```
59+
The logging stack includes Loki, Prometheus, and Grafana. Together these programs
60+
aggregate logs and data from Bitcoin RPC queries into a web-based dashboard.
6861

69-
To forward port `3000` and view the [Grafana](#grafana) dashboard run the `connect_logging` script:
62+
## Connect to logging dashboard
7063

71-
```bash
72-
./resources/scripts/connect_logging.sh
73-
```
74-
75-
It might take a couple minutes to get the pod running. If you see `error: unable to forward port because pod is not running. Current status=Pending`, hang tight.
76-
77-
The Grafana dashboard (and API) will be accessible without requiring authentication
78-
at `http://localhost:3000`.
64+
The logging stack including the user interface web server runs inside the kubernetes cluster.
65+
To access that from a local web browser, you must use kubernetes port-forwarding.
7966

80-
The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warnet network down && warnet network up` if necessary.
67+
Run the script `./resources/scripts/connect_logging.sh` to forward port 3000.
68+
The Grafana dashboard will then be available locally at `localhost:3000`.
8169

8270
### Prometheus
8371

@@ -86,40 +74,31 @@ to any Bitcoin Tank and configured to scrape any available RPC results.
8674

8775
The `bitcoin-exporter` image is defined in `resources/images/exporter` and
8876
maintained in the BitcoinDevProject dockerhub organization. To add the exporter
89-
in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file:
90-
91-
```xml
92-
<node id="0">
93-
<data key="version">27.0</data>
94-
<data key="exporter">true</data>
95-
</node>
96-
```
77+
in the Tank pod with Bitcoin Core add the `metricsExport: true` value to the node in the yaml file.
9778

9879
The default metrics are defined in the `bitcoin-exporter` image:
9980
- Block count
10081
- Number of inbound peers
10182
- Number of outbound peers
10283
- Mempool size (# of TXs)
10384

104-
Metrics can be configured by setting a `"metrics"` key to the node in the graphml file.
105-
The metrics value is a space-separated list of labels, RPC commands with arguments, and
85+
Metrics can be configured by setting an additional `metrics` value to the node in the yaml file. The metrics value is a space-separated list of labels, RPC commands with arguments, and
10686
JSON keys to resolve the desired data:
10787

10888
```
10989
label=method(arguments)[JSON result key][...]
11090
```
11191

112-
For example, the default metrics listed above are defined as:
92+
For example, the default metrics listed above would be explicitly configured as follows:
11393

114-
```xml
115-
<node id="0">
116-
<data key="version">27.0</data>
117-
<data key="exporter">true</data>
118-
<data key="metrics">blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]</data>
119-
</node>
94+
```yaml
95+
nodes:
96+
- name: tank-0000
97+
metricsExport: true
98+
metrics: blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]
12099
```
121100
122-
The data can be retrieved from the Prometheus exporter on port `9332`, example:
101+
The data can be retrieved directly from the Prometheus exporter container in the tank pod via port `9332`, example:
123102

124103
```
125104
# HELP blocks getblockcount()
@@ -138,7 +117,7 @@ mempool_size 0.0
138117
139118
### Grafana
140119
141-
Data from Prometheus exporters can be collected and fed into Grafana for a
120+
Data from Prometheus exporters is collected and fed into Grafana for a
142121
web-based interface.
143122
144123
#### Dashboards
-461 KB
Binary file not shown.

0 commit comments

Comments
 (0)