You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+9-11Lines changed: 9 additions & 11 deletions
Original file line number
Diff line number
Diff line change
@@ -5,24 +5,22 @@ Monitor and analyze the emergent behaviors of Bitcoin networks.
5
5
6
6
## Major Features
7
7
8
-
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology from a graphml file.
9
-
* Scenarios can be run across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
10
-
* Nodes can have traffic shaping parameters assigned to them via the graph using [tc-netem](https://manpages.ubuntu.com/manpages/trusty/man8/tc-netem.8.html) tool.
11
-
* Data from nodes can be collected and searched including log files and p2p messages.
12
-
* Performance data from containers can be monitored and visualized.
13
-
* Lightning Network nodes can be deployed and operated.
14
-
* Networks can be deployed using Kubernetes, e.g. via MiniKube (small network graphs) or a managed cluster for larger network graphs.
8
+
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology.
9
+
* Run scenarios of network behavior across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
10
+
* Collect and search data from nodes including log files and p2p messages.
11
+
* Monitor and visualize performance data from Bitcoin nodes.
12
+
* Connect to a large network running in a remote cluster, or a smaller network running locally.
15
13
16
14
## Documentation
17
15
18
16
-[Installation](/docs/install.md)
19
17
-[CLI Commands](/docs/warnet.md)
18
+
-[Network configuration with yaml files](/docs/config.md)
20
19
-[Scenarios](/docs/scenarios.md)
21
20
-[Monitoring](/docs/logging_monitoring.md)
22
-
-[Lightning Network](/docs/lightning.md)
21
+
-[Snapshots](/docs/snapshots.md)
22
+
-[Connecting to local nodes outside the cluster](/docs/connecting-local-nodes.md)
23
23
-[Scaling](/docs/scaling.md)
24
-
-[Connecting to local nodes](/docs/connecting-local-nodes.md)
### Recommended: use a virtual Python environment such as `venv`
13
+
14
+
```bash
15
+
python3 -m venv .venv # Use alternative venv manager if desired
16
+
source .venv/bin/activate
17
+
```
18
+
19
+
```bash
20
+
pip install --upgrade pip
21
+
pip install -e .
22
+
```
23
+
24
+
## Lint
25
+
3
26
This project primarily uses the `uv` python packaging tool: https://docs.astral.sh/uv/ along with the sister formatter/linter `ruff`https://docs.astral.sh/ruff/
4
27
5
28
With `uv` installed you can add/remove dependencies using `uv add <dep>` or `uv remove <dep>.
6
29
This will update the [`uv.lock`](https://docs.astral.sh/uv/guides/projects/#uvlock) file automatically.
7
30
31
+
8
32
`uv` can also run tools (like `ruff`) without external installation, simply run `uvx ruff check .` or `uvx ruff format .` to use a uv-managed format/lint on the project.
33
+
34
+
## Release process
35
+
36
+
Once a tag is pushed to GH this will start an image build using the tag
Copy file name to clipboardExpand all lines: docs/logging_monitoring.md
+36-57Lines changed: 36 additions & 57 deletions
Original file line number
Diff line number
Diff line change
@@ -1,31 +1,23 @@
1
1
# Logging and Monitoring
2
2
3
-
Warnet allows different granularity of logging.
4
-
5
3
## Logging
6
4
7
-
### Warnet network level logging
8
-
9
-
Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network.
10
-
11
-
Examples of information provided:
5
+
### Pod logs
12
6
13
-
- how many tanks are running
14
-
- what scenarios are running
15
-
- warnet RPC requests
7
+
The command `warnet logs` will bring up a menu of pods to print log output from,
8
+
such as Bitcoin tanks, or scenario commanders. Follow the output with the `-f` option.
16
9
17
-
Commands: `warnet network logs` or `warnet network logs --follow`.
18
-
19
-
See more details in [warnet](/docs/warnet.md#warnet-network-logs)
10
+
See command [`warnet logs`](/docs/warnet.md#warnet-logs)
20
11
21
12
### Bitcoin Core logs
22
13
23
-
These are tank level or pod level log output from a Bitcoin Core node, useful for things like net logging and transaction propagation, retrieved by RPC `debug-log` using its network name and graph node index.
14
+
Entire debug log files from a Bitcoin tank can be dumped by using the tank's
15
+
pod name.
24
16
25
17
Example:
26
18
27
19
```sh
28
-
$ warnet bitcoin debug-log 0
20
+
$ warnet bitcoin debug-log tank-0000
29
21
30
22
31
23
2023-10-11T17:54:39.616974Z Bitcoin Core version v25.0.0 (release build)
@@ -34,50 +26,46 @@ $ warnet bitcoin debug-log 0
34
26
... (etc)
35
27
```
36
28
37
-
For logs of lightning nodes, kubectl is required.
29
+
See command [`warnet bitcoin debug-log`](/docs/warnet.md#warnet-bitcoin-debug-log)
38
30
39
-
### Aggregated logs from all nodes
31
+
### Aggregated logs from all Bitcoin nodes
40
32
41
33
Aggregated logs can be searched using `warnet bitcoin grep-logs` with regex patterns.
42
34
35
+
See more details in [`warnet bitcoin grep-logs`](/docs/warnet.md#warnet-bitcoin-grep-logs)
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
See more details in [warnet](/docs/warnet.md#warnet-bitcoin-grep-logs)
58
51
59
52
## Monitoring and Metrics
60
53
61
54
## Install logging infrastructure
62
55
63
-
Ensure that [`helm`](https://helm.sh/docs/intro/install/) is installed, then simply run the following scripts:
56
+
If any tank in a network is configured with `collectLogs: true` or `metricsExport: true`
57
+
then the logging stack will be installed automatically when `warnet deploy` is executed.
64
58
65
-
```bash
66
-
./resources/scripts/install_logging.sh
67
-
```
59
+
The logging stack includes Loki, Prometheus, and Grafana. Together these programs
60
+
aggregate logs and data from Bitcoin RPC queries into a web-based dashboard.
68
61
69
-
To forward port `3000` and view the [Grafana](#grafana) dashboard run the `connect_logging` script:
62
+
## Connect to logging dashboard
70
63
71
-
```bash
72
-
./resources/scripts/connect_logging.sh
73
-
```
74
-
75
-
It might take a couple minutes to get the pod running. If you see `error: unable to forward port because pod is not running. Current status=Pending`, hang tight.
76
-
77
-
The Grafana dashboard (and API) will be accessible without requiring authentication
78
-
at `http://localhost:3000`.
64
+
The logging stack including the user interface web server runs inside the kubernetes cluster.
65
+
To access that from a local web browser, you must use kubernetes port-forwarding.
79
66
80
-
The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warnet network down && warnet network up` if necessary.
67
+
Run the script `./resources/scripts/connect_logging.sh` to forward port 3000.
68
+
The Grafana dashboard will then be available locally at `localhost:3000`.
81
69
82
70
### Prometheus
83
71
@@ -86,40 +74,31 @@ to any Bitcoin Tank and configured to scrape any available RPC results.
86
74
87
75
The `bitcoin-exporter` image is defined in `resources/images/exporter` and
88
76
maintained in the BitcoinDevProject dockerhub organization. To add the exporter
89
-
in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file:
90
-
91
-
```xml
92
-
<nodeid="0">
93
-
<datakey="version">27.0</data>
94
-
<datakey="exporter">true</data>
95
-
</node>
96
-
```
77
+
in the Tank pod with Bitcoin Core add the `metricsExport: true` value to the node in the yaml file.
97
78
98
79
The default metrics are defined in the `bitcoin-exporter` image:
99
80
- Block count
100
81
- Number of inbound peers
101
82
- Number of outbound peers
102
83
- Mempool size (# of TXs)
103
84
104
-
Metrics can be configured by setting a `"metrics"` key to the node in the graphml file.
105
-
The metrics value is a space-separated list of labels, RPC commands with arguments, and
85
+
Metrics can be configured by setting an additional `metrics` value to the node in the yaml file. The metrics value is a space-separated list of labels, RPC commands with arguments, and
106
86
JSON keys to resolve the desired data:
107
87
108
88
```
109
89
label=method(arguments)[JSON result key][...]
110
90
```
111
91
112
-
For example, the default metrics listed above are defined as:
92
+
For example, the default metrics listed above would be explicitly configured as follows:
0 commit comments