Skip to content

Commit

Permalink
docs: overhaul all (#585)
Browse files Browse the repository at this point in the history
  • Loading branch information
pinheadmz authored Sep 12, 2024
1 parent 39b04e8 commit f8a6c04
Show file tree
Hide file tree
Showing 11 changed files with 106 additions and 230 deletions.
20 changes: 0 additions & 20 deletions CONTRIBUTING.md

This file was deleted.

20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,24 +5,22 @@ Monitor and analyze the emergent behaviors of Bitcoin networks.

## Major Features

* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology from a graphml file.
* Scenarios can be run across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
* Nodes can have traffic shaping parameters assigned to them via the graph using [tc-netem](https://manpages.ubuntu.com/manpages/trusty/man8/tc-netem.8.html) tool.
* Data from nodes can be collected and searched including log files and p2p messages.
* Performance data from containers can be monitored and visualized.
* Lightning Network nodes can be deployed and operated.
* Networks can be deployed using Kubernetes, e.g. via MiniKube (small network graphs) or a managed cluster for larger network graphs.
* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology.
* Run scenarios of network behavior across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional).
* Collect and search data from nodes including log files and p2p messages.
* Monitor and visualize performance data from Bitcoin nodes.
* Connect to a large network running in a remote cluster, or a smaller network running locally.

## Documentation

- [Installation](/docs/install.md)
- [CLI Commands](/docs/warnet.md)
- [Network configuration with yaml files](/docs/config.md)
- [Scenarios](/docs/scenarios.md)
- [Monitoring](/docs/logging_monitoring.md)
- [Lightning Network](/docs/lightning.md)
- [Snapshots](/docs/snapshots.md)
- [Connecting to local nodes outside the cluster](/docs/connecting-local-nodes.md)
- [Scaling](/docs/scaling.md)
- [Connecting to local nodes](/docs/connecting-local-nodes.md)
- [Understanding network configuration](/docs/config.md)
- [Contributing](CONTRIBUTING.md)
- [Contributing](/docs/developer-notes.md)

![warnet-art](https://raw.githubusercontent.com/bitcoin-dev-project/warnet/main/docs/machines.webp)
3 changes: 1 addition & 2 deletions docs/connecting-local-nodes.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,7 @@
### Run Warnet network

```shell
warnet cluster deploy
warnet network start
warnet deploy path/to/network/directory
```

### Install Telepresence
Expand Down
54 changes: 54 additions & 0 deletions docs/developer-notes.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,62 @@
# Developer notes

## Contributing / Local Warnet Development

### Download the code repository

```bash
git clone https://github.com/bitcoin-dev-project/warnet
cd warnet
```

### Recommended: use a virtual Python environment such as `venv`

```bash
python3 -m venv .venv # Use alternative venv manager if desired
source .venv/bin/activate
```

```bash
pip install --upgrade pip
pip install -e .
```

## Lint

This project primarily uses the `uv` python packaging tool: https://docs.astral.sh/uv/ along with the sister formatter/linter `ruff` https://docs.astral.sh/ruff/

With `uv` installed you can add/remove dependencies using `uv add <dep>` or `uv remove <dep>.
This will update the [`uv.lock`](https://docs.astral.sh/uv/guides/projects/#uvlock) file automatically.


`uv` can also run tools (like `ruff`) without external installation, simply run `uvx ruff check .` or `uvx ruff format .` to use a uv-managed format/lint on the project.

## Release process

Once a tag is pushed to GH this will start an image build using the tag

### Prerequisites

- [ ] Update version in pyproject.toml
- [ ] Tag git with new version
- [ ] Push tag to GitHub

### Manual Builds

```bash
# Install build dependencies
pip install -e .[build]

# Remove previous release metadata
rm -i -Rf build/ dist/

# Build wheel
python3 -m build
```

#### Upload

```bash
# Upload to Pypi
python3 -m twine upload dist/*
```
97 changes: 0 additions & 97 deletions docs/lightning.md

This file was deleted.

93 changes: 36 additions & 57 deletions docs/logging_monitoring.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,23 @@
# Logging and Monitoring

Warnet allows different granularity of logging.

## Logging

### Warnet network level logging

Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network.

Examples of information provided:
### Pod logs

- how many tanks are running
- what scenarios are running
- warnet RPC requests
The command `warnet logs` will bring up a menu of pods to print log output from,
such as Bitcoin tanks, or scenario commanders. Follow the output with the `-f` option.

Commands: `warnet network logs` or `warnet network logs --follow`.

See more details in [warnet](/docs/warnet.md#warnet-network-logs)
See command [`warnet logs`](/docs/warnet.md#warnet-logs)

### Bitcoin Core logs

These are tank level or pod level log output from a Bitcoin Core node, useful for things like net logging and transaction propagation, retrieved by RPC `debug-log` using its network name and graph node index.
Entire debug log files from a Bitcoin tank can be dumped by using the tank's
pod name.

Example:

```sh
$ warnet bitcoin debug-log 0
$ warnet bitcoin debug-log tank-0000


2023-10-11T17:54:39.616974Z Bitcoin Core version v25.0.0 (release build)
Expand All @@ -34,50 +26,46 @@ $ warnet bitcoin debug-log 0
... (etc)
```

For logs of lightning nodes, kubectl is required.
See command [`warnet bitcoin debug-log`](/docs/warnet.md#warnet-bitcoin-debug-log)

### Aggregated logs from all nodes
### Aggregated logs from all Bitcoin nodes

Aggregated logs can be searched using `warnet bitcoin grep-logs` with regex patterns.

See more details in [`warnet bitcoin grep-logs`](/docs/warnet.md#warnet-bitcoin-grep-logs)

Example:

```sh
$ warnet bitcoin grep-logs 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d

warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
warnet_test_uhynisdj_tank_000007: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate
tank-0001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay
tank-0001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
tank-0001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d
tank-0002: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf
... (etc)
```

See more details in [warnet](/docs/warnet.md#warnet-bitcoin-grep-logs)

## Monitoring and Metrics

## Install logging infrastructure

Ensure that [`helm`](https://helm.sh/docs/intro/install/) is installed, then simply run the following scripts:
If any tank in a network is configured with `collectLogs: true` or `metricsExport: true`
then the logging stack will be installed automatically when `warnet deploy` is executed.

```bash
./resources/scripts/install_logging.sh
```
The logging stack includes Loki, Prometheus, and Grafana. Together these programs
aggregate logs and data from Bitcoin RPC queries into a web-based dashboard.

To forward port `3000` and view the [Grafana](#grafana) dashboard run the `connect_logging` script:
## Connect to logging dashboard

```bash
./resources/scripts/connect_logging.sh
```

It might take a couple minutes to get the pod running. If you see `error: unable to forward port because pod is not running. Current status=Pending`, hang tight.

The Grafana dashboard (and API) will be accessible without requiring authentication
at `http://localhost:3000`.
The logging stack including the user interface web server runs inside the kubernetes cluster.
To access that from a local web browser, you must use kubernetes port-forwarding.

The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warnet network down && warnet network up` if necessary.
Run the script `./resources/scripts/connect_logging.sh` to forward port 3000.
The Grafana dashboard will then be available locally at `localhost:3000`.

### Prometheus

Expand All @@ -86,40 +74,31 @@ to any Bitcoin Tank and configured to scrape any available RPC results.

The `bitcoin-exporter` image is defined in `resources/images/exporter` and
maintained in the BitcoinDevProject dockerhub organization. To add the exporter
in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file:

```xml
<node id="0">
<data key="version">27.0</data>
<data key="exporter">true</data>
</node>
```
in the Tank pod with Bitcoin Core add the `metricsExport: true` value to the node in the yaml file.

The default metrics are defined in the `bitcoin-exporter` image:
- Block count
- Number of inbound peers
- Number of outbound peers
- Mempool size (# of TXs)

Metrics can be configured by setting a `"metrics"` key to the node in the graphml file.
The metrics value is a space-separated list of labels, RPC commands with arguments, and
Metrics can be configured by setting an additional `metrics` value to the node in the yaml file. The metrics value is a space-separated list of labels, RPC commands with arguments, and
JSON keys to resolve the desired data:

```
label=method(arguments)[JSON result key][...]
```

For example, the default metrics listed above are defined as:
For example, the default metrics listed above would be explicitly configured as follows:

```xml
<node id="0">
<data key="version">27.0</data>
<data key="exporter">true</data>
<data key="metrics">blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]</data>
</node>
```yaml
nodes:
- name: tank-0000
metricsExport: true
metrics: blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"]
```
The data can be retrieved from the Prometheus exporter on port `9332`, example:
The data can be retrieved directly from the Prometheus exporter container in the tank pod via port `9332`, example:

```
# HELP blocks getblockcount()
Expand All @@ -138,7 +117,7 @@ mempool_size 0.0
### Grafana
Data from Prometheus exporters can be collected and fed into Grafana for a
Data from Prometheus exporters is collected and fed into Grafana for a
web-based interface.
#### Dashboards
Expand Down
Binary file removed docs/random_internet_as_graph_n100.png
Binary file not shown.
Loading

0 comments on commit f8a6c04

Please sign in to comment.