diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index 8d5671ce7..000000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,20 +0,0 @@ -# Contributing / Local Warnet Development - -## Download the code repository - -```bash -git clone https://github.com/bitcoin-dev-project/warnet -cd warnet -``` - -## Recommended: use a virtual Python environment such as `venv` - -```bash -python3 -m venv .venv # Use alternative venv manager if desired -source .venv/bin/activate -``` - -```bash -pip install --upgrade pip -pip install -e . -``` \ No newline at end of file diff --git a/README.md b/README.md index beeb264b1..383e7faf1 100644 --- a/README.md +++ b/README.md @@ -5,24 +5,22 @@ Monitor and analyze the emergent behaviors of Bitcoin networks. ## Major Features -* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology from a graphml file. -* Scenarios can be run across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional). -* Nodes can have traffic shaping parameters assigned to them via the graph using [tc-netem](https://manpages.ubuntu.com/manpages/trusty/man8/tc-netem.8.html) tool. -* Data from nodes can be collected and searched including log files and p2p messages. -* Performance data from containers can be monitored and visualized. -* Lightning Network nodes can be deployed and operated. -* Networks can be deployed using Kubernetes, e.g. via MiniKube (small network graphs) or a managed cluster for larger network graphs. +* Launch a bitcoin network with a specified number of nodes connected to each other according to a network topology. +* Run scenarios of network behavior across the network which can be programmed using the Bitcoin Core functional [test_framework language](https://github.com/bitcoin/bitcoin/tree/master/test/functional). +* Collect and search data from nodes including log files and p2p messages. +* Monitor and visualize performance data from Bitcoin nodes. +* Connect to a large network running in a remote cluster, or a smaller network running locally. ## Documentation - [Installation](/docs/install.md) - [CLI Commands](/docs/warnet.md) +- [Network configuration with yaml files](/docs/config.md) - [Scenarios](/docs/scenarios.md) - [Monitoring](/docs/logging_monitoring.md) -- [Lightning Network](/docs/lightning.md) +- [Snapshots](/docs/snapshots.md) +- [Connecting to local nodes outside the cluster](/docs/connecting-local-nodes.md) - [Scaling](/docs/scaling.md) -- [Connecting to local nodes](/docs/connecting-local-nodes.md) -- [Understanding network configuration](/docs/config.md) -- [Contributing](CONTRIBUTING.md) +- [Contributing](/docs/developer-notes.md) ![warnet-art](https://raw.githubusercontent.com/bitcoin-dev-project/warnet/main/docs/machines.webp) diff --git a/docs/connecting-local-nodes.md b/docs/connecting-local-nodes.md index 96407b3f2..89bade682 100644 --- a/docs/connecting-local-nodes.md +++ b/docs/connecting-local-nodes.md @@ -7,8 +7,7 @@ ### Run Warnet network ```shell -warnet cluster deploy -warnet network start +warnet deploy path/to/network/directory ``` ### Install Telepresence diff --git a/docs/developer-notes.md b/docs/developer-notes.md index 0c51bc8b3..14d1b8d5b 100644 --- a/docs/developer-notes.md +++ b/docs/developer-notes.md @@ -1,8 +1,62 @@ # Developer notes +## Contributing / Local Warnet Development + +### Download the code repository + +```bash +git clone https://github.com/bitcoin-dev-project/warnet +cd warnet +``` + +### Recommended: use a virtual Python environment such as `venv` + +```bash +python3 -m venv .venv # Use alternative venv manager if desired +source .venv/bin/activate +``` + +```bash +pip install --upgrade pip +pip install -e . +``` + +## Lint + This project primarily uses the `uv` python packaging tool: https://docs.astral.sh/uv/ along with the sister formatter/linter `ruff` https://docs.astral.sh/ruff/ With `uv` installed you can add/remove dependencies using `uv add ` or `uv remove . This will update the [`uv.lock`](https://docs.astral.sh/uv/guides/projects/#uvlock) file automatically. + `uv` can also run tools (like `ruff`) without external installation, simply run `uvx ruff check .` or `uvx ruff format .` to use a uv-managed format/lint on the project. + +## Release process + +Once a tag is pushed to GH this will start an image build using the tag + +### Prerequisites + +- [ ] Update version in pyproject.toml +- [ ] Tag git with new version +- [ ] Push tag to GitHub + +### Manual Builds + +```bash +# Install build dependencies +pip install -e .[build] + +# Remove previous release metadata +rm -i -Rf build/ dist/ + +# Build wheel +python3 -m build +``` + +#### Upload + +```bash +# Upload to Pypi +python3 -m twine upload dist/* +``` \ No newline at end of file diff --git a/docs/lightning.md b/docs/lightning.md deleted file mode 100644 index 72404bc4c..000000000 --- a/docs/lightning.md +++ /dev/null @@ -1,97 +0,0 @@ -# Lightning Network - -## Adding LN nodes to graph - -LN nodes can be added to any Bitcoin Core node by adding a data element with key -`"ln"` to the node in the graph file. The value is the LN implementation desired. - -**Currently only `lnd` is supported** - -Example: - -``` - - lnd - -``` - -## Adding LN channels to graph - -LN channels are represented in the graphml file as edges with extra data elements -that correspond to arguments to the lnd `openchannel` and `updatechanpolicy` RPC -commands. The keys are: - -- `"channel_open"` (arguments added to `openchannel`) -- `"target_policy"` or `"source_policy"` (arguments added to `updatechanpolicy`) - -The key `"channel_open"` is required to open a LN channel in warnet, and to -identify an edge in the graphml file as a LN channel. - -Example: - -``` - - --local_amt=100000 - --base_fee_msat=100 --fee_rate_ppm=5 --time_lock_delta=18 - --base_fee_msat=2200 --fee_rate_ppm=13 --time_lock_delta=20 - -``` - -A complete example graph with LN nodes and channels is included in the test -data directory: [ln.graphml](../test/data/ln.graphml) - -## Running the Lightning network - -When warnet is started with `warnet network start` the bitcoin containers will -be started first followed by the lightning node containers. It may require a few -automatic restarts before the lightning nodes start up and connect to their -corresponding bitcoin nodes. Use `warnet network status` to monitor container status -and wait for all containers to be `running`. - -To create the lightning channels specified in the graph file, run the included -scenario: - -`warnet scenarios run ln_init` - -This [scenario](../src/scenarios/ln_init.py) will generate blocks, fund the wallets -in the bitcoin nodes, and open the channels from the graph. Each of these steps -requires some waiting as transactions are confirmed in the warnet blockchain -and lightning nodes gossip their channel announcements to each other. -Use `warnet scenarios active` to monitor the status of the scenario. When it is -complete the subprocess will exit and it will indicate `Active: False`. At that -point, the lightning network is ready for activity. - -## sim-ln compatibility - -Warnet can export data required to run [sim-ln](https://github.com/bitcoin-dev-project/sim-ln) -with a warnet network. - -With a network running, execute: `warnet network export` with optional argument -`--network=` (default is "warnet"). This will copy all lightning -node credentials like SSL certificates and macaroons into a local directory as -well as generate a JSON file required by sim-ln. - -Example (see sim-ln docs for exact API): - -``` -$ warnet network export -/Users/bitcoin-dev-project/.warnet/warnet/warnet/simln - -$ ls /Users/bitcoin-dev-project/.warnet/warnet/warnet/simln -sim.json warnet_ln_000000_tls.cert warnet_ln_000001_tls.cert warnet_ln_000002_tls.cert -warnet_ln_000000_admin.macaroon warnet_ln_000001_admin.macaroon warnet_ln_000002_admin.macaroon - -$ sim-cli --data-dir /Users/bitcoin-dev-project/.warnet/warnet/warnet/simln -2023-11-18T16:58:28.731Z INFO [sim_cli] Connected to warnet_ln_000000 - Node ID: 031b1404744431b01ee4fa2bfc3c5caa1f1044ff5a9cb553d2c8ec6eb0f9d8040c. -2023-11-18T16:58:28.747Z INFO [sim_cli] Connected to warnet_ln_000001 - Node ID: 02318b75bd91bf6265b30fe97f8ebbb0eda85194cf9d4467d43374de0248c7bf05. -2023-11-18T16:58:28.760Z INFO [sim_cli] Connected to warnet_ln_000002 - Node ID: 0393aa24d777e2391b5238c485ecce08b35bd9aa4ddf4f2226016107c6829804d5. -2023-11-18T16:58:28.760Z INFO [sim_lib] Running the simulation forever. -2023-11-18T16:58:28.815Z INFO [sim_lib] Simulation is running on regtest. -2023-11-18T16:58:28.815Z INFO [sim_lib] Simulating 0 activity on 3 nodes. -2023-11-18T16:58:28.815Z INFO [sim_lib] Summary of results will be reported every 60s. -2023-11-18T16:58:28.826Z INFO [sim_lib] Generating random activity with multiplier: 2, average payment amount: 3800000. -2023-11-18T16:58:28.826Z INFO [sim_lib] Created network generator: network graph view with: 3 channels. -2023-11-18T16:58:28.826Z INFO [sim_lib] Started random activity producer for warnet_ln_000000(031b14...d8040c): activity generator for capacity: 50000000 with multiplier 2: 26.31578947368421 payments per month (0.03654970760233918 per hour). -2023-11-18T16:58:28.826Z INFO [sim_lib] Started random activity producer for warnet_ln_000001(02318b...c7bf05): activity generator for capacity: 100000000 with multiplier 2: 52.63157894736842 payments per month (0.07309941520467836 per hour). -2023-11-18T16:58:28.826Z INFO [sim_lib] Started random activity producer for warnet_ln_000002(0393aa...9804d5): activity generator for capacity: 50000000 with multiplier 2: 26.31578947368421 payments per month (0.03654970760233918 per hour). -``` diff --git a/docs/logging_monitoring.md b/docs/logging_monitoring.md index 07699469f..cd3a037dd 100644 --- a/docs/logging_monitoring.md +++ b/docs/logging_monitoring.md @@ -1,31 +1,23 @@ # Logging and Monitoring -Warnet allows different granularity of logging. - ## Logging -### Warnet network level logging - -Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network. - -Examples of information provided: +### Pod logs -- how many tanks are running -- what scenarios are running -- warnet RPC requests +The command `warnet logs` will bring up a menu of pods to print log output from, +such as Bitcoin tanks, or scenario commanders. Follow the output with the `-f` option. -Commands: `warnet network logs` or `warnet network logs --follow`. - -See more details in [warnet](/docs/warnet.md#warnet-network-logs) +See command [`warnet logs`](/docs/warnet.md#warnet-logs) ### Bitcoin Core logs -These are tank level or pod level log output from a Bitcoin Core node, useful for things like net logging and transaction propagation, retrieved by RPC `debug-log` using its network name and graph node index. +Entire debug log files from a Bitcoin tank can be dumped by using the tank's +pod name. Example: ```sh -$ warnet bitcoin debug-log 0 +$ warnet bitcoin debug-log tank-0000 2023-10-11T17:54:39.616974Z Bitcoin Core version v25.0.0 (release build) @@ -34,50 +26,46 @@ $ warnet bitcoin debug-log 0 ... (etc) ``` -For logs of lightning nodes, kubectl is required. +See command [`warnet bitcoin debug-log`](/docs/warnet.md#warnet-bitcoin-debug-log) -### Aggregated logs from all nodes +### Aggregated logs from all Bitcoin nodes Aggregated logs can be searched using `warnet bitcoin grep-logs` with regex patterns. +See more details in [`warnet bitcoin grep-logs`](/docs/warnet.md#warnet-bitcoin-grep-logs) + Example: ```sh $ warnet bitcoin grep-logs 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d -warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate -warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay -warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf -warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf -warnet_test_uhynisdj_tank_000001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d -warnet_test_uhynisdj_tank_000007: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf +tank-0001: 2023-10-11T17:44:48.716582Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d newupdate +tank-0001: 2023-10-11T17:44:48.717787Z [miner] Submitting wtx 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d to mempool for relay +tank-0001: 2023-10-11T17:44:48.717929Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf +tank-0001: 2023-10-11T17:44:48.718040Z [validation] TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf +tank-0001: 2023-10-11T17:44:48.723017Z [miner] AddToWallet 94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d +tank-0002: 2023-10-11T17:44:52.173199Z [validation] Enqueuing TransactionAddedToMempool: txid=94cacabc09b024b56dcbed9ccad15c90340c596e883159bcb5f1d2152997322d wtxid=0cc875e73bb0bd8f892b70b8d1e5154aab64daace8d571efac94c62b8c1da3cf ... (etc) ``` -See more details in [warnet](/docs/warnet.md#warnet-bitcoin-grep-logs) ## Monitoring and Metrics ## Install logging infrastructure -Ensure that [`helm`](https://helm.sh/docs/intro/install/) is installed, then simply run the following scripts: +If any tank in a network is configured with `collectLogs: true` or `metricsExport: true` +then the logging stack will be installed automatically when `warnet deploy` is executed. -```bash -./resources/scripts/install_logging.sh -``` +The logging stack includes Loki, Prometheus, and Grafana. Together these programs +aggregate logs and data from Bitcoin RPC queries into a web-based dashboard. -To forward port `3000` and view the [Grafana](#grafana) dashboard run the `connect_logging` script: +## Connect to logging dashboard -```bash -./resources/scripts/connect_logging.sh -``` - -It might take a couple minutes to get the pod running. If you see `error: unable to forward port because pod is not running. Current status=Pending`, hang tight. - -The Grafana dashboard (and API) will be accessible without requiring authentication -at `http://localhost:3000`. +The logging stack including the user interface web server runs inside the kubernetes cluster. +To access that from a local web browser, you must use kubernetes port-forwarding. -The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warnet network down && warnet network up` if necessary. +Run the script `./resources/scripts/connect_logging.sh` to forward port 3000. +The Grafana dashboard will then be available locally at `localhost:3000`. ### Prometheus @@ -86,14 +74,7 @@ to any Bitcoin Tank and configured to scrape any available RPC results. The `bitcoin-exporter` image is defined in `resources/images/exporter` and maintained in the BitcoinDevProject dockerhub organization. To add the exporter -in the Tank pod with Bitcoin Core add the `"exporter"` key to the node in the graphml file: - -```xml - - 27.0 - true - -``` +in the Tank pod with Bitcoin Core add the `metricsExport: true` value to the node in the yaml file. The default metrics are defined in the `bitcoin-exporter` image: - Block count @@ -101,25 +82,23 @@ The default metrics are defined in the `bitcoin-exporter` image: - Number of outbound peers - Mempool size (# of TXs) -Metrics can be configured by setting a `"metrics"` key to the node in the graphml file. -The metrics value is a space-separated list of labels, RPC commands with arguments, and +Metrics can be configured by setting an additional `metrics` value to the node in the yaml file. The metrics value is a space-separated list of labels, RPC commands with arguments, and JSON keys to resolve the desired data: ``` label=method(arguments)[JSON result key][...] ``` -For example, the default metrics listed above are defined as: +For example, the default metrics listed above would be explicitly configured as follows: -```xml - - 27.0 - true - blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"] - +```yaml +nodes: + - name: tank-0000 + metricsExport: true + metrics: blocks=getblockcount() inbounds=getnetworkinfo()["connections_in"] outbounds=getnetworkinfo()["connections_in"] mempool_size=getmempoolinfo()["size"] ``` -The data can be retrieved from the Prometheus exporter on port `9332`, example: +The data can be retrieved directly from the Prometheus exporter container in the tank pod via port `9332`, example: ``` # HELP blocks getblockcount() @@ -138,7 +117,7 @@ mempool_size 0.0 ### Grafana -Data from Prometheus exporters can be collected and fed into Grafana for a +Data from Prometheus exporters is collected and fed into Grafana for a web-based interface. #### Dashboards diff --git a/docs/random_internet_as_graph_n100.png b/docs/random_internet_as_graph_n100.png deleted file mode 100644 index 3e0fd3422..000000000 Binary files a/docs/random_internet_as_graph_n100.png and /dev/null differ diff --git a/docs/release-process.md b/docs/release-process.md deleted file mode 100644 index 6f4c28090..000000000 --- a/docs/release-process.md +++ /dev/null @@ -1,29 +0,0 @@ -# Release process - -Once a tag is pushed to GH this will start an image build using the tag - -## Prerequisites - -- [ ] Update version in pyproject.toml -- [ ] Tag git with new version -- [ ] Push tag to GitHub - -## Manual Builds - -```bash -# Install build dependencies -pip install -e .[build] - -# Remove previous release metadata -rm -i -Rf build/ dist/ - -# Build wheel -python3 -m build -``` - -### Upload - -```bash -# Upload to Pypi -python3 -m twine upload dist/* -``` diff --git a/docs/scaling.md b/docs/scaling.md index 3ae91886b..39b74208a 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -1,6 +1,6 @@ # Running large networks -> [NOTE] +> [!NOTE] > These changes are not required on multi-host/managed clusters When running a large number of containers on a single host machine, the system may run out of various resources. diff --git a/docs/scenarios.md b/docs/scenarios.md index 8b113473e..c2892c421 100644 --- a/docs/scenarios.md +++ b/docs/scenarios.md @@ -2,8 +2,11 @@ Scenarios are written using the Bitcoin Core test framework for functional testing, with some modifications: most notably that `self.nodes[]` represents an array of -containerized `bitcoind` nodes ("tanks"). Scenario files are run with a python interpreter -inside the server and can control many nodes in the network simultaneously. +containerized `bitcoind` nodes ("tanks"). + +Scenario files are run with a python interpreter inside their own pod called a "commander" +in kubernetes and many can be run simultaneously. The Commander is provided with +a JSON file describing the Bitcoin nodes it has access to via RPC. See [`resources/scenarios/`](../resources/scenarios/) for examples of how these can be written. When creating a new network default scenarios will be copied into your project directory for convenience. diff --git a/src/warnet/graph.py b/src/warnet/graph.py index 36b8020ba..6e5b3fd6b 100644 --- a/src/warnet/graph.py +++ b/src/warnet/graph.py @@ -16,17 +16,6 @@ def graph(): """Create and validate network graphs""" -@graph.command() -def import_json(infile: Path, outfile: Path, cb: str, ln_image: str): - """ - Create a cycle graph with nodes imported from lnd `describegraph` JSON file, - and additionally include 7 extra random outbounds per node. Include lightning - channels and their policies as well. - Returns XML file as string with or without --outfile option. - """ - raise Exception("Not Implemented") - - def custom_graph( num_nodes: int, num_connections: int,