diff --git a/README.md b/README.md index f58c98712..ff5b7ebb6 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,6 @@ Monitor and analyze the emergent behaviors of Bitcoin networks. - [Monitoring](/docs/logging_monitoring.md) - [Lightning Network](/docs/lightning.md) - [Scaling](/docs/scaling.md) -- [Connecting to local nodes](https://github.com/bitcoin-dev-project/warnet/blob/main/docs/ +- [Connecting to local nodes](https://github.com/bitcoin-dev-project/warnet/blob/main/docs/) ![warnet-art](https://raw.githubusercontent.com/bitcoin-dev-project/warnet/main/docs/machines.webp) diff --git a/docs/install.md b/docs/install.md index 6db83889b..e04d58267 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,26 +1,27 @@ # Install Warnet -Warnet requires _either_ Kubernetes or Docker to be installed in order to run the network. -Instructions for both can be found below. +Warnet requires Kubernetes in order to run the network. Kubernetes can be run +remotely or locally (with minikube or Docker Desktop). `kubectl` must be run +locally to administer the network. ## Dependencies ### Kubernetes Install [`kubectl`](https://kubernetes.io/docs/setup/) (or equivalent) and -configure your cluster. This can be done locally with `minikube` or using a -managed cluster. +configure your cluster. This can be done locally with `minikube` (or Docker Desktop) +or using a managed cluster. #### Docker engine with minikube If using Minikube to run a smaller-sized local cluster, you will require docker engine. -To install docker engine and its `compose` plugin for your system, see: https://docs.docker.com/engine/install/ +To install docker engine, see: https://docs.docker.com/engine/install/ e.g. For Ubuntu: ```bash # First uninstall any old versions -for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done +for pkg in docker.io docker-doc podman-docker containerd runc; do sudo apt-get remove $pkg; done # Add Docker's official GPG key: sudo apt-get update @@ -36,7 +37,7 @@ echo \ sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update -# Install the docker packages and docker compose plugin +# Install the docker packages sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin ``` @@ -50,28 +51,30 @@ The Docker daemon MUST be running before stating Warnet. - [Check Docker user/group permissions](https://stackoverflow.com/a/48957722/1653320) - or [`chmod` the Docker UNIX socket](https://stackoverflow.com/a/51362528/1653320) -##### macOS +## Install Warnet -On macOS, a bridge to the docker subnet is required, such as https://github.com/chipmk/docker-mac-net-connect +### Recommended: use a virtual Python environment such as `venv` ```bash -# Install via Homebrew -brew install chipmk/tap/docker-mac-net-connect +python3 -m venv .venv # Use alternative venv manager if desired +source .venv/bin/activate +``` -# Run the service and register it to launch at boot -sudo brew services start chipmk/tap/docker-mac-net-connect +```bash +pip install --upgrade pip +pip install warnet ``` -## Download Warnet +## Contributing / Local Warnet Development + +### Download the code repository ```bash git clone https://github.com/bitcoin-dev-project/warnet cd warnet ``` -## Install Warnet - -### Optional: use a virtual Python environment such as `venv` +### Recommended: use a virtual Python environment such as `venv` ```bash python3 -m venv .venv # Use alternative venv manager if desired @@ -82,3 +85,4 @@ source .venv/bin/activate pip install --upgrade pip pip install -e . ``` + diff --git a/docs/logging_monitoring.md b/docs/logging_monitoring.md index eb0b501d2..c205806aa 100644 --- a/docs/logging_monitoring.md +++ b/docs/logging_monitoring.md @@ -6,15 +6,17 @@ Warnet allows different granularity of logging. ### Warnet network level logging -For information like: +Fetch logs from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network. + +Examples of information provided: - how many tanks are running - what scenarios are running -- RPC requests +- warnet RPC requests -To run these `warcli network logs` or `warcli network logs --follow`. +Commands: `warcli network logs` or `warcli network logs --follow`. -These logs are fetched from the warnet RPC server `rpc-0`, which is in charge of orchestrating the network. +See more details in [warcli](/docs/warcli.md#warcli-network-logs) ### Bitcoin Core logs @@ -36,7 +38,7 @@ For logs of lightning nodes, kubectl is required. ### Aggregated logs from all nodes -Aggregated logs can be searched using RPC `grep-logs` with regex patterns. +Aggregated logs can be searched using `warcli bitcoin grep-logs` with regex patterns. Example: @@ -52,6 +54,8 @@ warnet_test_uhynisdj_tank_000007: 2023-10-11T17:44:52.173199Z [validation] Enque ... (etc) ``` +See more details in [warcli](/docs/warcli.md#warcli-bitcoin-grep-logs) + ## Monitoring and Metrics ## Install logging infrastructure @@ -73,7 +77,7 @@ It might take a couple minutes to get the pod running. If you see `error: unable The Grafana dashboard (and API) will be accessible without requiring authentication at `http://localhost:3000`. -The below logging scripts need to be installed before starting the network in order to collect the information for monitoring and metrics. `warcli network down && warcli network up` should do the trick. +The `install_logging` script will need to be installed before starting the network in order to collect the information for monitoring and metrics. Restart the network with `warcli network down && warcli network up` if necessary. ### Prometheus diff --git a/docs/running.md b/docs/running.md index 80d652d4e..04339a329 100644 --- a/docs/running.md +++ b/docs/running.md @@ -1,48 +1,32 @@ # Running Warnet -Warnet runs a server which can be used to manage multiple networks. On docker -this runs locally, but on Kubernetes this runs as a `statefulSet` in the -cluster. +Warnet runs a server which can be used to manage multiple networks. On Kubernetes +this runs as a `statefulSet` in the cluster. -If the `$XDG_STATE_HOME` environment variable is set, the server will log to -a file `$XDG_STATE_HOME/warnet/warnet.log`, otherwise it will use `$HOME/.warnet/warnet.log`. +See more details in [warcli](/docs/warcli.md), examples: -## Kubernetes - -// TODO - -### Install logging infrastructure - -First make sure you have `helm` installed, then simply run the following script: - -```bash -./scripts/install_logging.sh -``` - -To forward port to view Grafana dashboard: +To start the server run: ```bash -./scripts/connect_logging.sh +warcli cluster deploy ``` -## Kubernetes (e.g. minikube) - -To start the server run: +Start a network from a graph file: ```bash -warcli network start +warcli network start resources/graphs/default.graphml ``` -Make sure the tanks are running with: +Make sure all tanks are running with: ```bash - warcli network status +warcli network status ``` -Check if the edges of the nodes are connected with: +Check if the edges of the graph (bitcoin p2p connections) are complete: ```bash - warcli network connected +warcli network connected ``` _Optional_ Check out the logs with: diff --git a/docs/scaling.md b/docs/scaling.md index 7035affde..066827122 100644 --- a/docs/scaling.md +++ b/docs/scaling.md @@ -1,6 +1,6 @@ # Running large networks -When running a large number of containers on a single host machine (i.e. with the Docker interface), the system may run out of various resources. +When running a large number of containers on a single host machine, the system may run out of various resources. We recommend setting the following values in /etc/sysctl.conf: ```sh diff --git a/docs/scenarios.md b/docs/scenarios.md index 06292e6ed..acdc470b2 100644 --- a/docs/scenarios.md +++ b/docs/scenarios.md @@ -2,8 +2,8 @@ Scenarios are written using the Bitcoin Core test framework for functional testing, with some modifications: most notably that `self.nodes[]` represents an array of -dockerized `bitcoind` nodes. Scenario files can be run with a python interpreter -or from `warcli` commands and used to control many nodes in the network simultaneously. +containerized `bitcoind` nodes ("tanks"). Scenario files are run with a python interpreter +inside the server and can control many nodes in the network simultaneously. See [`src/warnet/scenarios`](../src/warnet/scenarios) for examples of how these can be written. @@ -13,7 +13,7 @@ To see available scenarios (loaded from the default directory): warcli scenarios available ``` -Once a scenarios is selected it can be run with `warcli scenarios run [--network=warnet] [scenario_params]`. +Once a scenario is selected it can be run with `warcli scenarios run [--network=warnet] [scenario_params]`. The [`miner_std`](../src/warnet/scenarios/miner_std.py) scenario is a good one to start with as it automates block generation: @@ -22,7 +22,7 @@ The [`miner_std`](../src/warnet/scenarios/miner_std.py) scenario is a good one t warcli scenarios run miner_std --allnodes --interval=5 ``` -This will run the run the scenario in the background until it exits or is stopped by the user. +This will run the scenario in a background thread on the server until it exits or is stopped by the user. Active scenarios can be listed and terminated by PID: @@ -44,7 +44,11 @@ $ warcli scenarios stop 14683 Stopped scenario with PID 14683. ``` -## Add scenarios +## Running a custom scenario -To add your own scenario make a copy of one of the existing python tests in src/scenarios/ and write the desired scenario. -Save this file back into the same `src/warnet/scenarios/` directory and it will be listed and available for running using the aforementioned commands. +You can write your own scenario file locally and upload it to the server with +the [run-file](/docs/warcli.md#warcli-scenarios-run-file) command (example): + +```bash +warcli scenarios run-file /home/me/bitcoin_attack.py +``` diff --git a/resources/scripts/connect_logging.sh b/resources/scripts/connect_logging.sh index 2bf7a05f0..20ea4a63e 100755 --- a/resources/scripts/connect_logging.sh +++ b/resources/scripts/connect_logging.sh @@ -8,7 +8,7 @@ echo "Grafana pod name: ${POD_NAME}" while true; do echo "Attempting to start Grafana port forwarding" - echo "(Hang tight... it might take a couple of minutes to get the pod running)" + echo "Please wait... it might take a few minutes for all logging pods to start" kubectl --namespace warnet-logging port-forward "${POD_NAME}" 3000 2>&1 echo "Grafana port forwarding exited with status: $?" sleep 5