Use Bitcoin regtest

Dear maintainers/contributors

Bitcoinlib version 0.6.13

Goal:

Use Bitcoin regtest for platform code testing having the availability to provide test users with credit and run our platform tests.

Problem:

When opening an existing wallet, the wallet.scan() function fails with error:
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']

Steps taken:

• add to providers.json:

"bitcoind": {
    "provider": "bitcoind",
    "network": "regtest",
    "client_class": "BitcoindClient()",
    "provider_coin_id": "",
    "url": "http://<user>:<pass>@bitcoin:8332",
    "api_key": "",
    "priority": 10,
    "denominator": 100000000,
    "network_overrides": null
},

• Change prefix_bech32 in network.jon for regtest:

  "regtest":
  {
    "description": "Bitcoin regtest",
    "currency_name": "regtest",
    "currency_name_plural": "bitcoins",
    "currency_symbol": "r฿",
    "currency_code": "rBTC",
    "prefix_address": "00",
    "prefix_address_p2sh": "05",
    "prefix_bech32": "bcrt",
    "prefix_wif": "80",
    "prefixes_wif": [
      ["0488B21E", "xpub", "public",  false, "legacy",      "p2pkh"],
      ["0488B21E", "xpub", "public",  true,  "legacy",      "p2sh"],
      ["0488ADE4", "xprv", "private", false, "legacy",      "p2pkh"],
      ["0488ADE4", "xprv", "private", true,  "legacy",      "p2sh"],
      ["049D7CB2", "ypub", "public",  false, "p2sh-segwit", "p2sh_p2wpkh"],
      ["0295B43F", "Ypub", "public",  true,  "p2sh-segwit", "p2sh_p2wsh"],
      ["049D7878", "yprv", "private", false, "p2sh-segwit", "p2sh_p2wpkh"],
      ["0295B005", "Yprv", "private", true,  "p2sh-segwit", "p2sh_p2wsh"],
      ["04B24746", "zpub", "public",  false, "segwit",      "p2wpkh"],
      ["02AA7ED3", "Zpub", "public",  true,  "segwit",      "p2wsh"],
      ["04B2430C", "zprv", "private", false, "segwit",      "p2wpkh"],
      ["02AA7A99", "Zprv", "private", true,  "segwit",      "p2wsh"]
    ],
    "bip44_cointype": 0,
    "denominator": 0.00000001,
    "dust_amount": 1000,
    "fee_default": null,
    "fee_min": 1000,
    "fee_max": 1000000,
    "priority": 0
  },

• have a functinoing bitcoin-core node running in regtest mode on url from providers.json

• create some wallets using bitcoinlib

• mine some coins on bitcoin node and send some to a bitcoinlib wallet using bitcoin-cli on bitcoin node.

• bitcoinlib commands:

>>> wallet = bitcoinlib.wallets.wallet_create_or_open("test-user")

>>> bitcoinlib.services.bitcoind.BitcoindClient().network
<Network: bitcoin>

>>> wallet.network
<Network: regtest>  

>>> bitcoinlib.services.bitcoind.BitcoindClient().gettransactions(wallet_address)
[
  <Transaction(id=8b7657..., inputs=1, outputs=2, status=confirmed, network=bitcoin)>,   
  <Transaction(id=0b281a...
]

>>> wallet.scan()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1849, in scan
    self.transactions_update_confirmations()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 3021, in transactions_update_confirmations
    srv = Service(network=network, providers=self.providers, cache_uri=self.db_cache_uri)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 145, in __init__
    self._blockcount = self.blockcount()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 495, in blockcount
    new_count = self._provider_execute('blockcount')
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 224, in _provider_execute
    raise ServiceError("No successful response from any serviceprovider: %s" % list(self.providers.keys()))
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']

>>> # set network to bitcoin for test
>>> wallet.network = bitcoinlib.networks.Network("bitcoin")
>>> w.scan()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1877, in scan
    if self.scan_key(key):
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 1800, in scan_key
    n_new = self.transactions_update(key_id=key.key_id)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/wallets.py", line 3102, in transactions_update
    srv = Service(network=network, providers=self.providers, cache_uri=self.db_cache_uri)
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 145, in __init__
    self._blockcount = self.blockcount()
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 495, in blockcount
    new_count = self._provider_execute('blockcount')
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/services.py", line 224, in _provider_execute
    raise ServiceError("No successful response from any serviceprovider: %s" % list(self.providers.keys()))
bitcoinlib.services.services.ServiceError: No successful response from any serviceprovider: ['bitcoind']
>>> # scan() hangs now, but throws similar error

Maybe I'm doing something wrong or have missed a configuration step?

Any help appreciated and thanks for the great project!

[mccwdev]

It seems related to this: c6efbde
fixed with this pull request: #352

[n1rna]

I'm not sure if the mentioned PR is addressing all the issues mentioned in the issue. I'm also facing the issue with the wrong chain param for the regtest network. The prefix_bech32 value must be bcrt @mccwdev

https://github.com/bitcoin/bitcoin/blob/master/src/kernel/chainparams.cpp#L522

After some more testing and digging i found out that

When wallet.scan() is called in our case, BitcoindClient.gettransactions(wallet_address, '', 20) is called and fails

error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/.local/lib/python3.8/site-packages/bitcoinlib/services/bitcoind.py", line 223, in gettransactions
    raise ClientError("Address %s not found in bitcoind wallet, use 'importpubkey' or 'importaddress' to add "
bitcoinlib.services.baseclient.ClientError: Address bcrt1qn... not found in bitcoind wallet, use 'importpubkey' or 'importaddress' to add address to wallet.

When changing res['iswatchonly'] = True
wallet.scan() works and the balance is updated.

[n1rna]

My workaround for now is pointing to a custom data directory as follows:

[locations]
# Location of BitcoinLib data, configuration and log files. Relative paths will be based in installation directory
data_dir=./data

In the BCL_CONFIG_FILE and then fixing the networks.json file for my own use case.

P.S. You need to replicate all the required files in the data directory: https://github.com/1200wd/bitcoinlib/tree/master/bitcoinlib/data

[raulcano]

I also set up a local environment with a Bitcoind Regtest and was experiencing similar issues.
While I don't address here the root cause or your original post, a work around that got me moving is to keep the bitcoind running in Regtest, but use an ElectrumX provider that connects to the bitcoind. My bitcoinlib would only then interact with ElectrumX and so far, it's working like charm.

A few set up instructions:

my project
|---myapp
|---bitcoin
|------bitcoin.env
|------bitcoin-core (here is the bitcoin folder where the regtest node is running)
|---.electrumx-data (for permanent storage of the electrumx)
|---electrumx
|------docker-compose.yml
|------start_electrumx.sh

• Set up your environment variables in bitcoin.env. This must include, at the very least:

RPC_USER=regtest
RPC_PASSWORD=regtest
RPC_PORT=18443

• Install Docker and get it running. Create a docker-compose.yml file like this:

services:
  electrumx:
    image: lukechilds/electrumx
    container_name: electrumx
    ports:
      - "50001:50001" # TCP server port
      - "50002:50002" # SSL server port
      - "50004:50004" # WebSocket server port
    volumes:
      - "../.electrumx-data:/data"
    environment:
      - DAEMON_URL=http://${RPC_USER:-regtest}:${RPC_PASSWORD:-regtest}@host.docker.internal:${RPC_PORT:-18443}
      - NET=regtest
    restart: unless-stopped
    network_mode: bridge

• Set up the providers.json with the data for your electrumX. So far, it only worked for me in port 50001 (TCP)

"electrumx.regtest": {
    "provider": "electrumx",
    "network": "regtest",
    "client_class": "ElectrumxClient",
    "provider_coin_id": "",
    "url": "localhost:50001,
    "api_key": "",
    "priority": 10,
    "denominator": 100000000,
    "network_overrides": null,
    "timeout": 3600
  }

• Run the bitcoind

bitcoind -conf=/path/to/your/bitcoin.conf -regtest

• Start the ElectrumX docker container. For this I created the script start_electrumx.sh

#!/usr/bin/env bash
# Start ElectrumX server for Bitcoin regtest network

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"

# Load environment variables from bitcoin.env
BITCOIN_ENV_FILE="$PROJECT_ROOT/bitcoin/bitcoin.env"
if [ ! -f "$BITCOIN_ENV_FILE" ]; then
    echo "Warning: bitcoin.env not found at $BITCOIN_ENV_FILE"
    echo "Using default values: regtest:regtest@host.docker.internal:18443"
    # Set defaults
    export RPC_USER=${RPC_USER:-regtest}
    export RPC_PASSWORD=${RPC_PASSWORD:-regtest}
    export RPC_PORT=${RPC_PORT:-18443}
else
    echo "Loading RPC configuration from bitcoin.env..."
    
    # Filter and process bitcoin.env file
    TEMP_FILE="${BITCOIN_ENV_FILE}.tmp"
    grep -v '^[[:space:]]*#' "$BITCOIN_ENV_FILE" | grep -v '^[[:space:]]*$' | grep '=' > "$TEMP_FILE"
    
    # Read and export variables from filtered file
    while IFS='=' read -r key value; do
        # Trim whitespace from key and value
        key=$(echo "$key" | xargs)
        value=$(echo "$value" | xargs)
        
        # Remove any quotes from the value
        value="${value//\"}"
        value="${value//\'}"
        
        # Only export RPC-related variables
        if [ "$key" = "RPC_USER" ] || [ "$key" = "RPC_PASSWORD" ] || [ "$key" = "RPC_PORT" ]; then
            export "$key=$value"
        fi
    done < "$TEMP_FILE"
    
    # Clean up temp file
    rm -f "$TEMP_FILE"
    
    # Set defaults if not found in env file
    export RPC_USER=${RPC_USER:-regtest}
    export RPC_PASSWORD=${RPC_PASSWORD:-regtest}
    export RPC_PORT=${RPC_PORT:-18443}
fi

echo "Using RPC credentials: ${RPC_USER}:***@host.docker.internal:${RPC_PORT}"
echo ""

# Create data directory if it doesn't exist
DATA_DIR="$PROJECT_ROOT/.electrumx-data"
if [ ! -d "$DATA_DIR" ]; then
    echo "Creating ElectrumX data directory: $DATA_DIR"
    mkdir -p "$DATA_DIR"
fi

# Change to the electrum directory where docker-compose.yml is located
cd "$SCRIPT_DIR"

# Check if Docker is running
if ! docker info > /dev/null 2>&1; then
    echo "Error: Docker is not running. Please start Docker Desktop and try again."
    exit 1
fi

echo "Starting ElectrumX server..."
echo "  Data directory: $DATA_DIR"
echo "  Network: regtest"
echo "  Port: 50001 (TCP)"
echo "  Port: 50002 (SSL)"
echo "  Port: 50004 (WebSocket)"
echo "  Bitcoin Core RPC: http://${RPC_USER}:***@host.docker.internal:${RPC_PORT}"
echo ""

# Start the container
docker-compose up -d

# Wait a moment for the container to start
sleep 2

# Check if the container is running
if docker ps | grep -q electrumx; then
    echo "✓ ElectrumX container is running"
    echo ""
    echo "Showing live logs (Ctrl+C to exit)..."
    echo ""
    echo "To stop the container: docker-compose -f $SCRIPT_DIR/docker-compose.yml down"
    echo "================================"
    echo ""
    
    # Show live logs
    docker-compose logs -f
else
    echo "✗ Failed to start ElectrumX container"
    echo "Check logs with: docker-compose logs"
    exit 1
fi

Hope this helps.
Good luck!