Merge bitcoin#22261: [p2p/mempool] Two small fixes to node broadcast logic

5a77abd [style] Clean up BroadcastTransaction() (John Newbery)
7282d4c [test] Allow rebroadcast for same-txid-different-wtxid transactions (glozow)
cd48372 [mempool] Allow rebroadcast for same-txid-different-wtxid transactions (John Newbery)
847b6ed [test] Test transactions are not re-added to unbroadcast set (Duncan Dean)
2837a9f [mempool] Only add a transaction to the unbroadcast set when it's added to the mempool (John Newbery)

Pull request description:

  1. Only add a transaction to the unbroadcast set when it's added to the mempool

      Currently, if BroadcastTransaction() is called to rebroadcast a
      transaction (e.g. by ResendWalletTransactions()), then we add the
      transaction to the unbroadcast set. That transaction has already been
      broadcast in the past, so peers are unlikely to request it again,
      meaning RemoveUnbroadcastTx() won't be called and it won't be removed
      from m_unbroadcast_txids.

      Net processing will therefore continue to attempt rebroadcast for the
      transaction every 10-15 minutes. This will most likely continue until
      the node connects to a new peer which hasn't yet seen the transaction
      (or perhaps indefinitely).

      Fix by only adding the transaction to the broadcast set when it's added to the mempool.

  2. Allow rebroadcast for same-txid-different-wtxid transactions

      There is some slightly unexpected behaviour when:

      - there is already transaction in the mempool (the "mempool tx")
      - BroadcastTransaction() is called for a transaction with the same txid
        as the mempool transaction but a different witness (the "new tx")

      Prior to this commit, if BroadcastTransaction() is called with
      relay=true, then it'll call RelayTransaction() using the txid/wtxid of
      the new tx, not the txid/wtxid of the mempool tx. For wtxid relay peers,
      in SendMessages(), the wtxid of the new tx will be taken from
      setInventoryTxToSend, but will then be filtered out from the vector of
      wtxids to announce, since m_mempool.info() won't find the transaction
      (the mempool contains the mempool tx, which has a different wtxid from
      the new tx).

      Fix this by calling RelayTransaction() with the wtxid of the mempool
      transaction in this case.

  The third commit is a comment/whitespace only change to tidy up the BroadcastTransaction() function.

ACKs for top commit:
  duncandean:
    reACK 5a77abd
  naumenkogs:
    ACK 5a77abd
  theStack:
    re-ACK 5a77abd
  lsilva01:
    re-ACK bitcoin@5a77abd

Tree-SHA512: d1a46d32a9f975220e5b432ff6633fac9be01ea41925b4958395b8d641680500dc44476b12d18852e5b674d2d87e4d0160b4483e45d3d149176bdff9f4dc8516

Author: fanquake

src/node/transaction.cpp

@@ -28,65 +28,83 @@ static TransactionError HandleATMPError(const TxValidationState& state, std::str
 
 TransactionError BroadcastTransaction(NodeContext& node, const CTransactionRef tx, std::string& err_string, const CAmount& max_tx_fee, bool relay, bool wait_callback)
 {
-    // BroadcastTransaction can be called by either sendrawtransaction RPC or wallet RPCs.
-    // node.peerman is assigned both before chain clients and before RPC server is accepting calls,
-    // and reset after chain clients and RPC sever are stopped. node.peerman should never be null here.
-    assert(node.peerman);
+    // BroadcastTransaction can be called by either sendrawtransaction RPC or the wallet.
+    // chainman, mempool and peerman are initialized before the RPC server and wallet are started
+    // and reset after the RPC sever and wallet are stopped.
+    assert(node.chainman);
     assert(node.mempool);
+    assert(node.peerman);
+
     std::promise<void> promise;
-    uint256 hashTx = tx->GetHash();
+    uint256 txid = tx->GetHash();
+    uint256 wtxid = tx->GetWitnessHash();
     bool callback_set = false;
 
-    { // cs_main scope
-    assert(node.chainman);
-    LOCK(cs_main);
-    // If the transaction is already confirmed in the chain, don't do anything
-    // and return early.
-    CCoinsViewCache &view = node.chainman->ActiveChainstate().CoinsTip();
-    for (size_t o = 0; o < tx->vout.size(); o++) {
-        const Coin& existingCoin = view.AccessCoin(COutPoint(hashTx, o));
-        // IsSpent doesn't mean the coin is spent, it means the output doesn't exist.
-        // So if the output does exist, then this transaction exists in the chain.
-        if (!existingCoin.IsSpent()) return TransactionError::ALREADY_IN_CHAIN;
-    }
-    if (!node.mempool->exists(hashTx)) {
-        // Transaction is not already in the mempool.
-        if (max_tx_fee > 0) {
-            // First, call ATMP with test_accept and check the fee. If ATMP
-            // fails here, return error immediately.
+    {
+        LOCK(cs_main);
+
+        // If the transaction is already confirmed in the chain, don't do anything
+        // and return early.
+        CCoinsViewCache &view = node.chainman->ActiveChainstate().CoinsTip();
+        for (size_t o = 0; o < tx->vout.size(); o++) {
+            const Coin& existingCoin = view.AccessCoin(COutPoint(txid, o));
+            // IsSpent doesn't mean the coin is spent, it means the output doesn't exist.
+            // So if the output does exist, then this transaction exists in the chain.
+            if (!existingCoin.IsSpent()) return TransactionError::ALREADY_IN_CHAIN;
+        }
+
+        if (auto mempool_tx = node.mempool->get(txid); mempool_tx) {
+            // There's already a transaction in the mempool with this txid. Don't
+            // try to submit this transaction to the mempool (since it'll be
+            // rejected as a TX_CONFLICT), but do attempt to reannounce the mempool
+            // transaction if relay=true.
+            //
+            // The mempool transaction may have the same or different witness (and
+            // wtxid) as this transaction. Use the mempool's wtxid for reannouncement.
+            wtxid = mempool_tx->GetWitnessHash();
+        } else {
+            // Transaction is not already in the mempool.
+            if (max_tx_fee > 0) {
+                // First, call ATMP with test_accept and check the fee. If ATMP
+                // fails here, return error immediately.
+                const MempoolAcceptResult result = AcceptToMemoryPool(node.chainman->ActiveChainstate(), *node.mempool, tx, false /* bypass_limits */,
+                                                                      true /* test_accept */);
+                if (result.m_result_type != MempoolAcceptResult::ResultType::VALID) {
+                    return HandleATMPError(result.m_state, err_string);
+                } else if (result.m_base_fees.value() > max_tx_fee) {
+                    return TransactionError::MAX_FEE_EXCEEDED;
+                }
+            }
+            // Try to submit the transaction to the mempool.
             const MempoolAcceptResult result = AcceptToMemoryPool(node.chainman->ActiveChainstate(), *node.mempool, tx, false /* bypass_limits */,
-                                                                  true /* test_accept */);
+                                                                  false /* test_accept */);
             if (result.m_result_type != MempoolAcceptResult::ResultType::VALID) {
                 return HandleATMPError(result.m_state, err_string);
-            } else if (result.m_base_fees.value() > max_tx_fee) {
-                return TransactionError::MAX_FEE_EXCEEDED;
             }
-        }
-        // Try to submit the transaction to the mempool.
-        const MempoolAcceptResult result = AcceptToMemoryPool(node.chainman->ActiveChainstate(), *node.mempool, tx, false /* bypass_limits */,
-                                                              false /* test_accept */);
-        if (result.m_result_type != MempoolAcceptResult::ResultType::VALID) {
-            return HandleATMPError(result.m_state, err_string);
-        }
 
-        // Transaction was accepted to the mempool.
+            // Transaction was accepted to the mempool.
 
-        if (wait_callback) {
-            // For transactions broadcast from outside the wallet, make sure
-            // that the wallet has been notified of the transaction before
-            // continuing.
-            //
-            // This prevents a race where a user might call sendrawtransaction
-            // with a transaction to/from their wallet, immediately call some
-            // wallet RPC, and get a stale result because callbacks have not
-            // yet been processed.
-            CallFunctionInValidationInterfaceQueue([&promise] {
-                promise.set_value();
-            });
-            callback_set = true;
-        }
-    }
+            if (relay) {
+                // the mempool tracks locally submitted transactions to make a
+                // best-effort of initial broadcast
+                node.mempool->AddUnbroadcastTx(txid);
+            }
 
+            if (wait_callback) {
+                // For transactions broadcast from outside the wallet, make sure
+                // that the wallet has been notified of the transaction before
+                // continuing.
+                //
+                // This prevents a race where a user might call sendrawtransaction
+                // with a transaction to/from their wallet, immediately call some
+                // wallet RPC, and get a stale result because callbacks have not
+                // yet been processed.
+                CallFunctionInValidationInterfaceQueue([&promise] {
+                    promise.set_value();
+                });
+                callback_set = true;
+            }
+        }
     } // cs_main
 
     if (callback_set) {
@@ -96,10 +114,7 @@ TransactionError BroadcastTransaction(NodeContext& node, const CTransactionRef t
     }
 
     if (relay) {
-        // the mempool tracks locally submitted transactions to make a
-        // best-effort of initial broadcast
-        node.mempool->AddUnbroadcastTx(hashTx);
-        node.peerman->RelayTransaction(hashTx, tx->GetWitnessHash());
+        node.peerman->RelayTransaction(txid, wtxid);
     }
 
     return TransactionError::OK;

test/functional/mempool_accept_wtxid.py

@@ -16,6 +16,7 @@
     CTxOut,
     sha256,
 )
+from test_framework.p2p import P2PTxInvStore
 from test_framework.script import (
     CScript,
     OP_0,
@@ -62,6 +63,8 @@ def run_test(self):
         parent_txid = node.sendrawtransaction(hexstring=raw_parent, maxfeerate=0)
         node.generate(1)
 
+        peer_wtxid_relay = node.add_p2p_connection(P2PTxInvStore())
+
         # Create a new transaction with witness solving first branch
         child_witness_script = CScript([OP_TRUE])
         child_witness_program = sha256(child_witness_script)
@@ -87,10 +90,13 @@ def run_test(self):
         assert_equal(child_one_txid, child_two_txid)
         assert child_one_wtxid != child_two_wtxid
 
-        self.log.info("Submit one child to the mempool")
+        self.log.info("Submit child_one to the mempool")
         txid_submitted = node.sendrawtransaction(child_one.serialize().hex())
         assert_equal(node.getrawmempool(True)[txid_submitted]['wtxid'], child_one_wtxid)
 
+        peer_wtxid_relay.wait_for_broadcast([child_one_wtxid])
+        assert_equal(node.getmempoolinfo()["unbroadcastcount"], 0)
+
         # testmempoolaccept reports the "already in mempool" error
         assert_equal(node.testmempoolaccept([child_one.serialize().hex()]), [{
             "txid": child_one_txid,
@@ -108,9 +114,18 @@ def run_test(self):
 
         # sendrawtransaction will not throw but quits early when the exact same transaction is already in mempool
         node.sendrawtransaction(child_one.serialize().hex())
+
+        self.log.info("Connect another peer that hasn't seen child_one before")
+        peer_wtxid_relay_2 = node.add_p2p_connection(P2PTxInvStore())
+
+        self.log.info("Submit child_two to the mempool")
         # sendrawtransaction will not throw but quits early when a transaction with the same non-witness data is already in mempool
         node.sendrawtransaction(child_two.serialize().hex())
 
+        # The node should rebroadcast the transaction using the wtxid of the correct transaction
+        # (child_one, which is in its mempool).
+        peer_wtxid_relay_2.wait_for_broadcast([child_one_wtxid])
+        assert_equal(node.getmempoolinfo()["unbroadcastcount"], 0)
 
 if __name__ == '__main__':
     MempoolWtxidTest().main()

test/functional/mempool_unbroadcast.py

@@ -92,6 +92,12 @@ def test_broadcast(self):
         self.disconnect_nodes(0, 1)
         node.disconnect_p2ps()
 
+        self.log.info("Rebroadcast transaction and ensure it is not added to unbroadcast set when already in mempool")
+        rpc_tx_hsh = node.sendrawtransaction(txFS["hex"])
+        mempool = node.getrawmempool(True)
+        assert rpc_tx_hsh in mempool
+        assert not mempool[rpc_tx_hsh]['unbroadcast']
+
     def test_txn_removal(self):
         self.log.info("Test that transactions removed from mempool are removed from unbroadcast set")
         node = self.nodes[0]