Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Router disables central node within route even if there is only one path #3685

Open
MaxFangX opened this issue Mar 26, 2025 · 8 comments · May be fixed by #3707
Open

Router disables central node within route even if there is only one path #3685

MaxFangX opened this issue Mar 26, 2025 · 8 comments · May be fixed by #3707

Comments

@MaxFangX
Copy link
Contributor

MaxFangX commented Mar 26, 2025

Consider this specific configuration:

  • Alice -> Bob 200k and 300k sat channels
  • Bob -> Charlie 600k sat private channel
  • Send a MPP payment Alice -> Charlie of size 350k sats

If Charlie includes just one route hint, the maximum payment amount for which LDK routing succeeds is around ~295.5k sats., i.e. it's not possible to send a MPP payment which uses more than one of Alice's channels.

Here's the relevant logs from testing this:

2025-03-25T23:36:49.040772Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Searching for a route from payer 031e18d11d9fdbc175f2fd73cfab07346146069fcf1ecfbeef6e55632a3bda7fe5 to payee node id 02deb1b2c0f91b77107faae1ce96820fbcb82ec541780db7e0306f92396770c669 with MPP and 2 first hops overriding the network graph of 0 nodes and 0 channels with a fee limit of 18446744073709551615 msat module="lightning::routing::router" line=2234 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.040799Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk:  First hop through 035a70d45eec7efb270319f116a9684250acb4ef282a26d21874878e7c5088f73b/114349209354241 can send between 1msat and 197460000msat (inclusive). module="lightning::routing::router" line=2244 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.040818Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk:  First hop through 035a70d45eec7efb270319f116a9684250acb4ef282a26d21874878e7c5088f73b/120946279120897 can send between 1msat and 296460000msat (inclusive). module="lightning::routing::router" line=2244 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.040858Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Building path from payee node id 02deb1b2c0f91b77107faae1ce96820fbcb82ec541780db7e0306f92396770c669 to payer 031e18d11d9fdbc175f2fd73cfab07346146069fcf1ecfbeef6e55632a3bda7fe5 for value 350000000 msat. module="lightning::routing::router" line=2362 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.040951Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Starting main path collection loop with 2 nodes pre-filled from first/last hops. module="lightning::routing::router" line=3052 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.040977Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Found a path back to us from the target with 2 hops contributing up to 295410000 msat: 
 [
    PathBuildingHop {
        source_node_id: NodeId(031e18d11d9fdbc175f2fd73cfab07346146069fcf1ecfbeef6e55632a3bda7fe5),
        target_node_id: Some(
            NodeId(035a70d45eec7efb270319f116a9684250acb4ef282a26d21874878e7c5088f73b),
        ),
        short_channel_id: Some(
            120946279120897,
        ),
        is_first_hop_target: false,
        total_fee_msat: 1050000,
        next_hops_fee_msat: 1050000,
        hop_use_fee_msat: 0,
        total_fee_msat - (next_hops_fee_msat + hop_use_fee_msat): 0,
        path_penalty_msat: 0,
        path_htlc_minimum_msat: 1,
        cltv_expiry_delta: 0,
    },
    PathBuildingHop {
        source_node_id: NodeId(035a70d45eec7efb270319f116a9684250acb4ef282a26d21874878e7c5088f73b),
        target_node_id: Some(
            NodeId(02deb1b2c0f91b77107faae1ce96820fbcb82ec541780db7e0306f92396770c669),
        ),
        short_channel_id: Some(
            1099588108288,
        ),
        is_first_hop_target: true,
        total_fee_msat: 1050000,
        next_hops_fee_msat: 0,
        hop_use_fee_msat: 0,
        total_fee_msat - (next_hops_fee_msat + hop_use_fee_msat): 1050000,
        path_penalty_msat: 0,
        path_htlc_minimum_msat: 0,
        cltv_expiry_delta: 72,
    },
] module="lightning::routing::router" line=3131 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041024Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Disabling route candidate route hint with SCID 1099588108288 for future path building iterations to avoid duplicates. module="lightning::routing::router" line=3176 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041047Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Ignoring route hint with SCID 1099588108288 due to insufficient value contribution (channel max Infinite). module="lightning::routing::router" line=2946 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041065Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Starting main path collection loop with 0 nodes pre-filled from first/last hops. module="lightning::routing::router" line=3052 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041085Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Ignoring route hint with SCID 1099588108288 due to insufficient value contribution (channel max Infinite). module="lightning::routing::router" line=2946 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041102Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Starting main path collection loop with 0 nodes pre-filled from first/last hops. module="lightning::routing::router" line=3052 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041118Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Have now collected 295410000 msat (seeking 1050000000 msat) in paths. Last path loop did not find a new path. module="lightning::routing::router" line=3232 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0
2025-03-25T23:36:49.041134Z TRACE (node):(app-node-run-server):(req)(srv):(preflight-pay-invoice): ldk: Ignored 2 candidate hops due to insufficient value contribution, 0 due to path length limit, 0 due to CLTV delta limit, 0 due to previous payment failure, 0 due to htlc_minimum_msat limit, 0 to avoid overpaying, 0 due to maximum total fee limit. Total: 2 ignored candidates. module="lightning::routing::router" line=3255 user_pk=bbf6c8a9 user_idx=0 trace_id=AH0oacZBCk6jhsFN from=app method=POST url=/app/preflight_pay_invoice version=HTTP/2.0

I tracked the issue down to this block in router.rs:

if !prevented_redundant_path_selection {
    // If we weren't capped by hitting a liquidity limit on a channel in the path,
    // we'll probably end up picking the same path again on the next iteration.
    // Decrease the available liquidity of a hop in the middle of the path.
    let victim_candidate = &payment_path.hops[(payment_path.hops.len()) / 2].0.candidate;
    let exhausted = u64::max_value();
    log_trace!(logger,
        "Disabling route candidate {} for future path building iterations to avoid duplicates.",
        LoggedCandidateHop(victim_candidate));
    if let Some(scid) = victim_candidate.short_channel_id() {
        *used_liquidities.entry(CandidateHopId::Clear((scid, false))).or_default() = exhausted;
        *used_liquidities.entry(CandidateHopId::Clear((scid, true))).or_default() = exhausted;
    }
}

If I disable this block, the payment succeeds. In this case, payment_path.hops.len() is 2, so we disable payment_path.hops[1], which happens to be the Bob -> Charlie hop.

If there is only one path from a sender to a recipient, it seems wrong to disable any hop within it, since it prevents other MPP shards from finding any route at all. I understand the motivation for varying our paths for MPPs, but it shouldn't cost at the cost of routing reliability.

MaxFangX added a commit to lexe-app/rust-lightning that referenced this issue Mar 27, 2025
The original motivation for the block was to ensure MPPs vary their
paths a bit by disabling the 'central' hop in any found paths. However,
we are disabling this block, as it:

1) causes MPP routing failures if there is only one path from payer -> payee
2) forces MPPs to be routed across less optimal paths

Issue: lightningdevkit#3685

The one possible privacy concern is that any hops which forward multiple shards
of the same MPP can link those shards together, as each shard uses the same
payment hash (since LDK implements Simplified Multipath Payments, not Atomic
Multipath Payments). But this concern seems moot, as sending the payment over a
single shard leaks almost the same information anyway.

As far as I (and SOTA AI models) can tell, disabling this block doesn't violate
any correctness constraints or security concerns in surrounding code.

License: PolyForm Noncommercial License 1.0.0
MaxFangX added a commit to lexe-app/rust-lightning that referenced this issue Mar 27, 2025
The original motivation for the block was to ensure MPPs vary their
paths a bit by disabling the 'central' hop in any found paths. However,
we are disabling this block, as it:

1) causes MPP routing failures if there is only one path from payer -> payee
2) forces MPPs to be routed across less optimal paths

Issue: lightningdevkit#3685

The one possible privacy concern is that any hops which forward multiple shards
of the same MPP can link those shards together, as each shard uses the same
payment hash (since LDK implements Simplified Multipath Payments, not Atomic
Multipath Payments). But this concern seems moot, as sending the payment over a
single shard leaks almost the same information anyway.

As far as I (and SOTA AI models) can tell, disabling this block doesn't violate
any correctness constraints or security concerns in surrounding code.

License: PolyForm Noncommercial License 1.0.0
@valentinewallace
Copy link
Contributor

Odd, I'm unable to reproduce this on 0.0.125 with this test that results in ~identical logs up until the part where your logs disable a channel:

#[test]
fn xxx() {
	let mut cfg = test_default_channel_config();
	cfg.channel_handshake_config.max_inbound_htlc_value_in_flight_percent_of_channel = 100;
	let mut high_fee_cfg = cfg.clone();
	high_fee_cfg.channel_config.forwarding_fee_base_msat = 1_050_000;
	let chanmon_cfgs = create_chanmon_cfgs(3);
	let node_cfgs = create_node_cfgs(3, &chanmon_cfgs);
	let node_chanmgrs = create_node_chanmgrs(3, &node_cfgs, &[Some(cfg.clone()), Some(high_fee_cfg.clone()), Some(cfg.clone())]);
	let nodes = create_network(3, &node_cfgs, &node_chanmgrs);

	create_unannounced_chan_between_nodes_with_value(&nodes, 0, 1, 200_000, 0);
	create_unannounced_chan_between_nodes_with_value(&nodes, 0, 1, 300_000, 0);
	create_unannounced_chan_between_nodes_with_value(&nodes, 1, 2, 600_000, 0);

	let amt_msat = 350_000_000;
	let invoice = crate::ln::invoice_utils::create_invoice_from_channelmanager(
		&nodes[2].node, &nodes[2].keys_manager, &nodes[2].logger, Currency::BitcoinTestnet,
		Some(amt_msat), "".to_string(), 100000000, None
	).unwrap();

	let (payment_hash, recipient_onion_fields, mut route_params) =
		crate::ln::bolt11_payment::payment_parameters_from_invoice(&invoice).unwrap();
	route_params.max_total_routing_fee_msat = Some(u64::MAX);

	nodes[0].node.send_payment(
		payment_hash, recipient_onion_fields, PaymentId([42; 32]), route_params, Retry::Attempts(0)
	).unwrap();
	assert!(nodes[0].node.list_recent_payments().len() == 1);
	check_added_monitors(&nodes[0], 2); // one monitor update per MPP part
	nodes[0].node.get_and_clear_pending_msg_events();
}

To confirm, this is on 0.0.125?

@MaxFangX
Copy link
Contributor Author

This is on 0.1.1, but my understanding is this behavior was added years ago. @valentinewallace

I don't think it should affect the reproducibility, but we tested with intercept hints.

I'll run your code and see if I can get it to reproduce.

@valentinewallace
Copy link
Contributor

Ah okay, if it's 0.1.1 let me send you an updated test since the interfaces changed a bit, gimme a sec.

@valentinewallace
Copy link
Contributor

valentinewallace commented Mar 28, 2025

@MaxFangX check out this commit based on 0.1.1: valentinewallace@e66c746

@MaxFangX
Copy link
Contributor Author

I'm assuming the 0.1.1 version didn't reproduce either?

@valentinewallace
Copy link
Contributor

valentinewallace commented Mar 28, 2025

That's correct :/ and the logs still look the same until they diverge.

@MaxFangX
Copy link
Contributor Author

MaxFangX commented Mar 28, 2025

Edit: Setting manually_accept_inbound_channels: false, fixed the test :/ Will keep working on it.

I imported Lexe's UserConfigs and now the test is failing at the create_unannounced_chan_between_nodes_with_value(&nodes, 0, 1, 200_000, 0); step:

---- ln::payment_tests::xxx stdout ----
Using Block Connection Style: HighlyRedundantTransactionsFirstSkippingBlocks
node 0 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 0355f8d2238a322d16b602bd0ceaad5b01019fb055971eaadcc9b29226a4da6c23
node 1 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 027f921585f2ac0c7c70e36110adecfd8fd14b8a99bfb3d000a283fcac358fce88
node 0 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 02888f2e3df341d21240f769afd83938fdc1878356769aae64141880d810c61dce
node 2 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 027f921585f2ac0c7c70e36110adecfd8fd14b8a99bfb3d000a283fcac358fce88
node 1 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 02888f2e3df341d21240f769afd83938fdc1878356769aae64141880d810c61dce
node 2 DEBUG [lightning::ln::channelmanager:11699] Generating channel_reestablish events for 0355f8d2238a322d16b602bd0ceaad5b01019fb055971eaadcc9b29226a4da6c23
thread 'ln::payment_tests::xxx' panicked at lightning/src/ln/functional_test_utils.rs:1512:26:
assertion `left == right` failed
  left: 0
 right: 1
stack backtrace:
   0: rust_begin_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
   1: core::panicking::panic_fmt
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:367:5
   4: lightning::ln::functional_test_utils::create_unannounced_chan_between_nodes_with_value
             at ./src/ln/functional_test_utils.rs:798:4
   5: lightning::ln::payment_tests::xxx
             at ./src/ln/payment_tests.rs:4582:2
   6: lightning::ln::payment_tests::xxx::{{closure}}
             at ./src/ln/payment_tests.rs:4572:9
   7: core::ops::function::FnOnce::call_once
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ops/function.rs:250:5
   8: core::ops::function::FnOnce::call_once
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

This is my first time working with LDK's test utils, I don't quite understand what's going on here.

Code here: https://github.com/lexe-app/rust-lightning/tree/max/repro-central-route-issue (you can cherry pick the last commit)

@MaxFangX
Copy link
Contributor Author

I've uploaded the full logs from our integration test which reliably reproduces this error here. The stdout output includes the results of calls to list_channels just prior to routing (sending) the payment.

https://gist.github.com/MaxFangX/3bfeb8328d7b55d624330965c7d56944

I've also updated the repro branch to include the exact commits in Lexe's LDK fork, including an extra commit which reverts our workaround for this issue.

Appreciate the help @valentinewallace, we already have a workaround for this so there's no rush.

@valentinewallace valentinewallace linked a pull request Apr 3, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants