Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VPP-1719] Policy based ipsec failing. #3182

Open
vvalderrv opened this issue Feb 2, 2025 · 11 comments
Open

[VPP-1719] Policy based ipsec failing. #3182

vvalderrv opened this issue Feb 2, 2025 · 11 comments

Comments

@vvalderrv
Copy link
Contributor

Description

Recent changes in master (post stable/1904) seem to have broken policy based ipsec (i.e., no explicit tunnel interface configured). This configuration/test case works in 1904.

A ping is done from the host side of node1 (source ip will be 192.168.30.166 routed to VPP interface 192.168.30.66) to 192.168.32.67 (which should be encapsulated in ipsec to node2.

The backtrace looks like this (with some debug printfs I added, seems to maybe be looping?):

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:223: ipsec_output_inline: START: packet received current_data 0 -pre_data_size -128

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:231: ipsec_output_inline: packet received from 192.168.30.166 to 192.168.32.67

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:234: ipsec_output_inline: last_sw_if_index 4294967295 sw_if_index0 4

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:245: ipsec_output_inline: p 0

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:281: packet received from 192.168.30.166 to 192.168.32.67 port 33505

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:283: sw_if_index0 4 spd_index0 0 spd_id 1

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:399: ipsec_output_inline: END next_node_index dpdk-esp4-encrypt (197)

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:223: ipsec_output_inline: START: packet received current_data -44 -pre_data_size -128

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:231: ipsec_output_inline: packet received from 192.168.25.66 to 192.168.25.67

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:234: ipsec_output_inline: last_sw_if_index 4294967295 sw_if_index0 4

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:245: ipsec_output_inline: p 0

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:281: packet received from 192.168.25.66 to 192.168.25.67 port 2030

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:283: sw_if_index0 4 spd_index0 0 spd_id 1

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:399: ipsec_output_inline: END next_node_index dpdk-esp4-encrypt (197)

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:223: ipsec_output_inline: START: packet received current_data -88 -pre_data_size -128

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:231: ipsec_output_inline: packet received from 192.168.25.66 to 192.168.25.67

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:234: ipsec_output_inline: last_sw_if_index 4294967295 sw_if_index0 4

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:245: ipsec_output_inline: p 0

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:281: packet received from 192.168.25.66 to 192.168.25.67 port 2030

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:283: sw_if_index0 4 spd_index0 0 spd_id 1

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:399: ipsec_output_inline: END next_node_index dpdk-esp4-encrypt (197)

Jul 09 13:46:02 dpdk2 vnet[2968]: ipsec_output_inline:223: ipsec_output_inline: START: packet received current_data -132 -pre_data_size -128

Jul 09 13:46:02 dpdk2 vnet[2968]: /home/chopps/net/w/vpp/src/vlib/buffer.h:232 (vlib_buffer_get_current) assertion `(signed) b->current_data >= (signed) -VLIB_BUFFER_PRE_DATA_SIZE' fails

Jul 09 13:46:02 dpdk2 vnet[2968]: received signal SIGABRT, PC 0x7fcaf32cee97

Jul 09 13:46:02 dpdk2 vnet[2968]: #0 0x00007fcaf3c98c73 unix_signal_handler + 0x25b

Jul 09 13:46:02 dpdk2 vnet[2968]: #1 0x00007fcaf39a6890 0x7fcaf39a6890

Jul 09 13:46:02 dpdk2 vnet[2968]: #2 0x00007fcaf32cee97 gsignal + 0xc7

Jul 09 13:46:02 dpdk2 vnet[2968]: #3 0x00007fcaf32d0801 abort + 0x141

Jul 09 13:46:02 dpdk2 vnet[2968]: #4 0x00005647559dd05d 0x5647559dd05d

Jul 09 13:46:02 dpdk2 vnet[2968]: #5 0x00007fcaf36b4d53 debugger + 0x9

Jul 09 13:46:02 dpdk2 vnet[2968]: #6 0x00007fcaf36b5122 _clib_error + 0x2c0

Jul 09 13:46:02 dpdk2 vnet[2968]: #7 0x00007fcaf48ad611 vlib_buffer_get_current + 0x56

Jul 09 13:46:02 dpdk2 vnet[2968]: #8 0x00007fcaf48aec70 ipsec_output_inline + 0x221

Jul 09 13:46:02 dpdk2 vnet[2968]: #9 0x00007fcaf48af995 ipsec4_output_node_fn + 0x2d

Jul 09 13:46:02 dpdk2 vnet[2968]: #10 0x00007fcaf3c3138c dispatch_node + 0x328

Jul 09 13:46:02 dpdk2 vnet[2968]: #11 0x00007fcaf3c31b44 dispatch_pending_node + 0x363

Jul 09 13:46:02 dpdk2 vnet[2968]: #12 0x00007fcaf3c337c4 vlib_main_or_worker_loop + 0xa31

Jul 09 13:46:02 dpdk2 vnet[2968]: #13 0x00007fcaf3c34027 vlib_main_loop + 0x1d

Jul 09 13:46:02 dpdk2 vnet[2968]: #14 0x00007fcaf3c34c3e vlib_main + 0x931

Assignee

Neale Ranns

Reporter

Christian Hopps

Comments

  • nranns (Thu, 11 Jul 2019 11:43:46 +0000):

    i've reverted the two changes that broke the endian comparisons.

https://gerrit.fd.io/r/c/20602/

https://gerrit.fd.io/r/c/20603/

these will need cherry-picking to 19.04.

i still have to do the change for getting the default any any ranges.

  • chopps (Thu, 11 Jul 2019 04:43:55 +0000): Right, just ran across this in 19.04 too.
  • nranns (Wed, 10 Jul 2019 13:41:22 +0000): for that dodgy policy matching: git revert 231c469
  • nranns (Wed, 10 Jul 2019 13:03:44 +0000): good point. that's another bug...
  • chopps (Wed, 10 Jul 2019 12:44:42 +0000): 2) shouldn't the ipsec encap packet have an outer IP of 192.168.25.67 and not match that policy?
  • nranns (Wed, 10 Jul 2019 12:30:02 +0000): ok

there are two reasons we end up in this mess. Firstly:

DBGvpp# sh ipsec spd

sh ipsec spd

spd 1

ip4-outbound:

[0] priority 100 action bypass type ip4-outbound protocol IPSEC_ESP

 local addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535

 remote addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535

 packets 0 bytes 0

[2] priority 10 action protect type ip4-outbound protocol any sa 20

 local addr range 192.168.30.0 - 192.168.30.255 port range 0 - 65535

 remote addr range 192.168.32.0 - 192.168.32.255 port range 0 - 65535

 packets 1 bytes 100

ip6-outbound:

ip4-inbound-protect:

[3] priority 10 action protect type ip4-inbound-protect protocol any sa 30

 local addr range 192.168.30.0 - 192.168.30.255 port range 0 - 65535

 remote addr range 192.168.32.0 - 192.168.32.255 port range 0 - 65535

 packets 0 bytes 0

ip6-inbound-protect:

ip4-inbound-bypass:

[1] priority 100 action bypass type ip4-inbound-bypass protocol IPSEC_ESP

 local addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535

 remote addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535

 packets 0 bytes 0

ip6-inbound-bypass:

DBGvpp#

note the IP ranges for your match any policies... not so good. so a work around is to specify them on the CLI when you add these policies. I'll update the CLI code though to do default the end of the range to the all ones address.

Secondly, as you suspected, the packet goes into a loop. this is because it keeps matching policy [2] which has protocol any and the encap addresses. I need to fix this recursive policy application. I'm considering my options

  • chopps (Wed, 10 Jul 2019 12:11:00 +0000): Yes, I tried to mark 1720 as a duplicate.
  • nranns (Wed, 10 Jul 2019 11:57:33 +0000): are 1719 and 1720 the same issue?
  • chopps (Tue, 9 Jul 2019 21:36:11 +0000): FWIW, if I force using non-dpdk backend it hit's the same assert.

[21:29:00 dpdk2:~]$ sudo vppctl show ipsec backend

IPsec AH backends available:

{{ Name Index Active}}

{{ crypto engine backend 0 yes}}

IPsec ESP backends available:

{{ Name Index Active}}

{{ crypto engine backend 0 yes}}

{{ dpdk backend 1 no}}

Jul 09 21:34:18 dpdk2 vnet[3959]: /home/chopps/net/w/vpp/src/vlib/buffer.h:232 (vlib_buffer_get_current) assertion `(signed) b->current_data >= (signed) -VLIB_BUFFER_PRE_DATA_SIZE' fails

Jul 09 21:34:18 dpdk2 vnet[3959]: received signal SIGABRT, PC 0x7f7758ef6e97

Jul 09 21:34:18 dpdk2 vnet[3959]: #0 0x00007f77598c0c73 unix_signal_handler + 0x25b

Jul 09 21:34:18 dpdk2 vnet[3959]: #1 0x00007f77595ce890 0x7f77595ce890

Jul 09 21:34:18 dpdk2 vnet[3959]: #2 0x00007f7758ef6e97 gsignal + 0xc7

Jul 09 21:34:18 dpdk2 vnet[3959]: #3 0x00007f7758ef8801 abort + 0x141

Jul 09 21:34:18 dpdk2 vnet[3959]: #4 0x00005589328f505d 0x5589328f505d

Jul 09 21:34:18 dpdk2 vnet[3959]: #5 0x00007f77592dcd13 debugger + 0x9

Jul 09 21:34:18 dpdk2 vnet[3959]: #6 0x00007f77592dd0e2 _clib_error + 0x2c0

Jul 09 21:34:18 dpdk2 vnet[3959]: #7 0x00007f775a4d272a vlib_buffer_get_current + 0x56

Jul 09 21:34:18 dpdk2 vnet[3959]: #8 0x00007f775a4d3d22 ipsec_output_inline + 0x1ba

Jul 09 21:34:18 dpdk2 vnet[3959]: #9 0x00007f775a4d471c ipsec4_output_node_fn + 0x2d

Jul 09 21:34:18 dpdk2 vnet[3959]: #10 0x00007f775985938c dispatch_node + 0x328

Jul 09 21:34:18 dpdk2 vnet[3959]: #11 0x00007f7759859b44 dispatch_pending_node + 0x363

Jul 09 21:34:18 dpdk2 vnet[3959]: #12 0x00007f775985b7c4 vlib_main_or_worker_loop + 0xa31

Jul 09 21:34:18 dpdk2 vnet[3959]: #13 0x00007f775985c027 vlib_main_loop + 0x1d

Jul 09 21:34:18 dpdk2 vnet[3959]: #14 0x00007f775985cc3e vlib_main + 0x931

  • chopps (Tue, 9 Jul 2019 21:30:35 +0000):

    $ sudo vppctl show ipsec backend

IPsec AH backends available:

{{ Name Index Active}}

{{ crypto engine backend 0 yes}}

IPsec ESP backends available:

{{ Name Index Active}}

{{ crypto engine backend 0 no}}

{{ dpdk backend 1 yes}}

  • nranns (Tue, 9 Jul 2019 16:08:57 +0000): Hi Chris,

 

which IPSec backend are you using?

sh ipsec backend

Original issue: https://jira.fd.io/browse/VPP-1719
@vvalderrv
Copy link
Contributor Author

i've reverted the two changes that broke the endian comparisons.
https://gerrit.fd.io/r/c/20602/
https://gerrit.fd.io/r/c/20603/
these will need cherry-picking to 19.04.

i still have to do the change for getting the default any any ranges.

@vvalderrv
Copy link
Contributor Author

Right, just ran across this in 19.04 too.

 

@vvalderrv
Copy link
Contributor Author

for that dodgy policy matching: git revert 231c469

@vvalderrv
Copy link
Contributor Author

good point. that's another bug...

@vvalderrv
Copy link
Contributor Author

2) shouldn't the ipsec encap packet have an outer IP of 192.168.25.67 and not match that policy?

@vvalderrv
Copy link
Contributor Author

ok

there are two reasons we end up in this mess. Firstly:

DBGvpp# sh ipsec spd
sh ipsec spd
spd 1
ip4-outbound:
[0] priority 100 action bypass type ip4-outbound protocol IPSEC_ESP
local addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535
remote addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535
packets 0 bytes 0
[2] priority 10 action protect type ip4-outbound protocol any sa 20
local addr range 192.168.30.0 - 192.168.30.255 port range 0 - 65535
remote addr range 192.168.32.0 - 192.168.32.255 port range 0 - 65535
packets 1 bytes 100
ip6-outbound:
ip4-inbound-protect:
[3] priority 10 action protect type ip4-inbound-protect protocol any sa 30
local addr range 192.168.30.0 - 192.168.30.255 port range 0 - 65535
remote addr range 192.168.32.0 - 192.168.32.255 port range 0 - 65535
packets 0 bytes 0
ip6-inbound-protect:
ip4-inbound-bypass:
[1] priority 100 action bypass type ip4-inbound-bypass protocol IPSEC_ESP
local addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535
remote addr range 0.0.0.0 - 0.0.0.0 port range 0 - 65535
packets 0 bytes 0
ip6-inbound-bypass:
DBGvpp#

note the IP ranges for your match any policies... not so good. so a work around is to specify them on the CLI when you add these policies. I'll update the CLI code though to do default the end of the range to the all ones address.

Secondly, as you suspected, the packet goes into a loop. this is because it keeps matching policy [2] which has protocol any and the encap addresses. I need to fix this recursive policy application. I'm considering my options

@vvalderrv
Copy link
Contributor Author

Yes, I tried to mark 1720 as a duplicate.

@vvalderrv
Copy link
Contributor Author

are 1719 and 1720 the same issue?

@vvalderrv
Copy link
Contributor Author

FWIW, if I force using non-dpdk backend it hit's the same assert.

[21:29:00 dpdk2:~]$ sudo vppctl show ipsec backend
IPsec AH backends available:
{{ Name Index Active}}
{{ crypto engine backend 0 yes}}
IPsec ESP backends available:
{{ Name Index Active}}
{{ crypto engine backend 0 yes}}
{{ dpdk backend 1 no}}

Jul 09 21:34:18 dpdk2 vnet[3959]: /home/chopps/net/w/vpp/src/vlib/buffer.h:232 (vlib_buffer_get_current) assertion `(signed) b->current_data >= (signed) -VLIB_BUFFER_PRE_DATA_SIZE' fails
Jul 09 21:34:18 dpdk2 vnet[3959]: received signal SIGABRT, PC 0x7f7758ef6e97
Jul 09 21:34:18 dpdk2 vnet[3959]: #0 0x00007f77598c0c73 unix_signal_handler + 0x25b
Jul 09 21:34:18 dpdk2 vnet[3959]: #1 0x00007f77595ce890 0x7f77595ce890
Jul 09 21:34:18 dpdk2 vnet[3959]: #2 0x00007f7758ef6e97 gsignal + 0xc7
Jul 09 21:34:18 dpdk2 vnet[3959]: #3 0x00007f7758ef8801 abort + 0x141
Jul 09 21:34:18 dpdk2 vnet[3959]: #4 0x00005589328f505d 0x5589328f505d
Jul 09 21:34:18 dpdk2 vnet[3959]: #5 0x00007f77592dcd13 debugger + 0x9
Jul 09 21:34:18 dpdk2 vnet[3959]: #6 0x00007f77592dd0e2 _clib_error + 0x2c0
Jul 09 21:34:18 dpdk2 vnet[3959]: #7 0x00007f775a4d272a vlib_buffer_get_current + 0x56
Jul 09 21:34:18 dpdk2 vnet[3959]: #8 0x00007f775a4d3d22 ipsec_output_inline + 0x1ba
Jul 09 21:34:18 dpdk2 vnet[3959]: #9 0x00007f775a4d471c ipsec4_output_node_fn + 0x2d
Jul 09 21:34:18 dpdk2 vnet[3959]: #10 0x00007f775985938c dispatch_node + 0x328
Jul 09 21:34:18 dpdk2 vnet[3959]: #11 0x00007f7759859b44 dispatch_pending_node + 0x363
Jul 09 21:34:18 dpdk2 vnet[3959]: #12 0x00007f775985b7c4 vlib_main_or_worker_loop + 0xa31
Jul 09 21:34:18 dpdk2 vnet[3959]: #13 0x00007f775985c027 vlib_main_loop + 0x1d
Jul 09 21:34:18 dpdk2 vnet[3959]: #14 0x00007f775985cc3e vlib_main + 0x931

 

@vvalderrv
Copy link
Contributor Author

$ sudo vppctl show ipsec backend
IPsec AH backends available:
{{ Name Index Active}}
{{ crypto engine backend 0 yes}}
IPsec ESP backends available:
{{ Name Index Active}}
{{ crypto engine backend 0 no}}
{{ dpdk backend 1 yes}}

@vvalderrv
Copy link
Contributor Author

Hi Chris,

 

which IPSec backend are you using?

sh ipsec backend

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vvalderrv