Large CPU spike in rmw_zenohd when a node restarts on another host

[oysstu]

Operating System:

Ubuntu 24.04

ROS version or commit hash:

Kilted

RMW Configuration (if applicable):

Two hosts with one zenohd router running on each with DEFAULT_RMW_ZENOH_ROUTER_CONFIG.json5 except an connect: endpoints TCP entry between the routers.

Steps to reproduce issue

Restarting a node (in this case an ublox driver) on host B causes a significant spike in rmw_zenohd CPU utilization on host A. The ublox node sets up a few topics of course, but I'm surprised that another host can be this severely affected by a node restarting on another host. See the following btop output. Every time the node on host B restarts, all cores on this CPU are loaded fully for about a second. It's the rmw_zenohd process that causes the CPU spike.

Thoughts? Are there any configuration options that may help alleviate this?

Expected behavior

Host B should not affect host A by simply restarting a node.

Actual behavior

100% CPU usage across four cores for about a second. This severely impacts other running processes.

Additional info

Relevant Zenoh debug output from Host A in attached file. Should be exactly when the spike happens.

host_a.txt

[oysstu]

There's also a spike in CPU usage when restarting the same process on localhost. It's smaller in magnitude though, but I'm not sure if that is due to it sharing CPU utilization with the process restarting or if rmw_zenohd actually uses less CPU.

Also, there is a spike in activity on the loopback interface of host A when the process on host B restarts. Very weird behavior. It spikes from ~150KiB/s to about 1500KiB/s. There is no bag recorder running there, and no other nodes should subscribe to the topics created by host B.