[Bug] Freezes and does not recover - no error

[stngrz]

Describe the bug

My app has ros2 dds topics communicating locally (~90 topics @ max) on the same host (A) and communicating with a cloud container, the container has about 10 topics (continuous), host (B). The topics on host A originate from multiple dockers containers, all using the same dds xml (the default dds xml for zenoh referenced here https://github.com/eclipse-zenoh/zenoh-plugin-ros2dds?tab=readme-ov-file#usage)

While in a steady state zenoh-bridge-ros2dds sends messages cleanly between hosts A to B with no issue. However, during the execution of my app I transition to various states/modes where the volume of topics can fluctuate greatly or not at all. This is where the issue emerges.

About 90% of these transitions (either many topic add/remove or none) create a mode where, on host A, the bridge freezes. I define freeze by not exiting, showing no errors - it just hangs. I don't see any corresponding events in logs on host B - as if B expects A to be connected and healthy. Otherwise it performs nominally - the issue is just during these stage/mode transitions. When i kill a docker running many topics i see this error as well (aka not using the apps change stage/mode feature).

To recover i simply stop zenoh and restart it - the system recovers and picks up using the data from Host B with no issue.

I've tried to review rosout to see if there is a difference in messages sent vs received on A and B but found nothing promising in the few samples I collected. I hoped it would reveal a problematic message being sent by my app on A - but nothing.

I expect that between app stages/mode transitions or add/remove of N topics the bridge continues to send and receive data or that it provide errors when its not sending/receiving data.

This occurs with and with out using custom zenoh json5 configs.

in the attached debug logs I've removed the customer name and replaced it with 'my-company'

Host A logs:
debug_start-wait5s-freeze-wait5s.txt
debug_start-freeze-wait10s.txt

To reproduce

1. Start zenoh bridge
2. add/remove topics / kill docker with many topics / stage change (sorry this is ill defined)
3. bridge freezes (no more updates from host B topics on A) with no crash or log output on host A

System info

• Platform: linux/aarch64
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.4 LTS
  Release: 22.04
  Codename: jammy

• CPU:
  Architecture: aarch64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
  CPU(s): 12
  On-line CPU(s) list: 0-11
  Vendor ID: ARM
  Model name: Cortex-A78AE

• Zenoh:
  zenoh-bridge-ros2dds v1.3.0 (also occurred on 1.21.1)

[stngrz]

I ran another test
2 local hosts(A,B) to 1 cloud container (C)

1. host A triggers a freeze
2. both A and B see the freeze - asserted via ros2 topic echo of topic from C no longer getting data on A,B tho still streaming on C
3. if reset zenoh on C then B resumes but not A - must reset on A for A to resume
4. if reset zenoh on A then both A and B resume
5. did not try reset on B when frozen

debug from bridge on C during just before and during freeze event

2host_freeze.txt

[stngrz]

Another data point, expecting that Local Host does not see zenoh pub if Cloud Host zenoh is not subbed

invalid state found
Cloud Host
pub = simulation
sub = none
Local Host
pub = zenoh_bridge
sub = perception_manager

[stngrz]

Trending, 2 instances where comms error in app correlates in time to zenoh WARN msg

[2m2025-04-03T03:27:31.740956Z[0m [33m WARN[0m net-0 ThreadId(04) [2mzenoh::net::routing::dispatcher::queries[0m[2m:[0m Didn't receive final reply Face{1, 3286a6f1a6ef7d83078bbe81e235feb}:1979 for Face{8, f42afe7db9b0a8ba91297963ae9c9080}:1438: Timeout(5s)!

i had a config on that end that is since been removed
said config:
{
mode: "peer",
listen: {
endpoints: ["tcp/0.0.0.0:7447"],
timeout_ms: 30000
},
transport: {
unicast: {
open_timeout: 30000,
accept_timeout: 30000,
max_links: 1
},
link: {
tx: {
lease: 30000,
keep_alive: 4,
batch_size: 65535,
queue: {
size: {
control: 4,
real_time: 4,
interactive_high: 4,
interactive_low: 4,
data_high: 4,
data: 4,
data_low: 4,
background: 4
},
congestion_control: {
block: {
wait_before_close: 30000000
}
}
}
}
}
}
}

[edu-rossrobotics]

Hi, did you manage to fix it? I have a similar issue where on my robot I launch a bunch of different nodes, when the first one comes up, everything is working but when I launch the node that controls the robot movements (it basically involves launching ros2control packages) I got the same issue:

2025-04-29T09:01:00.093590Z  INFO tokio-runtime-worker ThreadId(03) zenoh_plugin_ros2dds::routes_mgr: Route Service Client (ROS:/controller_manager/list_controllers <-> Zenoh:controller_manager/list_controllers) removed
2025-04-29T09:01:00.158524Z  INFO tokio-runtime-worker ThreadId(02) zenoh_plugin_ros2dds::discovered_entities: Undiscovered ROS Node /spawner_joint_state_broadcaster
2025-04-29T09:01:01.200437Z  INFO tokio-runtime-worker ThreadId(02) zenoh_plugin_ros2dds::discovered_entities: Undiscovered ROS Node /spawner_mk4_base_controller
2025-04-29T09:01:02.268224Z  INFO tokio-runtime-worker ThreadId(04) zenoh_plugin_ros2dds::discovered_entities: Undiscovered ROS Node /controller_manager
2025-04-29T09:01:05.381858Z  WARN                net-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{1, ebdefef004106e20966a87c2cb6f460b}:79 Didn't receive final reply for query Face{2, afa6a071508ccc64ac0aa86eaf70f153}:79: Timeout(5s)!

And all the communication stops working. But, just restarting the zenoh bit makes it work so I assume the issue is during the launch process.

[maxvijsseldijk]

We are having the same issue and have tested various versions of the zenoh bridge plugin for our setup. Here is my analysis:

The zenoh-bridge-dds plugin experiences communication failures when certain ROS2 nodes are stopped or undiscovered. The bridge stops functioning without reporting errors, indicating potential issues with node disconnect handling.

Steps to Reproduce

Scenario 1: Node Stopping

1. Set up zenoh-bridge-dds on both client and robot (versions 1.0.4+ on both)
2. Launch ROS2 system with a docker container which sets up various ROS2 topic publishers
3. Establish zenoh bridge communication between client and robot
4. Stop the docker container which publishes ROS2 topics
5. Expected: Bridge continues functioning
6. Actual: Bridge communication halts without error messages

Version Compatibility Testing

Client Version	Robot Version	Status
0.11.0	0.11.0	✅ Works
1.0.0	1.0.0	✅ Works
1.0.1	1.0.1	✅ Works
1.0.0	1.0.1	✅ Works
1.0.0	1.0.2	✅ Works
1.0.0	1.0.3	✅ Works
1.0.0	1.0.4	❌ No connection
1.0.4	1.0.4	❌ No connection
1.0.4	1.1.0	⚠️ Connects but crashes
1.1.0	1.0.0	⚠️ Works initially, slow, eventually crashes
1.0.4 till 1.5.1	1.0.3	⚠️ Connects but crashes
1.0.3	1.0.4 till 1.5.1	⚠️ Connects but crashes
1.4.0	1.4.0	⚠️ Connects but crashes
1.4.0	1.5.1	⚠️ Connects but crashes
1.5.1	1.4.0	⚠️ Connects but crashes
1.5.1	1.5.1	⚠️ Connects but crashes

Workaround

A solution for you would be to rollback to version 1.0.3 as that worked for us. I have not investigated what exactly causes the issue in the 1.0.4+ versions, but maybe the developers could look into this issues.