Service call reports z_reply_is_ok returned False Reason: Timeout

[mini-1235]

Generated by Generative AI

No

Operating System:

Linux XPS-15-9520 6.8.0-78-generic #78-Ubuntu SMP PREEMPT_DYNAMIC Tue Aug 12 11:34:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

ROS version or commit hash:

Rolling binary http://packages.ros.org/ros2/ubuntu noble/main amd64 ros-rolling-rmw-zenoh-cpp amd64 0.9.0-1noble.20250822.215306

RMW implementation (if applicable):

zenoh

RMW Configuration (if applicable):

No response

Client library (if applicable):

No response

'ros2 doctor --report' output

ros2 doctor --report

<COPY OUTPUT HERE>

Steps to reproduce issue

1. run nav2 with

ros2 launch nav2_bringup tb3_simulation_launch.py headless:=0

2. wait until initial transform timeout (around 1 minute) and see the error from zenoh

Expected behavior

Receive a response of false from the change_state service server, and nav2 abort bringup

[ERROR 1757400210.592444579] [global_costmap.global_costmap]: Failed to activate global_costmap because transform from base_link to map did not become available before timeout (on_activate() at /root/nav2_ws/src/navigation2/nav2_costmap_2d/src/costmap_2d_ros.cpp:304)
[component_container_isolated-7] [ERROR 1757400210.593177984] [lifecycle_manager_navigation]: Failed to change state for node: planner_server (changeStateForNode() at /root/nav2_ws/src/navigation2/nav2_lifecycle_manager/src/lifecycle_manager.cpp:268)
[component_container_isolated-7] [ERROR 1757400210.593291041] [lifecycle_manager_navigation]: Failed to bring up all requested nodes. Aborting bringup. (startup() at /root/nav2_ws/src/navigation2/nav2_lifecycle_manager/src/lifecycle_manager.cpp:335)

Actual behavior

The change_state service does not return false, instead I see a couple of errors from the zenoh side

[component_container_isolated-8] [ERROR 1757390757.072415008] [rmw_zenoh_cpp]: z_reply_is_ok returned False Reason: Timeout (add_new_query() at ./src/detail/rmw_service_data.cpp:252)
[component_container_isolated-8] 2025-09-09T04:05:58.074188Z  WARN ThreadId(12) zenoh::api::session: Received ReplyData for unknown Query: 77
[component_container_isolated-8] 2025-09-09T04:05:58.074224Z  WARN ThreadId(12) zenoh::api::session: Received ResponseFinal for unknown Request: 77
[component_container_isolated-8] [ERROR 1757390758.431290576] [rmw_zenoh_cpp]: Query queue depth of 10 reached, discarding oldest Query for service for 0/lifecycle_manager_navigation/is_active/std_srvs::srv::dds_::Trigger_/RIHS01_eeff2cd6fa5ad9d27cdf4dec64818317839b62f212a91e6b5304b634b2062c5f (add_new_query() at ./src/detail/rmw_service_data.cpp:252)

I believe it hangs over there and never receive the result from the service server side

Additional information

I was trying to setup rmw_zenoh for Nav2 in this PR ros-navigation/navigation2#5516, one of the system tests is failing here https://app.circleci.com/pipelines/github/ros-navigation/navigation2/16036/workflows/7e2c6499-4b74-4e47-9c88-cf5953de2fd1/jobs/46834. Note that this test passes on fastrtps / cyclonedds, and only fails on zenoh.

I will illustrate the working flow here:

1. After we launch the tb3 simulation, we are changing the states for the lifecycle nodes, the planner server, when calling on_activate, https://github.com/ros-navigation/navigation2/blob/ab4aaf863aea6a7d1e67e0c2ad536ec1f0c8ccf7/nav2_planner/src/planner_server.cpp#L180 will activate global costmap. However, for global costmap to be activated we typically need a transform from map -> odom -> robot_base_frame, which is unavailable unless we provide an initial pose.
2. Suppose we never provide the initial pose, then costmap ros should fail to activate when it reaches initial transform timeout, https://github.com/ros-navigation/navigation2/blob/ab4aaf863aea6a7d1e67e0c2ad536ec1f0c8ccf7/nav2_costmap_2d/src/costmap_2d_ros.cpp#L303-L310, planner will also return failure here https://github.com/ros-navigation/navigation2/blob/ab4aaf863aea6a7d1e67e0c2ad536ec1f0c8ccf7/nav2_planner/src/planner_server.cpp#L181-L183 and fail to change state
3. The lifecycle manager, who was trying to change planner server's state earlier https://github.com/ros-navigation/navigation2/blob/ab4aaf863aea6a7d1e67e0c2ad536ec1f0c8ccf7/nav2_lifecycle_manager/src/lifecycle_manager.cpp#L264-L269, should say "failed to change node for planner server" after

!node_map_[node_name]->change_state(transition, std::chrono::milliseconds(-1),
      service_timeout_

receiving the response(false) from service server. However, it seems that we are unable to get any response from the service server when running with rmw zenoh

[JEnoch]

rmw_zenoh is by default configured with a timeout of 60 seconds for queries (i.e. request to a Service).
According your log z_reply_is_ok returned False Reason: Timeout: this timeout is triggered, leading to just return from the callback and to not push any reply to rcl layer. See here: https://github.com/ros2/rmw_zenoh/blob/bc6969ea3373f770ba0fc12de6613bd8aa9e8df2/rmw_zenoh_cpp/src/detail/rmw_client_data.cpp#L413-420.

My understanding of RMW/RCL design is that it's not up to the RMW layer to send timeout error to the RCL layer. Actually, in rmw.h interface, the rmw_take_response() operation cannot return a timeout message, but only a valid and types ros_response:
https://github.com/ros2/rmw/blob/db761272b97dff7ffced709dc2b33236af0aabbd/rmw/include/rmw/rmw.h#L2092
Therefore, the RCL layer manages an application timeout when rclcpp::spin_until_future_complete() is called with a timeout value: https://github.com/ros2/rclcpp/blob/364419589207059e3ef43e875134b0762d5222d2/rclcpp/include/rclcpp/client.hpp#L598

I believe that's exactly what Nav2 is doing in ServiceClient::invoke() here:
https://github.com/ros-navigation/navigation2/blob/8aea1a895f23ca505da3629819b6d16f4d126a0f/nav2_ros_common/include/nav2_ros_common/service_client.hpp#L109
However in your case the timeout is set to std::chrono::milliseconds(-1), meaning infinite time. Thus I think the lifecycle manager remains blocked indefinitely because the service server didn't manage to send a reply before the 60 seconds queries_default_timeout.

Could you try to increase this config value to check if it make the test to pass ?

[mini-1235]

Thanks for the reply, I tried decreasing the initial_transform_timeout in Nav2 and increasing the queries_default_timeout in zenoh, both approaches allow the test to pass.

However, I don’t think these fully address the underlying issue. I also tried reverting the changes in #685, since it may not properly handle cases where the requested timeout exceeds queries_default_timeout. Would it make sense to revert the PR and find a better way to deal with the performance issues?

[JEnoch]

I believe the root of the issue is that in current DDS-oriented design of RCL/RWM interactions it's assumed that sending a Request is stateless for the RMW layer - which correct for DDS implementations. Therefore, the RCL doesn't pass any timeout to the RMW layer, and doesn't call it either on remove_pending_request().

Zenoh implementation of queries/reply is not stateless, for both the caller and the infrastructure (i.e. the routers routing the request). Thus it requires a timeout to eventually cleanup the request-related states.

Reverting #685 will set MAX_UINT64 as timeout for all the service queries, leading to possible memory leaks in the client, but also in the router(s). We've noticed this with some packages that periodically sends is_alive requests that are not replied until some point.
While the client well calls remove_pending_request(), this doesn't cleanup the Zenoh queries before the MAX_UINT64 timeout occurs.

So we could increase the queries_default_timeout in the default rmw_zenoh config files. But to which value ?
I think the ROS developper knows the maximum time a Service call could take in their system and should set queries_default_timeout accordingly.
If you think that Nav2 needs a "reasonably" higher than 60 seconds, we could increase queries_default_timeout in the default configuration.

[roncapat]

Hello, I'm getting the same error:

[rmw_zenoh_cpp]: z_reply_is_ok returned False Reason: Timeout

even when I'm not launching nav2.
The robot seems to be unaffected, but this message is printed at high frequency.
Can you confirm tihs error can be originated only by a service call?
How to track down (eg. by some clever log level setting / toggle which I don't know about in zenoh) the name of the offending service?

[JEnoch]

Can you confirm tihs error can be originated only by a service call?

Yes, by a service call, including actions' send_goal and cancel_goal.
That's the only place this log occurs:
https://github.com/search?q=repo%3Aros2%2Frmw_zenoh%20%22z_reply_is_ok%20returned%22&type=code

How to track down (eg. by some clever log level setting / toggle which I don't know about in zenoh) the name of the offending service?

If you set RUST_LOG=debug you might see just before some Zenoh logs about the key expression of the query in timeout.
But that's certainly a good idea to add the service name in this rmw_zenoh log ! I opened #798 for this.

[mini-1235]

I believe the root of the issue is that in current DDS-oriented design of RCL/RWM interactions it's assumed that sending a Request is stateless for the RMW layer - which correct for DDS implementations. Therefore, the RCL doesn't pass any timeout to the RMW layer, and doesn't call it either on remove_pending_request().

Zenoh implementation of queries/reply is not stateless, for both the caller and the infrastructure (i.e. the routers routing the request). Thus it requires a timeout to eventually cleanup the request-related states.

Reverting #685 will set MAX_UINT64 as timeout for all the service queries, leading to possible memory leaks in the client, but also in the router(s). We've noticed this with some packages that periodically sends is_alive requests that are not replied until some point. While the client well calls remove_pending_request(), this doesn't cleanup the Zenoh queries before the MAX_UINT64 timeout occurs.

Ok, I can understand that.

However, I don’t think increasing the default timeout in Zenoh or overriding it in Nav2’s repo is the right long-term solution.

I think the ROS developper knows the maximum time a Service call could take in their system and should set queries_default_timeout accordingly.
If you think that Nav2 needs a "reasonably" higher than 60 seconds, we could increase queries_default_timeout in the default configuration.

IMO, this really depends on the user’s configuration. For example, in our setup we have a localization module that may take several minutes before publishing the map -> odom transform. In such a case, we need a longer initial_transform_timeout and queries_default_timeout. While I could override the Zenoh config in my own application, this introduces two problems:

• It creates inconsistency with other RMW implementations.
• It is not documented in the ros2 repo(?) that a service call cannot exceed queries_default_timeout when using Zenoh. https://docs.ros.org/en/rolling/Installation/RMW-Implementations/Non-DDS-Implementations/Working-with-Zenoh.html

New users trying to use Nav2 with Zenoh might find this confusing if they haven’t learned the underlying Zenoh concepts.

For these reasons, I prefer to leave this ticket open until we can find a solution that can handle the performance/memory leak issues and our use case here

Hope that makes sense

[JEnoch]

@mini-1235 #820 should be a workaround for this issue, waiting for a definitive one addressing ros2/rmw#404.

However, I believe nav2 could also be improved not using any infinite timeout. I mean in this code:

  if (!node_map_[node_name]->change_state(transition, std::chrono::milliseconds(-1),
      service_timeout_) ||
    !(node_map_[node_name]->get_state(service_timeout_) == transition_state_map_[transition]))
  {
    RCLCPP_ERROR(get_logger(), "Failed to change state for node: %s", node_name.c_str());
    return false;
  }

Why not using service_timeout_ for both transition_timeout and wait_for_service_timeout parameters in call to change_state() ?

[mini-1235]

@mini-1235 #820 should be a workaround for this issue, waiting for a definitive one addressing ros2/rmw#404.

Thanks for the follow up @JEnoch!

Why not using service_timeout_ for both transition_timeout and wait_for_service_timeout parameters in call to change_state() ?

I think this is a special case where we really need infinite timeout, as explained by @SteveMacenski in ros-navigation/navigation2#4838 (comment)

[SteveMacenski]

Basically, due to other RMW's (at least historical, if not contemporary) quirks & arbitrary end-user code. Its the single only place in Nav2 where we do that however.