Skip to content

Declare token errors when using UDP transport #765

New issue
New issue
Open
Open
Declare token errors when using UDP transport#765
Assignees
JEnoch
Labels
bugSomething isn't working
@dwffls

Description

@dwffls
dwffls
opened on Sep 1, 2025

Generated by Generative AI

No

Operating System:

Linux base 6.8.0-65-generic #68~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 15 18:06:34 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

ROS version or commit hash:

jazzy

RMW implementation (if applicable):

rmw_zenoh_cpp

RMW Configuration (if applicable):

To enable UDP trasnport i've changed the router default config with the following changes:

{
  mode: "router",
  connect: {
    endpoints: [
      "tcp/10.0.4.9:7447",
      "udp/10.0.4.9:7447",
      "tcp/10.0.4.10:7447",
      "udp/10.0.4.10:7447",
    ],
  },
  listen: {
    endpoints: [
      "tcp/[::]:7447",
      "udp/[::]:7447",
    ]
  }

And for the sessions i've changed:

{
  mode: "peer",
  connect: {
    endpoints: [
      "tcp/localhost:7447",
      "udp/localhost:7447",
    ],
  },
  listen: {
    endpoints: [
      "tcp/localhost:0",
      "udp/localhost:0",
    ]
  }

Client library (if applicable):

rclcpp

'ros2 doctor --report' output

No response

Steps to reproduce issue

  1. I've changed the config's with the changes above
  2. Run routers on two different hosts
  3. And run publishers and subscribers on both hosts with rclcpp::SensorDataQos()

Expected behavior

UDP traffic between hosts and nodes, when reliability is set to best_effort

Actual behavior

UDP traffic appears to work sometimes, but not everytime. And after a certain period nodes do not want to connect anymore.
With this we get a lot of error messages with the running routers.

2025-09-01T10:14:07.532014Z  INFO ThreadId(02) zenoh::net::runtime: Using ZID: 6964828e84b015f8e2d1fab0db9f5d4e
2025-09-01T10:14:07.532833Z  WARN acc-0 ThreadId(04) zenoh_link_udp::unicast: Interceptors (e.g. Access Control, Downsampling) are not guaranteed to work on UDP when listening on 0.0.0.0 or [::]. Their usage is discouraged. See https://github.com/eclipse-zenoh/zenoh/issues/1126.
2025-09-01T10:14:07.533364Z  INFO ThreadId(02) zenoh::net::runtime::orchestrator: Zenoh can be reached at: tcp/[fe80::2f5c:38c8:d9c6:94fa]:7447
2025-09-01T10:14:07.533374Z  INFO ThreadId(02) zenoh::net::runtime::orchestrator: Zenoh can be reached at: tcp/10.0.4.10:7447
2025-09-01T10:14:07.533377Z  INFO ThreadId(02) zenoh::net::runtime::orchestrator: Zenoh can be reached at: udp/[fe80::2f5c:38c8:d9c6:94fa]:7447
2025-09-01T10:14:07.533379Z  INFO ThreadId(02) zenoh::net::runtime::orchestrator: Zenoh can be reached at: udp/10.0.4.10:7447
2025-09-01T10:14:07.638519Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 300!
2025-09-01T10:14:07.638550Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 250!
2025-09-01T10:14:07.641695Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 246!
2025-09-01T10:14:07.641779Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 102!
2025-09-01T10:14:07.642310Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 95!
2025-09-01T10:14:07.642348Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 215!
2025-09-01T10:14:07.642689Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 112!
2025-09-01T10:14:07.642772Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 137!
2025-09-01T10:14:07.643210Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 73!
2025-09-01T10:14:07.643310Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 275!
2025-09-01T10:14:07.643417Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 286!
2025-09-01T10:14:07.643422Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 232!
2025-09-01T10:14:07.643688Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 222!
2025-09-01T10:14:07.643831Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 240!
2025-09-01T10:14:07.643943Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 143!
2025-09-01T10:14:07.644088Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 206!
2025-09-01T10:14:07.644121Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 93!
2025-09-01T10:14:07.644281Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 223!
2025-09-01T10:14:07.644465Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 255!
2025-09-01T10:14:07.644557Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 287!
2025-09-01T10:14:07.644589Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 77!
2025-09-01T10:14:07.644805Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 299!
2025-09-01T10:14:07.644811Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 117!
2025-09-01T10:14:07.644959Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 67!
2025-09-01T10:14:07.645118Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 220!
2025-09-01T10:14:07.645166Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 285!
2025-09-01T10:14:07.645473Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 171!
2025-09-01T10:14:07.645479Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 135!
2025-09-01T10:14:07.645482Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 242!
2025-09-01T10:14:07.645672Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 243!
2025-09-01T10:14:07.645693Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 311!
2025-09-01T10:14:07.645779Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 78!
2025-09-01T10:14:07.645824Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 75!
2025-09-01T10:14:07.645947Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 247!
2025-09-01T10:14:07.645952Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 294!
2025-09-01T10:14:07.645955Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 164!
2025-09-01T10:14:07.646065Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 203!
2025-09-01T10:14:07.646112Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 94!
2025-09-01T10:14:07.646220Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 123!
2025-09-01T10:14:07.646241Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 274!
2025-09-01T10:14:07.646405Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 235!
2025-09-01T10:14:07.646759Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 189!
2025-09-01T10:14:07.646917Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 216!
2025-09-01T10:14:07.647029Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 185!
2025-09-01T10:14:07.647173Z ERROR  rx-1 ThreadId(07) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Declare token 0 for unknown scope 153!
Started Zenoh router with id 6964828e84b015f8e2d1fab0db9f5d4e
2025-09-01T10:14:28.143403Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare queryable 0 with unknown scope 274
2025-09-01T10:14:28.143788Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare liveliness token with unknown scope 274
2025-09-01T10:14:28.200316Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::pubsub: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare subscriber with unknown scope 38
2025-09-01T10:14:28.200357Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare queryable 0 with unknown scope 208
2025-09-01T10:14:28.200383Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare queryable 0 with unknown scope 157
2025-09-01T10:14:33.184653Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare queryable 0 with unknown scope 287
2025-09-01T10:14:33.184746Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::token: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare liveliness token with unknown scope 287
2025-09-01T10:14:33.231598Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::pubsub: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare subscriber with unknown scope 37
2025-09-01T10:14:33.231978Z ERROR  rx-0 ThreadId(06) zenoh::net::routing::dispatcher::queries: Face{3, 30f6ad0720cd468382c10f620f7c51cb} Undeclare queryable 0 with unknown scope 315

Aswell as error messages on running nodes:

ros2 topic echo /chatter --once
2025-09-01T11:08:04.499429Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 35 for unknown scope 35!
2025-09-01T11:08:04.499448Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 36 for unknown scope 36!
2025-09-01T11:08:04.499457Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 37 for unknown scope 37!
2025-09-01T11:08:04.499461Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 38 for unknown scope 38!
2025-09-01T11:08:04.499467Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 39 for unknown scope 39!
2025-09-01T11:08:04.499471Z ERROR rx-0 ThreadId(08) zenoh::net::routing::dispatcher::token: Face{17, eee95f5860618fa509eb6098b3736b9a} Declare token 40 for unknown scope 40!
2025-09-01T11:08:04.976521Z  WARN ThreadId(14) zenoh::net::runtime::orchestrator: Scouting delay elapsed before start conditions are met.
2025-09-01T11:08:04.985766Z ERROR rx-4 ThreadId(12) zenoh::net::routing::dispatcher::token: Face{18, 54cb1f7abc3c484aff51e915958b0439} Declare token 0 for unknown scope 4!
2025-09-01T11:08:04.985797Z ERROR rx-4 ThreadId(12) zenoh::net::routing::dispatcher::token: Face{18, 54cb1f7abc3c484aff51e915958b0439} Declare token 0 for unknown scope 5!
2025-09-01T11:08:04.985818Z ERROR rx-4 ThreadId(12) zenoh::net::routing::dispatcher::token: Face{18, 54cb1f7abc3c484aff51e915958b0439} Declare token 0 for unknown scope 6!
2025-09-01T11:08:14.979210Z  WARN net-0 ThreadId(02) zenoh::net::routing::dispatcher::interests: Face{18, 54cb1f7abc3c484aff51e915958b0439}:3 Didn't receive DeclareFinal for interest Face{1, c2435cc3e4546025e0a77fceb250fe1b}:0: Timeout(10s)!
2025-09-01T11:08:14.979327Z  WARN net-0 ThreadId(02) zenoh::net::routing::dispatcher::interests: Face{15, dadc7c6f7f88652844a5f35d8df7a3e0}:3 Didn't receive DeclareFinal for interest Face{1, c2435cc3e4546025e0a77fceb250fe1b}:0: Timeout(10s)!
2025-09-01T11:08:14.979366Z  WARN net-0 ThreadId(02) zenoh::net::routing::dispatcher::interests: Face{16, 81eebac27dcd92d6a3c69f952b5e424}:3 Didn't receive DeclareFinal for interest Face{1, c2435cc3e4546025e0a77fceb250fe1b}:0: Timeout(10s)!
2025-09-01T11:08:14.979392Z  WARN net-0 ThreadId(02) zenoh::net::routing::dispatcher::interests: Face{17, eee95f5860618fa509eb6098b3736b9a}:3 Didn't receive DeclareFinal for interest Face{1, c2435cc3e4546025e0a77fceb250fe1b}:0: Timeout(10s)!
2025-09-01T11:08:24.987894Z ERROR ThreadId(14) zenoh::api::session: error=close operation timed out! at /home/buildfarm/.cargo/git/checkouts/zenoh-cc237f2570fab813/c051173/zenoh/src/api/builders/close.rs:121.

Additional information

We've been trying to get UDP traffic to work with zenoh between multiple hosts. the implementation comes from this comment by @JEnoch on the ros disource. This all runs without any problems on a single host.

However when running this with multiple hosts we run into a lot of errors and nodes that don't want to pub/sub on topics.
Was hoping to get some help with getting this working for our robot.

The same also happens when setting listen/connect: ["tcp/[::]:7447, upd/{host_ip}:7447"] which suppresses the
WARN acc-0 ThreadId(04) zenoh_link_udp::unicast: Interceptors (e.g. Access Control, Downsampling) are not guaranteed to work on UDP when listening on 0.0.0.0 or [::]. Their usage is discouraged. See https://github.com/eclipse-zenoh/zenoh/issues/1126.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions