Discovery Server becomes unresponsive with a large number of participants

[thomasmoore-torc]

Overview

In our system, we are utilizing the Discovery Server and have a single participant that subscribes to all of the other topics in the system. We have noticed that some of the subscriptions in this participant will go unmatched despite the topic being successfully matched in other subscribers. While looking into this issue, we've discovered that it is possible to make the discovery server become unresponsive when there are a sufficient number of new participants. While the scenario below is likely more aggressive than most systems, it is effective in demonstrating how the discovery server can become unresponsive under load, which could be a contributing factor to our observed issue of subscribers not being matched.

Our testing of the below scenario was done on an Ubuntu 22.04 system with the current ROS2 Iron packages installed, which use Fast-DDS version 2.10.6. Similar results were observed with a compiled version of ROS2 Jazzy using Fast-DDS version 2.14.1. We have tried several things to attempt to improve the behavior, but have been unsuccessful:

1. Increase the network buffer sizes:

sudo sysctl -w net.core.wmem_max=268435456
sudo sysctl -w net.core.rmem_max=268435456
sudo sysctl -w net.core.wmem_default=4194304
sudo sysctl -w net.core.rmem_default=4194304
sudo sysctl -w net.core.netdev_max_backlog=3000

2. Set leaseAnnouncement and leaseDuration to DURATION_INFINITY in the XML config
3. Various values of clientAnnouncementPeriod in the XML config

Scenario

The scenario below is executed on a single machine with 5 terminals.

In terminal 1, start an instance of the fast-discovery-server:

source /opt/ros/iron/setup.bash
fast-discovery-server -i 0 -l 127.0.0.1

In terminal 2, start an instance of htop to monitor the CPU utilization of fast-discovery-server:

htop -p $(pidof fast-discovery-server)

In terminal 3, start 100 publishers, which will cause the CPU utilization of fast-discovery-server to increase significantly:

source /opt/ros/iron/setup.bash
export ROS_DISCOVERY_SERVER=127.0.0.1:11811
for i in {1..100}; do ros2 topic pub /chatter_$i std_msgs/String "data: Hello World $i" & done

In terminal 4, attempt to run ros2 topic list, which will fail to return any of the /chatter_{N} topics while the CPU utilization of fast-discovery-server remains high:

source /opt/ros/iron/setup.bash
export ROS_DISCOVERY_SERVER=127.0.0.1:11811
export ROS_SUPER_CLIENT=1
ros2 topic list --no-daemon

In terminal 5, start 100 subscribers, which will cause the CPU utilization of fast-discovery-server to increase significantly and will never echo any topic data or return:

source /opt/ros/iron/setup.bash
export ROS_DISCOVERY_SERVER=127.0.0.1:11811
for i in {1..100}; do ros2 topic echo --no-daemon --once /chatter_$i std_msgs/String & done

At this point, attempting to run ros2 topic list in terminal 4 will fail to show any of the /chatter_{N} topics as the discovery server is completely unresponsive.

As a data point, if ROS_DISCOVERY_SERVER is not set in terminals 2 and 5 such that simple discovery is utilized, the ros2 topic echo commands will eventually display a topic and exit. It does take some time, but it does eventually work.