yarpserver --read does not work as expected on Windows and Linux

[traversaro]

Describe the bug
It seems that trying to read an existing yarpserver configuration saved with yarpserver --write via yarpserver --read does not work on Windows, and instead the same error given by launching yarpserver without --read is printed.

To Reproduce
To install YARP:

conda create -n yarpenv
conda activate yarpenv
conda install -c conda-forge -c robotology yarp

Then launch it with a custom socket number

yarpserver.exe --socket 20000 --write

The output is:

(yarpenv) C:\Users\STraversaro>yarpserver.exe --socket 20000 --write

     ==========================
    | __  __ ___  ____   ____  |
    | \ \/ //   ||  _ \ |  _ \ |
    |  \  // /| || |/ / | |/ / |
    |  / // ___ ||  _ \ |  _/  |
    | /_//_/  |_||_| \_\|_|    |
    |                          |
     ==========================

Call with --help for information on available options
Using port database: :memory:
Using subscription database: :memory:
IP address: default
Port number: 20000
[ERROR] Database needs to be reset, IP or port number set.
[INFO] |yarp.os.Port| Port /root active at tcp://10.240.9.106:20000/
Registering name server with itself
 * register "/root" tcp "10.240.9.106" 20000
   + set "/root" offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer human mjpeg portmonitor priority rossrv shmem tcpros xmlrpc
   + set "/root" accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer human mjpeg portmonitor priority rossrv shmem tcpros xmlrpc
   + set "/root" ips "10.240.9.106" "127.0.0.1" "172.18.144.1" "172.17.224.1"
   + set "/root" process 17796
 * register fallback mcast "224.2.1.1" 10000
   + set fallback offers http name_ser local tcp fast_tcp mcast udp text text_ack bayer human mjpeg portmonitor priority rossrv shmem tcpros xmlrpc
   + set fallback accepts http name_ser local tcp fast_tcp mcast udp text text_ack bayer human mjpeg portmonitor priority rossrv shmem tcpros xmlrpc
   + set fallback ips "10.240.9.106" "127.0.0.1" "172.18.144.1" "172.17.224.1"
   + set fallback process 17796
 * set "/root" nameserver "true"
Name server can be browsed at http://10.240.9.106:20000/
Ok. Ready!

Then if I closed and launch again with --read:

(yarpenv) C:\Users\STraversaro>yarpserver.exe --read

     ==========================
    | __  __ ___  ____   ____  |
    | \ \/ //   ||  _ \ |  _ \ |
    |  \  // /| || |/ / | |/ / |
    |  / // ___ ||  _ \ |  _/  |
    | /_//_/  |_||_| \_\|_|    |
    |                          |
     ==========================

Call with --help for information on available options
Using port database: :memory:
Using subscription database: :memory:
IP address: default
Port number: 10000
PROBLEM: need to change settings in C:\Users\STraversaro\AppData\Roaming\yarp\config\yarp.conf
  Current settings: host 10.240.9.106 port 20000 family yarp
  Desired settings:  host 10.240.9.106 port 10000 family yarp
Please specify '--write' if it is ok to overwrite current settings, or
Please specify '--read' to use the current settings, or
delete C:\Users\STraversaro\AppData\Roaming\yarp\config\yarp.conf
[ERROR] Aborting.

Expected behavior
I would expect (based on the yarpserver output) that yarpserver --read would just read the configuration from C:\Users\STraversaro\AppData\Roaming\yarp\config\yarp.conf and do not complain that is different from the default.

Screenshots
N.a.

Configuration (please complete the following information):

• OS: Windows
• yarp version: 3.4.3 (conda binary build h2634132_7)
• compiler: Visual Studio 2019

Additional context
Complete environment for reproducibility:

(yarpenv) C:\Users\STraversaro>conda list
# packages in environment at C:\Users\STraversaro\Miniforge3\envs\yarpenv:
#
# Name                    Version                   Build  Channel
ace                       7.0.1                h0e60522_0    conda-forge
ca-certificates           2020.12.5            h5b45459_0    conda-forge
icu                       68.1                 h0e60522_0    conda-forge
jpeg                      9d                   he774522_0    conda-forge
libclang                  11.1.0          default_h5c34c98_1    conda-forge
libjpeg-turbo             2.1.0                h8ffe710_0    conda-forge
libpng                    1.6.37               ha81a0f5_2    conda-forge
openssl                   1.1.1k               h8ffe710_0    conda-forge
qt                        5.12.9               h5909a2a_4    conda-forge
robot-testing-framework   2.0.1                h0e60522_7    robotology
sdl                       1.2.15               h21ff451_1    conda-forge
sqlite                    3.35.5               h8ffe710_0    conda-forge
tinyxml                   2.6.2                h2d74725_2    conda-forge
vc                        14.2                 hb210afc_4    conda-forge
vs2015_runtime            14.28.29325          h5e1d092_4    conda-forge
yarp                      3.4.3                h2634132_7    robotology
ycm-cmake-modules         0.12.1               h0e60522_7    robotology
zlib                      1.2.11            h62dcd97_1010    conda-forge

[traversaro]

I tried the same on Linux, and a similar problem seems to be there as well, precise env in that case:

(yarpenv) traversaro@IITICUBLAP102:~$ conda list
# packages in environment at /home/traversaro/mambaforge/envs/yarpenv:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
ace                       7.0.1                h9c3ff4c_0    conda-forge
alsa-lib                  1.2.3                h516909a_0    conda-forge
ca-certificates           2020.12.5            ha878542_0    conda-forge
certifi                   2020.12.5        py39hf3d152e_1    conda-forge
dbus                      1.13.6               h48d8840_2    conda-forge
expat                     2.3.0                h9c3ff4c_0    conda-forge
fontconfig                2.13.1            hba837de_1005    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
gettext                   0.19.8.1          h0b5b191_1005    conda-forge
glib                      2.68.2               h9c3ff4c_0    conda-forge
glib-tools                2.68.2               h9c3ff4c_0    conda-forge
gst-plugins-base          1.18.4               hf529b03_2    conda-forge
gstreamer                 1.18.4               h76c114f_2    conda-forge
icu                       68.1                 h58526e2_0    conda-forge
jpeg                      9d                   h36c2ea0_0    conda-forge
krb5                      1.17.2               h926e7f8_0    conda-forge
ld_impl_linux-64          2.35.1               hea4e1c9_2    conda-forge
libclang                  11.1.0          default_ha53f305_1    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libevent                  2.1.10               hcdb4288_3    conda-forge
libffi                    3.3                  h58526e2_2    conda-forge
libgcc-ng                 9.3.0               h2828fa1_19    conda-forge
libglib                   2.68.2               h3e27bee_0    conda-forge
libgomp                   9.3.0               h2828fa1_19    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
libjpeg-turbo             2.1.0                h7f98852_0    conda-forge
libllvm11                 11.1.0               hf817b99_2    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libpq                     13.2                 hfd2b0eb_2    conda-forge
libstdcxx-ng              9.3.0               h6de172a_19    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libxcb                    1.13              h7f98852_1003    conda-forge
libxkbcommon              1.0.3                he3ba5ed_0    conda-forge
libxml2                   2.9.10               h72842e0_4    conda-forge
lz4-c                     1.9.3                h9c3ff4c_0    conda-forge
mysql-common              8.0.23               ha770c72_2    conda-forge
mysql-libs                8.0.23               h935591d_2    conda-forge
ncurses                   6.2                  h58526e2_4    conda-forge
nspr                      4.30                 h9c3ff4c_0    conda-forge
nss                       3.64                 hb5efdd6_0    conda-forge
openssl                   1.1.1k               h7f98852_0    conda-forge
pcre                      8.44                 he1b5a44_0    conda-forge
pip                       21.1.1             pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
python                    3.9.4           hffdb5ce_0_cpython    conda-forge
python_abi                3.9                      1_cp39    conda-forge
qt                        5.12.9               hda022c4_4    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
robot-testing-framework   2.0.1                h9c3ff4c_7    robotology
sdl                       1.2.15               he1b5a44_1    conda-forge
setuptools                49.6.0           py39hf3d152e_3    conda-forge
sqlite                    3.35.5               h74cdb3f_0    conda-forge
tinyxml                   2.6.2                h4bd325d_2    conda-forge
tk                        8.6.10               h21135ba_1    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yarp                      3.4.3                h59b5042_7    robotology
ycm-cmake-modules         0.12.1               h9c3ff4c_7    robotology
zlib                      1.2.11            h516909a_1010    conda-forge
zstd                      1.4.9                ha95c52a_0    conda-forge

[drdanz]

Some old issues/PR related to yarp server --read:

• yarp server & multiple eth interfaces #143
• problems running yarp server on a machine with multiple network interfaces #718
• yarpserver on computer with multiple network interfaces #1102
• arbitrary ip adress with yarpserver --read (ubuntu) #1404
• Fixed issue #1404 about 'yarp server --read command' #1457
• yarpserver --read does not work changing default port #1640

[traversaro]

To clarify, it is possible that I am not understanding correctly what yarpserver --read is supposed to do.

[drdanz]

According to the old issues, nobody does... 😆

[PeterBowman]

The --read (notabene not documented in yarpserver --help) and --write params are passed to the bootstrap server here:

yarp/src/libYARP_serversql/src/yarp/serversql/impl/NameServerContainer.cpp

Lines 124 to 126 in 8bde9a1

	if (!BootstrapServer::configFileBootstrap(contact,
	options.check("read"),
	options.check("write"))) {

And the error is triggered in said method: BootstrapServer::configFileBootstrap. It looks like the config file is always processed regardless of --read (see first if-else clause), but its presence mandates that no fallback is allowed in case of load error. Further down, sensible defaults are stored in the suggest contact variable based on said config file (pred), --read only seems to disable host setup (see if (!configFileRequired)). Later, both configs (stored and defaulted) are compared for changes, and the error reported in this issue is thrown because of a port mismatch (requested 10000 vs stored 20000) at least.

So, I think individual contact options need to be more closely inspected, especially the port one. The bootstrap server is fed with a contact template that defaults port/socket number to the one requested by the user, i.e. 10000 (ref1, ref2, ref3, ref4, ref5). We can later see in configFileBootstrap that suggest.setPort() is never called in that scenario (suggest's port is not zero), hence prev.getPort() != suggest.getPort() and the error is served.

[drdanz]

Sorry, a little slightly off topic digression, but the port number here is just the tip of the iceberg. The real problem, in my opinion, is that YARP lacks on configurability here, and the whole system is a bit broken.

Any "normal" server, allows you to choose one or more addresses and the ports to bind to.

Apparently YARP allows you to bind to one address, but that's not what happens, because YARP binds to all the ip addresses available. Instead, the IP passed to the configuration is the IP which is saved in the nameserver. The same happens when a port is opened while the YARP_IP.
Anyway, the ports are registered with all the available IP addresses (I'm not aware of any method to limit them, besides using software defined networks, or similar). You can also query all of them with yarp name get /port ips. Unfortunately AFAIK, this information is not used anywhere, and only the "main" IP is retrieved when a port tries to connect to another port.

Anyway, related to this issue, this line is clearly another problem: register fallback mcast "224.2.1.1" 10000. The port 10000 is not open, so that should be 20000 as well.

I'm quite sure that this used to work some time ago, but I don't remember any recent change to this part, the only one that comes to my mind is #1424, but that's long time ago. We should try bisecting to find out when this issue was introduced

Edit: After bisecting up to YARP 2.3.60, I realized that this never worked in the past. Anyway, up to 2e663e7 it did not cause an error, it was just ignored. The multicast port has never been working.

For reference this is the script I used for bisecting:

#!/bin/bash

rm -Rf build;
mkdir -p build;
(cd build && /opt/cmake-3.19/bin/cmake .. && make -j16 yarp) || exit 1
timeout 1 build/bin/yarp server --socket 20000 --write
timeout 1 build/bin/yarp server --read
[ $? -eq 124 ] && exit 0 || exit 3