Broken CommandTransport in cluster mode

[lbetz]

Describe the bug

I have configured both Icinga 2 nodes in the IcingaDB module on a dedicated running Icinga Web 2. If the first in the list is going down, every sent command get: 'Cannot traverse an already closed generator'.

To Reproduce

1. configure two running icinga2 instances in /etc/icingaweb2/modules/icingadb/commandtransports.ini, e.g.
   [server1.icinga.com]
   transport = "api"
   host = "server1.icinga.com"
   port = "5665"
   username = "icingaweb2"
   password = "icingaweb2"

[server2.icinga.com]
transport = "api"
host = "server2.icinga.com"
port = "5665"
username = "icingaweb2"
password = "icingaweb2"

2. stop icinga2 on server1
3. go to an server or service and send a 'check now' command

Expected behavior

IcingaDB module should try the second icicinga2 (server2) if the first one fails.

Your Environment

Include as many relevant details about the environment you experienced the problem in.

• Icinga DB Web version (System - About): 1.1.1
• Icinga Web 2 version (System - About): 2.12.1
• Web browser: Safari
• Icinga 2 version (icinga2 --version): 2.14.0
• Icinga DB version (icingadb --version): 1.1.1
• PHP version used (php --version): 8.0.30
• Server operating system and version: Alma Linux 9.2

[gbin2265]

Hello,

Can you also do the check that the icinga is not in progress with reloading it?

From this version , the API no longer accepts changes when it is reloading.

Icinga 2 v2.14.0
API: reject config modifications during reload with HTTP status 503.

This question is because reloading takes some time for us.
And if the command transporter refers to the host thas is reloading, then nothing happens.

[lbetz]

Icinga2 wasn't in progress of reload.

[gbin2265]

sorry my english is not very good and maybe not explained very well.

I just ask that they do 2 tests.

IF

the first server icinga2 is not running (your check)

OR

the first server is running but has the icinga2 status reloading (my check)

THEN

go to the second server