numbers of devices are not matching with namespace count after I do nvme connect with DHCHAP keys

[HaruChebrolu]

I have two gateways in a gateway group, connected to one subsystem with four namespaces. After performing an NVMe connect using DHCHAP keys, fewer devices appear than the total number of namespaces. However, this issue does not occur when connecting without DHCHAP keys.

Steps to reproduce:

1. Created a GW group with two gateways connected to 1 subsystem with 4 namespaces with bidirectional inband authentication.
2. nvme discover
3. nvme connect with DHCHAP keys.
4. Compared lsblk devices against total number of namespaces.
5. I continued to perform HA failover using daemon stop, IO didn't progress as well.

ceph -v
ceph version 19.3.0-7350-g7fcd9cba (7fcd9cbac04f022c8a95dfed302e308881bc1da4) squid (dev)

podman run --quiet --rm  quay.io/ceph/nvmeof-cli:latest  --server-address 10.0.66.242 --server-port 5500 subsystem add  --subsystem nqn.2016-06.io.spdk:cnode1 --dhchap-key DHHC-1:00:zRelgzEVRWKzcStoV1Wf4iM12Zfd/rfi01ZuQWAgcX2vCJim: --max-namespaces 32 --no-group-append
Adding subsystem nqn.2016-06.io.spdk:cnode1: Successful

podman run --quiet --rm  quay.io/ceph/nvmeof-cli:latest  --server-address 10.0.66.242 --server-port 5500 host add  --subsystem nqn.2016-06.io.spdk:cnode1 --dhchap-key DHHC-1:00:zRelgzEVRWKzcStoV1Wf4iM12Zfd/rfi01ZuQWAgcX2vCJim: --host nqn.2014-08.org.nvmexpress:uuid:bd6e1b0f-08e7-4eec-ad6b-a27b13e1e68e
Adding host nqn.2014-08.org.nvmexpress:uuid:bd6e1b0f-08e7-4eec-ad6b-a27b13e1e68e to nqn.2016-06.io.spdk:cnode1: Successful

nvme discover  --transport tcp --traddr 10.0.66.242 --trsvcid 8009 --output-format json 
"genctr":1,
  "records":[
    {
      "trtype":"tcp",
      "adrfam":"ipv4",
      "subtype":"nvme subsystem",
      "treq":"not required",
      "portid":0,
      "trsvcid":"4420",
      "subnqn":"nqn.2016-06.io.spdk:cnode1",
      "traddr":"10.0.66.242",
      "eflags":"none",
      "sectype":"none"
    },
    {
      "trtype":"tcp",
      "adrfam":"ipv4",
      "subtype":"nvme subsystem",
      "treq":"not required",
      "portid":1,
      "trsvcid":"4420",
      "subnqn":"nqn.2016-06.io.spdk:cnode1",
      "traddr":"10.0.66.184",
      "eflags":"none",
      "sectype":"none"
    }
  ]
}

nvme connect  --transport tcp --traddr 10.0.66.184 --trsvcid 4420 --nqn nqn.2016-06.io.spdk:cnode1 --dhchap-secret DHHC-1:00:zRelgzEVRWKzcStoV1Wf4iM12Zfd/rfi01ZuQWAgcX2vCJim: --dhchap-ctrl-secret DHHC-1:00:zRelgzEVRWKzcStoV1Wf4iM12Zfd/rfi01ZuQWAgcX2vCJim:
('connecting to device: nvme0\n', '')

Expected NVMe Targets : {'728a3269-4764-4888-897e-1262653656cf', 'f4959c03-571c-4e77-b705-a9be8eb566d6', 'cb4f1f8c-fc2a-411d-95fa-6c97c66c1bc1', 'df48ded2-0fd5-495c-a515-248ab4c1c2fb'} Vs LSBLK devices: {'f4959c03-571c-4e77-b705-a9be8eb566d6', 'cb4f1f8c-fc2a-411d-95fa-6c97c66c1bc1'}

node6.gz
node7.gz