pvc is always in pending

[wangshuai0709]

My Rook cluster and PVC are in different Kubernetes clusters.My PVC is for cross-cluster access with Rook.
ceph-csi-config configmap:

apiVersion: v1
data:
  cluster-mapping.json: '[]'
  config.json: |
    [
       {
         "clusterID": "c7c911ff-f040-421e-b7fa-e61996f4bc78",
         "monitors": [
           "10.113.208.30:31212",
           "10.113.208.31:31212",
           "10.113.208.32:31212"
         ]
       }
    ]
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ceph-csi-rbd
    meta.helm.sh/release-namespace: rook-ceph
  creationTimestamp: "2025-06-24T07:25:59Z"
  labels:
    app: ceph-csi-rbd
    app.kubernetes.io/managed-by: Helm
    chart: ceph-csi-rbd-3-canary
    component: nodeplugin
    heritage: Helm
    release: ceph-csi-rbd
  name: ceph-csi-config
  namespace: rook-ceph
  resourceVersion: "26784"
  uid: 25eecaba-25f6-44a5-a02e-bbbc21251a73

storageclass yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
provisioner: rbd.csi.ceph.com
parameters:
  clusterID: c7c911ff-f040-421e-b7fa-e61996f4bc78
  # ...... CephFS CSI ..........................................
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  # CephFS ............
  pool: replicapool
reclaimPolicy: Delete
volumeBindingMode: Immediate

rook-csi-rbd-node and rook-csi-rbd-provisioner secret is copied from the rook-ceph cluster k8s.

PVC events:

  Normal   Provisioning          3m28s (x12 over 22m)  rbd.csi.ceph.com_ceph-csi-rbd-provisioner-7bf85c4477-ljqc4_e6cb5dd9-b2bb-4fac-8371-eef5a086ba58  External provisioner is provisioning volume for claim "rook-ceph/rbd-fox-dev"
  Warning  ProvisioningFailed    2m28s (x12 over 21m)  rbd.csi.ceph.com_ceph-csi-rbd-provisioner-7bf85c4477-ljqc4_e6cb5dd9-b2bb-4fac-8371-eef5a086ba58  failed to provision volume with StorageClass "fast": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  118s (x82 over 22m)   persistentvolume-controller                                                                      Waiting for a volume to be created either by the external provisioner 'rbd.csi.ceph.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

ceph-csi-rbd-provisioner-7bf85c4477-ljqc4 pod log:

I0624 09:46:26.851037       1 utils.go:265] ID: 74 GRPC call: /csi.v1.Identity/Probe
I0624 09:46:26.851087       1 utils.go:266] ID: 74 GRPC request: {}
I0624 09:46:26.851113       1 utils.go:272] ID: 74 GRPC response: {}
I0624 09:46:43.860483       1 utils.go:265] ID: 75 Req-ID: pvc-7a673d77-c3aa-4053-b09c-173c5edb7573 GRPC call: /csi.v1.Controller/CreateVolume
I0624 09:46:43.860636       1 utils.go:266] ID: 75 Req-ID: pvc-7a673d77-c3aa-4053-b09c-173c5edb7573 GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-7a673d77-c3aa-4053-b09c-173c5edb7573","parameters":{"clusterID":"c7c911ff-f040-421e-b7fa-e61996f4bc78","csi.storage.k8s.io/pv/name":"pvc-7a673d77-c3aa-4053-b09c-173c5edb7573","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
I0624 09:46:43.860900       1 rbd_util.go:1410] ID: 75 Req-ID: pvc-7a673d77-c3aa-4053-b09c-173c5edb7573 setting disableInUseChecks: false image features: [] mounter: rbd
I0624 09:46:49.598978       1 utils.go:298] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 Slow GRPC call /csi.v1.Controller/CreateVolume (4m30s)
I0624 09:46:49.599163       1 utils.go:300] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 Slow GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-db7145c8-ab38-47ae-a9f4-024259cd5874","parameters":{"clusterID":"c7c911ff-f040-421e-b7fa-e61996f4bc78","csi.storage.k8s.io/pv/name":"pvc-db7145c8-ab38-47ae-a9f4-024259cd5874","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
I0624 09:47:13.878558       1 utils.go:298] ID: 73 Req-ID: pvc-7a673d77-c3aa-4053-b09c-173c5edb7573 Slow GRPC call /csi.v1.Controller/CreateVolume (1m29s)
I0624 09:47:13.878721       1 utils.go:300] ID: 73 Req-ID: pvc-7a673d77-c3aa-4053-b09c-173c5edb7573 Slow GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-7a673d77-c3aa-4053-b09c-173c5edb7573","parameters":{"clusterID":"c7c911ff-f040-421e-b7fa-e61996f4bc78","csi.storage.k8s.io/pv/name":"pvc-7a673d77-c3aa-4053-b09c-173c5edb7573","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
I0624 09:47:19.598567       1 utils.go:298] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 Slow GRPC call /csi.v1.Controller/CreateVolume (5m0s)
I0624 09:47:19.598777       1 utils.go:300] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 Slow GRPC request: {"capacity_range":{"required_bytes":5368709120},"name":"pvc-db7145c8-ab38-47ae-a9f4-024259cd5874","parameters":{"clusterID":"c7c911ff-f040-421e-b7fa-e61996f4bc78","csi.storage.k8s.io/pv/name":"pvc-db7145c8-ab38-47ae-a9f4-024259cd5874","csi.storage.k8s.io/pvc/name":"rbd-fox-dev","csi.storage.k8s.io/pvc/namespace":"rook-ceph","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"access_mode":{"mode":"SINGLE_NODE_WRITER"},"mount":{"fs_type":"ext4"}}]}
E0624 09:47:19.606551       1 controllerserver.go:243] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 failed to connect to volume : failed to get connection: connecting failed: rados: ret=-110, Connection timed out
E0624 09:47:19.606635       1 utils.go:270] ID: 68 Req-ID: pvc-db7145c8-ab38-47ae-a9f4-024259cd5874 GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-110, Connection timed out

The mon is ok and telnet 10.113.208.30 31212 is ok.
I don't know where the problem lies.

[Madhu-1]

@wangshuai0709 try below things and see if it works

• telet from the csi-rbdplugin container of the provisioner pod (this verifies you have network connectivity)
• Try running ceph/rbd commands from the same csi-rbdplugin container

[Madhu-1]

AFAIK Rook doesnt support service with NodePort, the option is to use Host Networking to run ceph cluster. Have you manually edited the service to have NodePort? How this is done. Please let me know if am wrong

[wangshuai0709]

I manually edited the service to have NodePort

[wangshuai0709]

Ok,Thank you for your help.

[Madhu-1]

Closing this one as its not supported configuration, suggested to use Hostnetwork in Cephcluster CR

[wangshuai0709]

Closing this one as its not supported configuration, suggested to use Hostnetwork in Cephcluster CR

OK,Thank you for you reply.