Helm ceph-csi-rbd - not enough permitions

[baznikin]

Describe the bug

It is not possible to provision with current (3.0.0) Helm chart version. See Steps to reproduce for details

Environment details

• Image/version of Ceph CSI driver :
• Helm chart version : 3.0.0
• Kernel version :
• Mounter used for mounting PVC (for cephfs its fuse or kernel. for rbd its
  krbd or rbd-nbd) :
• Kubernetes cluster version :
• Ceph cluster version :

Steps to reproduce

1. I installed ceph-csi-rbd using Helm chart:
   helm install ceph-csi-rbd ceph-csi/ceph-csi-rbd --namespace ceph-csi-rbd -f ceph-csi-rbd.yaml

2. With tolopogy.enabled=false (default value) i got error about nodes resource

Warning ProvisioningFailed 11m (x13 over 30m) rbd.csi.ceph.com_ceph-csi-rbd-provisioner-dfd44c7b6-5j5xz_00aa056a-c7ad-45c5-9631-1a4e9d984eea failed to get target node: nodes "worker-1-1" is forbidden: User "system:serviceaccount:ceph-csi-rbd:ceph-csi-rbd-provisioner" cannot get resource "nodes" in API group "" at the cluster scope

3. In order to enable it (see

   ceph-csi/charts/ceph-csi-rbd/templates/provisioner-rules-clusterrole.yaml

   Line 54 in 23e0874

   {{- if .Values.topology.enabled }}

   ) I set tolopogy.enabled=true, remove ClusterRole ceph-csi-rbd-provisioner-rules and upgrade Helm chart. Got another error:

Warning ProvisioningFailed 2m47s (x10 over 7m2s) rbd.csi.ceph.com_ceph-csi-rbd-provisioner-64cb55db6-zkm9w_dbd3b112-40ac-4ce7-b995-8cc381d6771e failed to provision volume with StorageClass "csi-rbd-sc": error generating accessibility requirements: no topology key found on CSINode worker-1-1

4. After setting tolopogy.enabled back to false, upgrading chart without deleting ClusterRole ceph-csi-rbd-provisioner-rules (so permitions for nodes remains in place) I got working provisioning:

Normal Provisioning 3m47s rbd.csi.ceph.com_ceph-csi-rbd-provisioner-dfd44c7b6-qnrmw_e39c8074-d8a3-4eda-8c43-b264a9bf9f9c External provisioner is provisioning volume for claim "db/data-redis-redis-ha-server-2"
Normal ExternalProvisioning 3m47s persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator
Normal ProvisioningSucceeded 3m47s rbd.csi.ceph.com_ceph-csi-rbd-provisioner-dfd44c7b6-qnrmw_e39c8074-d8a3-4eda-8c43-b264a9bf9f9c Successfully provisioned volume pvc-a5b443c3-cc37-414e-874d-bc288981688a

[Madhu-1]

@baznikin in tried to reproduce it with master build am not seeing any issue can you retry with v3.1.0 helm charts, if the issue persists can you provide the deployment, cluster role of rbd? and also the PVC -oyaml output

[baznikin]

@Madhu-1 sorry, I found workaround for my issue and live with it. My intention was to notify maintainer (who know his software better than me) about found lack of permissions. If you follow my steppes and it didn't reproduce with 3.0.0 then it fixed already and can be closed.

[Madhu-1]

closing this as not reproducible

[XtremeOwnageDotCom]

As a developer- I understand closing/skipping non-reproducible problems.

As a user- I really hate finding closed tickets like this.

[simonliska]

@XtremeOwnageDotCom Same problem with helm chart version 3.10 on k8s v1.24. Works for me with 3.9.0.

[XtremeOwnageDotCom]

The solution, is pretty simple.

You need to edit the cluster roles for ceph.

root@rke-master-1:~/Kubernetes/rke/ceph# kubectl get ClusterRole | grep ceph
ceph-csi-rbd-nodeplugin                                                2023-12-01T03:07:50Z
ceph-csi-rbd-provisioner                                               2023-12-01T03:07:51Z

For each of those roles, you need to add the ability to get nodes.

(Add this)

- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - get
  - list
  - watch

There is prob a fancy way of doing this with a patch, but, I did it the simple, stupid way.

kubectl get ClusterRole ceph-csi-rbd-nodeplugin -o yaml > t.yaml && nano t.yaml && kubectl apply -f t.yaml

As a note, 3.10 also has this issue.

[simonliska]

@XtremeOwnageDotCom #4302

[XtremeOwnageDotCom]

Until that PR hits a release- here is a patch command to fix the issue.

kubectl patch ClusterRole ceph-csi-rbd-nodeplugin --type=json -p='[{"op":"add","path":"/rules/-","value":{"apiGroups":[""],"resources":["nodes"],"verbs":["get","list","watch"]}}]'

[skliarie]

I used helm3 on kubernetes 1.25.4, and had to add this to ClusterRole (exactly as if .Values.topology.domainLabels is set):

  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list","watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]

otherwise got these errors from provisioner:

E0822 23:13:52.973668 1 reflector.go:150] k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:ceph-csi-rbd:ceph-csi-rbd-provisioner" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope

[iPraveenParihar]

E0822 23:13:52.973668 1 reflector.go:150] k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.CSINode: failed to list *v1.CSINode: csinodes.storage.k8s.io is forbidden: User "system:serviceaccount:ceph-csi-rbd:ceph-csi-rbd-provisioner" cannot list resource "csinodes" in API group "storage.k8s.io" at the cluster scope

@skliarie, #4798 should fix the above issue.
Thanks for reporting.

[laimis9133]

Just stepping in: this issue is still present on 3.12.1 chart on v1.27.4+rke2r1 Kubernetes. We faced this after upgrading from 3.9.0.

It took some time to figure out so just a small backtrace below to help out anyone searching:
Kubernetes is not able to provision any new PVC that runs on Ceph RBD (rbd.csi.ceph.com) and will show this description:

Events:
... ... persistentvolume-controller waiting for a volume to be created, either by external provisioner "rbd.csi.ceph.com" or manually created by system administrator

The log trace mentioned here can be found in the ceph-csi-rbd-provisioner pod behind the csi-provisioner container - it will not be visible if the container is not specified as the default logs are from csi-rbdplugin not csi-provisioner in the pod.

Mentioned workarounds solve this.