Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Invalid "invalid encryption kms configuration" error #3759

Closed
fabien-marchand opened this issue Apr 19, 2023 · 8 comments · Fixed by #3854
Closed

Invalid "invalid encryption kms configuration" error #3759

fabien-marchand opened this issue Apr 19, 2023 · 8 comments · Fixed by #3854
Assignees
@riya-singhal31
Labels
bug Something isn't working component/rbd Issues related to RBD
Milestone
release-v3.8.1

Comments

@fabien-marchand
Copy link

Describe the bug

After upgrading to v3.8.0, the following error appeared whereas I did not configure encryption:

csi-rbdplugin E0419 07:35:12.775698    6851 utils.go:210] ID: 69 Req-ID: 0001-0009-rook-ceph-0000000000000001-4aef4df9-b4a3-11ed-9485-b2ff204cea03 GRPC error: rpc error: code = Internal desc = invalid encryption kms configuration: missing encryption passphrase in secrets

Rolling back to v3.7.0 solved the issue.

This might be due to dc7ba68 which removed and else clause.
So before (in v3.7.0) the initKMS procedure was interrupted when no specific KMS configuration is present (which is my case).
Now (in v3.8.0) a non-nil error is returned:

diff --git a/internal/rbd/encryption.go b/internal/rbd/encryption.go
index c72113288..ea65f14aa 100644
--- a/internal/rbd/encryption.go
+++ b/internal/rbd/encryption.go
@@ -309,8 +309,6 @@ func (ri *rbdImage) initKMS(ctx context.Context, volOptions, credentials map[str
        kmsID, encType, err := ParseEncryptionOpts(ctx, volOptions, rbdDefaultEncryptionType)
        if err != nil {
                return err
-       } else if kmsID == "" {
-               return nil
        }

Environment details

  • Image/version of Ceph CSI driver : v3.8.0
  • Kernel version : 5.4.0
  • Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
    krbd or rbd-nbd) : krbd
  • Kubernetes cluster version : 1.25.4
  • Ceph cluster version : 17.2.6

Steps to reproduce

Steps to reproduce the behavior:

  1. Have the following StorageClass configuration:
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: rook-ceph-cluster
    meta.helm.sh/release-namespace: rook-ceph
    storageclass.kubernetes.io/is-default-class: "false"
  creationTimestamp: "2023-02-23T16:11:55Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ceph-block
  resourceVersion: "795634683"
  uid: fb7f65ef-0bff-49a0-bb52-5699bde7ce92
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  encrypted: "false"
  imageFeatures: layering
  imageFormat: "2"
  pool: ceph-blockpool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
  1. Create a pods with a PVC using this storage class

Actual results

The error about an "invalid encryption kms" is raised (as described above)

Expected behavior

No error while mounting Ceph image

Logs

"csi-rbdplugin" logs:

W0419 07:32:17.161034    6851 rbd_attach.go:241] kernel version "5.4.0-137-generic" doesn't support cookie feature
E0419 07:33:10.655993    6851 utils.go:210] ID: 23 Req-ID: 0001-0009-rook-ceph-0000000000000001-4aef4df9-b4a3-11ed-9485-b2ff204cea03 GRPC error: rpc error: code = Internal desc = invalid encryption kms configuration: missing encryption passphrase in secrets
@Madhu-1
Copy link
Collaborator

Madhu-1 commented Apr 19, 2023

@fabien-marchand do you have this problem if you dont set encrypted: "false" in storageclass?

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Apr 19, 2023 

[🎩︎]mrajanna@fedora rbd $]kubectl get sc rook-ceph-block -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2023-04-19T09:08:12Z"
  name: rook-ceph-block
  resourceVersion: "93479"
  uid: 39068deb-dc19-4afa-a73c-adf54451f8be
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  encrypted: "false"
  imageFeatures: layering
  imageFormat: "2"
  pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
[🎩︎]mrajanna@fedora rbd $]kubectl get pvc
NAME              STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
cephfs-pvc-fuse   Bound     pvc-a529aefa-b27f-4873-8d1c-ece81edae931   1Gi        RWO            rook-cephfs-fuse   83m
local-pvc-name    Bound     pvc-cf122973-e0de-477b-94c1-62a40fd1e3d6   1Gi        RWO            rook-cephfs        18h
rbd-pvc           Pending                                                                        rook-ceph-block    3m33s
[🎩︎]mrajanna@fedora rbd $]kubectl describe pvc/rbd-pvc
Name:          rbd-pvc
Namespace:     default
StorageClass:  rook-ceph-block
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                  From                                                                                                        Message
  ----     ------                ----                 ----                                                                                                        -------
  Normal   Provisioning          91s (x9 over 3m39s)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-7474686fb9-8d666_3a565fec-a1c7-460a-9a9a-51602cfcfc55  External provisioner is provisioning volume for claim "default/rbd-pvc"
  Warning  ProvisioningFailed    91s (x9 over 3m39s)  rook-ceph.rbd.csi.ceph.com_csi-rbdplugin-provisioner-7474686fb9-8d666_3a565fec-a1c7-460a-9a9a-51602cfcfc55  failed to provision volume with StorageClass "rook-ceph-block": rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
  Normal   ExternalProvisioning  4s (x16 over 3m39s)  persistentvolume-controller                                                                                 waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator

I0418 13:55:51.940204       1 cephcsi.go:199] Driver version: v3.8.0 and Git version: e13e72af0be97038f7cf158d3fb54c1cc6668020
I0418 13:55:51.940426       1 cephcsi.go:217] Initial PID limit is set to -1
I0418 13:55:51.940495       1 cephcsi.go:223] Reconfigured PID limit to -1 (max)
I0418 13:55:51.940687       1 cephcsi.go:248] Starting driver type: rbd with name: rook-ceph.rbd.csi.ceph.com
I0418 13:55:51.940850       1 driver.go:94] Enabling controller service capability: CREATE_DELETE_VOLUME
I0418 13:55:51.940867       1 driver.go:94] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0418 13:55:51.940871       1 driver.go:94] Enabling controller service capability: CLONE_VOLUME
I0418 13:55:51.940874       1 driver.go:94] Enabling controller service capability: EXPAND_VOLUME
I0418 13:55:51.940878       1 driver.go:107] Enabling volume access mode: SINGLE_NODE_WRITER
I0418 13:55:51.940881       1 driver.go:107] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0418 13:55:51.940884       1 driver.go:107] Enabling volume access mode: SINGLE_NODE_SINGLE_WRITER
I0418 13:55:51.940889       1 driver.go:107] Enabling volume access mode: SINGLE_NODE_MULTI_WRITER
W0418 13:55:51.940899       1 driver.go:178] replication service running on controller server is deprecated and replaced by CSI-Addons, see https://github.com/ceph/ceph-csi/issues/3314 for more details
I0418 13:55:51.940918       1 server.go:114] listening for CSI-Addons requests on address: &net.UnixAddr{Name:"/tmp/csi-addons.sock", Net:"unix"}
I0418 13:55:51.940992       1 server.go:126] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi-provisioner.sock", Net:"unix"}
I0418 13:55:52.194788       1 utils.go:195] ID: 1 GRPC call: /csi.v1.Identity/Probe
I0418 13:55:52.195965       1 utils.go:206] ID: 1 GRPC request: {}
I0418 13:55:52.196017       1 utils.go:212] ID: 1 GRPC response: {}
I0418 13:55:52.196671       1 utils.go:195] ID: 2 GRPC call: /csi.v1.Identity/GetPluginInfo
I0418 13:55:52.196724       1 utils.go:206] ID: 2 GRPC request: {}
I0418 13:55:52.196734       1 identityserver-default.go:39] ID: 2 Using default GetPluginInfo
I0418 13:55:52.196840       1 utils.go:212] ID: 2 GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"v3.8.0"}
I0418 13:55:52.197281       1 utils.go:195] ID: 3 GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0418 13:55:52.197314       1 utils.go:206] ID: 3 GRPC request: {}
I0418 13:55:52.197454       1 utils.go:212] ID: 3 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0418 13:55:52.198296       1 utils.go:195] ID: 4 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0418 13:55:52.198336       1 utils.go:206] ID: 4 GRPC request: {}
I0418 13:55:52.198365       1 controllerserver-default.go:72] ID: 4 Using default ControllerGetCapabilities
I0418 13:55:52.198710       1 utils.go:212] ID: 4 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0418 13:55:52.394020       1 utils.go:195] ID: 5 GRPC call: /csi.v1.Identity/Probe
I0418 13:55:52.394065       1 utils.go:206] ID: 5 GRPC request: {}
I0418 13:55:52.394082       1 utils.go:212] ID: 5 GRPC response: {}
I0418 13:55:52.394522       1 utils.go:195] ID: 6 GRPC call: /csi.v1.Identity/GetPluginInfo
I0418 13:55:52.394548       1 utils.go:206] ID: 6 GRPC request: {}
I0418 13:55:52.394554       1 identityserver-default.go:39] ID: 6 Using default GetPluginInfo
I0418 13:55:52.394579       1 utils.go:212] ID: 6 GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"v3.8.0"}
I0418 13:55:52.394848       1 utils.go:195] ID: 7 GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0418 13:55:52.394870       1 utils.go:206] ID: 7 GRPC request: {}
I0418 13:55:52.394948       1 utils.go:212] ID: 7 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0418 13:55:52.395325       1 utils.go:195] ID: 8 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0418 13:55:52.395343       1 utils.go:206] ID: 8 GRPC request: {}
I0418 13:55:52.395349       1 controllerserver-default.go:72] ID: 8 Using default ControllerGetCapabilities
I0418 13:55:52.395410       1 utils.go:212] ID: 8 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0418 13:55:52.395716       1 utils.go:195] ID: 9 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0418 13:55:52.395739       1 utils.go:206] ID: 9 GRPC request: {}
I0418 13:55:52.395747       1 controllerserver-default.go:72] ID: 9 Using default ControllerGetCapabilities
I0418 13:55:52.395828       1 utils.go:212] ID: 9 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0418 13:55:52.585394       1 utils.go:195] ID: 10 GRPC call: /csi.v1.Identity/Probe
I0418 13:55:52.585433       1 utils.go:206] ID: 10 GRPC request: {}
I0418 13:55:52.585448       1 utils.go:212] ID: 10 GRPC response: {}
I0418 13:55:52.585887       1 utils.go:195] ID: 11 GRPC call: /csi.v1.Identity/GetPluginInfo
I0418 13:55:52.585979       1 utils.go:206] ID: 11 GRPC request: {}
I0418 13:55:52.586037       1 identityserver-default.go:39] ID: 11 Using default GetPluginInfo
I0418 13:55:52.586084       1 utils.go:212] ID: 11 GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"v3.8.0"}
I0418 13:55:52.586441       1 utils.go:195] ID: 12 GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0418 13:55:52.586465       1 utils.go:206] ID: 12 GRPC request: {}
I0418 13:55:52.586649       1 utils.go:212] ID: 12 GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0418 13:55:52.587003       1 utils.go:195] ID: 13 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0418 13:55:52.587020       1 utils.go:206] ID: 13 GRPC request: {}
I0418 13:55:52.587026       1 controllerserver-default.go:72] ID: 13 Using default ControllerGetCapabilities
I0418 13:55:52.587097       1 utils.go:212] ID: 13 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0418 13:55:52.756795       1 utils.go:195] ID: 14 GRPC call: /csi.v1.Identity/GetPluginInfo
I0418 13:55:52.756822       1 utils.go:206] ID: 14 GRPC request: {}
I0418 13:55:52.756828       1 identityserver-default.go:39] ID: 14 Using default GetPluginInfo
I0418 13:55:52.756848       1 utils.go:212] ID: 14 GRPC response: {"name":"rook-ceph.rbd.csi.ceph.com","vendor_version":"v3.8.0"}
I0418 13:55:52.757330       1 utils.go:195] ID: 15 GRPC call: /csi.v1.Identity/Probe
I0418 13:55:52.757349       1 utils.go:206] ID: 15 GRPC request: {}
I0418 13:55:52.757364       1 utils.go:212] ID: 15 GRPC response: {}
I0418 13:55:52.757691       1 utils.go:195] ID: 16 GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0418 13:55:52.757711       1 utils.go:206] ID: 16 GRPC request: {}
I0418 13:55:52.757717       1 controllerserver-default.go:72] ID: 16 Using default ControllerGetCapabilities
I0418 13:55:52.757774       1 utils.go:212] ID: 16 GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":9}}}]}
I0419 09:08:19.830791       1 utils.go:195] ID: 17 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:19.832760       1 utils.go:206] ID: 17 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:19.835429       1 rbd_util.go:1308] ID: 17 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:19.836568       1 utils.go:210] ID: 17 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:20.350714       1 utils.go:195] ID: 18 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:20.353238       1 utils.go:206] ID: 18 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:20.354188       1 rbd_util.go:1308] ID: 18 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:20.354389       1 utils.go:210] ID: 18 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:21.364865       1 utils.go:195] ID: 19 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:21.366206       1 utils.go:206] ID: 19 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:21.368444       1 rbd_util.go:1308] ID: 19 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:21.368586       1 utils.go:210] ID: 19 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:23.381054       1 utils.go:195] ID: 20 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:23.382411       1 utils.go:206] ID: 20 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:23.383113       1 rbd_util.go:1308] ID: 20 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:23.383318       1 utils.go:210] ID: 20 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:27.394991       1 utils.go:195] ID: 21 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:27.395535       1 utils.go:206] ID: 21 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:27.396528       1 rbd_util.go:1308] ID: 21 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:27.396635       1 utils.go:210] ID: 21 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:35.402478       1 utils.go:195] ID: 22 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:35.403742       1 utils.go:206] ID: 22 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:35.403996       1 rbd_util.go:1308] ID: 22 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:35.404043       1 utils.go:210] ID: 22 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:08:51.410721       1 utils.go:195] ID: 23 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:08:51.411026       1 utils.go:206] ID: 23 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:08:51.411277       1 rbd_util.go:1308] ID: 23 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:08:51.411350       1 utils.go:210] ID: 23 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets
I0419 09:09:23.425599       1 utils.go:195] ID: 24 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC call: /csi.v1.Controller/CreateVolume
I0419 09:09:23.427457       1 utils.go:206] ID: 24 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","parameters":{"clusterID":"rook-ceph","csi.storage.k8s.io/pv/name":"pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe","csi.storage.k8s.io/pvc/name":"rbd-pvc","csi.storage.k8s.io/pvc/namespace":"default","encrypted":"false","imageFeatures":"layering","imageFormat":"2","pool":"replicapool"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}}]}
I0419 09:09:23.428102       1 rbd_util.go:1308] ID: 24 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe setting disableInUseChecks: false image features: [layering] mounter: rbd
E0419 09:09:23.428339       1 utils.go:210] ID: 24 Req-ID: pvc-d692a6a5-7e0f-4b46-9d7b-53ad6e16b0fe GRPC error: rpc error: code = InvalidArgument desc = invalid encryption kms configuration: missing encryption passphrase in secrets

PVC creation is also failing if encrypted: false is set

@fabien-marchand
Copy link
Author

fabien-marchand commented Apr 19, 2023 •
edited
Loading

do you have this problem if you dont set encrypted: "false" in storageclass?

yes the problem is still there

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Apr 19, 2023

do you have this problem if you dont set encrypted: "false" in storageclass?

yes the problem is still there

[🎩︎]mrajanna@fedora rbd $]kuberc get po/csi-rbdplugin-8kv9d -oyaml|grep 3.8.0
    image: quay.io/cephcsi/cephcsi:v3.8.0
    image: quay.io/cephcsi/cephcsi:v3.8.0
[🎩︎]mrajanna@fedora rbd $]kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
cephfs-pvc-fuse   Bound    pvc-a529aefa-b27f-4873-8d1c-ece81edae931   1Gi        RWO            rook-cephfs-fuse       141m
local-pvc-name    Bound    pvc-cf122973-e0de-477b-94c1-62a40fd1e3d6   1Gi        RWO            rook-cephfs            19h
rbd-pvc           Bound    pvc-a8ac2c08-fd42-4b2f-b9df-38c0744d5c40   1Gi        RWO            rook-ceph-block-test   37s
[🎩︎]mrajanna@fedora rbd $]kubectl get po
NAME                      READY   STATUS    RESTARTS      AGE
csicephfs-demo-pod-fuse   1/1     Running   0             141m
csirbd-demo-pod           1/1     Running   0             32s
rhel7                     1/1     Running   5 (23m ago)   19h
[🎩︎]mrajanna@fedora rbd $]

am not able to reproduce this one if you dont have encrypted parameter in storageclass

@fabien-marchand
Copy link
Author

Yes actually you're right: removing encrypted from the definition of the StorageClass does not trigger the issue (I reused a PVC created when this option was set, so it was present in the definition of the PV - I had to rewrite the PV then).

@Madhu-1
Copy link
Collaborator

Madhu-1 commented Apr 19, 2023

Yes actually you're right: removing encrypted from the definition of the StorageClass does not trigger the issue (I reused a PVC created when this option was set, so it was present in the definition of the PV - I had to rewrite the PV then).

yes that will be workaround for now until this gets fixed

@Madhu-1 Madhu-1 added bug Something isn't working component/rbd Issues related to RBD labels Apr 19, 2023
@Madhu-1 Madhu-1 added this to the release-v3.8.1 milestone Apr 19, 2023
@riya-singhal31 riya-singhal31 mentioned this issue May 24, 2023
Merged
@riya-singhal31
Copy link
Contributor

Closes with: #3854

@mergify mergify bot closed this as completed in #3854 Jun 6, 2023
@nics90
Copy link

nics90 commented Feb 7, 2025

@fabien-marchand : I'm using the kernel version as you mentioned above and in rbd plugin pod getting the below error:

"rbd_attach.go:241] kernel version "5.4.0-137-generic" doesn't support cookie feature"

By any chance did you face the issue in ceph that when network fluctuation happens then all rbd mount points are inaccessible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@riya-singhal31 riya-singhal31

Labels
bug Something isn't working component/rbd Issues related to RBD
Projects
None yet
Milestone
release-v3.8.1
Development

Successfully merging a pull request may close this issue.

fix: Invalid "invalid encryption kms configuration" error
4 participants
@nics90 @Madhu-1 @riya-singhal31 @fabien-marchand