Describe the bug
Deleting a PVC for a statically-provisioned RBD/CephFS volume whose
volumeHandle is a plain image/subvolume name, instead of ceph-csi's encoded
CSI identifier — causes DeleteVolume to fail with codes.Internal.
external-provisioner treats codes.Internal as retryable and retries forever,
so the PV is left stuck behind the
external-provisioner.volume.kubernetes.io/finalizer finalizer and is never
garbage collected. The only workaround is manually removing the finalizer.
Steps to reproduce
- Create a static PV/PVC pair pointing at an existing RBD image, but set
volumeHandle to the plain image name (e.g. my-image) rather than the
encoded CSI identifier ceph-csi normally generates, and set
persistentVolumeReclaimPolicy: Delete.
- Delete the PVC.
DeleteVolume is called on the plugin and fails to decode the volume ID.
Expected behavior
DeleteVolume should return a terminal error (codes.InvalidArgument) when
the volume ID can't be decoded as a ceph-csi CSI identifier, since retrying
can never succeed. This is the same pattern already used for the identical
error in RBD's ControllerExpandVolume and ControllerModifyVolume, which
return codes.InvalidArgument for ErrInvalidVolID with a "likely a static
provisioned volume" comment. DeleteVolume (RBD and CephFS) doesn't special
case this error and falls through to codes.Internal instead.
Actual results
DeleteVolume returns:
"rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID
(failed to decode CSI identifier, string underflow) (my-image)"
external-provisioner retries indefinitely since codes.Internal is treated as
retryable. The PV never gets garbage collected and stays stuck behind the
provisioner finalizer.
Logs
I0821 10:08:37.627043 1 event.go:389] "Event occurred" object="my-image" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID (failed to decode CSI identifier, string underflow) (my-image)"
E0821 10:08:37.715373 1 controller.go:1558] "Volume deletion failed" err="rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID (failed to decode CSI identifier, string underflow) (my-image)" PV="my-image"
Additional context
Note: docs/static-pvc.md documents that static PVs must use
persistentVolumeReclaimPolicy: Retain; a PV created with Delete (as in
this case) violates that documented requirement, but ceph-csi doesn't
currently validate this at PV-creation time.
Describe the bug
Deleting a PVC for a statically-provisioned RBD/CephFS volume whose
volumeHandleis a plain image/subvolume name, instead of ceph-csi's encodedCSI identifier — causes
DeleteVolumeto fail withcodes.Internal.external-provisioner treats
codes.Internalas retryable and retries forever,so the PV is left stuck behind the
external-provisioner.volume.kubernetes.io/finalizerfinalizer and is nevergarbage collected. The only workaround is manually removing the finalizer.
Steps to reproduce
volumeHandleto the plain image name (e.g.my-image) rather than theencoded CSI identifier ceph-csi normally generates, and set
persistentVolumeReclaimPolicy: Delete.DeleteVolumeis called on the plugin and fails to decode the volume ID.Expected behavior
DeleteVolumeshould return a terminal error (codes.InvalidArgument) whenthe volume ID can't be decoded as a ceph-csi CSI identifier, since retrying
can never succeed. This is the same pattern already used for the identical
error in RBD's
ControllerExpandVolumeandControllerModifyVolume, whichreturn
codes.InvalidArgumentforErrInvalidVolIDwith a "likely a staticprovisioned volume" comment.
DeleteVolume(RBD and CephFS) doesn't specialcase this error and falls through to
codes.Internalinstead.Actual results
DeleteVolumereturns:"rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID
(failed to decode CSI identifier, string underflow) (my-image)"
external-provisioner retries indefinitely since
codes.Internalis treated asretryable. The PV never gets garbage collected and stays stuck behind the
provisioner finalizer.
Logs
Additional context
Note:
docs/static-pvc.mddocuments that static PVs must usepersistentVolumeReclaimPolicy: Retain; a PV created withDelete(as inthis case) violates that documented requirement, but ceph-csi doesn't
currently validate this at PV-creation time.