Skip to content

DeleteVolume retries forever with codes.Internal for static PVs with non-decodable volume IDs #6489

Description

@fallmo

Describe the bug

Deleting a PVC for a statically-provisioned RBD/CephFS volume whose
volumeHandle is a plain image/subvolume name, instead of ceph-csi's encoded
CSI identifier — causes DeleteVolume to fail with codes.Internal.
external-provisioner treats codes.Internal as retryable and retries forever,
so the PV is left stuck behind the
external-provisioner.volume.kubernetes.io/finalizer finalizer and is never
garbage collected. The only workaround is manually removing the finalizer.

Steps to reproduce

  1. Create a static PV/PVC pair pointing at an existing RBD image, but set
    volumeHandle to the plain image name (e.g. my-image) rather than the
    encoded CSI identifier ceph-csi normally generates, and set
    persistentVolumeReclaimPolicy: Delete.
  2. Delete the PVC.
  3. DeleteVolume is called on the plugin and fails to decode the volume ID.

Expected behavior

DeleteVolume should return a terminal error (codes.InvalidArgument) when
the volume ID can't be decoded as a ceph-csi CSI identifier, since retrying
can never succeed. This is the same pattern already used for the identical
error in RBD's ControllerExpandVolume and ControllerModifyVolume, which
return codes.InvalidArgument for ErrInvalidVolID with a "likely a static
provisioned volume" comment. DeleteVolume (RBD and CephFS) doesn't special
case this error and falls through to codes.Internal instead.

Actual results

DeleteVolume returns:

"rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID
(failed to decode CSI identifier, string underflow) (my-image)"

external-provisioner retries indefinitely since codes.Internal is treated as
retryable. The PV never gets garbage collected and stays stuck behind the
provisioner finalizer.

Logs

I0821 10:08:37.627043       1 event.go:389] "Event occurred" object="my-image" fieldPath="" kind="PersistentVolume" apiVersion="v1" type="Warning" reason="VolumeFailedDelete" message="rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID (failed to decode CSI identifier, string underflow) (my-image)"
E0821 10:08:37.715373       1 controller.go:1558] "Volume deletion failed" err="rpc error: code = Internal desc = invalid VolumeID: error decoding volume ID (failed to decode CSI identifier, string underflow) (my-image)" PV="my-image"

Additional context

Note: docs/static-pvc.md documents that static PVs must use
persistentVolumeReclaimPolicy: Retain; a PV created with Delete (as in
this case) violates that documented requirement, but ceph-csi doesn't
currently validate this at PV-creation time.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions