Skip to content

NodeUnpublish stuck when the filesystem is shutdown #615

@tiagolobocastro

Description

@tiagolobocastro

What steps did you take and what happened:
When some goes awry there's a chance a filesystem may shutdown, example: #614
In this case, deleting the Pod is stuck because NodeUnpublish is stuck.

What did you expect to happen:
Expect to be able to delete the pod.

The output of the following commands will help us better understand what's going on:

  • kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin
I0121 15:24:24.353104       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:24.356452       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error
I0121 15:24:24.956720       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:24.961777       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error
I0121 15:24:25.964596       1 grpc.go:72] GRPC call: /csi.v1.Node/NodeUnpublishVolume requests {"target_path":"/var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount","volume_id":"pvc-f5e204ba-3797-47f9-bfe7-236d078292f7"}
E0121 15:24:25.968437       1 grpc.go:79] GRPC error: rpc error: code = Internal desc = unable to umount the volume pvc-f5e204ba-3797-47f9-bfe7-236d078292f7 err : Error checking if path exists: stat /var/lib/kubelet/pods/cef1876f-ff31-4d5e-a8c7-d8c8c1243138/volumes/kubernetes.io~csi/pvc-f5e204ba-3797-47f9-bfe7-236d078292f7/mount: input/output error

Environment:

  • LocalPV-ZFS version: 2.7.0-develop
  • Cloud provider or hardware configuration: qemu vm x86_64
  • OS (e.g. from /etc/os-release): NixOS 24.11 (Vicuna)
  • Kernel version: Linux nixos 6.6.69

Metadata

Metadata

Labels

kind/bugCategorizes issue or PR as related to a bug

Type

No type

Projects

Status

No status

Relationships

None yet

Development

No branches or pull requests

Issue actions