openebs-zfs-localpv-node stuck in CreateContainerConfigError

[kordonskim]

After updating zfs-kernel from version 2.3.1 to 2.3.2 openebs-zfs-localpv-node is no longer working.
After reverting back to 2.3.1 everything is working again.

The output of the following commands will help us better understand what's going on:

• kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin

Error from server (BadRequest): container "openebs-zfs-plugin" in pod "openebs-zfs-localpv-node-7wzsp" is waiting to start: CreateContainerConfigError

• kubectl describe pod

....
Events:
Type Reason Age From Message

---

Warning FailedMount 2m43s (x3200 over 4d12h) kubelet MountVolume.MountDevice failed for volume "pvc-1018ba41-76b6-4de9-8f30-8a001867c0dd" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name zfs.csi.openebs.io not found in the list of registered CSI drivers

• kubectl get events -n openebs-system

LAST SEEN TYPE REASON OBJECT MESSAGE
3m56s Normal Pulled pod/openebs-zfs-localpv-node-7wzsp Container image "openebs/zfs-driver:2.7.1" already present on machine
3m56s Warning Failed pod/openebs-zfs-localpv-node-7wzsp Error: failed to prepare subPath for volumeMount "chroot-zfs" of container "openebs-zfs-plugin"

• kubectl get pods -n openebs

NAME READY STATUS RESTARTS AGE
openebs-localpv-provisioner-f744f768f-47gd5 1/1 Running 0 68m
openebs-zfs-localpv-controller-5b7846bf9-lbdst 5/5 Running 0 68m
openebs-zfs-localpv-node-7wzsp 1/2 CreateContainerConfigError 0 68m

• kubectl get zv -A -o yaml

apiVersion: v1
items:

• apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
  creationTimestamp: "2025-02-26T23:46:03Z"
  finalizers:
  • zfs.openebs.io/finalizer
    generation: 2
    labels:
    kubernetes.io/nodename: n200
    name: pvc-1018ba41-76b6-4de9-8f30-8a001867c0dd
    namespace: openebs-system
    resourceVersion: "177039"
    uid: 8ae34ca9-136e-49f9-bc6c-88462a9fdfb0
    spec:
    capacity: "209715200"
    compression: "off"
    dedup: "off"
    fsType: zfs
    ownerNodeID: n200
    poolName: zroot/k3s
    quotaType: quota
    recordsize: 128k
    volumeType: DATASET
    status:
    state: Ready
• apiVersion: zfs.openebs.io/v1
  kind: ZFSVolume
  metadata:
  creationTimestamp: "2025-02-26T02:55:48Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2025-02-26T21:36:00Z"
  finalizers:
  • zfs.openebs.io/finalizer
    generation: 3
    labels:
    kubernetes.io/nodename: n200
    name: pvc-ae6ed66c-12d2-46b1-bb4e-65a96855fa90
    namespace: openebs
    resourceVersion: "159677"
    uid: 2aff258c-c5e6-4863-8ccb-2dadb0c478d5
    spec:
    capacity: "209715200"
    compression: "off"
    dedup: "off"
    fsType: zfs
    ownerNodeID: n200
    poolName: zroot/k3s
    quotaType: quota
    recordsize: 128k
    volumeType: DATASET
    status:
    state: Ready
    kind: List
    metadata:
    resourceVersion: ""

Environment:

• LocalPV-ZFS version

2.7.1 installed using helm chart 4.2.0

• Kubernetes version (use kubectl version):

Client Version: v1.33.0
Kustomize Version: v5.6.0
Server Version: v1.32.4+k3s1

• OS (e.g. from /etc/os-release):

NixOS 25.05 (Warbler)

[Abhinandan-Purkait]

Any thoughts here @niladrih @tiagolobocastro, seems like we have some issue on how we do this bit https://github.com/openebs/zfs-localpv/blob/develop/deploy/helm/charts/templates/configmap.yaml

[tiagolobocastro]

I have 2.3.2:

zfs-2.3.2-1
zfs-kmod-2.3.2-1

And it's working fine with my setup. I do have a slightly different way of configuring zfs, I use a "jump" script like so:

cat /host/sbin/zfs 
    #/bin/sh
    chroot /host /nix/store/wwnqymbkm8wv5yc6palbiignx7igfcas-zfs-user-2.3.2/bin/zfs "$@"

However in your case seems like it's the config map which is failing to mount as Abhinandan mentioned above

 3m56s Warning Failed pod/openebs-zfs-localpv-node-7wzsp Error: failed to prepare subPath for volumeMount "chroot-zfs" of container "openebs-zfs-plugin"

Anything on the kubelet logs which would add more info here?

[hrenard]

Might be related to an issue with util-linux, see: kubernetes/kubernetes#130999 & util-linux/util-linux#3474.