`initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown`

[khteh]

Ubuntu 25.04
Dockerfile:

# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
RUN apt install -y software-properties-common apt-transport-https curl sudo gnupg pipenv unzip dnsutils wget python3 python3-pip
RUN curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
RUN curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
RUN apt update -y
ENV NVIDIA_CONTAINER_TOOLKIT_VERSION 1.17.8-1
RUN apt install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}
RUN nvidia-ctk runtime configure --runtime=containerd

k8s:

clientVersion:
  buildDate: "2024-10-16T15:15:29Z"
  compiler: gc
  gitCommit: cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a
  gitTreeState: clean
  gitVersion: v1.27.16
  goVersion: go1.22.5
  major: "1"
  minor: "27"
  platform: linux/amd64
kustomizeVersion: v5.0.1
serverVersion:
  buildDate: "2024-10-16T15:16:32Z"
  compiler: gc
  gitCommit: cbb86e0d7f4a049666fac0551e8b02ef3d6c3d9a
  gitTreeState: clean
  gitVersion: v1.27.16
  goVersion: go1.22.5
  major: "1"
  minor: "27"
  platform: linux/amd64

Error:

  Warning  Failed     5s (x2 over 11s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli.real: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
  Warning  BackOff  2s (x4 over 5s)  kubelet  Back-off restarting failed container ollama in pod ollama-0_default(ada1f8c4-efb0-4eb4-9ae1-dd44e376fe78)

[khteh]

canonical/microk8s#5247

[elezar]

@khteh could you please provide more context on why you're installing the toolkit in a docker container?

[khteh]

I was referring to https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
"container" toolkit. So I assume I need it in a docker container, no?

[elezar]

The NVIDIA Container Toolkit is installed on the host system along with the NVIDIA GPU Driver. The toolkit is responsible for ensuring that the user-mode driver files and GPU devices nodes that are required for GPU usage are present in a container that is started.

[khteh]

So, is the error I saw in the pod log initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown due to issue I filed here?

canonical/microk8s#5247

[elezar]

The contents of the issue are the same. I would suggest that you follow up with the microk8s maintainers or check their documentation on what is required to start a cluster with GPU support. For example, I believe it is required to have the NVIDIA Driver installed on the host. If this is present, which version is installed (provide the output of nvidia-smi -L).

[khteh]

GPU 0: NVIDIA RTX A2000 Laptop GPU (UUID: GPU-9762feba-cea4-7981-7353-533400b79c72)

[khteh]

I still have doubt. Looking at the warning/error:

  Warning  Failed     5s (x2 over 11s)  kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli.real: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown
  Warning  BackOff  2s (x4 over 5s)  kubelet  Back-off restarting failed container ollama in pod ollama-0_default(ada1f8c4-efb0-4eb4-9ae1-dd44e376fe78)

Which system is looking for libnvidia-ml.so.1 and why is it missing? If it is the host, why does the pod start up complain about the file missing? Shoudn't this error appear in the host log system instead of in a pod?

[elezar]

Let's continue this discussion on the microk8s issue. Please provide more information on your system and veryfy that you've followed the microk8s documentation on starting a cluster with GPU support.