Glibc version incompatibility on Ubuntu 24.04 hosts with NVidia runtime

[Hboni]

Describe the bug
When I want to launch interface like brainvisa, on my Ubuntu 24.04 computer, I have an import error with the message :

ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /.singularity.d/libs/libGLdispatch.so.0)

This error occurs after an update of the drivers of my computer (NVIDIA). Thanks to @sapetnioc, we saw that libc version inside the container is 2.35, however the error mention the need of 2.38. We suspect that drivers of Ubuntu 24.04 requires recent libc which is not available in the container.
If I specify opengl=software I do not have the issue anymore, but the issue might happens more later.

To Reproduce
Steps to reproduce the behavior:

1. Install a brainvisa environment on a Ubuntu 24.04
2. Update nvidia drivers to <=550
3. Try to launch brainvisa

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

• Engine: Singularity
• Version of BrainVISA: master branch
• casa-dev image: 5.3-19 & 5.3-26

[denisri]

If libc 2.38 is backward-compatible with 2.35 (which should be OK) then we can try mounting the libc of the host system into the container in this situation. But automating this operation in casa-distro will not be easy, as we must do it only if the libc of the host system is newer than the one of the container. But to begin with, we must do the test.

[ylep]

The issue is confirmed on a newly remastered Ubuntu 24.04 workstation. New installs in NeuroSpin are now done with Ubuntu 24.04 unless 22.04 is specifically requested, so we have to find a solution relatively quickly. Pinning the issue.

From a technical point of view, I do not yet have a clear idea of the implications... but technically, the inclusion of these libraries is prompted by nvidia-container-cli (https://github.com/NVIDIA/libnvidia-container), so they surely are aware of this issue and maybe they already have a solution for it?

[denisri]

Could you try mounting the libc of the host system (Ubuntu 24.04) in the container (Ubuntu 22.04) ?

[denisri]

I have logged in your machine, and trying:
bv container_options='-B /usr/lib/x86_64-linux-gnu/libc.so.6,-B /lib64/ld-linux-x86-64.so.2' bash
it seemed to work.
However passing these options is not trivial for inexperienced users, and I would not know how to automatize it, knowing that they are needed only in some specific combinations of host/containers systems.

[denisri]

While we wait for a cleaner solution, maybe we can just add those libs to the site config in the neurospin install of brainvisa ?

[denisri]

I have done that (modified /drf/brainvisa/brainvisa-cea-5.1.2/conf/casa_distro.json on Neurospin network filesystem) and it seems to work. It would probably break the install if run from an older system but I assume all machines in NS are at least in Ubuntu 22.04 (?)
@ylep, can you test / confirm please ?

[ylep]

I did a quick test, the workaround seems to fix Ubuntu 24.04, unsurprisingly Ubuntu 22.04 still works. However, it seems to have broken Ubuntu 20.04 compatibility (we have few machines, but the graphical frontend of the cluster is one of them):

/bin/sh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /bin/sh)
/bin/sh: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /bin/sh)

[denisri]

Aha, yes if there are still machines older than the image, then the fix will break them :(
I didn't remember that there were also such systems on the network. OK I will remove the fix for now.

[denisri]

I have attempted a fix: brainvisa/casa-distro@c604093)

[denisri]

backported in brainvisa/casa-distro@5a5f0f7

[ylep]

The issue was still present on NeuroSpin's workstations, so I have deployed Denis' workaround to the shared install on /drf/brainvisa.

We need to decide how to proceed with that issue (new release?) because as of now, we cannot use BrainVISA on newly installed or updated NeuroSpin laptops; and external users may run into that issue as well...