Sharing GPU between Docker containers

[AvinashKushwah]

I am running multiple Docker containers using the following command, which exposes all GPU devices to each container:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

Scenario:

Container C1 runs Process P1, which internally needs to use GPU IDs 1, 2, 3, and 4.

Container C2 runs Process P2 at the same time.

Since both containers receive access to all GPU devices, I want to understand the expected behavior:

Question

Is it possible that P2 in C2 also selects GPUs 1, 2, 3, and 4, leading to GPU resource contention, crashes, or failed inference/training runs?

[elezar]

From a driver perspective there is no difference between a containerised application and a regular application and a container does not provide any additional resource isolation from a GPU perspective. This means that in the scenario you describe, both processes will have access to all GPUs and have the same resource contention issues that would occur were these to be running directly on the host.

[AvinashKushwah]

Then how does NVIDIA handle this scenario? For example, if two processes running on the host or inside containers need to run a model that requires multiple GPUs for inference, both processes might try to use the same GPU set or have overlapping GPU allocations, or Internally it provide different set of GPU to each process?