How to run Llama3-8b instruct model on multiple GPUs?

[aitechguy0105]

Deos llama.cpp support llama3-8b to run on multiple GPUs?

[gardner]

Why would you run it on multiple GPUs?

[aitechguy0105]

to test benchmark. other models seems to be supported on multiple GPUs.

[gardner]

It only actually runs inference on one GPU at a time. You get the benefit of the additional VRAM but not the benefit of the additional processing power. Llama 3 8B should work on multiple GPUs. Your question doesn't really provide a lot of context. To maximize the possibility of someone being able to understand your issue and to help you, please review this article on How To Ask Questions The Smart Way.

[aknirala]

One reason for trying to run on multiple GPUs, as in my case is, it does not fit on one. I have 2 Nividia 2080 tI, each has around 11GB of memory. By default it choses GPU:0. I was wondering if I can parallelize between the two GPUs to run it.

[gardner]

The Q4_K_M quant is about 4.92 GB. It should fit in 11 GB of VRAM with space left for inference.

It sounds like you might be interested in this discussion: #8725