GPU Underutilization and CPU Bottleneck with exllamav3

[DanielusG]

I’ve tested multiple models (e.g., Devstral, Qwen Coder 30B, Qwen3 4B, Qwen3 14B) using ExLLaMAv3 on my hardware, and I consistently encounter the same issue:

• GPU underutilization: The GPU usage peaks at only 80%, even during inference.
• Power consumption: The GPU draws 80–100W (out of a 160W power limit).
• CPU bottleneck: During inference, one CPU core consistently reaches 100% utilization, suggesting a potential bottleneck on the CPU side.

My hardware configuration:

• CPU: AMD Ryzen 9 5900X
• GPU: NVIDIA RTX 5060 Ti 16GB
• RAM: 32GB DDR4
• OS: Arch Linux with CUDA 13 installed

Typical use: Cline

I suspect the CPU might be the limiting factor here. Could there be a specific setting, driver issue, or resource allocation problem on my end causing this behavior?

Translated with Qwen3 14b

[turboderp]

100% usage on a single CPU core during inference is normal and expected. It's how CUDA synchronizes by default, with a spin-wait to minimize lag. There is a device flag (cudaDeviceScheduleYield) to change the behavior, but it's difficult to use in a PyTorch project, and it's not really important because unless you're actually CPU-bottlenecked, the reported 100% usage is going to be more like 60% actual CPU work and 40% busy-waiting for the CUDA stream to synchronize up to the final memcpy that transfers the next sampled token from device to host memory.

Short version: 100% CPU usage on one core is normal, and you should see that whether or not you're CPU bottlenecked.

The 80% GPU usage could indicate a bottleneck, though, although it's still strange. If you were CPU bottlenecked you should be seeing less GPU usage on the 4B model which has a higher ratio of CPU to GPU workload than the 14B model. Also, your CPU should have adequate single-core performance so you shouldn't be reaching that point to begin with.

I can't rule out that the power limit might be causing issues. I'll need to do some experiments to see if there's something specific about this particular workload that doesn't agree with how the firmware tries to hit a power target on 50xx cards. But I am also working on the next version which will have significantly less CPU overhead. I expect it to be ready sometime next week.