Issue running `llama.cpp` example with RX 9070

[hameerabbasi]

Steps to reproduce:

1. Clone repo (current main; commit 32a8324f086335de3743a23b9f54c95c0ffe4e8c).
2. cd {repo_dir}/examples/llama.cpp
3. Run the following:

pixi run benchmark | tee rock-the-conda.log
rocminfo | tee rocminfo.log
rocm-smi --showdriverversion | tee rocm-smi.driverversion.log

rock-the-conda.log
rocminfo.log
rocm-smi.driverversion.log

[traversaro]

It seems you have both gfx1201 and gfx1036 on your system, can you try to run the example one time with export HIP_VISIBLE_DEVICES=0 and another with export HIP_VISIBLE_DEVICES=1 to see if something changes? Otherwise, it could be that we did not build some dependency for gfx1201, I cleaned up the recipe to be consistent w.r.t. to that, I hope to push new builds soon.

[hameerabbasi]

GTX 9070/gfx1201 succeeds; but integrated graphics (gfx1036) fails with more-or-less the same error.

[traversaro]

GTX 9070/gfx1201 succeeds; but integrated graphics (gfx1036) fails with more-or-less the same error.

Ah ok, so that is the issue. This is kind of expected as we did not build the rocm package with gfx1036, also as gfx1036 has no official rocm support (see https://rocm.docs.amd.com/en/docs-7.0.0/compatibility/compatibility-matrix.html). The puzzling thing to me is that llama.cpp/rocm uses by default an unsupported GPU instead of the supported one, and the error is so confusing. Did you tried also with a non-conda install of llama.cpp+rocm ? My intuition is that this is not a conda-specific problem, but I may be wrong.

[hameerabbasi]

I run the following on my system:

docker run --privileged \
           --network=host \
           --device=/dev/kfd \
           --device=/dev/dri \
           --group-add video \
           --cap-add=SYS_PTRACE \
           --security-opt seccomp=unconfined \
           --shm-size 16G \
           --entrypoint "/app/llama-bench" \
           -e HIP_VISIBLE_DEVICES=0 \
           -v /home/habbasi/Code/rock-the-conda/examples/llama.cpp:/data \
           rocm/llama.cpp:llama.cpp-b6356_rocm6.4.3_ubuntu24.04_full \
           -m /data/gemma-3-1b-it-Q4_K_M.gguf

Strangely, the order of the devices is reversed inside the container; but the failure happens on both of them. I wonder if I'm doing something wrong; I removed --ipc=host because podman didn't seem to accept it with --shm-size 16G; however, adding any one fails on both devices.

So your solution is actually more bug-free than AMD in this situation. 😅