How to(Can I) load a bigger model with Metal build?

[Foxtr0t1337]

On my MacBook M2 I tried Metal and CPU.
Metal is very fast.
But it can't work with large models.
ggml_metal_graph_compute: command buffer 0 failed with status 5

Can I load part of the weight into GPU buffers and use CPU to compute the rest?
Is this supported right now?
If not will it be faster than CPU?

[ianscrivener]

Short answer - no
Long answer - read this recent dicussion (3026)

[Foxtr0t1337]

It did not answer my question about CPU + GPU using Metal.

[ianscrivener]

there are few discussions on this eg: Can we use both the CPU and GPU(and may the NE) on unified memory systems(Mac)?

https://github.com/search?q=repo%3Aggerganov%2Fllama.cpp+mac+metal&type=discussions

[KerfuffleV2]

There's #2182 which can let you adjust the unified memory split. Assuming your unified memory is close to your total memory, you'd pretty much only run into the limit if the model was bigger your available memory. At that point, it's getting read off the disk per token and that's way more likely to be the bottleneck than anything else. So even if you run the actual calculations on the GPU rather than CPU you probably won't see much of a performance increase - it's going to be spending most of its time waiting on IO.