Getting started - Trying to chat with Llama-3.2-3B on a M1 Macbook Air (CPU without ANY GPUs)

[jasonkaplan79]

For context, I did already search for related "getting started" discussions and was unable to find a CLEAR set of instructions on how to get started:

• Simple tutorial for beginers #1166
• Getting started. #2597
  I have spent several hours trying to figure this out on my own, finally posting to ask for help.

I have a basic MacBook Air (M1, 2020) with Apple M1 and 8GB of RAM. It has a CPU without ANY GPUs. I DO NOT want to run this using Docker.
Can llama.cpp support my hardware?

I want to run and be able to chat with: https://huggingface.co/meta-llama/Llama-3.2-3B

Here's what I tried:

1. Build locally on Mac CPU:

   • https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#build-llamacpp-locally
   • https://github.com/ggerganov/llama.cpp/blob/master/docs/build.md#cpu-build
   • git clone https://github.com/ggerganov/llama.cpp
   • cd llama.cpp
   • cmake -B build
   • cmake --build build --config Release
     The build part worked, but I was unable to proceed. Since I couldnt find a clear set of getting started instructions, I then jumped to try this: Simple tutorial for beginers #1166
     BUT IT DID NOT WORK (at least, I couldn't figure out what to do next)

2. Install from Mac Command line: https://github.com/ggerganov/llama.cpp/blob/master/docs/install.md

   • brew install llama.cpp
     The installation worked, but again, I was unable to proceed.
     What do I do next?

MY GOAL:

I want to do 2 things:

1. Chat with https://huggingface.co/meta-llama/Llama-3.2-3B from the Terminal / Command Line
2. More importantly, make the chat available from a web interface on my localhost. Ex: http://127.0.0.1:8000/chat (and eventually on a Digital ocean Droplet running a CPU - NO GPUs)

Is this possible?

[adarmiento]

This helped me achieving the same https://zahiralam.com/blog/installing-llama-3-2-on-mac-m1-m2-and-m3-your-gateway-to-ai-power/