After the directory change, what should be the change in this command

[ramanrewati]

! ./llama.cpp/main -m ./quantized_model/Q4_K_M.gguf -n 90 --repeat_penalty 1.0 --color -i -r "User:" -f llama.cpp/prompts/reason-act.txt
This line gives me error, bin/bash: line 1: ./llama.cpp/main: No such file or directory which is probably because the repository might have changed it's structure, what should the new cli command must be?

[danbev]

If you have built with cmake following the instructions in build.md the executable will be build/bin/llama-cli.

[ramanrewati]

i can inference it, but can't do the quantization, what is the exact syntax for cli quantization.
I tried the llama-quantize executable, but it fails to do so, I'm trying to quantise deepseek distilled-qwen-1.5B, here are the error logs.

!cd llama.cpp/build/bin && ./llama-quantize /content/quantized_model/FP16.gguf ./quantized_model/Q4_K_M.gguf q4_K_M
main: build = 4607 (aa6fb132)
main: built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu
main: quantizing '/content/quantized_model/FP16.gguf' to './quantized_model/Q4_K_M.gguf' as Q4_K_M
llama_model_loader: loaded meta data with 26 key-value pairs and 339 tensors from /content/quantized_model/FP16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Original_Model
llama_model_loader: - kv   3:                         general.size_label str              = 1.8B
llama_model_loader: - kv   4:                            general.license str              = mit
llama_model_loader: - kv   5:                          qwen2.block_count u32              = 28
llama_model_loader: - kv   6:                       qwen2.context_length u32              = 131072
llama_model_loader: - kv   7:                     qwen2.embedding_length u32              = 1536
llama_model_loader: - kv   8:                  qwen2.feed_forward_length u32              = 8960
llama_model_loader: - kv   9:                 qwen2.attention.head_count u32              = 12
llama_model_loader: - kv  10:              qwen2.attention.head_count_kv u32              = 2
llama_model_loader: - kv  11:                       qwen2.rope.freq_base f32              = 10000.000000
llama_model_loader: - kv  12:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  13:                          general.file_type u32              = 1
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = deepseek-r1-qwen
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151646
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151643
llama_model_loader: - kv  21:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  22:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  23:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {% if not add_generation_prompt is de...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  141 tensors
llama_model_loader: - type  f16:  198 tensors
llama_model_quantize: failed to quantize: basic_ios::clear: iostream error
main: failed to quantize model from '/content/quantized_model/FP16.gguf'

[danbev]

llama-quantize is documented here, but I can't spot anything wrong with the command your are using. Perhaps you can open a new discussion for this specific issue about quantization to get more eyes on it.