1. how to use it like chatgpt? 2. how to

[hiqsociety]

1. how to use it like chatgpt?
2. how to make it write more than 10000 words? it's not working here. it's always stopping real quick, like 500words only or so.
   ./main -m models/llama-2-7b-guanaco-qlora.Q2_K.gguf -p "these are the 10000 words as an original and unique article that passes artificial intelligence detection on how to use artificial intelligence to do seo:\nStep 1:" -n 400 -e

[staviq]

1. If you mean better interface, try the server example, it runs through the browser, or go with ooba webui or koboldcpp, they both use or can use llama.cpp as their backend.
2. You are telling it to stop at 400 tokens, that's what -n 400 does. But if you actually want 10k long output, you will need a model supporting big enough context, because otherwise the model will forget what you asked it to do halfway through. For coherent output with a single prompt, both the prompt and the answer has to fit in the context, and that model has a native context of only 4096. You can increase the context size with -c but RoPE scaling will be used and that tends to decrease generated output quality. You are also using Q2 quantized model, which makes its output quality noticeably worse from the base model, and it's not recommended for general use.

[hiqsociety]

@staviq thx for the explanation.

1. i have downloaded Q5 and Q8. how do i make it write 2500-10k words output?

-n 10000 ?
-c 10000 ?

2. how to use it like an api that can remember previous responses?

i've just tried out the server feature but it has a UI. I would like to use it like an API but when i used as api, it generates very few token and i'm not sure how to make it remember previous responses because it cant complete the article like chatgpt that just stops responding halfway.

[staviq]

If you run ./main -h you will get a short description of all parameters.

You can just do -n -1 that means no token limit at all ( but this can still "overflow" the context and forget your question )

You can do -n -2 which means token limit is the same as context size.

It should "work" with -c 10000 -n -2

If you want API, server does provide API, you can use it through http queries, without the browser interface. If you search this repo for "curl" you'll find examples of people calling the server API directly.

Hovewer, API does not "remember" previous questions, is universal, so it can't do any remembering, otherwise you would have to restart the API to have another conversation. Instead, typically you give it a prompt through the API, and you get a generated response, and in order to "continue" the conversation through API, you glue the previous question, and generated answer, with your follow-up question, and send that, appending the generated answer and subsequent questions to the conversation each time, on your end of the API.

[hiqsociety]

@staviq actually this is in server mode now and i have issues generated using server mode. previously i did -c 10240 but it didnt continue response.

./server --host 0.0.0.0 --port 8080 -m ./models/llama-2-13b-guanaco-qlora.Q8_0.gguf -c 10240 -n -2 --mlock -t 4
error: unknown argument: -n
usage: ./server [options]

options:
  -h, --help            show this help message and exit
  -v, --verbose         verbose output (default: disabled)
  -t N, --threads N     number of threads to use during computation (default: 3)
  -c N, --ctx-size N    size of the prompt context (default: 512)
  --rope-freq-base N    RoPE base frequency (default: 10000.0)
  --rope-freq-scale N   RoPE frequency scaling factor (default: 1)
  -b N, --batch-size N  batch size for prompt processing (default: 512)
  --memory-f32          use f32 instead of f16 for memory key+value (default: disabled)
                        not recommended: doubles context memory required and no measurable increase in quality
  --mlock               force system to keep model in RAM rather than swapping or compressing
  --no-mmap             do not memory-map model (slower load but may reduce pageouts if not using mlock)
  --numa                attempt optimizations that help on some NUMA systems
  -m FNAME, --model FNAME
                        model path (default: models/7B/ggml-model-f16.gguf)
  -a ALIAS, --alias ALIAS
                        set an alias for the model, will be added as `model` field in completion response
  --lora FNAME          apply LoRA adapter (implies --no-mmap)
  --lora-base FNAME     optional model to use as a base for the layers modified by the LoRA adapter
  --host                ip address to listen (default  (default: 127.0.0.1)
  --port PORT           port to listen (default  (default: 8080)
  --path PUBLIC_PATH    path from which to serve static files (default examples/server/public)
  -to N, --timeout N    server read/write timeout in seconds (default: 600)
  --embedding           enable embedding vector output (default: disabled)

[staviq]

Oh, I think I forgot to mention, with server you set token limit in the web UI, below the prompt template.

[hiqsociety]

@staviq
1.

i dont see the token limit, if it's in the webui,
2. how do i do the server api then?

[staviq]

"Predictions"

[KerfuffleV2]

The technical side of the issue aside:

"these are the 10000 words as an original and unique article that passes artificial intelligence detection"

If you expect this to work, unfortunately your idea of what LLMs are capable of isn't very realistic. Even really enormous models like ChatGPT which is probably at least 180B parameters and uses a bunch of tricks to increase its results quality wouldn't be able to pull it off. The 7B model is like 25 times smaller and doesn't have the benefit of those tricks.

LLMs (even big ones) are pretty well known for not being too great at counting. Saying something like 10,000 words might make it write more stuff but it's not going to be a specific number of words.

Telling it to write something that passes AI detection just... isn't going to work. You might as well tell it to hack into your bank account and add a few more digits to the balance. It's about as capable at both of those tasks.

Don't trust LLMs, large or small (but particularly small) to output true and factual information. They basically won't tell you when they can't do something, they'll just say they did and sound very confident.

[hiqsociety]

@KerfuffleV2 alrighty, so how do i get it to write really long articles with continuations like what chatgpt4 can do now?

by the way, bing chat can pass AI detection so i was wondering which llama model can pass too.

[KerfuffleV2]

so how do i get it to write really long articles with continuations like what chatgpt4 can do now?

You can't really expect these small models to have the same capabilities as ChatGPT4 which is a huge model, running on very expensive hardware, with lots of tricks to increase it's abilities and understanding of instructions. Even local models 70 times larger than the one you're using still don't really approach that level — and "that level" also has plenty of shortcomings as well.

That said, you can experiment with different prompts. At least for the main example, it's possible to specify --ignore-eos which prevents the LLM from ending its response. It will write stuff until it hits the context or token generation limit. However, that doesn't mean it will use that space to write what you want though.

by the way, bing chat can pass AI detection so i was wondering which llama model can pass too.

Basically what staviq said. Also, "AI detection" is an arms race. If models can evade it, the heuristics of AI detection can be improved or changed to detect it. As far as I know, absolutely 0 effort and development has gone into making these open source/publicly available models evade existing AI detection approaches. The answer may well be that no LLaMA model can pass.

[staviq]

I would like to point out "AI detection" doesn't exist. What exists, are classification algorithms, somewhat able to detect known patterns in currently existing models, sort of.

Current "AI detection" approaches can detect AI when it is an AI, they can detect AI when it's a human, can detect a human when it's a human, and can detect a human when it's an AI.

Which makes them completely useless as you cannot conclude anything from their results.

In order to be in any way usable, they would have to get rid of at least one of those situations, and that's currently mathematically impossible.

For now, "AI detection" is a "key in a drawer" paradox, where you can't put a key in a drawer and still use it to lock the same drawer. Because AIs are trained on human created data, and you want to detect an AI saying something which it learned from human created data.

[hiqsociety]

@KerfuffleV2

1. there is no ignore-eos in server mode. possible to add this as a feature?
2. i thought a higher mmlu or something for a model would make it understand when to end the article generation within the word count it was mentioned.

maybe i was comparing this to copy.ai which is doing a fantastic job. i always thought some llama model will be "smarter" in knowing exactly what needs to be done coz i was always referencing here:

https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard

[staviq]

Try just once with ./main. You can do just ./main -m yourmodel -p "your prompt" -c 10000 -n -2 --ignore-eos. This should force the model to do 10k long output.

Run this, and you will see yourself, all the problems we are trying to explain. Output will just be poor quality. Local LLMs are just not good enough for this, yet.

Maybe in a year or so, or maybe llama3 will be capable of this, but we are just not there yet. The whole AI thing is still in its infancy, compared to what researchers think it could be.

[KerfuffleV2]

there is no ignore-eos in server mode. possible to add this as a feature?

It looks like the server supports setting logit bias but maybe there isn't a way to do it in the existing interface. I've never used the server so I can't help you with that. If you can figure out how to set the logit bias, then just set token ids 1 (and possibly 2) to -inf.

i thought a higher mmlu or something for a model would make it understand when to end the article generation within the word count it was mentioned.

MMLU tests a certain kind of "intelligence" but that's not really going to guarantee anything. If you know a person's IQ, do you know if they're going to do what you ask them to? Also "higher MMLU" is comparative. Higher than what? Maybe it does follow instructions better than the models with lower MMLU but 7B models in general are way down at the bottom of the leaderboard whether you sort by MMLU or the average score.

I think you could benefit from doing some research on the capabilities of these models before trying to use them. Also possibly reading about the tests so you get a sense of what exactly the LLM is doing to perform better or worse when tested that way.

maybe i was comparing this to copy.ai which is doing a fantastic job.

copy.ai is probably using a model 20 times bigger than the one you're testing here.

[hiqsociety]

@KerfuffleV2 @staviq hmm... ok thx