You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+22-9Lines changed: 22 additions & 9 deletions
Original file line number
Diff line number
Diff line change
@@ -244,8 +244,7 @@ Each type of node owns a set of properties which affect the computational perfor
244
244
*`Input Prefix`: append before every user input
245
245
*`Input Suffix`: append after every user input
246
246
*`Should Output prompt`: whether the input prompt should be included in the output
247
-
*`Should Output Bos`: whether the special bos (beginning of sequence) token should be included in the output
248
-
*`Should Output Eos`: whether the special eos (ending of sequence) token should be included in the output
247
+
*`Should Output Special`: whether the special (e.g., beginning of sequence and ending of sequence) token should be included in the output
249
248
*`Context Size`: number of tokens the model can process at a time
250
249
*`N Predict`: number of new tokens to generate, generate infinite sequence if -1
251
250
*`N Keep`: when the model run out of `context size`, it starts to forget about earlier context, set this variable to force the model to keep a number of the earliest tokens to keep the conversation relevant
@@ -258,6 +257,8 @@ Each type of node owns a set of properties which affect the computational perfor
258
257
*`Min P`: only sample from tokens with at least this probability, disabledd if 0.0
259
258
*`N Thread`: number of cpu threads to use
260
259
*`N GPU Layer`: number of layer offloaded to GPU
260
+
*`Main GPU`: the main GPU for computation
261
+
*`Split Mode`: how the computation will be distributed if there are multiple GPU in your systemm (0: None, 1: Layer, 2: Row)
261
262
*`Escape`: process escape character in input prompt
262
263
*`N Batch`: maximum number of tokens per iteration during continuous batching
263
264
*`N Ubatch`: maximum batch size for computation
@@ -315,11 +316,10 @@ Suppose you want to generate a character with:
315
316
*`weapon`: either "sword", "bow", or "wand
316
317
*`description`: a text with minimum 10 character
317
318
318
-
You should first create a GDLlama node, and turn `Should Output prompt`, `Should Output Bos`, and `Should Output Eos` off either by inspector or by script:
319
+
You should first create a GDLlama node, and turn `Should Output prompt`and `Should Output Special` off either by inspector or by script:
319
320
```
320
321
should_output_prompt = false
321
-
should_output_bos = false
322
-
should_output_eos = false
322
+
should_output_special = false
323
323
```
324
324
325
325
@@ -408,15 +408,16 @@ Besides the functions and signals from GDEmbedding, LlmDB has a few more functio
408
408
*`has_id(id: String, p_table_name: String) -> bool`: whether the table has a specific id stored
409
409
*`split_text(text: String) -> PackedStringArray`: split a piece of text first by all `Absolute Separators`, then by one of the appropiate `Chunk Separators`, such that any text chunk is shorter than `Chunk Size` (measured in character), and the overlap is close to but not greater than `Chunk Overlap`. If the algorithm failed to satisfy the contraints, there will be an error message printed out and the returned chunk will be greater than the `Chunk Size`
410
410
*`store_text_by_id(id: String, text: String)`: split the text and store the chunks in the database, be aware that `store_meta` should have been called previously such that the `id` with the corresponding meta is already in the database
411
-
*`run_store_text_by_id(id: String, text: String) -> Error`: run `store_text_by_id` in background
411
+
*`run_store_text_by_id(id: String, text: String) -> Error`: run `store_text_by_id` in background, emits `store_text_finished` signal when finished
412
412
*`store_text_by_meta(meta_dict: Dictionary, text: String)`: split the text and store the chunks in the database with the metadata defined in `meta_dict`, be aware that the metadata should be valid, every key should be a name stored in the `.meta` property and the corresponding type should be correct
413
-
*`run_store_text_by_meta(meta_dict: Dictionary, text: String) -> Error` run `store_text_by_meta` in background
413
+
*`run_store_text_by_meta(meta_dict: Dictionary, text: String) -> Error` run `store_text_by_meta` in background, emits `store_text_finished` signal when finished
414
414
*`retrieve_similar_texts(text: String, where: String, n_results: int) -> PackedStringArray`: retrieve `n_results` most similar text chunks to `text`, `where` should be empty or an sql WHERE clause to filter the chunks by metadata
run `retrieve_similar_texts` in background, and emits a `retrieve_similar_texts_finished` signal once it is done
417
417
418
418
### Signals
419
419
420
+
*`store_text_finished`: emitted when `run_store_text_by_id` or `run_store_text_by_meta` is finished
420
421
*`retrieve_similar_texts_finished(array: PackedStringArray)`: contains an array of `String`, emitted when `run_retrieve_similar_texts` is finished
421
422
422
423
## LlmDBMetaData
@@ -436,14 +437,26 @@ There are 4 static functions to create LlmDBMetaData
436
437
437
438
# FAQ
438
439
439
-
1. Does it support languages other than English?
440
+
1. You have an issue, how to get more debug message?
441
+
442
+
Turn on `Verbose stdout` in `Project Settings`, consider running Godot from a terminal to get additional logging messages.
443
+
444
+
2. Does it support languages other than English?
440
445
441
446
Yes, the plugin uses utf8 encoding so it has multilingual support naturally. However, a language model may be trained with English data only and it won't be able to generate text other than English, choose the language model based on your need.
442
447
443
-
2. Observing strange tokens in generated text, such as `<eot_id>` when `Should Output Eos` is off.
448
+
3. Observing strange tokens in generated text, such as `<eot_id>` when `Should Output Special` is off.
444
449
445
450
You are always welcome to open an issue. However, be aware that the standard of GGUF format can be changed to support new features and models, such that the bug can come from the model side instead of within this plugin. For example, some older llama 3 GGUF model may not be compatible with the latest format, you may try to search for a newer model with fixes such as [this](https://huggingface.co/NikolayKozloff/Meta-Llama-3-8B-Instruct-bf16-correct-pre-tokenizer-and-EOS-token-Q8_0-Q6_k-Q4_K_M-GGUF/tree/main).
446
451
452
+
4. You are runningg Arch linux (or its derivatives such as Manjaro) and your Godot Editor crash.
453
+
454
+
The Arch build of Godot is bugged, download Godot from the official website instead.
455
+
456
+
5. You have a discrete GPU and you see `unable to load model` error, you have make sure that the model parameters are correctly set.
457
+
458
+
There is currently a bug on vulkan backend if you have multiple drivers installed for the same GPU, try to turn `Split Mode` to `NONE` (0) and set your `Main GPU` manually (starting from 0) to see if it works.
459
+
447
460
# Compile from source
448
461
Install build tools and Vulkan SDK for your operating system, then clone this repository
0 commit comments