Skip to content

Commit f01bd41

Browse files
pweglikpweglik
authored
fix: Improvements and fixes in useLLM (#278)
## Description Solving few issues: - Example app crashed when model was deleted during generation. This is prevented now and we throw error if library users try it. - state of hook and controller is properly reseted when changing models. - `preventLoad` props in hook so you can postpone loading model until you're ready - fixes in state management in hook ### Type of change - [x] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [x] iOS - [ ] Android ### Screenshots Preventing model swap during generation: https://github.com/user-attachments/assets/0cfec0e9-02fc-414c-ac3a-5f6e767ea7ea ### Related issues #235 #237 #241 #264 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings
1 parent 9d1e69e commit f01bd41

File tree

9 files changed

+251
-143
lines changed

9 files changed

+251
-143
lines changed

‎docs/docs/natural-language-processing/useLLM.md

Lines changed: 35 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,6 @@ import {
3838
LLAMA3_2_TOKENIZER_CONFIG,
3939
} from 'react-native-executorch';
4040

41-
const messageHistory = [
42-
{ role: 'user', content: 'Hello' },
43-
{ role: 'assistant', content: 'Hi, how can I help you?' },
44-
];
45-
4641
const llm = useLLM({
4742
modelSource: LLAMA3_2_1B,
4843
tokenizerSource: LLAMA3_2_TOKENIZER,
@@ -58,14 +53,12 @@ const useLLM: ({
5853
modelSource,
5954
tokenizerSource,
6055
tokenizerConfigSource,
61-
chatConfig,
62-
toolsConfig,
56+
preventLoad = false,
6357
}: {
6458
modelSource: ResourceSource;
6559
tokenizerSource: ResourceSource;
6660
tokenizerConfigSource: ResourceSource;
67-
chatConfig?: Partial<ChatConfig>;
68-
toolsConfig?: ToolsConfig;
61+
preventLoad?: boolean;
6962
}) => LLMType;
7063

7164
interface LLMType {
@@ -75,6 +68,13 @@ interface LLMType {
7568
isGenerating: boolean;
7669
downloadProgress: number;
7770
error: string | null;
71+
configure: ({
72+
chatConfig,
73+
toolsConfig,
74+
}: {
75+
chatConfig?: Partial<ChatConfig>;
76+
toolsConfig?: ToolsConfig;
77+
}) => void;
7878
forward: (input: string) => Promise<void>;
7979
generate: (messages: Message[], tools?: LLMTool[]) => Promise<void>;
8080
sendMessage: (message: string) => Promise<void>;
@@ -129,9 +129,33 @@ Given computational constraints, our architecture is designed to support only on
129129

130130
**`modelSource`** - A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) section.
131131

132-
**`tokenizerSource`** - URL to the JSON file which contains the tokenizer
132+
**`tokenizerSource`** - URL to the JSON file which contains the tokenizer.
133+
134+
**`tokenizerConfigSource`** - URL to the JSON file which contains the tokenizer config.
133135

134-
**`tokenizerConfigSource`** - URL to the JSON file which contains the tokenizer config
136+
**`preventLoad?`** - Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.
137+
138+
### Returns
139+
140+
| Field | Type | Description |
141+
| ------------------ | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
142+
| `messageHistory` | `Message[]` | History containing all messages in conversation. This field is updated after model responds to `sendMessage`. |
143+
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model. |
144+
| `isReady` | `boolean` | Indicates whether the model is ready. |
145+
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response. |
146+
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
147+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
148+
| `configure` | `({ chatConfig?: Partial<ChatConfig>, toolsConfig?: ToolsConfig }) => void` | Configures chat and tool calling. See more details in [configuring the model](#configuring-the-model). |
149+
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
150+
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
151+
| `generate` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
152+
| `forward` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage`. |
153+
| `interrupt` | `() => void` | Function to interrupt the current inference. |
154+
155+
## Configuring the model
156+
157+
To configure model (i.e. change system prompt, load initial conversation history or manage tool calling) you can use
158+
`configure` method. It accepts object with following fields:
135159

136160
**`chatConfig`** - Object configuring chat management, contains following properties:
137161

@@ -149,22 +173,6 @@ Given computational constraints, our architecture is designed to support only on
149173

150174
- **`displayToolCalls`** - If set to true, JSON tool calls will be displayed in chat. If false, only answers will be displayed.
151175

152-
### Returns
153-
154-
| Field | Type | Description |
155-
| ------------------ | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
156-
| `messageHistory` | `Message[]` | State of the generated response. This field is updated with each token generated by the model |
157-
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model |
158-
| `isReady` | `boolean` | Indicates whether the model is ready |
159-
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response |
160-
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
161-
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load |
162-
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
163-
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
164-
| `generate` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
165-
| `forward` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage` |
166-
| `interrupt` | `() => void` | Function to interrupt the current inference |
167-
168176
## Sending a message
169177

170178
In order to send a message to the model, one can use the following code:

0 commit comments

Comments
 (0)