Skip to content

Commit 846ac06

Browse files
mkopcinsmkopcins
authored
Merge branch 'v0.4.0-rc1' into @md/s2t_streaming
2 parents a4c262a + f01bd41 commit 846ac06

File tree

13 files changed

+271
-152
lines changed

13 files changed

+271
-152
lines changed

‎README.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,18 @@
88

99
**ExecuTorch** is a novel framework created by Meta that enables running AI models on devices such as mobile phones or microcontrollers. React Native ExecuTorch bridges the gap between React Native and native platform capabilities, allowing developers to run AI models locally on mobile devices with state-of-the-art performance, without requiring deep knowledge of native code or machine learning internals.
1010

11+
**Table of contents:**
12+
13+
- [Compatibility](#compatibility)
14+
- [Ready-made models 🤖](#readymade-models-)
15+
- [Documentation 📚](#documentation-)
16+
- [🦙 Quickstart - Running Llama](#-quickstart---running-llama)
17+
- [Minimal supported versions](#minimal-supported-versions)
18+
- [Examples 📲](#examples-)
19+
- [Warning](#warning)
20+
- [License](#license)
21+
- [What's next?](#whats-next)
22+
1123
## Compatibility
1224

1325
React Native Executorch supports only the [New React Native architecture](https://reactnative.dev/architecture/landing-page).

‎docs/docs/natural-language-processing/useLLM.md

Lines changed: 35 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -38,11 +38,6 @@ import {
3838
LLAMA3_2_TOKENIZER_CONFIG,
3939
} from 'react-native-executorch';
4040

41-
const messageHistory = [
42-
{ role: 'user', content: 'Hello' },
43-
{ role: 'assistant', content: 'Hi, how can I help you?' },
44-
];
45-
4641
const llm = useLLM({
4742
modelSource: LLAMA3_2_1B,
4843
tokenizerSource: LLAMA3_2_TOKENIZER,
@@ -58,14 +53,12 @@ const useLLM: ({
5853
modelSource,
5954
tokenizerSource,
6055
tokenizerConfigSource,
61-
chatConfig,
62-
toolsConfig,
56+
preventLoad = false,
6357
}: {
6458
modelSource: ResourceSource;
6559
tokenizerSource: ResourceSource;
6660
tokenizerConfigSource: ResourceSource;
67-
chatConfig?: Partial<ChatConfig>;
68-
toolsConfig?: ToolsConfig;
61+
preventLoad?: boolean;
6962
}) => LLMType;
7063

7164
interface LLMType {
@@ -75,6 +68,13 @@ interface LLMType {
7568
isGenerating: boolean;
7669
downloadProgress: number;
7770
error: string | null;
71+
configure: ({
72+
chatConfig,
73+
toolsConfig,
74+
}: {
75+
chatConfig?: Partial<ChatConfig>;
76+
toolsConfig?: ToolsConfig;
77+
}) => void;
7878
forward: (input: string) => Promise<void>;
7979
generate: (messages: Message[], tools?: LLMTool[]) => Promise<void>;
8080
sendMessage: (message: string) => Promise<void>;
@@ -129,9 +129,33 @@ Given computational constraints, our architecture is designed to support only on
129129

130130
**`modelSource`** - A string that specifies the location of the model binary. For more information, take a look at [loading models](../fundamentals/loading-models.md) section.
131131

132-
**`tokenizerSource`** - URL to the JSON file which contains the tokenizer
132+
**`tokenizerSource`** - URL to the JSON file which contains the tokenizer.
133+
134+
**`tokenizerConfigSource`** - URL to the JSON file which contains the tokenizer config.
133135

134-
**`tokenizerConfigSource`** - URL to the JSON file which contains the tokenizer config
136+
**`preventLoad?`** - Boolean that can prevent automatic model loading (and downloading the data if you load it for the first time) after running the hook.
137+
138+
### Returns
139+
140+
| Field | Type | Description |
141+
| ------------------ | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
142+
| `messageHistory` | `Message[]` | History containing all messages in conversation. This field is updated after model responds to `sendMessage`. |
143+
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model. |
144+
| `isReady` | `boolean` | Indicates whether the model is ready. |
145+
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response. |
146+
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
147+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load. |
148+
| `configure` | `({ chatConfig?: Partial<ChatConfig>, toolsConfig?: ToolsConfig }) => void` | Configures chat and tool calling. See more details in [configuring the model](#configuring-the-model). |
149+
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
150+
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
151+
| `generate` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
152+
| `forward` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage`. |
153+
| `interrupt` | `() => void` | Function to interrupt the current inference. |
154+
155+
## Configuring the model
156+
157+
To configure model (i.e. change system prompt, load initial conversation history or manage tool calling) you can use
158+
`configure` method. It accepts object with following fields:
135159

136160
**`chatConfig`** - Object configuring chat management, contains following properties:
137161

@@ -149,22 +173,6 @@ Given computational constraints, our architecture is designed to support only on
149173

150174
- **`displayToolCalls`** - If set to true, JSON tool calls will be displayed in chat. If false, only answers will be displayed.
151175

152-
### Returns
153-
154-
| Field | Type | Description |
155-
| ------------------ | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
156-
| `messageHistory` | `Message[]` | State of the generated response. This field is updated with each token generated by the model |
157-
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model |
158-
| `isReady` | `boolean` | Indicates whether the model is ready |
159-
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response |
160-
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
161-
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load |
162-
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
163-
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
164-
| `generate` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
165-
| `forward` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage` |
166-
| `interrupt` | `() => void` | Function to interrupt the current inference |
167-
168176
## Sending a message
169177

170178
In order to send a message to the model, one can use the following code:

0 commit comments

Comments
 (0)