Skip to content

Commit 270faa6

Browse files
pweglikNorbertKlockiewicz
pweglik
and
NorbertKlockiewicz
authored
feat: Improved LLM public API (#243)
Added generate function, and some little reformats ### Type of change - [ ] Bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation update (improves or adds clarity to existing documentation) ### Tested on - [x] iOS - [ ] Android ### Related issues #226 ### Checklist - [x] I have performed a self-review of my code - [x] I have updated the documentation accordingly - [x] My changes generate no new warnings --------- Co-authored-by: Norbert Klockiewicz <[email protected]>
1 parent 630f586 commit 270faa6

File tree

15 files changed

+113
-82
lines changed

15 files changed

+113
-82
lines changed

‎.cspell-wordlist.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,4 @@ Infima
4646
sublabel
4747
Aeonik
4848
Lexend
49+
finetuned

‎README.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,10 @@ function MyComponent() {
6565

6666
```tsx
6767
const handleGenerate = async () => {
68-
const prompt = 'The meaning of life is';
68+
const chat = [{ role: 'user', content: 'What is the meaning of life?' }];
6969

70-
// Generate text based on your desired prompt
71-
await llama.runInference(prompt);
70+
// Chat completion
71+
await llama.generate(chat);
7272
console.log('Llama says:', llama.response);
7373
};
7474
```

‎android/src/main/java/com/swmansion/rnexecutorch/LLM.kt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ class LLM(
3838
}
3939
}
4040

41-
override fun runInference(
41+
override fun forward(
4242
input: String,
4343
promise: Promise,
4444
) {

‎docs/docs/natural-language-processing/useLLM.md

Lines changed: 19 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -69,13 +69,14 @@ const useLLM: ({
6969
}) => LLMType;
7070

7171
interface LLMType {
72-
messageHistory: MessageType[];
72+
messageHistory: Message[];
7373
response: string;
7474
isReady: boolean;
7575
isGenerating: boolean;
7676
downloadProgress: number;
7777
error: string | null;
78-
runInference: (input: string) => Promise<void>;
78+
forward: (input: string) => Promise<void>;
79+
generate: (messages: Message[], tools?: LLMTool[]) => Promise<void>;
7980
sendMessage: (message: string) => Promise<void>;
8081
deleteMessage: (index: number) => void;
8182
interrupt: () => void;
@@ -85,12 +86,12 @@ type ResourceSource = string | number;
8586

8687
type MessageRole = 'user' | 'assistant' | 'system';
8788

88-
interface MessageType {
89+
interface Message {
8990
role: MessageRole;
9091
content: string;
9192
}
9293
interface ChatConfig {
93-
initialMessageHistory: MessageType[];
94+
initialMessageHistory: Message[];
9495
contextWindowLength: number;
9596
systemPrompt: string;
9697
}
@@ -136,7 +137,7 @@ Given computational constraints, our architecture is designed to support only on
136137

137138
- **`systemPrompt`** - Often used to tell the model what is its purpose, for example - "Be a helpful translator".
138139

139-
- **`initialMessageHistory`** - An array of `MessageType` objects that represent the conversation history. This can be used to provide initial context to the model.
140+
- **`initialMessageHistory`** - An array of `Message` objects that represent the conversation history. This can be used to provide initial context to the model.
140141

141142
- **`contextWindowLength`** - The number of messages from the current conversation that the model will use to generate a response. The higher the number, the more context the model will have. Keep in mind that using larger context windows will result in longer inference time and higher memory usage.
142143

@@ -150,18 +151,19 @@ Given computational constraints, our architecture is designed to support only on
150151

151152
### Returns
152153

153-
| Field | Type | Description |
154-
| ------------------ | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
155-
| `messageHistory` | `MessageType[]` | State of the generated response. This field is updated with each token generated by the model |
156-
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model |
157-
| `isReady` | `boolean` | Indicates whether the model is ready |
158-
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response |
159-
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
160-
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load |
161-
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
162-
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. |
163-
| `runInference` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage` |
164-
| `interrupt` | `() => void` | Function to interrupt the current inference |
154+
| Field | Type | Description |
155+
| ------------------ | ----------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
156+
| `messageHistory` | `Message[]` | State of the generated response. This field is updated with each token generated by the model |
157+
| `response` | `string` | State of the generated response. This field is updated with each token generated by the model |
158+
| `isReady` | `boolean` | Indicates whether the model is ready |
159+
| `isGenerating` | `boolean` | Indicates whether the model is currently generating a response |
160+
| `downloadProgress` | `number` | Represents the download progress as a value between 0 and 1, indicating the extent of the model file retrieval. |
161+
| `error` | <code>string &#124; null</code> | Contains the error message if the model failed to load |
162+
| `sendMessage` | `(message: string, tools?: LLMTool[]) => Promise<void>` | Method to add user message to conversation. After model responds, `messageHistory` will be updated with both user message and model response. |
163+
| `deleteMessage` | `(index: number) => void` | Deletes all messages starting with message on `index` position. After deletion `messageHistory` will be updated. |
164+
| `generate` | `(messages: Message[], tools?: LLMTool[]) => Promise<void>` | Runs model to complete chat passed in `messages` argument. It doesn't manage conversation context. |
165+
| `forward` | `(input: string) => Promise<void>` | Runs model inference with raw input string. You need to provide entire conversation and prompt (in correct format and with special tokens!) in input string to this method. It doesn't manage conversation context. It is intended for users that need access to the model itself without any wrapper. If you want simple chat with model consider using `sendMessage` |
166+
| `interrupt` | `() => void` | Function to interrupt the current inference |
165167

166168
## Sending a message
167169

0 commit comments

Comments
 (0)