@@ -66,7 +66,7 @@ python3 torchchat.py --help
66
66
### Download Weights
67
67
Most models use HuggingFace as the distribution channel, so you will need to create a HuggingFace account.
68
68
69
- [ prefix default] : HF_TOKEN="${SECRET_HF_TOKEN_PERIODIC}"
69
+ [ prefix default ] : HF_TOKEN="${SECRET_HF_TOKEN_PERIODIC}"
70
70
Create a HuggingFace user access token [ as documented here] ( https://huggingface.co/docs/hub/en/security-tokens ) .
71
71
Log into huggingface:
72
72
```
@@ -195,6 +195,22 @@ export TORCHCHAT_ROOT=${PWD}
195
195
./scripts/install_et.sh
196
196
```
197
197
198
+ ### Test it out using our Executorch runner
199
+ Build the runner
200
+ ``` bash
201
+ scripts/build_native.sh et
202
+ ```
203
+
204
+ ** Get a PTE file if you don't have one already**
205
+ ```
206
+ python3 torchchat.py export llama3 --quantize config/data/mobile.json --output-pte-path llama3.pte
207
+ ```
208
+
209
+ Execute using the runner
210
+ ``` bash
211
+ cmake-out/et_run llama3.pte -z ~ /.torchchat/model-cache/meta-llama/Meta-Llama-3-8B-Instruct/tokenizer.model -i " Once upon a time"
212
+ ```
213
+
198
214
### Export for mobile
199
215
The following example uses the Llama3 8B Instruct model.
200
216
@@ -216,7 +232,7 @@ For more details on quantization and what settings to use for your use
216
232
case visit our [ Quanitization documentation] ( docs/quantization.md ) or
217
233
run ` python3 torchchat.py export `
218
234
219
- [ end default] :
235
+ [ end default] :
220
236
221
237
### Deploy and run on iOS
222
238
@@ -344,18 +360,18 @@ solely responsible for complying with all such obligations.
344
360
345
361
346
362
### Disclaimer
347
- The torchchat Repository Content is provided without any guarantees about
348
- performance or compatibility. In particular, torchchat makes available
349
- model architectures written in Python for PyTorch that may not perform
350
- in the same manner or meet the same standards as the original versions
351
- of those models. When using the torchchat Repository Content, including
352
- any model architectures, you are solely responsible for determining the
353
- appropriateness of using or redistributing the torchchat Repository Content
354
- and assume any risks associated with your use of the torchchat Repository Content
355
- or any models, outputs, or results, both alone and in combination with
356
- any other technologies. Additionally, you may have other legal obligations
357
- that govern your use of other content, such as the terms of service for
358
- third-party models, weights, data, or other technologies, and you are
363
+ The torchchat Repository Content is provided without any guarantees about
364
+ performance or compatibility. In particular, torchchat makes available
365
+ model architectures written in Python for PyTorch that may not perform
366
+ in the same manner or meet the same standards as the original versions
367
+ of those models. When using the torchchat Repository Content, including
368
+ any model architectures, you are solely responsible for determining the
369
+ appropriateness of using or redistributing the torchchat Repository Content
370
+ and assume any risks associated with your use of the torchchat Repository Content
371
+ or any models, outputs, or results, both alone and in combination with
372
+ any other technologies. Additionally, you may have other legal obligations
373
+ that govern your use of other content, such as the terms of service for
374
+ third-party models, weights, data, or other technologies, and you are
359
375
solely responsible for complying with all such obligations.
360
376
361
377
@@ -401,4 +417,3 @@ code in this distribution is covered by the MIT and Apache Open Source
401
417
licenses.) However you may have other legal obligations that govern
402
418
your use of content, such as the terms of service for third-party
403
419
models.
404
-
0 commit comments