Skip to content

Commit c37d913

Browse files
committed
add ET Runner instructions
1 parent f54680e commit c37d913

File tree

1 file changed

+30
-15
lines changed

1 file changed

+30
-15
lines changed

README.md

Lines changed: 30 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ python3 torchchat.py --help
6666
### Download Weights
6767
Most models use HuggingFace as the distribution channel, so you will need to create a HuggingFace account.
6868

69-
[prefix default]: HF_TOKEN="${SECRET_HF_TOKEN_PERIODIC}"
69+
[prefix default]: HF_TOKEN="${SECRET_HF_TOKEN_PERIODIC}"
7070
Create a HuggingFace user access token [as documented here](https://huggingface.co/docs/hub/en/security-tokens).
7171
Log into huggingface:
7272
```
@@ -195,6 +195,22 @@ export TORCHCHAT_ROOT=${PWD}
195195
./scripts/install_et.sh
196196
```
197197

198+
### Test it out using our Executorch runner
199+
Build the runner
200+
```bash
201+
scripts/build_native.sh et
202+
```
203+
204+
**Get a PTE file if you don't have one already**
205+
```
206+
python3 torchchat.py export llama3 --quantize config/data/mobile.json --output-pte-path llama3.pte
207+
```
208+
209+
Execute using the runner
210+
```bash
211+
cmake-out/et_run llama3.pte -z ~/.torchchat/model-cache/meta-llama/Meta-Llama-3-8B-Instruct/tokenizer.model -i "Once upon a time"
212+
```
213+
198214
### Export for mobile
199215
The following example uses the Llama3 8B Instruct model.
200216

@@ -216,7 +232,7 @@ For more details on quantization and what settings to use for your use
216232
case visit our [Quanitization documentation](docs/quantization.md) or
217233
run `python3 torchchat.py export`
218234

219-
[end default]:
235+
[end default]:
220236

221237
### Deploy and run on iOS
222238

@@ -344,18 +360,18 @@ solely responsible for complying with all such obligations.
344360

345361

346362
### Disclaimer
347-
The torchchat Repository Content is provided without any guarantees about
348-
performance or compatibility. In particular, torchchat makes available
349-
model architectures written in Python for PyTorch that may not perform
350-
in the same manner or meet the same standards as the original versions
351-
of those models. When using the torchchat Repository Content, including
352-
any model architectures, you are solely responsible for determining the
353-
appropriateness of using or redistributing the torchchat Repository Content
354-
and assume any risks associated with your use of the torchchat Repository Content
355-
or any models, outputs, or results, both alone and in combination with
356-
any other technologies. Additionally, you may have other legal obligations
357-
that govern your use of other content, such as the terms of service for
358-
third-party models, weights, data, or other technologies, and you are
363+
The torchchat Repository Content is provided without any guarantees about
364+
performance or compatibility. In particular, torchchat makes available
365+
model architectures written in Python for PyTorch that may not perform
366+
in the same manner or meet the same standards as the original versions
367+
of those models. When using the torchchat Repository Content, including
368+
any model architectures, you are solely responsible for determining the
369+
appropriateness of using or redistributing the torchchat Repository Content
370+
and assume any risks associated with your use of the torchchat Repository Content
371+
or any models, outputs, or results, both alone and in combination with
372+
any other technologies. Additionally, you may have other legal obligations
373+
that govern your use of other content, such as the terms of service for
374+
third-party models, weights, data, or other technologies, and you are
359375
solely responsible for complying with all such obligations.
360376

361377

@@ -401,4 +417,3 @@ code in this distribution is covered by the MIT and Apache Open Source
401417
licenses.) However you may have other legal obligations that govern
402418
your use of content, such as the terms of service for third-party
403419
models.
404-

0 commit comments

Comments
 (0)