Skip to content

Commit ed30c98

Browse files
jeremyfowersramkrishna2910
jeremyfowers
and
ramkrishna2910
authored
Move llm source code into src/lemonade dir. Add HumanEval. (#262)
* Move lemonade source code into its own module * Reorganize into src/lemonade. Add HumanEval. Co-authored-by: Ramakrishnan Sivakumar <ramkrishna2910@gmail.com> Signed-off-by: Jeremy Fowers <80718789+jeremyfowers@users.noreply.github.com> --------- Signed-off-by: Jeremy Fowers <80718789+jeremyfowers@users.noreply.github.com> Co-authored-by: Ramakrishnan Sivakumar <ramkrishna2910@gmail.com>
1 parent 5b2d152 commit ed30c98

30 files changed

Lines changed: 630 additions & 121 deletions

‎.pylintrc‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ enable =
121121
no-init,
122122
abstract-method,
123123
invalid-overridden-method,
124-
arguments-differ,
124+
# arguments-differ,
125125
signature-differs,
126126
bad-staticmethod-argument,
127127
useless-super-delegation,

‎README.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,17 @@
55
[![OS - Windows | Linux](https://img.shields.io/badge/OS-windows%20%7C%20linux-blue)](https://github.com/onnx/turnkeyml/blob/main/docs/install.md "Check out our instructions")
66
[![Made with Python](https://img.shields.io/badge/Python-3.8,3.10-blue?logo=python&logoColor=white)](https://github.com/onnx/turnkeyml/blob/main/docs/install.md "Check out our instructions")
77

8-
We are on a mission to make it easy to use the most important tools in the ONNX ecosystem. TurnkeyML accomplishes this by providing no-code CLIs and low-code APIs for both general ONNX workflows as well as LLMs.
8+
We are on a mission to make it easy to use the most important tools in the ONNX ecosystem. TurnkeyML accomplishes this by providing no-code CLIs and low-code APIs for both general ONNX workflows with `turnkey` as well as LLMs with `lemonade`.
99

10-
| [**Turnkey LLM**](https://github.com/onnx/turnkeyml/tree/main/src/turnkeyml/llm) | [**Turnkey Classic**](https://github.com/onnx/turnkeyml/blob/main/docs/classic_getting_started.md) |
10+
| [**Lemonade**](https://github.com/onnx/turnkeyml/tree/main/src/turnkeyml/llm) | [**Turnkey**](https://github.com/onnx/turnkeyml/blob/main/docs/classic_getting_started.md) |
1111
|:----------------------------------------------: |:-----------------------------------------------------------------: |
12-
| Serve and benchmark LLMs on CPU, GPU, and NPU. <br/> [Click here to get started with turnkey-llm.](https://github.com/onnx/turnkeyml/tree/main/src/turnkeyml/llm) | Export and optimize ONNX models for CNNs, Transformers, and GNNs. <br/> [Click here to get started with turnkey classic.](https://github.com/onnx/turnkeyml/blob/main/docs/classic_getting_started.md) |
12+
| Serve and benchmark LLMs on CPU, GPU, and NPU. <br/> [Click here to get started with `lemonade`.](https://github.com/onnx/turnkeyml/blob/main/docs/lemonade_getting_started.md) | Export and optimize ONNX models for CNNs and Transformers. <br/> [Click here to get started with `turnkey`.](https://github.com/onnx/turnkeyml/blob/main/docs/classic_getting_started.md) |
1313
| <img src="img/llm_demo.png"/> | <img src="img/classic_demo.png"/> |
1414

1515

1616
## How It Works
1717

18-
The `turnkey` (classic) and `turnkey-llm` CLIs provide a set of `Tools` that users can invoke in a `Sequence`. The first `Tool` takes the input (`-i`), performs some action, and passes its state to the next `Tool` in the `Sequence`.
18+
The `turnkey` (CNNs and transformers) and `lemonade` (LLMs) CLIs provide a set of `Tools` that users can invoke in a `Sequence`. The first `Tool` takes the input (`-i`), performs some action, and passes its state to the next `Tool` in the `Sequence`.
1919

2020
You can read the `Sequence` out like a sentence. For example, the demo command above was:
2121

@@ -51,3 +51,4 @@ This project is licensed under the [Apache 2.0 License](https://github.com/onnx/
5151
## Attribution
5252

5353
TurnkeyML used code from other open source projects as a starting point (see [NOTICE.md](NOTICE.md)). Thank you Philip Colangelo, Derek Elkins, Jeremy Fowers, Dan Gard, Victoria Godsoe, Mark Heaps, Daniel Holanda, Brian Kurtz, Mariah Larwood, Philip Lassen, Andrew Ling, Adrian Macias, Gary Malik, Sarah Massengill, Ashwin Murthy, Hatice Ozen, Tim Sears, Sean Settle, Krishna Sivakumar, Aviv Weinstein, Xueli Xao, Bill Xing, and Lev Zlotnik for your contributions to that work.
54+

‎docs/humaneval_accuracy.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
# Using the HumanEval accuracy test tools
2+
3+
The HumanEval benchmark is a code generation and functional correctness evaluation framework designed to assess language models' ability to generate Python code. It consists of 164 handwritten programming problems, each containing a function signature, docstring, body, and several unit tests. This benchmark focuses on evaluating a model's capability to generate functionally correct code that passes the test cases, making it particularly useful for assessing code generation capabilities.
4+
5+
This tool provides an automated way to evaluate language models on the HumanEval benchmark. It handles the process of downloading the dataset, generating code completions, executing them in a secure environment, and calculating pass@k metrics.
6+
7+
## Dataset
8+
9+
The HumanEval dataset is automatically downloaded from [OpenAI's human-eval repository](https://github.com/openai/human-eval) when you first run the benchmark. The dataset contains programming problems that test various aspects of Python programming, including:
10+
11+
- Basic programming operations
12+
- String manipulation
13+
- Mathematical computations
14+
- List operations
15+
- Algorithm implementation
16+
- Data structure manipulation
17+
18+
## Running the Benchmark
19+
20+
```bash
21+
lemonade -i meta-llama/Llama-3.2-1B oga-load --device igpu --dtype int4 accuracy-humaneval --k-samples 1 --first-n-samples 5 --timeout 30.0
22+
```
23+
24+
### Optional arguments:
25+
26+
`--k-samples`: Number of completions to generate per prompt (default: 1). This parameter determines the k in pass@k metrics. For example:
27+
- `--k-samples 1`: Calculates pass@1 (single attempt per problem)
28+
- `--k-samples 10`: Calculates pass@10 (ten attempts per problem)
29+
- `--k-samples 100`: Calculates pass@100 (hundred attempts per problem)
30+
31+
Higher k values provide more robust evaluation but take longer to run.
32+
33+
`--first-n-samples`: Evaluate only the first N problems from the dataset (default: entire dataset). Useful for quick testing or when you want to evaluate a subset of problems.
34+
35+
`--timeout`: Maximum time in seconds allowed for each test case execution (default: 30.0). This prevents infinite loops or long-running code from blocking the evaluation.
36+
37+
`--data-dir`: Custom directory for storing the HumanEval dataset (default: "<lemonade_cache_dir>/data/humaneval").
38+
39+
## How It Works
40+
41+
1. **Dataset Preparation:**
42+
- On first run, the tool downloads the HumanEval dataset (HumanEval.jsonl.gz)
43+
- The dataset contains function signatures, docstrings, and test cases
44+
- Each problem is structured to test specific programming capabilities
45+
- You can evaluate only the first N problems using `--first-n-samples`
46+
47+
2. **Code Generation:**
48+
- For each programming problem, the model is provided with a prompt containing:
49+
- Function signature (e.g., `def sort_numbers(numbers):`)
50+
- Docstring describing the function's purpose and requirements
51+
- The model generates k code completions for the function body (controlled by `--k-samples`)
52+
- These k samples are used to calculate the pass@k metric
53+
54+
3. **Secure Execution:**
55+
- Generated code is executed in a secure sandbox environment maintained by OpenAI's human-eval library. For your awareness, OpenAI's policy is to disable code execution by default, however lemonade enables code execution by default by automatically setting the environment variable `HF_ALLOW_CODE_EVAL=1`. OpenAI provides the following code execution protections:
56+
- **Process Isolation**: Each code sample runs in a separate process to prevent interference
57+
- **Resource Limits**:
58+
- CPU time limit (controlled by `--timeout`)
59+
- Memory usage restrictions
60+
- Maximum output size restrictions
61+
- **Restricted Access**:
62+
- No network access
63+
- No file system access outside test directory
64+
- No subprocess creation
65+
- No system calls
66+
- **Module Restrictions**:
67+
- Only allows importing standard Python libraries needed for testing
68+
- Blocks potentially dangerous modules (os, sys, subprocess, etc.)
69+
These security measures are implemented through:
70+
- Python's built-in `resource` module for resource limits
71+
- AST (Abstract Syntax Tree) analysis for code validation
72+
- Process-level isolation using `multiprocessing`
73+
- Custom import hooks to restrict module access
74+
75+
4. **Evaluation Metrics:**
76+
- **pass@k**: Percentage of problems solved with k attempts
77+
- pass@1: Success rate with single attempt
78+
- pass@10: Success rate within 10 attempts
79+
- pass@100: Success rate within 100 attempts
80+
- A problem is considered solved if all test cases pass
81+
- Results are normalized to percentages
82+
83+
5. **Output Files:**
84+
The tool generates several output files in the results directory:
85+
- `evaluation_results.csv`: Contains prompts, completions, and expected answers
86+
- `humaneval_predictions.jsonl`: Raw model predictions in JSONL format
87+
- `humaneval_predictions.jsonl_results.jsonl`: Detailed evaluation results
88+
89+
## Example Results Format
90+
91+
The evaluation produces metrics in the following format:
92+
```json
93+
{
94+
"pass@1": 0.25, // 25% success rate with 1 attempt
95+
"pass@10": 0.45, // 45% success rate within 10 attempts
96+
"pass@100": 0.65 // 65% success rate within 100 attempts
97+
}
98+
```
99+
100+
## Limitations
101+
102+
1. **Resource Requirements**: Generating multiple samples per problem (high k values) can be computationally intensive and time-consuming.
103+
2. **Memory Usage**: Large language models may require significant memory, especially when generating multiple samples.
104+
105+
## References
106+
107+
1. [Evaluating Large Language Models Trained on Code](https://arxiv.org/abs/2107.03374)
108+
2. [OpenAI HumanEval Repository](https://github.com/openai/human-eval)

src/turnkeyml/llm/README.md renamed to docs/lemonade_getting_started.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,8 @@ Note that the `llm-prompt`, `accuracy-mmlu`, and `serve` tools can all be used w
7979
Lemonade is also available via API. Here's a quick example of how to benchmark an LLM:
8080

8181
```python
82-
import turnkeyml.llm.tools.torch_llm as tl
83-
import turnkeyml.llm.tools.chat as cl
82+
import lemonade.tools.torch_llm as tl
83+
import lemonade.tools.chat as cl
8484
from turnkeyml.state import State
8585

8686
state = State(cache_dir="cache", build_name="test")

‎examples/llm/leap_basic.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
"""
2+
This example demonstrates how to use the LEAP API to load a model for
3+
inference on CPU using the hf-cpu recipe, and then use it to generate
4+
the response to a prompt.
5+
6+
If you have a discrete GPU, you can try that by changing the recipe
7+
to hf-dgpu. Note: make sure to have torch+cuda installed when trying
8+
hf-dgpu.
9+
"""
10+
11+
from lemonade import leap
12+
13+
model, tokenizer = leap.from_pretrained("facebook/opt-125m", recipe="hf-cpu")
14+
15+
input_ids = tokenizer("This is my prompt", return_tensors="pt").input_ids
16+
response = model.generate(input_ids, max_new_tokens=30)
17+
18+
print(tokenizer.decode(response[0]))

‎examples/llm/leap_ryzenai_npu.py‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
"""
2+
This example demonstrates how to use the LEAP API to load a model for
3+
inference on a Ryzen AI NPU using the ryzenai-npu-load recipe,
4+
and then use it to generate the response to a prompt.
5+
6+
Note that this example will only run if the Ryzen AI NPU Private recipe is installed.
7+
See genai/docs/ryzenai_npu.md for instructions.
8+
9+
You can try the same model on CPU by changing the recipe to "hf-cpu".
10+
"""
11+
12+
from lemonade import leap
13+
14+
model, tokenizer = leap.from_pretrained(
15+
"meta-llama/Llama-2-7b-chat-hf", recipe="ryzenai-npu"
16+
)
17+
18+
input_ids = tokenizer("This is my prompt", return_tensors="pt").input_ids
19+
response = model.generate(input_ids, max_new_tokens=30)
20+
21+
print(tokenizer.decode(response[0]))

‎examples/llm/leap_streaming.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
"""
2+
This example demonstrates how to use the LEAP API to load a model for
3+
inference on CPU using the hf-cpu recipe, and then use a thread to
4+
generate a streaming the response to a prompt.
5+
6+
Note: this approach only works with recipes that support TextIteratorStreamer,
7+
i.e., huggingface-based recipes such as hf-cpu and ryzenai-npu.
8+
"""
9+
10+
from thread import Thread
11+
from transformers import TextIteratorStreamer
12+
from lemonade import leap
13+
14+
# Replace the recipe with "ryzenai-npu" to run on the RyzenAI NPU
15+
model, tokenizer = leap.from_pretrained(
16+
"meta-llama/Llama-2-7b-chat-hf", recipe="hf-cpu"
17+
)
18+
19+
input_ids = tokenizer("This is my prompt", return_tensors="pt").input_ids
20+
21+
streamer = TextIteratorStreamer(
22+
tokenizer,
23+
skip_prompt=True,
24+
)
25+
generation_kwargs = {
26+
"input_ids": input_ids,
27+
"streamer": streamer,
28+
"max_new_tokens": 30,
29+
}
30+
31+
thread = Thread(target=model.generate, kwargs=generation_kwargs)
32+
thread.start()
33+
34+
# Generate the response using streaming
35+
for new_text in streamer:
36+
print(new_text)
37+
38+
thread.join()

‎examples/llm/turnkey_llm.ipynb‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -85,7 +85,7 @@
8585
"outputs": [],
8686
"source": [
8787
"# Import the turnkey APIs\n",
88-
"from turnkeyml.llm import leap\n",
88+
"from lemonade import leap\n",
8989
"\n",
9090
"# Load the model on to RyzenAI NPU\n",
9191
"# NOTE: this takes a couple of minutes, but after you've done it once\n",
@@ -133,7 +133,7 @@
133133
"outputs": [],
134134
"source": [
135135
"# Import the turnkey APIs\n",
136-
"from turnkeyml.llm import leap\n",
136+
"from lemonade import leap\n",
137137
"\n",
138138
"# Load the model on iGPU\n",
139139
"igpu_model, igpu_tokenizer = leap.from_pretrained(\n",

‎examples/readme.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,3 +3,4 @@
33
This directory contains examples to help you learn how to use the tools. The examples are split up into two sub-directories:
44
1. `examples/cli`: a tutorial series for the `turnkey` CLI. This is the recommended starting point.
55
1. `examples/api`: scripts that demonstrate how to use the `turnkey.evaluate_files()` API.
6+
1. `examples/llm`: scripts that demonstrate the `lemonade` CLI for LLMs.

‎setup.py‎

Lines changed: 21 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,11 @@
33
with open("src/turnkeyml/version.py", encoding="utf-8") as fp:
44
version = fp.read().split('"')[1]
55

6+
67
setup(
78
name="turnkeyml",
89
version=version,
910
description="TurnkeyML Tools and Models",
10-
author="Jeremy Fowers, Daniel Holanda, Ramakrishnan Sivakumar, Victoria Godsoe",
1111
author_email="turnkeyml@amd.com",
1212
package_dir={"": "src", "turnkeyml_models": "models"},
1313
packages=[
@@ -17,10 +17,10 @@
1717
"turnkeyml.sequence",
1818
"turnkeyml.cli",
1919
"turnkeyml.common",
20-
"turnkeyml.llm",
21-
"turnkeyml.llm.tools",
22-
"turnkeyml.llm.tools.ort_genai",
23-
"turnkeyml.llm.tools.ryzenai_npu",
20+
"lemonade",
21+
"lemonade.tools",
22+
"lemonade.tools.ort_genai",
23+
"lemonade.tools.ryzenai_npu",
2424
"turnkeyml_models",
2525
"turnkeyml_models.graph_convolutions",
2626
"turnkeyml_models.selftest",
@@ -46,77 +46,61 @@
4646
"psutil",
4747
"wmi",
4848
"pytz",
49+
"tqdm",
4950
# Conditional dependencies for ONNXRuntime backends
5051
"onnxruntime >=1.10.1;platform_system=='Linux' and extra != 'llm-oga-cuda'",
5152
"onnxruntime-directml >=1.19.0;platform_system=='Windows' and extra != 'llm-oga-cuda'",
5253
"onnxruntime-gpu >=1.19.1;extra == 'llm-oga-cuda'",
5354
],
5455
extras_require={
5556
"llm": [
56-
"tqdm",
5757
"torch>=2.0.0",
5858
"transformers",
5959
"accelerate",
6060
"py-cpuinfo",
6161
"sentencepiece",
6262
"datasets",
63+
# Install human-eval from a forked repo with Windows support until the
64+
# PR (https://github.com/openai/human-eval/pull/53) is merged
65+
"human-eval @ git+https://github.com/ramkrishna2910/human-eval.git",
6366
"fastapi",
6467
"uvicorn[standard]",
6568
],
66-
"llm-oga-dml": [
69+
"llm-oga-igpu": [
6770
"onnxruntime-genai-directml==0.4.0",
68-
"tqdm",
6971
"torch>=2.0.0,<2.4",
7072
"transformers<4.45.0",
71-
"accelerate",
72-
"py-cpuinfo",
73-
"sentencepiece",
74-
"datasets",
75-
"fastapi",
76-
"uvicorn[standard]",
73+
"turnkeyml[llm]",
7774
],
7875
"llm-oga-cuda": [
7976
"onnxruntime-genai-cuda==0.4.0",
80-
"tqdm",
8177
"torch>=2.0.0,<2.4",
8278
"transformers<4.45.0",
83-
"accelerate",
84-
"py-cpuinfo",
85-
"sentencepiece",
86-
"datasets",
87-
"fastapi",
88-
"uvicorn[standard]",
79+
"turnkeyml[llm]",
8980
],
9081
"llm-oga-npu": [
91-
"transformers",
92-
"torch",
9382
"onnx==1.16.0",
9483
"onnxruntime==1.18.0",
9584
"numpy==1.26.4",
96-
"tqdm",
97-
"accelerate",
98-
"py-cpuinfo",
99-
"sentencepiece",
100-
"datasets",
101-
"fastapi",
102-
"uvicorn[standard]",
85+
"turnkeyml[llm]",
10386
],
10487
"llm-oga-hybrid": [
105-
"transformers",
106-
"torch",
10788
"onnx==1.16.1",
10889
"numpy==1.26.4",
109-
"datasets",
110-
"fastapi",
111-
"uvicorn[standard]",
90+
"turnkeyml[llm]",
91+
],
92+
"cuda": [
93+
"torch @ https://download.pytorch.org/whl/cu118/torch-2.3.1%2Bcu118-cp310-cp310-win_amd64.whl",
94+
"torchvision @ https://download.pytorch.org/whl/cu118/torchvision-0.18.1%2Bcu118-cp310-cp310-win_amd64.whl",
95+
"torchaudio @ https://download.pytorch.org/whl/cu118/torchaudio-2.3.1%2Bcu118-cp310-cp310-win_amd64.whl",
11296
],
11397
},
11498
classifiers=[],
11599
entry_points={
116100
"console_scripts": [
117101
"turnkey=turnkeyml:turnkeycli",
118-
"turnkey-llm=turnkeyml.llm:lemonadecli",
119-
"lemonade=turnkeyml.llm:lemonadecli",
102+
"turnkey-llm=lemonade:lemonadecli",
103+
"lemonade=lemonade:lemonadecli",
120104
]
121105
},
122106
python_requires=">=3.8, <3.12",

0 commit comments

Comments
 (0)