Skip to content

Commit 55ce091

Browse files
ATMxsp01Oscilloscope98
ATMxsp01
and
Oscilloscope98
authored
Add GLM4-Edge-V GPU example (#12596)
* Add GLM4-Edge-V examples * polish readme * revert wrong changes * polish readme * polish readme * little polish in reference info and indent * Small fix and sample output updates * Update main readme --------- Co-authored-by: Yuwen Hu <[email protected]>
1 parent 796ee57 commit 55ce091

File tree

4 files changed

+288
-0
lines changed

4 files changed

+288
-0
lines changed

‎README.md

+1
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,7 @@ Over 70 models have been optimized/verified on `ipex-llm`, including *LLaMA/LLaM
270270
| GLM-4 | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4) | [link](python/llm/example/GPU/HuggingFace/LLM/glm4) |
271271
| GLM-4V | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm-4v) | [link](python/llm/example/GPU/HuggingFace/Multimodal/glm-4v) |
272272
| GLM-Edge | | [link](python/llm/example/GPU/HuggingFace/LLM/glm-edge) | [Python link](python/llm/example/NPU/HF-Transformers-AutoModels/LLM) |
273+
| GLM-Edge-V | | [link](python/llm/example/GPU/HuggingFace/Multimodal/glm-edge-v) |
273274
| Mistral | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/mistral) | [link](python/llm/example/GPU/HuggingFace/LLM/mistral) |
274275
| Mixtral | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/mixtral) | [link](python/llm/example/GPU/HuggingFace/LLM/mixtral) |
275276
| Falcon | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/falcon) | [link](python/llm/example/GPU/HuggingFace/LLM/falcon) |

‎README.zh-CN.md

+1
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,7 @@ See the demo of running [*Text-Generation-WebUI*](https://ipex-llm.readthedocs.i
270270
| GLM-4 | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm4) | [link](python/llm/example/GPU/HuggingFace/LLM/glm4) |
271271
| GLM-4V | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/glm-4v) | [link](python/llm/example/GPU/HuggingFace/Multimodal/glm-4v) |
272272
| GLM-Edge | | [link](python/llm/example/GPU/HuggingFace/LLM/glm-edge) | [Python link](python/llm/example/NPU/HF-Transformers-AutoModels/LLM) |
273+
| GLM-Edge-V | | [link](python/llm/example/GPU/HuggingFace/Multimodal/glm-edge-v) |
273274
| Mistral | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/mistral) | [link](python/llm/example/GPU/HuggingFace/LLM/mistral) |
274275
| Mixtral | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/mixtral) | [link](python/llm/example/GPU/HuggingFace/LLM/mixtral) |
275276
| Falcon | [link](python/llm/example/CPU/HF-Transformers-AutoModels/Model/falcon) | [link](python/llm/example/GPU/HuggingFace/LLM/falcon) |

‎python/llm/example/GPU/HuggingFace/Multimodal/glm-edge-v/README.md

+164
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
# GLM-Edge-V
2+
In this directory, you will find examples on how you could apply IPEX-LLM INT4 optimizations on GLM-Edge-V models on [Intel GPUs](../../../README.md). For illustration purposes, we utilize the [THUDM/glm-edge-v-5b](https://huggingface.co/THUDM/glm-edge-v-5b) and [THUDM/glm-edge-v-2b](https://huggingface.co/THUDM/glm-edge-v-2b) (or [ZhipuAI/glm-edge-v-5b](https://www.modelscope.cn/models/ZhipuAI/glm-edge-v-5b) and [ZhipuAI/glm-edge-v-2b](https://www.modelscope.cn/models/ZhipuAI/glm-edge-v-2b) for ModelScope) as reference GLM-Edge-V models.
3+
4+
## 0. Requirements
5+
To run these examples with IPEX-LLM on Intel GPUs, we have some recommended requirements for your machine, please refer to [here](../../../README.md#requirements) for more information.
6+
7+
## Example: Predict Tokens using `generate()` API
8+
In the example [generate.py](./generate.py), we show a basic use case for a GLM-Edge-V model to predict the next N tokens using `generate()` API, with IPEX-LLM FP8 optimizations on Intel GPUs.
9+
### 1. Install
10+
#### 1.1 Installation on Linux
11+
We suggest using conda to manage environment:
12+
```bash
13+
conda create -n llm python=3.11
14+
conda activate llm
15+
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
16+
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
17+
18+
# install packages required for GLM-Edge-V
19+
pip install transformers==4.47.0
20+
pip install accelerate==0.33.0
21+
pip install "trl<0.12.0"
22+
23+
# [optional] only needed if you would like to use ModelScope as model hub
24+
pip install modelscope==1.11.0
25+
```
26+
27+
#### 1.2 Installation on Windows
28+
We suggest using conda to manage environment:
29+
```bash
30+
conda create -n llm python=3.11 libuv
31+
conda activate llm
32+
33+
# below command will install intel_extension_for_pytorch==2.1.10+xpu as default
34+
pip install --pre --upgrade ipex-llm[xpu] --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
35+
36+
# install packages required for GLM-Edge-V
37+
pip install transformers==4.47.0
38+
pip install accelerate==0.33.0
39+
pip install "trl<0.12.0"
40+
41+
# [optional] only needed if you would like to use ModelScope as model hub
42+
pip install modelscope==1.11.0
43+
```
44+
45+
### 2. Configures OneAPI environment variables for Linux
46+
47+
> [!NOTE]
48+
> Skip this step if you are running on Windows.
49+
50+
This is a required step on Linux for APT or offline installed oneAPI. Skip this step for PIP-installed oneAPI.
51+
52+
```bash
53+
source /opt/intel/oneapi/setvars.sh
54+
```
55+
56+
### 3. Runtime Configurations
57+
For optimal performance, it is recommended to set several environment variables. Please check out the suggestions based on your device.
58+
#### 3.1 Configurations for Linux
59+
<details>
60+
61+
<summary>For Intel Arc™ A-Series Graphics and Intel Data Center GPU Flex Series</summary>
62+
63+
```bash
64+
export USE_XETLA=OFF
65+
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
66+
export SYCL_CACHE_PERSISTENT=1
67+
```
68+
69+
</details>
70+
71+
<details>
72+
73+
<summary>For Intel Data Center GPU Max Series</summary>
74+
75+
```bash
76+
export LD_PRELOAD=${LD_PRELOAD}:${CONDA_PREFIX}/lib/libtcmalloc.so
77+
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=1
78+
export SYCL_CACHE_PERSISTENT=1
79+
export ENABLE_SDP_FUSION=1
80+
```
81+
> Note: Please note that `libtcmalloc.so` can be installed by `conda install -c conda-forge -y gperftools=2.10`.
82+
</details>
83+
84+
<details>
85+
86+
<summary>For Intel iGPU</summary>
87+
88+
```bash
89+
export SYCL_CACHE_PERSISTENT=1
90+
```
91+
92+
</details>
93+
94+
#### 3.2 Configurations for Windows
95+
<details>
96+
97+
<summary>For Intel iGPU and Intel Arc™ A-Series Graphics</summary>
98+
99+
```cmd
100+
set SYCL_CACHE_PERSISTENT=1
101+
```
102+
103+
</details>
104+
105+
106+
> [!NOTE]
107+
> For the first time that each model runs on Intel iGPU/Intel Arc™ A300-Series or Pro A60, it may take several minutes to compile.
108+
### 4. Running examples
109+
110+
```bash
111+
# for Hugging Face model hub
112+
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT --image-url-or-path IMAGE_URL_OR_PATH
113+
114+
# for ModelScope model hub
115+
python ./generate.py --repo-id-or-model-path REPO_ID_OR_MODEL_PATH --prompt PROMPT --n-predict N_PREDICT --image-url-or-path IMAGE_URL_OR_PATH --modelscope
116+
```
117+
118+
Arguments info:
119+
- `--repo-id-or-model-path REPO_ID_OR_MODEL_PATH`: argument defining the **Hugging Face** (e.g. `THUDM/glm-edge-v-5b` or `THUDM/glm-edge-v-2b`) or **ModelScope** (e.g. `ZhipuAI/glm-edge-v-5b` or `ZhipuAI/glm-edge-v-2b`) repo id for the GLM-Edge-V model to be downloaded, or the path to the checkpoint folder. It is default to be `'THUDM/glm-edge-v-5b'` for **Hugging Face** or `'ZhipuAI/glm-edge-v-5b'` for **ModelScope**.
120+
- `--image-url-or-path IMAGE_URL_OR_PATH`: argument defining the image to be infered. It is default to be `'http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg'`.
121+
- `--prompt PROMPT`: argument defining the prompt to be infered (with integrated prompt format for chat). It is default to be `'What is in the image?'`.
122+
- `--n-predict N_PREDICT`: argument defining the max number of tokens to predict. It is default to be `32`.
123+
- `--modelscope`: using **ModelScope** as model hub instead of **Hugging Face**.
124+
125+
#### Sample Output
126+
#### [THUDM/glm-edge-v-2b](https://huggingface.co/THUDM/glm-edge-v-2b)
127+
128+
```log
129+
Inference time: xxxx s
130+
-------------------- Prompt --------------------
131+
图片里有什么？
132+
-------------------- Output --------------------
133+
这张图片中，一个穿着粉色和白色条纹衣服的小女孩举着一只穿着粉色裙子的白色小熊玩偶。背景是砖墙
134+
```
135+
136+
```log
137+
Inference time: xxxx s
138+
-------------------- Prompt --------------------
139+
What is in the image?
140+
-------------------- Output --------------------
141+
A white teddy bear with pink accents is being held up by a person wearing a pink and white striped shirt. The bear is dressed in a
142+
```
143+
144+
#### [THUDM/glm-edge-v-5b](https://huggingface.co/THUDM/glm-edge-v-5b)
145+
146+
```log
147+
Inference time: xxxx s
148+
-------------------- Prompt --------------------
149+
图片里有什么？
150+
-------------------- Output --------------------
151+
这张图片展示了一个穿着红白条纹衣服的小女孩，她正在展示她手中的一个粉红色的白熊玩偶。背景是砖墙和一丛
152+
```
153+
154+
```log
155+
Inference time: xxxx s
156+
-------------------- Prompt --------------------
157+
What is in the image?
158+
-------------------- Output --------------------
159+
The image shows a young girl holding a small, white teddy bear. The girl is wearing a pink striped dress with a red bow at the neckline
160+
```
161+
162+
The sample input image is (which is fetched from [COCO dataset](https://cocodataset.org/#explore?id=264959)):
163+
164+
<a href="http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg"><img width=400px src="http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg" ></a>

‎python/llm/example/GPU/HuggingFace/Multimodal/glm-edge-v/generate.py

+122
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
#
2+
# Copyright 2016 The BigDL Authors.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
17+
import os
18+
import torch
19+
import time
20+
import argparse
21+
import requests
22+
23+
from ipex_llm.transformers import AutoModelForCausalLM
24+
25+
from PIL import Image
26+
27+
if __name__ == '__main__':
28+
parser = argparse.ArgumentParser(description='Predict Tokens using `generate()` API for glm-edge-v model')
29+
parser.add_argument('--repo-id-or-model-path', type=str,
30+
help='The Hugging Face or ModelScope repo id for the glm-edge-v model to be downloaded'
31+
', or the path to the checkpoint folder')
32+
parser.add_argument('--image-url-or-path', type=str,
33+
default='http://farm6.staticflickr.com/5268/5602445367_3504763978_z.jpg',
34+
help='The URL or path to the image to infer')
35+
parser.add_argument('--prompt', type=str, default="What is in the image?",
36+
help='Prompt to infer')
37+
parser.add_argument('--n-predict', type=int, default=32,
38+
help='Max tokens to predict')
39+
parser.add_argument('--modelscope', action="store_true", default=False,
40+
help="Use models from modelscope")
41+
42+
args = parser.parse_args()
43+
44+
if args.modelscope:
45+
from modelscope import AutoTokenizer, AutoImageProcessor
46+
model_hub = 'modelscope'
47+
else:
48+
from transformers import AutoTokenizer, AutoImageProcessor
49+
model_hub = 'huggingface'
50+
51+
model_path = args.repo_id_or_model_path if args.repo_id_or_model_path else \
52+
("ZhipuAI/glm-edge-v-5b" if args.modelscope else "THUDM/glm-edge-v-5b")
53+
image_path = args.image_url_or_path
54+
55+
# Load model in 4 bit,
56+
# which convert the relevant layers in the model into INT4 format
57+
# When running LLMs on Intel iGPUs for Windows users, we recommend setting `cpu_embedding=True` in the from_pretrained function.
58+
# This will allow the memory-intensive embedding layer to utilize the CPU instead of iGPU.
59+
model = AutoModelForCausalLM.from_pretrained(model_path,
60+
load_in_4bit=True,
61+
optimize_model=True,
62+
trust_remote_code=True,
63+
modules_to_not_convert=["vision"],
64+
use_cache=True,
65+
model_hub=model_hub)
66+
model = model.half().to('xpu')
67+
68+
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
69+
70+
query = args.prompt
71+
if os.path.exists(image_path):
72+
image = Image.open(image_path)
73+
else:
74+
image = Image.open(requests.get(image_path, stream=True).raw)
75+
76+
image_processor = AutoImageProcessor.from_pretrained(model_path, trust_remote_code=True)
77+
78+
with torch.inference_mode():
79+
pixel_values = image_processor(images=[image], return_tensors='pt').pixel_values
80+
pixel_values = pixel_values.to('xpu')
81+
82+
# The following code for generation is adapted from https://huggingface.co/THUDM/glm-edge-v-5b#inference
83+
messages = [{
84+
"role": "user",
85+
"content": [{"type": "image"},
86+
{"type": "text",
87+
"text": args.prompt}]
88+
}]
89+
90+
inputs = tokenizer.apply_chat_template(
91+
messages,
92+
add_generation_prompt=True,
93+
return_dict=True,
94+
tokenize=True,
95+
return_tensors="pt"
96+
)
97+
inputs = inputs.to('xpu')
98+
99+
generate_kwargs = {
100+
**inputs,
101+
"pixel_values": pixel_values,
102+
"max_new_tokens": args.n_predict,
103+
}
104+
105+
# ipex_llm model needs a warmup, then inference time can be accurate
106+
output = model.generate(**generate_kwargs)
107+
108+
st = time.time()
109+
output = model.generate(**generate_kwargs)
110+
torch.xpu.synchronize()
111+
end = time.time()
112+
113+
output_str = tokenizer.decode(
114+
output[0][len(inputs["input_ids"][0]):],
115+
skip_special_tokens=True
116+
)
117+
118+
print(f'Inference time: {end-st} s')
119+
print('-'*20, 'Prompt', '-'*20)
120+
print(args.prompt)
121+
print('-'*20, 'Output', '-'*20)
122+
print(output_str)

0 commit comments

Comments
 (0)