Skip to content

Commit 3b51b95

Browse files
Added a fix for llm-chatbot int8 weight compression in case fp16 model already exists (#1479)
1 parent 13aee00 commit 3b51b95

File tree

1 file changed

+4
-1
lines changed

1 file changed

+4
-1
lines changed

notebooks/254-llm-chatbot/254-llm-chatbot.ipynb

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -554,7 +554,10 @@
554554
" return\n",
555555
" int8_model_dir.mkdir(parents=True, exist_ok=True)\n",
556556
" if not model_configuration[\"remote\"]:\n",
557-
" if not fp16_model_dir.exists():\n",
557+
" if fp16_model_dir.exists():\n",
558+
" ov_model = OVModelForCausalLM.from_pretrained(fp16_model_dir, compile=False,\n",
559+
" ov_config={\"CACHE_DIR\": \"\"})\n",
560+
" else:\n",
558561
" ov_model = OVModelForCausalLM.from_pretrained(\n",
559562
" pt_model_id, export=True, compile=False\n",
560563
" ).half()\n",

0 commit comments

Comments
 (0)