Skip to content

Commit da84641

Browse files
gericdonggericdong
committed
feat: add function calling and code execution examples to multimodal live api websockets notebook
1 parent 93443fd commit da84641

File tree

1 file changed

+189
-0
lines changed

1 file changed

+189
-0
lines changed

‎gemini/multimodal-live-api/intro_multimodal_live_api.ipynb

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,8 @@
104104
"- Text-to-text generation\n",
105105
"- Text-to-audio generation\n",
106106
"- Text-to-audio conversation\n",
107+
"- Function calling\n",
108+
"- Code execution\n",
107109
"\n",
108110
"See the [Multimodal Live API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live) page for more details."
109111
]
@@ -590,6 +592,193 @@
590592
"await main()"
591593
]
592594
},
595+
{
596+
"cell_type": "markdown",
597+
"metadata": {
598+
"id": "f214d0c3bee0"
599+
},
600+
"source": [
601+
"### **Example 4**: Function calling\n",
602+
"\n",
603+
"You can use function calling to create a description of a function, then pass that description to the model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
604+
"\n",
605+
"**Notes**:\n",
606+
"\n",
607+
"- All functions must be declared at the start of the session by sending tool definitions as part of the `setup` message.\n",
608+
"- Currently only one tool is supported in the API."
609+
]
610+
},
611+
{
612+
"cell_type": "code",
613+
"execution_count": null,
614+
"metadata": {
615+
"id": "8a7595aee24a"
616+
},
617+
"outputs": [],
618+
"source": [
619+
"# Set model generation_config\n",
620+
"CONFIG = {\"response_modalities\": [\"TEXT\"]}\n",
621+
"\n",
622+
"# Define function declarations\n",
623+
"TOOLS = {\n",
624+
" \"function_declarations\": {\n",
625+
" \"name\": \"get_current_weather\",\n",
626+
" \"description\": \"Get the current weather in the given location\",\n",
627+
" \"parameters\": {\n",
628+
" \"type\": \"OBJECT\",\n",
629+
" \"properties\": {\"location\": {\"type\": \"STRING\"}},\n",
630+
" },\n",
631+
" }\n",
632+
"}\n",
633+
"\n",
634+
"headers = {\n",
635+
" \"Content-Type\": \"application/json\",\n",
636+
" \"Authorization\": f\"Bearer {bearer_token[0]}\",\n",
637+
"}\n",
638+
"\n",
639+
"# Connect to the server\n",
640+
"async with connect(SERVICE_URL, additional_headers=headers) as ws:\n",
641+
" # Setup the session\n",
642+
" await ws.send(\n",
643+
" json.dumps(\n",
644+
" {\n",
645+
" \"setup\": {\n",
646+
" \"model\": MODEL,\n",
647+
" \"generation_config\": CONFIG,\n",
648+
" \"tools\": TOOLS,\n",
649+
" }\n",
650+
" }\n",
651+
" )\n",
652+
" )\n",
653+
"\n",
654+
" # Receive setup response\n",
655+
" raw_response = await ws.recv(decode=False)\n",
656+
" setup_response = json.loads(raw_response.decode())\n",
657+
"\n",
658+
" # Send text message\n",
659+
" text_input = \"Get the current weather in Santa Clara, San Jose and Mountain View\"\n",
660+
" display(Markdown(f\"**Input:** {text_input}\"))\n",
661+
"\n",
662+
" msg = {\n",
663+
" \"client_content\": {\n",
664+
" \"turns\": [{\"role\": \"user\", \"parts\": [{\"text\": text_input}]}],\n",
665+
" \"turn_complete\": True,\n",
666+
" }\n",
667+
" }\n",
668+
"\n",
669+
" await ws.send(json.dumps(msg))\n",
670+
"\n",
671+
" responses = []\n",
672+
"\n",
673+
" # Receive chucks of server response\n",
674+
" async for raw_response in ws:\n",
675+
" response = json.loads(raw_response.decode(\"UTF-8\"))\n",
676+
"\n",
677+
" if (tool_call := response.get(\"toolCall\")) is not None:\n",
678+
" for function_call in tool_call[\"functionCalls\"]:\n",
679+
" responses.append(f\"FunctionCall: {str(function_call)}\\n\")\n",
680+
"\n",
681+
" if (server_content := response.get(\"serverContent\")) is not None:\n",
682+
" if server_content.get(\"turnComplete\", True):\n",
683+
" break\n",
684+
"\n",
685+
" # Print the server response\n",
686+
" display(Markdown(\"**Response >** {}\".format(\"\\n\".join(responses))))"
687+
]
688+
},
689+
{
690+
"cell_type": "markdown",
691+
"metadata": {
692+
"id": "ad6b585deadb"
693+
},
694+
"source": [
695+
"### **Example 5**: Code execution\n",
696+
"\n",
697+
"You can use code execution capability to generate and execute Python code directly within the API.\n",
698+
"\n",
699+
"In this example, you initialize the code execution tool by passing `code_execution` in the `Tools` configuration, and register this tool with the model at the start of the session by sending tool definitions as part of the `setup` message."
700+
]
701+
},
702+
{
703+
"cell_type": "code",
704+
"execution_count": null,
705+
"metadata": {
706+
"id": "acbbd8c0155e"
707+
},
708+
"outputs": [],
709+
"source": [
710+
"# Set model generation_config\n",
711+
"CONFIG = {\"response_modalities\": [\"TEXT\"]}\n",
712+
"\n",
713+
"# Set code execution\n",
714+
"TOOLS = {\"code_execution\": {}}\n",
715+
"\n",
716+
"headers = {\n",
717+
" \"Content-Type\": \"application/json\",\n",
718+
" \"Authorization\": f\"Bearer {bearer_token[0]}\",\n",
719+
"}\n",
720+
"\n",
721+
"# Connect to the server\n",
722+
"async with connect(SERVICE_URL, additional_headers=headers) as ws:\n",
723+
" # Setup the session\n",
724+
" await ws.send(\n",
725+
" json.dumps(\n",
726+
" {\n",
727+
" \"setup\": {\n",
728+
" \"model\": MODEL,\n",
729+
" \"generation_config\": CONFIG,\n",
730+
" \"tools\": TOOLS,\n",
731+
" }\n",
732+
" }\n",
733+
" )\n",
734+
" )\n",
735+
"\n",
736+
" # Receive setup response\n",
737+
" raw_response = await ws.recv(decode=False)\n",
738+
" setup_response = json.loads(raw_response.decode())\n",
739+
"\n",
740+
" # Send text message\n",
741+
" text_input = \"Write code to calculate the 15th fibonacci number then find the nearest palindrome to it\"\n",
742+
" display(Markdown(f\"**Input:** {text_input}\"))\n",
743+
"\n",
744+
" msg = {\n",
745+
" \"client_content\": {\n",
746+
" \"turns\": [{\"role\": \"user\", \"parts\": [{\"text\": text_input}]}],\n",
747+
" \"turn_complete\": True,\n",
748+
" }\n",
749+
" }\n",
750+
"\n",
751+
" await ws.send(json.dumps(msg))\n",
752+
"\n",
753+
" responses = []\n",
754+
"\n",
755+
" # Receive chucks of server response\n",
756+
" async for raw_response in ws:\n",
757+
" response = json.loads(raw_response.decode(\"UTF-8\"))\n",
758+
"\n",
759+
" if (server_content := response.get(\"serverContent\")) is not None:\n",
760+
" model_turn = server_content.get(\"modelTurn\")\n",
761+
" if (parts := model_turn.get(\"parts\")) is not None:\n",
762+
" if parts[0].get(\"text\"):\n",
763+
" responses.append(parts[0][\"text\"])\n",
764+
" for part in parts:\n",
765+
" if (executable_code := part.get(\"executableCode\")) is not None:\n",
766+
" display(\n",
767+
" Markdown(\n",
768+
" f\"\"\"**Executable code:**\n",
769+
"```py\n",
770+
"{executable_code.get(\"code\")}\n",
771+
"```\n",
772+
" \"\"\"\n",
773+
" )\n",
774+
" )\n",
775+
" if server_content.get(\"turnComplete\", False):\n",
776+
" break\n",
777+
"\n",
778+
" # Print the server response\n",
779+
" display(Markdown(f\"**Response >** {''.join(responses)}\"))"
780+
]
781+
},
593782
{
594783
"cell_type": "markdown",
595784
"metadata": {

0 commit comments

Comments
 (0)