Skip to content

Commit bd9a9c6

Browse files
gericdonggericdong
committed
feat: add function calling and code execution examples for live api
1 parent 825c62e commit bd9a9c6

File tree

1 file changed

+166
-0
lines changed

1 file changed

+166
-0
lines changed

‎gemini/multimodal-live-api/intro_multimodal_live_api_genai_sdk.ipynb

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,8 @@
104104
"- Text-to-text generation\n",
105105
"- Text-to-audio generation\n",
106106
"- Text-to-audio conversation\n",
107+
"- Function calling\n",
108+
"- Code execution\n",
107109
"\n",
108110
"See the [Multimodal Live API](https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/multimodal-live) page for more details."
109111
]
@@ -186,9 +188,12 @@
186188
"from IPython.display import Audio, Markdown, display\n",
187189
"from google import genai\n",
188190
"from google.genai.types import (\n",
191+
" FunctionDeclaration,\n",
189192
" LiveConnectConfig,\n",
190193
" PrebuiltVoiceConfig,\n",
191194
" SpeechConfig,\n",
195+
" Tool,\n",
196+
" ToolCodeExecution,\n",
192197
" VoiceConfig,\n",
193198
")\n",
194199
"import numpy as np"
@@ -314,6 +319,53 @@
314319
" display(Markdown(f\"**Response >** {''.join(response)}\"))"
315320
]
316321
},
322+
{
323+
"cell_type": "code",
324+
"execution_count": null,
325+
"metadata": {
326+
"id": "a715adea5343"
327+
},
328+
"outputs": [],
329+
"source": [
330+
"get_current_weather = FunctionDeclaration(\n",
331+
" name=\"get_current_weather\",\n",
332+
" description=\"Get current weather in the given location\",\n",
333+
" parameters={\n",
334+
" \"type\": \"OBJECT\",\n",
335+
" \"properties\": {\n",
336+
" \"location\": {\n",
337+
" \"type\": \"STRING\",\n",
338+
" },\n",
339+
" },\n",
340+
" },\n",
341+
")\n",
342+
"\n",
343+
"config = LiveConnectConfig(\n",
344+
" response_modalities=[\"TEXT\"],\n",
345+
" tools=[get_current_weather],\n",
346+
")\n",
347+
"\n",
348+
"async with client.aio.live.connect(\n",
349+
" model=MODEL_ID,\n",
350+
" config=config,\n",
351+
") as session:\n",
352+
" text_input = \"Get the current weather in Santa Clara, San Jose and Mountain View\"\n",
353+
" display(Markdown(f\"**Input:** {text_input}\"))\n",
354+
"\n",
355+
" await session.send(input=text_input, end_of_turn=True)\n",
356+
"\n",
357+
" response = []\n",
358+
"\n",
359+
" async for message in session.receive():\n",
360+
" print(message)\n",
361+
" if message.text:\n",
362+
" response.append(message.text)\n",
363+
" if message.tool_call:\n",
364+
" print(message.function_response)\n",
365+
"\n",
366+
" display(Markdown(f\"**Response >** {''.join(response)}\"))"
367+
]
368+
},
317369
{
318370
"cell_type": "markdown",
319371
"metadata": {
@@ -458,6 +510,120 @@
458510
"await main()"
459511
]
460512
},
513+
{
514+
"cell_type": "markdown",
515+
"metadata": {
516+
"id": "907da7836dcf"
517+
},
518+
"source": [
519+
"### **Example 4**: Function calling\n",
520+
"\n",
521+
"You can use function calling to create a description of a function, then pass that description to the model in a request. The response from the model includes the name of a function that matches the description and the arguments to call it with.\n",
522+
"\n",
523+
"**Notes**:\n",
524+
"\n",
525+
"- All functions must be declared at the start of the session by sending tool definitions.\n",
526+
"- Currently only one tool is supported in the API."
527+
]
528+
},
529+
{
530+
"cell_type": "code",
531+
"execution_count": null,
532+
"metadata": {
533+
"id": "0f4657af21e3"
534+
},
535+
"outputs": [],
536+
"source": [
537+
"get_current_weather = FunctionDeclaration(\n",
538+
" name=\"get_current_weather\",\n",
539+
" description=\"Get current weather in the given location\",\n",
540+
" parameters={\n",
541+
" \"type\": \"OBJECT\",\n",
542+
" \"properties\": {\n",
543+
" \"location\": {\n",
544+
" \"type\": \"STRING\",\n",
545+
" },\n",
546+
" },\n",
547+
" },\n",
548+
")\n",
549+
"\n",
550+
"config = LiveConnectConfig(\n",
551+
" response_modalities=[\"TEXT\"],\n",
552+
" tools=[Tool(function_declarations=[get_current_weather])],\n",
553+
")\n",
554+
"\n",
555+
"async with client.aio.live.connect(\n",
556+
" model=MODEL_ID,\n",
557+
" config=config,\n",
558+
") as session:\n",
559+
" text_input = \"Get the current weather in Santa Clara, San Jose and Mountain View\"\n",
560+
" display(Markdown(f\"**Input:** {text_input}\"))\n",
561+
"\n",
562+
" await session.send(input=text_input, end_of_turn=True)\n",
563+
"\n",
564+
" async for message in session.receive():\n",
565+
" if message.tool_call:\n",
566+
" for function_call in message.tool_call.function_calls:\n",
567+
" display(Markdown(f\"**FunctionCall >** {str(function_call)}\"))"
568+
]
569+
},
570+
{
571+
"cell_type": "markdown",
572+
"metadata": {
573+
"id": "23cb7ab89311"
574+
},
575+
"source": [
576+
"### **Example 5**: Code Execution\n",
577+
"\n",
578+
" You can use code execution capability to generate and execute Python code directly within the API.\n",
579+
"\n",
580+
" In this example, you initialize the code execution tool by passing `code_execution` in a `Tool` definition, and register this tool with the model when a session is initiated."
581+
]
582+
},
583+
{
584+
"cell_type": "code",
585+
"execution_count": null,
586+
"metadata": {
587+
"id": "631b0c59c516"
588+
},
589+
"outputs": [],
590+
"source": [
591+
"config = LiveConnectConfig(\n",
592+
" response_modalities=[\"TEXT\"],\n",
593+
" tools=[Tool(code_execution=ToolCodeExecution())],\n",
594+
")\n",
595+
"\n",
596+
"async with client.aio.live.connect(\n",
597+
" model=MODEL_ID,\n",
598+
" config=config,\n",
599+
") as session:\n",
600+
" text_input = \"Write code to calculate the 15th fibonacci number then find the nearest palindrome to it\"\n",
601+
" display(Markdown(f\"**Input:** {text_input}\"))\n",
602+
"\n",
603+
" await session.send(input=text_input, end_of_turn=True)\n",
604+
"\n",
605+
" response = []\n",
606+
"\n",
607+
" async for message in session.receive():\n",
608+
" if message.text:\n",
609+
" response.append(message.text)\n",
610+
" if message.server_content.model_turn.parts:\n",
611+
" for part in message.server_content.model_turn.parts:\n",
612+
" if part.executable_code:\n",
613+
" display(\n",
614+
" Markdown(\n",
615+
" f\"\"\"\n",
616+
"**Executable code:**\n",
617+
"```py\n",
618+
"{part.executable_code.code}\n",
619+
"```\n",
620+
"\"\"\"\n",
621+
" )\n",
622+
" )\n",
623+
"\n",
624+
" display(Markdown(f\"**Response >** {''.join(response)}\"))"
625+
]
626+
},
461627
{
462628
"cell_type": "markdown",
463629
"metadata": {

0 commit comments

Comments
 (0)