00-playground/53-building-agentic-rag/L3_Building_an_Agent_Reasoning_Loop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64ec2f0f",
   "metadata": {},
   "source": [
    "# Lesson 3: Building an Agent Reasoning Loop"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0d7f1cf",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b07baa43-7a51-4c39-91cc-aa0d9619b69f",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from helper import get_openai_api_key\n",
    "OPENAI_API_KEY = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "dcfa86a3-c7f2-41fa-b8b6-5617659ec36a",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d3af4bb",
   "metadata": {},
   "source": [
    "## Load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d8bfb34",
   "metadata": {
    "tags": []
   },
   "source": [
    "To download this paper, below is the needed code:\n",
    "\n",
    "#!wget \"https://openreview.net/pdf?id=VtmBAGCN7o\" -O metagpt.pdf\n",
    "\n",
    "**Note**: The pdf file is included with this lesson. To access it, go to the `File` menu and select`Open...`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb741560",
   "metadata": {},
   "source": [
    "## Setup the Query Tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "77464fb2-5ace-4839-9032-a020df8d4259",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from utils import get_doc_tools\n",
    "\n",
    "vector_tool, summary_tool = get_doc_tools(\"metagpt.pdf\", \"metagpt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40aae3fc",
   "metadata": {},
   "source": [
    "## Setup Function Calling Agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ff4f5199-d02c-47b0-a9ab-cf72c8a506a3",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\", temperature=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "9365d78d-8e9f-4f22-8d57-35a4c6aa6baf",
   "metadata": {
    "height": 181,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionCallingAgentWorker\n",
    "from llama_index.core.agent import AgentRunner\n",
    "\n",
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    [vector_tool, summary_tool], \n",
    "    llm=llm, \n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5a9535d7-0baf-4905-ad16-5fb903d33b85",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the agent roles in MetaGPT, and then how they communicate with each other.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"agent roles in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "Agent roles in MetaGPT include Product Manager, Architect, Project Manager, Engineer, QA Engineer, CEO, Director of the AI Initiative, researchers, individuals contributing to evaluation metric design, creators of illustrative materials, reviewers and editors of papers, and individuals providing comprehensive feedback.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"how agents communicate with each other in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "Agents in MetaGPT communicate with each other through structured communication interfaces, utilizing documents and diagrams as structured outputs rather than dialogue. They share information using a publish-subscribe mechanism, where they exchange messages directly in a shared message pool. Additionally, agents in MetaGPT use a subscription mechanism based on their role-specific interests to extract relevant information, allowing them to select information to follow based on their role profiles. This approach enables directional information exchange among different roles and facilitates efficient communication within the multi-agent system.\n",
      "=== LLM Response ===\n",
      "In MetaGPT, the agent roles include Product Manager, Architect, Project Manager, Engineer, QA Engineer, CEO, Director of the AI Initiative, researchers, individuals contributing to evaluation metric design, creators of illustrative materials, reviewers and editors of papers, and individuals providing comprehensive feedback.\n",
      "\n",
      "Agents in MetaGPT communicate with each other through structured communication interfaces, utilizing documents and diagrams as structured outputs rather than dialogue. They share information using a publish-subscribe mechanism, where they exchange messages directly in a shared message pool. Additionally, agents use a subscription mechanism based on their role-specific interests to extract relevant information, enabling directional information exchange among different roles and facilitating efficient communication within the multi-agent system.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\n",
    "    \"Tell me about the agent roles in MetaGPT, \"\n",
    "    \"and then how they communicate with each other.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bcf74ec4-559f-4284-9ed0-817d26951c54",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "page_label: 1\n",
      "file_name: metagpt.pdf\n",
      "file_path: metagpt.pdf\n",
      "file_type: application/pdf\n",
      "file_size: 16911937\n",
      "creation_date: 2025-03-11\n",
      "last_modified_date: 2024-06-24\n",
      "\n",
      "Preprint\n",
      "METAGPT: M ETA PROGRAMMING FOR A\n",
      "MULTI -AGENT COLLABORATIVE FRAMEWORK\n",
      "Sirui Hong1∗, Mingchen Zhuge2∗, Jonathan Chen1, Xiawu Zheng3, Yuheng Cheng4,\n",
      "Ceyao Zhang4,Jinlin Wang1,Zili Wang ,Steven Ka Shing Yau5,Zijuan Lin4,\n",
      "Liyang Zhou6,Chenyu Ran1,Lingfeng Xiao1,7,Chenglin Wu1†,J¨urgen Schmidhuber2,8\n",
      "1DeepWisdom,2AI Initiative, King Abdullah University of Science and Technology,\n",
      "3Xiamen University,4The Chinese University of Hong Kong, Shenzhen,\n",
      "5Nanjing University,6University of Pennsylvania,\n",
      "7University of California, Berkeley,8The Swiss AI Lab IDSIA/USI/SUPSI\n",
      "ABSTRACT\n",
      "Remarkable progress has been made on automated problem solving through so-\n",
      "cieties of agents based on large language models (LLMs). Existing LLM-based\n",
      "multi-agent systems can already solve simple dialogue tasks. Solutions to more\n",
      "complex tasks, however, are complicated through logic inconsistencies due to\n",
      "cascading hallucinations caused by naively chaining LLMs. Here we introduce\n",
      "MetaGPT, an innovative meta-programming framework incorporating efficient\n",
      "human workflows into LLM-based multi-agent collaborations. MetaGPT en-\n",
      "codes Standardized Operating Procedures (SOPs) into prompt sequences for more\n",
      "streamlined workflows, thus allowing agents with human-like domain expertise\n",
      "to verify intermediate results and reduce errors. MetaGPT utilizes an assembly\n",
      "line paradigm to assign diverse roles to various agents, efficiently breaking down\n",
      "complex tasks into subtasks involving many agents working together. On col-\n",
      "laborative software engineering benchmarks, MetaGPT generates more coherent\n",
      "solutions than previous chat-based multi-agent systems. Our project can be found\n",
      "at https://github.com/geekan/MetaGPT.\n",
      "1 I NTRODUCTION\n",
      "Autonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en-\n",
      "hance and replicate human workflows. In real-world applications, however, existing systems (Park\n",
      "et al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023;\n",
      "Liang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle to achieve\n",
      "effective, coherent, and accurate problem-solving processes, particularly when there is a need for\n",
      "meaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou\n",
      "et al., 2023; Qian et al., 2023).\n",
      "Through extensive collaborative practice, humans have developed widely accepted Standardized\n",
      "Operating Procedures (SOPs) across various domains (Belbin, 2012; Manifesto, 2001; DeMarco &\n",
      "Lister, 2013). These SOPs play a critical role in supporting task decomposition and effective coor-\n",
      "dination. Furthermore, SOPs outline the responsibilities of each team member, while establishing\n",
      "standards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe-\n",
      "cution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001;\n",
      "DeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company,\n",
      "Product Managers analyze competition and user needs to create Product Requirements Documents\n",
      "(PRDs) using a standardized structure, to guide the developmental process.\n",
      "Inspired by such ideas, we design a promising GPT -based Meta -Programming framework called\n",
      "MetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al.,\n",
      "2023), MetaGPT requires agents to generate structured outputs, such as high-quality requirements\n",
      "∗These authors contributed equally to this work.\n",
      "†Chenglin Wu (alexanderwu@fuzhi.ai) is the corresponding author, affiliated with DeepWisdom.\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(response.source_nodes[0].get_content(metadata_mode=\"all\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6b28c184-0b65-4e38-808e-d91a285aaefe",
   "metadata": {
    "height": 64,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the evaluation datasets used.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"evaluation datasets used in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The evaluation datasets used in MetaGPT include HumanEval, MBPP, and the SoftwareDev dataset.\n",
      "=== LLM Response ===\n",
      "The evaluation datasets used in MetaGPT include HumanEval, MBPP, and the SoftwareDev dataset.\n"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\n",
    "    \"Tell me about the evaluation datasets used.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d9586cef-21b5-4732-b95d-619462b4aaf6",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me the results over one of the above datasets.\n",
      "=== Calling Function ===\n",
      "Calling function: vector_tool_metagpt with args: {\"query\": \"results over HumanEval dataset\"}\n",
      "=== Function Output ===\n",
      "MetaGPT achieved 85.9% and 87.7% Pass rates over the HumanEval dataset.\n",
      "=== LLM Response ===\n",
      "MetaGPT achieved 85.9% and 87.7% Pass rates over the HumanEval dataset.\n"
     ]
    }
   ],
   "source": [
    "response = agent.chat(\"Tell me the results over one of the above datasets.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cc4e983",
   "metadata": {},
   "source": [
    "## Lower-Level: Debuggability and Control"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "55abad72-b189-471a-accc-1621fd19c804",
   "metadata": {
    "height": 130,
    "tags": []
   },
   "outputs": [],
   "source": [
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    [vector_tool, summary_tool], \n",
    "    llm=llm, \n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "18e911aa-4640-4f89-99c8-6cdf6aff07c6",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [],
   "source": [
    "task = agent.create_task(\n",
    "    \"Tell me about the agent roles in MetaGPT, \"\n",
    "    \"and then how they communicate with each other.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5eaf0b88-e03a-4dd9-91f6-f6f0c8758e64",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the agent roles in MetaGPT, and then how they communicate with each other.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"agent roles in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The agent roles in MetaGPT include Product Manager, Architect, Project Manager, Engineer, Developer, and QA Engineer. Each role has specific responsibilities tailored to different aspects of the software development process, such as analyzing user requirements, translating requirements into system design components, distributing tasks, implementing design and writing code, and ensuring software quality through testing procedures. These roles work together in a structured workflow facilitated by efficient communication mechanisms like documents, diagrams, and subscriptions to develop software solutions effectively.\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a8e77fac-8734-4071-a672-b3a9f30e2bf1",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num completed for task d2f575ee-b65d-4ad3-9cf3-9d7a019cab4a: 1\n",
      "The agent roles in MetaGPT include Product Manager, Architect, Project Manager, Engineer, Developer, and QA Engineer. Each role has specific responsibilities tailored to different aspects of the software development process, such as analyzing user requirements, translating requirements into system design components, distributing tasks, implementing design and writing code, and ensuring software quality through testing procedures. These roles work together in a structured workflow facilitated by efficient communication mechanisms like documents, diagrams, and subscriptions to develop software solutions effectively.\n"
     ]
    }
   ],
   "source": [
    "completed_steps = agent.get_completed_steps(task.task_id)\n",
    "print(f\"Num completed for task {task.task_id}: {len(completed_steps)}\")\n",
    "print(completed_steps[0].output.sources[0].raw_output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "db8de410-4b82-4daf-93da-28da57cbb0bb",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Num upcoming steps for task d2f575ee-b65d-4ad3-9cf3-9d7a019cab4a: 1\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TaskStep(task_id='d2f575ee-b65d-4ad3-9cf3-9d7a019cab4a', step_id='3bf46cd3-6565-4de5-8cda-abb7450e7b8d', input=None, step_state={}, next_steps={}, prev_steps={}, is_ready=True)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "upcoming_steps = agent.get_upcoming_steps(task.task_id)\n",
    "print(f\"Num upcoming steps for task {task.task_id}: {len(upcoming_steps)}\")\n",
    "upcoming_steps[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dc352582-2c17-46ef-ba80-0f571e920c3c",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: What about how agents share information?\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"how agents share information in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "Agents in MetaGPT share information through a structured communication protocol that includes a shared message pool. This pool allows agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange information, obtain directional guidance from other roles, and access public information from the environment within the framework of MetaGPT. Additionally, agents review previous feedback before each project, adjust constraint prompts based on this feedback, and summarize how they can improve in a constraint prompt by evaluating received feedback during the development of previous projects. This information is stored in long-term memory for future updates.\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(\n",
    "    task.task_id, input=\"What about how agents share information?\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "be80661f-81b1-45fc-b0ba-33a04dae849b",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=== LLM Response ===\n",
      "Agents in MetaGPT share information through a structured communication protocol that includes a shared message pool. This pool allows agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange information, obtain directional guidance from other roles, and access public information from the environment within the framework of MetaGPT. Additionally, agents review previous feedback before each project, adjust constraint prompts based on this feedback, and summarize how they can improve in a constraint prompt by evaluating received feedback during the development of previous projects. This information is stored in long-term memory for future updates.\n",
      "True\n"
     ]
    }
   ],
   "source": [
    "step_output = agent.run_step(task.task_id)\n",
    "print(step_output.is_last)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4496328c-e6d5-4722-a8df-78a73a441b3c",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = agent.finalize_response(task.task_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "601d1bed-78b2-4512-87ac-aec5ce5d8494",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "assistant: Agents in MetaGPT share information through a structured communication protocol that includes a shared message pool. This pool allows agents to publish structured messages and subscribe to relevant messages based on their profiles. By utilizing this communication interface, agents can efficiently exchange information, obtain directional guidance from other roles, and access public information from the environment within the framework of MetaGPT. Additionally, agents review previous feedback before each project, adjust constraint prompts based on this feedback, and summarize how they can improve in a constraint prompt by evaluating received feedback during the development of previous projects. This information is stored in long-term memory for future updates.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7bb2ef28",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}