00-playground/53-building-agentic-rag/L4_Building_a_Multi-Document_Agent.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0b523e0a",
   "metadata": {},
   "source": [
    "# Lesson 4: Building a Multi-Document Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a323703",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b9625ab2-71b6-4fd0-904e-42df80d3215f",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from helper import get_openai_api_key\n",
    "OPENAI_API_KEY = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3221a474-5817-4db2-af46-e029042a75a5",
   "metadata": {
    "height": 47,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20adaa26",
   "metadata": {},
   "source": [
    "## 1. Setup an agent over 3 papers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48b71ff6",
   "metadata": {},
   "source": [
    "**Note**: The pdf files are included with this lesson. To access these papers, go to the `File` menu and select`Open...`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ed10a24b-d65c-4b98-a93a-94ccdb8900d0",
   "metadata": {
    "height": 200,
    "tags": []
   },
   "outputs": [],
   "source": [
    "urls = [\n",
    "    \"https://openreview.net/pdf?id=VtmBAGCN7o\",\n",
    "    \"https://openreview.net/pdf?id=6PmJoRfdaK\",\n",
    "    \"https://openreview.net/pdf?id=hSyW5go0v8\",\n",
    "]\n",
    "\n",
    "papers = [\n",
    "    \"metagpt.pdf\",\n",
    "    \"longlora.pdf\",\n",
    "    \"selfrag.pdf\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0d8f3185-3221-4b00-bd38-41d36e4a3307",
   "metadata": {
    "height": 164,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting tools for paper: metagpt.pdf\n",
      "Getting tools for paper: longlora.pdf\n",
      "Getting tools for paper: selfrag.pdf\n"
     ]
    }
   ],
   "source": [
    "from utils import get_doc_tools\n",
    "from pathlib import Path\n",
    "\n",
    "paper_to_tools_dict = {}\n",
    "for paper in papers:\n",
    "    print(f\"Getting tools for paper: {paper}\")\n",
    "    vector_tool, summary_tool = get_doc_tools(paper, Path(paper).stem)\n",
    "    paper_to_tools_dict[paper] = [vector_tool, summary_tool]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "0e541bdd-14e1-41b6-81b5-b1bfda078d07",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [],
   "source": [
    "initial_tools = [t for paper in papers for t in paper_to_tools_dict[paper]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "bff58c52",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "from llama_index.llms.openai import OpenAI\n",
    "\n",
    "llm = OpenAI(model=\"gpt-3.5-turbo\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2f2c6a9f",
   "metadata": {
    "height": 30
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(initial_tools)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a124a438-5609-402e-8642-69d1088cb9ad",
   "metadata": {
    "height": 181,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionCallingAgentWorker\n",
    "from llama_index.core.agent import AgentRunner\n",
    "\n",
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    initial_tools, \n",
    "    llm=llm, \n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "17409d4c-05a9-4bf4-b74f-75135fa3cb6b",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the evaluation dataset used in LongLoRA, and then tell me about the evaluation results\n",
      "=== Calling Function ===\n",
      "Calling function: vector_tool_longlora with args: {\"query\": \"evaluation dataset\"}\n",
      "=== Function Output ===\n",
      "PG19 test split\n",
      "=== Calling Function ===\n",
      "Calling function: vector_tool_longlora with args: {\"query\": \"evaluation results\"}\n",
      "=== Function Output ===\n",
      "The evaluation results show that the models achieve better perplexity with longer context sizes. Increasing the context window size leads to improved perplexity values. Additionally, the models are fine-tuned on different context lengths, such as 100k, 65536, and 32768, and achieve promising results on these extremely large settings. However, there is some perplexity degradation observed on small context sizes for the extended models, which is a known limitation of Position Interpolation.\n",
      "=== LLM Response ===\n",
      "The evaluation dataset used in LongLoRA is the PG19 test split. \n",
      "\n",
      "Regarding the evaluation results, the models in LongLoRA achieve better perplexity with longer context sizes. Increasing the context window size leads to improved perplexity values. The models are fine-tuned on different context lengths, such as 100k, 65536, and 32768, and achieve promising results on these extremely large settings. However, there is some perplexity degradation observed on small context sizes for the extended models, which is a known limitation of Position Interpolation.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\n",
    "    \"Tell me about the evaluation dataset used in LongLoRA, \"\n",
    "    \"and then tell me about the evaluation results\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ace340b1-761f-4058-be41-68cf131541e4",
   "metadata": {
    "height": 62,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Give me a summary of both Self-RAG and LongLoRA\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_selfrag with args: {\"input\": \"Self-RAG\"}\n",
      "=== Function Output ===\n",
      "Self-RAG is a framework that enhances the quality and factuality of a large language model by incorporating retrieval and self-reflection mechanisms. It involves training a critic model and a generator model to generate text with reflection tokens for self-evaluation during the inference phase. The system aims to control its behavior by dynamically deciding when to retrieve text passages and generating special tokens to evaluate its own predictions after each generated segment. Self-RAG demonstrates significant performance improvements in various tasks compared to state-of-the-art language models and retrieval-augmented models.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_longlora with args: {\"input\": \"LongLoRA\"}\n",
      "=== Function Output ===\n",
      "LongLoRA is an efficient method for extending the context length of Large Language Models (LLMs) while minimizing computational costs. It combines shifted sparse attention (S2-Attn) with LoRA to effectively fine-tune models to longer context lengths. This approach allows for significant context extension while retaining the original model architectures and compatibility with existing techniques. Additionally, LongLoRA demonstrates strong empirical results on various tasks and is available for fine-tuning LLMs up to 100k context on a single 8×A100 machine.\n",
      "=== LLM Response ===\n",
      "Self-RAG is a framework that enhances the quality and factuality of a large language model by incorporating retrieval and self-reflection mechanisms. It involves training a critic model and a generator model to generate text with reflection tokens for self-evaluation during the inference phase. The system aims to control its behavior by dynamically deciding when to retrieve text passages and generating special tokens to evaluate its own predictions after each generated segment. Self-RAG demonstrates significant performance improvements in various tasks compared to state-of-the-art language models and retrieval-augmented models.\n",
      "\n",
      "LongLoRA is an efficient method for extending the context length of Large Language Models (LLMs) while minimizing computational costs. It combines shifted sparse attention (S2-Attn) with LoRA to effectively fine-tune models to longer context lengths. This approach allows for significant context extension while retaining the original model architectures and compatibility with existing techniques. Additionally, LongLoRA demonstrates strong empirical results on various tasks and is available for fine-tuning LLMs up to 100k context on a single 8×A100 machine.\n",
      "assistant: Self-RAG is a framework that enhances the quality and factuality of a large language model by incorporating retrieval and self-reflection mechanisms. It involves training a critic model and a generator model to generate text with reflection tokens for self-evaluation during the inference phase. The system aims to control its behavior by dynamically deciding when to retrieve text passages and generating special tokens to evaluate its own predictions after each generated segment. Self-RAG demonstrates significant performance improvements in various tasks compared to state-of-the-art language models and retrieval-augmented models.\n",
      "\n",
      "LongLoRA is an efficient method for extending the context length of Large Language Models (LLMs) while minimizing computational costs. It combines shifted sparse attention (S2-Attn) with LoRA to effectively fine-tune models to longer context lengths. This approach allows for significant context extension while retaining the original model architectures and compatibility with existing techniques. Additionally, LongLoRA demonstrates strong empirical results on various tasks and is available for fine-tuning LLMs up to 100k context on a single 8×A100 machine.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\"Give me a summary of both Self-RAG and LongLoRA\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7eede70c",
   "metadata": {},
   "source": [
    "## 2. Setup an agent over 11 papers"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18771e69",
   "metadata": {},
   "source": [
    "### Download 11 ICLR papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "60d01d2c-547f-4054-b0fe-ed9b1a9cc3b5",
   "metadata": {
    "height": 472,
    "tags": []
   },
   "outputs": [],
   "source": [
    "urls = [\n",
    "    \"https://openreview.net/pdf?id=VtmBAGCN7o\",\n",
    "    \"https://openreview.net/pdf?id=6PmJoRfdaK\",\n",
    "    \"https://openreview.net/pdf?id=LzPWWPAdY4\",\n",
    "    \"https://openreview.net/pdf?id=VTF8yNQM66\",\n",
    "    \"https://openreview.net/pdf?id=hSyW5go0v8\",\n",
    "    \"https://openreview.net/pdf?id=9WD9KwssyT\",\n",
    "    \"https://openreview.net/pdf?id=yV6fD7LYkF\",\n",
    "    \"https://openreview.net/pdf?id=hnrB5YHoYu\",\n",
    "    \"https://openreview.net/pdf?id=WbWtOYIzIK\",\n",
    "    \"https://openreview.net/pdf?id=c5pwL0Soay\",\n",
    "    \"https://openreview.net/pdf?id=TpD2aG1h0D\"\n",
    "]\n",
    "\n",
    "papers = [\n",
    "    \"metagpt.pdf\",\n",
    "    \"longlora.pdf\",\n",
    "    \"loftq.pdf\",\n",
    "    \"swebench.pdf\",\n",
    "    \"selfrag.pdf\",\n",
    "    \"zipformer.pdf\",\n",
    "    \"values.pdf\",\n",
    "    \"finetune_fair_diffusion.pdf\",\n",
    "    \"knowledge_card.pdf\",\n",
    "    \"metra.pdf\",\n",
    "    \"vr_mcl.pdf\"\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b77426cb",
   "metadata": {
    "tags": []
   },
   "source": [
    "To download these papers, below is the needed code:\n",
    "\n",
    "\n",
    "    #for url, paper in zip(urls, papers):\n",
    "         #!wget \"{url}\" -O \"{paper}\"\n",
    "    \n",
    "    \n",
    "**Note**: The pdf files are included with this lesson. To access these papers, go to the `File` menu and select`Open...`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ea5ee34d-02ac-4537-ae20-7ef6c5767172",
   "metadata": {
    "height": 164,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting tools for paper: metagpt.pdf\n",
      "Getting tools for paper: longlora.pdf\n",
      "Getting tools for paper: loftq.pdf\n",
      "Getting tools for paper: swebench.pdf\n",
      "Getting tools for paper: selfrag.pdf\n",
      "Getting tools for paper: zipformer.pdf\n",
      "Getting tools for paper: values.pdf\n",
      "Getting tools for paper: finetune_fair_diffusion.pdf\n",
      "Getting tools for paper: knowledge_card.pdf\n",
      "Getting tools for paper: metra.pdf\n",
      "Getting tools for paper: vr_mcl.pdf\n"
     ]
    }
   ],
   "source": [
    "from utils import get_doc_tools\n",
    "from pathlib import Path\n",
    "\n",
    "paper_to_tools_dict = {}\n",
    "for paper in papers:\n",
    "    print(f\"Getting tools for paper: {paper}\")\n",
    "    vector_tool, summary_tool = get_doc_tools(paper, Path(paper).stem)\n",
    "    paper_to_tools_dict[paper] = [vector_tool, summary_tool]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e35d52c",
   "metadata": {},
   "source": [
    "### Extend the Agent with Tool Retrieval"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "20154923-873e-4941-9a3a-4926ab5f9b8c",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [],
   "source": [
    "all_tools = [t for paper in papers for t in paper_to_tools_dict[paper]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "671582f9-70d7-4a8f-b813-58b2a068ca72",
   "metadata": {
    "height": 164,
    "tags": []
   },
   "outputs": [],
   "source": [
    "# define an \"object\" index and retriever over these tools\n",
    "from llama_index.core import VectorStoreIndex\n",
    "from llama_index.core.objects import ObjectIndex\n",
    "\n",
    "obj_index = ObjectIndex.from_objects(\n",
    "    all_tools,\n",
    "    index_cls=VectorStoreIndex,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c3929882-e9dc-46ca-b495-53e3ed60340e",
   "metadata": {
    "height": 45,
    "tags": []
   },
   "outputs": [],
   "source": [
    "obj_retriever = obj_index.as_retriever(similarity_top_k=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ba9cfecd-fe14-4da8-b9ba-b3d485d98a03",
   "metadata": {
    "height": 79,
    "tags": []
   },
   "outputs": [],
   "source": [
    "tools = obj_retriever.retrieve(\n",
    "    \"Tell me about the eval dataset used in MetaGPT and SWE-Bench\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c270ffbf-69c7-48ea-a028-9ba25221cde5",
   "metadata": {
    "height": 30,
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ToolMetadata(description='Useful for summarization questions related to swebench', name='summary_tool_swebench', fn_schema=<class 'llama_index.core.tools.types.DefaultToolFnSchema'>, return_direct=False)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tools[2].metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9cc0a0b6-9858-4348-9ae0-1cd4160f3fb7",
   "metadata": {
    "height": 266,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.agent import FunctionCallingAgentWorker\n",
    "from llama_index.core.agent import AgentRunner\n",
    "\n",
    "agent_worker = FunctionCallingAgentWorker.from_tools(\n",
    "    tool_retriever=obj_retriever,\n",
    "    llm=llm, \n",
    "    system_prompt=\"\"\" \\\n",
    "You are an agent designed to answer queries over a set of given papers.\n",
    "Please always use the tools provided to answer a question. Do not rely on prior knowledge.\\\n",
    "\n",
    "\"\"\",\n",
    "    verbose=True\n",
    ")\n",
    "agent = AgentRunner(agent_worker)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "a250cf1a-e011-4994-bcca-4e0294f20864",
   "metadata": {
    "height": 98,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Tell me about the evaluation dataset used in MetaGPT and compare it against SWE-Bench\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_metagpt with args: {\"input\": \"evaluation dataset used in MetaGPT\"}\n",
      "=== Function Output ===\n",
      "The evaluation dataset used in MetaGPT includes HumanEval and MBPP.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_swebench with args: {\"input\": \"evaluation dataset used in SWE-Bench\"}\n",
      "=== Function Output ===\n",
      "The evaluation dataset used in SWE-Bench consists of task instances extracted from various open source repositories, including real GitHub issues and pull requests from Python repositories. It includes task instructions, issue text, retrieved files, documentation, example patch files, and prompts for generating patch files. The dataset aims to establish a baseline for approaches to software engineering tasks, encourages experimentation with different methodologies, and covers a diverse set of task types beyond bug fixes. Models like ChatGPT-3.5, GPT-4, Claude 2, and SWE-Llama were evaluated on this dataset using BM25 retrieval and the \"oracle\" retrieval setting.\n",
      "=== LLM Response ===\n",
      "The evaluation dataset used in MetaGPT includes HumanEval and MBPP. \n",
      "\n",
      "The evaluation dataset used in SWE-Bench consists of task instances extracted from various open source repositories, including real GitHub issues and pull requests from Python repositories. It includes task instructions, issue text, retrieved files, documentation, example patch files, and prompts for generating patch files. The dataset aims to establish a baseline for approaches to software engineering tasks, encourages experimentation with different methodologies, and covers a diverse set of task types beyond bug fixes. Models like ChatGPT-3.5, GPT-4, Claude 2, and SWE-Llama were evaluated on this dataset using BM25 retrieval and the \"oracle\" retrieval setting.\n",
      "assistant: The evaluation dataset used in MetaGPT includes HumanEval and MBPP. \n",
      "\n",
      "The evaluation dataset used in SWE-Bench consists of task instances extracted from various open source repositories, including real GitHub issues and pull requests from Python repositories. It includes task instructions, issue text, retrieved files, documentation, example patch files, and prompts for generating patch files. The dataset aims to establish a baseline for approaches to software engineering tasks, encourages experimentation with different methodologies, and covers a diverse set of task types beyond bug fixes. Models like ChatGPT-3.5, GPT-4, Claude 2, and SWE-Llama were evaluated on this dataset using BM25 retrieval and the \"oracle\" retrieval setting.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\n",
    "    \"Tell me about the evaluation dataset used \"\n",
    "    \"in MetaGPT and compare it against SWE-Bench\"\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "8084c8cb-98ed-4835-aaa4-5b0c7254be6d",
   "metadata": {
    "height": 96,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Added user message to memory: Compare and contrast the LoRA papers (LongLoRA, LoftQ). Analyze the approach in each paper first. \n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_longlora with args: {\"input\": \"Compare and contrast the approach in the LongLoRA paper.\"}\n",
      "=== Function Output ===\n",
      "The LongLoRA paper introduces an efficient fine-tuning approach for extending the context sizes of pre-trained large language models (LLMs) with limited computation cost. It utilizes shifted sparse attention (S2-Attn) during training to approximate long context efficiently, while retaining the original attention architecture for inference. This method allows for extending the context window of LLMs while reducing computational costs compared to full fine-tuning approaches. In contrast, traditional low-rank adaptation (LoRA) for context extension is found to be less effective and efficient in comparison. LongLoRA combines S2-Attn with LoRA to achieve strong empirical results on various tasks with different LLM sizes. The approach focuses on trainable embedding and normalization layers as key components for successful long context fine-tuning, ensuring performance while minimizing additional trainable parameters.\n",
      "=== Calling Function ===\n",
      "Calling function: summary_tool_loftq with args: {\"input\": \"Compare and contrast the approach in the LoftQ paper.\"}\n",
      "=== Function Output ===\n",
      "The approach in the LoftQ paper focuses on simultaneously quantizing a Large Language Model (LLM) and finding a suitable low-rank initialization for LoRA fine-tuning. This method integrates low-rank approximation with quantization to align with the original pre-trained weights, providing a better initialization point for subsequent LoRA fine-tuning and improving generalization in downstream tasks. In contrast, traditional quantization methods like QLoRA primarily concentrate on quantization techniques and may overlook the importance of subsequent LoRA fine-tuning. LoftQ actively addresses the quantization discrepancy by jointly optimizing the initial values of the quantized backbone and low-rank adapters, offering a more effective approach for downstream task performance improvement. Additionally, LoftQ iterates through alternating optimization steps to refine the quantized weights and low-rank adapters, demonstrating stability and robustness while outperforming baseline methods like QLoRA in terms of convergence and performance across various datasets and quantization levels.\n",
      "=== LLM Response ===\n",
      "The LongLoRA paper introduces an efficient fine-tuning approach for extending the context sizes of pre-trained large language models (LLMs) with limited computation cost. It utilizes shifted sparse attention (S2-Attn) during training to approximate long context efficiently, while retaining the original attention architecture for inference. This method allows for extending the context window of LLMs while reducing computational costs compared to full fine-tuning approaches. In contrast, traditional low-rank adaptation (LoRA) for context extension is found to be less effective and efficient in comparison. LongLoRA combines S2-Attn with LoRA to achieve strong empirical results on various tasks with different LLM sizes. The approach focuses on trainable embedding and normalization layers as key components for successful long context fine-tuning, ensuring performance while minimizing additional trainable parameters.\n",
      "\n",
      "On the other hand, the approach in the LoftQ paper focuses on simultaneously quantizing a Large Language Model (LLM) and finding a suitable low-rank initialization for LoRA fine-tuning. This method integrates low-rank approximation with quantization to align with the original pre-trained weights, providing a better initialization point for subsequent LoRA fine-tuning and improving generalization in downstream tasks. In contrast, traditional quantization methods like QLoRA primarily concentrate on quantization techniques and may overlook the importance of subsequent LoRA fine-tuning. LoftQ actively addresses the quantization discrepancy by jointly optimizing the initial values of the quantized backbone and low-rank adapters, offering a more effective approach for downstream task performance improvement. Additionally, LoftQ iterates through alternating optimization steps to refine the quantized weights and low-rank adapters, demonstrating stability and robustness while outperforming baseline methods like QLoRA in terms of convergence and performance across various datasets and quantization levels.\n"
     ]
    }
   ],
   "source": [
    "response = agent.query(\n",
    "    \"Compare and contrast the LoRA papers (LongLoRA, LoftQ). \"\n",
    "    \"Analyze the approach in each paper first. \"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "36cee1e5",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}