Lesson-4 Retrieval QA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson 4: Q&A over Documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **1. Components of the Retrieval Q&A Chain**\n",
    "\n",
    "#### **1.1 Document Store**\n",
    "The document store is where all your documents are stored and indexed for retrieval. Popular options include:\n",
    "\n",
    "- **FAISS (Facebook AI Similarity Search):** For vector-based similarity search.\n",
    "- **Pinecone:** A scalable vector database for high-performance retrieval.\n",
    "- **Weaviate or Chroma:** Modern alternatives with feature-rich capabilities.\n",
    "\n",
    "Here, we're using **DocArrayInMemorySearch**, as it is suitable for small scale applications like this one, whereas the above DBs are more suited for large-scale applications\n",
    "\n",
    "The document store allows for the efficient retrieval of documents based on vector similarity.\n",
    "\n",
    "#### **1.2 Embedding Model**\n",
    "The embedding model converts documents and user queries into dense vector representations. These embeddings capture semantic meaning and are essential for similarity searches\n",
    "\n",
    "#### **1.3 Retriever**\n",
    "The retriever is responsible for searching the document store and returning the most relevant documents based on the query embedding. Two main types of retrieval methods are used:\n",
    "\n",
    "- **Similarity-based retrieval:** Finds documents closest to the query in vector space.\n",
    "- **Hybrid retrieval:** Combines traditional keyword search with vector similarity.\n",
    "\n",
    "#### **1.4 Large Language Model (LLM)**\n",
    "The LLM interprets the retrieved documents and generates an accurate and contextually appropriate answer to the user’s query.\n",
    "\n",
    "#### **1.5 Chain Logic**\n",
    "Chains in LangChain enable the combination of multiple components into a coherent pipeline. For Retrieval Q&A, the chain typically involves:\n",
    "\n",
    "- Embedding the query.\n",
    "- Retrieving relevant documents.\n",
    "- Answer generation using the LLM.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And again, Andrew Ng's lesson used deprecated classes, so here I use the latest ones, as suggested by LangChain: https://python.langchain.com/docs/versions/migrating_chains/retrieval_qa/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_ollama import ChatOllama, OllamaLLM, OllamaEmbeddings\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.vectorstores import DocArrayInMemorySearch\n",
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain import hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Creating Document Store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1 Loading Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n",
      "page_content='SUFIA: Language-Guided Augmented Dexterity\n",
      "for Robotic Surgical Assistants\n",
      "Masoud Moghani1, Lars Doorenbos 2, William Chung-Ho Panitch 3,\n",
      "Sean Huver4, Mahdi Azizian 4, Ken Goldberg 3, Animesh Garg 1,4,5\n",
      "Abstract— In this work, we present SUFIA , the first frame-\n",
      "work for natural language-guided augmented dexterity for\n",
      "robotic surgical assistants. SUFIA incorporates the strong\n",
      "reasoning capabilities of large language models (LLMs) with\n",
      "perception modules to implement high-level planning and low-\n",
      "level control of a robot for surgical sub-task execution. This\n",
      "enables a learning-free approach to surgical augmented dexterity\n",
      "without any in-context examples or motion primitives. SUFIA\n",
      "uses a human-in-the-loop paradigm by restoring control to\n",
      "the surgeon in the case of insufficient information, mitigating\n",
      "unexpected errors for mission-critical tasks. We evaluate SUFIA\n",
      "on four surgical sub-tasks in a simulation environment and two\n",
      "sub-tasks on a physical surgical robotic platform in the lab,\n",
      "demonstrating its ability to perform common surgical sub-tasks\n",
      "through supervised autonomous operation under challenging\n",
      "physical and workspace conditions.\n",
      "Project website: orbit-surgical.github.io/sufia\n",
      "I. I NTRODUCTION\n",
      "Recently, one prominent trend in surgery has been the\n",
      "increasing adoption of robotic surgical assistants (RSAs) in\n",
      "operating rooms. These RSAs are often controlled via local\n",
      "or remote teleoperation through a console by a trained human\n",
      "surgeon using hand controllers or other input peripherals,\n",
      "thereby enabling the surgeon to perform tasks with enhanced\n",
      "precision, dexterity, and control during an operation [ 1]. The\n",
      "teleoperated surgical procedures often involve tedious, repeti-\n",
      "tive, or time-consuming sub-tasks. Augmented dexterity in\n",
      "surgery holds the potential to simplify the surgical workflow,\n",
      "reduce surgeon fatigue, and improve patient outcomes [ 2],\n",
      "[3].\n",
      "Learning-based approaches such as reinforcement and\n",
      "imitation learning learn policies to solve specific surgical sub-\n",
      "tasks [4], [5]. However, complex, long-horizon surgical sub-\n",
      "tasks are often computationally expensive, require extensive\n",
      "domain knowledge and reward engineering, and involve\n",
      "time-consuming dataset curation. Furthermore, the lack of\n",
      "generalizability limits the utility of learning-based models in\n",
      "safety-critical applications where unseen, in-domain variations\n",
      "are prevalent. As a result, most surgical robotic platforms\n",
      "still lack any level of autonomous capabilities [6].\n",
      "In recent years, Large Language Models (LLMs) have\n",
      "received considerable attention for their ability to respond\n",
      "naturally to textual prompts and have been integrated into var-\n",
      "ious domains, including the field of robotics and autonomous\n",
      "agents [7]. Language and vision models have demonstrated\n",
      "considerable promise in long-horizon robot planning and\n",
      "1University of Toronto, 2University of Bern, 3University of California,\n",
      "Berkeley, 4NVIDIA, 5Georgia Institute of Technology\n",
      "moghani@cs.toronto.edu, animesh.garg@gatech.edu\n",
      "High-levelPlanning Low-levelCode Feedback\n",
      "LLM Generation\n",
      "RobotActionPerception\n",
      "Surgeon: \n",
      "“Please lift the needle”\n",
      "Assistant:\n",
      "“The needle has been successfully \n",
      "lifted to the correct position”\n",
      "Re-planning Autonomy Delegation\n",
      "needle\n",
      "Fig. 1: An overview of SUFIA automating the lifting of a\n",
      "suture needle from a surgical site. SUFIA receives commands\n",
      "from a surgeon in natural language and converts them to high-\n",
      "level planning and low-level control code. If a task requires object\n",
      "interaction, SUFIA queries a perception module for object state\n",
      "information and generates low-level trajectories and robot actions\n",
      "accordingly. SUFIA can assist a surgeon with open-ended tasks,\n",
      "such as moving the robot in a desired motion to help complete a\n",
      "surgical task. In times of inefficient information, SUFIA delegates\n",
      "full control back to the surgeon.\n",
      "control [ 8], [ 9], [ 10]. While these efforts still require pre-\n",
      "trained skills and motion primitives, they have demonstrated\n",
      "the potential of unified many-modality models for addressing\n",
      "a variety of complex tasks involving improved generalization\n",
      "to novel objects and unseen tasks.\n",
      "In surgical settings, LLMs have the additional potential\n",
      "to aid interaction between a human surgeon and a robot via\n",
      "natural language teleoperation. This empowers the surgeon\n",
      "with the ability to use both fine-grained manual control\n",
      "and autonomous natural language conversational control in\n",
      "commanding the RSA to perform a sub-task. This approach\n",
      "promises both more natural human-robot coordination and\n",
      "the potential for developing general-purpose models for\n",
      "autonomous surgery beyond the capability of current task-by-\n",
      "arXiv:2405.05226v1  [cs.RO]  8 May 2024' metadata={'source': 'SuFIA.pdf', 'page': 0}\n"
     ]
    }
   ],
   "source": [
    "loader = PyPDFLoader(\n",
    "    file_path=\"SuFIA.pdf\",\n",
    "    extract_images=True,\n",
    "    )\n",
    "\n",
    "pages = [page for page in loader.lazy_load()]\n",
    "\n",
    "print(len(pages)) #the pdf has 8 pages, and this prints 8\n",
    "print(pages[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'langchain_core.documents.base.Document'>\n"
     ]
    }
   ],
   "source": [
    "print(type(pages[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4681\n"
     ]
    }
   ],
   "source": [
    "print(len(pages[0].page_content))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.2 Configuring the DB with document and embedding "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\milap\\envs\\langchain_course_env\\Lib\\site-packages\\pydantic\\_migration.py:283: UserWarning: `pydantic.error_wrappers:ValidationError` has been moved to `pydantic:ValidationError`.\n",
      "  warnings.warn(f'`{import_path}` has been moved to `{new_location}`.')\n"
     ]
    }
   ],
   "source": [
    "embeddings = OllamaEmbeddings(model=\"llama3.2\")\n",
    "\n",
    "db = DocArrayInMemorySearch.from_documents(pages, embeddings)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "llm = ChatOllama(model=\"llama3.2\")  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Creating the QnA chain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`db.as_retriever()` wraps the vector db into a retriever, and returns a `VectorStoreRetriever`.\\\n",
    "It acts acts as a bridge between a vector store and the query processing logic in a chain or pipeline.\\\n",
    "It Converts user queries into embeddings ( using the embedding model associated with the vector store) and \\\n",
    "retrieves the top-k (default k=4,by performing a similarity search in the vector store) most relevant documents using the vector store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\milap\\envs\\langchain_course_env\\Lib\\site-packages\\langsmith\\client.py:256: LangSmithMissingAPIKeyWarning: API key must be provided when using hosted LangSmith API\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "#See the full prompt at: https://smith.langchain.com/hub/langchain-ai/retrieval-qa-chat?tab=0\n",
    "retrieval_qa_chat_prompt = hub.pull(\"langchain-ai/retrieval-qa-chat\")\n",
    "\n",
    "combine_docs_chain = create_stuff_documents_chain(llm, retrieval_qa_chat_prompt)\n",
    "rag_chain = create_retrieval_chain(db.as_retriever(), combine_docs_chain)\n",
    "\n",
    "response = rag_chain.invoke({\"input\": \"How can LLMs be used in a surgical setting?\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Large Language Models (LLMs) have the potential to transform various fields, including surgery. Here are some ways LLMs can be used in a surgical setting:\n",
      "\n",
      "1. **Preoperative planning**: LLMs can help surgeons analyze patient data, such as medical history, imaging results, and laboratory tests, to provide personalized treatment plans.\n",
      "2. **Surgical documentation**: AI-powered LLMs can assist with surgical documentation by automatically generating reports, summaries of procedures, and even assisting with medical dictation.\n",
      "3. **Procedural guidance**: LLMs can be used to provide step-by-step instructions for surgeons during complex procedures, such as minimally invasive surgery or robotic-assisted surgeries.\n",
      "4. **Patient education**: AI-powered chatbots or virtual assistants built on LLMs can educate patients about their conditions, treatment options, and post-operative care, improving patient engagement and outcomes.\n",
      "5. **Annotated medical texts**: LLMs can be trained to analyze large volumes of annotated medical text data, enabling the creation of high-quality reference materials for surgical training and education.\n",
      "6. **Surgical simulation**: AI-powered LLMs can create simulated surgical environments, allowing surgeons to practice complex procedures in a safe and controlled setting.\n",
      "7. **Decision support systems**: LLMs can be integrated into decision support systems (DSS) that provide surgeons with evidence-based recommendations for diagnosis, treatment, and patient management.\n",
      "8. **Automated medical text analysis**: AI-powered LLMs can analyze unstructured clinical data, such as free-text notes or emails, to extract relevant information and identify potential issues or discrepancies.\n",
      "9. **Robotic-assisted surgery integration**: LLMs can be used to optimize robotic-assisted surgeries by providing real-time guidance on instrument placement, tissue tracking, and surgical technique optimization.\n",
      "10. **Research and development**: AI-powered LLMs can aid in the analysis of large datasets related to surgical outcomes, patient behavior, and treatment efficacy, driving research and innovation in surgery.\n",
      "\n",
      "While these applications hold promise, it's essential to note that LLMs should be used responsibly and under the guidance of human clinicians in a surgical setting.\n"
     ]
    }
   ],
   "source": [
    "print(response[\"answer\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Document(metadata={'source': 'SuFIA.pdf', 'page': 6}, page_content='SuFIA LLM Planner: To begin the task of finding the suturing site, I will first need to detect \\nthe location of the suturing site within the environment. I will call the `detect_object` function \\nwith the parameter \"suturing site\" to obtain its position.\\n```python\\ndetect_object(\"suturing site\")\\n```\\nSystem execution output: Object not found!\\nSuFIA LLM Planner: Since the \"suturing site\" object was not found in the environment, I \\nwill defer control to the surgeon to provide further instructions or to adjust the environment so \\nthat the suturing site can be detected.\\n```python\\ntransfer_control()\\n```\\nControl of the robot is transferred to the Surgeon.\\n(a) (b)\\n(c)\\nSurgeon: Please find the suturing site.\\nFig. 6: Interactive human-in-the-loop approach. (a) An overview of the environment showing the dVRK robotic arm and endoscope\\ncamera as well as a needle and a suturing pad in ORBIT -Surgical, (b) RGB image from the endoscope camera focused on the needle as\\ninput to the perception module, (c) delegation of control back to the surgeon due to the inability of the system to identify a suturing site.\\nThe uncertainty and occlusion in a surgical scene might lead to undesired object localization and retrieval. A human-in-the-loop approach\\nenables a fail-safe paradigm for interactive robotic surgical assistants.\\nLLM Function Call for Verification:\\n# Verify that the needle has been picked up and \\nlifted\\nverify_object(\"needle\", lifted_position, 0.015)\\nFunction Output: \\nverify_object output: False\\nLLM Planner: \\nThe verification step has returned `False`, indicating that \\nthe needle has not been successfully picked up and lifted \\nto the specified position. Therefore, I will need to re-open \\nthe gripper and plan a new trajectory to complete the task.\\nLLM Function Call for Verification:\\n# Verify that the needle has been picked up and \\nlifted\\nverify_object(\"needle\", lifted_position, 0.015)\\nFunction Output: \\nverify_object output: True\\n(a) Failed execution\\n(b) Re-planning and successful \\nexecution\\nFig. 7: Re-planning. (a) A failed execution as a result of not\\nfinding a proper grasping point on the needle. The LLM verification\\nstep indicates that the task was not executed successfully. (b) LLM\\nplanner proposes a new plan to re-identify and lift the needle by\\nthe robot.\\nH. Limitations\\nThe current best results are obtained with API calls to\\nGPT-4 Turbo. Due to the generation speed of OpenAI’s\\nAPI, SUFIA does not operate in real-time; for real-world\\nneedle handover trials, the API calls invoked during planning\\n(including sensing and replanning) took an average of 25.5\\nseconds to complete, out of an average total trial time of\\n61.4 seconds. However, with open-source models constantly\\nimproving, running a quantized open-source LLM on-device\\nwill soon be a viable way to improve inference time greatly.\\nMoreover, while we incorporate two measures specifically\\ndesigned to improve safety and reliability, deploying au-\\ntonomous or semi-autonomous RSAs in real-world scenarios\\nstill has the potential to bring risks from unexpected circum-\\nstances an AI system might not be able to handle.\\nVI. C ONCLUSION\\nWe present SUFIA , a modular framework for natural\\nsurgeon-robot interaction. We show that our training-free\\napproach, which uses pre-trained LLMs to provide low-level\\ncontrol of surgical robots, can successfully interact with\\nsmall surgical objects and execute surgeon commands for\\nautomating surgical sub-tasks. Safety is bolstered through\\nre-planning capabilities and a human-in-the-loop approach.\\nWe evaluate the efficacy of SUFIA for common surgical\\nsub-tasks in simulated and physical experiments in the lab\\nand show that the proposed method succeeds across different\\nsub-tasks with various difficulty levels. These results suggest\\nthat language-guided autonomy has the potential to enhance\\nsurgeon’s efficiency in surgical procedures.\\nIn future work, we plan to test the viability of quantized\\nopen-source LLMs on-device to improve inference time.\\nThis will also address any privacy concerns stemming from\\ntransmitting highly sensitive medical information to off-site\\nservers. Furthermore, we intend to explore the usefulness of\\nfine-tuned large language and vision models in S UFIA.\\nREFERENCES\\n[1] M. Hwang, D. Seita, B. Thananjeyan, J. Ichnowski, S. Paradis, D. Fer,\\nT. Low, and K. Goldberg, “Applying depth-sensing to automated\\nsurgical manipulation with a da vinci robot,” in 2020 international\\nsymposium on medical robotics (ISMR). IEEE, 2020, pp. 22–29.\\n[2] S. Sen, A. Garg, D. V . Gealy, S. McKinley, Y . Jen, and K. Goldberg,\\n“Automating multi-throw multilateral surgical suturing with a mechani-\\ncal needle guide and sequential convex optimization,” in 2016 IEEE\\ninternational conference on robotics and automation (ICRA). IEEE,\\n2016, pp. 4178–4185.\\n[3] H. Lin, B. Li, X. Chu, Q. Dou, Y . Liu, and K. W. S. Au, “End-to-\\nend learning of deep visuomotor policy for needle picking,” in 2023\\nIEEE/RSJ International Conference on Intelligent Robots and Systems\\n(IROS). IEEE, 2023, pp. 8487–8494.\\n[4] P. M. Scheikl, B. Gyenes, R. Younis, C. Haas, G. Neumann, M. Wagner,\\nand F. Mathis-Ullrich, “LapGym - An open source framework for\\nreinforcement learning in robot-assisted laparoscopic surgery,” Journal\\nof Machine Learning Research, vol. 24, no. 368, pp. 1–42, 2023.\\n[5] J. Xu, B. Li, B. Lu, Y .-H. Liu, Q. Dou, and P.-A. Heng, “Surrol:\\nAn open-source reinforcement learning centered and dvrk compatible\\nplatform for surgical robot learning,” in 2021 IEEE/RSJ International\\nConference on Intelligent Robots and Systems (IROS). IEEE, 2021,\\npp. 1821–1828.\\n[6] A. Attanasio, B. Scaglioni, E. De Momi, P. Fiorini, and P. Valdastri,\\n“Autonomy in surgical robotics,” Annual Review of Control, Robotics,\\nand Autonomous Systems, vol. 4, pp. 651–679, 2021.\\n[7] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen,\\nJ. Tang, X. Chen, Y . Lin et al., “A survey on large language model\\nbased autonomous agents,” arXiv preprint arXiv:2308.11432, 2023.\\n[8] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter,'),\n",
       " Document(metadata={'source': 'SuFIA.pdf', 'page': 4}, page_content='1 2 3 4\\nFig. 4: Physical Needle Handover task. (1) Starting workspace configuration. The needle is placed in a fixed position within the\\nworkspace, and the gripper positions are randomized. In this stage, the SUFIA LLM planner queries for and identifies the pose of the\\nsuture needle, determines which robot arm is closest to it, and plans a trajectory for that robot arm to reach the suture needle. (2) The\\nclosest robot arm approaches and grasps the suture needle. (3) The suture needle is lifted to a neutral handover position. At this stage, the\\nSUFIA LLM planner detects the pose of the suture needle at the handover position and plans a trajectory for the second robot arm to\\napproach the needle. (4) The second robot arm descends and grasps the needle, then the first robot arm releases the needle after the second\\nrobot arm has grasped it. We provide task videos at orbit-surgical.github.io/sufia\\nFailure ModesExperiment Success Rate Planning Steps\\n(P) (E)\\nSim Experiments\\nNeedle Lift 100 % 6 0 0\\nNeedle Handover 90 % 14 - 16 1 0\\nVessel Dilation 60 % 6 - 8 3 1\\nShunt Insertion 70 % 8 - 9 3 0\\nPhysical Experiments\\nNeedle Lift 100 % 6 0 0\\nNeedle Handover 50 % 14 - 18 2 3\\nTABLE I: Evaluation Success rate and planning steps required\\nfor surgical sub-tasks automation (10 trials for each experiment).\\nFailure modes: (P) denotes planning and (E) denotes execution\\nfailures. Sim experiments are carried out in ORBIT -Surgical, a high-\\nfidelity surgical simulation framework. Physical Experiments are\\nperformed on a dVRK surgical platform.\\ntask, all planning failures were due to not rotating the\\ngrippers to grasp the vessel’s rim correctly. In the Shunt\\nInsertion task, the planning failures were from incorrect\\nlift height calculations before insertion.\\nWe observed that the performance of SUFIA was relatively\\nrobust to the more complex physics and observation spaces\\nof the physical environment, with 0 and 2 planning fail-\\nures encountered during the Needle Lift and Needle\\nHandover experiments, respectively. This aligns closely\\nwith the framework’s performance in simulation. However, we\\nfound that hysteresis and encoder mismatch within the cable-\\ndriven dVRK resulted in variation between the commanded\\nand actual gripper positions. Although SUFIA was often\\nable to recover from the failures induced by this mismatch\\nthrough its re-planning behavior, the lack of explicit servoing\\ncan result in dropping the needle during more complicated\\nhandovers.\\nD. Task Prompt Analysis\\nSimple tasks such as Needle Lift require a simple\\nprompt to function properly. The surgeon can specify a\\nposition to transfer the needle to or allow the LLM to\\ndetermine a specific lift height above the table.\\nMore sophisticated prompts are needed for tasks that re-\\nquire several steps for successful completion. In the Needle\\nHandover task, the surgeon can provide additional notes\\nfor SUFIA to consider (e.g., \"please note that for a handover,\\neach robot should grasp the needle from the side closest to\\nit.\"). The sequence in which the robot arms grasp and hand\\nover to each other, as well as the location of the handover, can\\neither be specified directly or left for the SUFIA to decide\\nbased on the distance to the needle or other environmental\\nstates.\\nThe SUFIA planner may suggest unnecessary steps that\\nmay not be required for task completion and may potentially\\nelongate task execution time. For instance, in the Vessel\\nDilation task, the vanilla prompt for dilating a vessel can\\nsometimes lead to an additional step of \"Lift the vessel slightly\\nby moving the end-effector upwards to provide clearance from\\nthe table.\" The surgeon can provide additional information\\nabout the fact that the clamps are holding the vessel vertically\\nto eliminate the suggestion of lifting steps in dilating the\\nvessel. Similarly, in the Shunt Insertion task, additional\\ninformation such as \"please lift the small tube by a specific\\namount off of the table and horizontally insert it\" helps to\\nachieve better planning and execution.\\nVision language models (VLMs) can also be incorporated\\nin SUFIA to enhance the general visual understanding of the\\nLLM planner. For instance, in the Vessel Dilation task,\\nGPT4-Vision [12] can provide the planner with environmental\\ncontext regarding the orientation of the vessel phantom. In\\nthis example, the VLM response can complement the user\\nprompt: I see a vertical yellow tube on the\\nright side of the image. It appears to\\nbe standing upright on one of its ends\\non a flat surface. While useful for providing\\ngeneral visual context, similar to [ 44], we find GPT4-Vision\\nunreliable as a standalone perception module for detecting\\n(small) objects’ spatial states and omit it for the remainder\\nof our experiments.\\nThe prompts used for the tasks are as follows:\\nNeedle Lift – \"Pick up the needle and lift it.\"'),\n",
       " Document(metadata={'source': 'SuFIA.pdf', 'page': 1}, page_content='task automation approaches.\\nIn this work, we present SUFIA (Surgical First Interactive\\nAutonomy Assistant), a framework for natural interaction\\nbetween a human surgeon and a surgical robot to provide\\ninteractive surgical autonomy. As shown in Fig. 1, SUFIA\\ntakes in sub-task commands from a surgeon and outputs\\na high-level natural language task plan, as well as low-\\nlevel Python code snippets for execution, if requested. A\\nperception module grounds perceived surgical objects in the\\nscene regardless of variations in shape, size, and pose and\\naccounts for the characteristics of their often small, slender\\nshapes. SUFIA also incorporates re-planning and human-in-\\nthe-loop control as safety measures. Our primary contributions\\nare as follows:\\n• A general formulation for natural language interaction\\nbetween a surgeon and a robot.\\n• A language-based control approach to facilitate surgical\\nsub-task implementations.\\n• A systematic evaluation of the generalization of our\\napproach to various surgical sub-tasks, showing its per-\\nformance and robustness for challenging workspace condi-\\ntions.\\nII. R ELATED WORK\\nA. Large Language Models for Robotics\\nLarge Language Models (LLMs) are state-of-the-art nat-\\nural language processing systems built on the transformer\\narchitecture [11]. LLMs are pre-trained with self-supervised\\nobjectives on vast amounts of text corpora, enabling these\\nmodels to exhibit impressive language understanding and\\ngeneration capabilities and perform a wide range of tasks.\\nThey are typically further fine-tuned with labeled data and\\nRLHF to create general-purpose assistants [ 12], [13] or more\\nspecialized models for use cases such as coding [ 14], [15]\\nor report generation [16].\\nIn robotics, LLMs have been recently employed to address\\nthe high-level planning aspect of robotic control [ 17], [18].\\nThese models still require trajectory generators through cost\\nor reward functions to compute the trajectory. Other works\\nfocused on leveraging LLMs to design reward functions [ 19],\\n[20] to acquire complex skills via reinforcement learning.\\nHowever, most of these research works perform well on\\npredefined tasks and still require expensive training time\\nto generalize. Building on a recent work [ 21] that revealed\\nthe potential of LLMs to directly reason trajectory paths for\\nrobot arms, SUFIA incorporates LLMs to directly control\\nthe gripper poses to perform surgical sub-tasks. This enables\\nthe surgeon to naturally interact with the robot by asking\\nfor a complete task (e.g., pick the needle, insert the soft\\ntube) or an open-ended task (e.g., move the needle in semi-\\ncircular motion) to help complete a sub-task. Our work\\ndiffers from [ 21] in that we do not rely on a separate object\\ndetector for validation, incorporate further safety mechanisms\\nby delegation, and show results for surgical scenes, where we\\nadditionally study domain-relevant axes such as variations in\\nneedle shape.\\nB. Surgical Augmented Dexterity\\nAugmented dexterity has been attempted for several sub-\\ntasks with varying levels of autonomy [ 22], [23], [6] such as\\ndexterous needle picking and handling [ 24], [3], suturing [ 2],\\n[25], and tissue manipulation [ 26], [27], [28]. In particular,\\nin contrast to full automation, an often-explored paradigm\\nin surgical robotics is augmented dexterity [ 29], in which\\nminimal surgical sub-tasks are automated under human\\nsupervision, enabling more precise actuation with less effort\\nexpended.\\nHowever, these works largely rely on access to expensive,\\ntask-specific surgical hardware and software. In order to\\nenable wider exploration of robotic automation in surgery,\\nprior work has often focused on reducing the hardware barrier\\nby adapting traditional robotic arm geometries for medical\\nsub-tasks [ 30], [ 31], or designing novel, lower-cost, multi-\\npurpose medical robotic systems [32], [33], [34]. Additionally,\\nlearning robust perception and control models for surgical\\ntasks often requires gathering very large and expensive in-vivo\\ndatasets to avoid safety-critical failure cases [ 35], [ 36]. In\\nthis work, we propose an alternate approach to this software\\nbarrier by relying on a general-purpose, natural language-\\nguided framework for surgical augmented dexterity across\\nmultiple tasks.\\nIII. P ROBLEM FORMULATION\\nWe focus on a novel approach to surgical augmented\\ndexterity. In contrast to previous methods, we are investigating\\nthe potential of a generalist framework using large language\\nmodels to address surgical augmented dexterity rather than\\ntraining individual models for isolated tasks. We now briefly\\ndetail the assumptions with respect to the environment and\\navailable tools in our work. We do not provide any policies,\\ntrajectory optimizers, or in-context examples to the LLM [ 21].\\nInstead, we expect the LLM to reason over automating a\\nbenchmark simulated surgical sub-task with their internal\\nknowledge and access to limited environment information\\nthrough pre-defined function calls available in an API. All\\nof our experiments are carried out on the da Vinci Research\\nKit (dVRK) robot platform [ 33]. The initial position and\\norientation of the dVRK grippers are available from the robot\\ncontroller.\\nWe assume access to a single RGB-D camera with a known\\nintrinsic matrix, allowing for transformation between the\\ncamera’s perspective and the world coordinate space. With\\nthis, we design a perception module for the LLM to interact\\nwith and query object information. This module identifies\\nand retrieves the pose information of objects present in the\\nscene. For this, in simulation, we assume access to an instance\\nsegmentation model that, given an object name, outputs the\\nsegmentation maps of all instances of the queried object. In\\nthe physical experiments, we train a segmentation network\\nbased on the architecture from [ 37] for a needle segmentation\\nmodel.\\nIV. S UFIA\\nWe propose SUFIA , a framework for natural interaction\\nbetween surgeons and robots. SUFIA uses a human-in-the-'),\n",
       " Document(metadata={'source': 'SuFIA.pdf', 'page': 5}, page_content='(N1) (N2) (N3)\\n(N4) (N5)\\nFig. 5: Needle variations in simulation. We consider five instances\\nof simulated suture needles (N1 - N5) with various sizes and shapes\\nto conduct the generalizability experiment in O RBIT -Surgical.\\nNeedle Handover – \"Pick up the needle with the arm\\nclosest to it, move it directly to the handover location between\\nthe two arms, and keep holding the needle. Grasp the right\\nside of the needle with the other robot arm, then right after\\nthat, release the needle from the first robot and stay put.\"\\nVessel Dilation – \"Grasp the vessel from its leftmost\\nside with robot 0 and pull it backward to the left by 5\\nmillimeters while holding on to it to dilate. When grasping\\nthe vessel, grasp it 15 millimeters below the left point.\"\\nShunt Insertion – \"Lift the small shunt from the middle\\nand insert it into the left opening of the large tube. Approach\\nthe large tube from the left. Only lift the tube by 8 millimeters\\nand move horizontally to insert.\"\\nE. Perception Adaptation and Domain Variation\\nWhile we envision domain-specific perception models for\\napplications of our framework in a given surgical environment\\n(e.g., a needle segmentation network in our real-world\\nphysical experimentation), here we investigate whether a\\ngeneral-purpose segmentation model, LangSAM [ 45], can\\nbe utilized by the perception module in simulated surgical\\nenvironments to enhance the generalizability of the SUFIA\\nframework to various object shapes.\\nWe found that due to the slender shape of dVRK\\narms, the use of LangSAM in the perception module\\nsometimes returned both the dVRK arm and the suture\\nneedle when prompted to find the \"needle.\" However,\\ndescriptive adjectives (e.g. \"round\" or \"small white needle\")\\nenable LangSAM to correctly identify and segment the\\nsuture needle well enough for SUFIA to generate the\\nrequired steps to grasp and lift it. GPT-4 was also able to\\nreason over the sizes of the segmented objects and properly\\ndetermine the object of interest; here is an example of the\\nLLM planner: \" there are two objects detected\\nas \"white needle,\" but only one of them\\nhas dimensions that match a needle\\n(Width: 0.011, Length: 0.032, Height:\\n0.002). The second object’s dimensions\\nare too large to be the needle we are\\ninterested in. Therefore, we will focus\\non the first object with the correct\\ndimensions.\"\\nTo study the generalizability of the perception module\\nacross various needle sizes and shapes, we conduct a study\\nwith five different needles, three needles in different sizes (N1\\nPerception Module N1 N2 N3 N4 N5\\nIsaac Sim Camera 5 / 5 4 / 5 5 / 5 5 / 5 4 / 5\\nLangSAM 4 / 5 5 / 5 4 / 5 3 / 5 3 / 5\\nTABLE II: Domain variation evaluation in simulation. We report\\nthe success rates for lifting suture needles with varied sizes and\\nirregular shapes (suture needles N1 - N5) over 5 trial runs with two\\nvariations of the perception module.\\n- N3; Fig. 5) and two irregular shapes (N4 and N5; Fig. 5).\\nTABLE II shows the performance of SUFIA to lift various\\nneedles, which is robust to their shape and size.\\nF . LLMs Investigation\\nHere, we investigate the effect of different LLMs on the\\nperformance of the needle lift task. We use the same prompt\\nand needle locations for all LLMs. As the error handling in\\nSUFIA would, in principle, allow an LLM to keep trying\\nendlessly until it generates code where no exceptions are\\nraised, we limit the number of errors to five before terminating\\nthe program.\\nNone of the open-source LLMs can perform the simple\\ntask of needle picking and have a hard time following\\nthe instructions in the prompt. All models struggle with\\nunderstanding that detect_object() will print its result\\nrather than return it as a variable in a Python script. When\\nfaced with errors, Mixtral [ 46] typically only outputs updated\\ncode snippets when asked to improve a code block rather\\nthan the whole code. CodeLlama [ 15] calls many undefined\\nfunctions, such as get_end_effector_pose(), despite\\nthe end-effector pose being given in the prompt. Llama 2 [ 47]\\nhas a variety of mistakes related to understanding the steps\\nin the task, such as forgetting to close the gripper or moving\\nit down before lifting the needle.\\nGPT3.5 Turbo similarly misunderstands\\ndetect_object(), often assigning its value to a\\nvariable called needle_position, despite the prompt\\nstating the function does not return anything. Beyond that,\\nGPT3.5 Turbo does consistently define a proper plan to\\nlift the needle, but even when it calls detect_object()\\ncorrectly, the information is not incorporated successfully.\\nAll in all, in our experiments, only GPT-4 Turbo could\\nfollow all instructions and appropriately plan and execute the\\nrelatively simple task of lifting a suture needle.\\nG. Re-planning\\nTo illustrate the benefits of our safety modules, we provide\\nan example in the Needle Lift environment in Fig. 7.\\nIn the first row, SUFIA executes the plan it came up with\\nto perform the task desired by the user, i.e., orienting its\\ngripper with the needle, moving to a position where it can\\ngrab it, and picking it up. While picking it up, we move the\\nneedle to a different position. Because SUFIA validates the\\nexpected and observed position of the objects it manipulates,\\nit correctly identifies the needle is not where it should be.\\nBased on the newly observed state, SUFIA devises a new\\nplan to proceed with the user instruction, finally lifting it to\\nthe desired height.')]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response[\"context\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Explanation\n",
    "\n",
    "The final `rag_chain` builds a full Retrieval-Augmented Generation (RAG) chain.\n",
    "\n",
    "**Key Components:**\n",
    "The flow of the chain would be as: \n",
    "1. **Retriever (`db.as_retriever()`):**\n",
    "   - Converts `db` (a vector database) into a retriever object.\n",
    "   - Responsible for retrieving the most relevant documents from the vector store based on the query embedding.\n",
    "\n",
    "2. **Combiner (`combine_docs_chain`):**\n",
    "   - Takes the documents retrieved by the retriever.\n",
    "   - Uses the LLM to generate a final response.\n",
    "\n",
    "**Result:** \n",
    "- `rag_chain` orchestrates the entire process:\n",
    "  - Takes a user query.\n",
    "  - Uses the retriever to fetch relevant documents.\n",
    "  - Passes those documents to the `combine_docs_chain` for response generation.\n",
    "\n",
    "`create_retrieval_chain()` and `combine_docs_chain()` are helper functions which create the chain (RunnableSequence) as specified by their names.\\\n",
    "So, in the above `rag_chain`, the `combine_docs_chain()` takes input from the retriever, inserts that input into its prompt as input variable, and then passes that prompt to the \\\n",
    "specified llm, which then returns a response.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
      "  context: RunnableLambda(format_docs)\n",
      "}), kwargs={}, config={'run_name': 'format_inputs'}, config_factories=[])\n",
      "| ChatPromptTemplate(input_variables=['context', 'input'], optional_variables=['chat_history'], input_types={'chat_history': list[typing.Annotated[typing.Union[typing.Annotated[langchain_core.messages.ai.AIMessage, Tag(tag='ai')], typing.Annotated[langchain_core.messages.human.HumanMessage, Tag(tag='human')], typing.Annotated[langchain_core.messages.chat.ChatMessage, Tag(tag='chat')], typing.Annotated[langchain_core.messages.system.SystemMessage, Tag(tag='system')], typing.Annotated[langchain_core.messages.function.FunctionMessage, Tag(tag='function')], typing.Annotated[langchain_core.messages.tool.ToolMessage, Tag(tag='tool')], typing.Annotated[langchain_core.messages.ai.AIMessageChunk, Tag(tag='AIMessageChunk')], typing.Annotated[langchain_core.messages.human.HumanMessageChunk, Tag(tag='HumanMessageChunk')], typing.Annotated[langchain_core.messages.chat.ChatMessageChunk, Tag(tag='ChatMessageChunk')], typing.Annotated[langchain_core.messages.system.SystemMessageChunk, Tag(tag='SystemMessageChunk')], typing.Annotated[langchain_core.messages.function.FunctionMessageChunk, Tag(tag='FunctionMessageChunk')], typing.Annotated[langchain_core.messages.tool.ToolMessageChunk, Tag(tag='ToolMessageChunk')]], FieldInfo(annotation=NoneType, required=True, discriminator=Discriminator(discriminator=<function _get_type at 0x000001B3D3D09080>, custom_error_type=None, custom_error_message=None, custom_error_context=None))]]}, partial_variables={'chat_history': []}, metadata={'lc_hub_owner': 'langchain-ai', 'lc_hub_repo': 'retrieval-qa-chat', 'lc_hub_commit_hash': 'b60afb6297176b022244feb83066e10ecadcda7b90423654c4a9d45e7a73cebc'}, messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context'], input_types={}, partial_variables={}, template='Answer any use questions based solely on the context below:\\n\\n<context>\\n{context}\\n</context>'), additional_kwargs={}), MessagesPlaceholder(variable_name='chat_history', optional=True), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], input_types={}, partial_variables={}, template='{input}'), additional_kwargs={})])\n",
      "| ChatOllama(model='llama3.2')\n",
      "| StrOutputParser() kwargs={} config={'run_name': 'stuff_documents_chain'} config_factories=[]\n"
     ]
    }
   ],
   "source": [
    "print(combine_docs_chain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bound=RunnableAssign(mapper={\n",
      "  context: RunnableBinding(bound=RunnableLambda(lambda x: x['input'])\n",
      "           | VectorStoreRetriever(tags=['DocArrayInMemorySearch'], vectorstore=<langchain_community.vectorstores.docarray.in_memory.DocArrayInMemorySearch object at 0x000001B3F51FE420>, search_kwargs={}), kwargs={}, config={'run_name': 'retrieve_documents'}, config_factories=[])\n",
      "})\n",
      "| RunnableAssign(mapper={\n",
      "    answer: RunnableBinding(bound=RunnableBinding(bound=RunnableAssign(mapper={\n",
      "              context: RunnableLambda(format_docs)\n",
      "            }), kwargs={}, config={'run_name': 'format_inputs'}, config_factories=[])\n",
      "            | ChatPromptTemplate(input_variables=['context', 'input'], optional_variables=['chat_history'], input_types={'chat_history': list[typing.Annotated[typing.Union[typing.Annotated[langchain_core.messages.ai.AIMessage, Tag(tag='ai')], typing.Annotated[langchain_core.messages.human.HumanMessage, Tag(tag='human')], typing.Annotated[langchain_core.messages.chat.ChatMessage, Tag(tag='chat')], typing.Annotated[langchain_core.messages.system.SystemMessage, Tag(tag='system')], typing.Annotated[langchain_core.messages.function.FunctionMessage, Tag(tag='function')], typing.Annotated[langchain_core.messages.tool.ToolMessage, Tag(tag='tool')], typing.Annotated[langchain_core.messages.ai.AIMessageChunk, Tag(tag='AIMessageChunk')], typing.Annotated[langchain_core.messages.human.HumanMessageChunk, Tag(tag='HumanMessageChunk')], typing.Annotated[langchain_core.messages.chat.ChatMessageChunk, Tag(tag='ChatMessageChunk')], typing.Annotated[langchain_core.messages.system.SystemMessageChunk, Tag(tag='SystemMessageChunk')], typing.Annotated[langchain_core.messages.function.FunctionMessageChunk, Tag(tag='FunctionMessageChunk')], typing.Annotated[langchain_core.messages.tool.ToolMessageChunk, Tag(tag='ToolMessageChunk')]], FieldInfo(annotation=NoneType, required=True, discriminator=Discriminator(discriminator=<function _get_type at 0x000001B3D3D09080>, custom_error_type=None, custom_error_message=None, custom_error_context=None))]]}, partial_variables={'chat_history': []}, metadata={'lc_hub_owner': 'langchain-ai', 'lc_hub_repo': 'retrieval-qa-chat', 'lc_hub_commit_hash': 'b60afb6297176b022244feb83066e10ecadcda7b90423654c4a9d45e7a73cebc'}, messages=[SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=['context'], input_types={}, partial_variables={}, template='Answer any use questions based solely on the context below:\\n\\n<context>\\n{context}\\n</context>'), additional_kwargs={}), MessagesPlaceholder(variable_name='chat_history', optional=True), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=['input'], input_types={}, partial_variables={}, template='{input}'), additional_kwargs={})])\n",
      "            | ChatOllama(model='llama3.2')\n",
      "            | StrOutputParser(), kwargs={}, config={'run_name': 'stuff_documents_chain'}, config_factories=[])\n",
      "  }) kwargs={} config={'run_name': 'retrieval_chain'} config_factories=[]\n"
     ]
    }
   ],
   "source": [
    "print(rag_chain)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain_course_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}