Lesson-5 Evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lesson-5: Evaluating the LLM\n",
    "Evaluating the responses using a LLM itself, which is called LLM-assisted evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Create the QnA chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Loads the Langsmith API key from the .env file, used for accessing the hub\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.output_parsers import StrOutputParser\n",
    "from langchain_core.runnables import RunnablePassthrough\n",
    "from langchain_ollama import ChatOllama, OllamaLLM, OllamaEmbeddings\n",
    "from langchain.document_loaders import PyPDFLoader\n",
    "from langchain.vectorstores import DocArrayInMemorySearch\n",
    "from langchain.chains import create_retrieval_chain\n",
    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
    "from langchain import hub"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\milap\\envs\\langchain_course_env\\Lib\\site-packages\\pydantic\\_migration.py:283: UserWarning: `pydantic.error_wrappers:ValidationError` has been moved to `pydantic:ValidationError`.\n",
      "  warnings.warn(f'`{import_path}` has been moved to `{new_location}`.')\n"
     ]
    }
   ],
   "source": [
    "loader = PyPDFLoader(\n",
    "    file_path=\"SuFIA.pdf\",\n",
    "    extract_images=True,\n",
    ")\n",
    "pages = loader.load()\n",
    "\n",
    "embeddings = OllamaEmbeddings(model=\"llama3.2\")\n",
    "\n",
    "db = DocArrayInMemorySearch.from_documents(pages, embeddings)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\milap\\envs\\langchain_course_env\\Lib\\site-packages\\langsmith\\client.py:256: LangSmithMissingAPIKeyWarning: API key must be provided when using hosted LangSmith API\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "llm = ChatOllama(model=\"llama3.2\")\n",
    "\n",
    "# See full prompt at https://smith.langchain.com/hub/rlm/rag-prompt\n",
    "prompt = hub.pull(\"rlm/rag-prompt\")\n",
    "\n",
    "def format_docs(docs):\n",
    "    return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
    "\n",
    "qa_chain = (\n",
    "    {\n",
    "        \"context\": db.as_retriever() | format_docs,\n",
    "        \"question\": RunnablePassthrough(),\n",
    "    }\n",
    "    | prompt\n",
    "    | llm\n",
    "    | StrOutputParser()\n",
    ")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. LLM Assisted Evalution "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate examples using the LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\milap\\envs\\langchain_course_env\\Lib\\site-packages\\langchain\\chains\\llm.py:369: UserWarning: The apply_and_parse method is deprecated, instead pass an output parser directly to LLMChain.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "from langchain.evaluation.qa import QAGenerateChain\n",
    "\n",
    "example_generator_chain = QAGenerateChain.from_llm(ChatOllama(model=\"llama3.2\"))\n",
    "\n",
    "qa_pairs = example_generator_chain.apply_and_parse(\n",
    "    input_list=[{\"doc\": doc.page_content} for doc in pages[0:2]] #test for first two pages\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'qa_pairs': {'query': 'What is the primary limitation of learning-based approaches to robotic surgical assistants, as mentioned in the introduction?', 'answer': 'The primary limitation of learning-based approaches, such as reinforcement and imitation learning, is that complex, long-horizon surgical sub-tasks are often computationally expensive, require extensive domain knowledge and reward engineering, and involve time-consuming dataset curation, which limits their generalizability in safety-critical applications.'}}, {'qa_pairs': {'query': 'What is the primary contribution of SUFIA, a framework for natural interaction between a human surgeon and a surgical robot?', 'answer': 'The primary contributions of SUFIA are (1) a general formulation for natural language interaction between a surgeon and a robot, (2) a language-based control approach to facilitate surgical sub-task implementations, and (3) a systematic evaluation of the generalization of its approach to various surgical sub-tasks.'}}]\n",
      "2\n"
     ]
    }
   ],
   "source": [
    "print(qa_pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_examples = [pair[\"qa_pairs\"] for pair in qa_pairs]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will make predictions of these queries by invoking the `qa_chain`, then use the LLM to evaluate those predictions using the generated answers in the `qa_pairs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.evaluation.qa import QAEvalChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_queries = [qa_pair[\"query\"] for qa_pair in qa_examples]\n",
    "predictions = qa_chain.batch(\n",
    "    inputs=input_queries\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['The text describes a framework called SUFIA (Surgeon-in-the-Loop Framework for Augmented Dexterity in Robotics) that enables natural language-guided augmented dexterity for robotic surgical assistants. The framework combines the strengths of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution.\\n\\nSUFIA receives commands from a surgeon in natural language, converts them into high-level planning and low-level control code, and queries a perception module for object state information when necessary. The framework can assist a surgeon with open-ended tasks, such as moving the robot in a desired motion to help complete a surgical task.\\n\\nIn times of insufficient information, SUFIA delegates full control back to the surgeon. This approach enables natural human-robot coordination and has the potential to develop general-purpose models for autonomous surgery beyond the capability of current task-by-task learning methods.\\n\\nThe framework is evaluated on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions.', \"The answer is that none of the open-source LLMs (Large Language Models) were able to successfully perform the task of lifting a suture needle, except for GPT-4 Turbo. The other models struggled with understanding the instructions and incorporating the information correctly. However, even GPT-3.5 Turbo was unable to follow all instructions and execute the plan effectively.\\n\\nDespite the challenges, the SUFIA framework's safety modules were able to re-plan and correct the task, ensuring that the needle was lifted to the desired height. The framework's validation of expected and observed positions helped it adapt to the changing situation and complete the task successfully.\"]\n"
     ]
    }
   ],
   "source": [
    "print(predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "qa_predict=[{\"query\": query, \"answer\": answer} for query, answer in zip(input_queries, predictions)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "eval_chain = QAEvalChain.from_llm(ChatOllama(model=\"llama3.2\"))\n",
    "graded_outputs = eval_chain.evaluate(\n",
    "    examples=qa_examples,\n",
    "    predictions=qa_predict,\n",
    "    prediction_key=\"answer\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'results': \"INCORRECT\\n\\nThe student's answer provides a detailed description of the SUFIA framework, its capabilities, and its evaluation, but it does not address the primary limitation of learning-based approaches as mentioned in the introduction. The true answer specifically highlights that complex surgical tasks are computationally expensive, require domain knowledge and reward engineering, and involve dataset curation issues, which limits their generalizability.\"}, {'results': 'INCORRECT'}]\n"
     ]
    }
   ],
   "source": [
    "print(graded_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example 0:\n",
      "Question: What is the primary limitation of learning-based approaches to surgical robotic platforms, according to the document?\n",
      "Real Answer: \n",
      "The primary limitation of learning-based approaches, such as reinforcement and imitation learning, is that complex, long-horizon surgical sub-tasks are often computationally expensive, require extensive domain knowledge and reward engineering, and involve time-consuming dataset curation, which limits their generalizability in safety-critical applications.\n",
      "Predicted Answer:\n",
      "The text describes a framework called SUFIA (Surgeon-in-the-Loop Framework for Augmented Dexterity in Robotics) that enables natural language-guided augmented dexterity for robotic surgical assistants. The framework combines the strengths of large language models (LLMs) with perception modules to implement high-level planning and low-level control of a robot for surgical sub-task execution.\n",
      "\n",
      "SUFIA receives commands from a surgeon in natural language, converts them into high-level planning and low-level control code, and queries a perception module for object state information when necessary. The framework can assist a surgeon with open-ended tasks, such as moving the robot in a desired motion to help complete a surgical task.\n",
      "\n",
      "In times of insufficient information, SUFIA delegates full control back to the surgeon. This approach enables natural human-robot coordination and has the potential to develop general-purpose models for autonomous surgery beyond the capability of current task-by-task learning methods.\n",
      "\n",
      "The framework is evaluated on four surgical sub-tasks in a simulation environment and two sub-tasks on a physical surgical robotic platform in the lab, demonstrating its ability to perform common surgical sub-tasks through supervised autonomous operation under challenging physical and workspace conditions.\n",
      "\n",
      "Predicted Grade: INCORRECT\n",
      "\n",
      "The student's answer provides a detailed description of the SUFIA framework, its capabilities, and its evaluation, but it does not address the primary limitation of learning-based approaches as mentioned in the introduction. The true answer specifically highlights that complex surgical tasks are computationally expensive, require domain knowledge and reward engineering, and involve dataset curation issues, which limits their generalizability.\n",
      "\n",
      "Example 1:\n",
      "Question: What is the primary contribution of SUFIA, a framework for natural interaction between a human surgeon and a surgical robot?\n",
      "Real Answer: \n",
      "The primary contributions of SUFIA are (1) a general formulation for natural language interaction between a surgeon and a robot, (2) a language-based control approach to facilitate surgical sub-task implementations, and (3) a systematic evaluation of the generalization of its approach to various surgical sub-tasks.\n",
      "Predicted Answer:\n",
      "The answer is that none of the open-source LLMs (Large Language Models) were able to successfully perform the task of lifting a suture needle, except for GPT-4 Turbo. The other models struggled with understanding the instructions and incorporating the information correctly. However, even GPT-3.5 Turbo was unable to follow all instructions and execute the plan effectively.\n",
      "\n",
      "Despite the challenges, the SUFIA framework's safety modules were able to re-plan and correct the task, ensuring that the needle was lifted to the desired height. The framework's validation of expected and observed positions helped it adapt to the changing situation and complete the task successfully.\n",
      "\n",
      "Predicted Grade: INCORRECT\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i, eg in enumerate(qa_examples):\n",
    "    print(f\"Example {i}:\")\n",
    "    print(\"Question: \" + qa_predict[i]['query'])\n",
    "    print(\"Real Answer: \\n\" + qa_examples[i]['answer'])\n",
    "    print(\"Predicted Answer:\\n\" + qa_predict[i]['answer'])\n",
    "    print(\"\\nPredicted Grade: \" + graded_outputs[i]['results'])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE**: A reason why both queries are graded incorrect, and we can see that they're actually incorrect, is because, when predicting answer for these queries, the `qa_chain` has context of entire document, while the queries from the `example_generator_chain`, had context of two pages only.\\\n",
    "Another reason could be the inherently performance issue related to Llama's architecture."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "langchain_course_env",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}