dspy_breakdown.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "afbec709-c5b0-4c68-a4db-7a0f5c0ae228",
   "metadata": {},
   "source": [
    "# Programming (Not Prompting) Your LLM with DSPy\n",
    "\n",
    "DSPy (Declarative Self-improving Python) is a framework from Stanford NLP that treats language models as programmable functions rather than prompt templates. It provides a PyTorch-like interface for defining, composing, and optimizing LLM operations. Instead of writing and maintaining complex prompts, developers specify input/output signatures and let DSPy handle prompt engineering and optimization. The framework enables systematic improvement of LLM pipelines through techniques like automatic prompt tuning and self-improvement.\n",
    "\n",
    "<img src=\"./media/dspy_workflow.png\" width=500>\n",
    "\n",
    "The DSPy workflow follows 4 main steps:\n",
    "1. Define your program using signatures and modules\n",
    "2. Create measurable success metrics that clearly show your program's performance\n",
    "3. Compile your program and optimize towards success metrics\n",
    "4. Collect additional data and iterate\n",
    "\n",
    "We'll look through and apply all the various approaches DSPy offers across these steps in this notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5737a40f-6a4d-4035-80b1-61604cc9a6b0",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup\n",
    "\n",
    "<img src=\"./media/dspy.png\" width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "fdad4261-30d2-491c-a42d-ed3c522a9a44",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dspy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "504e5e46-4f4c-4d11-b9ee-19a380cb36d1",
   "metadata": {},
   "source": [
    "Configure LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef325de5-72b8-44c9-92e4-7f5ae602aec1",
   "metadata": {},
   "source": [
    "**Configure LLM**\n",
    "\n",
    "DSPy by default caches responses and models across your environment. Unless explicitly stated otherwise, configuring a language model will use that language model for all subsequent calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "83eb594d-b7a6-4c73-883f-a2ccf671b5c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "lm = dspy.LM('openai/gpt-4o-mini')\n",
    "dspy.configure(lm=lm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "44e58448-f7f3-4857-a5af-697bc6787bc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['This is a test! How can I assist you further?']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm(messages=[{\"role\": \"user\", \"content\": \"Say this is a test!\"}])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e580490e-d9c9-4c49-8a8a-1b366e66b9b4",
   "metadata": {},
   "source": [
    "---\n",
    "## Signatures\n",
    "\n",
    "DSPy Signatures follow the same approach as regular function signatures but are defined in natural language. This is the core of the \"prompting\" that DSPy aims to replace. Instead of telling the LLM what to do, we take the approach of declaring what the LLM will do.\n",
    "\n",
    "The format looks like:\n",
    "\n",
    "```python \n",
    "'input -> output' \n",
    "```\n",
    "\n",
    "Where your `input` and `output` can be anything you'd like. It's also possible to define multiple inputs, outputs, types, or more well defined schemas.\n",
    "\n",
    "<img src=\"./media/signatures.png\" width=600>\n",
    "\n",
    "Behind the scenes, this is still a language model prompt, but it aims to be more modular than static, changing wording and structure based on your natural language signature. While this may seem counter intuitive as we're abstracting away from prompting, DSPy has set this up in a way that allows for easy switching in and out of models, and algorithmic optimizations that we will highlight later."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fa1e67a-77b9-4d48-b405-e21f45d0b5d3",
   "metadata": {},
   "source": [
    "### Simple Input & Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "5245d195-9a30-4017-bb17-d2000ff97be9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Response:  The sky appears blue due to a phenomenon called Rayleigh scattering. When sunlight enters the Earth's atmosphere, it collides with molecules and small particles in the air. Sunlight is made up of different colors, each with varying wavelengths. Blue light has a shorter wavelength and is scattered in all directions more than other colors with longer wavelengths, such as red or yellow. This scattering causes the sky to look predominantly blue to our eyes during the day.\n"
     ]
    }
   ],
   "source": [
    "qna = dspy.Predict('question -> answer')\n",
    "\n",
    "response = qna(question=\"Why is the sky blue?\")\n",
    "\n",
    "print(\"Response: \", response.answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "id": "54e1c151-4eba-4ba6-82f6-1c1982561639",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary:  The market for our products is highly competitive, driven by rapid technological advancements and changing industry standards. Key competitive factors include product performance, range of offerings, customer access, distribution channels, software support, adherence to industry standards, manufacturing capabilities, pricing, and overall system costs. Our competitiveness hinges on our ability to predict customer demands and deliver quality products at competitive prices. We anticipate increased competition from both established players and new entrants, potentially offering lower prices or superior features. Additionally, competition may arise from companies specializing in GPUs, CPUs, DPUs, and high-performance interconnect products. Some competitors may possess greater resources, making it challenging for us to keep pace with market changes. The competitive landscape is expected to intensify in the future.\n"
     ]
    }
   ],
   "source": [
    "sum = dspy.Predict('document -> summary')\n",
    "\n",
    "document = \"\"\"\n",
    "The market for our products is intensely competitive and is characterized by rapid technological change and evolving industry standards. \n",
    "We believe that theprincipal competitive factors in this market are performance, breadth of product offerings, access to customers and partners and distribution channels, softwaresupport, conformity to industry standard APIs, manufacturing capabilities, processor pricing, and total system costs. \n",
    "We believe that our ability to remain competitive will depend on how well we are able to anticipate the features and functions that customers and partners will demand and whether we are able todeliver consistent volumes of our products at acceptable levels of quality and at competitive prices. \n",
    "We expect competition to increase from both existing competitors and new market entrants with products that may be lower priced than ours or may provide better performance or additional features not provided by our products. \n",
    "In addition, it is possible that new competitors or alliances among competitors could emerge and acquire significant market share.\n",
    "A significant source of competition comes from companies that provide or intend to provide GPUs, CPUs, DPUs, embedded SoCs, and other accelerated, AI computing processor products, and providers of semiconductor-based high-performance interconnect products based on InfiniBand, Ethernet, Fibre Channel,and proprietary technologies. \n",
    "Some of our competitors may have greater marketing, financial, distribution and manufacturing resources than we do and may bemore able to adapt to customers or technological changes. \n",
    "We expect an increasingly competitive environment in the future.\n",
    "\"\"\"\n",
    "\n",
    "response = sum(document=document)\n",
    "\n",
    "print(\"Summary: \", response.summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9b6d5b4-9b8e-415c-b8aa-e5304e99fd6c",
   "metadata": {},
   "source": [
    "### Multiple Inputs and Outputs\n",
    "\n",
    "<img src=\"./media/multiple_signature.png\" width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "268ea25f-3f78-4b3a-82ed-ea7b67a3f800",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer:  Your name is Adam Lucek.\n",
      "\n",
      "Citation:  Context provided by the user.\n"
     ]
    }
   ],
   "source": [
    "multi = dspy.Predict('question, context -> answer, citation')\n",
    "\n",
    "question = \"What's my name?\"\n",
    "context = \"The user you're talking to is Adam Lucek, AI youtuber extraordinaire\"\n",
    "\n",
    "response = multi(question=question, context=context)\n",
    "\n",
    "print(\"Answer: \", response.answer)\n",
    "print(\"\\nCitation: \", response.citation)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "972af06a-bca0-4e00-97c5-935b3c26d21e",
   "metadata": {},
   "source": [
    "### Type Hints with Outputs\n",
    "\n",
    "<img src=\"./media/input_type.png\" width=400>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "eb8da6a7-2dc2-4703-86a9-922066b7247f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment Classification:  negative\n",
      "\n",
      "Confidence:  0.85\n",
      "\n",
      "Reasoning:  The phrase \"I didn't really like it\" clearly indicates a negative sentiment towards whatever is being discussed. The use of \"didn't like\" suggests dissatisfaction, and the uncertainty expressed by \"I don't quite know\" reinforces a lack of positive feelings. The confidence level is high at 0.85 due to the explicit negative language used.\n"
     ]
    }
   ],
   "source": [
    "emotion = dspy.Predict('input -> sentiment: str, confidence: float, reasoning: str')\n",
    "\n",
    "text = \"I don't quite know, I didn't really like it\"\n",
    "\n",
    "response = emotion(input=text)\n",
    "\n",
    "print(\"Sentiment Classification: \", response.sentiment)\n",
    "print(\"\\nConfidence: \", response.confidence)\n",
    "print(\"\\nReasoning: \", response.reasoning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8ef8c6-f2bb-4d8f-8ca0-0c5381304a7f",
   "metadata": {},
   "source": [
    "### Class Based Signatures\n",
    "\n",
    "For more advanced signatures, DSPy allows you to define a pydantic class or data structure schema instead of the simple inline string approach. These classes inherit from `dspy.Signature` to start, but you must define your inputs with `dspy.InputField()` and outputs with `dspy.OutputField()`.\n",
    "\n",
    "An optional `desc` argument can be passed within each field to add additional context as a description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "d9b8c1ba-09f3-4f1b-8c6c-496e7804df93",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Transformed Text:  In a quaint coffee shop, where dreams brew and swirl,  \n",
      "The finest lattes dance, a creamy, frothy whirl.  \n",
      "A new barista, skilled, with hands that weave delight,  \n",
      "Crafts magic with the espresso, morning's purest light.\n",
      "\n",
      "Style Metrics:  {'formality': 0.7, 'complexity': 0.6, 'emotiveness': 0.8}\n",
      "\n",
      "Preserverd Keywords:  ['coffee shop', 'lattes', 'barista', 'espresso machine']\n"
     ]
    }
   ],
   "source": [
    "from typing import Literal\n",
    "\n",
    "class TextStyleTransfer(dspy.Signature):\n",
    "    \"\"\"Transfer text between different writing styles while preserving content.\"\"\"\n",
    "    text: str = dspy.InputField()\n",
    "    source_style: Literal[\"academic\", \"casual\", \"business\", \"poetic\"] = dspy.InputField()\n",
    "    target_style: Literal[\"academic\", \"casual\", \"business\", \"poetic\"] = dspy.InputField()\n",
    "    preserved_keywords: list[str] = dspy.OutputField()\n",
    "    transformed_text: str = dspy.OutputField()\n",
    "    style_metrics: dict[str, float] = dspy.OutputField(desc=\"Scores for formality, complexity, emotiveness\")\n",
    "\n",
    "\n",
    "text = \"This coffee shop makes the best lattes ever! Their new barista really knows what he's doing with the espresso machine.\"\n",
    "\n",
    "style_transfer = dspy.Predict(TextStyleTransfer)\n",
    "\n",
    "response = style_transfer(\n",
    "    text=text,\n",
    "    source_style=\"casual\",\n",
    "    target_style=\"poetic\"\n",
    ")\n",
    "\n",
    "print(\"Transformed Text: \", response.transformed_text)\n",
    "print(\"\\nStyle Metrics: \", response.style_metrics)\n",
    "print(\"\\nPreserverd Keywords: \", response.preserved_keywords)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5e87a8c-27cb-4d81-ad87-ebe2f901569f",
   "metadata": {},
   "source": [
    "---\n",
    "## Modules\n",
    "\n",
    "<img src=\"./media/modules.png\" width=1000>\n",
    "\n",
    "Modules are where we apply different prompting frameworks to signatures. We've already been using the basic `Predict` module in our signature examples prior, but there exist many more popular strategies and variants. Here are the current available modules: \n",
    "\n",
    "* `ChainOfThought`: Implements chain-of-thought prompting by prepending a reasoning step before generating outputs. The module automatically adds a \"Let's think step by step\" prefix to encourage structured thinking. Use this when you need the model to break down complex problems into smaller steps.\n",
    "\n",
    "* `ProgramOfThought`: Generates executable Python code to solve problems, with built-in error handling and code regeneration capabilities. Use this for mathematical or algorithmic problems that are better solved through actual code execution.\n",
    "\n",
    "* `ReAct`: Implements Reasoning + Acting by interleaving thoughts, actions (via tools), and observations in a structured loop. Use this when your task requires multi-step reasoning and interaction with external tools or APIs.\n",
    "\n",
    "And a few helpers:\n",
    "\n",
    "* `MultiChainComparison`: Takes multiple reasoning attempts (default 3) and combines them into a single, more accurate response by comparing different reasoning paths. Use this when you need higher accuracy and can afford multiple attempts at solving a problem.\n",
    "\n",
    "* `majority`: A utility function that takes multiple completions and returns the most common response after normalizing the text. Use this when you want to implement simple voting among multiple completion attempts to increase reliability.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10e865e5-3600-4091-8c5b-05b9ec883754",
   "metadata": {},
   "source": [
    "### [Chain of Thought](https://github.com/stanfordnlp/dspy/blob/main/dspy/predict/chain_of_thought.py)\n",
    "\n",
    "<img src=\"./media/cot_module.png\" width=300>\n",
    "\n",
    "ChainOfThought works by modifying the prompt signature to include an explicit reasoning step before the output. When initialized with a signature, it creates an extended signature by prepending a \"reasoning\" field with the prefix \"Reasoning: Let's think step by step in order to\". This reasoning field forces the language model to write out its thought process before providing the final answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "aee04b2f-1e45-479b-909a-c24b7133a71e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment:  Mixed\n",
      "\n",
      "Reasoning:  The statement expresses a conflicting sentiment. The word \"phenomenal\" indicates a strong positive reaction, suggesting that the experience was impressive or outstanding. However, the phrase \"but I hated it\" introduces a negative sentiment, indicating a strong dislike or aversion to the same experience. This juxtaposition creates a complex emotional response, where the speaker acknowledges something as remarkable while simultaneously expressing a strong negative feeling towards it.\n"
     ]
    }
   ],
   "source": [
    "# Define the Signature and Module\n",
    "cot_emotion = dspy.ChainOfThought('input -> sentiment: str')\n",
    "\n",
    "# Example\n",
    "text = \"That was phenomenal, but I hated it!\"\n",
    "\n",
    "# Run\n",
    "cot_response = cot_emotion(input=text)\n",
    "\n",
    "# Output\n",
    "print(\"Sentiment: \", cot_response.sentiment)\n",
    "# Inherently added reasoning\n",
    "print(\"\\nReasoning: \", cot_response.reasoning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2372287-82c1-4a36-b933-cdae87034ed9",
   "metadata": {},
   "source": [
    "### [Program of Thought](https://github.com/stanfordnlp/dspy/blob/main/dspy/predict/program_of_thought.py)\n",
    "\n",
    "<img src=\"./media/program_of_thought.png\" width=700>\n",
    "\n",
    "ProgramOfThought solves tasks by generating executable Python code rather than working directly with natural language outputs. When given a task, PoT first generates Python code using a ChainOfThought predictor, then executes that code in an isolated Python interpreter. If the code generates any errors, PoT enters a refinement loop where it shows the error to the language model, gets corrected code, and tries executing again, for up to a maximum number of iterations (default 3). The final output comes from actually running the successful code rather than from the language model directly. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "96a677a8-c6b6-4efa-97ef-f8f295b11b81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Error in code execution\n"
     ]
    }
   ],
   "source": [
    "# Define the Signature\n",
    "class MathAnalysis(dspy.Signature):\n",
    "    \"\"\"Analyze a dataset and compute various statistical metrics.\"\"\"\n",
    "    \n",
    "    numbers: list[float] = dspy.InputField(desc=\"List of numerical values to analyze\")\n",
    "    required_metrics: list[str] = dspy.InputField(desc=\"List of metrics to calculate (e.g. ['mean', 'variance', 'quartiles'])\")\n",
    "    analysis_results: dict[str, float] = dspy.OutputField(desc=\"Dictionary containing the calculated metrics\")\n",
    "\n",
    "# Create the module\n",
    "math_analyzer = dspy.ProgramOfThought(MathAnalysis)\n",
    "\n",
    "# Example\n",
    "data = [1.5, 2.8, 3.2, 4.7, 5.1, 2.3, 3.9]\n",
    "metrics = ['mean', 'median']\n",
    "\n",
    "# Run\n",
    "pot_response = math_analyzer(\n",
    "    numbers=data,\n",
    "    required_metrics=metrics\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "559da509-f4e0-42ee-931b-d9c1da78c277",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reasoning:  The provided code correctly calculates the mean and median of the given list of numbers. The mean is computed by summing all the numbers and dividing by the count of numbers, while the median is determined by sorting the list and finding the middle value (or the average of the two middle values if the count is even). The output matches the expected results for both metrics.\n",
      "\n",
      "Results:  {'mean': 3.357142857142857, 'median': 3.2}\n"
     ]
    }
   ],
   "source": [
    "print(\"Reasoning: \", pot_response.reasoning)\n",
    "print(\"\\nResults: \", pot_response.analysis_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ebcd1149-5e3e-422a-a1f3-f73d0d3178f3",
   "metadata": {},
   "source": [
    "### [Reasoning + Acting (ReAct)](https://github.com/stanfordnlp/dspy/blob/main/dspy/predict/react.py)\n",
    "\n",
    "<img src=\"./media/react.png\" width=700>\n",
    "\n",
    "ReAct enables interactive problem-solving by combining reasoning with tool usage. It works by maintaining a trajectory of thought-action pairs, where at each step the model explains its reasoning, selects a tool to use, provides arguments for that tool, and then observes the tool's output to inform its next step. Each iteration consists of four parts: a thought explaining the strategy, selection of a tool name from the available tools, arguments to pass to that tool, and the observation from running the tool. This continues until either the model chooses to \"finish\" or reaches the maximum number of iterations. Here's a simple example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "7fbf0317-40ee-4664-8bb3-27bcb0961599",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer:  The Baltimore Orioles won the World Series in 1983, and England won the World Cup in 1966.\n",
      "\n",
      "Reasoning:  The Baltimore Orioles won the 1983 World Series, defeating the Philadelphia Phillies four games to one. Additionally, England won the 1966 FIFA World Cup, beating West Germany 4–2 in the final match.\n"
     ]
    }
   ],
   "source": [
    "# Define a Tool\n",
    "def wikipedia_search(query: str) -> list[str]:\n",
    "    \"\"\"Retrieves abstracts from Wikipedia.\"\"\"\n",
    "    # Existing Wikipedia Abstracts Server\n",
    "    results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) \n",
    "    return [x['text'] for x in results]\n",
    "\n",
    "# Define ReAct Module\n",
    "react_module = dspy.ReAct('question -> response', tools=[wikipedia_search])\n",
    "\n",
    "# Example\n",
    "text = \"Who won the world series in 1983 and who won the world cup in 1966?\"\n",
    "\n",
    "# Run\n",
    "react_response = react_module(question=text)\n",
    "\n",
    "print(\"Answer: \", react_response.response)\n",
    "print(\"\\nReasoning: \", react_response.reasoning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a8f2ed-8fb6-4fac-a476-0c8056dc0a13",
   "metadata": {},
   "source": [
    "### [Multi Chain Comparison](https://github.com/stanfordnlp/dspy/blob/main/dspy/predict/multi_chain_comparison.py)\n",
    "\n",
    "<img src=\"./media/multi_chain.png\" width=700>\n",
    "\n",
    "MultiChainComparison is a meta-predictor that synthesizes multiple existing completions into a single, more robust prediction. It doesn't generate predictions itself, but instead takes M different completions (default 3) from other predictors - these could be from the same predictor with different temperatures, different predictors entirely, or repeated calls with the same settings. These completions are formatted as \"Student Attempt #1:\", \"Student Attempt #2:\", etc., with each attempt packaged as «I'm trying to \\[rationale] I'm not sure but my prediction is \\[answer]». The module then prompts the model to analyze these attempts holistically with \"Accurate Reasoning: Thank you everyone. Let's now holistically...\" to synthesize a final answer. This approach helps mitigate individual prediction errors by having the model explicitly compare and critique multiple solution paths before making its final decision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "5d346cb9-e837-472f-be07-974645808183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment: Positive\n",
      "\n",
      "Reasoning: The phrase \"That was phenomenal!\" clearly indicates strong positive feelings. The word \"phenomenal\" is a superlative that suggests something is remarkable or exceptional, reinforcing the positive sentiment expressed by the speaker. All reasoning attempts correctly identify this sentiment as positive.\n",
      "\n",
      "Completion 1:  Prediction(\n",
      "    reasoning='The phrase \"That was phenomenal!\" expresses strong positive feelings about an experience or event. The use of the word \"phenomenal\" indicates that the subject exceeded expectations and was highly impressive.',\n",
      "    sentiment='Positive'\n",
      ")\n",
      "\n",
      "Completion 2:  Prediction(\n",
      "    reasoning='The phrase \"That was phenomenal!\" expresses a strong positive reaction. The use of the word \"phenomenal\" indicates that the speaker is extremely impressed or pleased with something. This suggests a high level of enthusiasm and admiration.',\n",
      "    sentiment='Positive'\n",
      ")\n",
      "\n",
      "Completion 3:  Prediction(\n",
      "    reasoning='The phrase \"That was phenomenal!\" expresses a strong positive reaction to an experience or event. The use of the word \"phenomenal\" conveys excitement and high praise, indicating that the speaker found something to be extraordinary or outstanding.',\n",
      "    sentiment='positive'\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "# Run CoT completions with increasing temperatures\n",
    "text = \"That was phenomenal!\"\n",
    "\n",
    "cot_completions = []\n",
    "for i in range(3):\n",
    "    # Temperature increases: 0.7, 0.8, 0.9\n",
    "    temp_config = dict(temperature=0.7 + (0.1 * i))\n",
    "    completion = cot_emotion(input=text, config=temp_config)\n",
    "    cot_completions.append(completion)\n",
    "\n",
    "# Synthesize with MultiChainComparison\n",
    "mcot_emotion = dspy.MultiChainComparison('input -> sentiment', M=3)\n",
    "final_result = mcot_emotion(completions=cot_completions, input=text)\n",
    "\n",
    "print(f\"Sentiment: {final_result.sentiment}\")\n",
    "print(f\"\\nReasoning: {final_result.rationale}\")\n",
    "\n",
    "for i in range(3):\n",
    "    print(f\"\\nCompletion {i+1}: \", cot_completions[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10dce03c-432f-4541-bf29-27a8a6798b5d",
   "metadata": {},
   "source": [
    "### [Majority](https://github.com/stanfordnlp/dspy/blob/main/dspy/predict/aggregation.py)\n",
    "\n",
    "<img src=\"./media/majority.png\" width=700>\n",
    "\n",
    "Majority is a utility function that implements a basic voting mechanism across multiple completions to determine the most common answer. It works by taking either a Prediction object (which contains completions) or a list of completions directly, then normalizes their values for the target field (either specified or defaults to the last output field). The normalization process, handled by normalize_text, helps manage slight variations in text that should be considered the same answer (returning None for answers that should be ignored). In cases of ties, earlier completions are prioritized. The function is particularly useful when combined with modules that generate multiple completions (like running predictors with different temperatures) and you want a simple way to find the most common response. The function returns a new Prediction object containing just the winning completion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "75c0d15a-74c1-4379-a2ea-4948ea23f562",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most common sentiment: Positive\n"
     ]
    }
   ],
   "source": [
    "# Example Completions From Prior Multi-Chain\n",
    "majority_result = dspy.majority(cot_completions, field='sentiment')\n",
    "\n",
    "# Results\n",
    "print(f\"Most common sentiment: {majority_result.sentiment}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d88698ff-7227-4a65-ac74-ca6e915603de",
   "metadata": {},
   "source": [
    "---\n",
    "## Evaluators\n",
    "\n",
    "While modules are the building blocks of your program, you may have realized there's limited ability to actually tune or change your modules directly like you would iterate on prompt chains. This is where DSPy starts to differentiate itself, as it aims to tune performance of your modules through measuring against defined metrics.\n",
    "\n",
    "As such, you need to deeply consider the optimal state of your LLM output and how you would measure it. This can be as simple as accuracy for classification tasks, or more complex like faithfulness to retrieved context."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fb19ae3-476c-46bf-99c2-29c3a5952891",
   "metadata": {},
   "source": [
    "### Example Data Type\n",
    "\n",
    "The data type for DSPy evaluators and metrics is the `Example` object. In essence it's just a `dict` but handles the formatting that the DSPy backend expects. The fields can be anything you'd like, but make sure they match up to your current input and output formatting for your module.\n",
    "\n",
    "Your training set data will consist of a list of examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "8b62e852-043a-42df-8c80-b49d92f631c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example({'question': 'What is my name?', 'answer': 'Your name is Adam Lucek'}) (input_keys=None)\n",
      "What is my name?\n",
      "Your name is Adam Lucek\n"
     ]
    }
   ],
   "source": [
    "qa_pair = dspy.Example(question=\"What is my name?\", answer=\"Your name is Adam Lucek\")\n",
    "\n",
    "print(qa_pair)\n",
    "print(qa_pair.question)\n",
    "print(qa_pair.answer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "3ebbc2bf-6495-4cb4-aec5-dbf59ba74f96",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example({'excerpt': 'I really love programming!', 'classification': 'Positive', 'confidence': 0.95}) (input_keys=None)\n",
      "I really love programming!\n",
      "Positive\n",
      "0.95\n"
     ]
    }
   ],
   "source": [
    "classification_pair = dspy.Example(excerpt=\"I really love programming!\", classification=\"Positive\", confidence=0.95)\n",
    "\n",
    "print(classification_pair)\n",
    "print(classification_pair.excerpt)\n",
    "print(classification_pair.classification)\n",
    "print(classification_pair.confidence)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eddb0021-4117-4ed7-940f-321390b337f9",
   "metadata": {},
   "source": [
    "You may also explicitly label `inputs` and `labels` using the `.with_inputs()` method. Anything not specified in `.with_inputs()` is then expected to either be labels or metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4a82e7a2-7f61-414d-b16a-d628731431b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Example with Input fields only: Example({'article': 'Placeholder for Article'}) (input_keys={'article'})\n",
      "\n",
      "Example object Non-Input fields only: Example({'summary': 'Expected Summary'}) (input_keys=None)\n"
     ]
    }
   ],
   "source": [
    "article_summary = dspy.Example(article = \"Placeholder for Article\", summary= \"Expected Summary\").with_inputs(\"article\")\n",
    "\n",
    "input_key_only = article_summary.inputs()\n",
    "non_input_key_only = article_summary.labels()\n",
    "\n",
    "print(\"Example with Input fields only:\", article_summary.inputs())\n",
    "print(\"\\nExample object Non-Input fields only:\", article_summary.labels())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56367549-d4a2-4aef-8aa6-82d1769291bb",
   "metadata": {},
   "source": [
    "### Metrics\n",
    "\n",
    "<img src=\"./media/metrics.png\" width=600>\n",
    "\n",
    "Now that we understand the data format, we must consider our metrics. Metrics are critical to DSPy as the framework will optimize your modules towards defined metrics.\n",
    "\n",
    "DSPy defines metrics concisely *A metric is just a function that will take examples from your data and the output of your system and return a score that quantifies how good the output is. What makes outputs from your system good or bad?*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8601d33c-8668-418e-851d-3e176977bed8",
   "metadata": {},
   "source": [
    "#### Simple Metrics\n",
    "\n",
    "<img src=\"./media/simple_metrics.png\" width=300>\n",
    "\n",
    "Starting simply, setup and run validation for exact matches across a sentiment classification module"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5401ccf6-34da-4710-9e35-ecf3a2652510",
   "metadata": {},
   "source": [
    "**Setup Module**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "f05e5d2c-dde9-4de6-a788-7ffa6375d44b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple Tweet Sentiment Classification Module\n",
    "from typing import Literal\n",
    "\n",
    "class TwtSentiment(dspy.Signature):\n",
    "    tweet: str = dspy.InputField(desc=\"Candidate tweet for classificaiton\")\n",
    "    sentiment: Literal[\"positive\", \"negative\", \"neutral\"] = dspy.OutputField()\n",
    "\n",
    "twt_sentiment = dspy.ChainOfThought(TwtSentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4911e354-a446-428f-9628-46dca7567db4",
   "metadata": {},
   "source": [
    "**Format Dataset**\n",
    "\n",
    "We'll grab some example tweet and sentiment pairs from the [MTEB Tweeet Sentiment Extraction](https://huggingface.co/datasets/mteb/tweet_sentiment_extraction) dataset. This will be our dataset we validate against."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "3ac4fac1-2c9e-4f82-9746-3350530be540",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Formatting Examples\n",
    "examples = []\n",
    "num_examples = 50\n",
    "\n",
    "with open(\"./datasets/tweets.jsonl\", 'r', encoding='utf-8') as f:\n",
    "    for i, line in enumerate(f):\n",
    "        if num_examples and i >= num_examples:\n",
    "            break\n",
    "            \n",
    "        data = json.loads(line.strip())\n",
    "        example = dspy.Example(\n",
    "            tweet=data['text'],\n",
    "            sentiment=data['label_text']\n",
    "        ).with_inputs(\"tweet\")\n",
    "        examples.append(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "606ea004-dee1-40da-9c2b-eca82718f4e3",
   "metadata": {},
   "source": [
    "**Defining Metric**\n",
    "\n",
    "The metric takes in an example, a prediction and an optional trace (we'll discuss the trace at a later point). In this case, it will return `True` or `False` depending on whether the llm predicted sentiment is the same as our ground truth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "710d6e00-5c7e-448d-b1f7-6689e9a31167",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_answer(example, pred, trace=None):\n",
    "    return example.sentiment.lower() == pred.sentiment.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1901b202-c843-42af-a771-2e8cf23427c2",
   "metadata": {},
   "source": [
    "**Running A Manual Evaluation**\n",
    "\n",
    "For each tweet in the examples it will run a prediction with our examples defined inputs (the tweet), this is then ran through our `validate_answer` metric which returns True or False and is then stored in our scores list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "6fa9b3dd-753a-47f6-83fc-1a889a856a61",
   "metadata": {},
   "outputs": [],
   "source": [
    "scores = []\n",
    "for x in examples:\n",
    "    pred = twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "8a0d6d7c-ef85-4e13-b281-62ccf4d536b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Accuracy:  0.76\n"
     ]
    }
   ],
   "source": [
    "accuracy = sum(scores) / len(scores)\n",
    "print(\"Baseline Accuracy: \", accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78f2fff-d437-460e-aed3-387a358779b3",
   "metadata": {},
   "source": [
    "#### Intermediate Metrics\n",
    "\n",
    "<img src=\"./media/inter_metrics.png\" width=300>\n",
    "\n",
    "While these direct ground truth comparisons are good, we've seen the introduction of LLM-as-a-judge approaches assist in comparing and judging long form outputs.\n",
    "\n",
    "Let's implement some LLM based metrics:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f61a28ba-5be9-4d23-9016-9e727f33400e",
   "metadata": {},
   "source": [
    "**Setup Module**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "180432ca-6b0d-483a-9b33-94f66257ef31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# CoT For Summarizing a Dialogue\n",
    "\n",
    "dialog_sum = dspy.ChainOfThought(\"dialogue: str -> summary: str\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "832c4f3c-22f0-4db9-8a25-644dc1f77b67",
   "metadata": {},
   "source": [
    "**Format Dataset**\n",
    "\n",
    "Our dataset for this example comes from [DialogSum](https://github.com/cylnlp/dialogsum), a collection of dialogues and corresponding summaries. We can use their summaries as the \"gold\" standard to test against with fuzzy metrics from an LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "348b0d4e-106d-41a5-95c2-7d3b3fb820d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "num_examples = 20\n",
    "df = pd.read_csv(\"./datasets/dialogsum.csv\")\n",
    "    \n",
    "# Limit the number of examples\n",
    "if num_examples:\n",
    "    df = df.head(num_examples)\n",
    "\n",
    "dialogsum_examples = []\n",
    "\n",
    "for _, row in df.iterrows():\n",
    "    example = dspy.Example(\n",
    "        dialogue=row['dialogue'],\n",
    "        summary=row['summary']\n",
    "    ).with_inputs('dialogue')\n",
    "    \n",
    "    dialogsum_examples.append(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07eef3ee-ef51-47cb-9645-f9bf39119e11",
   "metadata": {},
   "source": [
    "**Metric Signature**\n",
    "\n",
    "Now that we're using modules within our metrics, we need a dynamic signature that we can apply to metric predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "3bb1fcbc-3a35-4850-b715-8a5dcf021f86",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the signature for automatic assessments.\n",
    "class Assess(dspy.Signature):\n",
    "    \"\"\"Assess the quality of a dialog summary along the specified dimension.\"\"\"\n",
    "\n",
    "    assessed_text = dspy.InputField()\n",
    "    assessment_question = dspy.InputField()\n",
    "    assessment_answer: bool = dspy.OutputField()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "431562b1-3a95-4d8e-8cac-87919ff7b484",
   "metadata": {},
   "source": [
    "**Metric Definition**\n",
    "\n",
    "We'll be using an LLM to assess whether the generated dialogue summary is accurate in comparison to the original quesiton, and concise in comparison to the expected summary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "id": "ecae2695-43c4-49dd-b40e-c15cd01fff64",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dialog_metric(gold, pred, trace=None):\n",
    "    dialogue, gold_summary, generated_summary = gold.dialogue, gold.summary, pred.summary\n",
    "    \n",
    "    # Define Assessment Questions\n",
    "    accurate_question = f\"Given this original dialog: '{dialogue}', does the summary accurately represent what was discussed without adding or changing information?\"\n",
    "    \n",
    "    concise_question = f\"\"\"Compare the level of detail in the generated summary with the gold summary:\n",
    "    Gold summary: '{gold_summary}'\n",
    "    Is the generated summary appropriately detailed - neither too sparse nor too verbose compared to the gold summary?\"\"\"\n",
    "\n",
    "    # Run Predictions\n",
    "    accurate = dspy.Predict(Assess)(assessed_text=generated_summary, assessment_question=accurate_question)\n",
    "    concise = dspy.Predict(Assess)(assessed_text=generated_summary, assessment_question=concise_question)\n",
    "    \n",
    "    # Extract boolean assessment answers\n",
    "    accurate, concise = [m.assessment_answer for m in [accurate, concise]]\n",
    "    \n",
    "    # Calculate score - accuracy is required for any points\n",
    "    score = (accurate + concise) if accurate else 0\n",
    "    \n",
    "    if trace is not None:\n",
    "        return score >= 2\n",
    "        \n",
    "    return score / 2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0928f747-b822-457f-8ffe-42322316d2a8",
   "metadata": {},
   "source": [
    "**Running Evaluation**\n",
    "\n",
    "Similar manual evaluation to what we did earlier!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5f0d7289-f9a4-4003-9d7a-92d1a9f8a23e",
   "metadata": {},
   "outputs": [],
   "source": [
    "intermediate_scores = []\n",
    "for x in dialogsum_examples:\n",
    "    pred = dialog_sum(**x.inputs())\n",
    "    score = dialog_metric(x, pred)\n",
    "    intermediate_scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "7bc22618-4874-4ebc-92fc-ad812b05d9a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dialog Metric Score:  0.85\n"
     ]
    }
   ],
   "source": [
    "final_score = sum(intermediate_scores) / len(intermediate_scores)\n",
    "print(\"Dialog Metric Score: \", final_score)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e7856a-473c-464b-b5ac-601bbf65546d",
   "metadata": {},
   "source": [
    "#### Advanced Metrics with Tracing in DSPy\n",
    "\n",
    "<img src=\"./media/advan_metrics.png\" width=300>\n",
    "\n",
    "DSPy's documentation highlights two key points about using modules as metrics:\n",
    "\n",
    "1. If your metric is itself a DSPy program, one of the most powerful ways to iterate is to compile (optimize) your metric itself. That's usually easy because the output of the metric is usually a simple value (e.g., a score out of 5) so the metric's metric is easy to define and optimize by collecting a few examples.\n",
    "\n",
    "2. When your metric is used during evaluation runs, DSPy will not try to track the steps of your program. But during compiling (optimization), DSPy will trace your LM calls. The trace will contain inputs/outputs to each DSPy predictor and you can leverage that to validate intermediate steps for optimization.\n",
    "\n",
    "Digging into the second point with our prior example, the metric operates in two modes:\n",
    "\n",
    "**Standard Evaluation (trace=None)**: Returns a normalized score (0-1) based on accuracy and conciseness of the summary, requiring factual accuracy as a gating factor.\n",
    "\n",
    "**Compilation Mode (trace available)**: During compilation, DSPy provides us with the trace of our ChainOfThought module `(dialog_sum)`. While our standard evaluation returns a normalized score between 0-1, in compilation mode we alter the return logic to instead provide a binary success criterion `(score >= 2)`. This binary signal helps DSPy optimize more effectively during compilation by providing a clear success/failure signal for each example.\n",
    "\n",
    "```python\n",
    "def dialog_metric(gold, pred, trace=None):\n",
    "    dialogue, gold_summary, generated_summary = gold.dialogue, gold.summary, pred.summary\n",
    "    \n",
    "    # LLM-based assessment using Assess signature\n",
    "    accurate = dspy.Predict(Assess)(assessed_text=generated_summary, assessment_question=accurate_question)\n",
    "    concise = dspy.Predict(Assess)(assessed_text=generated_summary, assessment_question=concise_question)\n",
    "    \n",
    "    if trace is not None:\n",
    "        # During compilation: Can access and validate CoT reasoning steps\n",
    "        # We're not doing anything with it currently but you can access in this way\n",
    "        reasoning_steps = [output.reasoning for *_, output in trace if hasattr(output, 'reasoning')]\n",
    "        # Return binary success criteria for optimization\n",
    "        return score >= 2  # Requires both accuracy and conciseness\n",
    "    \n",
    "    return score / 2.0  # Normalized evaluation score\n",
    "```\n",
    "\n",
    "The trace functionality is particularly valuable for complex modules like our ChainOfThought implementation as it alters how DSPy handles optimization. During compilation, instead of returning normalized scores, we provide binary success signals based on specific criteria (score >= 2). This binary feedback helps DSPy more effectively optimize the model by providing clear success/failure signals for each example.\n",
    "\n",
    "This dual-mode evaluation strategy serves two distinct purposes. During normal evaluation, we get detailed normalized scores to assess model performance. During compilation, we switch to binary success criteria to guide optimization more effectively. This approach helps us maintain rich evaluation metrics while providing clearer signals for model improvement during the compilation phase. We could also further complicate this by including signals from intermediate steps that are generally obfuscated."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7902c886-81fd-45d5-97e6-943ff66c548f",
   "metadata": {},
   "source": [
    "---\n",
    "## Optimization\n",
    "\n",
    "<img src=\"./media/optimizers.png\" width=500>\n",
    "\n",
    "So now that we have some modules and metrics we're measuring against, we can take the final step of optimizing our programs. This takes the guesswork out of tweaking and editing prompts by automatically testing, assessing and iterating against measurable values.\n",
    "\n",
    "DSPy offers a few ways to optimize your programs, copied over [from the docs](https://dspy.ai/learn/optimization/optimizers/):\n",
    "\n",
    "**Automatic Few-Shot Learning**\n",
    "These optimizers extend the signature by automatically generating and including optimized examples within the prompt sent to the model, implementing few-shot learning.\n",
    "\n",
    "- `LabeledFewShot`: Simply constructs few-shot examples (demos) from provided labeled input and output data points. Requires k (number of examples for the prompt) and trainset to randomly select k examples from.\n",
    "\n",
    "- `BootstrapFewShot`: Uses a teacher module (which defaults to your program) to generate complete demonstrations for every stage of your program, along with labeled examples in trainset. Parameters include max_labeled_demos (the number of demonstrations randomly selected from the trainset) and max_bootstrapped_demos (the number of additional examples generated by the teacher). The bootstrapping process employs the metric to validate demonstrations, including only those that pass the metric in the \"compiled\" prompt. Advanced: Supports using a teacher program that is a different DSPy program that has compatible structure, for harder tasks.\n",
    "\n",
    "- `BootstrapFewShotWithRandomSearch`: Applies BootstrapFewShot several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of BootstrapFewShot, with the addition of num_candidate_programs, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, LabeledFewShot optimized program, BootstrapFewShot compiled program with unshuffled examples and num_candidate_programs of BootstrapFewShot compiled programs with randomized example sets.\n",
    "\n",
    "- `KNNFewShot`: Uses k-Nearest Neighbors algorithm to find the nearest training example demonstrations for a given input example. These nearest neighbor demonstrations are then used as the trainset for the BootstrapFewShot optimization process. See this notebook for an example.\n",
    "\n",
    "**Automatic Instruction Optimization**\n",
    "These optimizers produce optimal instructions for the prompt and, in the case of MIPROv2 can also optimize the set of few-shot demonstrations.\n",
    "\n",
    "- `COPRO`: Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the trainset). Parameters include depth which is the number of iterations of prompt improvement the optimizer runs over.\n",
    "\n",
    "- `MIPROv2`: Generates instructions and few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules.\n",
    "\n",
    "**Automatic Finetuning**\n",
    "This optimizer is used to fine-tune the underlying LLM(s).\n",
    "\n",
    "- `BootstrapFinetune`: Distills a prompt-based DSPy program into weight updates. The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM.\n",
    "\n",
    "**Program Transformations**\n",
    "- `Ensemble`: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063e1119-620f-4cee-bf2a-8c7cf60da190",
   "metadata": {},
   "source": [
    "**Loading Train and Test for Tweets**\n",
    "\n",
    "For our examples, we'll be optimizing the tweet sentiment classification module from before. While classification tasks are not the best examples for LLM applications, it will still allow us to understand in a lightweight way what's going on behind each optimizer so we can better apply them to more advanced programs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "a30d0ce0-e86c-45b4-9511-5988b6c5286f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Formatting Examples\n",
    "twitter_train = []\n",
    "twitter_test = []\n",
    "train_size = 100  # how many for train \n",
    "test_size = 200   # how many for test\n",
    "\n",
    "with open(\"./datasets/tweets.jsonl\", 'r', encoding='utf-8') as f:\n",
    "   for i, line in enumerate(f):\n",
    "       if i >= (train_size + test_size):\n",
    "           break\n",
    "           \n",
    "       data = json.loads(line.strip())\n",
    "       example = dspy.Example(\n",
    "           tweet=data['text'],\n",
    "           sentiment=data['label_text']\n",
    "       ).with_inputs(\"tweet\")\n",
    "       \n",
    "       if i < train_size:\n",
    "           twitter_train.append(example)\n",
    "       else:\n",
    "           twitter_test.append(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d022044-e3e8-4ced-bcb7-b9eebefc3b9d",
   "metadata": {},
   "source": [
    "**Candidate Program**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8fbdbe86-ed1c-4147-afb0-8b19164e2bc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Simple Tweet Sentiment Classification Module\n",
    "from typing import Literal\n",
    "\n",
    "class TwtSentiment(dspy.Signature):\n",
    "    tweet: str = dspy.InputField(desc=\"Candidate tweet for classificaiton\")\n",
    "    sentiment: Literal[\"positive\", \"negative\", \"neutral\"] = dspy.OutputField()\n",
    "\n",
    "base_twt_sentiment = dspy.Predict(TwtSentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d58fe87c-ed1b-4882-a26b-7408846baab6",
   "metadata": {},
   "source": [
    "**Simple Metrics**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "daf38c22-e76c-456c-aecb-876ae4a9bd3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def validate_answer(example, pred, trace=None):\n",
    "    return example.sentiment.lower() == pred.sentiment.lower()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb80f619-c994-4d93-be40-8e213edd6d4b",
   "metadata": {},
   "source": [
    "**Baseline Score**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "98020703-6a83-40c1-88d8-432a4cdb7244",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Baseline Accuracy:  0.69\n"
     ]
    }
   ],
   "source": [
    "baseline_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = base_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    baseline_scores.append(score)\n",
    "\n",
    "base_accuracy = baseline_scores.count(True) / len(baseline_scores)\n",
    "print(\"Baseline Accuracy: \", base_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3dc33d56-3516-4b78-b198-97a5b52921fa",
   "metadata": {},
   "source": [
    "**Example Tweet We'll Run Each Program Through**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "eda44269-775f-45b2-9062-74657de17d40",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Expected Positive Label\n",
    "example_tweet = \"Hi! Waking up, and not lazy at all. You would be proud of me, 8 am here!!! Btw, nice colour, not burnt.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2955e0d2-2740-4f8b-a747-94b0d142002d",
   "metadata": {},
   "source": [
    "### Automatic Few Shot Learning\n",
    "\n",
    "<img src=\"./media/auto_fewshot.png\" width=300>\n",
    "\n",
    "These optimizers are focused around providing the best examples either by finding similar examples for your query in the training data during inference, or by generating optimized examples to use from the program itself."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1328534c-1aa2-465a-94d8-371fd3ae9c06",
   "metadata": {},
   "source": [
    "#### LabeledFewShot\n",
    "\n",
    "<img src=\"./media/labeled_few_shot.png\" width=600>\n",
    "\n",
    "The simplest optimizer. Randomly selects k examples from your training data to use as demonstrations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "b1ce3158-a10c-4aef-b820-e9980a716496",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dspy.teleprompt import LabeledFewShot\n",
    "\n",
    "lfs_optimizer = LabeledFewShot(k=16)  # Use 16 examples in prompts\n",
    "\n",
    "lfs_twt_sentiment = lfs_optimizer.compile(base_twt_sentiment, trainset=twitter_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "60b3937a-2920-45f3-bdf5-6bb8c2ad7509",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labeled Few Shot Accuracy:  0.695\n"
     ]
    }
   ],
   "source": [
    "lfs_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = lfs_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    lfs_scores.append(score)\n",
    "\n",
    "lfs_accuracy = lfs_scores.count(True) / len(lfs_scores)\n",
    "print(\"Labeled Few Shot Accuracy: \", lfs_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "de9c155d-eed3-4e23-9336-e735d434470f",
   "metadata": {},
   "outputs": [],
   "source": [
    "lfs_twt_sentiment.save(\"./optimized/lfs_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "d1630be8-c24e-490b-a3b5-8d54ada53c38",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(lfs_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "726fefdf-eba3-45f8-8e10-346a64855aca",
   "metadata": {},
   "source": [
    "#### BootstrapFewShot \n",
    "\n",
    "<img src=\"./media/bootstrap_fewshot.png\" width=900>\n",
    "\n",
    "Generates high-quality examples by executing your program and keeping only successful runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c4e5087f-25d0-4c4d-b474-a50b1ba7e484",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  4%|█▋                                        | 4/100 [00:00<00:00, 199.89it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 4 examples for up to 1 rounds, amounting to 4 attempts.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "from dspy.teleprompt import BootstrapFewShot\n",
    "\n",
    "bsfs_optimizer = BootstrapFewShot(\n",
    "    metric=validate_answer,          # Function to evaluate quality\n",
    "    max_bootstrapped_demos=4,        # Generated examples\n",
    "    max_labeled_demos=16,            # Examples from training data\n",
    "    metric_threshold=1               # Minimum quality threshold\n",
    ")\n",
    "\n",
    "bsfs_twt_sentiment = bsfw_optimizer.compile(base_twt_sentiment, trainset=twitter_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "dd4424fe-00e9-427d-83cb-43d76d46b73a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Labeled Few Shot Accuracy:  0.715\n"
     ]
    }
   ],
   "source": [
    "bsfs_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = bsfw_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    bsfs_scores.append(score)\n",
    "\n",
    "bsfs_accuracy = bsfs_scores.count(True) / len(bsfs_scores)\n",
    "print(\"Bootstrap Few Shot Accuracy: \", bsfs_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "98dfaacd-8b6b-419b-b143-4399b11766f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "bsfs_twt_sentiment.save(\"./optimized/bsfs_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "1295d6d0-0a3c-423e-927e-793d1f507f4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(bsfs_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8e406d2-f445-4395-87ed-20848f5bd5ba",
   "metadata": {},
   "source": [
    "#### BootstrapFewShotWithRandomSearch\n",
    "\n",
    "<img src=\"./media/bsfswrs_diagram.png\" width=900>\n",
    "\n",
    "Extends BootstrapFewShot by trying multiple random sets of examples to find the best performing combination."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e90ef5e8-6ae9-485a-bb04-150262146ab9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dspy.teleprompt import BootstrapFewShotWithRandomSearch\n",
    "\n",
    "bsfswrs_optimizer = BootstrapFewShotWithRandomSearch(\n",
    "    metric=validate_answer,\n",
    "    num_candidate_programs=16,\n",
    "    max_bootstrapped_demos=4,\n",
    "    max_labeled_demos=16\n",
    ")\n",
    "\n",
    "bsfswrs_twt_sentiment = bsfswrs_optimizer.compile(base_twt_sentiment, trainset=twitter_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e37f3146-8c97-4772-b739-39cf97409a0c",
   "metadata": {},
   "source": [
    "<img src=\"./media/bsfswrs.png\" width=800>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "0788556f-d7cb-4884-b7c2-b3f4fead1ff2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrap Few Shot With Random Search Accuracy:  0.7\n"
     ]
    }
   ],
   "source": [
    "bsfswrs_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = bsfswrs_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    bsfswrs_scores.append(score)\n",
    "\n",
    "bsfswrs_accuracy = bsfswrs_scores.count(True) / len(bsfswrs_scores)\n",
    "print(\"Bootstrap Few Shot With Random Search Accuracy: \", bsfswrs_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "d35f2dfa-a39c-4cd9-bdea-080866a8eafc",
   "metadata": {},
   "outputs": [],
   "source": [
    "bsfswrs_twt_sentiment.save(\"./optimized/bsfswrs_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "e2a2e40e-0149-4460-9636-52558459de6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(bsfswrs_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76d73c88-f8c9-49f1-996b-5efb5da74f64",
   "metadata": {},
   "source": [
    "#### KNNFewShot\n",
    "\n",
    "<img src=\"./media/knn_diagram.png\" width=800>\n",
    "\n",
    "Dynamically selects relevant examples based on similarity to the input."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c02bc4e-c009-4865-aa38-417aaf3c8205",
   "metadata": {},
   "source": [
    "**Defining an Embedding Function**\n",
    "\n",
    "As KNN retrieval relies on vector similarity, we need a quick embedding function. This is a very simple setup that uses OpenAI's api."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "37169d2d-3136-4837-8270-833e698243d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "import numpy as np\n",
    "\n",
    "client = OpenAI()\n",
    "\n",
    "def openai_embeddings(texts):\n",
    "    if isinstance(texts, str):\n",
    "        texts = [texts]\n",
    "    \n",
    "    response = client.embeddings.create(\n",
    "        model=\"text-embedding-3-small\",\n",
    "        input=texts\n",
    "    )\n",
    "    \n",
    "    # Convert to numpy array\n",
    "    embeddings = np.array([embedding.embedding for embedding in response.data], dtype=np.float32)\n",
    "    \n",
    "    # If single text, return single embedding\n",
    "    if len(embeddings) == 1:\n",
    "        return embeddings[0]\n",
    "    return embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2ebf5a88-c119-4626-b2cf-ceecfbc5f7bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dspy.teleprompt import KNNFewShot\n",
    "\n",
    "knn_optimizer = KNNFewShot(\n",
    "    k=5,                               # Number of neighbors to use\n",
    "    trainset=twitter_train,            # Dataset for finding neighbors\n",
    "    vectorizer=openai_embeddings       # Function to convert inputs to vectors\n",
    ")\n",
    "\n",
    "knn_twt_sentiment = knn_optimizer.compile(base_twt_sentiment, trainset=twitter_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bb82a5c-62c9-4653-b27f-e19a3cd47563",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "knn_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = knn_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    knn_scores.append(score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "05ed64de-325a-47d5-90e3-3f3afecad10a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "KNN Few Shot Accuracy:  0.7\n"
     ]
    }
   ],
   "source": [
    "knn_accuracy = knn_scores.count(True) / len(knn_scores)\n",
    "print(\"KNN Few Shot Accuracy: \", knn_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "63a2ee91-273f-4395-9410-3bd002ef1517",
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_twt_sentiment.save(\"./optimized/knn_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "9fba2b9a-58ed-4228-8520-553f4eaa7290",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " 80%|████████████████████████████████████         | 4/5 [00:02<00:00,  1.69it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrapped 4 full traces after 4 examples for up to 1 rounds, amounting to 4 attempts.\n",
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(knn_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d2e86af-1c2e-4db3-af5d-5f289943e8f7",
   "metadata": {},
   "source": [
    "### Instruction Optimization\n",
    "\n",
    "<img src=\"./media/auto_instr.png\" width=300>\n",
    "\n",
    "These optimizers improve the actual instructions and prompts given to the model, enhancing zero-shot performance rather than the few shot setups shown above."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8f241e6-2a53-4536-99fb-aadb72d7d0c9",
   "metadata": {},
   "source": [
    "#### COPRO (Coordinate Prompt Optimization)\n",
    "\n",
    "<img src=\"./media/copro_diagram.png\" width=1000>\n",
    "\n",
    "Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the trainset). Parameters include depth which is the number of iterations of prompt improvement the optimizer runs over."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e9b82ca-a0c9-4138-9f47-a1c522cbf43d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from dspy.teleprompt import COPRO\n",
    "\n",
    "copro_optimizer = COPRO(\n",
    "    metric=validate_answer,              # Metric to Optimize Against\n",
    "    prompt_model= dspy.LM('openai/gpt-4o'), # Different Model for Prompt Generation\n",
    "    breadth=10,                          # New prompts per iteration\n",
    "    depth=3,                             # Number of improvement rounds\n",
    "    init_temperature=1.4                 # Creativity in generation\n",
    ")\n",
    "\n",
    "copro_twt_sentiment = copro_optimizer.compile(base_twt_sentiment, trainset=twitter_train, eval_kwargs={'num_threads': 6, 'display_progress': True})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "83e3e525-c203-4972-b2f2-223938102aa1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CORPO Accuracy:  0.71\n"
     ]
    }
   ],
   "source": [
    "corpo_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = copro_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    corpo_scores.append(score)\n",
    "\n",
    "corpo_accuracy = corpo_scores.count(True) / len(corpo_scores)\n",
    "print(\"CORPO Accuracy: \", corpo_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "f2854fac-3397-47e2-abf1-67e83cd6d02b",
   "metadata": {},
   "outputs": [],
   "source": [
    "copro_twt_sentiment.save(\"./optimized/copro_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "174c5403-2455-4d80-9cdb-66db486d6f29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(copro_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40895940-3c94-45c4-b445-195a557d53e4",
   "metadata": {},
   "source": [
    "#### MIPROv2 (Multiprompt Instruction Proposal Optimizer Version 2)\n",
    "\n",
    "<img src=\"./media/mipro_diagram.png\" width=1000>\n",
    "\n",
    "Generates instructions and few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03a3594f-fa3d-4fbf-bf34-121ebd79bfab",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from dspy.teleprompt import MIPROv2\n",
    "\n",
    "mipro_optimizer = MIPROv2(\n",
    "    metric=validate_answer,\n",
    "    prompt_model= dspy.LM('openai/gpt-4o'), # Different Model for Prompt Generation\n",
    "    num_candidates=10,                      # Instructions to try\n",
    ")\n",
    "\n",
    "mipro_twt_sentiment = mipro_optimizer.compile(base_twt_sentiment, trainset=twitter_train, valset=twitter_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "d07efad0-e1c1-4c2e-a233-91d53fa7e0ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MIPRO Accuracy:  0.715\n"
     ]
    }
   ],
   "source": [
    "mipro_scores = []\n",
    "for x in twitter_test:\n",
    "    pred = mipro_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    mipro_scores.append(score)\n",
    "\n",
    "mipro_accuracy = mipro_scores.count(True) / len(mipro_scores)\n",
    "print(\"MIPRO Accuracy: \", mipro_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "e303d3b1-6a3b-4056-b27a-8612f8fc4071",
   "metadata": {},
   "outputs": [],
   "source": [
    "mipro_twt_sentiment.save(\"./optimized/mipro_twt_sentiment.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "id": "7de48564-3c5d-4d91-a5f9-f175870c3065",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(mipro_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abcc4488-71b8-4480-a04e-39816bcf07ee",
   "metadata": {},
   "source": [
    "### Automatic Finetuning\n",
    "\n",
    "<img src=\"./media/auto_ft.png\" width=300>\n",
    "\n",
    "Once you have a well optimized program, you may want to start looking for even further optimizations. Ideally, you would use a large and expensive model first to get the best performance, then transfer that knowledge to an optimized smaller model (or continually train an existing model)\n",
    "\n",
    "DSPy offers a solution to automatically use your best programs to create training data for downstream finetuning with `BootstrapFinetune`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd130178-ddcb-4af8-8f7d-0a860b55211a",
   "metadata": {},
   "source": [
    "#### BootstrapFinetune\n",
    "\n",
    "<img src=\"./media/bootstrap_finetune_diagram.png\" width=1000>\n",
    "\n",
    "Creates fine-tuned versions of language models based on successful program executions. In this example we'll instill our best performing program from MIPROv2 directly into gpt-4o-mini."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "30dcedb7-58c0-4ac7-9e2d-ec2d73534b50",
   "metadata": {},
   "outputs": [],
   "source": [
    "dspy.settings.experimental = True"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fa389d0-9a6a-4813-85c4-d4219c34e663",
   "metadata": {},
   "source": [
    "**Grabbing some additional data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "3bba076e-0d87-4758-9283-74f7808eec62",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# Formatting Examples\n",
    "bsft_twitter_train = []\n",
    "bsft_twitter_test = []\n",
    "train_size = 500  # how many for train \n",
    "test_size = 200    # how many for test\n",
    "\n",
    "with open(\"./datasets/tweets.jsonl\", 'r', encoding='utf-8') as f:\n",
    "   for i, line in enumerate(f):\n",
    "       if i >= (train_size + test_size):\n",
    "           break\n",
    "           \n",
    "       data = json.loads(line.strip())\n",
    "       example = dspy.Example(\n",
    "           tweet=data['text'],\n",
    "           sentiment=data['label_text']\n",
    "       ).with_inputs(\"tweet\")\n",
    "       \n",
    "       if i < train_size:\n",
    "           bsft_twitter_train.append(example)\n",
    "       else:\n",
    "           bsft_twitter_test.append(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "228c3889-ec33-40b5-9d07-a9214a3d0dd0",
   "metadata": {},
   "source": [
    "**Teacher and Student**\n",
    "\n",
    "At it's core `BootstrapFinetune` is meant to use our best optimized program to create training data to fine tune a language model. As such we need a teacher model that will be used across our data to create the examples, and then a student program with a target model to be fine tuned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "88c88c7a-a3cc-4049-b502-55f820ee43ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First make a deep copy of your optimized MIPRO program as the teacher\n",
    "teacher = mipro_twt_sentiment.deepcopy()\n",
    "\n",
    "# Create student as a copy but with your target model\n",
    "student = mipro_twt_sentiment.deepcopy()\n",
    "student.set_lm(dspy.LM(\"gpt-4o-mini-2024-07-18\"))  # e.g., mistral or whatever model you want to fine-tune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "ae4f2657-2964-46f7-a2f3-68b2e94923d7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[BootstrapFinetune] Preparing the student and teacher programs...\n",
      "[BootstrapFinetune] Bootstrapping data...\n",
      "Average Metric: 362.00 / 500 (72.4%): 100%|██| 500/500 [00:00<00:00, 628.33it/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024/12/30 00:58:17 INFO dspy.evaluate.evaluate: Average Metric: 362 / 500 (72.4%)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[BootstrapFinetune] Preparing the train data...\n",
      "[BootstrapFinetune] Collected data for 500 examples\n",
      "[BootstrapFinetune] After filtering with the metric, 362 examples remain\n",
      "[BootstrapFinetune] Using 362 data points for fine-tuning the model: gpt-4o-mini-2024-07-18\n",
      "[BootstrapFinetune] Starting LM fine-tuning...\n",
      "[BootstrapFinetune] 1 fine-tuning job(s) to start\n",
      "[BootstrapFinetune] Starting 1 fine-tuning job(s)...\n",
      "[OpenAI Provider] Validating the data format\n",
      "[OpenAI Provider] Saving the data to a file\n",
      "[OpenAI Provider] Data saved to /Users/adamlucek/.dspy_cache/finetune/798b39e1a18373a3.jsonl\n",
      "[OpenAI Provider] Uploading the data to the provider\n",
      "[OpenAI Provider] Starting remote training\n",
      "[OpenAI Provider] Job started with the OpenAI Job ID ftjob-L8D3vni8wlEyuCOAhIgzuFHF\n",
      "[OpenAI Provider] Waiting for training to complete\n",
      "[OpenAI Provider] 2024-12-30 00:58:23 Validating training file: file-Sh4DqQsYEY5UaqJEHGy37y\n",
      "[OpenAI Provider] 2024-12-30 01:02:36 Fine-tuning job started\n",
      "[OpenAI Provider] The OpenAI estimated time remaining is: 0:09:13.388291\n",
      "[OpenAI Provider] 2024-12-30 01:05:02 Step 11/1086: training loss=0.14\n",
      "[OpenAI Provider] 2024-12-30 01:05:27 Step 32/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:05:45 Step 53/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:06:01 Step 66/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:06:29 Step 94/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:06:44 Step 110/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:07:10 Step 132/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:07:28 Step 152/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:07:53 Step 177/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:08:12 Step 194/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:08:28 Step 211/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:08:53 Step 232/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:09:13 Step 253/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:09:38 Step 279/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:09:56 Step 297/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:10:18 Step 317/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:10:38 Step 334/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:10:56 Step 353/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:11:05 Step 361/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:11:41 Step 363/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:11:57 Step 375/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:12:26 Step 403/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:12:44 Step 421/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:13:00 Step 437/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:13:25 Step 457/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:13:43 Step 478/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:14:09 Step 501/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:14:26 Step 519/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:14:52 Step 543/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:15:07 Step 556/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:15:34 Step 583/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:15:50 Step 599/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:16:16 Step 621/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:16:34 Step 641/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:16:59 Step 665/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:17:17 Step 683/1086: training loss=0.01\n",
      "[OpenAI Provider] 2024-12-30 01:17:32 Step 699/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:17:57 Step 720/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:18:41 Step 740/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:18:59 Step 758/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:19:17 Step 772/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:19:42 Step 796/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:20:01 Step 816/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:20:21 Step 835/1086: training loss=0.03\n",
      "[OpenAI Provider] 2024-12-30 01:20:46 Step 857/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:21:04 Step 877/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:21:30 Step 901/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:21:47 Step 919/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:22:04 Step 934/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:22:29 Step 957/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:22:47 Step 977/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:23:13 Step 1001/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:23:30 Step 1019/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:23:56 Step 1043/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:24:12 Step 1056/1086: training loss=0.00\n",
      "[OpenAI Provider] 2024-12-30 01:24:39 Step 1085/1086: training loss=0.00\n",
      "[OpenAI Provider] Attempting to retrieve the trained model\n",
      "[OpenAI Provider] Model retrieved: ft:gpt-4o-mini-2024-07-18:personal::AjxtAzey\n",
      "[BootstrapFinetune] Job 1/1 is done\n",
      "[BootstrapFinetune] Updating the student program with the fine-tuned LMs...\n",
      "[BootstrapFinetune] BootstrapFinetune has finished compiling the student program\n"
     ]
    }
   ],
   "source": [
    "from dspy.teleprompt import BootstrapFinetune\n",
    "\n",
    "bsft_optimizer = BootstrapFinetune(\n",
    "    metric=validate_answer,          # Used to filter training data\n",
    "    num_threads=16                   # For parallel processing\n",
    ")\n",
    "\n",
    "bsft_twt_sentiment = bsft_optimizer.compile(\n",
    "    student=student,\n",
    "    trainset=bsft_twitter_train,\n",
    "    teacher=teacher\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "ecf21fde-43c7-4f83-99a5-32aee4cd014a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bootstrap Fine Tune Accuracy:  0.725\n"
     ]
    }
   ],
   "source": [
    "bsft_scores = []\n",
    "for x in bsft_twitter_test:\n",
    "    pred = bsft_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    bsft_scores.append(score)\n",
    "\n",
    "bsft_accuracy = bsft_scores.count(True) / len(bsft_scores)\n",
    "print(\"Bootstrap Fine Tune Accuracy: \", bsft_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "cdbce889-116d-4000-ae0e-03ac488f107f",
   "metadata": {},
   "outputs": [],
   "source": [
    "bsft_twt_sentiment.save(\"./optimized/bsft_twt_sentiment.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "id": "bb535b64-324e-48bd-97af-757cd8e33aaf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(bsft_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19dcfe42-f555-4027-9f37-867e9432e7af",
   "metadata": {},
   "source": [
    "### Choosing an Optimizer\n",
    "\n",
    "From DSPy's [Documentation](https://dspy.ai/learn/optimization/optimizers):\n",
    "\n",
    "- If you have very few examples (around 10), start with `BootstrapFewShot`.\n",
    "- If you have more data (50 examples or more), try `BootstrapFewShotWithRandomSearch`.\n",
    "- If you prefer to do instruction optimization only (i.e. you want to keep your prompt 0-shot), use `MIPROv2` configured for 0-shot optimization to optimize.\n",
    "- If you’re willing to use more inference calls to perform longer optimization runs (e.g. 40 trials or more), and have enough data (e.g. 200 examples or more to prevent overfitting) then try `MIPROv2`.\n",
    "- If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very efficient program, finetune a small LM for your task with `BootstrapFinetune`.\n",
    "\n",
    "Can't choose one? Try the [Ensemble](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/ensemble.py) compiler to combine multiple optimized programs together, then process the output's in some way (i.e. majority, weighted majority, etc) to get to a final output! \n",
    "\n",
    "<img src=\"./media/ensemble_diagram.png\" width=800>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "405d6fa5-8189-47f2-9841-ae76f48abae2",
   "metadata": {},
   "source": [
    "### Optimizing Optimized Programs\n",
    "\n",
    "As emphasized, running just one iteration of optimization is usually not enough. Iterate across your metrics, programs, and metrics in programs!\n",
    "\n",
    "DSPy has a built in function that encourages this, **[BetterTogether](https://github.com/stanfordnlp/dspy/blob/main/dspy/teleprompt/bettertogether.py)**\n",
    "\n",
    "<img src=\"./media/better_together.png\" width=400>\n",
    "\n",
    "But we'll go ahead and do it manually to see if it makes a difference!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "192e8175-d747-4ce7-b6e6-788e87c3001c",
   "metadata": {},
   "source": [
    "**Grabbing Unseen Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "id": "79112317-00eb-4b07-b9b5-e4e74c71be5b",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "import json\n",
    "# Formatting Examples\n",
    "final_twitter_train = []\n",
    "final_twitter_test = []\n",
    "train_size = 300  # how many for train \n",
    "test_size = 500    # how many for test\n",
    "start_row = 1500   # start reading from this row\n",
    "\n",
    "with open(\"./datasets/tweets.jsonl\", 'r', encoding='utf-8') as f:\n",
    "   for i, line in enumerate(f):\n",
    "       # Skip until we reach start_row\n",
    "       if i < start_row:\n",
    "           continue\n",
    "           \n",
    "       # Adjust the index for our collection logic\n",
    "       collection_index = i - start_row\n",
    "       \n",
    "       if collection_index >= (train_size + test_size):\n",
    "           break\n",
    "           \n",
    "       data = json.loads(line.strip())\n",
    "       example = dspy.Example(\n",
    "           tweet=data['text'],\n",
    "           sentiment=data['label_text']\n",
    "       ).with_inputs(\"tweet\")\n",
    "       \n",
    "       if collection_index < train_size:\n",
    "           final_twitter_train.append(example)\n",
    "       else:\n",
    "           final_twitter_test.append(example)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85cdaeb4-a4f9-4947-9e63-b96b716dff0b",
   "metadata": {},
   "source": [
    "**Optimizing our Fine Tuned Program with MIPROv2**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3b9117e-5710-4f97-be8b-e920a7a0e8b3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "mipro_optimizer = MIPROv2(\n",
    "    metric=validate_answer,\n",
    "    prompt_model= dspy.LM('openai/gpt-4o'), # Different Model for Prompt Generation\n",
    "    num_candidates=10,                      # Instructions to try\n",
    ")\n",
    "\n",
    "mipro_bsft_twt_sentiment = mipro_optimizer.compile(bsft_twt_sentiment, trainset=final_twitter_train, valset=final_twitter_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "id": "038447be-c965-4b64-9183-09a46a21802e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MIPROv2 After Bootstrap Fine Tune Accuracy:  0.744\n"
     ]
    }
   ],
   "source": [
    "final_scores = []\n",
    "for x in final_twitter_test:\n",
    "    pred = mipro_bsft_twt_sentiment(**x.inputs())\n",
    "    score = validate_answer(x, pred)\n",
    "    final_scores.append(score)\n",
    "\n",
    "mipro_bsft_accuracy = final_scores.count(True) / len(final_scores)\n",
    "print(\"MIPROv2 After Bootstrap Fine Tune Accuracy: \", mipro_bsft_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "937124eb-a41a-4ade-bc83-7bd63fd919c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "mipro_bsft_twt_sentiment.save(\"./optimized/mipro_bsft_twt_sentiment.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 149,
   "id": "4bffb932-7700-404d-8159-86e65337829c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive\n"
     ]
    }
   ],
   "source": [
    "print(mipro_bsft_twt_sentiment(tweet=example_tweet).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db021148-d0b7-4695-8b5b-51d133ae3f79",
   "metadata": {},
   "source": [
    "---\n",
    "## Final Thoughts"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32bba4b2-c462-407f-a1c5-bd21570738ab",
   "metadata": {},
   "source": [
    "Check out DSPy's [official documentation](https://dspy.ai/), which this notebook is essentially a code forward exploration of. They have plenty more [tutorials](https://dspy.ai/tutorials/) and [guides](https://dspy.ai/learn/) that are actively being updated as part of their latest (Dec 2024) release!\n",
    "\n",
    "Overall DSPy provides an interesting approach to applying language models within programs, abstracting away from trial and error via prompting by adding rigour around clear metric definition and optimization. Rather than work with difficult to interpret or tune text strings, they offer a clean base template that can be further optimized through algorithmic approaches, applying automated ways to coordinate or generate few shot examples, directly change the instructions given to the LLM, or a combination of the two.\n",
    "\n",
    "Inspired by deep learning frameworks, DSPy offers a powerful way to reliably optimize and iterate on LLM applications in a systematic and controlled way, with the entire ecosystem growing by the day. Go give [the DSPy repo](https://github.com/stanfordnlp/dspy/tree/main) a star!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "948f1044-3022-49a5-a5ac-07fd5010eb00",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}