add batch speedup evaluation to fastdraft_deepseek notebook (#2734)

Added a section to fastdraft_deepseek notebook to evaluate the speedup
of deepseek with speculative decoding over multiple examples

Author: shira-g

supplementary_materials/notebooks/fastdraft-deepseek/fastdraft_deepseek.ipynb

@@ -309,6 +309,137 @@
     "print(f\"End to end speedup with FastDraft and speculative decoding is {ar_gen_time / sd_gen_time:.2f}x\")"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Evaluate Speculative Decoding Speedup On Multiple Examples\n",
+    "\n",
+    "In this section we compare auto-regressive generation and speculative-decoding generation with DeepSeek-R1-Distill-Llama-8B model on multiple examples. \n",
+    "We use 40 example-prompts taken from [MT-Bench](https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts) and from [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) datasets.\n",
+    "We loop over these examples and measure generation times, first without speculative-decoding and later with speculative-decoding. Eventually we compare generation times for both methods and compute the average speedup gain."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 1. Run target model without speculative decoding\n",
+    "As in previous section, we will first run generation without speculative-decoding, but this time we will run it over 40 examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import openvino_genai as ov_genai\n",
+    "import sys\n",
+    "import time\n",
+    "from tqdm import tqdm\n",
+    "from llm_pipeline_with_hf_tokenizer import LLMPipelineWithHFTokenizer\n",
+    "\n",
+    "print(f\"Loading model from {model_dir}\")\n",
+    "\n",
+    "# Define scheduler\n",
+    "scheduler_config = ov_genai.SchedulerConfig()\n",
+    "scheduler_config.num_kv_blocks = 2048 // 16\n",
+    "scheduler_config.dynamic_split_fuse = False\n",
+    "scheduler_config.max_num_batched_tokens = 2048\n",
+    "\n",
+    "pipe = LLMPipelineWithHFTokenizer(model_dir, device, scheduler_config=scheduler_config)\n",
+    "\n",
+    "generation_config = ov_genai.GenerationConfig()\n",
+    "generation_config.max_new_tokens = 1024\n",
+    "\n",
+    "print(\"Loading prompts...\")\n",
+    "import json\n",
+    "f= open('prompts.json')\n",
+    "prompts = json.load(f)\n",
+    "prompts = [[{\"role\": \"user\", \"content\": p }] for p in prompts]\n",
+    "\n",
+    "times_auto_regressive = []\n",
+    "for prompt in tqdm(prompts):\n",
+    "    start_time = time.perf_counter()\n",
+    "    result = pipe.generate(prompt, generation_config, apply_chat_template=True)\n",
+    "    end_time = time.perf_counter()\n",
+    "    times_auto_regressive.append(end_time - start_time)\n",
+    "print(\"Done\")\n",
+    "\n",
+    "import gc\n",
+    "\n",
+    "del pipe\n",
+    "gc.collect()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 2. Run target model with speculative decoding\n",
+    "Now we will run generation with speculative-decoding over the same 40 examples."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print(f\"Loading draft from {draft_model_path}\")\n",
+    "\n",
+    "# Define scheduler for the draft\n",
+    "\n",
+    "draft_scheduler_config = ov_genai.SchedulerConfig()\n",
+    "draft_scheduler_config.num_kv_blocks = 2048 // 16\n",
+    "draft_scheduler_config.dynamic_split_fuse = False\n",
+    "draft_scheduler_config.max_num_batched_tokens = 2048\n",
+    "\n",
+    "draft_model = ov_genai.draft_model(draft_model_path, device, scheduler_config=draft_scheduler_config)\n",
+    "\n",
+    "pipe = LLMPipelineWithHFTokenizer(model_dir, device, scheduler_config=scheduler_config, draft_model=draft_model)\n",
+    "\n",
+    "\n",
+    "generation_config = ov_genai.GenerationConfig()\n",
+    "generation_config.num_assistant_tokens = 3\n",
+    "generation_config.max_new_tokens = 2048\n",
+    "\n",
+    "times_speculative_decoding = []\n",
+    "\n",
+    "print(\"Running Speculative Decoding generation...\")\n",
+    "for prompt in tqdm(prompts):\n",
+    "    start_time = time.perf_counter()\n",
+    "    result = pipe.generate(prompt, generation_config, apply_chat_template=True)\n",
+    "    end_time = time.perf_counter()\n",
+    "    times_speculative_decoding.append((end_time - start_time))\n",
+    "print(\"Done\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### 3. Calculate speedup\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "avg_speedup = sum([x / y for x, y in zip(times_auto_regressive, times_speculative_decoding)]) / len(prompts)\n",
+    "print(f\"average speedup: {avg_speedup:.2f}\")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We see that by using speculative-decoding with FastDraft we can accelerate DeepSeek-R1-Distill-Llama-8B generation by ~1.5x on avarage."
+   ]
+  },
   {
    "attachments": {},
    "cell_type": "markdown",

supplementary_materials/notebooks/fastdraft-deepseek/prompts.json

@@ -0,0 +1,44 @@
+[
+    "which word does not belong to the other: Hammer, Screwdriver, Nail, Wood",
+    "Each problem consists of three statements. Based on the first two statements, the third statement may be true, false, or uncertain.\n1. Oranges cost more than apples.\n2. Oranges cost less than bananas.\n3. Bananas cost more than apples and bananas cost more than orange.\nIf the first two statements are true, then the third statement is",
+    "Parents have complained to the principal about bullying during recess. The principal wants to quickly resolve this, instructing recess aides to be vigilant. Which situation should the aides report to the principal?\na) An unengaged girl is sitting alone on a bench, engrossed in a book and showing no interaction with her peers.\nb) Two boys engaged in a one-on-one basketball game are involved in a heated argument regarding the last scored basket.\nc) A group of four girls has surrounded another girl and appears to have taken possession of her backpack.\nd) Three boys are huddled over a handheld video game, which is against the rules and not permitted on school grounds.",
+    "Which number is bigger 9.11 or 9.9?",
+    "A tech startup invests $8000 in software development in the first year, and then invests half of that amount in software development in the second year.\nWhat's the total amount the startup invested in software development over the two years?",
+    "In a survey conducted at a local high school, preferences for a new school color were measured: 58% of students liked the color blue, 45% preferred green, and 22% liked both colors. If we randomly pick a student from the school, what's the probability that they would like neither blue nor green?",
+    "Lily has a rubber ball that she drops from the top of a wall. the wall is 2 meters tall. how long will it take for the ball to reach the ground?",
+    "Draft a professional email seeking your supervisor's feedback on the 'Quarterly Financial Report' you prepared. Ask specifically about the data analysis, presentation style, and the clarity of conclusions drawn. Keep the email short and to the point.",
+    "Suppose you are a mathematician and poet. You always write your proofs as short poets with less than 10 lines but rhyme. Prove the square root of 2 is irrational number.",
+    "Extract the following information from the presented texts: The name of the book, the author, the main character, the year of publication. Output in the format of \"main character, book, author, year of publication\", one book per line.\na) In the realm of wizarding literature, a true standout is the work of J.K. Rowling. One of her books that left an indelible mark is 'Harry Potter and the Philosopher's Stone'. This iconic tale, published in 1997, tells the story of Harry, a young orphan who discovers his magical abilities on his 11th birthday. Soon, he finds himself at the Hogwarts School of Witchcraft and Wizardry, a place teeming with magic and adventure, located somewhere in Scotland.\nb) The magic of Middle-earth has entranced readers worldwide, thanks to the brilliance of J.R.R. Tolkien. In one of his seminal works, 'The Lord of the Rings: The Fellowship of the Ring', published in 1954, we meet Frodo Baggins, a brave hobbit tasked with the perilous quest of destroying the One Ring. The epic journey takes him from the peaceful Shire to the tumultuous regions of Middle-earth.\nc) In a galaxy far, far away, the imagination of L.E. Starlighter gives us 'The Prism Galaxy Chronicles: The Awakening of the Starcaster'. Published in 2028, the story is about Zylo, a humble spaceship mechanic, who unexpectedly discovers he's a Starcaster - a rare individual with the power to manipulate stardust. Set against the backdrop of an interstellar empire in turmoil, Zylo's destiny unfolds on numerous alien worlds, each with its unique cosmic charm.",
+    "Given the following data, identify the company with the highest profit in 2021 and provide its CEO's name:\na) Company X, with CEO Amy Williams, reported $30 billion in revenue and a $3 billion profit in 2021.\nb) Company Y, led by CEO Mark Thompson, posted a $60 billion revenue and a $6 billion profit in the same year.\nc) Company Z, under CEO Sarah Johnson, announced a $20 billion revenue and a $7 billion profit in 2021.\nd) Company W, managed by CEO James Smith, revealed a $300 billion revenue with a $21 billion profit in 2021.\ne) Company V, with CEO Lisa Brown, reported a $200 billion revenue and a $25 billion profit in 2021.\nf) Company U, under CEO John White, posted a $180 billion revenue and a $20 billion profit in the same year.",
+    "Which is a species of fish? Tope or Rope",
+    "Identify which instrument is string or percussion: Cantaro, Gudok",
+    "Identify which instrument is string or woodwind: Panduri, Zurna",
+    "Identify which instrument is string or percussion: Kpanlogo, Shamisen",
+    "Which of these are rappers? Eminem, Michael Jackson, Rihanna, 50 Cent",
+    "Classify each of the following pieces of equipment according to the sport they are used for, either basketball, football, or soccer: shin guards, shooting sleeve, penalty flag, corner flag, kicking tee, and goalie gloves.",
+    "Identify which animal species is alive or extinct: Palaeophis, Giant Tortoise",
+    "If we were playing a game where we had to identify things that can be found inside a house, which of these would we call out: car, chair, table, park, cloud, microwave.",
+    "Classify the following as either dark-colored beers or light colored beers: porter, pilsner, stout, amber, lager",
+    "Categorize each of the following instruments as either string or keyboard: Guitar, Violin, piano, harmonium, cello, accordion, banjo",
+    "What are common ingredients of a full english breakfast?",
+    "Classify each of the following as either a bird, animal, reptile or insect: tiger, heron, eagle, alligator, snake, spider, ant, dog, cat, rhinoceros, kingfisher, chameleon, hornet, butterfly",
+    "Which is a species of fish? Sea dragon or Red bearded",
+    "Categorize where each of these household items belong: bed, couch, desk",
+    "Separate the following suburbs into those that border, and do not border, the Brisbane River: Indooroopilly, Bulimba, St Lucia, Newstead, Wilston, West End, Toowong, Bowen Hills and, Wooloongabba.",
+    "Choose the word which is different from the rest: hangar, platform, dock, park, bus stand",
+    "Identify which instrument is string or percussion: Den-den daiko, Luc huyen cam",
+    "Pick if these would be useful or not useful for a high school student to put in their backpack. Notebooks, textbook, desk lamp, pencil pouch, beach ball, pillow, laptop.",
+    "Classify the following numbers as 'prime' or 'composite' - 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16.",
+    "Classify the below characters based on whether they are created by Marvel comics or D.C. comics\n Spider-Man, Iron Man, Captain America, Thor, Doctor Strange, Superman, Batman, Wonder Woman, Flash, Aquaman",
+    "Which of the following are colors: red, black, yellow, orange, sun, sunflower, chips, book, white, pink, blue, keyboard.",
+    "Which sports would be easiest to find success in if you’re not tall: baseball, soccer, basketball, bowling.",
+    "What are the classifications of Academic Degrees?",
+    "I'm creating a class to explain to kids the difference between items that are powered and can be plugged in and items that aren't. Please divide the following things into those you can plug in and those that you can't: table, hairdryer, television, chair, computer, fridge, comb, flowers.",
+    "Identify which instrument is string or woodwind: Wheelharp, Clarinet",
+    "Tell me whether these cities are in Texas: Austin, Houston, New York, Chicago, Miami, Dallas",
+    "Classify the following types of cars as \"economy\" or \"luxury\": Ford, Chevrolet, Lamborghini, Ferrari, Mercedes, Honda, Lexus, Toyota, Nissan, Subaru",
+    "Classify each of the following grades as being in elementary or high school: 10th grade, 3rd grade, 4th grade, 12th grade, 1st grade.",
+    "Which of the following are ice cream toppings and which are salad dressings: thousand island, chocolate sauce, hot fudge, balsamic vinaigrette, whipped cream, and Caesar."
+]
+
+