main.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# liaisons-experiments - Large Language Models Benchmarkings for Relation-based Argument Mining\n",
    "\n",
    "This notebook make a evaluation of the Large Language Model landscape ability for micro-scale Relation-based Argument Mining tasks.\n",
    "\n",
    "## About the Task\n",
    "\n",
    "This work is a modest continuation of a previous work (Gorur et al., 2024), limiting the computing cost by highly reducing the size of the dataset.\n",
    "\n",
    "The actual task of this evaluation consists in measuring each model capability to predict the logical relation between two arguments on controversial topics collected from wikipedia (Bar-Haim et al., 2017).  \n",
    "Predicted relation can be of either 2 or 3 classes depending of the relation dimension configuration:\n",
    "- When *binary*, a relation can either be *support* (e.g., \"Arg A logically supports Arg B\") or *attack* (e.g., \"Arg A logically contradicts Arg B\")\n",
    "- When *ternary*, a relation can either be *support* (e.g., \"Arg A logically supports Arg B\"), *attack* (e.g., \"Arg A logically contradicts Arg B\"), or *unrelated* (e.g., \"Arg A is logically irevelant to Arg B\")  \n",
    "  \n",
    "For example, the first argument `ASEAN has subscribed to the notion of democratic peace` `attack` the second argument `This house would disband ASEAN`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from dotenv import load_dotenv\n",
    "\n",
    "# Load the .env file to safely retrieve HuggingFace token for the task dataset,\n",
    "# but also for the platform-hosted LLM\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selected Models\n",
    "\n",
    "Capitalizing on the growing trend of open-source LLMs, this research investigates models like phi3, gemma2, and llama3 that are accessible even to users without specialized hardware.\n",
    "\n",
    "Expanding the evaluation, this work also included larger, platform-hosted models (e.g., gpt-3.5-turbo-0125, gemini-1.5-pro...). Their ease of scaling makes them particularly attractive for further macro-scale argument mining feature development.\n",
    "\n",
    "### Hyperparameters Configuration\n",
    "\n",
    "Following previous hyperparamters search (Gorur et al., 2024), `temperature` and `top_p` have respectively been set to 0.7 and 1 for better results. The `max_tokens` hyperparameter also have been set to a minimum to generate the expected classes (\"support\"/\"attack\"/\"unrelated\") to enable a crucial computation cost cut. However, the minimum value needed differs from the tokenizer used by each models, leading the a variation of this value.\n",
    "\n",
    "### Pipeline Acceleration\n",
    "\n",
    "Taking advantage of platform-hosted models infrastructure, the benchmarking framework propose a multithreading feature, configurable through the `num_workers` parameter. Enabling a critical performance improvement.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from liaisons_experiments.experiments.multi_experiment import MultiExperiment\n",
    "from tqdm.notebook import tqdm\n",
    "from langchain_community.chat_models import ChatOllama\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain_anthropic import ChatAnthropic\n",
    "from langchain_google_genai import GoogleGenerativeAI, HarmCategory, HarmBlockThreshold\n",
    "import os\n",
    "\n",
    "self_hosted_llms = [\n",
    "    (ChatOllama(\n",
    "        model=\"llama3:8b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 5\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"phi3:3.8b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 10\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"phi3:14b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 3\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma:2b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 10\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma:2b-it\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 10\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma:7b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 5\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma:7b-it\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 5\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:2b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 10\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:2b-it\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 10\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:9b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 5\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:9b-it\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 5\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:27b\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 2\n",
    "    }),\n",
    "    (ChatOllama(\n",
    "        model=\"gemma2:27b-it\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "    ), {\n",
    "        \"num_workers\": 2\n",
    "    }),\n",
    "]\n",
    "\n",
    "platform_hosted_llms = [\n",
    "    (ChatOpenAI(\n",
    "        model=\"gpt-3.5-turbo-0125\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_OPENAI_API_KEY\"],\n",
    "    ), {\n",
    "        \"num_workers\": 16,\n",
    "    }),\n",
    "    (ChatOpenAI(\n",
    "        model=\"gpt-4-turbo-2024-04-09\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_OPENAI_API_KEY\"],\n",
    "    ), {\n",
    "        \"num_workers\": 1,\n",
    "    }),\n",
    "    (ChatOpenAI(\n",
    "        model=\"gpt-4o-2024-05-13\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_OPENAI_API_KEY\"],\n",
    "    ), {\n",
    "        \"num_workers\": 1,\n",
    "    }),\n",
    "    (ChatOpenAI(\n",
    "        model=\"gpt-4o-mini-2024-07-18\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=2,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_OPENAI_API_KEY\"],\n",
    "    ), {\n",
    "        \"num_workers\": 16,\n",
    "    }),\n",
    "    (ChatAnthropic(\n",
    "        model=\"claude-3-haiku-20240307\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=3,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_ANTHROPIC_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 2,\n",
    "    }),\n",
    "    (ChatAnthropic(\n",
    "        model=\"claude-3-sonnet-20240229\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=3,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_ANTHROPIC_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 2,\n",
    "    }),\n",
    "    (ChatAnthropic(\n",
    "        model=\"claude-3-opus-20240229\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=3,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_ANTHROPIC_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 1,\n",
    "    }),\n",
    "    (ChatAnthropic(\n",
    "        model=\"claude-3-5-sonnet-20240620\",\n",
    "        temperature=0.7,\n",
    "        max_tokens=3,\n",
    "        top_p=1,\n",
    "        api_key=os.environ[\"LIAISONS_EXPERIMENTS_ANTHROPIC_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 2,\n",
    "    }),\n",
    "    (GoogleGenerativeAI(\n",
    "        model=\"gemini-1.5-flash\",\n",
    "        temperature=0.7,\n",
    "        max_output_tokens=2,\n",
    "        top_p=1,\n",
    "        safety_settings={\n",
    "            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "            HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,\n",
    "        },\n",
    "        google_api_key=os.environ[\"LIAISONS_EXPERIMENTS_GOOGLE_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 16,\n",
    "    }),\n",
    "    (GoogleGenerativeAI(\n",
    "        model=\"gemini-1.5-pro\",\n",
    "        temperature=0.7,\n",
    "        max_output_tokens=2,\n",
    "        top_p=1,\n",
    "        safety_settings={\n",
    "            HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "            HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "            HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,\n",
    "        },\n",
    "        google_api_key=os.environ[\"LIAISONS_EXPERIMENTS_GOOGLE_API_KEY\"],\n",
    "    ),{\n",
    "        \"num_workers\": 5,\n",
    "    }),\n",
    "]\n",
    "\n",
    "exps = MultiExperiment([*self_hosted_llms, *platform_hosted_llms], tqdm=tqdm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "hf_token = os.environ.get(\"LIAISONS_HUGGING_FACE_API_KEY\")\n",
    "\n",
    "dataset = load_dataset(\"coding-kelps/liaisons-claim-stance-sample\", token=hf_token)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "def plot_binary_results(binary_results, title: str | None = None):\n",
    "    binary_plot_results = pd.merge(binary_results.f1_scores, binary_results.metadata) \\\n",
    "        .melt(id_vars='model_name', var_name='Metric', value_name='Value')\n",
    "\n",
    "    # Set the size of the plot\n",
    "    plt.figure(figsize=(14, 8))\n",
    "\n",
    "    # Define a list of colors for the palette\n",
    "    colors = [\"#1F77B4\", \"#FF7F0F\", \"#2BA02B\", \"#D62727\"]\n",
    "\n",
    "    # Create a grouped bar plot\n",
    "    ax = sns.barplot(data=binary_plot_results, x='model_name', y='Value', hue='Metric', palette=colors)\n",
    "\n",
    "    plt.title(title)\n",
    "    plt.xlabel(\"Model Name\")\n",
    "    plt.ylabel(\"Benchmarks\")\n",
    "\n",
    "    # Fix ticks position to avoid hazardous position\n",
    "    # https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.set_xticklabels.html\n",
    "    ax.set_xticks(ax.get_xticks())\n",
    "    # Rotate labels and align to the right\n",
    "    ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n",
    "\n",
    "    # Show the plot\n",
    "    plt.tight_layout()\n",
    "    plt.show()\n",
    "\n",
    "def plot_ternary_results(ternary_results, title: str | None = None):\n",
    "    ternary_plot_results = pd.merge(ternary_results.f1_scores, ternary_results.metadata) \\\n",
    "        .melt(id_vars='model_name', var_name='Metric', value_name='Value')\n",
    "    \n",
    "    # Set the size of the plot\n",
    "    plt.figure(figsize=(14, 8))\n",
    "    \n",
    "    \n",
    "    # Define a list of colors for the palette\n",
    "    colors = [\"#1F77B4\", \"#FF7F0F\", \"#9467BD\", \"#2BA02B\", \"#D62727\"]\n",
    "    \n",
    "    # Create a grouped bar plot\n",
    "    ax = sns.barplot(data=ternary_plot_results, x='model_name', y='Value', hue='Metric', palette=colors)\n",
    "    \n",
    "    plt.title(title)\n",
    "    plt.xlabel(\"Model Name\")\n",
    "    plt.ylabel(\"Benchmarks\")\n",
    "    \n",
    "    # Fix ticks position to avoid hazardous position\n",
    "    # https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.set_xticklabels.html\n",
    "    ax.set_xticks(ax.get_xticks())\n",
    "    # Rotate labels and align to the right\n",
    "    ax.set_xticklabels(ax.get_xticklabels(), rotation=45, ha=\"right\", rotation_mode=\"anchor\")\n",
    "    \n",
    "    # Show the plot\n",
    "    plt.tight_layout()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prompting Techniques\n",
    "\n",
    "This benchmark builds upon previous research by Gorur et al. (2024) utilizing \"few-shot\" prompting. This technique involves providing X examples of desired behavior before presenting the actual prompt.  \n",
    "\n",
    "However, a significant discrepancy emerged between prior results and our findings. To address this gap, we also explored \"augmented few-shot\" prompting, which incorporates an additional instructional line within the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def binary_ibm_few_shot_prompting(parent_argment: str, child_argument: str) -> str:\n",
    "    prompt = f\"\"\"\n",
    "Arg1: Even in the case of provocateurs, it can be an effective strategy to call their bluff, by offering them a chance to have a rational conversation. In this case, the failure to do so is their responsibility alone.\n",
    "Arg2: No-platforming hinders productive discourse.\n",
    "Relation: attack\n",
    "\n",
    "Arg1: A country used to receiving ODA may be perpetually bound to depend on handouts (pp. 197).\n",
    "Arg2: Government structures adapt to handle and distribute incoming ODA. As the funding from ODA is significant, countries have vested bureaucratic interest to remain bound to aid (pp. 197).\n",
    "Relation: support\n",
    "\n",
    "Arg1: Elections would limit the influence of lobbyists on the appointment of Supreme Court judges.\n",
    "Arg2: The more individuals take part in a decision, as would be the case in a popular vote compared to a vote in the Senate, the harder it is to sway the outcome.\n",
    "Relation: support\n",
    "\n",
    "Arg1: ChatGPT will reach AGI level before 2030.\n",
    "Arg2: To reach AGI it should be able to generate its own goals and intentions: where would it draw these from?\n",
    "Relation: attack\n",
    "\n",
    "Arg1: {parent_argment}\n",
    "Arg2: {child_argument}\n",
    "Relation: \n",
    "\"\"\"\n",
    "    \n",
    "    return prompt\n",
    "\n",
    "def binary_ibm_augmented_few_shot_prompting(parent_argment: str, child_argument: str) -> str:\n",
    "    prompt = f\"\"\"\n",
    "Arg1: Even in the case of provocateurs, it can be an effective strategy to call their bluff, by offering them a chance to have a rational conversation. In this case, the failure to do so is their responsibility alone.\n",
    "Arg2: No-platforming hinders productive discourse.\n",
    "Relation: attack\n",
    "\n",
    "Arg1: A country used to receiving ODA may be perpetually bound to depend on handouts (pp. 197).\n",
    "Arg2: Government structures adapt to handle and distribute incoming ODA. As the funding from ODA is significant, countries have vested bureaucratic interest to remain bound to aid (pp. 197).\n",
    "Relation: support\n",
    "\n",
    "Arg1: Elections would limit the influence of lobbyists on the appointment of Supreme Court judges.\n",
    "Arg2: The more individuals take part in a decision, as would be the case in a popular vote compared to a vote in the Senate, the harder it is to sway the outcome.\n",
    "Relation: support\n",
    "\n",
    "Arg1: ChatGPT will reach AGI level before 2030.\n",
    "Arg2: To reach AGI it should be able to generate its own goals and intentions: where would it draw these from?\n",
    "Relation: attack\n",
    "\n",
    "---\n",
    "\n",
    "What the relation between Arg1 and Arg2, respond with one word: support or attack:\n",
    "\n",
    "Arg1: {parent_argment}\n",
    "Arg2: {child_argument}\n",
    "Relation: \n",
    "\"\"\"\n",
    "    \n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "binary_df = dataset['binary'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "few_shot_binary_results = exps.run_from_df(binary_df, binary_ibm_few_shot_prompting, relation_dim=\"binary\")\n",
    "\n",
    "plot_binary_results(few_shot_binary_results, title=\"Large Language Models for binary argumentative relation prediction over the IBM Debater preprocessed dataset sample using few shot prompting\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "augmented_few_shot_binary_results = exps.run_from_df(binary_df, binary_ibm_augmented_few_shot_prompting, relation_dim=\"binary\")\n",
    "\n",
    "plot_binary_results(augmented_few_shot_binary_results, title=\"Large Language Models for binary argumentative relation prediction over the IBM Debater preprocessed dataset sample using few augmented shot prompting\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ternary_ibm_few_shot_prompting(parent_argment: str, child_argument: str) -> str:\n",
    "    prompt = f\"\"\"\n",
    "Arg1: Even in the case of provocateurs, it can be an effective strategy to call their bluff, by offering them a chance to have a rational conversation. In this case, the failure to do so is their responsibility alone.\n",
    "Arg2: No-platforming hinders productive discourse.\n",
    "Relation: attack\n",
    "\n",
    "Arg1: ChatGPT will reach AGI level before 2030.\n",
    "Arg2: Government structures adapt to handle and distribute incoming ODA. As the funding from ODA is significant, countries have vested bureaucratic interest to remain bound to aid (pp. 197).\n",
    "Relation: unrelated\n",
    "\n",
    "Arg1: Elections would limit the influence of lobbyists on the appointment of Supreme Court judges.\n",
    "Arg2: The more individuals take part in a decision, as would be the case in a popular vote compared to a vote in the Senate, the harder it is to sway the outcome.\n",
    "Relation: support\n",
    "\n",
    "Arg1: A country used to receiving ODA may be perpetually bound to depend on handouts (pp. 197).\n",
    "Arg2: To reach AGI it should be able to generate its own goals and intentions: where would it draw these from?\n",
    "Relation: unrelated\n",
    "\n",
    "Arg1: {parent_argment}\n",
    "Arg2: {child_argument}\n",
    "Relation: \n",
    "\"\"\"\n",
    "    \n",
    "    return prompt\n",
    "\n",
    "def ternary_ibm_augmented_few_shot_prompting(parent_argment: str, child_argument: str) -> str:\n",
    "    prompt = f\"\"\"\n",
    "Arg1: Even in the case of provocateurs, it can be an effective strategy to call their bluff, by offering them a chance to have a rational conversation. In this case, the failure to do so is their responsibility alone.\n",
    "Arg2: No-platforming hinders productive discourse.\n",
    "Relation: attack\n",
    "\n",
    "Arg1: ChatGPT will reach AGI level before 2030.\n",
    "Arg2: Government structures adapt to handle and distribute incoming ODA. As the funding from ODA is significant, countries have vested bureaucratic interest to remain bound to aid (pp. 197).\n",
    "Relation: unrelated\n",
    "\n",
    "Arg1: Elections would limit the influence of lobbyists on the appointment of Supreme Court judges.\n",
    "Arg2: The more individuals take part in a decision, as would be the case in a popular vote compared to a vote in the Senate, the harder it is to sway the outcome.\n",
    "Relation: support\n",
    "\n",
    "Arg1: A country used to receiving ODA may be perpetually bound to depend on handouts (pp. 197).\n",
    "Arg2: To reach AGI it should be able to generate its own goals and intentions: where would it draw these from?\n",
    "Relation: unrelated\n",
    "\n",
    "---\n",
    "\n",
    "What the relation between Arg1 and Arg2, respond with one word: support, attack, or unrelated:\n",
    "\n",
    "Arg1: {parent_argment}\n",
    "Arg2: {child_argument}\n",
    "Relation: \n",
    "\"\"\"\n",
    "    \n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ternary_df = dataset['ternary'].to_pandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "few_shot_ternary_results = exps.run_from_df(ternary_df, ternary_ibm_few_shot_prompting, relation_dim=\"ternary\")\n",
    "\n",
    "plot_ternary_results(few_shot_ternary_results, title=\"Large Language Models for ternary argumentative relation prediction over the IBM Debater preprocessed dataset sample using few shot prompting\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "augmented_few_shot_ternary_results = exps.run_from_df(ternary_df, ternary_ibm_augmented_few_shot_prompting, relation_dim=\"ternary\")\n",
    "\n",
    "plot_ternary_results(augmented_few_shot_ternary_results, title=\"Large Language Models for ternary argumentative relation prediction over the IBM Debater preprocessed dataset sample using augmented few shot prompting\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bibliography\n",
    "- Bar-Haim, R., Bhattacharya, I., Dinuzzo, F., Saha, A., and Slonim, N. (2017). Stance classification of context dependent claims. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 251–261.\n",
    "- Gorur, D., Rago, A. and Toni, F. (2024). Can Large Language Models perform Relation-based Argument Mining? [online] arXiv.org. doi:https://doi.org/10.48550/arXiv.2402.11243."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}