11_Multi_Class_Classification.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "11 Multi Class Classification.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPWNOFvn1SCWX1AdS9Jn8me",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/sandipanpaul21/Logistic-regression-in-python/blob/main/11_Multi_Class_Classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Binary Classifiers for Multi-Class Classification**\n",
        "\n",
        "- Classification is a predictive modeling problem that involves assigning a class label to an example.\n",
        "- **Binary classification** are those tasks where examples are assigned exactly one of two classes. \n",
        "- **Binary Classification:** Classification tasks with two classes.\n",
        "- **Multi-class classification** is those tasks where examples are assigned exactly one of more than two classes.\n",
        "- **Multi-class Classification:** Classification tasks with more than two classes.\n",
        "\n",
        "*Some algorithms are designed for binary classification problems. Examples include:*\n",
        "\n",
        "1. Logistic Regression\n",
        "2. Support Vector Machines\n",
        "\n",
        "\n",
        "Instead, heuristic methods can be used to split a multi-class classification problem into multiple binary classification datasets and train a binary classification model each.\n",
        "\n",
        "Two examples of these heuristic methods include:\n",
        "\n",
        "1. One-vs-Rest (OvR)\n",
        "2. One-vs-One (OvO)"
      ],
      "metadata": {
        "id": "VyVM3wn3FDmZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-Vs-Rest (OvR) for Multi-Class Classification**\n",
        "\n",
        "- **One-vs-rest (OvR)** for short, also referred to as **One-vs-All (OvA)** is a heuristic method for using binary classification algorithms for multi-class classification.\n",
        "\n",
        "- It involves splitting the multi-class dataset into multiple binary classification problems. \n",
        "- A binary classifier is then trained on each binary classification problem and predictions are made using the model that is the most confident.\n",
        "\n",
        "- For example, \n",
        "  - Given a multi-class classification problem with examples for each class ‘red,’ ‘blue,’ and ‘green‘. \n",
        "  - This could be divided into three binary classification datasets as follows:\n",
        "    1. Binary Classification Problem 1: red vs [blue, green]\n",
        "    2. Binary Classification Problem 2: blue vs [red, green]\n",
        "    3. Binary Classification Problem 3: green vs [red, blue]\n",
        "\n",
        "**Possible Issue**\n",
        "- A possible downside of this approach is that it requires one model to be created for each class. \n",
        "- For example, three classes requires three models. \n",
        "  - This could be an issue for large datasets (e.g. millions of rows) \n",
        "  - Very large numbers of classes (e.g. hundreds of classes).\n",
        "\n",
        "**Approach of OvA or OvR**\n",
        "- The obvious approach is to use a one-versus-the-rest approach (also called one-vs-all), in which we train C binary classifiers, fc(x), where the data from class c is treated as positive, and the data from all the other classes is treated as negative.\n",
        "- This approach requires that each model predicts a class membership probability or a probability-like score. The argmax of these scores (class index with the largest score) is then used to predict a class.\n",
        "- This approach is commonly used for algorithms that naturally predict numerical class membership probability or score, such as: Logistic Regression. \n",
        "As such, the implementation of these algorithms in the scikit-learn library implements the OvR strategy by default when using these algorithms for multi-class classification.\n",
        "\n",
        "**Python Example**\n",
        "- We can demonstrate this with an example on a 3-class classification problem using the LogisticRegression algorithm. \n",
        "- The strategy for handling multi-class classification can be set via the “multi_class” argument and can be set to “ovr” for the one-vs-rest strategy."
      ],
      "metadata": {
        "id": "LNYaXKSJFjUq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# logistic regression for multi-class classification using built-in one-vs-rest\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "\n",
        "# define dataset\n",
        "X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\n",
        "\n",
        "# define model\n",
        "model = LogisticRegression(multi_class='ovr')\n",
        "\n",
        "# fit model\n",
        "model.fit(X, y)\n",
        "\n",
        "# make predictions\n",
        "yhat = model.predict(X)"
      ],
      "metadata": {
        "id": "LoAiNMAzFjEX"
      },
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- The scikit-learn library also provides a separate OneVsRestClassifier class that allows the one-vs-rest strategy to be used with any classifier.\n",
        "\n",
        "- This class can be used to use a binary classifier like Logistic Regression or Perceptron for multi-class classification, or even other classifiers that natively support multi-class classification."
      ],
      "metadata": {
        "id": "VoM4ch_1FqKC"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "AcL956kJBPes"
      },
      "outputs": [],
      "source": [
        "# logistic regression for multi-class classification using a one-vs-rest\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.multiclass import OneVsRestClassifier\n",
        "\n",
        "# define dataset\n",
        "X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\n",
        "\n",
        "# define model\n",
        "model = LogisticRegression()\n",
        "\n",
        "# define the ovr strategy\n",
        "ovr = OneVsRestClassifier(model)\n",
        "\n",
        "# fit model\n",
        "ovr.fit(X, y)\n",
        "\n",
        "# make predictions\n",
        "yhat = ovr.predict(X)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**One-Vs-One for Multi-Class Classification**\n",
        "\n",
        "- One-vs-One (OvO for short) is another heuristic method for using binary classification algorithms for multi-class classification.\n",
        "\n",
        "- Like one-vs-rest, one-vs-one splits a multi-class classification dataset into binary classification problems. \n",
        "\n",
        "- Unlike one-vs-rest that splits it into one binary dataset for each class, the one-vs-one approach splits the dataset into one dataset for each class versus every other class.\n",
        "- For example\n",
        "  - Consider a multi-class classification problem with four classes: ‘red,’ ‘blue,’ and ‘green,’ ‘yellow.’ \n",
        "  - This could be divided into six binary classification datasets as follows:\n",
        "    \n",
        "    Binary Classification Problem 1: red vs. blue\n",
        "\n",
        "    Binary Classification Problem 2: red vs. green\n",
        "\n",
        "    Binary Classification Problem 3: red vs. yellow\n",
        "\n",
        "    Binary Classification Problem 4: blue vs. green\n",
        "\n",
        "    Binary Classification Problem 5: blue vs. yellow\n",
        "\n",
        "    Binary Classification Problem 6: green vs. yellow\n",
        "\n",
        "-  The formula for calculating the number of binary datasets, and in turn, models, is as follows:\n",
        "\n",
        "    (NumClasses * (NumClasses – 1)) / 2\n",
        "\n",
        "- We can see that for four classes, this gives us the expected value of six binary classification problems:\n",
        "\n",
        "  (NumClasses * (NumClasses – 1)) / 2\n",
        "  \n",
        "  (4 * (4 – 1)) / 2\n",
        "  \n",
        "  (4 * 3) / 2\n",
        "  \n",
        "  12 / 2\n",
        "  \n",
        "  6\n",
        "\n",
        "Each binary classification model may predict one class label and the model with the most predictions or votes is predicted by the one-vs-one strategy.\n",
        "\n",
        "- An alternative is to introduce K(K − 1)/2 binary discriminant functions, one for every possible pair of classes. \n",
        "- This is known as a **one-versus-one classifier**. \n",
        "- Each point is then classified according to a *majority vote amongst* the discriminant functions.\n",
        "- Similarly, if the binary classification models predict a numerical class membership, such as a *probability, then the argmax of the sum of the scores (class with the largest sum score) is predicted as the class label.*\n",
        "\n",
        "The support vector machine implementation in the scikit-learn is provided by the SVC class and supports the one-vs-one method for multi-class classification problems. This can be achieved by setting the “decision_function_shape” argument to ‘ovo‘."
      ],
      "metadata": {
        "id": "KrhI1hBuFqaP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# SVM for multi-class classification using built-in one-vs-one\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.svm import SVC\n",
        "\n",
        "# define dataset\n",
        "X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\n",
        "\n",
        "# define model\n",
        "model = SVC(decision_function_shape='ovo')\n",
        "\n",
        "# fit model\n",
        "model.fit(X, y)\n",
        "\n",
        "# make predictions\n",
        "yhat = model.predict(X)"
      ],
      "metadata": {
        "id": "Avulnt8zL071"
      },
      "execution_count": 3,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "- The scikit-learn library also provides a separate OneVsOneClassifier class that allows the one-vs-one strategy to be used with any classifier.\n",
        "\n",
        "- This class can be used with a binary classifier like SVM, Logistic Regression or Perceptron for multi-class classification, or even other classifiers that natively support multi-class classification.\n",
        "\n",
        "- It is very easy to use and requires that a classifier that is to be used for binary classification be provided to the OneVsOneClassifier as an argument."
      ],
      "metadata": {
        "id": "SraJXdoKL0h3"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# SVM for multi-class classification using one-vs-one\n",
        "from sklearn.datasets import make_classification\n",
        "from sklearn.svm import SVC\n",
        "from sklearn.multiclass import OneVsOneClassifier\n",
        "\n",
        "# define dataset\n",
        "X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, n_redundant=5, n_classes=3, random_state=1)\n",
        "\n",
        "# define model\n",
        "model = SVC()\n",
        "\n",
        "# define ovo strategy\n",
        "ovo = OneVsOneClassifier(model)\n",
        "\n",
        "# fit model\n",
        "ovo.fit(X, y)\n",
        "\n",
        "# make predictions\n",
        "yhat = ovo.predict(X)"
      ],
      "metadata": {
        "id": "DRGfJE3nL15l"
      },
      "execution_count": 4,
      "outputs": []
    }
  ]
}