Histology_tissue_classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/GabrielCabas/Summer_Course_ML/blob/main/Histology_tissue_classification.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UjCaOERsaGvL"
      },
      "source": [
        "Let's download data as a first step to win time 😀"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JtOYEnmyVyiW"
      },
      "outputs": [],
      "source": [
        "import kagglehub\n",
        "CRC_FOLDER = imrankhan77_crc_val_he_7k_path = kagglehub.dataset_download('imrankhan77/crc-val-he-7k')\n",
        "print(CRC_FOLDER)\n",
        "NCT_FOLDER = imrankhan77_nct_crc_he_100k_path = kagglehub.dataset_download('imrankhan77/nct-crc-he-100k')\n",
        "print(NCT_FOLDER)\n",
        "\n",
        "print('Data source import complete.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-JiFoOEMRtXd"
      },
      "source": [
        "# Histology tissue classification using Deep Learning models\n",
        "\n",
        "The main purpose of this practical notebook is to apply a deep learning model to solve a basic tissue classficiation task in histology images."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ckliCRoeTjJg"
      },
      "source": [
        "For this session, we will use two colon adenocarcinoma datasets:  [nct-crc-he-100k](https://www.kaggle.com/datasets/imrankhan77/nct-crc-he-100k) and [crc-val-he-7k](https://www.kaggle.com/datasets/imrankhan77/crc-val-he-7k). These two datasets contain tissue tiles classified with nine labels:\n",
        "\n",
        "*   ADI: Adipose\n",
        "*   STR: Strome\n",
        "*   TUM: Tumour\n",
        "*   NORM: Normal tissue\n",
        "*   BACK: Background\n",
        "*   LYM: Lymphome\n",
        "*   DEB: Debris\n",
        "*   MUC: Mucose\n",
        "*   MUS: Musculus\n",
        "\n",
        "Each dataset has 100K and 7K tiles images and are mutually excluyent between them. i.e, There aren't any slide repeated between NCT and CRC\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pEC_cBxzUhZS"
      },
      "source": [
        "In addition, we will use a foundational model for histopathology as feature extractor: UNI *(Chen, 2024)*. You can check for more information about it on [Huggingface](https://huggingface.co/MahmoodLab/UNI) 🤗 and the [arcticle](https://www.nature.com/articles/s41591-024-02857-3)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mHYLk8z9VhiG"
      },
      "source": [
        "We will apply some Machine Learning algorithms using the extracted features, and then, select and optimize for our final prediction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ID_ncabxVv82"
      },
      "source": [
        "LET'S GO"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WS2KYweKVxqf"
      },
      "source": [
        "## Dependences, tokens, data and GPUs\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cksptU8HW-ub"
      },
      "source": [
        "First of all, let's install [Optuna](https://optuna.org/). We will use it later for model optimization. All other packages and libraries are included in google collab base environment, so we don't need to install it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": true,
        "id": "N575a9qdpFUR"
      },
      "outputs": [],
      "source": [
        "!pip install optuna"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YhaitKzlXIij"
      },
      "source": [
        "We will disable warnings for to simplify the visualization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "H5zV8DJOr40q"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMCFfLXQXY8D"
      },
      "source": [
        "Google collab provides free but limited access to a NVIDIA T4 (suficient for this demo). For an unlimited use of NVIDIA P40 on cloud, you can this Kaggle [notebook](https://www.kaggle.com/code/gabrielcabas/histology-tissue-classification).  Kaggle provides two NVIDIA T4 and one  NVIDIA P100 . Otherwise, you can always download this notebook and use it on your laptop or adapt it to use on an HPC."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lj3oXOQ9azDJ"
      },
      "source": [
        "Our last dependence is the permission of the authors of UNI to use this model. **Only for this time** we will use my token as a group. If you decide to use it further please request access via Huggingface.\n",
        "\n",
        "Use Secrets to insert and import the token\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "k0J5uEIEVyiY"
      },
      "outputs": [],
      "source": [
        "#Token\n",
        "from huggingface_hub import login\n",
        "from google.colab import userdata\n",
        "token = userdata.get('huggingface_token')\n",
        "login(token = token)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m5ngNv5rb6wt"
      },
      "source": [
        "And import the necessary libraries"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "XSFghM1IVyiX"
      },
      "outputs": [],
      "source": [
        "import numpy as np # linear algebra\n",
        "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
        "import os #Operative system calls and files\n",
        "import matplotlib.pyplot as plt #Data visualization\n",
        "import seaborn as sns #Wrapper for plt\n",
        "from PIL import Image #Image open\n",
        "import tqdm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "g6eYXZbLVyiY"
      },
      "outputs": [],
      "source": [
        "#ML dependences\n",
        "import torch\n",
        "import torchvision\n",
        "from torch.utils.data import DataLoader\n",
        "import timm\n",
        "from timm.data import resolve_data_config\n",
        "from timm.data.transforms_factory import create_transform\n",
        "from torchvision import transforms\n",
        "from sklearn.decomposition import PCA\n",
        "from sklearn.ensemble import RandomForestClassifier\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.metrics import confusion_matrix\n",
        "import optuna"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Cn-_Be5QcE28"
      },
      "source": [
        "This is just a function for to clear the GPU and free memory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "znvNnkzwVyiY"
      },
      "outputs": [],
      "source": [
        "def clear_gpu(): #Function to clear cache in emergency case D:\n",
        "    torch.cuda.empty_cache()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "JEHLPi_1VyiZ"
      },
      "outputs": [],
      "source": [
        "clear_gpu()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mN1STjnRcGWn"
      },
      "source": [
        "And because everyone loves gg-plot, we will import this styles"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "VbvKUrgQVyiZ"
      },
      "outputs": [],
      "source": [
        "plt.style.use('ggplot')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sW7jXs5cVyia"
      },
      "source": [
        "## Exploratory data analysis"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "atArjS7hVyiZ"
      },
      "outputs": [],
      "source": [
        "NCT_FOLDER = f\"{NCT_FOLDER}/NCT-CRC-HE-100K\"\n",
        "CRC_FOLDER = f\"{CRC_FOLDER}/CRC-VAL-HE-7K\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UdcdCi_zct28"
      },
      "source": [
        " This function only list files in main folders and stores it on a dataframe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "Dat8e9bxVyib"
      },
      "outputs": [],
      "source": [
        "def list_files(dataset_folder):\n",
        "    data = []\n",
        "    for folder, subdir, file in os.walk(dataset_folder):\n",
        "        label = os.path.basename(folder)\n",
        "        if file != []:\n",
        "            tiles = pd.DataFrame(file, columns=[\"tile\"])\n",
        "            tiles[\"label\"] = label\n",
        "            data.append(tiles)\n",
        "    data = pd.concat(data)\n",
        "    return data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YwwoxCrRc1qX"
      },
      "source": [
        "We will use NCT dataset for training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jj35m35lVyic"
      },
      "outputs": [],
      "source": [
        "nct_df = list_files(NCT_FOLDER)\n",
        "nct_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9lD0utEXc9xu"
      },
      "source": [
        " And CRC for testing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "r0xPQWoQVyic"
      },
      "outputs": [],
      "source": [
        "crc_df = list_files(CRC_FOLDER)\n",
        "crc_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KNEjLuFRdqB7"
      },
      "source": [
        "Let's show the labels distribution over tiles"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BGEI-1ppVyic"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(15,5))\n",
        "plt.subplot(1,2,1)\n",
        "plt.title(\"NCT-CRC-HE-100k\")\n",
        "sns.histplot(nct_df, hue=\"label\", x=\"label\")\n",
        "plt.subplot(1,2,2)\n",
        "plt.title(\"CRC-VAL-HE-100k\")\n",
        "sns.histplot(crc_df, hue=\"label\", x=\"label\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D-RHWRy7hfbd"
      },
      "source": [
        "This function show some random tiles per label in each dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "KGLopfqdLz7K"
      },
      "outputs": [],
      "source": [
        "def show_tiles(df, dataset_folder):\n",
        "  plt.figure(figsize= (15, 30))\n",
        "  sample_n = 5\n",
        "  plt.suptitle(dataset_folder.split(\"/\")[-1], size=20)\n",
        "  labels = df.label.unique()\n",
        "  for id_label, label in enumerate(labels):\n",
        "      sample = df[df.label == label].sample(5).reset_index(drop=True)\n",
        "      for index, row in sample.iterrows():\n",
        "          path = f\"{dataset_folder}/{label}/{row.tile}\"\n",
        "          img = Image.open(path)\n",
        "          plt.subplot(len(labels), sample_n, (id_label*sample_n) + index+1)\n",
        "          if index == 0:\n",
        "              plt.ylabel(label)\n",
        "          plt.imshow(img)\n",
        "  plt.tight_layout()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xLL4RhQ8KMm1"
      },
      "outputs": [],
      "source": [
        "show_tiles(nct_df, NCT_FOLDER)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UKYdXPjQMsyf",
        "collapsed": true
      },
      "outputs": [],
      "source": [
        "show_tiles(crc_df, CRC_FOLDER)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QCFc__9zVyid"
      },
      "source": [
        "## Import model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OQhUf_oXhlp2"
      },
      "source": [
        "Let's use cuda for to import model and data in the GPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "JhZQoaAgVyid"
      },
      "outputs": [],
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "stPpQ9Flkxcd"
      },
      "source": [
        "And to use timm (Huggingface) to download UNI model and mount it in the GPU"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "thnHBKLiVyid"
      },
      "outputs": [],
      "source": [
        "feature_extractor = timm.create_model(\"hf-hub:MahmoodLab/uni\", pretrained=True, init_values=1e-5, dynamic_img_size=True)\n",
        "transform = create_transform(**resolve_data_config(feature_extractor.pretrained_cfg, model=feature_extractor))\n",
        "feature_extractor.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2A-mp6-xk521"
      },
      "source": [
        "We will use standard image normalization"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "7Qgh4gaoVyid"
      },
      "outputs": [],
      "source": [
        "transform = transforms.Compose(\n",
        "        [\n",
        "            transforms.Resize(224),\n",
        "            transforms.ToTensor(),\n",
        "            transforms.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),\n",
        "        ]\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dOb6K_Kqk-G8"
      },
      "source": [
        "Let's use pytorch to import Imagefolders and create generators to evaluate the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "rGlvGlBGVyie"
      },
      "outputs": [],
      "source": [
        "image_folder = torchvision.datasets.ImageFolder(NCT_FOLDER, transform = transform)\n",
        "train_loader = DataLoader(image_folder, batch_size=256, shuffle=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "RO-gbEKeVyie"
      },
      "outputs": [],
      "source": [
        "image_folder = torchvision.datasets.ImageFolder(CRC_FOLDER, transform = transform)\n",
        "test_loader = DataLoader(image_folder, batch_size=256, shuffle=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bSby96rAVyie"
      },
      "source": [
        "## Feature extraction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TR2OZBcolEc1"
      },
      "source": [
        "The labels and the ordinal encoding"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZAP842nRVyie"
      },
      "outputs": [],
      "source": [
        "image_folder.class_to_idx\n",
        "idx_to_class = {v: k for k, v in image_folder.class_to_idx.items()}\n",
        "idx_to_class"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gTqm3Z7hlSwL"
      },
      "source": [
        "This functions simply applies the model to the batch of images, we will limit this process to a low number of batches.\n",
        "This is the time consuming process of the notebook.\n",
        "To use it in all the images should take ~ 1:30 hrs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "gSoCy3SeVyie"
      },
      "outputs": [],
      "source": [
        "def feature_extraction(loader, sample_length):\n",
        "    with torch.inference_mode():\n",
        "        embeddings = []\n",
        "        targets = []\n",
        "        it = iter(loader)\n",
        "        for index in tqdm.tqdm(range(sample_length)):\n",
        "            a = next(it)\n",
        "            x = a[0].to(device)\n",
        "            targets.append(a[1])\n",
        "            features = feature_extractor(x).cpu().detach().numpy()\n",
        "            embeddings.append(features)\n",
        "        embeddings = np.concatenate(embeddings)\n",
        "        targets = np.concatenate(targets)\n",
        "        return embeddings, targets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SQbIsoOMlc28"
      },
      "source": [
        "Let's use the function and extract the features on X_train and X_test"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "e-rCy5NwVyif"
      },
      "outputs": [],
      "source": [
        "X_train, y_train = feature_extraction(train_loader, 10)\n",
        "X_test, y_test = feature_extraction(test_loader, 5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oZgkPWTUVyif"
      },
      "outputs": [],
      "source": [
        "print(X_train.shape)\n",
        "print(X_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RvXkLYXDVyif"
      },
      "source": [
        "## Features visualization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NqEdIbT3lhOy"
      },
      "source": [
        "We will use a PCA to show how does the features are reduced and if there's a trivial separation (clusters) between them"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fTwoXIsKVyif"
      },
      "outputs": [],
      "source": [
        "pca = PCA(n_components = 10)\n",
        "transformed = pca.fit_transform(X_train)\n",
        "trans = pd.DataFrame(transformed)\n",
        "trans[\"target\"] = y_train\n",
        "trans.target = trans.target.replace(idx_to_class)\n",
        "trans"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "PC_values = np.arange(pca.n_components_) + 1\n",
        "plt.plot(PC_values, pca.explained_variance_ratio_, 'o-', linewidth=2, color='blue')\n",
        "plt.title('Scree Plot')\n",
        "plt.xlabel('Principal Component')\n",
        "plt.ylabel('Variance Explained')\n",
        "plt.show()"
      ],
      "metadata": {
        "id": "j0TeLx4VLDBC"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KepIDjhQlqYi"
      },
      "source": [
        "As you might see, the explained variance for the first two components is slow. Neverhteless there are two highly separated clusters in the III and IV quadrant. These are LYM and BACK. You can watch the sample images to conclude if they are very different images from rest.\n",
        "\n",
        "The other separation is not trivial, so we need to apply a more advanced classification. Here is where Machine Learning comes to action"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z5-DWJTRVyif"
      },
      "outputs": [],
      "source": [
        "plt.figure(figsize=(10,10))\n",
        "ax = sns.scatterplot(data = trans, x= 0, y= 1 , hue= \"target\", alpha=1, s=20)\n",
        "sns.move_legend(ax, \"upper left\", bbox_to_anchor=(1, 1))\n",
        "plt.title(\"PCA features\")\n",
        "plt.xlabel(f\"PC1: {round(pca.explained_variance_ratio_[0] * 100, 2)}\")\n",
        "plt.ylabel(f\"PC2: {round(pca.explained_variance_ratio_[1] * 100, 2)}\")\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UV9GERSIVyif"
      },
      "source": [
        "## Train model\n",
        "We can use Scikit-learn to explore some algorithms, and to decide how is going to be our optimized model.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eCH5f4pNVyig"
      },
      "outputs": [],
      "source": [
        "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
        "from sklearn.neural_network import MLPClassifier\n",
        "from sklearn.model_selection import cross_val_score"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o77jFjR1CtcE"
      },
      "outputs": [],
      "source": [
        "def train_model(model, X_train, y_train, X_test, y_test):\n",
        "  model.fit(X_train, y_train)\n",
        "  train_acc = model.score(X_train, y_train)\n",
        "  cv_acc = cross_val_score(model, X_train, y_train) #We wil apply k-fold cross validation on X_train to obtain a most consistent metric.\n",
        "  test_acc = model.score(X_test, y_test)\n",
        "  print(\"cv_accuracy\", cv_acc.mean(), sep=\":\")\n",
        "  print(\"test_accuracy\", test_acc, sep=\":\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZTdXmT9WCiod"
      },
      "outputs": [],
      "source": [
        "train_model(AdaBoostClassifier(), X_train, y_train, X_test, y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z1npz_LeVyig"
      },
      "outputs": [],
      "source": [
        "train_model(RandomForestClassifier(), X_train, y_train, X_test, y_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8sB_fFaaVyig"
      },
      "outputs": [],
      "source": [
        "train_model(MLPClassifier(), X_train, y_train, X_test, y_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RK4LOXnEowjL"
      },
      "source": [
        "## Model optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UW-lCwbImaAI"
      },
      "source": [
        "Finally, we will find a better hyperparameters combination using Optuna. This is a bayesian optimizer that searches over some given options and tries to reach the best performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5SRX2BJmyfn"
      },
      "source": [
        "We will construct a multi layer perceptron using only three hidden layers. This layers may have a different number of neurons and the number of different combinations is huge."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LZC_bXDho8Yb"
      },
      "outputs": [],
      "source": [
        "def objective(trial):\n",
        "    neurons_1 = trial.suggest_int(\"neurons_1\", 50, 200)\n",
        "    neurons_2 = trial.suggest_int(\"neurons_2\", 50, 200)\n",
        "    neurons_3 = trial.suggest_int(\"neurons_3\", 50, 200)\n",
        "    classifier_obj = MLPClassifier(hidden_layer_sizes=(neurons_1, neurons_2, neurons_3), early_stopping=True, learning_rate=\"adaptive\")\n",
        "    classifier_obj.fit(X_train, y_train)\n",
        "    accuracy = classifier_obj.score(X_test, y_test)\n",
        "    return accuracy\n",
        "\n",
        "study = optuna.create_study(direction='maximize')\n",
        "study.optimize(objective, n_trials=100, show_progress_bar=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtjTXGmynL3l"
      },
      "source": [
        "Once the optimization is done, we can get the best combination of hyperparameters and the metric obtained."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MN1oRqLVqYem"
      },
      "outputs": [],
      "source": [
        "study.best_params"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Dukmy49pnTlV"
      },
      "source": [
        "This is a very tinny example to apply Deep learning in histology, so it's not surprising that the metrics on this datasets are usually very high."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nmQwQ5GAsb82"
      },
      "outputs": [],
      "source": [
        "round(study.best_value, 3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMPcwSnwngJc"
      },
      "source": [
        "Let's retrain our model with our optimizer hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xwvW8UvwHrxb"
      },
      "outputs": [],
      "source": [
        "model = MLPClassifier(hidden_layer_sizes=(study.best_params[\"neurons_1\"], study.best_params[\"neurons_2\"], study.best_params[\"neurons_3\"]),\n",
        "                      early_stopping=True, learning_rate=\"adaptive\")\n",
        "model.fit(X_train, y_train)\n",
        "model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mXBUdknenkt_"
      },
      "source": [
        "We can visualize the search over time"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "144B4kh4IzJw"
      },
      "outputs": [],
      "source": [
        "optuna.visualization.plot_optimization_history(study)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fvgo5hDvnuQF"
      },
      "source": [
        "Always is a good practice to watch the confussion matrix, so we can analyze the performance and the possible errors our model is doing."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DHUXCXW34xs_"
      },
      "outputs": [],
      "source": [
        "cf_matrix = confusion_matrix(y_test, model.predict(X_test))\n",
        "sns.heatmap(cf_matrix, annot=True, cmap='Blues',fmt='')\n",
        "plt.title(\"Test Confussion matrix\")\n",
        "plt.ylabel(\"Y true\")\n",
        "plt.xlabel(\"Y pred\")\n",
        "plt.yticks(np.arange(len(idx_to_class))+0.5, list(idx_to_class.values()))\n",
        "plt.xticks(np.arange(len(idx_to_class))+0.5, list(idx_to_class.values()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gZI6oqvfn_g6"
      },
      "source": [
        "And that's all. Good luck!"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": [],
      "include_colab_link": true
    },
    "kaggle": {
      "accelerator": "gpu",
      "dataSources": [
        {
          "datasetId": 1775623,
          "sourceId": 2897650,
          "sourceType": "datasetVersion"
        },
        {
          "datasetId": 1775969,
          "sourceId": 2898182,
          "sourceType": "datasetVersion"
        }
      ],
      "dockerImageVersionId": 30823,
      "isGpuEnabled": true,
      "isInternetEnabled": true,
      "language": "python",
      "sourceType": "notebook"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.12"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}