C3_W3_Lab_1_permutation_method__M.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Permutation Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lecture notebook, we'll demonstrate the permutation method on a dataset you worked on in Week 1 of Course 2. By going through this short exercise, you'll hopefully develop some intuition on how to do the related task in this week's programming assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and Loading the Pretrained Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's import the packages we'll be using:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import pickle\n",
    "from util import cindex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we load the model we trained during the said assignment in Course 2. This accepts a patient's age, systolic blood pressure, diastolic blood pressure, and cholesterol to output a risk score (i.e. probability of an event)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_X = pickle.load(open('./pretained_model/C2M1_model.model', 'rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also use the same test dataset from that programming assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# patient data (test set in the previous assignment)\n",
    "X_baseline = pd.read_csv('./data/X_data_normalized.csv',index_col=0)\n",
    "\n",
    "# corresponding patient outcome\n",
    "y = pd.read_csv('./data/y_data_test.csv',index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's print the first 10 cells to peek into the dataset. Take note that these are already standardized and normalized as you did previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Systolic_BP</th>\n",
       "      <th>Diastolic_BP</th>\n",
       "      <th>Cholesterol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4320</th>\n",
       "      <td>0.747640</td>\n",
       "      <td>-1.211321</td>\n",
       "      <td>-1.003548</td>\n",
       "      <td>-1.072461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>-0.758669</td>\n",
       "      <td>-1.476605</td>\n",
       "      <td>-1.541427</td>\n",
       "      <td>-0.434487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5689</th>\n",
       "      <td>0.067833</td>\n",
       "      <td>0.399011</td>\n",
       "      <td>0.904060</td>\n",
       "      <td>0.694137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>-0.257881</td>\n",
       "      <td>0.655984</td>\n",
       "      <td>1.157183</td>\n",
       "      <td>0.700589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1370</th>\n",
       "      <td>0.800621</td>\n",
       "      <td>-0.664058</td>\n",
       "      <td>0.228552</td>\n",
       "      <td>1.366445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>-0.988683</td>\n",
       "      <td>0.935349</td>\n",
       "      <td>0.752583</td>\n",
       "      <td>-0.709739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5783</th>\n",
       "      <td>0.145403</td>\n",
       "      <td>0.759190</td>\n",
       "      <td>0.155382</td>\n",
       "      <td>1.790223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3484</th>\n",
       "      <td>0.149429</td>\n",
       "      <td>-0.852439</td>\n",
       "      <td>0.671971</td>\n",
       "      <td>0.477288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1145</th>\n",
       "      <td>-0.250499</td>\n",
       "      <td>-0.173969</td>\n",
       "      <td>0.006586</td>\n",
       "      <td>-0.360930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>-0.757791</td>\n",
       "      <td>-0.781030</td>\n",
       "      <td>0.431331</td>\n",
       "      <td>-0.617740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Age  Systolic_BP  Diastolic_BP  Cholesterol\n",
       "4320  0.747640    -1.211321     -1.003548    -1.072461\n",
       "2006 -0.758669    -1.476605     -1.541427    -0.434487\n",
       "5689  0.067833     0.399011      0.904060     0.694137\n",
       "472  -0.257881     0.655984      1.157183     0.700589\n",
       "1370  0.800621    -0.664058      0.228552     1.366445\n",
       "1457 -0.988683     0.935349      0.752583    -0.709739\n",
       "5783  0.145403     0.759190      0.155382     1.790223\n",
       "3484  0.149429    -0.852439      0.671971     0.477288\n",
       "1145 -0.250499    -0.173969      0.006586    -0.360930\n",
       "130  -0.757791    -0.781030      0.431331    -0.617740"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_baseline.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## C-index and Baseline Performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's get the c-index of the trained model using our test dataset. Just to refresh:\n",
    "\n",
    "* The c-index measures the discriminatory power of a risk score. \n",
    "* Intuitively, a higher c-index indicates that the model's prediction is in agreement with the actual outcomes of a pair of patients.\n",
    "* The formula for the c-index is\n",
    "\n",
    "$$ \\mbox{cindex} = \\frac{\\mbox{concordant} + 0.5 \\times \\mbox{ties}}{\\mbox{permissible}} $$\n",
    "\n",
    "* A permissible pair is a pair of patients who have different outcomes.\n",
    "* A concordant pair is a permissible pair in which the patient with the higher risk score also has the worse outcome.\n",
    "* A tie is a permissible pair where the patients have the same risk score.\n",
    "\n",
    "\n",
    "Computing this for our test dataset will give us our baseline performance. For efficiency, we will be using the `concordance_index()` method from the [lifelines](https://lifelines.readthedocs.io/en/latest/lifelines.utils.html) package instead of the homemade c-index function we developed in the Course 2 assignment. This is wrapped in a `cindex()` function in the `util` library we imported earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline c-index is 0.8182\n"
     ]
    }
   ],
   "source": [
    "# get the patient's risk scores by feeding the baseline data to the trained model\n",
    "scores = model_X.predict_proba(X_baseline)[:, 1]\n",
    "\n",
    "# measure the model's cindex given the patient outcome and computed risk scores above\n",
    "c_index = cindex(y.values, scores)\n",
    "\n",
    "print(f\"baseline c-index is {c_index:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This should look familiar as this is also the result you got in the Course 2 assignment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Importance and the Permutation Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed in the lecture videos, one way we can compute the importance of a feature in a given model is by shuffling the values of a particular column in our test set. Evaluating the performance of the model after this shuffling will allow us to quantify the feature's importance. For this exercise, we'll choose to shuffle the `Age` column. You will do this programmatically in this week's assignment but for now, you'll use a dataset that is shuffled beforehand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Systolic_BP</th>\n",
       "      <th>Diastolic_BP</th>\n",
       "      <th>Cholesterol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4320</th>\n",
       "      <td>-0.494544</td>\n",
       "      <td>-1.211321</td>\n",
       "      <td>-1.003548</td>\n",
       "      <td>-1.072461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>-0.289638</td>\n",
       "      <td>-1.476605</td>\n",
       "      <td>-1.541427</td>\n",
       "      <td>-0.434487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5689</th>\n",
       "      <td>0.425693</td>\n",
       "      <td>0.399011</td>\n",
       "      <td>0.904060</td>\n",
       "      <td>0.694137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>-2.244324</td>\n",
       "      <td>0.655984</td>\n",
       "      <td>1.157183</td>\n",
       "      <td>0.700589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1370</th>\n",
       "      <td>-0.078010</td>\n",
       "      <td>-0.664058</td>\n",
       "      <td>0.228552</td>\n",
       "      <td>1.366445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>-1.475322</td>\n",
       "      <td>0.935349</td>\n",
       "      <td>0.752583</td>\n",
       "      <td>-0.709739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5783</th>\n",
       "      <td>0.694167</td>\n",
       "      <td>0.759190</td>\n",
       "      <td>0.155382</td>\n",
       "      <td>1.790223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3484</th>\n",
       "      <td>-0.695555</td>\n",
       "      <td>-0.852439</td>\n",
       "      <td>0.671971</td>\n",
       "      <td>0.477288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1145</th>\n",
       "      <td>-0.761429</td>\n",
       "      <td>-0.173969</td>\n",
       "      <td>0.006586</td>\n",
       "      <td>-0.360930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>0.021294</td>\n",
       "      <td>-0.781030</td>\n",
       "      <td>0.431331</td>\n",
       "      <td>-0.617740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Age  Systolic_BP  Diastolic_BP  Cholesterol\n",
       "4320 -0.494544    -1.211321     -1.003548    -1.072461\n",
       "2006 -0.289638    -1.476605     -1.541427    -0.434487\n",
       "5689  0.425693     0.399011      0.904060     0.694137\n",
       "472  -2.244324     0.655984      1.157183     0.700589\n",
       "1370 -0.078010    -0.664058      0.228552     1.366445\n",
       "1457 -1.475322     0.935349      0.752583    -0.709739\n",
       "5783  0.694167     0.759190      0.155382     1.790223\n",
       "3484 -0.695555    -0.852439      0.671971     0.477288\n",
       "1145 -0.761429    -0.173969      0.006586    -0.360930\n",
       "130   0.021294    -0.781030      0.431331    -0.617740"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# patient data permuted at the Age column\n",
    "X_permuted_1 = pd.read_csv('./data/X_data_normalized_perm1.csv',index_col=0)\n",
    "\n",
    "# Let's print the first 10 cells and compare with the baseline table above.\n",
    "# All are in the same place except for the values in the Age column\n",
    "X_permuted_1.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's see how our model performs with this shuffled input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c-index for 1st permutation is 0.6749\n"
     ]
    }
   ],
   "source": [
    "# get the patient's risk scores by feeding the baseline data with shuffled Age column\n",
    "scores = model_X.predict_proba(X_permuted_1)[:, 1]\n",
    "\n",
    "# measure the c-index given the patient outcome and newly computed risk scores\n",
    "c_index = cindex(y.values, scores)\n",
    "print(f\"c-index for 1st permutation is {c_index:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a big drop in the performance. To quantify the importance of this feature, we simply subtract this drop from the baseline performance. This results in `0.8182 - 0.6749 = 0.1433`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Permutations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result above (i.e. `0.1433`) describes the feature importance of `Age` in our model given the shuffled dataset. However, you might deduce that the predicted risk scores are greatly affected by how the Age column is rearranged. If we arrange it differently, then the model will predict different values and this might lead to a different c-index, and consequently a different value of the feature importance. To illustrate, let's import two more datasets with different permutations of the `Age` column. Since there is a random element when doing the permutation, the resulting order of the elements in the `Age` column will be different for each shuffle. You can see that in the two datasets below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Systolic_BP</th>\n",
       "      <th>Diastolic_BP</th>\n",
       "      <th>Cholesterol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4320</th>\n",
       "      <td>0.728099</td>\n",
       "      <td>-1.211321</td>\n",
       "      <td>-1.003548</td>\n",
       "      <td>-1.072461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>0.008176</td>\n",
       "      <td>-1.476605</td>\n",
       "      <td>-1.541427</td>\n",
       "      <td>-0.434487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5689</th>\n",
       "      <td>0.604035</td>\n",
       "      <td>0.399011</td>\n",
       "      <td>0.904060</td>\n",
       "      <td>0.694137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>-1.179030</td>\n",
       "      <td>0.655984</td>\n",
       "      <td>1.157183</td>\n",
       "      <td>0.700589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1370</th>\n",
       "      <td>0.835747</td>\n",
       "      <td>-0.664058</td>\n",
       "      <td>0.228552</td>\n",
       "      <td>1.366445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>0.872547</td>\n",
       "      <td>0.935349</td>\n",
       "      <td>0.752583</td>\n",
       "      <td>-0.709739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5783</th>\n",
       "      <td>0.498609</td>\n",
       "      <td>0.759190</td>\n",
       "      <td>0.155382</td>\n",
       "      <td>1.790223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3484</th>\n",
       "      <td>0.324675</td>\n",
       "      <td>-0.852439</td>\n",
       "      <td>0.671971</td>\n",
       "      <td>0.477288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1145</th>\n",
       "      <td>0.671958</td>\n",
       "      <td>-0.173969</td>\n",
       "      <td>0.006586</td>\n",
       "      <td>-0.360930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>-0.925788</td>\n",
       "      <td>-0.781030</td>\n",
       "      <td>0.431331</td>\n",
       "      <td>-0.617740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Age  Systolic_BP  Diastolic_BP  Cholesterol\n",
       "4320  0.728099    -1.211321     -1.003548    -1.072461\n",
       "2006  0.008176    -1.476605     -1.541427    -0.434487\n",
       "5689  0.604035     0.399011      0.904060     0.694137\n",
       "472  -1.179030     0.655984      1.157183     0.700589\n",
       "1370  0.835747    -0.664058      0.228552     1.366445\n",
       "1457  0.872547     0.935349      0.752583    -0.709739\n",
       "5783  0.498609     0.759190      0.155382     1.790223\n",
       "3484  0.324675    -0.852439      0.671971     0.477288\n",
       "1145  0.671958    -0.173969      0.006586    -0.360930\n",
       "130  -0.925788    -0.781030      0.431331    -0.617740"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 2nd permutation\n",
    "X_permuted_2 = pd.read_csv('./data/X_data_normalized_perm2.csv',index_col=0)\n",
    "\n",
    "X_permuted_2.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Systolic_BP</th>\n",
       "      <th>Diastolic_BP</th>\n",
       "      <th>Cholesterol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4320</th>\n",
       "      <td>-0.593886</td>\n",
       "      <td>-1.211321</td>\n",
       "      <td>-1.003548</td>\n",
       "      <td>-1.072461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2006</th>\n",
       "      <td>0.303244</td>\n",
       "      <td>-1.476605</td>\n",
       "      <td>-1.541427</td>\n",
       "      <td>-0.434487</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5689</th>\n",
       "      <td>-1.755198</td>\n",
       "      <td>0.399011</td>\n",
       "      <td>0.904060</td>\n",
       "      <td>0.694137</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>472</th>\n",
       "      <td>-0.500154</td>\n",
       "      <td>0.655984</td>\n",
       "      <td>1.157183</td>\n",
       "      <td>0.700589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1370</th>\n",
       "      <td>-0.109112</td>\n",
       "      <td>-0.664058</td>\n",
       "      <td>0.228552</td>\n",
       "      <td>1.366445</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1457</th>\n",
       "      <td>0.439787</td>\n",
       "      <td>0.935349</td>\n",
       "      <td>0.752583</td>\n",
       "      <td>-0.709739</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5783</th>\n",
       "      <td>1.335356</td>\n",
       "      <td>0.759190</td>\n",
       "      <td>0.155382</td>\n",
       "      <td>1.790223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3484</th>\n",
       "      <td>-0.428345</td>\n",
       "      <td>-0.852439</td>\n",
       "      <td>0.671971</td>\n",
       "      <td>0.477288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1145</th>\n",
       "      <td>0.199869</td>\n",
       "      <td>-0.173969</td>\n",
       "      <td>0.006586</td>\n",
       "      <td>-0.360930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>130</th>\n",
       "      <td>0.146330</td>\n",
       "      <td>-0.781030</td>\n",
       "      <td>0.431331</td>\n",
       "      <td>-0.617740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Age  Systolic_BP  Diastolic_BP  Cholesterol\n",
       "4320 -0.593886    -1.211321     -1.003548    -1.072461\n",
       "2006  0.303244    -1.476605     -1.541427    -0.434487\n",
       "5689 -1.755198     0.399011      0.904060     0.694137\n",
       "472  -0.500154     0.655984      1.157183     0.700589\n",
       "1370 -0.109112    -0.664058      0.228552     1.366445\n",
       "1457  0.439787     0.935349      0.752583    -0.709739\n",
       "5783  1.335356     0.759190      0.155382     1.790223\n",
       "3484 -0.428345    -0.852439      0.671971     0.477288\n",
       "1145  0.199869    -0.173969      0.006586    -0.360930\n",
       "130   0.146330    -0.781030      0.431331    -0.617740"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 3rd permutation\n",
    "X_permuted_3 = pd.read_csv('./data/X_data_normalized_perm3.csv',index_col=0)\n",
    "\n",
    "X_permuted_3.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's evaluate the performance of the model with these new datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c-index for 2nd permutation: 0.6554\n"
     ]
    }
   ],
   "source": [
    "# get the patient's risk scores by feeding a dataset with a different permutation of the Age column\n",
    "scores = model_X.predict_proba(X_permuted_2)[:, 1]\n",
    "\n",
    "# measure c-index\n",
    "c_index = cindex(y.values, scores)\n",
    "\n",
    "print(f\"c-index for 2nd permutation: {c_index:.4f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "c-index for 3rd permutation: 0.6428\n"
     ]
    }
   ],
   "source": [
    "# get the patient's risk scores by feeding a dataset with a different permutation of the Age column\n",
    "scores = model_X.predict_proba(X_permuted_3)[:, 1]\n",
    "\n",
    "# measure c-index\n",
    "c_index = cindex(y.values, scores)\n",
    "\n",
    "print(f\"c-index for 3rd permutation: {c_index:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are slight differences in the resulting c-index depending on how the values are shuffled. This will then result in different values of the feature importance as well. Let's summarize our results in a table for clarity:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Dataset      | C-index | Importance |\n",
    "|:---------------|:--------:| :--------: |\n",
    "|Baseline       | 0.8182| n/a |\n",
    "|1st permutation| 0.6749 | 0.1433 |\n",
    "|2nd permutation| 0.6554 | 0.1628 |\n",
    "|3rd permutation| 0.6428 | 0.1754 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To take these variations into account, we can take the mean of these different results to have a single value of the feature importance. More formally:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{align}\n",
    "I_x  = \\left\\lvert perf - perf_x  \\right\\rvert\n",
    "\\end{align}\n",
    "\n",
    "where $I_x$ is the importance of feature $x$ and\n",
    "\\begin{align}\n",
    "perf_x  = \\frac{1}{n}\\cdot \\sum_{i=1}^{n} perf_i^{sx}\n",
    "\\end{align}\n",
    "\n",
    "where $perf_i^{sx}$ is the performance with the feature $x$ shuffled in the $i$th permutation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Applying this to our results, that would be `abs(0.8182 - mean([0.6749, 0.6554, 0.6428]) = 0.1605` . We would want to get more permutations to get a more accurate value of the feature importance. You will get the chance to work this out in the week's programming assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it for this lecture on the permutation method! We can now measure the global importance of a feature in the predictions of a given model. In the next section of the course, you will learn how to determine the importance of a feature for an individual patient in the dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}