Identify_Fraudulent_Activities.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fraudelent Activities Detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal<br>\n",
    "### Build a machine learning model to predict the probability that the first transaction of a new user is fraudulent.<br>\n",
    "#### 1. For each user, determine her country based on the numeric IP address.<br>\n",
    "#### 2. Build a model to predict whether an activity is fraudulent or not. Explain how different assumptions about the cost of false positives vs false negatives would impact the model.<br>\n",
    "#### 3. Your boss is a bit worried about using a model she doesn't understand for something as important as fraud detection. How would you explain her how the model is making the predictions? Not from a mathematical perspective (she couldn't care less about that), but from a user perspective. What kinds of users are more likely to be classified as at risk? What are their characteristics?<br>\n",
    "#### 4. Let's say you now have this model which can be used live to predict in real time if an activity is fraudulent or not. From a product perspective, how would you use it? That is,what kind of different user experiences would you build based on the model output?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# each new user's first transaction\n",
    "data = pd.read_csv(r'C:\\Users\\lshen\\Downloads\\Fraud\\Fraud_Data.csv')\n",
    "ip = pd.read_csv(r'C:\\Users\\lshen\\Downloads\\Fraud\\IpAddress_to_Country.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id          signup_time        purchase_time  purchase_value  \\\n",
       "0    22058  2015-02-24 22:55:49  2015-04-18 02:47:11              34   \n",
       "1   333320  2015-06-07 20:39:50  2015-06-08 01:38:54              16   \n",
       "2     1359  2015-01-01 18:52:44  2015-01-01 18:52:45              15   \n",
       "3   150084  2015-04-28 21:13:25  2015-05-04 13:54:50              44   \n",
       "4   221365  2015-07-21 07:09:52  2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address  class  \n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08      0  \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08      0  \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09      1  \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09      0  \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08      0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id             int64\n",
       "signup_time        object\n",
       "purchase_time      object\n",
       "purchase_value      int64\n",
       "device_id          object\n",
       "source             object\n",
       "browser            object\n",
       "sex                object\n",
       "age                 int64\n",
       "ip_address        float64\n",
       "class               int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(151112, 11)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "151112"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if there is any duplicate values\n",
    "len(data.user_id.unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id           0\n",
       "signup_time       0\n",
       "purchase_time     0\n",
       "purchase_value    0\n",
       "device_id         0\n",
       "source            0\n",
       "browser           0\n",
       "sex               0\n",
       "age               0\n",
       "ip_address        0\n",
       "class             0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>lower_bound_ip_address</th>\n",
       "      <th>upper_bound_ip_address</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>16777216.0</td>\n",
       "      <td>16777471</td>\n",
       "      <td>Australia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>16777472.0</td>\n",
       "      <td>16777727</td>\n",
       "      <td>China</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>16777728.0</td>\n",
       "      <td>16778239</td>\n",
       "      <td>China</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16778240.0</td>\n",
       "      <td>16779263</td>\n",
       "      <td>Australia</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>16779264.0</td>\n",
       "      <td>16781311</td>\n",
       "      <td>China</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   lower_bound_ip_address  upper_bound_ip_address    country\n",
       "0              16777216.0                16777471  Australia\n",
       "1              16777472.0                16777727      China\n",
       "2              16777728.0                16778239      China\n",
       "3              16778240.0                16779263  Australia\n",
       "4              16779264.0                16781311      China"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ip.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Determine the country based on ip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create country with NA values\n",
    "data['country'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>class</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id          signup_time        purchase_time  purchase_value  \\\n",
       "0    22058  2015-02-24 22:55:49  2015-04-18 02:47:11              34   \n",
       "1   333320  2015-06-07 20:39:50  2015-06-08 01:38:54              16   \n",
       "2     1359  2015-01-01 18:52:44  2015-01-01 18:52:45              15   \n",
       "3   150084  2015-04-28 21:13:25  2015-05-04 13:54:50              44   \n",
       "4   221365  2015-07-21 07:09:52  2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address  class  country  \n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08      0      NaN  \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08      0      NaN  \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09      1      NaN  \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09      0      NaN  \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08      0      NaN  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# look up the country in ip table and update country value\n",
    "# some ip_addres don't have corresponding values, check if length is 0, then remain nan value\n",
    "for i in range(len(data)):\n",
    "    c = ip[(ip.lower_bound_ip_address <=data.ip_address[i])&(ip.upper_bound_ip_address>=data.ip_address[i])].country\n",
    "    if len(c) == 1:\n",
    "        data.loc[i,'country'] = c.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United States                           58049\n",
       "NaN                                     21966\n",
       "China                                   12038\n",
       "Japan                                    7306\n",
       "United Kingdom                           4490\n",
       "Korea Republic of                        4162\n",
       "Germany                                  3646\n",
       "France                                   3161\n",
       "Canada                                   2975\n",
       "Brazil                                   2961\n",
       "Italy                                    1944\n",
       "Australia                                1844\n",
       "Netherlands                              1680\n",
       "Russian Federation                       1616\n",
       "India                                    1310\n",
       "Taiwan; Republic of China (ROC)          1237\n",
       "Mexico                                   1121\n",
       "Sweden                                   1090\n",
       "Spain                                    1027\n",
       "South Africa                              838\n",
       "Switzerland                               785\n",
       "Poland                                    729\n",
       "Argentina                                 661\n",
       "Indonesia                                 649\n",
       "Norway                                    609\n",
       "Colombia                                  602\n",
       "Turkey                                    568\n",
       "Viet Nam                                  550\n",
       "Romania                                   525\n",
       "Denmark                                   490\n",
       "                                        ...  \n",
       "Congo The Democratic Republic of The        3\n",
       "Faroe Islands                               3\n",
       "Bhutan                                      2\n",
       "Saint Kitts and Nevis                       2\n",
       "Burkina Faso                                2\n",
       "Bermuda                                     2\n",
       "Lesotho                                     2\n",
       "Gibraltar                                   2\n",
       "Liechtenstein                               2\n",
       "Fiji                                        2\n",
       "Benin                                       2\n",
       "Maldives                                    2\n",
       "Burundi                                     1\n",
       "Dominica                                    1\n",
       "South Sudan                                 1\n",
       "Myanmar                                     1\n",
       "Cape Verde                                  1\n",
       "Gambia                                      1\n",
       "San Marino                                  1\n",
       "Nauru                                       1\n",
       "Guadeloupe                                  1\n",
       "Vanuatu                                     1\n",
       "Bonaire; Sint Eustatius; Saba               1\n",
       "Yemen                                       1\n",
       "Madagascar                                  1\n",
       "Niger                                       1\n",
       "Turkmenistan                                1\n",
       "British Indian Ocean Territory              1\n",
       "Saint Martin                                1\n",
       "Tajikistan                                  1\n",
       "Name: country, Length: 182, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.country.value_counts(dropna = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build a Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Feature Engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.1  I have a hypothesis that if it's a very short time between signup and purchase, there is a large chance that it's a fraud. So change the data type to datetime and also create a new feature of time difference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['signup_time']=pd.to_datetime(data.signup_time)\n",
    "data['purchase_time']=pd.to_datetime(data.purchase_time)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id                    int64\n",
       "signup_time       datetime64[ns]\n",
       "purchase_time     datetime64[ns]\n",
       "purchase_value             int64\n",
       "device_id                 object\n",
       "source                    object\n",
       "browser                   object\n",
       "sex                       object\n",
       "age                        int64\n",
       "ip_address               float64\n",
       "class                      int64\n",
       "country                   object\n",
       "dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['purchase_signup_diff'] =  (data.purchase_time - data.signup_time)/np.timedelta64(1,'s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>class</th>\n",
       "      <th>country</th>\n",
       "      <th>purchase_signup_diff</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>Japan</td>\n",
       "      <td>4506682.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>17944.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>1</td>\n",
       "      <td>United States</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>492085.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>4361461.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id         signup_time       purchase_time  purchase_value  \\\n",
       "0    22058 2015-02-24 22:55:49 2015-04-18 02:47:11              34   \n",
       "1   333320 2015-06-07 20:39:50 2015-06-08 01:38:54              16   \n",
       "2     1359 2015-01-01 18:52:44 2015-01-01 18:52:45              15   \n",
       "3   150084 2015-04-28 21:13:25 2015-05-04 13:54:50              44   \n",
       "4   221365 2015-07-21 07:09:52 2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address  class        country  \\\n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08      0          Japan   \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08      0  United States   \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09      1  United States   \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09      0            NaN   \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08      0  United States   \n",
       "\n",
       "   purchase_signup_diff  \n",
       "0             4506682.0  \n",
       "1               17944.0  \n",
       "2                   1.0  \n",
       "3              492085.0  \n",
       "4             4361461.0  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.2 Hypothesis: if multiple users use the same device, might have a larger chance of fraud.<br>\n",
    "Create a feature with total number of occurance of this device"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['device_occ'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "counts = data.device_id.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):\n",
    "    data.loc[i,'device_occ'] = counts[data.loc[i,'device_id']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     1.0\n",
       "1     1.0\n",
       "2    12.0\n",
       "3     1.0\n",
       "4     1.0\n",
       "Name: device_occ, dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.device_occ.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.3 Hypothesis: if multiple users use same ip address, might have a larger chance of fraud.<br>\n",
    "Create a feature with counts of ip address"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['ip_occ'] = np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "counts_ip = data.ip_address.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):\n",
    "    data.loc[i,'ip_occ'] = counts_ip[data.loc[i,'ip_address']]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     1.0\n",
       "1     1.0\n",
       "2    12.0\n",
       "3     1.0\n",
       "4     1.0\n",
       "Name: ip_occ, dtype: float64"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.ip_occ.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.4 Day of week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['signup_dayofweek'] = data.signup_time.dt.dayofweek\n",
    "data['purchase_dayofweek'] = data.purchase_time.dt.dayofweek"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.5 Week of year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['signup_weekofyear'] = data.signup_time.dt.week\n",
    "data['purchase_weekofyear'] = data.purchase_time.dt.week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>class</th>\n",
       "      <th>country</th>\n",
       "      <th>purchase_signup_diff</th>\n",
       "      <th>device_occ</th>\n",
       "      <th>ip_occ</th>\n",
       "      <th>signup_dayofweek</th>\n",
       "      <th>purchase_dayofweek</th>\n",
       "      <th>signup_weekofyear</th>\n",
       "      <th>purchase_weekofyear</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>Japan</td>\n",
       "      <td>4506682.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>17944.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>1</td>\n",
       "      <td>United States</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>492085.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>4361461.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id         signup_time       purchase_time  purchase_value  \\\n",
       "0    22058 2015-02-24 22:55:49 2015-04-18 02:47:11              34   \n",
       "1   333320 2015-06-07 20:39:50 2015-06-08 01:38:54              16   \n",
       "2     1359 2015-01-01 18:52:44 2015-01-01 18:52:45              15   \n",
       "3   150084 2015-04-28 21:13:25 2015-05-04 13:54:50              44   \n",
       "4   221365 2015-07-21 07:09:52 2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address  class        country  \\\n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08      0          Japan   \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08      0  United States   \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09      1  United States   \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09      0            NaN   \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08      0  United States   \n",
       "\n",
       "   purchase_signup_diff  device_occ  ip_occ  signup_dayofweek  \\\n",
       "0             4506682.0         1.0     1.0                 1   \n",
       "1               17944.0         1.0     1.0                 6   \n",
       "2                   1.0        12.0    12.0                 3   \n",
       "3              492085.0         1.0     1.0                 1   \n",
       "4             4361461.0         1.0     1.0                 1   \n",
       "\n",
       "   purchase_dayofweek  signup_weekofyear  purchase_weekofyear  \n",
       "0                   5                  9                   16  \n",
       "1                   0                 23                   24  \n",
       "2                   3                  1                    1  \n",
       "3                   0                 18                   19  \n",
       "4                   2                 30                   37  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.6 replace country NA values with str"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.country.fillna('Not_found', inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "user_id                 0\n",
       "signup_time             0\n",
       "purchase_time           0\n",
       "purchase_value          0\n",
       "device_id               0\n",
       "source                  0\n",
       "browser                 0\n",
       "sex                     0\n",
       "age                     0\n",
       "ip_address              0\n",
       "class                   0\n",
       "country                 0\n",
       "purchase_signup_diff    0\n",
       "device_occ              0\n",
       "ip_occ                  0\n",
       "signup_dayofweek        0\n",
       "purchase_dayofweek      0\n",
       "signup_weekofyear       0\n",
       "purchase_weekofyear     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.7 keep top 50 countries, name the rest \"Others\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the top 50 countries\n",
    "country_counts = data.country.value_counts().head(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(data)):\n",
    "    if data.loc[i,'country'] not in country_counts:\n",
    "        data.loc[i,'country'] = 'Others'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United States                      58049\n",
       "Not_found                          21966\n",
       "China                              12038\n",
       "Japan                               7306\n",
       "United Kingdom                      4490\n",
       "Korea Republic of                   4162\n",
       "Others                              4050\n",
       "Germany                             3646\n",
       "France                              3161\n",
       "Canada                              2975\n",
       "Brazil                              2961\n",
       "Italy                               1944\n",
       "Australia                           1844\n",
       "Netherlands                         1680\n",
       "Russian Federation                  1616\n",
       "India                               1310\n",
       "Taiwan; Republic of China (ROC)     1237\n",
       "Mexico                              1121\n",
       "Sweden                              1090\n",
       "Spain                               1027\n",
       "South Africa                         838\n",
       "Switzerland                          785\n",
       "Poland                               729\n",
       "Argentina                            661\n",
       "Indonesia                            649\n",
       "Norway                               609\n",
       "Colombia                             602\n",
       "Turkey                               568\n",
       "Viet Nam                             550\n",
       "Romania                              525\n",
       "Denmark                              490\n",
       "Hong Kong                            471\n",
       "Finland                              460\n",
       "Austria                              435\n",
       "Ukraine                              429\n",
       "Chile                                417\n",
       "Belgium                              409\n",
       "Iran (ISLAMIC Republic Of)           389\n",
       "Egypt                                359\n",
       "Czech Republic                       349\n",
       "Thailand                             291\n",
       "New Zealand                          278\n",
       "Israel                               272\n",
       "Saudi Arabia                         264\n",
       "Venezuela                            251\n",
       "Ireland                              240\n",
       "European Union                       238\n",
       "Greece                               231\n",
       "Portugal                             229\n",
       "Hungary                              211\n",
       "Malaysia                             210\n",
       "Name: country, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.country.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "51"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.country.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create dummy values for categorical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>class</th>\n",
       "      <th>country</th>\n",
       "      <th>purchase_signup_diff</th>\n",
       "      <th>device_occ</th>\n",
       "      <th>ip_occ</th>\n",
       "      <th>signup_dayofweek</th>\n",
       "      <th>purchase_dayofweek</th>\n",
       "      <th>signup_weekofyear</th>\n",
       "      <th>purchase_weekofyear</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>Japan</td>\n",
       "      <td>4506682.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>17944.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>1</td>\n",
       "      <td>United States</td>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>0</td>\n",
       "      <td>Not_found</td>\n",
       "      <td>492085.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>0</td>\n",
       "      <td>United States</td>\n",
       "      <td>4361461.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>37</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id         signup_time       purchase_time  purchase_value  \\\n",
       "0    22058 2015-02-24 22:55:49 2015-04-18 02:47:11              34   \n",
       "1   333320 2015-06-07 20:39:50 2015-06-08 01:38:54              16   \n",
       "2     1359 2015-01-01 18:52:44 2015-01-01 18:52:45              15   \n",
       "3   150084 2015-04-28 21:13:25 2015-05-04 13:54:50              44   \n",
       "4   221365 2015-07-21 07:09:52 2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address  class        country  \\\n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08      0          Japan   \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08      0  United States   \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09      1  United States   \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09      0      Not_found   \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08      0  United States   \n",
       "\n",
       "   purchase_signup_diff  device_occ  ip_occ  signup_dayofweek  \\\n",
       "0             4506682.0         1.0     1.0                 1   \n",
       "1               17944.0         1.0     1.0                 6   \n",
       "2                   1.0        12.0    12.0                 3   \n",
       "3              492085.0         1.0     1.0                 1   \n",
       "4             4361461.0         1.0     1.0                 1   \n",
       "\n",
       "   purchase_dayofweek  signup_weekofyear  purchase_weekofyear  \n",
       "0                   5                  9                   16  \n",
       "1                   0                 23                   24  \n",
       "2                   3                  1                    1  \n",
       "3                   0                 18                   19  \n",
       "4                   2                 30                   37  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "source_dummies = pd.get_dummies(data.source,prefix = 'source_').iloc[:,1:]\n",
    "browser_dummies = pd.get_dummies(data.browser,prefix = 'browser_').iloc[:,1:]\n",
    "sex_dummies = pd.get_dummies(data.sex,prefix = 'sex_').iloc[:,1:]\n",
    "country_dummies = pd.get_dummies(data.country,prefix = 'country_').iloc[:,1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.concat([data, source_dummies,browser_dummies, sex_dummies,country_dummies],axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>user_id</th>\n",
       "      <th>signup_time</th>\n",
       "      <th>purchase_time</th>\n",
       "      <th>purchase_value</th>\n",
       "      <th>device_id</th>\n",
       "      <th>source</th>\n",
       "      <th>browser</th>\n",
       "      <th>sex</th>\n",
       "      <th>age</th>\n",
       "      <th>ip_address</th>\n",
       "      <th>...</th>\n",
       "      <th>country__Sweden</th>\n",
       "      <th>country__Switzerland</th>\n",
       "      <th>country__Taiwan; Republic of China (ROC)</th>\n",
       "      <th>country__Thailand</th>\n",
       "      <th>country__Turkey</th>\n",
       "      <th>country__Ukraine</th>\n",
       "      <th>country__United Kingdom</th>\n",
       "      <th>country__United States</th>\n",
       "      <th>country__Venezuela</th>\n",
       "      <th>country__Viet Nam</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>22058</td>\n",
       "      <td>2015-02-24 22:55:49</td>\n",
       "      <td>2015-04-18 02:47:11</td>\n",
       "      <td>34</td>\n",
       "      <td>QVPSPJUOCKZAR</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>M</td>\n",
       "      <td>39</td>\n",
       "      <td>7.327584e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>333320</td>\n",
       "      <td>2015-06-07 20:39:50</td>\n",
       "      <td>2015-06-08 01:38:54</td>\n",
       "      <td>16</td>\n",
       "      <td>EOGFQPIZPYXFZ</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Chrome</td>\n",
       "      <td>F</td>\n",
       "      <td>53</td>\n",
       "      <td>3.503114e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1359</td>\n",
       "      <td>2015-01-01 18:52:44</td>\n",
       "      <td>2015-01-01 18:52:45</td>\n",
       "      <td>15</td>\n",
       "      <td>YSSKYOSJHPPLJ</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Opera</td>\n",
       "      <td>M</td>\n",
       "      <td>53</td>\n",
       "      <td>2.621474e+09</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>150084</td>\n",
       "      <td>2015-04-28 21:13:25</td>\n",
       "      <td>2015-05-04 13:54:50</td>\n",
       "      <td>44</td>\n",
       "      <td>ATGTXKYKUDUQN</td>\n",
       "      <td>SEO</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>41</td>\n",
       "      <td>3.840542e+09</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>221365</td>\n",
       "      <td>2015-07-21 07:09:52</td>\n",
       "      <td>2015-09-09 18:40:53</td>\n",
       "      <td>39</td>\n",
       "      <td>NAUITBZFJKHWW</td>\n",
       "      <td>Ads</td>\n",
       "      <td>Safari</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>4.155831e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 76 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   user_id         signup_time       purchase_time  purchase_value  \\\n",
       "0    22058 2015-02-24 22:55:49 2015-04-18 02:47:11              34   \n",
       "1   333320 2015-06-07 20:39:50 2015-06-08 01:38:54              16   \n",
       "2     1359 2015-01-01 18:52:44 2015-01-01 18:52:45              15   \n",
       "3   150084 2015-04-28 21:13:25 2015-05-04 13:54:50              44   \n",
       "4   221365 2015-07-21 07:09:52 2015-09-09 18:40:53              39   \n",
       "\n",
       "       device_id source browser sex  age    ip_address        ...          \\\n",
       "0  QVPSPJUOCKZAR    SEO  Chrome   M   39  7.327584e+08        ...           \n",
       "1  EOGFQPIZPYXFZ    Ads  Chrome   F   53  3.503114e+08        ...           \n",
       "2  YSSKYOSJHPPLJ    SEO   Opera   M   53  2.621474e+09        ...           \n",
       "3  ATGTXKYKUDUQN    SEO  Safari   M   41  3.840542e+09        ...           \n",
       "4  NAUITBZFJKHWW    Ads  Safari   M   45  4.155831e+08        ...           \n",
       "\n",
       "   country__Sweden country__Switzerland  \\\n",
       "0                0                    0   \n",
       "1                0                    0   \n",
       "2                0                    0   \n",
       "3                0                    0   \n",
       "4                0                    0   \n",
       "\n",
       "   country__Taiwan; Republic of China (ROC)  country__Thailand  \\\n",
       "0                                         0                  0   \n",
       "1                                         0                  0   \n",
       "2                                         0                  0   \n",
       "3                                         0                  0   \n",
       "4                                         0                  0   \n",
       "\n",
       "   country__Turkey  country__Ukraine  country__United Kingdom  \\\n",
       "0                0                 0                        0   \n",
       "1                0                 0                        0   \n",
       "2                0                 0                        0   \n",
       "3                0                 0                        0   \n",
       "4                0                 0                        0   \n",
       "\n",
       "   country__United States  country__Venezuela  country__Viet Nam  \n",
       "0                       0                   0                  0  \n",
       "1                       1                   0                  0  \n",
       "2                       1                   0                  0  \n",
       "3                       0                   0                  0  \n",
       "4                       1                   0                  0  \n",
       "\n",
       "[5 rows x 76 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "76"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_col = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(12,len(data.columns)):\n",
    "    feature_col.append(data.columns[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.8 Feature columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "feature_col.append('purchase_value')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "65"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(feature_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Build Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = data[feature_col]\n",
    "y = data['class']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>purchase_signup_diff</th>\n",
       "      <th>device_occ</th>\n",
       "      <th>ip_occ</th>\n",
       "      <th>signup_dayofweek</th>\n",
       "      <th>purchase_dayofweek</th>\n",
       "      <th>signup_weekofyear</th>\n",
       "      <th>purchase_weekofyear</th>\n",
       "      <th>source__Direct</th>\n",
       "      <th>source__SEO</th>\n",
       "      <th>browser__FireFox</th>\n",
       "      <th>...</th>\n",
       "      <th>country__Switzerland</th>\n",
       "      <th>country__Taiwan; Republic of China (ROC)</th>\n",
       "      <th>country__Thailand</th>\n",
       "      <th>country__Turkey</th>\n",
       "      <th>country__Ukraine</th>\n",
       "      <th>country__United Kingdom</th>\n",
       "      <th>country__United States</th>\n",
       "      <th>country__Venezuela</th>\n",
       "      <th>country__Viet Nam</th>\n",
       "      <th>purchase_value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>4506682.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>17944.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>23</td>\n",
       "      <td>24</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>492085.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>18</td>\n",
       "      <td>19</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>44</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4361461.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>30</td>\n",
       "      <td>37</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>39</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 65 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   purchase_signup_diff  device_occ  ip_occ  signup_dayofweek  \\\n",
       "0             4506682.0         1.0     1.0                 1   \n",
       "1               17944.0         1.0     1.0                 6   \n",
       "2                   1.0        12.0    12.0                 3   \n",
       "3              492085.0         1.0     1.0                 1   \n",
       "4             4361461.0         1.0     1.0                 1   \n",
       "\n",
       "   purchase_dayofweek  signup_weekofyear  purchase_weekofyear  source__Direct  \\\n",
       "0                   5                  9                   16               0   \n",
       "1                   0                 23                   24               0   \n",
       "2                   3                  1                    1               0   \n",
       "3                   0                 18                   19               0   \n",
       "4                   2                 30                   37               0   \n",
       "\n",
       "   source__SEO  browser__FireFox       ...        country__Switzerland  \\\n",
       "0            1                 0       ...                           0   \n",
       "1            0                 0       ...                           0   \n",
       "2            1                 0       ...                           0   \n",
       "3            1                 0       ...                           0   \n",
       "4            0                 0       ...                           0   \n",
       "\n",
       "   country__Taiwan; Republic of China (ROC)  country__Thailand  \\\n",
       "0                                         0                  0   \n",
       "1                                         0                  0   \n",
       "2                                         0                  0   \n",
       "3                                         0                  0   \n",
       "4                                         0                  0   \n",
       "\n",
       "   country__Turkey  country__Ukraine  country__United Kingdom  \\\n",
       "0                0                 0                        0   \n",
       "1                0                 0                        0   \n",
       "2                0                 0                        0   \n",
       "3                0                 0                        0   \n",
       "4                0                 0                        0   \n",
       "\n",
       "   country__United States  country__Venezuela  country__Viet Nam  \\\n",
       "0                       0                   0                  0   \n",
       "1                       1                   0                  0   \n",
       "2                       1                   0                  0   \n",
       "3                       0                   0                  0   \n",
       "4                       1                   0                  0   \n",
       "\n",
       "   purchase_value  \n",
       "0              34  \n",
       "1              16  \n",
       "2              15  \n",
       "3              44  \n",
       "4              39  \n",
       "\n",
       "[5 rows x 65 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X,y, test_size = 0.3, random_state = 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(151112, 65) (151112,) (105778, 65) (45334, 65) (105778,) (45334,)\n"
     ]
    }
   ],
   "source": [
    "print (X.shape, y.shape,X_train.shape, X_test.shape, y_train.shape, y_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(n_jobs=50,random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "            max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, n_estimators=10, n_jobs=50,\n",
       "            oob_score=False, random_state=0, verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.9, 0.1],\n",
       "       [1. , 0. ],\n",
       "       [1. , 0. ],\n",
       "       ...,\n",
       "       [1. , 0. ],\n",
       "       [1. , 0. ],\n",
       "       [1. , 0. ]])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9574491551594829\n"
     ]
    }
   ],
   "source": [
    "# calculate the accuracy of the trained classifier\n",
    "from sklearn import metrics\n",
    "print (metrics.accuracy_score(y_test, y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.2 Cross validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import KFold\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "kf = KFold(n_splits=10,random_state = 0)\n",
    "modelCV = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring = 'accuracy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = cross_val_score(modelCV, X_train, y_train, cv = kf, scoring = scoring)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10-fold cross validation average accuracy: 0.9546692324782885\n"
     ]
    }
   ],
   "source": [
    "print ('10-fold cross validation average accuracy: {}'.format(results.mean()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The avg accuacy score is very close to model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.3 Confusion Matrix, ROC curve--Evaluate false positive and false negative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[41098    79]\n",
      " [ 1850  2307]]\n"
     ]
    }
   ],
   "source": [
    "conf_matrix = confusion_matrix(y_test, y_pred)\n",
    "print (conf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>Predicted Value</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Actual Value</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>41098</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1850</td>\n",
       "      <td>2307</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "Predicted Value      0     1\n",
       "Actual Value                \n",
       "0                41098    79\n",
       "1                 1850  2307"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# same as above\n",
    "pd.crosstab(y_test, y_pred,rownames=['Actual Value'], colnames=['Predicted Value'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 79 false positive and 1850 false negative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ROC(Receiver Operating Characteristic) Curve typically feature true positive rate on the Y axis and false positive rate on the X axis. This means that the top left corner of the plot is the \"ideal\" point - a false positive rate of zero, and true positive is rate of one. It does mean that a larger area under the curve(AUC) is usually better.<br>\n",
    "The “steepness” of ROC curves is also important, since it is ideal to maximize the true positive rate while minimizing the false positive rate.<br>\n",
    "ROC curves are typically used in binary classification to study the output of a classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_roc_auc_score = roc_auc_score(y_test, y_pred)\n",
    "fpr, tpr, thresholds = roc_curve(y_test, rf.predict_proba(X_test)[:,1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x231e9105358>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEWCAYAAAB42tAoAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd4lGXWwOHfIZRQklAShNBC71KMKBYEEUVWbGvBjrqLveun2EVduyi7rF1QF0XRVbCsYAERFTFIkR6aEAIktBQgIeV8fzxvwiSEZIBMJpmc+7rmypS3nHmTzJmni6pijDHGHEyNYAdgjDGmcrNEYYwxplSWKIwxxpTKEoUxxphSWaIwxhhTKksUxhhjSmWJwhwyEblMRGYEO45gE5HWIpIpImEVeM44EVERqVlR5wwkEVkqIgMPYz/7G6xAYuMoqjYRWQ8cBeQBmcDXwM2qmhnMuEKRd63/pqrfBjGGOGAdUEtVc4MVhxeLAh1VdXWAzxNHJXnP1ZWVKELDcFVtAPQG+gCjgxzPYQnmt+RQ+YZ+KOx6G39ZogghqroFmI5LGACISB0ReV5ENojIVhF5VUTq+rx+jogsFJF0EVkjIkO956NE5C0R2Swim0TkiYIqFhEZKSJzvPuvisjzvnGIyFQRudO7Hysin4hIqoisE5FbfbZ7VEQ+FpH/iEg6MLL4e/LieNfb/08ReVBEavjE8ZOI/FNE0kRkhYgMLrZvae/hJxEZKyI7gEdFpL2IfC8i20Vkm4hMEpGG3vbvAa2Bz73qpv8rXg0kIrNE5HHvuBkiMkNEon3iudJ7D9tF5CERWS8ip5X0uxSRuiLygrd9mojM8f29AZd5v9NtIvKAz379ROQXEdnlve9/iUhtn9dVRG4SkUQg0XvuZRHZ6P0NzBeRk322DxOR+72/jQzv9VYiMtvbZJF3PS72tj/L+3vaJSI/i8jRPsdaLyL3ishiYLeI1PS9Bl7sCV4cW0XkRW/XgnPt8s7V3/dv0Nu3u4h8IyI7vH3vL+m6msOkqnarwjdgPXCad78l8Afwss/rLwHTgMZABPA58JT3Wj8gDRiC+9LQAujivfYZ8BpQH2gKzAOu814bCczx7g8ANrK/GrMRsBeI9Y45H3gYqA20A9YCZ3jbPgrkAOd629Yt4f29C0z1Yo8DVgHX+sSRC9wB1AIu9t5PYz/fQy5wC1ATqAt08K5FHSAG9wH1UknX2nscByhQ03s8C1gDdPKONwt42nutG65q8CTvWjzvvffTDvJ7He/t3wIIA07w4io45xveOXoB2UBXb79jgOO99xQHLAdu9zmuAt/g/h7qes9dDjTx9rkL2AKEe6/dg/ub6gyId74mPsfq4HPsvkAKcJwX81XeNavjc/0WAq18zl14TYFfgCu8+w2A40u6ziX8DUYAm73Yw73HxwX7fzOUbkEPwG5H+At0/2iZQIb3z/Qd0NB7TYDdQHuf7fsD67z7rwFjSzjmUd6HT12f5y4BZnr3ff9JBdgADPAe/x343rt/HLCh2LFHAxO8+48Cs0t5b2FeHN18nrsOmOUTRzJekvKemwdc4ed72HCwc3vbnAssKHaty0oUD/q8fiPwtXf/YeADn9fqAfsoIVHgkuZeoFcJrxWcs2Wx9zziIO/hduBTn8cKnFrG+95ZcG5gJXDOQbYrniheAR4vts1K4BSf63dNCX+/BYliNvAYEH2Q93ywRHGJ7+/JbuV/s3rC0HCuqn4rIqcA7wPRwC7ct+J6wHwRKdhWcB/A4L7ZfVXC8drgvqFv9tmvBq7kUISqqohMxv2zzgYuBf7jc5xYEdnls0sY8KPP4wOO6SMa9+37T5/n/sR9yy6wSb1PC5/XY/18D0XOLSJNgXHAybhvpTVwH5qHYovP/T24b8Z4MRWeT1X3iMj2gxwjGvfNeM2hnkdEOgEvAvG4331NXKnOV/H3fRfwNy9GBSK9GMD9jZQWh682wFUicovPc7W945Z47mKuBcYAK0RkHfCYqn7hx3kPJUZzGKyNIoSo6g/ARFy1BsA23DfT7qra0LtFqWv4BvdP276EQ23EfRuP9tkvUlW7H+TUHwAXiEgbXCniE5/jrPM5RkNVjVDVYb5hl/KWtuGqZ9r4PNca2OTzuIX4ZALv9WQ/30Pxcz/lPXe0qkbiqmSklO0PxWZc1SDg2iBw1T0l2QZkUfLvpiyvACtwvZEigfsp+h7A53147RH3AhcBjVS1Ia76rmCfg/2NlGQj8GSx33c9Vf2gpHMXp6qJqnoJrprwGeBjEalf2j6HEaM5DJYoQs9LwBAR6a2q+bi67LHet2VEpIWInOFt+xZwtYgMFpEa3mtdVHUzMAN4QUQivdfaeyWWA6jqAiAVeBOYrqoFJYh5QLrXgFnXaxjtISLH+vNGVDUP+Ah4UkQivER0J/tLLOA+VG4VkVoiciHQFfjqUN+DJwJXjbdLRFrg6ud9bcW1sxyOj4HhInKC17j8GAd+gAPg/d7eBl4U1xkgzGvArePHeSKAdCBTRLoAN/ixfS7u91dTRB7GlSgKvAk8LiIdxTlaRAoSXPHr8QZwvYgc521bX0T+IiIRfsSNiFwuIjHe+y/4G8rzYsvn4Nf+C6CZiNwurvNGhIgc5885jX8sUYQYVU3FNQA/5D11L7AamCuuZ9G3uIZJVHUecDUwFvct8gf2f3u/EldtsAxX/fIx0LyUU38AnIar+iqIJQ8YjuuFtQ73TflNIOoQ3tItuHaWtcAc7/hv+7z+K9DRO/aTwAWqWlClc6jv4TFcg2wa8CXw32KvPwU86PXoufsQ3gOqutR7L5NxpYsMXMNv9kF2uRvXiPwbsAP3Dduf/9e7cdV/GbgP7g/L2H468D9cJ4E/cSUZ3+qhF3HJegYuAb2Fa0QH18b0jnc9LlLVBFwb1b9w13s1JfRkK8VQYKmIZAIv49pdslR1D+53+5N3ruN9d1LVDFwnhOG4KrlEYNAhnNeUwQbcmSpLREbiBsCdFOxYDpWINMB9a+6oquuCHY8xpbEShTEVRESGi0g9r979eVyJYX1wozKmbJYojKk45+Aa2pNx1WUj1Ir0pgqwqidjjDGlshKFMcaYUlW5AXfR0dEaFxcX7DCMMaZKmT9//jZVjTmcfatcooiLiyMhISHYYRhjTJUiIn+WvVXJrOrJGGNMqSxRGGOMKZUlCmOMMaWyRGGMMaZUliiMMcaUyhKFMcaYUgUsUYjI2yKSIiJLDvK6iMg4EVktIotFpG+gYjHGGHP4AlmimIibNvhgzsTNd9MRGIVbcMUYY0w5y83LP6L9AzbgTlVni0hcKZucA7zrTYo2V0Qaikhzb8EZY4wxh2HvvjxWbElnaXI6SzelEfm/z+n263dHdMxgjsxuQdEFUpK85w5IFCIyClfqoHXr1hUSnDHGVHZpe3JYmpzmkoL3c01qJvkKLdO28uR3r3FK4jy2xnU6ovMEM1GUtAxkiVPZqurrwOsA8fHxNt2tMaZaUVW2pGexdFN6kaSwadfewm2aRYbTPTaSM3s0o1vzSE69Yhi1khPhhRc46tZboVatwz5/MBNFEtDK53FL3Dz9xhhTbeXnK+u27y5MCMuSXXLYsXsfACLQtkl9+rRuyOXHt6F7bCTdYyNp0qAO/Pwz9IyFiAiY8DZER0OrVmWcsWzBTBTTgJtFZDJwHJBm7RPGmOpkX24+q7ZmeMnAlRKWb05n9748AGqFCZ2OiuC0rk3pHhtF99hIujaPpH6dYh/d27fDHTfDm2/CI4/Ao49Cnz7lFmfAEoWIfAAMBKJFJAl4BKgFoKqvAl8Bw3ALsO8Brg5ULMYYE2yZ2bks3+wamJd6pYTElAxy8lxtev3aYXSLjeTC+FZ080oJHZtGULtmKZ1TVeHdd+Huu2HnTrjnHncrZ4Hs9XRJGa8rcFOgzm+MMcGyLTO7SFvCsuR01m/fTcGCok3q16ZbbCQDOrUrrDqKa1KfGjVKarotxb33wnPPwQknwKuvQs+e5f9mqILrURhjTGWhqiTt3OvT88glh63p2YXbtGxUl+6xkZzXp4WXFKI4KrIOIoeYFArs3Qu7d7v2h2uvhY4d3c8agRsWZ4nCGGP8kJuXz9ptu11S8Ol9lJ6VC0ANgQ5NG3BC+2i6x0a66qPmUUTVO/zeRgf4+mu46Sbo3Rs++QQ6d3a3ALNEYYwxxWTl5LFiS0aRksKKzelk57oRznVq1qBL80jO6hVbWEro0iyC8FphgQkoORluvx2mTHGJ4eabA3Oeg7BEYYyp1tL25LB08/5uqEuT01iTupu8fNegEBlek26xkYVdUXu0iKJddH1qhlXQnKrffQfnnQf79sHjj7vG6jp1KubcHksUxphqQVVJychmyaaiI5mTdu4ftHZUZB26x0ZxRvdmhSWFlo3qHn57wpHIyXGD5Hr1gmHD4IknoEOHio8DSxTGmBCUn6/8uWNPkaqjZclpbMvcV7hN2+j69GrVkEuPa104RiG6QcV+Uy9Rejo89BD8+iv89JNrtJ48OaghWaIwxlRp+3LzSUzJKOyGujQ5jeWbM8jMdo3MNWu4QWuDOjd1pYQWUXRtHkmD4oPWgk0VPv4YbrsNtmyBG2+E7GyoVy/YkVmiMMZUHbsLBq35VB0lbs1knzeNdr3aYXRtHsn5ffd3Re14VAPq1AxQI3N5SU2Fq66C//3PjaieOhWOPTbYURWyRGGMqZS2Fw5a2z/n0TqfQWuN69eme2wkV58UV1h1FNekPmGHOmitMoiMhG3b4KWXXPfXmpXro7lyRWOMqXZUlU279hZpS1ianM7mtKzCbVo0dIPWzuntlRRaRNIsMjw4jczlZfZsePJJNx6iQQOYOzegg+aOhCUKY0yFyctX1qZmFqk6WpqcTtreHMANWmsf04Dj2jYuLCV0i42kYb3aQY68HG3b5rq4TpwIcXGwfj306FFpkwRYojDGBEhWTh4rt2QUSQortqSTlePaE2rXrEGXZhEM69mMbgUzozaLpG7tSt6ecLhUYcIElyTS02H0aHjwwUrRWF0WSxTGmCOWnpWzf8CaN05hdWpm4aC1iPCadGseyaX92hRWHbWPaUCtihq0Vln85z/QrZubwK9792BH4zdLFMaYQ5KSnnVA1dGGHXsKX28aUYfusZEM6XZUYc+jVo2DNGgt2PbsgX/8A66/Hlq2dO0RUVGVupqpJJYojDElys9XNuzYc0BS2Ja5f2bUuCb16NkiiouPbVWYFGIiKsGgtcrgq69cD6b166FFC7jhBmjUKNhRHRZLFMYYcvLySdyaWWT9hGWb04sMWuvQtAGndIopnO+oa/MIIsLLcWbUUJGU5Cbw++QT6NoVfvgBBgwIdlRHxBKFMdXMnn25LN+cUdgNdWlyOiu3ZBQOWqtbK4yuzSOKrJ/QqVkVGLRWWTz5JHz5patyuusuqF31e2yJFoxeqSLi4+M1ISEh2GEYUyXs3L2vWNVRGmu37R+01qherSLdULvHRtE2uooOWgumefOgbl23wtz27ZCWBu3aBTuqIkRkvqrGH86+VqIwJgSoKslpWUXWY16WnEayz6C12KhwureIYniv2MLk0Dyqig9aC7a0NLj/fnjlFTjrLJg2DZo0cbcQYonCmComL19Zty2T4tNb7NzjBq2JQLvo+sTHNS6sOuoeG0mj+lW/CqTSUIUPP4Q77oCUFLjlFrdWRIiyRGFMJZadm8eqLfsbmZckp7FicwZ7c/IAqB1Wg87NIgrXT+gW6xqZ69W2f+2A+s9/4MorIT4evvgCjjkm2BEFlP01GVNJZPgOWvNKCqtTMsktGLRWpyZdYyMZ0a9VYSmhQ9NqOGgtWLKzYe1a15PpoosgN9cli7DQb+S3RGFMEKRkZBVZP2Fpcjp/bt8/aC3GG7Q2uGvTwqTQqlE9algjc3DMnOnGQezZA4mJbinSq68OdlQVxhKFMQGkWvKgtdSM/YPWWjeuR/fYSC48pmVhUmgaGR7EqE2hlBS4+2547z3Xi+n11yt8verKwBKFMeUkJy+fNamZLN3k08i8OZ2MLDdoLayG0LFpA07uGF2kS2qkDVqrnFavhn79IDMTHnjA3erWDXZUQWGJwpjDsHdfHsu3FF0/YcWWDPblukFr4bVq0LV5JOf03t8VtdNREYTXCv367CovPd0tJNS+PVx7LVxzjWuXqMYsURhThl17ig9aS2dtaiZeGzMN69Wie2wkI0+I87qjRtI2uoENWqtqdu+GMWPgjTdg8WI3id9zzwU7qkrBEoUxHlVlc1rRmVGXJaezadfewm2aR4XTPTaSYT2b0yM2ku4tooi1QWtV3+efw803w4YNrhRRBdaIqEiWKEy1lJ+vrNu+u8iAtaXJ6ezYvQ9wg9baRtenb5tGXNG/TeHAtcY2aC205Oa6rq6ffurWh/jxRzjppGBHVelYojAhLzs3r8jMqEuT01m+OZ09+/YPWuvUrAFDuh5F9xau6qhLs0jq17F/j5Cl6r4N1KwJzZvD00+7UdYhMIFfINh/ggkpmdm5RcYmLE1OZ3VKBjl5rkGhfu0wusdGcVH8/vUTOjRtQO2aNmit2pg7160T8cYb0LcvjB8f7IgqPUsUpsralpntprXYlFaYHNb7DFqLblCbbrFRDOwcU5gU2jS2QWvV1s6dbgK/116D2Fj32PgloIlCRIYCLwNhwJuq+nSx11sD7wANvW3uU9WvAhmTqXpUlaSde4uUEpYmp7E1ff+gtVaN69K9eRR/7dvSqz6KomlEHWtkNs6HH8Ktt8K2bW5Rocceg4iIYEdVZQQsUYhIGDAeGAIkAb+JyDRVXeaz2YPAR6r6ioh0A74C4gIVk6n8cvPyWZO6u8j6CcuS00n3GbTWIaYBJ7aPLlw/oVtsJFF1bdCaKcWKFRAXB19/DX36BDuaKieQJYp+wGpVXQsgIpOBcwDfRKFApHc/CkgOYDymksnKyWP55v2T4C1LTmPFlgyyfQatdWkWWWT9hM7NbNCa8UNWFjzzjGuDGD7cVTk9+GC1mMAvEAKZKFoAG30eJwHHFdvmUWCGiNwC1AdOK+lAIjIKGAXQunXrcg/UBF7anpwipYSlyems8Rm0Fhlek+6xUVxxfJvCqqN20fWpaTOjmkP17bdw441u8r677nKJopaVOI9EIBNFSZXDxdddvQSYqKoviEh/4D0R6aGq+UV2Un0deB3cUqgBidaUmx2797Fgw84iSSFp5/5Ba80i3aC1M3s0o5tXUmjZqK61J5gjs3Ur3HknvP8+dOgAM2bAkCHBjiokBDJRJAGtfB635MCqpWuBoQCq+ouIhAPRQEoA4zLlLD9fWZKcxswVqcxcmcKipF2F3dTbNqlP71YNuey4NoXTWzRpUP1m3zQV4Jtv4OOP4eGHYfRoCLcZeMtLIBPFb0BHEWkLbAJGAJcW22YDMBiYKCJdgXAgNYAxmXKStieH2YmpzFqZyg+rUtiWuQ8R6NWyIbcP7sQJHZrQtXkkDWzQmgmkRYtcFdMFF8Bll8GJJ0LbtsGOKuQE7L9YVXNF5GZgOq7r69uqulRExgAJqjoNuAt4Q0TuwFVLjVRVq1qqhFSVZZvTmbUylVkrU5j/507y1U2Id0qnGAZ2jmFAxxgrLZiKkZkJjzwCL7/sejOde64bZW1JIiCkqn0ux8fHa0JCQrDDqBbSs3L4KXGbSw6rUgrHLfRsEcWgzjGc0rkpvVs1tFlSTcX67DO45RZISoJRo+Cpp6Bx42BHVemJyHxVjT+cfa1ewBRSVVZtzWTWyhRmrkwhYf1OcvOViPCaDOgUw8BOMZzSOYamEVb3a4Lkjz/gvPOgZ083iO6EE4IdUbVgiaKa252dy89rtjNzZQqzVqSQnJYFQNfmkfx9QDsGdW5K39YNrZuqCZ6cHDer66mnugTx5ZeuN5N1ea0wliiqGVVlTepuZq1MYdbKVOat28G+vHwa1KnJiR2acOvgjpzSOYbmUdVzyUdTyfz8M1x/PSxdCitXum6vw4YFO6pqxxJFNbB3Xx5z17pSw8yVKWzc4cY0dGzagJEnxjGwcwzxbRrbDKqm8tixA+67z83w2qoV/Pe/LkmYoLBEEaL+3L6bmStSmLkylblrt5Odm0/dWmGc2KEJ1w1oz8DOMbRsZKt4mUooKwt694bkZDey+tFHoUGDYEdVrVmiCBFZOXnMW7eDmStT+GFlKmu37QagXXR9LjuuDYO6xHBsXGObJ8lUXklJbp3q8HB4/HGXLHr1CnZUBksUVdrGHXuYtSqVWStS+HnNdvbm5FGnZg36t2/CVSe4KqU2TeoHO0xjSrd3r+vi+swzbmT18OFw1VXBjsr48CtRiEhtoLWqrg5wPKYU+3LzSVjvSg2zVqaSmJIJuLUYLopvycDOTTm+XRPq1rZSg6kiZsxwE/itWQOXXw79+gU7IlOCMhOFiPwFeBGoDbQVkd7AI6p6XqCDM7A5bW/haOg5idvYvS+P2mE1OK5dYy4+thWDujSlXXR9m1DPVD233AL/+hd07OhmfB08ONgRmYPwp0QxBjc9+EwAVV0oItb9IEBy8vL5/c+dzFqVyswVKazYkgFAbFQ45/RpwaDOTTmhfRPq2xxKpirKy3M/w8Lg+OMhOhruvdcm8Kvk/Pm0yVHVXcW+sVateT8quZT0LGatSuWHlanMTkwlIyuXmjWE+LhGjD6zC4O6NKVj0wZWajBV2++/uzERV1zhShOXXRbsiIyf/EkUy0XkIqCGNxPsbcDcwIYV2vLylYUbdxVOlbFkUzoAR0XWYViP5gzqEsOJHaKJCLeRpyYEZGS4qb/HjYOYGGjePNgRmUPkT6K4GXgYyAf+i5sNdnQggwpF2zOzmZ2YyswVrtSwa08OYTWEvq0bcs8ZnRnUuSldm0dYqcGElhkz4Jpr3JiI66+Hf/wDGjYMdlTmEPmTKM5Q1XuBewueEJHzcUnDHER+vvLHpjRvNHQqi73FfKIb1GZwl6MY1CWGkzvEEFXPSg0mhNWuDU2bwiefwHHFV0I2VUWZ04yLyO+q2rfYc/NV9ZiARnYQlXma8V179jE7cRuzVqTww6pUtu92i/n0btWQQZ2bMqhzU7rHRlLDpuU2oSonB158EdLT4ckn3XP5+VDDpocJtoBMMy4iZ+CWKW0hIi/6vBSJq4aq9lSVpcnphRPs/b7BLebTyFvMZ1CXppzcMYbG9WsHO1RjAm/OnP0T+F144f4EYUmiyiut6ikFWAJkAUt9ns8A7gtkUJVZelYOcxK3MdMrNaRkuMV8jm4Zxc2ndmRg5xh6tbTFfEw1sn276+L61lvQujV8/jmcdVawozLl6KCJQlUXAAtEZJKqZlVgTJWOqvL54s1Mmvsn8/90i/lEFizm07kpp3SKISbClgA11dT27TB5Mvzf/7neTfVt2phQ409jdgsReRLoBhSOilHVTgGLqhJZuSWDR6YtYe7aHbSPqc+oAe0Y1KUpfVrZYj6mGlu+HD76yK1b3akTbNhgy5GGMH8SxUTgCeB54EzgaqpBG0VGVg4vfZvIxJ/XExFekyfP68GIY1tblZKp3vbscY3Uzz3npv6+9lo346sliZDmT6Kop6rTReR5VV0DPCgiPwY6sGBRVT5buIl/fLWCbZnZjDi2Nf93RmcaWYO0qe6+/tpN4LdunZvd9bnn3AA6E/L8SRTZ4kaBrRGR64FNQNPAhhUcy5LTeWTaEn5bv5NerRry5pXx9Gplg4OMITPTTb3RpAnMnAkDBwY7IlOB/EkUdwANgFuBJ4Eo4JpABlXR0vbmMPabVbz7y3qi6tbi6fN7clF8KxvvYKq3vDz44AO45BJXzfTtt9ClC9SxjhvVTZmJQlV/9e5mAFcAiEjLQAZVUfLzlU9+T+KZr1ewffc+LjuuNXef3pmG9ayayVRz8+fDdde5n3Xrwl//aqvNVWOlJgoRORZoAcxR1W0i0h03lcepQJVOFks2pfHw1CX8vmEXfVo3ZOLV/ejRIirYYRkTXGlp8NBDMH68m3pj8mQ4//xgR2WCrLSR2U8BfwUW4RqwP8XNHPsMcH3FhFf+0vbk8PyMlUz69U8a1avNsxcczQV9W1o1kzHgSg7ffw833QRPPAFR9uXJlF6iOAfopap7RaQxkOw9XlkxoZWv/HxlyvyNPPP1Snbt2ceV/eO4Y0gnourapHymmlu71vVeiohwXV9r1IBjjw12VKYSKS1RZKnqXgBV3SEiK6pqklictIuHpi5l0cZdHBvXiMfOPo5usZHBDsuY4Nq3D55/Hh5/HG69FZ55xmZ4NSUqLVG0E5GCqcQFiPN5jKpW+orLnbv38ez0lUz+bQNN6tfhxYt6cV6fFrbmgzGzZ7sJ/JYvhwsucInCmIMoLVH8tdjjfwUykPI2/88dXPtOAhlZuVx9QltuH9KRSFsxzhgYOxbuvBPi4uDLL2HYsGBHZCq50iYF/K4iAylPm3bt5br35tOwbi0mjzqeLs2smslUc/n5sHu3a4f4y18gNRUefBDq1Qt2ZKYKCLlZ7fbsy+Xv7ySQnZPPm1fFW5IwZulSOOUUGDnSPe7UyS1JaknC+CmgiUJEhorIShFZLSIlrmEhIheJyDIRWSoi7x/J+VSVe6YsZvmWdMZd0ocOTSOO5HDGVG179sDo0dC7t2uLOOssKGNFS2NK4s8UHgCISB1VzT6E7cOA8cAQIAn4TUSmqeoyn206AqOBE1V1p4gc0RxS//p+NV/+sZnRZ3ZhUJeQnI7KGP8sWOAGyq1fD1dfDc8+C9HRwY7KVFFllihEpJ+I/AEkeo97icg//Th2P2C1qq5V1X3AZNzYDF9/B8ar6k4AVU05pOh9fL1kCy98s4rz+rRg1IB2h3sYY6q2ghJD69bu9sMP8PbbliTMEfGn6mkccBawHUBVFwGD/NivBbDR53GS95yvTkAnEflJROaKyFA/jnuAFVvSufOjhfRq1ZCnzu9p3V9N9ZObCy+9BIMHu8n8mjRxSWLAgGBHZkKAP4mihqr+Wey5PD/2K+nTungFaU2gIzAQuAR4U0QOmNdbREaJSIKIJKSmphZ5bcfuffztnQQa1KnJ61ccQ3itMD9CMyaEzJsH/frBHXdAeDikpwc7IhNi/EkUG0WkH6AiEiYitwOr/NgvCWjl87glbhot+gUCAAAgAElEQVSQ4ttMVdUcVV0HrMQljiJU9XVVjVfV+BifhVJUlVs/WEBKRjavXxnPUZHhxXc1JnRlZro5mY4/HrZuhSlT3LiIRo2CHZkJMf4kihuAO4HWwFbgeO+5svwGdBSRtiJSGxgBTCu2zWd41VgiEo2rilrrX+gwZX4Sc1Zv4+GzutHbFhgy1U2tWjBrFtxyy/4R1lbtagLAn15Puao64lAPrKq5InIzMB0IA95W1aUiMgZIUNVp3muni8gyXHXWPaq63Z/jb8vM5h9fLSe+TSMu7df6UMMzpmpavRrGjHHTgEdEuPUiwq0kbQLLn0Txm4isBD4E/quqGf4eXFW/Ar4q9tzDPvcVV1q5099jFnjii2Xszs7lqfN72hThJvRlZ7surk8+CbVrw9//DiefbEnCVIgyq55UtT3wBHAM8IeIfCYih1zCKE8/Jqby2cJkbjilPR2PskF1JsTNnOlWl3v4YTj3XFixwiUJYyqIXyOzVfVnVb0V6AukA5MCGlVpsQCPTltK2+j63DioQ7DCMKZiqLpSRE4OfP21W3EuNjbYUZlqpsyqJxFpgBsoNwLoCkwFTghwXAe1c/c+tqfu5jXrCmtCVX4+vPUWDB0KrVrBe+9Bw4Zu7WpjgsCfEsUSXE+nZ1W1g6repaq/Bjiug9qansUxbRpxerejghWCMYGzeDGcdBKMGgVvvumea97ckoQJKn8as9upan7AI/FTbr5yy6kdbPS1CS2ZmfDYY26tiEaNYOJEuPLKYEdlDFBKohCRF1T1LuATETlgyslgrXBXs4YwoGNM2RsaU5U8+ii88AL87W/w9NNuCg5jKonSShQfej8r1cp2NWvUsO6wJjRs3OgWE+rSBe67z/VoOumkYEdlzAEO2kahqvO8u11V9TvfG65ROyhaNLK6WlPF5ebCiy9C165w3XXuuehoSxKm0vKnMfuaEp67trwD8Ve92tbTyVRhc+dCfDzcdRcMHAjvvBPsiIwpU2ltFBfjusS2FZH/+rwUAewKdGDGhJwvv4Thw904iP/+11U1WacMUwWU1kYxD7cGRUvcSnUFMoAFgQzKmJChCsnJ0KIFnHaam6fpttvcPE3GVBEHTRTetN/rgG8rLhxjQsiqVXDjje7nsmXQoAE8+GCwozLmkB20jUJEfvB+7hSRHT63nSKyo+JCNKaKycpy3V179oSEBBg92gbMmSqttKqnguVObbFdY/y1ZYtbfjQxES65xPVuatYs2FEZc0RK6x5bMBq7FRCmqnlAf+A6oH4FxGZM1ZGT434edZRLFDNmwPvvW5IwIcGf7rGf4ZZBbQ+8ixtD8X5AozKmqsjPh1dfhfbtISnJ9WJ6800YMiTYkRlTbvxJFPmqmgOcD7ykqrcALQIbljFVwKJFcMIJcMMN0LHj/lKFMSHGn0SRKyIXAlcAX3jP1QpcSMZUcqpw991wzDGwdq2bBvzbb6Ft22BHZkxA+DsyexBumvG1ItIW+CCwYRlTiYnAzp1w7bWwciVcfrkNnDMhTdyy1WVsJFITKFhObrWq5gY0qlLEx8drQkJCsE5vqqs//3QD5R5+GPr2dW0TNfxaINKYSkFE5qtq/OHsW+ZfuoicDKwG3gLeBlaJyImHczJjqpycHHj2WejWDb75xpUgwJKEqVb8WbhoLDBMVZcBiEhX4D3gsDKTMVXGzz+72V2XLIFzzoFx46B162BHZUyF8ydR1C5IEgCqulxEagcwJmMqh2+/hbQ0+OwzlyiMqabKbKMQkYlANq4UAXAZUE9VrwpsaCWzNgoTMKquB1NMDJx5JmRnu6qnBg2CHZkxRyygbRTA9cAa4P+Ae4G1uNHZxoSOFSvg1FPhqqtgwgT3XJ06liSMoYyqJxHpCbQHPlXVZysmJGMq0N698I9/wDPPQP368Nprbt1qY0yh0maPvR83fcdlwDciUtJKd8ZUbZ9/Dk88ARdf7EoVo0ZZjyZjiimtRHEZcLSq7haRGOArXPdYY6q2LVtg4UIYOhQuvBDi4qBfv2BHZUylVdpXp2xV3Q2gqqllbGtM5ZeXB//+N3TuDFdc4aqdRCxJGFOG0koU7XzWyhagve/a2ap6fkAjM6Y8/f47XH89/PabW5L03/+2xYSM8VNpieKvxR7/K5CBGBMw69a5UkN0tFsjYsQIm5vJmENQ2prZ31VkIMaUK1X44w84+mg3q+uECTB8ODRsGOzIjKlyrN3BhJ516+Css6BPH1i82D13xRWWJIw5TAFNFCIyVERWishqEbmvlO0uEBEVEZs/yhy+ffvg6aehe3f44Qd4/nk3mZ8x5oj4M9cTACJSR1WzD2H7MGA8MARIAn4TkWm+80Z520UAtwK/+ntsYw6Ql+dWm5s/H84/H156CVq1CnZUxoQEf6YZ7ycifwCJ3uNeIvJPP47dD7d2xVpV3QdMBkqaWe1x4Fkgy/+wjfGkp7ufYWFwzTVuAN0nn1iSMKYc+VP1NA44C9gOoKqLcCvelaUFsNHncRLF1toWkT5AK1X9glKIyCgRSRCRhNTUVD9ObUKeKkycCO3awdSp7rkbb3RtE8aYcuVPoqihqn8Wey7Pj/1K6n9YOFWtiNTArXVxV1kHUtXXVTVeVeNjYmL8OLUJacuWwcCBcPXV0KULtG8f7IiMCWn+JIqNItIPUBEJE5HbgVV+7JcE+Jb/WwLJPo8jgB7ALBFZDxwPTLMGbVOqZ5+FXr3cYkJvvgmzZ0OPHsGOypiQ5k+iuAG4E2gNbMV9oN/gx36/AR1FpK230NEIYFrBi6qapqrRqhqnqnHAXOBsVbXFJsyBCtZNadYMLrvMTeB37bU2gZ8xFaDMXk+qmoL7kD8kqporIjcD04Ew4G1VXSoiY4AEVZ1W+hGMAZKT4bbb4OST4dZb4cor3c0YU2HKTBQi8gY+bQsFVHVUWfuq6le4WWd9n3v4INsOLOt4phopmMDvgQfcKnMnnBDsiIyptvwZR/Gtz/1w4DyK9mYypnwtXOgWD5o/H04/3SUMa7A2Jmj8qXr60PexiLwHfBOwiIxJS3NVTh9+6NaLsAn8jAkqv0dm+2gLtCnvQEw1pgpTpkBioqtqOuUUWLsWwsODHZkxBv9GZu8UkR3ebReuNHF/4EMz1cKaNTBsmFuKdOpU1x4BliSMqURKLVGIiAC9gE3eU/mqekDDtjGHLDvbTdr3xBNQqxa8/LIbWV3zcAq5xphAKrVE4SWFT1U1z7tZkjDlY+NGePxxN+XG8uWu66slCWMqJX9GK80Tkb4Bj8SEvtRU+Je3UGKHDm4qjilToEWL0vczxgTVQROFiBR8vTsJlyxWisjvIrJARH6vmPBMSMjPh7fecvMy3XknrFzpnm/XLrhxGWP8UlpZfx7QFzi3gmIxoWjJErjhBpgzx42ufvVV6Nw52FEZYw5BaYlCAFR1TQXFYkLNvn1uwNy+ffD22zBypI2JMKYKKi1RxIjInQd7UVVfDEA8JhR8/70bC1G7Nnz0katyio4OdlTGmMNUWmN2GNAANx14STdjikpKgr/+FQYPhnffdc+ddJIlCWOquNJKFJtVdUyFRWKqrtxc15vpoYfcZH5PPeWmAjfGhIQy2yiMKdMVV8DkyXDmmTB+PLRtG+yIjDHlqLREMbjCojBVz65dboBcgwZw002uyumvf7XGamNC0EHbKFR1R0UGYqoIVVd66NrVVTWBa4e44AJLEsaEKFtH0vhv9Wo44wy45BJo2RIuvzzYERljKoAlCuOf99+HHj3g119dw/XcuXDMMcGOyhhTAWwWNlO6nBw3u2t8vKteevZZiI0NdlTGmApkJQpTspQU15vp4ovd406d4D//sSRhTDVkicIUlZ8Pr7/u5mP68EPo3t2NjTDGVFtW9WT2W7vWNVD/8gsMHAivvOKm3zDGVGuWKMx+UVFufMQ777hqJ+vuaozBqp7MtGlw/vmueqlJEzct+JVXWpIwxhSyRFFdbdgA554L55wDq1bB5s3u+Rr2J2GMKco+Faqb3Fx4/nk3snrGDHjmGViwwA2gM8aYElgbRXWTlwdvvgmnngr//CfExQU7ImNMJWcliupg5064917IyIA6deCnn1zbhCUJY4wfLFGEMlWYNMl1cX3hBZg50z3fpIk1Vhtj/GaJIlStWgVDhrhxEXFxkJAAZ58d7KiMMVWQtVGEqttvd8nh3/+GUaMgLCzYERljqihLFKHkm29cNVOrVm5UdZ060KxZsKMyxlRxAa16EpGhIrJSRFaLyH0lvH6niCwTkcUi8p2ItAlkPCFryxa49FI4/XTX3RWgTRtLEsaYchGwRCEiYcB44EygG3CJiHQrttkCIF5VjwY+Bp4NVDwhKT8fXn3VlSI++QQeecSNkTDGmHIUyBJFP2C1qq5V1X3AZOAc3w1Udaaq7vEezgVs1NeheOopuOEGt4DQ4sXw6KMQHh7sqIwxISaQbRQtgI0+j5OA40rZ/lrgfyW9ICKjgFEArVu3Lq/4qqaMDNi2Ddq2heuvdz8vucS6uxpjAiaQJYqSPrm0xA1FLgfigedKel1VX1fVeFWNj4mJKccQqxBV+PRT6NbNLSak6sZDXHqpJQljTEAFMlEkAa18HrcEkotvJCKnAQ8AZ6tqdgDjqbr+/NONgTj/fGjcGMaNs+RgjKkwgax6+g3oKCJtgU3ACOBS3w1EpA/wGjBUVVMCGEvV9csvcNpp7v7zz8Ntt0FN69VsjKk4AStRqGoucDMwHVgOfKSqS0VkjIgUDBF+DmgATBGRhSIyLVDxVDnp6e5n375wzTWwfDncdZclCWNMhRPVEpsNKq34+HhNSEgIdhiBs3073HefmwJ86VJo0CDYERljQoCIzFfV+MPZ1+Z6qixU4d133ZiICRNcg7W1QxhjKgGrx6gM0tLcanOzZkH//m4Q3dFHBzsqY4wBLFEEl6orNURGQnQ0vP46XHutLUdqjKlU7BMpWKZPdw3VSUkuWUyZAn//uyUJY0ylY59KFW3zZhgxAoYOhT17IMV6BRtjKjdLFBVp/HjXWP3ZZ/DYY25+pr59gx2VMcaUytooKtL8+XDccS5hdOwY7GiMMcYvVqIIpPR0t9Lc/Pnu8b//7domLEkYY6oQSxSBoAoffwxdu7p5mX74wT0fHm5jI4wxVY4livK2bh2cdRZceCE0bermarrzzmBHZYwxh80SRXmbNAlmz4axY+G331ybhDHGVGHWmF0efvwRsrPdLK/33AMjR0JLW6zPGBMarERxJLZtczO7DhgAY8a45+rUsSRhjAkpVqI4HKowcaIrPaSlwb33wkMPBTsqcxhycnJISkoiKysr2KEYUy7Cw8Np2bIltWrVKrdjWqI4HF995UoSJ57oJvDr0SPYEZnDlJSUREREBHFxcYj1SDNVnKqyfft2kpKSaNu2bbkd16qe/LVnD/z0k7s/bBhMneoarS1JVGlZWVk0adLEkoQJCSJCkyZNyr2EbInCH//7n0sIZ54Ju3a5sRBnn20T+IUISxImlATi79k+6UqzaZMbDzFsmGuk/vxzaNgw2FEZY0yFskRxMCkp0K0bfPEFPPEELFoEp5wS7KhMCAoLC6N379706NGD4cOHs2vXrnI57vr16+kRgKrRRx99lBYtWtC7d2969+7NfffdV+7nKLBw4UK++uqrg76+YMEC/va3vwXs/OXhqaeeokOHDnTu3Jnp06eXuM3JJ59ceD1jY2M599xzAUhLS2P48OH06tWL7t27M2HCBABSU1MZOnRohb0HSxTFbdrkfjZtCo8/DkuWwAMPQO3awY3LhKy6deuycOFClixZQuPGjRk/fnywQyrTHXfcwcKFC1m4cCFPP/203/vl5eUd0nnKShT/+Mc/uOWWW/w+Xm5u7iGd/0gtW7aMyZMns3TpUr7++mtuvPHGEq/Bjz/+WHg9+/fvz/nnnw/A+PHj6datG4sWLWLWrFncdddd7Nu3j5iYGJo3b85PBe2mAWa9ngqkpcGDD8Jrr8HcuW7671tvDXZUpgI99vlSliWnl+sxu8VG8sjw7n5v379/fxYvXgxAZmYm55xzDjt37iQnJ4cnnniCc845h/Xr13PmmWdy0kkn8fPPP9OiRQumTp1K3bp1mT9/Ptdccw316tXjpJNOKjxuVlYWN9xwAwkJCdSsWZMXX3yRQYMGMXHiRD777DPy8vJYsmRJ4QfRe++9R506dfjqq69o3LixX7F/99133H333eTm5nLsscfyyiuvUKdOHeLi4rjmmmuYMWMGN998M8ceeyw33XQTqamp1KtXjzfeeIMuXbowZcoUHnvsMcLCwoiKiuLbb7/l4YcfZu/evcyZM4fRo0dz8cUXF54vIyODxYsX06tXLwDmzZvH7bffzt69e6lbty4TJkygc+fOTJw4kS+//JKsrCx2797N999/z3PPPcdHH31EdnY25513Ho899hgA5557Lhs3biQrK4vbbruNUaNG+f27K8nUqVMZMWIEderUoW3btnTo0IF58+bRv3//ErfPyMjg+++/Lyw5iAgZGRmoKpmZmTRu3JiaNWsWxjpp0iROPPHEI4rRH1aiUIWPPnIT+I0fD9dfD+3bBzsqUw3l5eXx3XffcfbZZwOuP/ynn37K77//zsyZM7nrrrtQVQASExO56aabWLp0KQ0bNuSTTz4B4Oqrr2bcuHH88ssvRY5dUEr5448/+OCDD7jqqqsKe8YsWbKE999/n3nz5vHAAw9Qr149FixYQP/+/Xn33XdLjHXs2LGFVSXTp08nKyuLkSNH8uGHH/LHH3+Qm5vLK6+8Urh9eHg4c+bMYcSIEYwaNYp//vOfzJ8/n+eff54bb7wRgDFjxjB9+nQWLVrEtGnTqF27NmPGjOHiiy9m4cKFRZIEQEJCQpGqtS5dujB79mwWLFjAmDFjuP/++wtf++WXX3jnnXf4/vvvmTFjBomJicybN4+FCxcyf/58Zs+eDcDbb7/N/PnzSUhIYNy4cWzfvv2A937HHXcUvnffW0klq02bNtGqVavCxy1btmRTQa1FCT799FMGDx5MZGQkADfffDPLly8nNjaWnj178vLLL1PD60QTHx/Pjz/+eNBjlafqXaJQhfPPdwsJ9e0L06ZBfHywozJBcijf/MvT3r176d27N+vXr+eYY45hyJAhgOsTf//99zN79mxq1KjBpk2b2Lp1KwBt27ald+/eABxzzDGsX7+etLQ0du3axSleW9oVV1zB//73PwDmzJlTWEXTpUsX2rRpw6pVqwAYNGgQERERREREEBUVxfDhwwHo2bNnYemmuDvuuIO777678PGiRYto27YtnTp1AuCqq65i/Pjx3H777QCFH/KZmZn8/PPPXHjhhYX7ZmdnA3DiiScycuRILrroosKql9Js3ryZmJiYwsdpaWlcddVVJCYmIiLk5OQUvjZkyJDCktGMGTOYMWMGffr0KYwpMTGRAQMGMG7cOD799FMANm7cSGJiIk2aNCly3rFjx5YZW4GCxO6rtF5JH3zwQZE2l+nTp9O7d2++//571qxZw5AhQzj55JOJjIykadOmJCcn+x3LkaieiSInB2rVct1cTzoJTj0VbrwRwsKCHZmphgraKNLS0jjrrLMYP348t956K5MmTSI1NZX58+dTq1Yt4uLiCksBderUKdw/LCyMvXv3oqoH/RAq6QOrgO+xatSoUfi4Ro0aftfpl3Z8gPr16wOQn59Pw4YNWbhw4QHbvPrqq/z66698+eWX9O7du8RtfNWtW7fIeIGHHnqIQYMG8emnn7J+/XoGDhx4wPkLYh09ejTXXXddkePNmjWLb7/9ll9++YV69eoxcODAEscj3HHHHcycOfOA50eMGHFAw37Lli3ZuHFj4eOkpCRiY2NLfD/bt29n3rx5hYkKYMKECdx3332ICB06dKBt27asWLGCfv36kZWVRd26dQ9ydcpX9at6mjULjj7aDZgDuOsuuOUWSxIm6KKiohg3bhzPP/88OTk5pKWl0bRpU2rVqsXMmTP5888/S92/YcOGREVFMWfOHAAmTZpU+NqAAQMKH69atYoNGzbQuXPncou9S5curF+/ntWrVwPw3nvvFZZsfEVGRtK2bVumTJkCuA/tRYsWAbBmzRqOO+44xowZQ3R0NBs3biQiIoKMjIwSz9m1a9fC84ErUbRo0QKAiRMnHjTWM844g7fffpvMzEzAVQ+lpKSQlpZGo0aNqFevHitWrGDu3Lkl7j927NjChmffW0m9v84++2wmT55MdnY269atIzExkX79+pV43ClTpnDWWWcRHh5e+Fzr1q357rvvANi6dSsrV66kXbt2gPs9BqJXW0mqT6JITYWrroJBg9xMrxERwY7ImAP06dOHXr16MXnyZC677DISEhKIj49n0qRJdOnSpcz9J0yYwE033UT//v2LfNss6G3Ts2dPLr74YiZOnFikJHGkwsPDmTBhAhdeeCE9e/akRo0aXH/99SVuO2nSJN56663CLp9TvS9t99xzDz179qRHjx4MGDCAXr16MWjQIJYtW0bv3r358MMPixynS5cupKWlFSaS//u//2P06NGceOKJpfauOv3007n00kvp378/PXv25IILLiAjI4OhQ4eSm5vL0UcfzUMPPcTxxx9/xNele/fuXHTRRXTr1o2hQ4cyfvx4wrwvpcOGDStSdTR58mQuueSSIvs/9NBD/Pzzz/Ts2ZPBgwfzzDPPEB0dDcDMmTP5y1/+csQx+kPKKjJWNvHx8ZqQkHBoO33wAdx0E2Rmuon8HngA6tULTICmSlm+fDldu3YNdhjmMI0dO5aIiIhKP5YiEAYMGMDUqVNp1KjRAa+V9HctIvNV9bAaYatHiSI3103BsXAhPPmkJQljQsQNN9xQriWjqiI1NZU777yzxCQRCKFZoti92w2Wa93aNVIXvEeb08cUYyUKE4qsRFGWL76A7t3hmWfA6/6HiCUJc1BV7cuSMaUJxN9z6CSKpCQ3JmL4cKhf300B/tJLwY7KVHLh4eFs377dkoUJCQXrUfj2nCoPoTOOYu1amD4dnnoK7rzT5mYyfmnZsiVJSUmkpqYGOxRjykXBCnflqWoninnz4Jdf4Lbb3LrVGzZAsVGUxpSmVq1a5boSmDGhKKBVTyIyVERWishqETlgNIqI1BGRD73XfxWROL8OvGuXa6Q+/nh48UXXeA2WJIwxJgAClihEJAwYD5wJdAMuEZFuxTa7Ftipqh2AscAzZR54xw7o0sXN8nrrrfDHH65NwhhjTEAEskTRD1itqmtVdR8wGTin2DbnAO949z8GBktZ6/itXw+tWsFvv7nGam+WRWOMMYERyDaKFsBGn8dJwHEH20ZVc0UkDWgCbPPdSERGAQUTw2dLQsISjjkmIEFXMdEUu1bVmF2L/exa7GfXYr/DntwrkImipJJB8T6I/myDqr4OvA4gIgmHO2gk1Ni12M+uxX52Lfaza7GfiBzi3Ef7BbLqKQlo5fO4JVB88vTCbUSkJhAF7AhgTMYYYw5RIBPFb0BHEWkrIrWBEcC0YttMA67y7l8AfK828skYYyqVgFU9eW0ONwPTgTDgbVVdKiJjgARVnQa8BbwnIqtxJYkRfhz69UDFXAXZtdjPrsV+di32s2ux32Ffiyo3KaAxxpiKFTpzPRljjAkISxTGGGNKVWkTRcCm/6iC/LgWd4rIMhFZLCLfiUibYMRZEcq6Fj7bXSAiKiIh2zXSn2shIhd5fxtLReT9io6xovjxP9JaRGaKyALv/2RYMOIMNBF5W0RSRGTJQV4XERnnXafFItLXrwOraqW74Rq/1wDtgNrAIqBbsW1uBF717o8APgx23EG8FoOAet79G6rztfC2iwBmA3OB+GDHHcS/i47AAqCR97hpsOMO4rV4HbjBu98NWB/suAN0LQYAfYElB3l9GPA/3Bi244Ff/TluZS1RBGb6j6qpzGuhqjNVdY/3cC5uzEoo8ufvAuBx4FkgqyKDq2D+XIu/A+NVdSeAqqZUcIwVxZ9roUDBfD9RHDimKySo6mxKH4t2DvCuOnOBhiLSvKzjVtZEUdL0Hy0Oto2q5gIF03+EGn+uha9rcd8YQlGZ10JE+gCtVPWLigwsCPz5u+gEdBKRn0RkrogMrbDoKpY/1+JR4HIRSQK+Am6pmNAqnUP9PAEq73oU5Tb9Rwjw+32KyOVAPHBKQCMKnlKvhYjUwM1CPLKiAgoif/4uauKqnwbiSpk/ikgPVd0V4Ngqmj/X4hJgoqq+ICL9ceO3eqhqfuDDq1QO63OzspYobPqP/fy5FojIacADwNmqml1BsVW0sq5FBNADmCUi63F1sNNCtEHb3/+Rqaqao6rrgJW4xBFq/LkW1wIfAajqL0A4bsLA6savz5PiKmuisOk/9ivzWnjVLa/hkkSo1kNDGddCVdNUNVpV41Q1Dtdec7aqHvZkaJWYP/8jn+E6OiAi0biqqLUVGmXF8OdabAAGA4hIV1yiqI7r304DrvR6Px0PpKnq5rJ2qpRVTxq46T+qHD+vxXNAA2CK156/QVXPDlrQAeLntagW/LwW04HTRWQZkAfco6rbgxd1YPh5Le4C3hCRO3BVLSND8YuliHyAq2qM9tpjHgFqAajqq7j2mWHAamAPcLVfxw3Ba2WMMaYcVdaqJ2OMMZWEJQpjjDGlskRhjDGmVJYojDHGlMoShTHGmFJZojCVjojkichCn1tcKdvGHWymzEM85yxv9tFF3pQXnQ/jGNeLyJXe/ZEiEuvz2psi0q2c4/xNRHr7sc/tIlLvSM9tqi9LFKYy2quqvX1u6yvovJepai/cZJPPHerOqvqqqr7rPRwJxPq89jdVXVYuUe6P89/4F+ftgCUKc9gsUZgqwSs5/Cgiv3u3E0rYpruIzPNKIYtFpKP3/OU+z78mImFlnG420MHbd7C3hsEf3lz/dbznn5b9a4A87z33qIjcLSIX4ObcmuSds65XEogXkRtE5Blt1K0AAAMTSURBVFmfmEeKyD8PM85f8JnQTUReEZEEcWtPPOY9dysuYc0UkZnec6eLyC/edZwiIg3KOI+p5ixRmMqork+106fecynAEFXtC1wMjCthv+uBl1W1N+6DOsmbruFi4ETv+TzgsjLOPxz4Q0TCgYnAxaraEzeTwQ0i0hg4D+iuqkcDT/jurKofAwm4b/69VXWvz8sfA+f7PL4Y+PAw4xyKm6ajwAOqGg8cDZwiIker6jjcXD6DVHWQN5XHg8Bp3rVMAO4s4zymmquUU3iYam+v92HpqxbwL69OPg83b1FxvwAPiEhL4L+qmigig4FjgN+86U3q4pJOSSaJyF5gPW4a6s7AOlVd5b3+DnAT8C/cWhdvisiXgN9Tmqtqqois9ebZSfTO8ZN33EOJsz5uugrfFcouEpFRuP/r5rgFehYX2/d47/mfvPPUxl03Yw7KEoWpKu4AtgK9cCXhAxYlUtX3ReRX4C/AdBH5G25a5XdUdbQf57jMdwJBESlxfRNvbqF+uEnmRgA3A6cewnv5ELgIWAF8qqoq7lPb7zhxq7g9DYwHzheRtsDdwLGqulNEJuImvitOgG9U9ZJDiNdUc1b1ZKqKKGCzt37AFbhv00WISDtgrVfdMg1XBfP/7d29SgQxFIbh92stFCy01MLCSxC8AjsLwUq8CW/DVhYLsRMRrBYFRWS7xcrfxXuwEJEFQfBYnIzFMhvdUviebkLIZKbIISchuQI2JM2VOrP6+53iz8CipKXyvAX0Sk5/JiLOyIXitp1H7+Sx521OgXXyjoTjUjZRPyPik0whrZS01TQwBN4kzQNrY/rSB1abb5I0Jaltdmb2w4HC/os9YFtSn0w7DVvqbAKPkm6BZfLKxwE5oF5IugcuybTMryLigzxd80TSA/AFdMhBt1va65GznVGHQKdZzB5p9xUYAAsRcVPKJu5nWfvYBXYi4o68H/sJOCDTWY194FzSdUS8kDuyjsp7+uS/MhvLp8eamVmVZxRmZlblQGFmZlUOFGZmVuVAYWZmVQ4UZmZW5UBhZmZVDhRmZlb1Da0OoKWgExquAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x231e90517b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "plt.plot(fpr, tpr, label='Random Forest (area = %0.2f)' % rf_roc_auc_score)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Receiver operating characteristic')\n",
    "plt.legend(loc=\"lower right\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the ROC, if we care about minimising false positive, we would choose a cutoff that would give us about 0.5 true positive rate. <br>\n",
    "However if we care about maximizing the true positive, we will have to decrease the cut-off. This way, we will classify more events as \"1\"-- true positive goes up, meanwhile, false positive will also go up.<br>\n",
    "In this case, I believe we should increase the true positive rate. Because we want to catch more fraud, just to be cautious. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.2.4 Feature Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get a score telling how importance each feature is in classifying\n",
    "feature_importance = pd.DataFrame(list(zip(X.columns,rf.feature_importances_)), columns = ['feature','score'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>device_occ</td>\n",
       "      <td>0.242327</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ip_occ</td>\n",
       "      <td>0.141782</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>purchase_weekofyear</td>\n",
       "      <td>0.135216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>purchase_signup_diff</td>\n",
       "      <td>0.124294</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>signup_weekofyear</td>\n",
       "      <td>0.099185</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>purchase_value</td>\n",
       "      <td>0.071053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>purchase_dayofweek</td>\n",
       "      <td>0.037560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>signup_dayofweek</td>\n",
       "      <td>0.035797</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>sex__M</td>\n",
       "      <td>0.011800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>source__SEO</td>\n",
       "      <td>0.009839</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>browser__IE</td>\n",
       "      <td>0.009460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>61</th>\n",
       "      <td>country__United States</td>\n",
       "      <td>0.008155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>browser__Safari</td>\n",
       "      <td>0.007013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>browser__FireFox</td>\n",
       "      <td>0.006581</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>country__Not_found</td>\n",
       "      <td>0.006226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>source__Direct</td>\n",
       "      <td>0.004936</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>country__China</td>\n",
       "      <td>0.004416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>country__Japan</td>\n",
       "      <td>0.003083</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>60</th>\n",
       "      <td>country__United Kingdom</td>\n",
       "      <td>0.002930</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>country__Korea Republic of</td>\n",
       "      <td>0.002449</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>browser__Opera</td>\n",
       "      <td>0.002428</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>country__Germany</td>\n",
       "      <td>0.002191</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>country__Canada</td>\n",
       "      <td>0.001977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>country__Others</td>\n",
       "      <td>0.001923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>country__France</td>\n",
       "      <td>0.001874</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>country__Brazil</td>\n",
       "      <td>0.001811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>country__Netherlands</td>\n",
       "      <td>0.001631</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>country__Italy</td>\n",
       "      <td>0.001626</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>country__Australia</td>\n",
       "      <td>0.001378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>country__Russian Federation</td>\n",
       "      <td>0.001100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>country__South Africa</td>\n",
       "      <td>0.000779</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>country__Colombia</td>\n",
       "      <td>0.000762</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55</th>\n",
       "      <td>country__Switzerland</td>\n",
       "      <td>0.000700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>country__Norway</td>\n",
       "      <td>0.000692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>country__Denmark</td>\n",
       "      <td>0.000640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>53</th>\n",
       "      <td>country__Spain</td>\n",
       "      <td>0.000589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>country__Belgium</td>\n",
       "      <td>0.000553</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>country__Finland</td>\n",
       "      <td>0.000539</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>63</th>\n",
       "      <td>country__Viet Nam</td>\n",
       "      <td>0.000525</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>country__Egypt</td>\n",
       "      <td>0.000489</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>country__Turkey</td>\n",
       "      <td>0.000484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>country__Greece</td>\n",
       "      <td>0.000471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>country__Austria</td>\n",
       "      <td>0.000465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>country__Chile</td>\n",
       "      <td>0.000459</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>country__Indonesia</td>\n",
       "      <td>0.000425</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>country__Hong Kong</td>\n",
       "      <td>0.000420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>country__New Zealand</td>\n",
       "      <td>0.000412</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>country__Saudi Arabia</td>\n",
       "      <td>0.000387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>country__Iran (ISLAMIC Republic Of)</td>\n",
       "      <td>0.000387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>country__Romania</td>\n",
       "      <td>0.000368</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>country__Hungary</td>\n",
       "      <td>0.000342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>country__Portugal</td>\n",
       "      <td>0.000320</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>country__Israel</td>\n",
       "      <td>0.000318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>59</th>\n",
       "      <td>country__Ukraine</td>\n",
       "      <td>0.000301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>country__Malaysia</td>\n",
       "      <td>0.000292</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>57</th>\n",
       "      <td>country__Thailand</td>\n",
       "      <td>0.000291</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>country__Ireland</td>\n",
       "      <td>0.000288</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>country__Czech Republic</td>\n",
       "      <td>0.000274</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>62</th>\n",
       "      <td>country__Venezuela</td>\n",
       "      <td>0.000267</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>country__European Union</td>\n",
       "      <td>0.000251</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>65 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                feature     score\n",
       "1                            device_occ  0.242327\n",
       "2                                ip_occ  0.141782\n",
       "6                   purchase_weekofyear  0.135216\n",
       "0                  purchase_signup_diff  0.124294\n",
       "5                     signup_weekofyear  0.099185\n",
       "64                       purchase_value  0.071053\n",
       "4                    purchase_dayofweek  0.037560\n",
       "3                      signup_dayofweek  0.035797\n",
       "13                               sex__M  0.011800\n",
       "8                           source__SEO  0.009839\n",
       "10                          browser__IE  0.009460\n",
       "61               country__United States  0.008155\n",
       "12                      browser__Safari  0.007013\n",
       "9                      browser__FireFox  0.006581\n",
       "45                   country__Not_found  0.006226\n",
       "7                        source__Direct  0.004936\n",
       "20                       country__China  0.004416\n",
       "38                       country__Japan  0.003083\n",
       "60              country__United Kingdom  0.002930\n",
       "39           country__Korea Republic of  0.002449\n",
       "11                       browser__Opera  0.002428\n",
       "28                     country__Germany  0.002191\n",
       "18                      country__Canada  0.001977\n",
       "46                      country__Others  0.001923\n",
       "27                      country__France  0.001874\n",
       "17                      country__Brazil  0.001811\n",
       "42                 country__Netherlands  0.001631\n",
       "37                       country__Italy  0.001626\n",
       "14                   country__Australia  0.001378\n",
       "50          country__Russian Federation  0.001100\n",
       "..                                  ...       ...\n",
       "52                country__South Africa  0.000779\n",
       "21                    country__Colombia  0.000762\n",
       "55                 country__Switzerland  0.000700\n",
       "44                      country__Norway  0.000692\n",
       "23                     country__Denmark  0.000640\n",
       "53                       country__Spain  0.000589\n",
       "16                     country__Belgium  0.000553\n",
       "26                     country__Finland  0.000539\n",
       "63                    country__Viet Nam  0.000525\n",
       "24                       country__Egypt  0.000489\n",
       "58                      country__Turkey  0.000484\n",
       "29                      country__Greece  0.000471\n",
       "15                     country__Austria  0.000465\n",
       "19                       country__Chile  0.000459\n",
       "33                   country__Indonesia  0.000425\n",
       "30                   country__Hong Kong  0.000420\n",
       "43                 country__New Zealand  0.000412\n",
       "51                country__Saudi Arabia  0.000387\n",
       "34  country__Iran (ISLAMIC Republic Of)  0.000387\n",
       "49                     country__Romania  0.000368\n",
       "31                     country__Hungary  0.000342\n",
       "48                    country__Portugal  0.000320\n",
       "36                      country__Israel  0.000318\n",
       "59                     country__Ukraine  0.000301\n",
       "40                    country__Malaysia  0.000292\n",
       "57                    country__Thailand  0.000291\n",
       "35                     country__Ireland  0.000288\n",
       "22              country__Czech Republic  0.000274\n",
       "62                   country__Venezuela  0.000267\n",
       "25              country__European Union  0.000251\n",
       "\n",
       "[65 rows x 2 columns]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_importance.sort_values(by = 'score', ascending = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Observations:<br>\n",
    "- Using the same device and ip address are the most important ones that detect a fraud.<br>\n",
    "- Sign up/Purchse time difference<br>\n",
    "- Purchase value<br>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}