Mini-Project Part 4 Supervised Learning (2).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ca4c60c3",
   "metadata": {},
   "source": [
    "## Mini-Project Part 4: Supervised Learning (2)\n",
    "### Naive Bayes, SVM, and Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4439f587",
   "metadata": {},
   "source": [
    "Continue to use the dataset about adaptivity to online education. Encode data to integers (code from part 1)\n",
    "###### Results at bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "a22bb293",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Age</th>\n",
       "      <th>Network Type</th>\n",
       "      <th>Class Duration</th>\n",
       "      <th>Gender_Boy</th>\n",
       "      <th>Gender_Girl</th>\n",
       "      <th>Education Level_College</th>\n",
       "      <th>Education Level_School</th>\n",
       "      <th>Education Level_University</th>\n",
       "      <th>Institution Type_Government</th>\n",
       "      <th>Institution Type_Non Government</th>\n",
       "      <th>...</th>\n",
       "      <th>Financial Condition_Mid</th>\n",
       "      <th>Financial Condition_Poor</th>\n",
       "      <th>Financial Condition_Rich</th>\n",
       "      <th>Internet Type_Mobile Data</th>\n",
       "      <th>Internet Type_Wifi</th>\n",
       "      <th>Self Lms_No</th>\n",
       "      <th>Self Lms_Yes</th>\n",
       "      <th>Device_Computer</th>\n",
       "      <th>Device_Mobile</th>\n",
       "      <th>Device_Tab</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Age  Network Type  Class Duration  Gender_Boy  Gender_Girl  \\\n",
       "0    5             4               2           1            0   \n",
       "1    5             4               1           0            1   \n",
       "2    4             4               1           0            1   \n",
       "\n",
       "   Education Level_College  Education Level_School  \\\n",
       "0                        0                       0   \n",
       "1                        0                       0   \n",
       "2                        1                       0   \n",
       "\n",
       "   Education Level_University  Institution Type_Government  \\\n",
       "0                           1                            0   \n",
       "1                           1                            0   \n",
       "2                           0                            1   \n",
       "\n",
       "   Institution Type_Non Government  ...  Financial Condition_Mid  \\\n",
       "0                                1  ...                        1   \n",
       "1                                1  ...                        1   \n",
       "2                                0  ...                        1   \n",
       "\n",
       "   Financial Condition_Poor  Financial Condition_Rich  \\\n",
       "0                         0                         0   \n",
       "1                         0                         0   \n",
       "2                         0                         0   \n",
       "\n",
       "   Internet Type_Mobile Data  Internet Type_Wifi  Self Lms_No  Self Lms_Yes  \\\n",
       "0                          0                   1            1             0   \n",
       "1                          1                   0            0             1   \n",
       "2                          0                   1            1             0   \n",
       "\n",
       "   Device_Computer  Device_Mobile  Device_Tab  \n",
       "0                0              0           1  \n",
       "1                0              1           0  \n",
       "2                0              1           0  \n",
       "\n",
       "[3 rows x 26 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "data = pd.read_csv(r\"C:\\Users\\roryq\\Downloads\\online_adapt.csv\")\n",
    "\n",
    "# Encode `Age` to integers, 1, 2, 3, 4, 5, 6.\n",
    "age_mapper = {'26-30':6, '21-25':5, '16-20':4, '11-15':3, '6-10':2, '1-5':1}\n",
    "age_t = data['Age'].replace(age_mapper)\n",
    "\n",
    "# Encode `Network Type` to integers, 2, 3, 4.\n",
    "net_mapper = {'2G':2, '3G':3, '4G':4}\n",
    "net_t = data['Network Type'].replace(net_mapper)\n",
    "\n",
    "# Encode `Class Duration` to integers, 0, 1, 2.\n",
    "class_mapper = {'0':0, '1-3':1, '3-6':2}\n",
    "class_t = data['Class Duration'].replace(class_mapper)\n",
    "\n",
    "# Replace `Age`, `Network Type`, `Class Duration` by their corresponding numeric versions.\n",
    "data['Age'] = age_t\n",
    "data['Network Type'] = net_t\n",
    "data['Class Duration'] = class_t\n",
    "\n",
    "# One-hot encode the rest of the variables except for the response variable, `Adaptivity Level`.\n",
    "y = data['Adaptivity Level']\n",
    "data1 = pd.get_dummies(data.drop('Adaptivity Level', axis=1), dtype=int )\n",
    "data1.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c2ef5106",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1205, 26)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cc04e71",
   "metadata": {},
   "source": [
    "###  Gaussian Naive Bayes \n",
    "Fit a Gaussian Naive Bayes model on `data1` and `y`. \n",
    "\n",
    "\n",
    "1. Split data (80% training; 20% test) (set `random_state=100`)\n",
    "2. Calculate the confusion matrix for the test set.\n",
    "3. Report the accuracy as well as the precision, recall, and the F1 score for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "25cff178",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        High       0.57      0.41      0.47        32\n",
      "         Low       0.75      0.58      0.65       119\n",
      "    Moderate       0.50      0.70      0.58        90\n",
      "\n",
      "    accuracy                           0.60       241\n",
      "   macro avg       0.61      0.56      0.57       241\n",
      "weighted avg       0.63      0.60      0.60       241\n",
      "\n",
      "Accuracy:  0.6016597510373444\n",
      "  \n",
      "Confusion matrix:\n",
      "[[13  3  7]\n",
      " [ 3 69 20]\n",
      " [16 47 63]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.metrics import classification_report, accuracy_score\n",
    "\n",
    "# Create features and target for each model in this part\n",
    "# Target is adaptability level\n",
    "# Features are all other variables in the data set (available in readme file)\n",
    "features = data1\n",
    "target = y\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features, target, random_state=100, test_size=.2)\n",
    "\n",
    "# select classifer for Gaussian\n",
    "classifier_NB = GaussianNB()\n",
    "\n",
    "# Create model with classifier and train\n",
    "model = classifier_NB.fit(X_train, y_train)  \n",
    "\n",
    "# Create predictions from model with test data\n",
    "ypred=model.predict(X_test)\n",
    "\n",
    "# Find accuracy\n",
    "#create confusion matrix with accuracy for test and predicted values from model\n",
    "accuracy = accuracy_score(y_test,ypred)\n",
    "report = classification_report(ypred, y_test)\n",
    "cm = confusion_matrix(y_test, ypred)\n",
    "\n",
    "print(\"Classification report:\")\n",
    "\n",
    "print(report)\n",
    "print(\"Accuracy: \",accuracy)\n",
    "print(\"  \")\n",
    "print(\"Confusion matrix:\")\n",
    "print(cm)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f9d8f53",
   "metadata": {},
   "source": [
    "###  Bernoulli Naive Bayes \n",
    "\n",
    "The performance of the Gaussian Naive Bayes model is very poor. Let's try to figure out the reason. Check the original data again. Most of the features are categorical variables. Though we've converted them to integers or dummy variables, they are not normal in nature. Probably a Bernoulli Naive Bayes model will fit the data better."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b3352e",
   "metadata": {},
   "source": [
    "All variables are `object`, i.e. categorical variables! Now, extract the `y` variable and convert all features to dummies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "acd67193",
   "metadata": {},
   "outputs": [],
   "source": [
    "y=data['Adaptivity Level']\n",
    "X_dummy = pd.get_dummies(data.drop(['Adaptivity Level'], axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b310a7d",
   "metadata": {},
   "source": [
    "Now, use a Bernoulli Naive model to fit the data, i.e. `X_dummy` and `y`.\n",
    "\n",
    "1. Split data (80% training; 20% test) (set `random_state=100`)\n",
    "2. Calculate the confusion matrix for the test set.\n",
    "3. Report the accuracy as well as the precision, recall, and the F1 score for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e83f0c45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        High       0.70      0.67      0.68        24\n",
      "         Low       0.59      0.62      0.60        87\n",
      "    Moderate       0.70      0.68      0.69       130\n",
      "\n",
      "    accuracy                           0.66       241\n",
      "   macro avg       0.66      0.65      0.66       241\n",
      "weighted avg       0.66      0.66      0.66       241\n",
      "\n",
      "Accuracy:  0.6556016597510373\n",
      "  \n",
      "Confusion matrix:\n",
      "[[16  2  5]\n",
      " [ 1 54 37]\n",
      " [ 7 31 88]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.naive_bayes import BernoulliNB\n",
    "\n",
    "# Specify features and target for model\n",
    "features = X_dummy\n",
    "target = y\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features, target, random_state=100, test_size=.2)\n",
    "\n",
    "# select classifer for Bernoulli\n",
    "classifer_BNB = BernoulliNB()\n",
    "\n",
    "# Create model with classifier and train\n",
    "model_BNB = classifer_BNB.fit(X_train, y_train) \n",
    "\n",
    "# Create predictions from model with test data\n",
    "ypred1=model_BNB.predict(X_test)\n",
    "\n",
    "# Find accuracy\n",
    "#create confusion matrix with accuracy for test and predicted values from model\n",
    "accuracy = accuracy_score(y_test,ypred1)\n",
    "report = classification_report(ypred1, y_test)\n",
    "cm = confusion_matrix(y_test, ypred1)\n",
    "\n",
    "print(\"Classification report:\")\n",
    "print(report)\n",
    "print(\"Accuracy: \",accuracy)\n",
    "print(\"  \")\n",
    "print(\"Confusion matrix:\")\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ce2bf8b",
   "metadata": {},
   "source": [
    "###  Support Vector Machine\n",
    "#### Linear Support Vector Machine\n",
    "Fit a SVM with a linear kernel.\n",
    "\n",
    "\n",
    "0. Use `data1` and `y` for this question because SVM can take any type of variables.\n",
    "1. Split data (80% training; 20% test) (set `random_state=100`)\n",
    "2. Use the standard scaler to preprocess features.\n",
    "3. Use `LinearSVC` from `sklearn.svm`.\n",
    "4. Report the test set accuary, precision, recall, and the F1 score. \n",
    "5. Print out the confusion matrix for the test set.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "5b770cf2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.7012448132780082\n",
      "  \n",
      "\u001b[1m Poor Accuracy\n",
      "  \n",
      "\u001b[0m Confusion matrix:\n",
      "[[ 8  7  8]\n",
      " [ 0 64 28]\n",
      " [ 6 23 97]]\n",
      "  \n",
      "Classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        High       0.35      0.57      0.43        14\n",
      "         Low       0.70      0.68      0.69        94\n",
      "    Moderate       0.77      0.73      0.75       133\n",
      "\n",
      "    accuracy                           0.70       241\n",
      "   macro avg       0.60      0.66      0.62       241\n",
      "weighted avg       0.72      0.70      0.71       241\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load libraries\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn import datasets\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "features= data1\n",
    "target=y\n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features_standardized, target, random_state=100, test_size=.2)\n",
    "\n",
    "# select classifer for svc\n",
    "svc = LinearSVC(C=1.0, dual=False)\n",
    "\n",
    "# Create model with svc and train model with data\n",
    "model2 = svc.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "# Create predictions from model with test data\n",
    "ypred2=model2.predict(X_test)\n",
    "\n",
    "# Find accuracy\n",
    "#create confusion matrix with accuracy for test and predicted values from model\n",
    "accuracy = accuracy_score(y_test,ypred2)\n",
    "report = classification_report(ypred2, y_test)\n",
    "cm = confusion_matrix(y_test, ypred2)\n",
    "\n",
    "\n",
    "\n",
    "# Check Model accuracy\n",
    "print(\"Accuracy: \",accuracy)\n",
    "print(\"  \")\n",
    "if accuracy< 0.75:\n",
    "    print(\"\\033[1m Poor Accuracy\")\n",
    "else:\n",
    "    print(\"Acceptable Accuracy\")\n",
    "print(\"  \")\n",
    "print(\"\\033[0m Confusion matrix:\")\n",
    "print(cm)\n",
    "print(\"  \")\n",
    "print(\"Classification report:\")\n",
    "print(report)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d2a489c",
   "metadata": {},
   "source": [
    "####  Kernal SVM\n",
    "##### The performance of the linear SVM is not very good. Try to fit a nonlinear SVM.\n",
    "\n",
    "\n",
    "1. `kernel` should be `rbf`.\n",
    "2. Set `C=1e6` to create a hard margin.\n",
    "3. `gamma='scale'`\n",
    "4. Report the confusion matrix, accuracy, precision, recall, and the F1 socre for the test data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "3be04c1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1m Accuracy:  0.9377593360995851\n",
      "\u001b[1m Significantly Improved Accuracy\n",
      " \u001b[0m \n",
      "Classification report:\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "        High       0.91      0.91      0.91        23\n",
      "         Low       0.95      0.93      0.94        94\n",
      "    Moderate       0.94      0.95      0.94       124\n",
      "\n",
      "    accuracy                           0.94       241\n",
      "   macro avg       0.93      0.93      0.93       241\n",
      "weighted avg       0.94      0.94      0.94       241\n",
      "\n",
      "  \n",
      "Confusion matrix:\n",
      "[[ 21   0   2]\n",
      " [  1  87   4]\n",
      " [  1   7 118]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "\n",
    "features= data1\n",
    "target=y\n",
    "\n",
    "# Scale features \n",
    "scaler = StandardScaler()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    features_standardized, target, random_state=100, test_size=.2)\n",
    "\n",
    "# Set params (nonlinear) with gamma scale\n",
    "clf = SVC(kernel='rbf', C=1E6, gamma='scale')\n",
    "# Train model with data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Save predictions of model from test data\n",
    "ypred3=clf.predict(X_test)\n",
    "accuracy = accuracy_score(y_test,ypred3)\n",
    "report = classification_report(ypred3, y_test)\n",
    "cm = confusion_matrix(y_test, ypred3)\n",
    "\n",
    "\n",
    "# Check if accuracy has improved from linear model\n",
    "print(\"\\033[1m Accuracy: \",accuracy)\n",
    "if accuracy< 0.75:\n",
    "    print(\"\\033[1m Poor Accuracy\")\n",
    "else:\n",
    "    print(\"\\033[1m Significantly Improved Accuracy\")\n",
    "print(\" \\033[0m \")\n",
    "\n",
    "'\n",
    "\n",
    "# Print model report and confusion matrix\n",
    "print(\"Classification report:\")\n",
    "print(report)\n",
    "print(\"  \")\n",
    "print(\"Confusion matrix:\")\n",
    "print(cm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a68db1ae",
   "metadata": {},
   "source": [
    "####  Search the \"Optimal\" SVM model\n",
    "Use grid search and cross-validation to tune the hyperparameters `C` and `gamma`.\n",
    "\n",
    "1. For `C`, consider these values: top 5 likely values\n",
    "2. For `gamma`, consider values: top four likely values\n",
    "3. Let `kernel='rbf'`.\n",
    "4. Use 5-fold CV\n",
    "5. Report the best parameters\n",
    "6. Report the score (accuracy) of the test set \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "121aab2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 29.1 s\n",
      "Wall time: 8.34 s\n",
      " \n",
      " \u001b[1m Accuracy: 0.8962655601659751\n",
      "Best Params: {'svc__C': 1000.0, 'svc__gamma': 0.1}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.decomposition import PCA\n",
    "\n",
    "\n",
    "features= data1\n",
    "target=y\n",
    "\n",
    "# Set svc and PCA specs\n",
    "pca = PCA( random_state=0)\n",
    "svc = SVC(kernel='rbf', class_weight='balanced')\n",
    "\n",
    "# Make model with pca and svc specifications from above\n",
    "# Use make_pipeline to create a pipe \n",
    "    # Because GridSearchCV only accept a pipeline for the model argument. \n",
    "    # We already preprocessed the data in previous parts, it is not necessary to do it again.\n",
    "model = make_pipeline(pca, svc)\n",
    "\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(data1, target,\n",
    "                                                random_state=0, test_size=.2)\n",
    "\n",
    "# Create param search space\n",
    "param_grid = {'svc__C': [1.0, 1e1, 1e2, 1e3, 1e4],\n",
    "              'svc__gamma': [1, 0.1, 0.01, 0.001]}\n",
    "\n",
    "# Create search for model estimator \n",
    "# fit with training data\n",
    "grid = GridSearchCV(model, param_grid, cv=5, refit=True)\n",
    "\n",
    "%time grid.fit(X_train,y_train)\n",
    "\n",
    "# Select model with best estimator from grid search\n",
    "model_best = grid.best_estimator_\n",
    "\n",
    "# Save model predictions from test data\n",
    "yfit = model_best.predict(X_test)\n",
    "print(' ')\n",
    "print(' \\033[1m Accuracy:',np.mean(yfit==y_test))\n",
    "print('Best Params:',grid.best_params_)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d28c4a7",
   "metadata": {},
   "source": [
    "###  Neural Network\n",
    "####  Neural Network with One Hidden Layer\n",
    "\n",
    "1. Normalize `data1` before spliting the data.\n",
    "2. There should be 50 nodes in the hidden layer.\n",
    "3. Set `max_iter=100000`. \n",
    "4. Calculate the accuracy for the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c36c4813",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.8576158940397351\n"
     ]
    }
   ],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "from sklearn.preprocessing import Normalizer\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "\n",
    "features= data1\n",
    "target=y\n",
    "\n",
    "# Scale all features with normalizers and transform features for the model\n",
    "scaler = Normalizer()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(features_standardized, target, stratify=y,\n",
    "                                                    random_state=100)\n",
    "\n",
    "\n",
    "\n",
    "# Create and train model with mpl parameters we set\n",
    "mlp = MLPClassifier(solver='lbfgs', random_state=100, hidden_layer_sizes=[50], max_iter= 100000)\n",
    "mlp.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "# Save the predictions of model from test data\n",
    "ypred3=mlp.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test,ypred3)\n",
    "print(\"Accuracy: \",accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4df19904",
   "metadata": {},
   "source": [
    "####  Neural Network with Two Hidden Layers\n",
    "\n",
    "1. There should be 10 nodes in the first hidden layer and 10 nodes in the second hidden layer.\n",
    "2. Set `max_iter=9999`. \n",
    "3. Set `random_state=0` and `solver='lbfgs'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "98dca72a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy:  0.7582781456953642\n"
     ]
    }
   ],
   "source": [
    "# Same code as above nueral network, \n",
    "# Except adding a second layer and small param tweeks\n",
    "\n",
    "features= data1\n",
    "target=y\n",
    "\n",
    "scaler = Normalizer()\n",
    "\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(features_standardized, target, stratify=y,\n",
    "                                                    random_state=0)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "mlp = MLPClassifier(solver='lbfgs', random_state=0, \n",
    "    hidden_layer_sizes=[10,10], # Set nodes in first and second layer\n",
    "    max_iter= 1000)\n",
    "\n",
    "mlp.fit(X_train, y_train)\n",
    "\n",
    "\n",
    "\n",
    "ypred3=mlp.predict(X_test)\n",
    "\n",
    "accuracy = accuracy_score(y_test,ypred3)\n",
    "print(\"Accuracy: \",accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47dcc466",
   "metadata": {},
   "source": [
    "####  Regularized Neural Network\n",
    "Now consider 10 values for the regularization hyperparameter `alpha`: `np.linspace(0.001, 0.010, 10)`. Which value will be optimal? i.e. Which value yields the best test accuracy? \n",
    "\n",
    "\n",
    "1. Use two hidden layers `[50,50]`. Set `max_iter=100000000`.\n",
    "2. Write a `for` loop to fit the mlp model for each value of `alpha` and calculate the accuracy of the test set in each iteration.\n",
    "3. print out the test accuracy. Which value of `alpha` is the optimal one?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "fec78103",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9271523178807947 0.001\n",
      "0.9039735099337748 0.002\n",
      "0.9039735099337748 0.003\n",
      "0.9105960264900662 0.004\n",
      "0.9271523178807947 0.005\n",
      "0.9072847682119205 0.006\n",
      "0.9006622516556292 0.007\n",
      "0.9072847682119205 0.008\n",
      "0.890728476821192 0.009000000000000001\n",
      "0.9006622516556292 0.01\n"
     ]
    }
   ],
   "source": [
    "features= data1\n",
    "target=y\n",
    "\n",
    "# Scale all features with normalizers and transform features for the model\n",
    "scaler = Normalizer()\n",
    "features_standardized = scaler.fit_transform(features)\n",
    "\n",
    "# Specify train split random state and target/features for model\n",
    "X_train, X_test, y_train, y_test = train_test_split(features_standardized, target, stratify=y,\n",
    "                                                    random_state=0)\n",
    "# Specify the possible hyperparameters for alpha\n",
    "# 10 values from 0.001 to 0.01\n",
    "alpha= np.linspace(0.001, 0.010, 10)\n",
    "\n",
    "# For each possible alpha value (10 diff values) fit a neural network and print the accuracy\n",
    "for x in alpha:\n",
    "    mlp = MLPClassifier(solver='lbfgs', random_state=0,\n",
    "                            hidden_layer_sizes=[50,50],\n",
    "                            alpha=x, max_iter=10000)\n",
    "    mlp.fit(X_train, y_train)\n",
    "    \n",
    "    ypred3=mlp.predict(X_test)\n",
    "\n",
    "    accuracy1 = accuracy_score(y_test,ypred3)\n",
    "    print(accuracy1, x)\n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "    \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "183ab563",
   "metadata": {},
   "source": [
    "#### Optimal Alpha is .005 or .001"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05a35ce1",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "+ Naive bayes and guassian models were the worst (less than 70%)\n",
    "+ **Non linear SVM model is the best (93% Accurate)**\n",
    "+ Nueral networks were fairly accurate\n",
    "    + Regularized with aplha as 0.001 or 0.005 was 2nd best (92.7% accuracy)\n",
    "    + 2 layer neural network was worse than single layer\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d79bdc7c",
   "metadata": {},
   "source": [
    "+ Given future feature data we could predict a students online learning adaptability level, using our non linear svm model.  This model could be used to adjust the teaching style, or budget allocation of learning materials to best accomodate the predicted adaptability level of students."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f68dc8e7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}