Multiple Linear Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Multiple Linear Regression</h1>\n",
    "\n",
    "- MLR: y = b0 + b1*x1 + ...... + bn*xn\n",
    "- Assumptions of a Linear Regression;\n",
    "    - Linearity\n",
    "    - Homoscedasticity\n",
    "    - Multivariate Normality\n",
    "    - Independence of Errors\n",
    "    - Lack of Mulitcollinearity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Building a Model**\n",
    "- There are 5 methods of building a model\n",
    "\n",
    "    **1. All-in**\n",
    "        - One is if you have prior knowledge if you know that these exact variavle are the ones are your true \n",
    "          predictors you dont have to build anything you already know that this is the case.\n",
    "        - Other one is you have to.\n",
    "        - Preparing for backward elimination\n",
    "    **2. Backward Elimination**\n",
    "        - Step 1: Select a significaance level to stay in the model(eg Significance Level = 0.05)\n",
    "        - Step 2: Fit the full model with akk possible predictors.\n",
    "        - Step 3: Consider the predictor with the hghest p-value, if p > sl, go to step 4, otherwise go to FIN (Keep \n",
    "                  the previous mode.).\n",
    "        - Step 4: Remove the predictor.\n",
    "        - Step 5: Fit model without this variables.\n",
    "    **3. Forward Selection**\n",
    "        - Step 1: Select a significaance level to enter in the model(eg Significance Level = 0.05)\n",
    "        - Step 2: Fit all the simple regression model Y ~ Xn, then select the one with the lowest p-value.\n",
    "        - Step 3: Keep this variable and fit all possible models with one extra predictor added to the one you already \n",
    "                  have.\n",
    "        - Step 4: Consider the predictor with the lowest p-value. If p < sl, go to step 3, otherwise go to FIN.\n",
    "    **4. Bidirectional Elimination**\n",
    "        - Step 1: Select a significance level to enter and to stay in the model. (eg SLENTER = 0.05, SLSTAY = 0.05)\n",
    "        - Step 2: Perfomr the next step of Forward Selection. (new variable must have p < SLENTER to enter)\n",
    "        - Step 3: Perform ALL step of Backward Elemination. (old variable must have p < SLSTAY to stay)\n",
    "        - Step 4: No new variable can enter and no old variabel can exit.\n",
    "        - Now your model is ready.\n",
    "    **5. Score Comparison/All Possible Models**\n",
    "        - Step 1: Select a criterion of goodness of fit. (eg Akaike criterion)\n",
    "        - Step 2: Construct all possible regression model; 2^n - 1 total combination.\n",
    "        - Step 3: Select the one with the best criterion\n",
    "\n",
    "**Importing Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy allows us to work with array.\n",
    "import numpy as np\n",
    "\n",
    "# Maptplotlib which allows us to plot some chart.\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Pandas allows us to not only import the datasets but also create the matrix of features(independent) and \n",
    "# dependent variable.\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Importing the Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_csv(\"50_Startups.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The independent variable usally in the first columns of dataset and dependent variable usally in the last columns of the data sets.\n",
    "- X is Independent Variable.\n",
    "- Y is Dependent Variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[165349.2 136897.8 471784.1 'New York']\n",
      " [162597.7 151377.59 443898.53 'California']\n",
      " [153441.51 101145.55 407934.54 'Florida']\n",
      " [144372.41 118671.85 383199.62 'New York']\n",
      " [142107.34 91391.77 366168.42 'Florida']\n",
      " [131876.9 99814.71 362861.36 'New York']\n",
      " [134615.46 147198.87 127716.82 'California']\n",
      " [130298.13 145530.06 323876.68 'Florida']\n",
      " [120542.52 148718.95 311613.29 'New York']\n",
      " [123334.88 108679.17 304981.62 'California']\n",
      " [101913.08 110594.11 229160.95 'Florida']\n",
      " [100671.96 91790.61 249744.55 'California']\n",
      " [93863.75 127320.38 249839.44 'Florida']\n",
      " [91992.39 135495.07 252664.93 'California']\n",
      " [119943.24 156547.42 256512.92 'Florida']\n",
      " [114523.61 122616.84 261776.23 'New York']\n",
      " [78013.11 121597.55 264346.06 'California']\n",
      " [94657.16 145077.58 282574.31 'New York']\n",
      " [91749.16 114175.79 294919.57 'Florida']\n",
      " [86419.7 153514.11 0.0 'New York']\n",
      " [76253.86 113867.3 298664.47 'California']\n",
      " [78389.47 153773.43 299737.29 'New York']\n",
      " [73994.56 122782.75 303319.26 'Florida']\n",
      " [67532.53 105751.03 304768.73 'Florida']\n",
      " [77044.01 99281.34 140574.81 'New York']\n",
      " [64664.71 139553.16 137962.62 'California']\n",
      " [75328.87 144135.98 134050.07 'Florida']\n",
      " [72107.6 127864.55 353183.81 'New York']\n",
      " [66051.52 182645.56 118148.2 'Florida']\n",
      " [65605.48 153032.06 107138.38 'New York']\n",
      " [61994.48 115641.28 91131.24 'Florida']\n",
      " [61136.38 152701.92 88218.23 'New York']\n",
      " [63408.86 129219.61 46085.25 'California']\n",
      " [55493.95 103057.49 214634.81 'Florida']\n",
      " [46426.07 157693.92 210797.67 'California']\n",
      " [46014.02 85047.44 205517.64 'New York']\n",
      " [28663.76 127056.21 201126.82 'Florida']\n",
      " [44069.95 51283.14 197029.42 'California']\n",
      " [20229.59 65947.93 185265.1 'New York']\n",
      " [38558.51 82982.09 174999.3 'California']\n",
      " [28754.33 118546.05 172795.67 'California']\n",
      " [27892.92 84710.77 164470.71 'Florida']\n",
      " [23640.93 96189.63 148001.11 'California']\n",
      " [15505.73 127382.3 35534.17 'New York']\n",
      " [22177.74 154806.14 28334.72 'California']\n",
      " [1000.23 124153.04 1903.93 'New York']\n",
      " [1315.46 115816.21 297114.46 'Florida']\n",
      " [0.0 135426.92 0.0 'California']\n",
      " [542.05 51743.15 0.0 'New York']\n",
      " [0.0 116983.8 45173.06 'California']]\n"
     ]
    }
   ],
   "source": [
    "x = dataset.iloc[:, :-1].values      # iloc is index of location\n",
    "y = dataset.iloc[:, -1].values\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[192261.83 191792.06 191050.39 182901.99 166187.94 156991.12 156122.51\n",
      " 155752.6  152211.77 149759.96 146121.95 144259.4  141585.52 134307.35\n",
      " 132602.65 129917.04 126992.93 125370.37 124266.9  122776.86 118474.03\n",
      " 111313.02 110352.25 108733.99 108552.04 107404.34 105733.54 105008.31\n",
      " 103282.38 101004.64  99937.59  97483.56  97427.84  96778.92  96712.8\n",
      "  96479.51  90708.19  89949.14  81229.06  81005.76  78239.91  77798.83\n",
      "  71498.49  69758.98  65200.33  64926.08  49490.75  42559.73  35673.41\n",
      "  14681.4 ]\n"
     ]
    }
   ],
   "source": [
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Encoding Categorical Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.0 0.0 1.0 165349.2 136897.8 471784.1]\n",
      " [1.0 0.0 0.0 162597.7 151377.59 443898.53]\n",
      " [0.0 1.0 0.0 153441.51 101145.55 407934.54]\n",
      " [0.0 0.0 1.0 144372.41 118671.85 383199.62]\n",
      " [0.0 1.0 0.0 142107.34 91391.77 366168.42]\n",
      " [0.0 0.0 1.0 131876.9 99814.71 362861.36]\n",
      " [1.0 0.0 0.0 134615.46 147198.87 127716.82]\n",
      " [0.0 1.0 0.0 130298.13 145530.06 323876.68]\n",
      " [0.0 0.0 1.0 120542.52 148718.95 311613.29]\n",
      " [1.0 0.0 0.0 123334.88 108679.17 304981.62]\n",
      " [0.0 1.0 0.0 101913.08 110594.11 229160.95]\n",
      " [1.0 0.0 0.0 100671.96 91790.61 249744.55]\n",
      " [0.0 1.0 0.0 93863.75 127320.38 249839.44]\n",
      " [1.0 0.0 0.0 91992.39 135495.07 252664.93]\n",
      " [0.0 1.0 0.0 119943.24 156547.42 256512.92]\n",
      " [0.0 0.0 1.0 114523.61 122616.84 261776.23]\n",
      " [1.0 0.0 0.0 78013.11 121597.55 264346.06]\n",
      " [0.0 0.0 1.0 94657.16 145077.58 282574.31]\n",
      " [0.0 1.0 0.0 91749.16 114175.79 294919.57]\n",
      " [0.0 0.0 1.0 86419.7 153514.11 0.0]\n",
      " [1.0 0.0 0.0 76253.86 113867.3 298664.47]\n",
      " [0.0 0.0 1.0 78389.47 153773.43 299737.29]\n",
      " [0.0 1.0 0.0 73994.56 122782.75 303319.26]\n",
      " [0.0 1.0 0.0 67532.53 105751.03 304768.73]\n",
      " [0.0 0.0 1.0 77044.01 99281.34 140574.81]\n",
      " [1.0 0.0 0.0 64664.71 139553.16 137962.62]\n",
      " [0.0 1.0 0.0 75328.87 144135.98 134050.07]\n",
      " [0.0 0.0 1.0 72107.6 127864.55 353183.81]\n",
      " [0.0 1.0 0.0 66051.52 182645.56 118148.2]\n",
      " [0.0 0.0 1.0 65605.48 153032.06 107138.38]\n",
      " [0.0 1.0 0.0 61994.48 115641.28 91131.24]\n",
      " [0.0 0.0 1.0 61136.38 152701.92 88218.23]\n",
      " [1.0 0.0 0.0 63408.86 129219.61 46085.25]\n",
      " [0.0 1.0 0.0 55493.95 103057.49 214634.81]\n",
      " [1.0 0.0 0.0 46426.07 157693.92 210797.67]\n",
      " [0.0 0.0 1.0 46014.02 85047.44 205517.64]\n",
      " [0.0 1.0 0.0 28663.76 127056.21 201126.82]\n",
      " [1.0 0.0 0.0 44069.95 51283.14 197029.42]\n",
      " [0.0 0.0 1.0 20229.59 65947.93 185265.1]\n",
      " [1.0 0.0 0.0 38558.51 82982.09 174999.3]\n",
      " [1.0 0.0 0.0 28754.33 118546.05 172795.67]\n",
      " [0.0 1.0 0.0 27892.92 84710.77 164470.71]\n",
      " [1.0 0.0 0.0 23640.93 96189.63 148001.11]\n",
      " [0.0 0.0 1.0 15505.73 127382.3 35534.17]\n",
      " [1.0 0.0 0.0 22177.74 154806.14 28334.72]\n",
      " [0.0 0.0 1.0 1000.23 124153.04 1903.93]\n",
      " [0.0 1.0 0.0 1315.46 115816.21 297114.46]\n",
      " [1.0 0.0 0.0 0.0 135426.92 0.0]\n",
      " [0.0 0.0 1.0 542.05 51743.15 0.0]\n",
      " [1.0 0.0 0.0 0.0 116983.8 45173.06]]\n"
     ]
    }
   ],
   "source": [
    "# One Hot Encoding it helps to change the Category to Binary Vector Number.\n",
    "# California = 1 0 0\n",
    "# New York = 0 0 1\n",
    "# Florida = 0 1 0\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Creating an object of the column transform\n",
    "ct = ColumnTransformer(transformers =[('encoder', OneHotEncoder(), [3])], remainder = 'passthrough')\n",
    "\n",
    "# Fit and Transform\n",
    "x = np.array(ct.fit_transform(x))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Splitting the dataset into Training Set and Test Set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.0 1.0 0.0 55493.95 103057.49 214634.81]\n",
      " [0.0 0.0 1.0 46014.02 85047.44 205517.64]\n",
      " [0.0 1.0 0.0 75328.87 144135.98 134050.07]\n",
      " [1.0 0.0 0.0 46426.07 157693.92 210797.67]\n",
      " [0.0 1.0 0.0 91749.16 114175.79 294919.57]\n",
      " [0.0 1.0 0.0 130298.13 145530.06 323876.68]\n",
      " [0.0 1.0 0.0 119943.24 156547.42 256512.92]\n",
      " [0.0 0.0 1.0 1000.23 124153.04 1903.93]\n",
      " [0.0 0.0 1.0 542.05 51743.15 0.0]\n",
      " [0.0 0.0 1.0 65605.48 153032.06 107138.38]\n",
      " [0.0 0.0 1.0 114523.61 122616.84 261776.23]\n",
      " [0.0 1.0 0.0 61994.48 115641.28 91131.24]\n",
      " [1.0 0.0 0.0 63408.86 129219.61 46085.25]\n",
      " [1.0 0.0 0.0 78013.11 121597.55 264346.06]\n",
      " [1.0 0.0 0.0 23640.93 96189.63 148001.11]\n",
      " [1.0 0.0 0.0 76253.86 113867.3 298664.47]\n",
      " [0.0 0.0 1.0 15505.73 127382.3 35534.17]\n",
      " [0.0 0.0 1.0 120542.52 148718.95 311613.29]\n",
      " [1.0 0.0 0.0 91992.39 135495.07 252664.93]\n",
      " [1.0 0.0 0.0 64664.71 139553.16 137962.62]\n",
      " [0.0 0.0 1.0 131876.9 99814.71 362861.36]\n",
      " [0.0 0.0 1.0 94657.16 145077.58 282574.31]\n",
      " [1.0 0.0 0.0 28754.33 118546.05 172795.67]\n",
      " [1.0 0.0 0.0 0.0 116983.8 45173.06]\n",
      " [1.0 0.0 0.0 162597.7 151377.59 443898.53]\n",
      " [0.0 1.0 0.0 93863.75 127320.38 249839.44]\n",
      " [1.0 0.0 0.0 44069.95 51283.14 197029.42]\n",
      " [0.0 0.0 1.0 77044.01 99281.34 140574.81]\n",
      " [1.0 0.0 0.0 134615.46 147198.87 127716.82]\n",
      " [0.0 1.0 0.0 67532.53 105751.03 304768.73]\n",
      " [0.0 1.0 0.0 28663.76 127056.21 201126.82]\n",
      " [0.0 0.0 1.0 78389.47 153773.43 299737.29]\n",
      " [0.0 0.0 1.0 86419.7 153514.11 0.0]\n",
      " [1.0 0.0 0.0 123334.88 108679.17 304981.62]\n",
      " [1.0 0.0 0.0 38558.51 82982.09 174999.3]\n",
      " [0.0 1.0 0.0 1315.46 115816.21 297114.46]\n",
      " [0.0 0.0 1.0 144372.41 118671.85 383199.62]\n",
      " [0.0 0.0 1.0 165349.2 136897.8 471784.1]\n",
      " [1.0 0.0 0.0 0.0 135426.92 0.0]\n",
      " [1.0 0.0 0.0 22177.74 154806.14 28334.72]]\n"
     ]
    }
   ],
   "source": [
    "# Importing Package\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Dividing training and test set.\n",
    "# The best ratio is 80 - 20 for trainging and testing respectively.\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 0)\n",
    "\n",
    "print(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 96778.92  96479.51 105733.54  96712.8  124266.9  155752.6  132602.65\n",
      "  64926.08  35673.41 101004.64 129917.04  99937.59  97427.84 126992.93\n",
      "  71498.49 118474.03  69758.98 152211.77 134307.35 107404.34 156991.12\n",
      " 125370.37  78239.91  14681.4  191792.06 141585.52  89949.14 108552.04\n",
      " 156122.51 108733.99  90708.19 111313.02 122776.86 149759.96  81005.76\n",
      "  49490.75 182901.99 192261.83  42559.73  65200.33]\n"
     ]
    }
   ],
   "source": [
    "print(y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Training the MLR model on the Training Set**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearRegression()"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Importing Package\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "reg = LinearRegression()\n",
    "\n",
    "# Fitting the model\n",
    "reg.fit(x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Preparing the Test Set results**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[103015.2  103282.38]\n",
      " [132582.28 144259.4 ]\n",
      " [132447.74 146121.95]\n",
      " [ 71976.1   77798.83]\n",
      " [178537.48 191050.39]\n",
      " [116161.24 105008.31]\n",
      " [ 67851.69  81229.06]\n",
      " [ 98791.73  97483.56]\n",
      " [113969.44 110352.25]\n",
      " [167921.07 166187.94]]\n"
     ]
    }
   ],
   "source": [
    "# Predicting\n",
    "y_pred =  reg.predict(x_test)\n",
    "\n",
    "# Getting only 2 decimal\n",
    "np.set_printoptions(precision = 2)\n",
    "\n",
    "# Concatenating and Reshaping\n",
    "print(np.concatenate((y_pred.reshape(len(y_pred), 1), y_test.reshape(len(y_test), 1)), axis = 1))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}