Data Preprocessing Tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Data Preprocessing Tools</h1>\n",
    "\n",
    "**Importing the Libararies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy allows us to work with array.\n",
    "import numpy as np\n",
    "\n",
    "# Maptplotlib which allows us to plot some chart.\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Pandas allows us to not only import the datasets but also create the matrix of features(independent) and \n",
    "# dependent variable.\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Importing the Dataset**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = pd.read_csv(\"Data.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The independent variable usally in the first columns of dataset and dependent variable usally in the last columns of the data sets.\n",
    "- X is Independent Variable.\n",
    "- Y is Dependent Variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['France' 44.0 72000.0]\n",
      " ['Spain' 27.0 48000.0]\n",
      " ['Germany' 30.0 54000.0]\n",
      " ['Spain' 38.0 61000.0]\n",
      " ['Germany' 40.0 nan]\n",
      " ['France' 35.0 58000.0]\n",
      " ['Spain' nan 52000.0]\n",
      " ['France' 48.0 79000.0]\n",
      " ['Germany' 50.0 83000.0]\n",
      " ['France' 37.0 67000.0]]\n"
     ]
    }
   ],
   "source": [
    "x = dataset.iloc[:, :-1].values      # iloc is index of location\n",
    "y = dataset.iloc[:, -1].values\n",
    "\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['No' 'Yes' 'No' 'No' 'Yes' 'Yes' 'No' 'Yes' 'No' 'Yes']\n"
     ]
    }
   ],
   "source": [
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Taking Care of Missing Data**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['France' 44.0 72000.0]\n",
      " ['Spain' 27.0 48000.0]\n",
      " ['Germany' 30.0 54000.0]\n",
      " ['Spain' 38.0 61000.0]\n",
      " ['Germany' 40.0 63777.77777777778]\n",
      " ['France' 35.0 58000.0]\n",
      " ['Spain' 38.77777777777778 52000.0]\n",
      " ['France' 48.0 79000.0]\n",
      " ['Germany' 50.0 83000.0]\n",
      " ['France' 37.0 67000.0]]\n"
     ]
    }
   ],
   "source": [
    "# Replacing the missing value by the average of all the age and salary.\n",
    "\n",
    "# Scikit Learn contain all the machinary models that can be build simply creating an object of some classes.\n",
    "from sklearn.impute import SimpleImputer\n",
    "imputer = SimpleImputer(missing_values = np.nan, strategy = \"mean\")\n",
    "imputer.fit(x[:, 1:3])\n",
    "\n",
    "# Updating the data \n",
    "x[:, 1:3] = imputer.transform(x[:, 1:3])\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Encoding Categorical Data</h3>\n",
    "\n",
    "- **Encoding the Independent Variable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1.0 0.0 0.0 44.0 72000.0]\n",
      " [0.0 0.0 1.0 27.0 48000.0]\n",
      " [0.0 1.0 0.0 30.0 54000.0]\n",
      " [0.0 0.0 1.0 38.0 61000.0]\n",
      " [0.0 1.0 0.0 40.0 63777.77777777778]\n",
      " [1.0 0.0 0.0 35.0 58000.0]\n",
      " [0.0 0.0 1.0 38.77777777777778 52000.0]\n",
      " [1.0 0.0 0.0 48.0 79000.0]\n",
      " [0.0 1.0 0.0 50.0 83000.0]\n",
      " [1.0 0.0 0.0 37.0 67000.0]]\n"
     ]
    }
   ],
   "source": [
    "# One Hot Encoding it helps to change the Category to Binary Vector Number.\n",
    "# France = 1 0 0\n",
    "# Spain = 0 0 1\n",
    "# Germany = 0 1 0\n",
    "\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "\n",
    "# Creating an object of the column transform\n",
    "ct = ColumnTransformer(transformers =[('encoder', OneHotEncoder(), [0])], remainder = 'passthrough')\n",
    "\n",
    "# Fit and Transform\n",
    "x = np.array(ct.fit_transform(x))\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Encoding the Dependent Variable**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 0 0 1 1 0 1 0 1]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "# Label Encoder for 1 variable, so we dont have to add any argument.\n",
    "le = LabelEncoder()\n",
    "\n",
    "# Fit and Transform\n",
    "y = le.fit_transform(y)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Splitting the dataset into Training Set and Test Set**\n",
    "- Training sets are going to train model on existing observation.\n",
    "- Test sets are going the evaluate the performance of your model on new observation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.0 0.0 1.0 38.77777777777778 52000.0]\n",
      " [0.0 1.0 0.0 40.0 63777.77777777778]\n",
      " [1.0 0.0 0.0 44.0 72000.0]\n",
      " [0.0 0.0 1.0 38.0 61000.0]\n",
      " [0.0 0.0 1.0 27.0 48000.0]\n",
      " [1.0 0.0 0.0 48.0 79000.0]\n",
      " [0.0 1.0 0.0 50.0 83000.0]\n",
      " [1.0 0.0 0.0 35.0 58000.0]]\n"
     ]
    }
   ],
   "source": [
    "# Importing Package\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Dividing training and test set.\n",
    "# The best ratio is 80 - 20 for trainging and testing respectively.\n",
    "x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 1)\n",
    "\n",
    "print(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.0 1.0 0.0 30.0 54000.0]\n",
      " [1.0 0.0 0.0 37.0 67000.0]]\n"
     ]
    }
   ],
   "source": [
    "print(x_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1 0 0 1 1 0 1]\n"
     ]
    }
   ],
   "source": [
    "print(y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0 1]\n"
     ]
    }
   ],
   "source": [
    "print(y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Feature Scaling**\n",
    "- This allows our feature on the same scale.\n",
    "- Two main techniques for Feature Scaling are;\n",
    "    - Standardisation\n",
    "        - Xstand = (x - mean(x)) / standard deviation(x)\n",
    "        - The value come between -3 and +3.\n",
    "    - Normalisation\n",
    "        - Xnorm = (x - min(x)) / (max(x) - min(x)\n",
    "        - the value come between 0 and 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.0 0.0 1.0 -0.19159184384578545 -1.0781259408412425]\n",
      " [0.0 1.0 0.0 -0.014117293757057777 -0.07013167641635372]\n",
      " [1.0 0.0 0.0 0.566708506533324 0.633562432710455]\n",
      " [0.0 0.0 1.0 -0.30453019390224867 -0.30786617274297867]\n",
      " [0.0 0.0 1.0 -1.9018011447007988 -1.420463615551582]\n",
      " [1.0 0.0 0.0 1.1475343068237058 1.232653363453549]\n",
      " [0.0 1.0 0.0 1.4379472069688968 1.5749910381638885]\n",
      " [1.0 0.0 0.0 -0.7401495441200351 -0.5646194287757332]]\n",
      "[[0.0 1.0 0.0 -1.0 -1.0]\n",
      " [1.0 0.0 0.0 1.0 1.0]]\n"
     ]
    }
   ],
   "source": [
    "# Importing Package\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "sc = StandardScaler()\n",
    "x_train[:, 3:] = sc.fit_transform(x_train[:, 3:])\n",
    "x_test[:, 3:] = sc.fit_transform(x_test[:, 3:])\n",
    "\n",
    "print(x_train)\n",
    "print(x_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}