Day_1_Linear_and_Polynomial_Regression.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNEzZPxGoI5k0nItMt9DKBc",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ebrarkiziloglu/ML-Workshops/blob/main/Day_1_Linear_and_Polynomial_Regression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import warnings\n",
        "warnings.filterwarnings('ignore')\n",
        "\n",
        "# fundamental modules\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "# sklearn\n",
        "from sklearn.preprocessing import PolynomialFeatures\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "# you can use for visualization\n",
        "import seaborn as sns\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "# Read and split the data:\n",
        "df = pd.read_csv('path_to_data')\n",
        "df.head()\n",
        "features = []\n",
        "X = df[features]\n",
        "y = df[\"name_of_the_y_column\"]\n",
        "train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=2)"
      ],
      "metadata": {
        "id": "T-adCSCh8ugX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Lineaer Regression\n",
        "\n",
        "## Simple Lineaer Regression\n",
        "\n",
        "m: # of data points\n",
        "\n",
        "h(x) = w0 + w1*x\n",
        "\n",
        "Minimize the error -> Cost function = minimize the sum of (y - h(x))^2\n",
        "\n",
        "Σ (y - h(x))^2 = Σ (w0 + w1.x - y)^2\n",
        "\n",
        "### Finding the w0 and w1:\n",
        "\n",
        "Take the derivative w.r.t. w0: \n",
        "\n",
        "->  0 = Σ 2 * (w0 + w1.x - y)\n",
        "->  m * w0 + Σ w1.x = Σ y\n",
        "\n",
        "#### w0 = μ(y) - w1*μ(x) \n",
        "\n",
        "Take the derivative w.r.t. w1: \n",
        "\n",
        "Put wo = μ(y) - w1*μ(x) \n",
        "\n",
        "0 = Σ 2 * (μ(y) - w1*μ(x) + w1.x - y) * (x - μ(x)) -> Complete the calculations\n",
        "\n",
        "#### w1 = [Σ (y-μ(y))(x-μ(x))]/[Σ (x-μ(x))^2]\n",
        "\n",
        "\n",
        "## Linear Regression with Multiple Features\n",
        "\n",
        "m: # of data points\n",
        "\n",
        "n: # of features + 1\n",
        "\n",
        "X(mxn) = matrix, first column is full of 1's. Each row corresponds to one data point.\n",
        "\n",
        "W = \\[ ]nx1 column vector of weights.\n",
        "\n",
        "Minimize the error -> Cost function J(W) = (X.W - Y)^T (X.W - Y)\n",
        "\n",
        "Take the gradiant this time!\n",
        "\n",
        "W = ( (X^T.X)^-1 ) X^T.Y          -> Matrix multiplication is costly! \n",
        "\n",
        "**Solution**: Use **Machine Learning** instead:\n",
        "\n",
        "### Gradient Descent\n",
        "* Take the derivative of the cost function. \n",
        "* Take a small step towards that direction. \n",
        "* Continue improving the direction.\n",
        "\n",
        "#### Problems:\n",
        "1. Being stuck at the local optima.\n",
        "2. Overfitting: Memorizing the data. \n",
        "\n",
        "**Solution** to this is to divide the data into **Training**, **Validation**, and **Test** parts.\n",
        "\n"
      ],
      "metadata": {
        "id": "e23CebmlqjOt"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "linear_model = LinearRegression()\n",
        "linear_model.fit(train_X, train_y)\n",
        "linear_model.coef_        # gives the weights except for the constant coeff.\n",
        "linear_model.intercept_   # gives the constant weight\n",
        "\n",
        "# Following scores will be in the descending order, since this model fits best with the training data: \n",
        "linear_model.score(train_X, train_y)\n",
        "linear_model.score(X, y)\n",
        "linear_model.score(val_X, val_y)"
      ],
      "metadata": {
        "id": "v0WiECDzq6hc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Using the normal equation for the Linear Regression:\n",
        "X_normal = np.hstack((np.ones((train_X.shape[0], 1)), train_X))\n",
        "# W = ( (X^T.X)^-1 ) X^T.Y\n",
        "weights = np.linalg.solve(np.dot(np.transpose(X_normal), X_normal), np.dot(np.transpose(X_normal), train_y))\n",
        "weights"
      ],
      "metadata": {
        "id": "HUmT9XvG9I3m"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Polynomial Regression\n",
        "\n",
        "h(x) = w0 + w1.x + w2.x^2\n",
        "\n"
      ],
      "metadata": {
        "id": "4rXkL2agTmul"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Non-linear Regression:\n",
        "N_poly_degree = 2\n",
        "poly = PolynomialFeatures(N_poly_degree)\n",
        "poly_X = poly.fit_transform(train_X)\n",
        "poly_model = LinearRegression()\n",
        "poly_model = poly_model.fit(poly_X, train_y)\n",
        "poly_model.coef_\n",
        "poly_model.intercept_\n",
        "\n",
        "# Following scores will be in the descending order, since this model fits best with the training data: \n",
        "poly_model.score(poly_X, train_y)\n",
        "ply_val = poly.transform(val_X)\n",
        "poly_model.score(ply_val, val_y)"
      ],
      "metadata": {
        "id": "f8u-0XLr8Rar"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}