03302019_99p4ProbBased_LeNet_Lab.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "03302019 99p4ProbBased LeNet-Lab.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/preritt/CS231n-/blob/master/03302019_99p4ProbBased_LeNet_Lab.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "HFVqYNiRCy47",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# LeNet Lab\n",
        "![LeNet Architecture](https://github.com/sujaybabruwad/LeNet-in-Tensorflow/blob/master/lenet.png?raw=1)\n",
        "Source: Yan LeCun"
      ]
    },
    {
      "metadata": {
        "id": "k7PXv7fjCy5B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Load Data\n",
        "\n",
        "Load the MNIST data, which comes pre-loaded with TensorFlow.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "t4iqynjrCy5E",
        "colab_type": "code",
        "outputId": "7185342c-6808-459a-b2ff-638f670e5e34",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 199
        }
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.examples.tutorials.mnist import input_data\n",
        "\n",
        "mnist = input_data.read_data_sets(\"MNIST_data/\", reshape=False)\n",
        "X_train, y_train           = mnist.train.images, mnist.train.labels\n",
        "X_validation, y_validation = mnist.validation.images, mnist.validation.labels\n",
        "X_test, y_test             = mnist.test.images, mnist.test.labels\n",
        "\n",
        "assert(len(X_train) == len(y_train))\n",
        "assert(len(X_validation) == len(y_validation))\n",
        "assert(len(X_test) == len(y_test))\n",
        "\n",
        "print()\n",
        "print(\"Image Shape: {}\".format(X_train[0].shape))\n",
        "print()\n",
        "print(\"Training Set:   {} samples\".format(len(X_train)))\n",
        "print(\"Validation Set: {} samples\".format(len(X_validation)))\n",
        "print(\"Test Set:       {} samples\".format(len(X_test)))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Extracting MNIST_data/train-images-idx3-ubyte.gz\n",
            "Extracting MNIST_data/train-labels-idx1-ubyte.gz\n",
            "Extracting MNIST_data/t10k-images-idx3-ubyte.gz\n",
            "Extracting MNIST_data/t10k-labels-idx1-ubyte.gz\n",
            "\n",
            "Image Shape: (28, 28, 1)\n",
            "\n",
            "Training Set:   55000 samples\n",
            "Validation Set: 5000 samples\n",
            "Test Set:       10000 samples\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "pV8kvNmMCy5T",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The MNIST data that TensorFlow pre-loads comes as 28x28x1 images.\n",
        "\n",
        "However, the LeNet architecture only accepts 32x32xC images, where C is the number of color channels.\n",
        "\n",
        "In order to reformat the MNIST data into a shape that LeNet will accept, we pad the data with two rows of zeros on the top and bottom, and two columns of zeros on the left and right (28+2+2 = 32).\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "sBWJCBw1Cy5U",
        "colab_type": "code",
        "outputId": "5cda8fae-1365-405c-f538-71438d35969b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "\n",
        "# Pad images with 0s\n",
        "X_train      = np.pad(X_train, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
        "X_validation = np.pad(X_validation, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
        "X_test       = np.pad(X_test, ((0,0),(2,2),(2,2),(0,0)), 'constant')\n",
        "    \n",
        "print(\"Updated Image Shape: {}\".format(X_train[0].shape))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Updated Image Shape: (32, 32, 1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cW7cCH1xCy5c",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Visualize Data\n",
        "\n",
        "View a sample from the dataset.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "h7UbBX8RCy5e",
        "colab_type": "code",
        "outputId": "a1a03fca-0a8e-47c5-df58-5c4f17a6677f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "cell_type": "code",
      "source": [
        "import random\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "\n",
        "index = random.randint(0, len(X_train))\n",
        "image = X_train[index].squeeze()\n",
        "\n",
        "plt.figure(figsize=(1,1))\n",
        "plt.imshow(image, cmap=\"gray\")\n",
        "print(y_train[index])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAFcAAABYCAYAAACAnmu5AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAABdxJREFUeJztnF9oW1Ucxz9xUnAFRWyLNDA6RX4z\ndJTZDDpBM1txKqIM7XwYzkmZujkok8oUV3AvKu5B8MWHiSLzxXZlUu2wxbmtlDHaDZVlk4ODVQot\ntVVasxU6thwfcm+W1SbNvc1Jb9rzgUuSe88999dvf/mdf7+ckNYaixnuWGoDljNWXINYcQ1ixTWI\nFdcgVlyD3On3RhH5FGgANNCqlBoqmFXLBF+eKyIx4CGl1CagBfisoFYtE/yGhSbgOwCl1O/AvSJy\nd7bCoVBIx+NxHQqFSuLwYmsukfyGhfuB8xmfJ5xz/85X+MKFC9TW1lJKo8FC2Oo75s4hlOvi+vXr\n0VoTCuUsFhi82Jrrn+A3LIyS8lSXamDMZ13LFr/i9gEvAYjII8CoUipRMKuWCSG/sUVEPgYeB5LA\nW0qp37I+JBTSyzgsZC3oW1wvrFRx7QjNIFZcg1hxDVKofu6yIRqNAnDx4kWGhlLTJTt37vRVl/Vc\ng1jPdYjFYgAcO3YMgOHhYXbt2rWoOq3nGsT2c4FIJEJnZycAq1evpqamhqqqKiYmJha8N1c/F621\n8SP1GK1JTawH5nCnDfv6+vT4+LgeHx/XDQ0NnmzN9XfbsGCQFd2g7d+/H4Cmpiba2toAOHv2bMHq\nt55rkpUYc2OxmI7FYjqRSOhEIqE7Ojp0eXm5Li8vd+OojblBZ0V1xcLhMAADAwMAjI6OArB9+3aG\nh4fT5ex8rkcqKyvp6ekBYM2aNQA0NjYCcOnSpdvK2vncUqDUGjS343/w4EFdXV2tq6urc5YvKyvT\nZWVlenBwUM/OzurZ2VkdjUZ1NBrN1UjZBi3olNwg4sCBA+nXrq4u4FbDNB9u41VfX58eKJw7d86w\nlQ6lEhbq6up0XV1dum86MjJyW9907tHe3q7b29t1MpnUyWRSd3Z2evmq27AQdAIfFiorKwE4fPgw\nkJoSBGhububatWv/K79jxw4A9u3bB8Dp06cBaG1tNW7rXKznGiTQg4hwOJxedtm4cSMA8XgcgN7e\nXiYnJwGYmZkBYN26dezevRvAjfVMT08DcPTo0byXbQo1iMgrLIjIJ8BjTvmPgCHgCLCKVALeK0qp\n2bysWUEs6Lki8gTwjlLqWRG5D/gFOAEcV0p1isiHwIhS6vOsD/HpuVeuXEkPVd17M+11l2HcaxUV\nFfT39wO3YuzY2NhtZfOhmJ7bDww676eAcmAz8KZz7nugDcgqrl+6u7vTX323T5s5DxCJRIBUjgGk\nQsa2bdsAb2KawlPMFZHXSYWHLUqpKufcg8ARpdSj2e6Lx+O6trZ2sbYGlcXFXAAReYHUj0ueAv7I\np3IXE5nlkUiE3t5eAG7cuAHAhg0bmJqaWnTdHsNC1mt5dcVEZAvwPvCMUmoauCoidzmXw6QyzS1z\nWNBzReQe4BDwpFLqH+f0T8CLwDfO64/GLJxDTU0NACdPnkx3wdauXVusx3sin7DwMlABdIiIe+5V\n4AsReQP4E/jajHmlTaAHEZm4PQN3UHH9+nW2bt0KwOXLlxdvZAYrZpnHnVs4deoUcKsrtmfPHmPd\nLbvMUwIEflbMDQfuSK2lpQUIxiBhIaznGiTwMXcpsDG3BLDiGsSKaxArrkGsuAax4hrEimsQK65B\nrLgGKcoIbaViPdcgVlyDWHENYsU1iBXXIFZcg1hxDVKUNbSgb2Q8T4rs80A98LdT5JBSqsdrvcbF\nzdzIWEQeBr4ENpl+br44KbK1jn1uiuzPwHtKqR8WU3cxwoKnjYyXgH6g2XnvpsiuKkTFxQgLnjYy\nLjZKqZuA+8uVFuA4cBPYKyJvA38Be5VSk17rXooGLZBLwBkpsntJ/SThXaVUI/Ar8IGfOovhuYHf\nyDgjRfZpJ0X2RMblbnxmzRfDcwO9kXFGiuxzboqsiHSJyANOkc1A3E/dxj1XKXVGRM6LyBmcjYxN\nP9Mj86XIfgV8KyIzwFXgNT8V2/lcg9gRmkGsuAax4hrEimsQK65BrLgGseIa5D/RredT6GslEwAA\nAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 72x72 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "metadata": {
        "id": "1W6py5mlCy5n",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Preprocess Data\n",
        "\n",
        "Shuffle the training data.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "UYJZs7GgCy5p",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from sklearn.utils import shuffle\n",
        "\n",
        "X_train, y_train = shuffle(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Q4WFIdiuCy53",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup TensorFlow\n",
        "The `EPOCH` and `BATCH_SIZE` values affect the training speed and model accuracy.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "est-t83SCy55",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import tensorflow as tf\n",
        "\n",
        "EPOCHS = 100\n",
        "BATCH_SIZE = 128"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "cWVHSK0pCy59",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## TODO: Implement LeNet-5\n",
        "Implement the [LeNet-5](http://yann.lecun.com/exdb/lenet/) neural network architecture.\n",
        "\n",
        "This is the only cell you need to edit.\n",
        "### Input\n",
        "The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.\n",
        "\n",
        "### Architecture\n",
        "**Layer 1: Convolutional.** The output shape should be 28x28x6.\n",
        "\n",
        "**Activation.** Your choice of activation function.\n",
        "\n",
        "**Pooling.** The output shape should be 14x14x6.\n",
        "\n",
        "**Layer 2: Convolutional.** The output shape should be 10x10x16.\n",
        "\n",
        "**Activation.** Your choice of activation function.\n",
        "\n",
        "**Pooling.** The output shape should be 5x5x16.\n",
        "\n",
        "**Flatten.** Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using `tf.contrib.layers.flatten`, which is already imported for you.\n",
        "\n",
        "**Layer 3: Fully Connected.** This should have 120 outputs.\n",
        "\n",
        "**Activation.** Your choice of activation function.\n",
        "\n",
        "**Layer 4: Fully Connected.** This should have 84 outputs.\n",
        "\n",
        "**Activation.** Your choice of activation function.\n",
        "\n",
        "**Layer 5: Fully Connected (Logits).** This should have 10 outputs.\n",
        "\n",
        "### Output\n",
        "Return the result of the 2nd fully connected layer."
      ]
    },
    {
      "metadata": {
        "id": "Vgad6Ny0OjqN",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "connection_probability = tf.Variable(.9999)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "X1nJmcI7Cy5-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "from tensorflow.contrib.layers import flatten\n",
        "\n",
        "def LeNet(x, test_mode = False):    \n",
        "    # Hyperparameters\n",
        "    mu = 0\n",
        "    sigma = 0.1\n",
        "    layer_depth = {\n",
        "        'layer_1' : 6,\n",
        "        'layer_2' : 16,\n",
        "        'layer_3' : 120,\n",
        "        'layer_f1' : 84\n",
        "    }\n",
        "\n",
        "    \n",
        "    # TODO: Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.\n",
        "    conv1_w = tf.Variable(tf.truncated_normal(shape = [5,5,1,6],mean = mu, stddev = sigma))\n",
        "    conv1_b = tf.Variable(tf.zeros(6))\n",
        "    conv1 = tf.nn.conv2d(x,conv1_w, strides = [1,1,1,1], padding = 'VALID') + conv1_b \n",
        "    # TODO: Activation.\n",
        "    conv1 = tf.nn.relu(conv1)\n",
        "\n",
        "    # TODO: Pooling. Input = 28x28x6. Output = 14x14x6.\n",
        "    pool_1 = tf.nn.max_pool(conv1,ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID')\n",
        "    \n",
        "    # TODO: Layer 2: Convolutional. Output = 10x10x16.\n",
        "    conv2_w = tf.Variable(tf.truncated_normal(shape = [5,5,6,16], mean = mu, stddev = sigma))\n",
        "    conv2_b = tf.Variable(tf.zeros(16))\n",
        "    conv2 = tf.nn.conv2d(pool_1, conv2_w, strides = [1,1,1,1], padding = 'VALID') + conv2_b\n",
        "    # TODO: Activation.\n",
        "    conv2 = tf.nn.relu(conv2)\n",
        "    \n",
        "    ##### Inset probability connection from x to conv2\n",
        "    conv1p_w = tf.Variable(tf.truncated_normal(shape = [5,5,1,16], mean = mu, stddev = sigma))\n",
        "    conv1p_b = tf.Variable(tf.zeros(16))\n",
        "    conv2_2nd_input = tf.nn.conv2d(x, conv1p_w, strides = [1,3,3,1], padding = 'VALID') + conv1p_b\n",
        "    conv2_2nd_input = tf.nn.relu(conv2_2nd_input) #result = tf.cond(x < y, lambda: tf.add(x, z), lambda: tf.square(y))\n",
        "#     conv2 = tf.cond(tf.random.uniform(shape = tf.shape(connection_probability)) < connection_probability,lambda: conv2 + conv2_2nd_input, lambda: conv2 )\n",
        "    conv2 = tf.cond(tf.logical_and(tf.random.uniform(shape = tf.shape(connection_probability)) < connection_probability, tf.equal(test_mode,False)),lambda: conv2 + conv2_2nd_input, lambda: conv2 )\n",
        "\n",
        "#     if (np.random.random_sample()<connection_probability and test_mode == False):\n",
        "#         conv2 = conv2 + conv2_2nd_input\n",
        "#     if test_mode == True:\n",
        "#       conv2 = conv2 + conv2_2nd_input*connection_probability*0\n",
        "    \n",
        "    # TODO: Pooling. Input = 10x10x16. Output = 5x5x16.\n",
        "    pool_2 = tf.nn.max_pool(conv2, ksize = [1,2,2,1], strides = [1,2,2,1], padding = 'VALID') \n",
        "    \n",
        "    # TODO: Flatten. Input = 5x5x16. Output = 400.\n",
        "    fc1 = flatten(pool_2)\n",
        "    fdense = fc1\n",
        "    \n",
        "    # TODO: Layer 3: Fully Connected. Input = 400. Output = 120.\n",
        "    fc1_w = tf.Variable(tf.truncated_normal(shape = (400,120), mean = mu, stddev = sigma))\n",
        "    fc1_b = tf.Variable(tf.zeros(120))\n",
        "    fc1 = tf.matmul(fc1,fc1_w) + fc1_b\n",
        "    \n",
        "    # TODO: Activation.\n",
        "    fc1 = tf.nn.relu(fc1)\n",
        "\n",
        "    # TODO: Layer 4: Fully Connected. Input = 120. Output = 84.\n",
        "    fc2_w = tf.Variable(tf.truncated_normal(shape = (120,84), mean = mu, stddev = sigma))\n",
        "    fc2_b = tf.Variable(tf.zeros(84))\n",
        "    fc2 = tf.matmul(fc1,fc2_w) + fc2_b\n",
        "    # TODO: Activation.\n",
        "    fc2 = tf.nn.relu(fc2)\n",
        "    \n",
        "    \n",
        "    #################\n",
        "    ##### Inset probability connection from x to conv2\n",
        "    fc2p_w = tf.Variable(tf.truncated_normal(shape = [400,84], mean = mu, stddev = sigma))\n",
        "    fc2p_b = tf.Variable(tf.zeros(84))\n",
        "    fc2_2nd_input = tf.matmul(fdense,fc2p_w) + fc2p_b\n",
        "    fc2_2nd_input = tf.nn.relu(fc2_2nd_input)\n",
        "    fc2 = tf.cond(tf.logical_and(tf.random.uniform(shape = tf.shape(connection_probability)) < connection_probability, tf.equal(test_mode,False)),lambda: fc2 + fc2_2nd_input, lambda: fc2 )\n",
        "\n",
        "#     if (np.random.random_sample()<connection_probability and test_mode == False):\n",
        "#         fc2 = fc2 + fc2_2nd_input\n",
        "#     if test_mode == True:\n",
        "#         fc2 = fc2 + fc2_2nd_input*connection_probability*0\n",
        "    \n",
        "    ################\n",
        "    \n",
        "    # TODO: Layer 5: Fully Connected. Input = 84. Output = 10.\n",
        "    fc3_w = tf.Variable(tf.truncated_normal(shape = (84,10), mean = mu , stddev = sigma))\n",
        "    fc3_b = tf.Variable(tf.zeros(10))\n",
        "    logits = tf.matmul(fc2, fc3_w) + fc3_b\n",
        "    return logits"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aCx837c9Cy6E",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Features and Labels\n",
        "Train LeNet to classify [MNIST](http://yann.lecun.com/exdb/mnist/) data.\n",
        "\n",
        "`x` is a placeholder for a batch of input images.\n",
        "`y` is a placeholder for a batch of output labels.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "Mrb2FHFHCy6G",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "x = tf.placeholder(tf.float32, (None, 32, 32, 1))\n",
        "y = tf.placeholder(tf.int32, (None))\n",
        "one_hot_y = tf.one_hot(y, 10)\n",
        "is_testing= tf.placeholder(tf.bool) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "uk0pcAjmCy6M",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Training Pipeline\n",
        "Create a training pipeline that uses the model to classify MNIST data.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "3oYy_Y3FCy6N",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "rate = 0.001\n",
        "\n",
        "logits = LeNet(x,is_testing)\n",
        "cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits = logits, labels = one_hot_y)\n",
        "loss_operation = tf.reduce_mean(cross_entropy)\n",
        "optimizer = tf.train.AdamOptimizer(learning_rate = rate)\n",
        "training_operation = optimizer.minimize(loss_operation)\n",
        "new_prob = connection_probability.assign(connection_probability/1.0005)\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "KrMtHo6rCy6S",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Model Evaluation\n",
        "Evaluate how well the loss and accuracy of the model for a given dataset.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "OMR9XsLBCy6U",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(one_hot_y, 1))\n",
        "accuracy_operation = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
        "saver = tf.train.Saver()\n",
        "\n",
        "def evaluate(X_data, y_data):\n",
        "    num_examples = len(X_data)\n",
        "    total_accuracy = 0\n",
        "    sess = tf.get_default_session()\n",
        "    for offset in range(0, num_examples, BATCH_SIZE):\n",
        "        batch_x, batch_y = X_data[offset:offset+BATCH_SIZE], y_data[offset:offset+BATCH_SIZE]\n",
        "        accuracy = sess.run(accuracy_operation, feed_dict={x: batch_x, y: batch_y, is_testing: True})\n",
        "        total_accuracy += (accuracy * len(batch_x))\n",
        "    return total_accuracy / num_examples"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "2Jw7iox3Cy6X",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train the Model\n",
        "Run the training data through the training pipeline to train the model.\n",
        "\n",
        "Before each epoch, shuffle the training set.\n",
        "\n",
        "After each epoch, measure the loss and accuracy of the validation set.\n",
        "\n",
        "Save the model after training.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "c98G07hDQG2g",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import math"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "8QC1kt_DL1yU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def augment(images, labels,\n",
        "            resize=None, # (width, height) tuple or None\n",
        "            horizontal_flip=False,\n",
        "            vertical_flip=False,\n",
        "            rotate=0, # Maximum rotation angle in degrees\n",
        "            crop_probability=0, # How often we do crops\n",
        "            crop_min_percent=0.6, # Minimum linear dimension of a crop\n",
        "            crop_max_percent=1.,  # Maximum linear dimension of a crop\n",
        "            mixup=0):  # Mixup coeffecient, see https://arxiv.org/abs/1710.09412.pdf\n",
        "  if resize is not None:\n",
        "    images = tf.image.resize_bilinear(images, resize)\n",
        "  \n",
        "  # My experiments showed that casting on GPU improves training performance\n",
        "  if images.dtype != tf.float32:\n",
        "    images = tf.image.convert_image_dtype(images, dtype=tf.float32)\n",
        "    images = tf.subtract(images, 0.5)\n",
        "    images = tf.multiply(images, 2.0)\n",
        "  labels = tf.to_float(labels)\n",
        "\n",
        "  with tf.name_scope('augmentation'):\n",
        "    shp = tf.shape(images)\n",
        "    batch_size, height, width = shp[0], shp[1], shp[2]\n",
        "    width = tf.cast(width, tf.float32)\n",
        "    height = tf.cast(height, tf.float32)\n",
        "\n",
        "    # The list of affine transformations that our image will go under.\n",
        "    # Every element is Nx8 tensor, where N is a batch size.\n",
        "    transforms = []\n",
        "    identity = tf.constant([1, 0, 0, 0, 1, 0, 0, 0], dtype=tf.float32)\n",
        "    if horizontal_flip:\n",
        "      coin = tf.less(tf.random_uniform([batch_size], 0, 1.0), 0.5)\n",
        "      flip_transform = tf.convert_to_tensor(\n",
        "          [-1., 0., width, 0., 1., 0., 0., 0.], dtype=tf.float32)\n",
        "      transforms.append(\n",
        "          tf.where(coin,\n",
        "                   tf.tile(tf.expand_dims(flip_transform, 0), [batch_size, 1]),\n",
        "                   tf.tile(tf.expand_dims(identity, 0), [batch_size, 1])))\n",
        "\n",
        "    if vertical_flip:\n",
        "      coin = tf.less(tf.random_uniform([batch_size], 0, 1.0), 0.5)\n",
        "      flip_transform = tf.convert_to_tensor(\n",
        "          [1, 0, 0, 0, -1, height, 0, 0], dtype=tf.float32)\n",
        "      transforms.append(\n",
        "          tf.where(coin,\n",
        "                   tf.tile(tf.expand_dims(flip_transform, 0), [batch_size, 1]),\n",
        "                   tf.tile(tf.expand_dims(identity, 0), [batch_size, 1])))\n",
        "\n",
        "    if rotate > 0:\n",
        "      angle_rad = rotate / 180 * math.pi\n",
        "      angles = tf.random_uniform([batch_size], -angle_rad, angle_rad)\n",
        "      transforms.append(\n",
        "          tf.contrib.image.angles_to_projective_transforms(\n",
        "              angles, height, width))\n",
        "\n",
        "    if crop_probability > 0:\n",
        "      crop_pct = tf.random_uniform([batch_size], crop_min_percent,\n",
        "                                   crop_max_percent)\n",
        "      left = tf.random_uniform([batch_size], 0, width * (1 - crop_pct))\n",
        "      top = tf.random_uniform([batch_size], 0, height * (1 - crop_pct))\n",
        "      crop_transform = tf.stack([\n",
        "          crop_pct,\n",
        "          tf.zeros([batch_size]), top,\n",
        "          tf.zeros([batch_size]), crop_pct, left,\n",
        "          tf.zeros([batch_size]),\n",
        "          tf.zeros([batch_size])\n",
        "      ], 1)\n",
        "\n",
        "      coin = tf.less(\n",
        "          tf.random_uniform([batch_size], 0, 1.0), crop_probability)\n",
        "      transforms.append(\n",
        "          tf.where(coin, crop_transform,\n",
        "                   tf.tile(tf.expand_dims(identity, 0), [batch_size, 1])))\n",
        "\n",
        "    if transforms:\n",
        "      images = tf.contrib.image.transform(\n",
        "          images,\n",
        "          tf.contrib.image.compose_transforms(*transforms),\n",
        "          interpolation='BILINEAR') # or 'NEAREST'\n",
        "\n",
        "    def cshift(values): # Circular shift in batch dimension\n",
        "      return tf.concat([values[-1:, ...], values[:-1, ...]], 0)\n",
        "\n",
        "    if mixup > 0:\n",
        "      mixup = 1.0 * mixup # Convert to float, as tf.distributions.Beta requires floats.\n",
        "      beta = tf.distributions.Beta(mixup, mixup)\n",
        "      lam = beta.sample(batch_size)\n",
        "      ll = tf.expand_dims(tf.expand_dims(tf.expand_dims(lam, -1), -1), -1)\n",
        "      images = ll * images + (1 - ll) * cshift(images)\n",
        "      labels = lam * labels + (1 - lam) * cshift(labels)\n",
        "\n",
        "  return images, labels"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "siI7jT2RUvYP",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def augment_data(dataset, dataset_labels, augementation_factor=1, use_random_rotation=True, use_random_shear=True, use_random_shift=True, use_random_zoom=True):\n",
        "\taugmented_image = []\n",
        "\taugmented_image_labels = []\n",
        "\n",
        "\tfor num in range (0, dataset.shape[0]):\n",
        "\n",
        "\t\tfor i in range(0, augementation_factor):\n",
        "\t\t\t# original image:\n",
        "\t\t\taugmented_image.append(dataset[num])\n",
        "\t\t\taugmented_image_labels.append(dataset_labels[num])\n",
        "\n",
        "\t\t\tif use_random_rotation:\n",
        "\t\t\t\taugmented_image.append(tf.contrib.keras.preprocessing.image.random_rotation(dataset[num], 20, row_axis=0, col_axis=1, channel_axis=2))\n",
        "\t\t\t\taugmented_image_labels.append(dataset_labels[num])\n",
        "\n",
        "\t\t\tif use_random_shear:\n",
        "\t\t\t\taugmented_image.append(tf.contrib.keras.preprocessing.image.random_shear(dataset[num], 0.2, row_axis=0, col_axis=1, channel_axis=2))\n",
        "\t\t\t\taugmented_image_labels.append(dataset_labels[num])\n",
        "\n",
        "\t\t\tif use_random_shift:\n",
        "\t\t\t\taugmented_image.append(tf.contrib.keras.preprocessing.image.random_shift(dataset[num], 0.2, 0.2, row_axis=0, col_axis=1, channel_axis=2))\n",
        "\t\t\t\taugmented_image_labels.append(dataset_labels[num])\n",
        "\n",
        "# \t\t\tif use_random_zoom:\n",
        "# \t\t\t\taugmented_image.append(tf.contrib.keras.preprocessing.image.random_zoom(dataset[num], 0.9, row_axis=0, col_axis=1, channel_axis=2))\n",
        "# \t\t\t\taugmented_image_labels.append(dataset_labels[num])\n",
        "\n",
        "\treturn np.array(augmented_image), np.array(augmented_image_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "Of2PvjkIQikg",
        "colab_type": "code",
        "outputId": "a00aae1f-f60b-4bf0-d7d6-f267d90bf5e9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "(X_train.shape)"
      ],
      "execution_count": 133,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(55000, 32, 32, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "metadata": {
        "id": "RbERCioKL6dc",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# These can be any tensors of matching type and dimensions.\n",
        "images = tf.placeholder(tf.uint8, shape=(None, None, None, 3))\n",
        "labels = tf.placeholder(tf.uint64, shape=(None))\n",
        "\n",
        "images, labels = augment(tf.convert_to_tensor(X_train), tf.convert_to_tensor(y_train),\n",
        "                         horizontal_flip=True, rotate=15, crop_probability=0.8, mixup=4)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "-bwtFMk5U0HH",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "images, labels = augment_data(X_train, y_train)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VZA2k-ZQR7kU",
        "colab_type": "code",
        "outputId": "474fb476-1e60-4c08-8a6e-1d5b9ba594df",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "cell_type": "code",
      "source": [
        "images.shape"
      ],
      "execution_count": 136,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(220000, 32, 32, 1)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 136
        }
      ]
    },
    {
      "metadata": {
        "id": "MbueMHwfEbi4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "EPOCHS = 66"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "bKU1MIyJCy6Y",
        "colab_type": "code",
        "outputId": "0cba6b3c-408e-4525-9b60-46c1e77e01b7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 4872
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.Session() as sess:\n",
        "    sess.run(tf.global_variables_initializer())\n",
        "    num_examples = len(X_train)\n",
        "    best_accuracy_valid = 0\n",
        "    \n",
        "    print(\"Training...\")\n",
        "    print()\n",
        "    for i in range(EPOCHS):\n",
        "        X_train, y_train = shuffle(images, labels)\n",
        "        for offset in range(0, num_examples, BATCH_SIZE):\n",
        "            end = offset + BATCH_SIZE\n",
        "            batch_x, batch_y = X_train[offset:end], y_train[offset:end]\n",
        "            tr_op = sess.run([training_operation], feed_dict={x: batch_x, y: batch_y, is_testing : False})\n",
        "            \n",
        "            prob = sess.run(new_prob)\n",
        "        print(prob)\n",
        "        validation_accuracy = evaluate(X_validation, y_validation)\n",
        "        print(\"EPOCH {} ...\".format(i+1))\n",
        "        print(\"Validation Accuracy = {:.3f}\".format(validation_accuracy))\n",
        "        print()\n",
        "        if (validation_accuracy >= best_accuracy_valid):\n",
        "          best_accuracy_valid = validation_accuracy\n",
        "          saver.save(sess, './lenet')\n",
        "        \n",
        "#     saver.save(sess, './lenet')\n",
        "    print(\"Model saved\")"
      ],
      "execution_count": 137,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Training...\n",
            "\n",
            "0.8065128\n",
            "EPOCH 1 ...\n",
            "Validation Accuracy = 0.963\n",
            "\n",
            "0.65052795\n",
            "EPOCH 2 ...\n",
            "Validation Accuracy = 0.979\n",
            "\n",
            "0.5247114\n",
            "EPOCH 3 ...\n",
            "Validation Accuracy = 0.979\n",
            "\n",
            "0.42322883\n",
            "EPOCH 4 ...\n",
            "Validation Accuracy = 0.983\n",
            "\n",
            "0.34137297\n",
            "EPOCH 5 ...\n",
            "Validation Accuracy = 0.983\n",
            "\n",
            "0.27534926\n",
            "EPOCH 6 ...\n",
            "Validation Accuracy = 0.985\n",
            "\n",
            "0.22209492\n",
            "EPOCH 7 ...\n",
            "Validation Accuracy = 0.989\n",
            "\n",
            "0.17914018\n",
            "EPOCH 8 ...\n",
            "Validation Accuracy = 0.989\n",
            "\n",
            "0.14449328\n",
            "EPOCH 9 ...\n",
            "Validation Accuracy = 0.987\n",
            "\n",
            "0.11654733\n",
            "EPOCH 10 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.09400628\n",
            "EPOCH 11 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "0.07582486\n",
            "EPOCH 12 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "0.0611598\n",
            "EPOCH 13 ...\n",
            "Validation Accuracy = 0.989\n",
            "\n",
            "0.049331047\n",
            "EPOCH 14 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.039790098\n",
            "EPOCH 15 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "0.032094438\n",
            "EPOCH 16 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.025887143\n",
            "EPOCH 17 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.020880401\n",
            "EPOCH 18 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.016841987\n",
            "EPOCH 19 ...\n",
            "Validation Accuracy = 0.988\n",
            "\n",
            "0.013584643\n",
            "EPOCH 20 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0109572895\n",
            "EPOCH 21 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "0.008838077\n",
            "EPOCH 22 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0071287365\n",
            "EPOCH 23 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0057499935\n",
            "EPOCH 24 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.004637909\n",
            "EPOCH 25 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0037409083\n",
            "EPOCH 26 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0030173932\n",
            "EPOCH 27 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.0024338092\n",
            "EPOCH 28 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "0.0019630946\n",
            "EPOCH 29 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "0.0015834197\n",
            "EPOCH 30 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "0.0012771745\n",
            "EPOCH 31 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0010301609\n",
            "EPOCH 32 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "0.000830921\n",
            "EPOCH 33 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.000670215\n",
            "EPOCH 34 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.00054059096\n",
            "EPOCH 35 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.0004360369\n",
            "EPOCH 36 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.00035170466\n",
            "EPOCH 37 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.00028368252\n",
            "EPOCH 38 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "0.00022881645\n",
            "EPOCH 39 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "0.00018456187\n",
            "EPOCH 40 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "0.00014886643\n",
            "EPOCH 41 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "0.00012007465\n",
            "EPOCH 42 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "9.685145e-05\n",
            "EPOCH 43 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "7.811978e-05\n",
            "EPOCH 44 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "6.30109e-05\n",
            "EPOCH 45 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "5.0824183e-05\n",
            "EPOCH 46 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "4.0994473e-05\n",
            "EPOCH 47 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "3.306585e-05\n",
            "EPOCH 48 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "2.667071e-05\n",
            "EPOCH 49 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "2.151243e-05\n",
            "EPOCH 50 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "1.7351793e-05\n",
            "EPOCH 51 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "1.3995846e-05\n",
            "EPOCH 52 ...\n",
            "Validation Accuracy = 0.990\n",
            "\n",
            "1.128896e-05\n",
            "EPOCH 53 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "9.105599e-06\n",
            "EPOCH 54 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "7.3445153e-06\n",
            "EPOCH 55 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "5.9240365e-06\n",
            "EPOCH 56 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "4.778285e-06\n",
            "EPOCH 57 ...\n",
            "Validation Accuracy = 0.989\n",
            "\n",
            "3.8541334e-06\n",
            "EPOCH 58 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "3.1087186e-06\n",
            "EPOCH 59 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "2.5074705e-06\n",
            "EPOCH 60 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "2.0225089e-06\n",
            "EPOCH 61 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "1.6313427e-06\n",
            "EPOCH 62 ...\n",
            "Validation Accuracy = 0.991\n",
            "\n",
            "1.3158307e-06\n",
            "EPOCH 63 ...\n",
            "Validation Accuracy = 0.992\n",
            "\n",
            "1.0613409e-06\n",
            "EPOCH 64 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "8.560711e-07\n",
            "EPOCH 65 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "6.905012e-07\n",
            "EPOCH 66 ...\n",
            "Validation Accuracy = 0.993\n",
            "\n",
            "Model saved\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "yVIeYA_cCy6e",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Evaluate the Model\n",
        "Once you are completely satisfied with your model, evaluate the performance of the model on the test set.\n",
        "\n",
        "Be sure to only do this once!\n",
        "\n",
        "If you were to measure the performance of your trained model on the test set, then improve your model, and then measure the performance of your model on the test set again, that would invalidate your test results. You wouldn't get a true measure of how well your model would perform against real data.\n",
        "\n",
        "You do not need to modify this section."
      ]
    },
    {
      "metadata": {
        "id": "O4b5lSD5H_q9",
        "colab_type": "code",
        "outputId": "4a97ff23-13fd-4f3d-b5e0-8d3aa32124b7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.Session() as sess:\n",
        "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
        "\n",
        "    test_accuracy = evaluate(X_validation, y_validation)\n",
        "    print(\"Validation Accuracy = {:.3f}\".format(test_accuracy))"
      ],
      "execution_count": 140,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from ./lenet\n",
            "Validation Accuracy = 0.993\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bJONJ7CeCy6f",
        "colab_type": "code",
        "outputId": "7f6c55ea-6292-4636-874e-2d595fa5431b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "with tf.Session() as sess:\n",
        "    saver.restore(sess, tf.train.latest_checkpoint('.'))\n",
        "\n",
        "    test_accuracy = evaluate(X_test, y_test)\n",
        "    print(\"Test Accuracy = {:.3f}\".format(test_accuracy))"
      ],
      "execution_count": 139,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Restoring parameters from ./lenet\n",
            "Test Accuracy = 0.993\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "qvRqnUrrCy6m",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}