Semi_Supervised_Deep_Learning_Workshop_Exercise.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Semi Supervised Deep Learning Workshop Exercise.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "TPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8EMoTPBwCftl"
      },
      "source": [
        "# Semi-Supervised Deep Learning\n",
        "\n",
        "Welcome to the hands-on component of the Semi-Supervised Deep Learning (SSDL) workshop. You should by this stage have a high level understanding of a few semi-supervised techniques for training a deep learning model. The exercises in this notebook are intended to help you solidify your understanding by guiding you through the process of implementing some of the techniques that you have learned.\n",
        "\n",
        "It is worth noting that these exercises will focus more on the implementation of SSDL and less so on getting the best possible results from a dataset or model architecture. We do not aim to prove that SSDL will be successful on the toy dataset and model architecture we will be using. Instead, we will look at how SSDL can be implemented and applied to deep learning at a broad level."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M2XuKeJWQV--"
      },
      "source": [
        "## Let's start with the data\n",
        "\n",
        "We will be using the popular [CIFAR-10](https://www.cs.toronto.edu/~kriz/cifar.html) dataset. The CIFAR-10 dataset consists of 60000 32x32 colour images across 10 classes, with 6000 images per class. Of these 6000 images per class, 5000 are intended as training images and 10000 reserved for validating your model.\n",
        "\n",
        "The following code downloads the CIFAR-10 dataset and preprocesses it into a slightly different format for your convenience. Specifically it:\n",
        "\n",
        "*   Normalises all images pixel values to a range 0 to 1.\n",
        "*   Removes redundant dimensions of size 1.\n",
        "*   Splits the training set into 10 lists with each list containing all images for a class.\n",
        "\n",
        "The final data structure is named \"dataset\" and is a Python dictionary with the following structure:\n",
        "\n",
        "```\n",
        "{\n",
        "  'train_images' : <contains all 50000 training images>\n",
        "  'train_labels' : <contains the integer labels for the 50000 training images>\n",
        "  'train_images_grouped' : <a list of 10 lists, each containing the training images for each class>\n",
        "  'test_images' : <contains all 10000 test images>\n",
        "  'test_labels' : <contains the integer labels for the 10000 test images>\n",
        "}\n",
        "```\n",
        "\n",
        "Run the code below to prepare your data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JkN7shTctK3M"
      },
      "source": [
        "# Load the modules we will be using today\n",
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers as L\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow import keras\n",
        "import random\n",
        "import pickle\n",
        "\n",
        "# Preprocess the dataset for convenience\n",
        "(train_images, train_labels), (test_images, test_labels) = keras.datasets.cifar10.load_data()\n",
        "\n",
        "# Normalise the images and remove redundant dimensions\n",
        "train_images = train_images / 255.0\n",
        "test_images = test_images / 255.0\n",
        "train_labels = np.squeeze(train_labels)\n",
        "test_labels = np.squeeze(test_labels)\n",
        "\n",
        "# Populate the \"dataset\" dictionary with the CIFAR-10 data\n",
        "dataset = {}\n",
        "dataset['train_images'] = train_images\n",
        "dataset['train_labels'] = train_labels\n",
        "dataset['train_images_grouped'] = [ [] for i in range(10) ]\n",
        "dataset['test_images'] = test_images\n",
        "dataset['test_labels'] = test_labels\n",
        "\n",
        "for i in range(train_labels.shape[0]):\n",
        "  dataset['train_images_grouped'][train_labels[i]].append(train_images[i])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l_q3OI0CW3uA"
      },
      "source": [
        "Note that the \"dataset\" object does not contain \"train_labels_grouped\" as the index of each list can be used to represent the integer label for that list.\n",
        "\n",
        "Next, spend some time in the code block below to explore and understand the structure and contents of the dataset you will be working with. This understanding is important for you to complete your model training script in a later stage of this exercise.\n",
        "\n",
        "Some example code has been provided for your convenience. Can you identify the 10 object classes in this dataset?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2UUd03rL3WRe"
      },
      "source": [
        "### Inspect the dataset ###\n",
        "print('Dataset keys:', [k for k in dataset.keys()])\n",
        "print('Training images per class:', [len(c) for c in dataset['train_images_grouped']])\n",
        "print('Image shape:', dataset['train_images_grouped'][0][0].shape)\n",
        "\n",
        "fig=plt.figure(figsize=(20, 20))\n",
        "rows = 10 # Show all 10 classes\n",
        "columns = 5 # Show 5 samples per class\n",
        "for i in range(rows):\n",
        "  for j in range(columns):\n",
        "    fig.add_subplot(rows, columns, (i)*columns+j+1)\n",
        "    plt.imshow(dataset['train_images_grouped'][i][j])\n",
        "plt.show()\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Hwc_yUtdY2QV"
      },
      "source": [
        "## Partially labelled data\n",
        "\n",
        "You would be aware that all images in your dataset have been provided a corresponding label. If you were to train a model with all the data as is, you would have a fully supervised model. \n",
        "\n",
        "In the spirit of semi-supervised learning, we will need to simulate the lack of labelled data. One simply way to do so is to extract a small subset of the images and their corresponding labels; you may then pretend that everything else does not have a label.\n",
        "\n",
        "The code below extracts this \"supervised\" data subset. Note that all object classes have an equal number of extracted samples. It is suggested that you start with extracting only 1% of the dataset so that testing and debugging of your subsequent code runs quicker. You may certainly increase this portion at a later stage."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_G2rX1H5KJCj"
      },
      "source": [
        "# Extract the first 1% of labelled samples\n",
        "take_first_X = 50\n",
        "train_images_supervised = []; train_labels_supervised = []\n",
        "for i in range(10):\n",
        "    train_images_supervised += dataset['train_images_grouped'][i][:take_first_X]\n",
        "    train_labels_supervised += [i] * take_first_X\n",
        "\n",
        "train_images_supervised = np.asarray(train_images_supervised)\n",
        "train_labels_supervised = np.asarray(train_labels_supervised, np.int32)\n",
        "\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mu9hbr7nKoa6"
      },
      "source": [
        "## Defining the model architecture\n",
        "\n",
        "Now that we have our data ready, let's turn our attention to the model architecture. Remember that our aim is not to get the best possible performance out of the data we have, but to focus on learning how to implement SSDL techniques. With that in mind, we will be looking at simple toy models that we would be able to build from the ground up.\n",
        "\n",
        "You would be familiar with various SSDL techniques by now. One commonality among many of these techniques is a neural network that has two \"heads\" that can be trained on two different tasks.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Rtd2pC4Ne2tr"
      },
      "source": [
        "![image.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAgEAAAGOCAYAAAAO6NDxAAAgAElEQVR4Ae2dTaslR37mS9/A/Q0M5Q/gRX8AeVXGQws3YvDWvSwzCyMtDEZYgoGhrxbt3WBqIa9GtREWsjCSaOOGKmO0sN3tbmYoQ1c3xojRgFHDUMyoGNAZnlt+KuPGjYjMOPlyIiN/AYfMjIzX3z9vPk++nHPvnEgQgAAEIAABCBySwJ1DzppJQwACEIAABCBwwgRwEEAAAhCAAAQOSgATcNDAM20IQAACEIAAJoBjAAIQgAAEIHBQApiAgwaeaUMAAhCAAAQwARwDEIAABCAAgYMSwAQcNPBMGwIQgAAEIIAJ4BiAAAQgAAEIHJQAJuCggWfaEIAABCAAAUwAxwAEIAABCEDgoAQwAQcNPNOGAAQgAAEIYAI4BiAAAQhAAAIHJYAJOGjgmTYELkHg8ePHpzt37pyurq4u0T19QgACEQFMQASETQiUCDx9+vRaxCRkEjRSHQFMQB0vSkNgbQKYgLUJ035XBHQFe+/evdPdu3dP9+/f72puW0wGE7AFZfqAwHQCmIDprCgJgWvxlxGQAdDdgDnJdxWOdGscEzDniKEuBJYnMO8stvx4aBECzRKwgGn58OHDaxOg9XMTJuBcctSDAASWIoAJWIok7XRPILz6t4CnHgnoDoEeGcRJ+S6v/dqOP2rXyf2FZcL9LmdzEpbzPi/9+CIuKzMTJ88tbC/uN9yn9dR81a76dVmtu//47ofzXVbLOClP9fRxuXhccR22IQCBMoHbf2nl8uyFwGEJSMRCsbOQx0AkUGE571e+TYDyLLaxIGpfqm2bAgmmk+9IhGLucqFAWoy1dLKYhu1ZjMMxqW3Px2MO56H2NLewbeW5T/fnsaps3L7yxuagMvp4LG6XJQQgcD4BTMD57Kh5IAIpcbSIhuIlJDmhUn4onhbUUBBV32IZirNRS1hDEYzbdLk4X/VikfZYwzGp7VQ5t5vbbz5mkZuDmYVzjsfqvuJ8betDggAEliPAX9RyLGmpYwKpq2sLXyjKQiChivOcHwpuzgS4rxTOcJ/7t/CG5WOzEG+7bEpowzG6nJdxeedrGe4LxxmW8ZhtArw9ZQ5h+2GbrEMAAucTwAScz46aByIgAUpdIStfnzBpe44JyF1tq49QXC2gHkO8DMdQMgEulzMlntvYfvVvAxGO0/W19JhjExCP3dsem+qG7Ydtsg4BCJxP4ObZ6/x2qAmBbglYuCxMqWV4JVsSXIukYOVEVcKnPlIpFFePK+w7VUd5pTFZaD2ecIxhe2P7Q5EOxxm24THHJmDKHML2wzZZhwAEzieQPtOc3x41IdAdAQuaRDBOFjULqfanBNflQoG1qFoQ3ba2JXiqEyflh31NFcbUmNR23J7K6ZNLuf0ScbVlMc/NwfnhnKfOYWq53NjJhwAEbhPABNxmQg4EbhCIhfLGzuBNfpsEC51F3AYgJWK5tpWvT5h8h8D9aJ/7svi6vMqGeVNNgMU8FGmN38bD+1NmxmU8Bo0/NBQeq/LD9p0fjldtxHNQvbBf98MSAhA4n8DNs8z57VATAl0SsOjFAhVONlXGdw8kXPooSRBjEQsNgsu5bYmg62sZCqrLaOn+w7LxeKeagFx7YX++gxH2F4p6WDYsY5OgvLj8lDmoXswv7It1CECgngAmoJ4ZNSAAAQhAAAJdEMAEdBFGJgEBCEAAAhCoJ4AJqGdGDQhAAAIQgEAXBDABXYSRSUAAAhCAAATqCWAC6plRAwIQgAAEINAFAUxAF2FkEhCAAAQgAIF6ApiAembUgAAEIAABCHRBABPQRRiZBAQgAAEIQKCeACagnhk1IAABCEAAAl0QwAR0EUYmAQEIQAACEKgngAmoZ0YNCEAAAhCAQBcEMAFdhJFJQAACEIAABOoJYALqmVEDAhCAAAQg0AUBTEAXYWQSEIAABCAAgXoCmIB6ZtSAAAQgAAEIdEEAE9BFGI8xiU8//fT6/9A/ePDg9Nlnn52ePHly+vrrr48xeWZ5UQI6znS86bjT8Xd1dXXS8UiCwN4JYAL2HsEDjf/tt98+pT6YggMdBBtNNRb91HGnPBIE9k4AE7D3CB5o/LkTcZyPKTjQQbHQVKeKfnysLdQ9zUDgYgQwARdDT8e1BMITsG/Lhnm5dUxBLen+y58j+j6OwuOsf1LMsHcCmIDeI9zR/FIn3zknc94p6OjgGJnKksdJ6jgc6Z7dEGiWACag2dAwsJjAlJPvkif7uH+290NgzeNgynG4H1KM9OgEMAFHPwJ2NP9zTr5risGO0HU/1C3jfM5x2H0AmOBuCWACdhu64w18iZPvlmJxvAhtN+NLxnGJ43A7UvQEgTIBTECZD3sbIrDGyfeSYtIQ2uaH0lKc1jgOmw8AA+yWACag29D2N7EtTr4tiU1/EZw+o5bjsMVxOJ0UJSEwjwAmYB4/am9I4BIn35bFaEP0q3e1J86XOA5XDwAdHJYAJuCwod/fxFs4+e5JrFqO8J45tnActhxbxrYvApiAfcXr0KNt8eS7ZzHb8mDqiVOLx+GWsaSvvghgAvqKZ9ez2cPJtyexm3Mw9cxhD8fhnNhR91gEMAHHiveuZ7vHk2/PYhgeTEeZp+a8x+MwjBXrEAgJYAJCGqw3TaCHk28vYtnLPM454Hs4Ds+ZN3X6JIAJ6DOuXc6qx5PvXDHdMtD6Xwv84ybuBGx5zNHX+gQwAeszpoeFCPRoAmI0tabgvffei5s4ffHFF6e33nrr9Prrr59effXVSR+VVR3VTSX1E/JPrfu/7PX+j5nCuadYkQeBPRHABOwpWgcf6xFPvlNMQXhYSMTv3bs3SfhTBkF1U0YgZO/1o4h+yFfrnr+WJAjsnQAmYO8RPND4OfmeTjYFORa6mk+Je02e2ohT2F/vV/rx3OPtkEW8j20I7I0AJmBvETvweDn5DsHPsQgfAXz88cdDhZE1lbVRUBtxyvUXlzvCNiyOEOXjzBETcJxY736mnHyHEOZYWMi1rE2lurn+avvooTwseogiczABTIBJsGyeACffIUQ5FiUhH2qn10p1c/2lW+o7FxZ9x/dos8MEHC3iO54vJ98heDkWJSEfaqfXSnVz/aVb6jsXFn3H92izW8wEPH/+/KQXhv76r//69NVXX50++OCD0/vvv3/613/91+ultp8+fXq9rjL/8A//cL2upbZd1vW+/PLLyfXUruu5P7en5SeffHL62c9+9rK/R48eXa+r3kcfffSyb5XN1VN9f549e3a046SJ+XLyHcKQY1ES8qF2eq1UN9dfuqW+c2FxufiiM8uzn20CJPgSVYm5BFjiLZFUsHpKmpPmJiOgecq4YAa2jTAn34F3jkVJyIfa6bVS3Vx/6Zb6zoXF9vFFZ9ZjPssE6Epawqg7AL2J/hhy3/XACIyRWm4/J9+BZY5FSciH2um1Ut1cf+mW+s6FxbbxRWfWveA82wRIBHWb/egiqDsCn3/++bZ/FQftjZPvEPgci5KQD7XTa6W6uf7SLfWdC4vt4ovOvGC9ps6cZQLszI529Z879GUCjm6GcmyWzOfkO9DMsSgJ+VA7vVaqm+sv3VLfubDYJr7ozE3Oa+lMtQnQsxm9TIfo3QyQ7oqIDWk9Apx8B7Y5FiUhH2qn10p1c/2lW+o7FxbrxxedSTNeQ2eqTYDeAdCLcaSbBORa9XIkaT0CnHwHtjkWJSEfaqfXSnVz/aVb6jsXFuvHF51JM15DZ6pMgN1Zenjk6vkVd0jWOw44+Q5scyxKQj7UTq+V6ub6S7fUdy4s1o0vOlPmu7TOVJmANVxIebr72qs7AfoKIWkdApx8B645FiUhH2qn10p1c/2lW+o7FxbrxhedKfNdWmeqTQAvA+YDpLsAcmmkdQhw8h245liUhHyonV4r1c31l26p71xYrBtfmQB0Js94aZ2pMgG8EJgPjPbowNVXOUjrEODkO3DNsSgJ+VA7vVaqm+sv3VLfubBYN77oTJnv0jpTZQJ0G4Jn3uUA6WeHSesQ4OQ7cM2xKAn5UDu9Vqqb6y/dUt+5sFg3vujMON8ldabKBOjrCaQyAb3VSlqHACffgWuORUnIh9rptVLdXH/plvrOhcW68UVnxvkuqTNVJmBJ9zE+zX2WgNF6cePkO7DNsSgJ+VA7vVaqm+sv3VLfubBYN76cQ8f5LsmoygTw+wDbBme8t2OV4OQ7xDvHoiTkQ+30Wqlurr90S33nwmLd+KIz43wvZgKW7Hh8mvsswQG8Xtw4+Q5scyxKQj7UTq+V6ub6S7fUdy4s1o0vOjPOd0mdqboTQHDGgwOjcUbnluDkO5DLsSgJ+VA7vVaqm+sv3VLfubBYN76cQ8f5LsmoygQs6T7Gp7leiXfeeef0a7/2a6c//MM/XLwTfb2FtA4BTr4D1xyLkpAPtdNrpbq5/tIt9Z0Li3Xji86M811SZ6pMwJLuY3ya65XQye7OnTsvPzIFSyV+LGgpkrfb4eQ7MMmxKAn5UDu9Vqqb6y/dUt+5sFg3vujMON8ldabKBHzwwQfjo1upxK9+9avThx9+ePrzP//z05/+6Z9eX8X/7u/+7un3f//3TxJxfbTvxz/+8egIYhMgQ/Drv/7r1/VHK48U6OUAHpnmRXZz8h2w51iUhHyonV4r1c31l26p71xYrBtfdGac75I6U2UC9HOOWyYJusT9N3/zN69v34dX72PrMgi5K/yUCXB76ktm4tz06NGjc6tSb4QAJ98BUI5FSciH2um1Ut1cf+mW+s6FxbrxRWfG+S6pM1UmYEn3kZumrvh/9KMfXQu/hdlLXa3rROWrf+ePLWUIwjsEJRPgtmQGfvnLX+aGmc3//PPPs/vYMY8AJ9+BX45FSciH2um1Ut1cf+mW+s6FxbrxRWfG+S6pM1UmYO3fxc+Jv4RZL/LJIIRJ4m7R9lLl1I7vIDhfS5kHtTHFBLie6tSYgS0O4JDBkdY5+Q7RzrEoCflQO71WqpvrL91S37mwWDe+l9YZ/SvjMPWuM1UmQL/pvEaSMEtsLby5pcqESUKfKhveztd6XEZ3FOK8eFt3AsI8vYcwJfGvhKdQOq8MJ9+BW45FSciH2um1Ut1cf+mW+s6FxbrxRWfG+S6pM1UmYI2rXF216+pdguuv7aWE24IcXpXnjIOcm5MMhtt3G6WlxD98dBAajSlfKeR3r01++SUn34FpjkVJyIfa6bVS3Vx/6Zb6zoXFuvFFZ8a/ur6kzlSZgKUdmq6uQ4G2+Eq4c0IdXpGnbtOonoTcSaYh11YqPxyDDUfYT3iXwX2EyzUO4LD9I69z8h2in2NREvKhdnqtVDfXX7ql6bn3799P/n3eu3fv9PDhw+kNbVhyLRYbTqHpro6qM9/97ndf/i1sqTNVJmDJZzXhFbbFOJx4fDveZVTPKXdbX8bCKdWP24qXYT31r/aVwjbCMu4jXPbyQxfhnFpZX+PkqzeR4+Mg3P6zP/uzVqZ/Yxw5FiUhv9FAYqNUN9dfopmqLIm9eMsMXF1dXX+0/q1vfes6/9vf/vbpJz/5SVWbaxdei8Xa495L++jMi7vipXgtqTNVJmCJq9zS8/9QYHXrPTwZez281e+81FL9KOXMRKqO8lxPv0nguw7xY4fQrMSBWoJR3CbbLwiscfJ9/Pjx9XF29+7dlyJkMdKyNQHysZBjURJy180tS3Vz/eXamppvE6A4hOnf/u3fro2B/iZlBFpKa7FoaY6XHMsS59A96sz3vve9G5q3lc5UmYAl3Ed4VZ0SYou8RDi1X3m6Ta/b9rn9LlNqI1fX/fuPQP3InITl4xcUXVbLJQ7gsD3WBwJrnHxtAiRGe0o5FiUhH5tfqW6uv7E2x/bnTIDreX9LjwbWYuE5H315VJ3x3S9rzVY6U2UC5v6S01RR1l0AOTnDiJe6QtcLhXF+uK36tXcBXF/1ZFYUhNgAqEx4xyL+g13iAI7bZPsFgTVOvpiA4ehq0QT81V/91fXfecqkKXa/93u/9/I84Ls5w4yGNbWjOwr+G9e68s5JaxyH54yj1zpL6Mwrr7zyMtaOebxEZ14cQVUmYM5VrkQ5JahxYLwdvoznPC/VTu5xgcpov4yCy6+x9EuD8R/iHEZxW3vc1ldX9GtWz549W3z4a5x8a0yAy2qpdwUkOjq2SuKjxwlThaoGWI5FScjH2i/VzfU31ubYfl/pi2ku+e833K87AyF7PbqxyP/xH/9xWPT6MU9c1rG7UXDixlosJnbfXLGl/+bnnEPRmfrDo8oEfPnll/U9/HuNkqj7j3xPy9zzmrku9mzAjVTUCUF/xPosbQbWOPla2FNXmjFSl7XY+GU238aLb1m7vPZLmCRUU0Qv7je1nWNREvJUO2FeqW6uv7D+OetTePi84Pb1voCYKg5aD5MNl396VvttAOKycbzCdkrra7Eo9dnyvqX/5tGZ4Z/bbaEzVSbgXIe29lW5TxJbLXWnIXcnwCeflv9o1xxbeEJY2gyscfK1UPtqPnwpML5d7LISoPCFQa3r2JMohclmISyr/Wpn7nGSY1ES8nBsqfVS3Vx/qXZq8s4xAX5EIBMmluFHeYqFY+c7BorrUmktFkuNb+t2lv6bR2demICtdKbKBJx7lXvus/lQ1MceAYRlU+v6ut/YS4mup/cNJPK5j2455dK5B7AFs+fl3DsDa5x8LeyOfbjUVWWYXDYlKBZ8l/dXD6fcYXCdmmWORUnIx9ov1c31N9bm2P6pJkDGy0n8wzil1v3VTpdV7JZKS7Ho+W/dczvnbx6deaE9a+hM6m+gygScc/Uy9WXA1B9ymOe39nO/DRCWTa2PvQQS1lFZGYbUpxQYAdZBf07yH03PS33/N/5d7hpWS518wz4t7FPE2mVTJsBi5rZLZV1mzjLHoiTkY/2V6ub6G2tzbL+55UTaV/2hIbOwp+4EqB19fOvfZXPtj40vtX8pFj3/rXtu5/zNX1pnvvnmm+vfiAk1Yep66zqTOp6rTIACW5uWehdAcHVlPjUYcTmZkbFvFMR1Uttqp5SW/rWrUl8t7kvdGjznRJCa21In37BtiYPijAk4Xf9jLRuBkJHW12CvdsdMgJ/xh8/va27x646A4psybvEcp26vxWJq/62VW/pvfs868xd/8Re705kqE1D7S05zRDsWYInvnLsKGsucdxP0OCL3kkb4R3nOARzW3/t6eEJYSvzNZI2T71omwO8JTDEXnl/NMsfCIq5lbSrVzfVX20dcvmQC9DKlzgPxuxZ+1KJHBL7iD9v1owDluaze+YjLhsYirD+2vhaLsX5b3b/03/yedeYXv/jF7nSmygTUXuVOfQYfC35qWyJ+bnv++d9z7kpI/PV7AWOPAfwHqj+IIyfNf2nxN881Tr5rmQCN2V9D48VAR/D20ibA37TQFbvWzU4GIBZvteLb/DIC/uaFlqoXvj+gsjYT2qd6+sTvcNweWT5njeMw31v7e5b+m9+zzuhRwt50psoE1F7l6soiJehhnkVWt/t1pa6r7RiiXixU0u187R/7xP2qPSXdSRir6/3qS78WOFX8rzs4na4F0OtHXD5//ny1aa9x8rUJCAXCQqGl9ju5rPLjZDEL8/08OxQq394O2w3rTF3PsShdzY+1Xaqb62+szbH9fps/PCeIlziNXamLr7mrvmKo9lLPlBUz7Xc/qjfWfm7sa7HI9dd6/tJ/83vWGZmAvelMlQmodWgSeP/RxUudcHIiG5sAXcnLJIQf3RVQ/VSKXx7M1dfdhaVT7QG8dP89t7fGyde3i+Pj09uh4NsEhLebzVuiEl+Bap+EyledalPrajN1deu2pixzLEpCPtZuqW6uv7E2e9wPi3Wjis6M811SZ6pMQM2zGgm0T6TxUmJeusKORTyuH27LdYVJ7Yb7x9ZzRiJss2adnw2uoVVXlpPvwCvHoiTkQ+30Wqlurr90S33nwmLd+KIz43yX1JkqE1DjPnJv4usqv2QANP3SHYRY1H2r39hqTYAeQSyZahgt2e8R2uLkO0Q5x6Ik5EPt9Fqpbq6/dEt958Ji3fjWnEPRmfmxqDIBqWdtuSHo2Xos2NpW0MZS/Dgg1Y7zUiJe8+NEU8YzNt5wf80BHNZjfZwAJ9+BUY5FSciH2um1Ut1cf+mW+s6FxbrxXUJn/uZv/mZ0kOjMC0RVJqDml5xSJiC8atcVu8pIsHXlr33adpK466380ics73pe6pFDqa72Lf0oQH0veZvGc2H5ggAn3+FIyLEoCflQO71WqpvrL91S37mwWDe+S+iMXtBTQmfGY1VlAmquclPf6ZcwO+W+7heWcdk9LWsY7WleLYyVk+8QhRyLkpAPtdNrpbq5/tIt9Z0Li3XjW3MOzemMTQA6Mx6rKhNQc5Wbejbvl/i0z8/9FSTlh7dmXG58+O2VqHGx7Y2+7RFx8h3ik2NREvKhdnqtVDfXX7qlvnNhsW580ZlxvkvqTJUJqHFomoaFPn5+b4MQPx5wuTB/HEdbJWqeZ7U18vZHw8l3iFGORUnIh9rptVLdXH/plvrOhcW68UVnxvkuqTOrmoDw6l4Cr6t+Jz2PlxkIk78a6B8HCvftZb32AN7LvFoYJyffIQo5FiUhH2qn10p1c/2lW+o7Fxbrxrf2HIrOzItHlQmouU2jYcX/O0B3BnI/0BN+1UMv7e011XzHda9zvNS4OfkO5HMsSkI+1E6vlerm+ku31HcuLNaN7xI6o9/wTyV05jaVKhNQ69DUnQTdt/m11FV+bAR0VyD8Wl/prf/bU2grp/bXrtoafduj4eQ7xCfHoiTkQ+30Wqlurr90S33nwmLd+KIz43yX1JkqE/DJJ5+Mjy4qEb4EGJoB3cKRQfBXBL1PjwTixwRRk01vnnMANz2hhgbHyXcIRo5FSciH2um1Ut1cf+mW+s6FxbrxRWfG+S6pM1Um4Nz/kCdRD6/0LfjxUo8Llv7xnnGcy5ZY0qEtO7L9t8bJd4hhjkVJyIfa6bVS3Vx/6Zb6zoXFuvFFZ8b5LqkzVSZgrvvQbwDEwu9tnYDW+PGecZzLluCdgGV5hq1x8h1o5FiUhHyonV4r1c31l26p71xYrBtfdGac75I6U2UClnIfeuavbwrIFOiXAVPfFBjH0GaJuQdwm7NqY1ScfIc45FiUhHyonV4r1c31l26p71xYrBtfdGac75I6U2UCHj16ND66g5dY8vubB0d5a/qcfAckORYlIR9qp9dKdXP9pVvqOxcW68YXnRnnu6TOVJmAJd3H+DT3WWLJX3LaJ4H1Rs3Jd2CbY1ES8qF2eq1UN9dfuqW+c2GxbnzRmXG+S+pMlQlY0n2MT3OfJTiA14sbJ9+BbY5FSciH2um1Ut1cf+mW+s6FxbrxRWfG+S6pM1UmQC8jPHv2bHyEBy3x/Pnz01dffXXQ2a8/bU6+A+Mci5KQD7XTa6W6uf7SLfWdC4t144vOlPkurTPVJuDLL78sj/DAe2UAzvmO64GRVU2dk++AK8eiJORD7fRaqW6uv3RLfefCYt34ygSgM3nGS+tMlQnQXQCudPPB0c9dLvVma76X4+7h5DvEPseiJORD7fRaqW6uv3RLfefCYt34ojNlvkvrTJUJ0G2IJV9IKE91f3ufPHmyv0HvaMScfIdg5ViUhHyonV4r1c31l26p71xYrBtfdKbMd2mdqTIBGtrnn3/O3YBEjOTOlvwBh0QXh8/i5DscAjkWJSEfaqfXSnVz/aVb6jsXFuvHF51JM15DZ6pNgIamAMmtkV4Q8IsaPMda94jg5DvwzbEoCflQO71WqpvrL91S37mw2Ca+6MxNzmvpzFkmQLcjuOp9ESAFRj9usfQtmpvhZ0sEOPkOx0GORUnIh9rptVLdXH/plvrOhcU28UVnBs5r6sxZJkBD0wuCR3dqdmZH5zAcquuucfId+OZYlIR8qJ1eK9XN9Zduqe9cWGwXX3TmdH3XfU0OZ5sAHQb6b0/6StwRfztAQfnoo4+4A7Dd+YA7AQHrnBCVhDyonlwt1c31l2yo80xYbBtgdGZdnZllAnQo6GpYX4uTGdAzcX16NAWap5IOSP+YhYwAaTsCnHwH1jkWJSEfaqfXSnVz/aVb6jsXFtvHF51Zj/lsE+ChWfwlkLpC1luM+mlDXy1rXc/Odetc63reI+OgddXVUp+xenaFYdlwXV9h1M9OKk/9yaBo3eIdlg3Xw3qag+tpqW3v17gRf0d92yUn34F3jsXrr79+sph//PHHQ4WRNZV1PbURp1x/cbkjbMPiclFGZ5Znv5gJWH5otAiBmwQ4+Z5OX3/99bWBzrF46623Xoq5Rb12qTbiFPYnI6xxHDWFLI7KgHn3QwAT0E8su5/JEU++Fv3PPvvs9ODBgxvvRZhHGPgvvvjidO/evbONgOqqjTi5r3Cp8WhcRzMFIYOYE9sQ2BsBTMDeInbg8R7h5DtF9EMO77333q0jQiKuq/nw0cDY3QCVVZ2UAVAH6ifsN7V+FFMQzv0WfDIgsDMCmICdBezIw+3x5Fsr+mIQiu2Wx4Ou+Et3JML4xOPs6fFBOM8t+dMXBNYggAlYgyptrkKgh5PvXNFvRUx7mcc5B2oPx+E586ZOnwQwAX3GtctZ7fHkexSxPMo89Ye1x+OwyxMCk1qEACZgEYw0sgWBPZx8jySGpZj3zGEPx2EpNuyDQEgAExDSYL1pAi2efHsWuyUPhp44tXgcLhkr2joWAUzAseK969m2cPLtScwueTDsmWMLx+ElY0fffRHABPQVz65nc4mT757Fak8Hw544X+I43FMsGeu+CGAC9hWvQ492i5PvnsSo54Oh5ThscRz2HFvm1hYBTEBb8WA0BQJrnHxbFpsCirN26X9l6LPH1FKc1jgO9xgTxtwHAUxAH3E8xCyWOPm2JCZbB23PJiBmdck4LnEcxvNhGwKXIoAJuBR5+q0mcM7J95JiUT3BlSv0ZAJiVFvG+ZzjMB4v2xBohQAmoJVIMI5RAlNOvluKweiAGyvQswmIUa95HEw5DuPxsA2BVglgAlqNDJ5U3/QAAB/eSURBVOO6RSB18l3zZH9rADvPOJIJiEO15HGSOg7j/tiGwF4IYAL2EinGeePnWvlHNvUHxJFNQExrjinABMQ02d4zAUzAnqN3sLGHJ9/Sevhf9nSyJ70ggAnIHwnnmAIdgyQI7J0AJmDvETzQ+HPCj+hPOwgwAdM4qdRUUzC9RUpCoE0CmIA248KoEgQ+/fTT09XV1QnRT8CZkIUJmAApUyQ2BToOdTySILB3ApiAvUeQ8UNgIgFMwERQFIPAgQhgAg4UbKZ6bAKYgGPHn9lDIEUAE5CiQh4EOiSACegwqEwJAjMJYAJmAqQ6BPZCABOwl0gxTghsRwATsB1reoLARQlgAi6Kn84h0CQBTECTYWFQEFieACZgeaa0CIG9E8AE7D2CjB8CEwlgAiaCohgEDkQAE3CgYDPVYxPABBw7/sweAikCmIAUFfIg0CEBTECHQWVKEJhJABMwEyDVIbAXApiAvUSKcUJgOwKYgO1Y0xMELkoAE3BR/HQOgSYJYAKaDAuDgkCZwNOnT0937tw53b9/v1ww2IsJCGCwCgEIXBPABHAgNEvAQiexiz/37t1rdtxbDMxsMAFb0KYPCPRLABPQb2x3P7PHjx9fi7/+YxvpJgFMwE0ebEEAAucRwAScx41aGxBY0gTozsHdu3c3GPXtLtTv0ncuMAG3OZMDAQjUE8AE1DOjxkYEMAF50JiAPBv2QAAC0wlgAqazouTGBKaYAD0q0PsCDx8+vDE65enqW/nx+wTa9iMG75eo6ord9dyY89yG63m/lhZkl9FSeR5bmK/1cKzu32Vydys0F5fR0n3yTkAYCdYhAIFaApiAWmKU34zAFBOgwVioPTAJo4XSebnHAaEIh+JskdXSyaIelkuNUfvD2/8aX7gdt6c2nCz23tbSeR6L+9QcMQEhKdYhAIFaApiAWmKU34xAKHbhVXAs8C4nQfR6fMUuIU1dZdsETBXTWHhz7YaQUibAJiMeZ5zv+YTGQ23Xjlt1+IpgGBXWIQABEcAEcBw0S8ACGAtlasC++pdIS5jjlBNri6n6mpLi9mNTkGojZQJK/YZt+u6D7wK4fZuFqeZF9TABpscSAhAwAUyASbBsjkCNCdDgJZ76xFfN2neOCbDQul0vbTK8f8yklEyA24yXFndMQHOHJQOCQFcEMAFdhbOvydSYAN8JkOBKUONUawLct8XY7ant2ATEZVzWy5IJGLsDgQkwRZYQgMAaBG6fLdfohTYhcAYBC/HYlbZvrWvpOrEw15qAnPiGJkBTksDrU0opEzD1LkI4t7AP58fzDMvE6zwOiImwDQEIYAI4BpolYEEfMwGxMEsYlRdeZTtP4hsmi2lYVvudr6WTjETcl8uFY1Rbvluguq7ndrz0mOK+ZRrCPG2rXyf3qTxMgKmwhAAEziEwnFnOqU0dCKxIwFfLErv446tvC2ks7irvMh6ixVT7LO4W1FB0Xd53A9y3tiXoocCrrNtwOS3jFO4L+4r7ULlwv9sJx+55aYkJMCGWEIDAOQRun63OaYU6EIBA8wR4HNB8iBggBDYngAnYHDkdQuAyBDABl+FOrxBomQAmoOXoMDYILEgAE7AgTJqCQCcEMAGdBJJpQGCMACZgjBD7IXA8ApiA48WcGR+UACbgoIFn2hAoEMAEFOCwCwI9EcAE9BRN5gKBZQhgApbhSCsQaJ4AJqD5EDFACGxOABOwOXI6hMBlCGACLsOdXiHQMgFMQMvRYWwQWJAAJmBBmDQFgU4IYAI6CSTTgMAYAUzAGCH2Q+B4BDABx4s5Mz4oAUzAQQPPtCFQIIAJKMBhV1sE/I94wt/h17p/S7+t0bY3GkxAezFhRBC4NAFMwKUjQP+TCUjsEfzJuG4VxATcQkIGBA5PABNw+ENgPwC2NgHqL/6PgfuhdXukmIDbTMiBwNEJYAKOfgTsaP6YgHnBwgTM40dtCPRIABPQY1Q7ndNUExC/O3B1dXWLyMOHD0/xuwUupPLxPm2rjtL9+/ev9z99+tRVbuQ7U/tdz3W07eT9YV9xmyob7p9zZwITYPIsIQABExjOSM5hCYFGCUwxAXEZC61E2Enr4bbyJbSqGyZtp0TXgh4LtvPdhvtW23F/jx8/vu4zNCg2H9rnFNdVOzYjLjN1iQmYSopyEDgOAUzAcWK9+5lKlMOrYq9bYC2isTjn8kMgqTJLmYDYXKjfXNthvk3EuaIfzk/rmICYCNsQgAAmgGNgNwQkkClB9QR01Z7a71v/4RW263hpExCWCQXZ5bT0FX9sNpzvshZxtR2mXL7KxHPQGGR2ljACmIAwCqxDAAIigAngONgNAQliSuQ9AQmo7w6klqHAW7DjcmGZtU1A3Le34zlq2/tSjyc8/7ElJmCMEPshcDwCmIDjxXy3M5YYxgIZTia+ig73hes2C+GVfM2dAJcN66t9Gwv3lbviz+W7Xm7pftXPOQkTcA416kCgbwKYgL7j29XsxkyARTIW5xiCrqpjIXXdKXcCXDbux1fs7q8k9ip7zlX9ufU0JkyAI8MSAhAwAUyASbBsnsCYCdAEJPAqFyYJe5gXt6P9vt0emgDfMQjb0rrFPTQSatNtuLzLyTTEye8pxPvUpvO0DPsotRe3n9rGBKSokAeBYxPABBw7/ruafSzeucGHgpwyBapnwfbSRiA0AXG5cJ/Lu75EXaKtbacx0Y7bUF0bALdhI+J+QlPgMlOXmICppCgHgeMQGM5Yx5kzM4XAIQlgAg4ZdiYNgSIBTEARDzsh0A8BTEA/sWQmEFiKACZgKZK0A4HGCWACGg8Qw4PABQhgAi4AnS4hcAkCmIBLUKdPCLRNABPQdnwYHQQWI4AJWAwlDUGgGwKYgG5CyUQgUCaACSjzYS8EjkgAE3DEqDPnQxLABBwy7EwaAkUCmIAiHnZCoB8CmIB+YslMILAUAUzAUiRpBwKNE8AENB4ghgeBCxDABFwAOl1C4BIEMAGXoE6fEGibACag7fgwOggsRgATsBhKGoJANwQwAd2EkolAoEwAE1Dmw14IHJEAJuCIUWfOhySACThk2Jk0BIoEMAFFPOyEQD8EMAH9xJKZQGApApiApUjSDgQaI/D8+fPTRx999PJjE+C8Tz755KQyJAhA4LgEMAHHjT0zPwCBn/3sZyeLf7zUPhIEIHBsApiAY8ef2XdOQFf6sfhr+4MPPjg9e/as89kzPQhAYIwAJmCMEPshsHMCqbsB3AXYeVAZPgQWIoAJWAgkzUCgVQLx3QDuArQaKcYFge0JYAK2Z06PENicQHg3gLsAm+OnQwg0SwAT0GxoGBgEliPguwHcBViOKS1BoAcCmIAeosgcIDCBgO4AcBdgAiiKQOBABDABBwo2Uz02Ad0N4BsBxz4GmD0EYgKYgJgI2xCAAAQgAIGDEMAEHCTQTBMCEIAABCAQE8AExETYhgAEIAABCByEACbgIIFmmhCAAAQgAIGYACYgJsI2BCAAAQhA4CAEMAEHCTTThAAEIAABCMQEMAExEbYhAAEIQAACByGACThIoJkmBCAAAQhAICaACYiJsA0BCEAAAhA4CAFMwEECzTQhAAEIQAACMQFMQEyEbQhAAAIQgMBBCGACDhJopgkBCEAAAhCICWACYiJsQwACEIAABA5CABNwkEAzTQhAAAIQgEBMABMQE2EbAhCAAAQgcBACmICDBJppQgACEIAABGICmICYCNsQ2DGBJ0+enN54443Td77zndOrr746+lE5lVe9Uvr0009PV1dXpwcPHpw+++yz6/Jff/11qQr7IACBHRDABOwgSAwRAlMISMinCH+uTMkIvP3226fUB1MwJTKUgUC7BDAB7caGkUGgisCbb745ywTojkAupQxAKg9TkCNIPgTaJIAJaDMujAoC1QRee+21lybg448/nlRf5XxnQI8GcikUfD0OkNiHebl1TEGOKPkQaIMAJqCNODAKCMwm8Fu/9VsvBb2mMZsALXMpFHmX0TsBeoSAKTARlhDYHwFMwP5ixoghkCSwtQmIB4EpiImwDYH2CWAC2o8RI4TAJAKXNgHxIDEFMRG2IdAeAUxAezFhRBA4i0BrJiCeBKYgJsI2BC5PABNw+RgwAggsQqB1ExBPElMQE2EbAtsTwARsz5weIbAKgb2ZgBgCpiAmwjYE1ieACVifMT1AYBMCezcBMSRMQUyEbQgsTwATsDxTWoTARQj0ZgJiiJiCmAjbEJhPABMwnyEtQKAJAr2bgBgypiAmwjYE6glgAuqZUQMCTRI4mgmIg4ApiImwDYFxApiAcUaUgMAuCBzdBMRBOtcU/PSnP42bYhsC3RLABHQbWiZ2NAKYgHLEp5qCd999t9wQeyHQEQFMQEfBZCrHJoAJqIt/yRTUtURpCOyXACZgv7Fj5BC4QQATcANH9UbqnyRVN0IFCOyMACZgZwFjuBDIEcAE5MhMy+/VBDx9+vT0yiuvnO7fvz8NBKUORQATcKhwM9meCWAC5kV3LRPw+PHj0507d259thLllAm4d+/e9Xi0b27y/Laaz9zxUv8mAUzATR5sQWC3BDAB80K3lgl4+PDhteBq6WThlBivnTABaxPed/uYgH3Hj9FD4CUBTMBLFGetbGkCNMCrq6trcyBDsGZKmYBz+vN4l7h7cE7/1FmHACZgHa60CoHNCWAC5iHf2gT4bkB4h2DeDNK1MQFpLuS+IIAJ4EiAQCcEMAHzArm1CYgfE3hbon337t3ruwTh4wKbhvD9gtSM/bzf5VImQM/vtT9OcV1d/au+2wqXHpv3q2yY3G9YJ35vwHPS0mNy+W+++SZs7tY44rZuFGZjMoHbR8HkqhSEAARaIoAJmBeNrU2ARU9iqWQTIBGM7w5oW2/4h/lxfbVhEXebFlm1GYqm64bEZDz0CZPq/fznP7/Oyj0OSJkA9/v973//ZXMuZ/OgHS6nfkITEY/FdcMyakf1SfMIYALm8aM2BJohgAmYF4otTYAFPxQ154Vi7RnFIp7Kt6CGRkHltB1/RTA2Ae67JKo1JkAiLpGOr+bjfjzmkEM4Zo/H5bzt+bOcTwATMJ8hLUCgCQKYgHlhWNsESMjDTyxosUB6NhbA999/31kvlxZbZZREeswExKbgZQfBSqn98Eo+ddXuZuJ9npvmHiZta8whI7ML88I6rJ9HABNwHjdqQaA5ApiAeSFZ2wTEQhePdswEWATjpW+vl0R6zASojfhRQDy+UvuhCcgJu9qrNQExs3DuqTsm8ZjZHieACRhnRAkI7IIAJmBemFo3Aak7AeGMSyI9xQRIYEup1H7KBKh8nOaaALfnOxepPlyG5TQC5ahPa4NSEIBAAwQwAfOC0KoJ0KwksmNXvr6TEF89a3vMBFjgS7faXUZCHqZY2D3e1DsBbsP95O4aeMzxXMJ+pzAJy7OeJoAJSHMhFwK7I4AJmBeylk2AxTMWRQltmKdb+uEVvfZpOxZMX0mHxFQmfiSgbX87wG2F/al+ygR4vOG3Ayz4oZlxXtymtmVcnK86atMpV8/7WU4ngAmYzoqSEGiaACZgXnhaNgGamQTRgu6lRTKcuY1AKOq/8Ru/ceNOQsoEqA2362Xcvutpv8U8ZQLUloRaQu62tIzby4m5yoUmQO2F81JboSkI5896HQFMQB0vSkOgWQKYgHmhWcsEzBsVtSGwLgFMwLp8aR0CmxHABMxDjQmYx4/a+ySACdhn3Bg1BG4RwATcQlKVgQmowtVN4Y8++uj6vYZuJlQ5EUxAJTCKQ6BVApiAeZHBBMzjt9fa+uqlPkc1A5iAvR65jBsCEQFMQASkchMTUAmsk+I2AV4ezQxgAjo5kJkGBDAB846BGhNgwWD54iq6Rw5HMQOYgHnnDWpDoBkCmIB5ocAE9Cvo55iUDz744PTkyZPT8+fP5x1YjdfGBDQeIIYHgakEMAFTSaXL1ZiAdAvk7pFAbBCOIv6OFSbAJFhCYOcEMAHzAogJmMdvr7VtAo4m/o4XJsAkWEJg5wQwAfMCiAmYx2+vtY8q/o4XJsAkWEJg5wQwAfMCiAmYx2+vtXt/5j8WF0zAGCH2Q2AnBDAB8wKFCZjHj9r7JIAJ2GfcGDUEbhHABNxCUpWBCajCReFOCGACOgkk04AAJmDeMYAJmMeP2vskgAnYZ9wYNQRuEcAE3EJSlYEJqMJF4U4IYAI6CSTTgAAmYN4xgAmYx4/a+ySACdhn3Bg1BG4RwATcQlKVgQmowkXhTghgAjoJJNOAACZg3jGACZjHj9r7JIAJ2GfcGDUEbhHABNxCUpWBCajCReFOCGACOgkk04AAJmDeMYAJmMeP2vskgAnYZ9wYNQRuEcAE3EJSlYEJqMJF4U4IYAI6CSTTgAAmYN4xgAmYx4/a+ySACdhn3Bg1BG4RwATcQlKVgQmowkXhTghgAjoJJNOAACZg3jGACZjHj9r7JIAJ2GfcGDUEbhHABNxCUpWBCajCReFOCGACOgkk04AAJmDeMYAJmMeP2vskgAnYZ9wYNQRuEcAE3EJSlYEJqMJF4U4IYAI6CSTTgAAmYN4xgAmYx4/a+ySACdhn3Bg1BG4RwATcQlKV8c4775xsBKoqUhgCOyaACdhx8Bg6BEICmICQxvj6119/fXry5Mnps88+Oz148OClAZARIEHgKAQwAUeJNPPsngAmoBzikuj7DoCW7777brkh9kKgIwKYgI6CyVSOTQATcDP+U0U/NAC6I/DTn/70ZkNsQaBjApiAjoPL1I5F4Ogm4FzR1+MAPRZQfRIEjkYAE3C0iDPfbgkczQQg+t0eykxsQwKYgA1h0xUE1iTQuwlA9Nc8emj7qAQwAUeNPPPujkBvJgDR7+4QZUINEsAENBgUhgSBcwjs3QQg+udEnToQmEcAEzCPH7Uh0AyBvZkARL+ZQ4eBHJgAJuDAwWfqfRFo3QQg+n0db8ymDwKYgD7iyCwgcGrNBCD6HJQQaJ8AJqD9GDFCCEwicGkTgOhPChOFINAUAUxAU+FgMBA4n8DWJgDRPz9W1IRAKwQwAa1EgnFAYCaB11577fTqq69ef/7yL/9yUmsff/zxyzrf+c53snXCn9ZN/cOdcH+4rp/h5Rf5sljZAYGLE8AEXDwEDAACyxB48803Xwq6zUDN8o033sgOJBT20jqin0XIDgg0SQAT0GRYGBQE6gno9+9rRD8uq/q5lBN+RD9HjHwI7IMAJmAfcWKUEJhEQEKuK3rd2o9FPrWtcipfMgDq+NNPPz1dXV2dEP1JYaAQBHZDABOwm1AxUAhAAAIQgMCyBDABy/KkNQhAYGECT//n/164RZqDAARMABNgEiwhAIEmCXz4d/9y+qP3/r7JsTEoCOydACZg7xFk/BDonMB/fvhPp9/5kx+efvDhf+98pkwPAtsTwARsz5weIQCBCgLf+8HfXpsAGYEf/uMXFTUpCgEIjBHABIwRYj8EIHAxAs/+7/97aQBkAjACFwsFHXdKABPQaWCZFgR6IPBPv/jqlgmQEVA+CQIQmE8AEzCfIS1AAAIrEdBLgb4DEC7/43/50ennfGtgJeo0eyQCmIAjRZu5QmBnBPxSYGgAtP6f/uvnp//2o6c7mw3DhUB7BDAB7cWEEUEAAv9OwC8F6spf6zIF/G4AhwcEliOACViOJS1BAAILEtBLgbral+hr/e/+x/+6NgILdkFTEDg8AUzA4Q8BAEBgPwR0N+DLX/2f/QyYkUKgcQKYgMYDxPAgAIGBwINP/pkfDRpwsAaB2QQwAbMR0gAEILAVAT0W0PsBWpIgAIH5BDAB8xnSAgQgsCEB/XywvjpIggAE5hPABMxnSAsQgMCGBPSCoO4GfLNhn3QFgV4JYAJ6jSzzgkDHBPSCIF8V7DjATG0zApiAzVDTEQQgsBQBfXWQ/yq4FE3aOTIBTMCRo8/cIbBTAnox8D/8yQ/5uuBO48ew2yGACWgnFowEAhCoIKC7AfoFQRIEIHA+AUzA+eyoCQEIXJCA7gbo3QC9KEiCAATOI4AJOI8btSAAgQYI/PAfv9jlTwnfvXv3dO/evQYInk6PHz8+3blz5+WniUExiM0IYAI2Q01HEIDAGgT0HwVlBlpKEvhQWLX+9OnwXw8vYQLu379/PSaJfpg0tqurqzDrupzGuFVS/xrHw4cPt+qSfv6dACaAQwECENg1gX/6xVfXvxvQwiQk9BKz+Crf+Y8ePboeZismQKKr8aaMASaghSNq/TFgAtZnTA8QgMDKBPSCoF4UvHSaKu5Ty609n5wJWKtf341Yq33arSeACahnRg0IQKAxAvrPgr/zJz+86A8I1QhqKybAt+HjOwFrhRcTsBbZ89vFBJzPjpoQgEBDBPxzwpf650J+D2AKkpQJiF/Q02368D0CtevHCtqnj0Q1TDYi3q9tJwu+23SZcOl3A1LjUzthWa2H5iEem/b78Udqbtrv8XvcYXvqL1XPY4znpW2N22OMH8m4PMubBDABN3mwBQEI7JiAfkXwj977+4vMQAKkz5QUi6yELRYtC5rbs8iGIqg6Fs6UkEoQnVRP2zYBynee23DZeHzu26Ktcqrj+apvr7sNm6Kwv9ydgNTYnaelk01BOA7PQXPzPFLjdRssbxIYjpCb+WxBAAIQ2CUBfVvgEu8HxMJZgjelrEXQwmYB9HbcvsUwzve294ei7Ly4zXh8OfF226ml2nzllVduvPGfayeeq9pT3VDs3YfH7Hl4O55Dri+3w/IFAUwARwIEINAVAT0O0H8Z3PpHhCSc+kxJscim6lgYwyth3+qOBU/1lef9qfYslhZPlXFe3F48vng71X6cpzYl5OrDKSfMnqvH4brh3N2G9mme3uc5hPNS2VxfboflCwKYAI4ECECgOwL+2uCW/2nQt7+nwEyJqsXMQu6lxc7tOl/L+ErZAukyYV23H4ql8yy+7iMeX6ovl/VSfbnfcKk+nHLC7Loeh7ZlILzt+lp6jp6b5xDOS+XcV5wftsX66YQJ4CiAAAS6JPDh3/3L9R2BrYyAxSglXDHgWGQtWGFdC6PFLm7DdUKRDcuoD4mx2/T4QlF0nsu4fjw+tRO/s+CyWrqdcKxq89w7AWpHdcP23J/a1Xi8z32H81JZ84nz3Q7LFwQwARwJEIBAtwR8R0DLtZPEZkwsPYZYZONtlZPIhWLnuuFS+yV2qTRFLC2gYyZg7C6H9msOYZpjAsRSJiA1t1jcPYdY7ONy4dhYHwhgAgYWrEEAAh0SkAHQbwhsYQQs3PFVsw2CvzIXi34ssi4fmoA/+IM/uPF8PRZ5tRmKeSyO8bZC7bywnvLj8bmvUJQ1Rgt/SnA1dn3Uh1OuP3MLx6H5hvNXGy6XahMTYMp1S0xAHS9KQwACOySgRwIyAlv9jwELYLgMRSoWWSFVXljewivhc4rLhGIYGge3E4qqBTgch/PCch5LbGQ8HretZdiWjYz3q3z8OEBth+U8fot7PA7nu00t4zKeQzgW9ZMyJubIciCACRhYsAYBCHRMQEZA3xp48Mk/dzxLpgaBOgKYgDpelIYABHZMQF8f1P8Z0G8JbPXC4I5xMfQDEMAEHCDITBECELhJQN8c0OMB/cKg/u8ACQJHJYAJOGrkmTcEIHCyGdDdga3eFwA7BFoigAloKRqMBQIQuAgBmYHv/eBvr+8O+N8S69sE+uhOgR4d6BcIMQoXCQ+drkgAE7AiXJqGAAT2RUCiL0Oglwf1j4hkDPTRC4V6j0CPD0gQ6IkAJqCnaDIXCEAAAhCAQAUBTEAFLIpCAAIQgAAEeiKACegpmswFAhCAAAQgUEEAE1ABi6IQgAAEIACBnghgAnqKJnOBAAQg0BCB3/7t377+KeSGhsRQIgKYgAgImxCAAAT2RsD/NyD8Bz/hHPzPeOLf1w/LrLGOCViD6rJtYgKW5UlrEIAABDYngAnYHHk3HWICugklE4EABI5KABNw1MjPnzcmYD5DWoAABCBwUQKYgIvi33XnmIBdh4/BQwACEDidzjUBV1dX1y/u3blz53p57969JE7v9/Lhw4e3yinP+7XUttrTOqldAkSn3dgwMghAAAKTCJxjAvQSoQRadZ3u3r170sfp8ePHp1deecWb10vX0z4nGwCZCie1Y1PgPJbtEcAEtBcTRgQBCECgioBNgEU3t7TgS8BVJr6iz+WHg3FfseCn7iLIQKgfUrsEiE67sWFkEIAABCYRsDBP/YqgHwPYFLgTtxMKvPd56TLuy9upOnxF0NTaXWIC2o0NI4MABCAwiYCF2MIcV4p/J8AmIHfHIBR03x2Iy7ov9x3Wcf+YAJNod4kJaDc2jAwCEIDAJAIWYgtzXClnAlSvlPS4QLf0Q4GP+/J2WMZtYgJMot0lJqDd2DAyCEAAApMIWIinmgBf3cfvBMSdqb34mX6qL5XhnYCY3j62MQH7iBOjhAAEIJAlkBLmsHB8J0D7/PU91Q2TBN15urrXtkyDk9/6Dw2Hy4WmQvX8cV2W7RHABLQXE0YEAQhAoIrAOSZAHfhK32KtpQ2AB2AD4TISehmB0ASorI2Ay8k4OM9tsWyPACagvZgwIghAAAIQgMAmBDABm2CmEwhAAAIQgEB7BDAB7cWEEUEAAhCAAAQ2IYAJ2AQznUAAAhCAAATaI4AJaC8mjAgCEIAABCCwCQFMwCaY6QQCEIAABCDQHgFMQHsxYUQQgAAEIACBTQhgAjbBTCcQgAAEIACB9ghgAtqLCSOCAAQgAAEIbEIAE7AJZjqBAAQgAAEItEcAE9BeTBgRBCAAAQhAYBMCmIBNMNMJBCAAAQhAoD0CmID2YsKIIAABCEAAApsQwARsgplOIAABCEAAAu0RwAS0FxNGBAEIQAACENiEACZgE8x0AgEIQAACEGiPACagvZgwIghAAAIQgMAmBDABm2CmEwhAAAIQgEB7BDAB7cWEEUEAAhCAAAQ2IfD/AY2vOPKSHbDjAAAAAElFTkSuQmCC)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZHWDeiEvfG9M"
      },
      "source": [
        "The diagram above is an example of a model with two \"heads\", one a decoder that regenerates the original image as part of an autoencoder used in the unsupervised learning stage, the other a classification head that learns to classify images based on supervised (labelled) data. We will be using this model architecture as our base model in this exercise. \n",
        "\n",
        "The following section invites you to complete the code for such a model using TensorFlow. There is no need for you to construct a deep, complex model; just 2-3 layers in each of the encoder, decoder and classification modules are sufficient for the purpose of this exercise. You may wish to use the following neural network layers to construct your model:\n",
        "\n",
        "*   [keras.layers.Conv2D](https://keras.io/api/layers/convolution_layers/convolution2d/)\n",
        "*   [keras.layers.Dense](https://keras.io/api/layers/core_layers/dense/)\n",
        "*   [keras.layers.Conv2DTranspose](https://keras.io/api/layers/convolution_layers/convolution2d_transpose/)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Zc3K1-BHVYs"
      },
      "source": [
        "### Define the model architecture ###\n",
        "class SimpleModel(tf.keras.Model):\n",
        "  def __init__(self):\n",
        "    super(SimpleModel, self).__init__()\n",
        "\n",
        "    # The init function runs once when the model is created.\n",
        "    # This is where you should initialise the weights/layers of your network. \n",
        "    # Some layers have been created for you. You may add more layers if you wish or stick to what is already provided.\n",
        "\n",
        "    ### Encoder layers ###\n",
        "    self.enConv1 = L.Conv2D(16,(3,3), padding='same', activation='relu')\n",
        "    self.enConv2 = L.Conv2D(32,(3,3), padding='same', activation='relu')\n",
        "    self.enConv3 = L.Conv2D(64,(3,3), padding='same', activation='relu')\n",
        "\n",
        "    ### Classification head layers ###\n",
        "    self.dense1 = L.Dense(32, activation='relu')\n",
        "    self.dense2 = L.Dense(10, activation=None)\n",
        "\n",
        "    ### Decoder head layers ###\n",
        "    # Creating an output the same shape as the input using deconvolution layers may be tricky.\n",
        "    # Using padding='same' makes it easier.\n",
        "    self.deConv1 = L.Conv2DTranspose(32, (3,3), padding='same', strides=(2,2), activation='relu')\n",
        "    self.deConv2 = L.Conv2DTranspose(1, (3,3), padding='same', strides=(2,2), activation='sigmoid')\n",
        "    \n",
        "\n",
        "  def call(self, x):\n",
        "    # The call function runs each time you invoke the model during training or inference.\n",
        "    # This is where you should define the flow of information through your model.\n",
        "\n",
        "    # TODO: Build your model data flow graph here by invoking the layers you created above.\n",
        "\n",
        "    # Encoder - the first two lines have been filled in for you below. Complete the rest of the encoder.\n",
        "    x = self.enConv1(x)\n",
        "    x = L.MaxPooling2D((2, 2), strides=(2, 2))(x)\n",
        "\n",
        "\n",
        "    # Classification head\n",
        "    # Hint: Flatten the output of your encoder then pass them into the dense classification layers.\n",
        "    \n",
        "\n",
        "    # Decoder head\n",
        "    # Hint: Pass the output of your encoder into the deconvolution layers here\n",
        "    \n",
        "\n",
        "    # Don't forget to return the output-pointers of your model (classification and decoder heads)\n",
        "    return "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbDYLLq_ToDk"
      },
      "source": [
        "## Testing data flow through the model\n",
        "\n",
        "We should test the construction of our model as good practice before moving on to the next task. Create an instance of the model in the code block below and pass randomly generated data through it. While this does not verify the learning abilities of the model, it does at least allow us to confirm that data is able to flow through the model without errors.\n",
        "\n",
        "Fix any errors that arise."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SO4pyy9LVYYt"
      },
      "source": [
        "# TODO: Create an instance of the model and pass randomly generated data through it.\n",
        "\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uu58NPWrWYqk"
      },
      "source": [
        "## Defining the loss function\n",
        "\n",
        "Since our model has two heads, our loss function would consist of two components - a separate loss for each of the heads. There are two ways in which we could utilise the loss for each of the two heads: \n",
        "\n",
        "1. We could utilise each loss separately across two distinct training stages (e.g. unsupervised learning using the decoder head followed by supervised training using the classification head).\n",
        "2. We could combine the losses and perform unsupervised and supervised learning simultaneously.\n",
        "\n",
        "We will be taking the latter approach in this exercise.\n",
        "\n",
        "For simplicity's sake, we will use a [standard cross entropy loss](https://www.tensorflow.org/api_docs/python/tf/nn/sparse_softmax_cross_entropy_with_logits) for the classification head, and a standard mean-squared-error (L2 loss) for the autoencoder. \n",
        "\n",
        "Write your loss function in the code block below. The function should have two modes (supervised and unsupervised training) controlled by an IF-statement. \n",
        "\n",
        "    if mode=='supervised':\n",
        "      ...\n",
        "    elif mode=='unsupervised':\n",
        "      ...\n",
        "\n",
        "Since the model has access to labels in the supervised mode, the final loss would just be a linear combination of the classification loss and the autoencoder's reconstruction loss. On the other hand, the unsupervised mode would have a loss function with the classification loss being a constant zero."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2BQIv1Ifbsb0"
      },
      "source": [
        "### Define the loss function ###\n",
        "\n",
        "def SemiSupervised_CrossEntropy_or_MSE(labels, inputs, logits, regen_img, mode='supervised'):\n",
        "\n",
        "  # TODO: Write the code for your loss function here. \n",
        "  # labels: the labels for the supervised portion of data.\n",
        "  # inputs: the training images (labels for the unsupervised portion of data).\n",
        "  # logits: raw output of your classification head.\n",
        "  # regen_img: the output of your decoder head (the regenerated image).\n",
        "  # mode: Your loss function should have a supervised mode and an unsupervised mode.\n",
        "\n",
        "  if mode == 'supervised':\n",
        "    # Calculate the loss for a supervised batch here.\n",
        "    classification_loss = #TODO\n",
        "\n",
        "    reconstruction_loss = #TODO\n",
        "\n",
        "    return classification_loss + reconstruction_loss\n",
        "\n",
        "  else:\n",
        "    # Calculate the loss for an unsupervised batch here.\n",
        "    classification_loss = #TODO\n",
        "\n",
        "    reconstruction_loss = #TODO\n",
        "\n",
        "    return classification_loss + reconstruction_loss\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o0ILDM5ScNQF"
      },
      "source": [
        "## Defining the validation function\n",
        "\n",
        "Finally, before we proceed onto the main training script, we should define a validation function. The validation function has been completed for you below to ensure standardisation in testing procedure across all students completing this exercise. This is to facilitate consistent comparison of results and discussion between students.\n",
        "\n",
        "The validation function receives as input a pointer to your trained model and the entire test set, including both images and labels. The test set can be retrieved directly from your prepared dataset object. The function then prints onto screen the validation accuracy and a 10x10 confusion matrix where the rows represent the labels and the columns represent the model predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0ztCwX2eUkO"
      },
      "source": [
        "### Define a validation function ###\n",
        "def ValidateModel(model, test_images, test_labels):\n",
        "  predictions, seg = model.predict(test_images)\n",
        "  predicted_classes = np.argmax(predictions, axis=-1)\n",
        "  accuracy = 1. * np.sum(predicted_classes == test_labels) / test_labels.shape[0]\n",
        "  print('Validation accuracy =', accuracy)\n",
        "  confusion_matrix = np.zeros((10,10))\n",
        "  for i in range(test_labels.shape[0]):\n",
        "    confusion_matrix[test_labels[i],predicted_classes[i]] += 1\n",
        "  print(confusion_matrix)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bRNvpx25ecTr"
      },
      "source": [
        "## Training the model\n",
        "\n",
        "Finally, it's model training time!\n",
        "\n",
        "The code below is a skeleton training script that you should modify to your own needs. Your first task is to validate all the code you have prepared so far by training a model and demonstrating that it is able to learn (reduce it's loss) from both supervised and unsupervised data.\n",
        "\n",
        "Once you have performed this first step, proceed to complete as many of the exercises below which look at evaluating various SSDL training strategies.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "31GEg1kPjrsi"
      },
      "source": [
        "### The main training script ###\n",
        "\n",
        "# Create an instance of your model and define an optimiser\n",
        "model = SimpleModel()\n",
        "optimiser = tf.keras.optimizers.Adam(learning_rate=1e-4)\n",
        "\n",
        "# Main training and validation loop\n",
        "max_train_batches = 3000\n",
        "batch_size = 32\n",
        "train_batch = 0\n",
        "while train_batch < max_train_batches:\n",
        "    train_batch += 1\n",
        "\n",
        "    # Design a decision process to determine if you should use a supervised or unsupervised batch in this iteration\n",
        "    # A random number generator is a good start\n",
        "\n",
        "\n",
        "    if ... : # if supervised\n",
        "      # Randomly sample a supervised batch from your extracted supervised dataset\n",
        "      input_batch = []; label_batch = []\n",
        "      for i in range(batch_size):\n",
        "        x = np.random.randint(0, train_images_supervised.shape[0])\n",
        "        input_batch.append(train_images_supervised[x])\n",
        "        label_batch.append(train_labels_supervised[x])\n",
        "        \n",
        "    else: # if unsupervised\n",
        "      # Randomly sample an unsupervised batch from the full train set\n",
        "      input_batch = []\n",
        "      for i in range(batch_size):\n",
        "        x = np.random.randint(0, dataset['train_images'].shape[0])\n",
        "        input_batch.append(dataset['train_images'][x])\n",
        "\n",
        "    with tf.GradientTape() as tape:\n",
        "      # Infer a batch through the model\n",
        "      input_batch = np.asarray(input_batch)\n",
        "      logits, regen_img = model(input_batch, training=True)\n",
        "      \n",
        "      # Calculate the loss depending on training mode\n",
        "      if ... : # Supervised mode\n",
        "          loss = SemiSupervised_CrossEntropy_or_MSE( ... , mode='supervised')\n",
        "      else: # Unsupervised mode\n",
        "          loss = SemiSupervised_CrossEntropy_or_MSE( ... , mode='unsupervised')\n",
        "\n",
        "      # Perform gradient descent on the loss    \n",
        "      gradients = tape.gradient(loss, model.trainable_variables)\n",
        "      optimiser.apply_gradients(zip(gradients, model.trainable_variables))\n",
        "    \n",
        "    \n",
        "    # Validation at every certain number of batches\n",
        "    if train_batch % 250 == 0:\n",
        "      print('Validation at train batch', train_batch)\n",
        "      ValidateModel(model, dataset['test_images'], dataset['test_labels'])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vV-dFiJ2u_kL"
      },
      "source": [
        "## Simple SSDL experiments\n",
        "\n",
        "Here are some simple experiments that you should be able to try in a short period of time. Don't forget to note down your observations or conclusions for dicussion.\n",
        "\n",
        "1. Many SSDL techniques utilise unsupervised learning to train the model's feature extractors as a first step, then supervised learning to fine tune the model's classifiers. Modify your batch sampling strategy such that you have a high chance of sampling an unsupervised batch at the start of training, then gradually decrease that probability as you progress through training.\n",
        "\n",
        "2. You may choose to give the classification and reconstruction (autoencoder) loss different weightings throughout the training process.\n",
        "\n",
        "3. Remember, if you followed our recommendations, that you are utilising only 1% of all labelled data as your supervised dataset. Try increasing the size of your supervised dataset and observe its effect on training results.\n",
        "\n",
        "4. We have so far assumed that all object classes have an equal number of labelled data points. What happens if only some classes have labelled data? Try training a model by excluding the labels for the last 3 classes. How does the model perform differently?\n",
        "\n",
        "5. Implement data augmentation techniques to leverage on the smoothness assumption of SSDL. You may choose to add random noise to your images, stretch, warp, rotate, flip, adjust the brightness and contrast or even discolour them.\n",
        "\n",
        "6. Self-training: Use your partially trained model to provide predictions on some unlabelled data. Take the most confident predictions and add them as labels to your labelled dataset. Retrain and repeat. Does model performance improve over time?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "llprAX44vf_c"
      },
      "source": [
        "## Advanced SSDL experiments\n",
        "\n",
        "Here are some more complex experiments that you may wish to attempt in your free time. You should check that you have all the theoretical knowledge needed to complete these exercises before ending your workshop session. Feel free to bring them up for discussion if you have any queries.\n",
        "\n",
        "1. Incremental and selective labelling: Infer some unlabelled data through your partially trained model. Provide labels for the least confident predictions. Continue training your model with your expanded dataset. Repeat this process until satisfactory results are achieved.\n",
        "\n",
        "2. Increase the size and diversity of your unsupervised dataset by incorporating images from other datasets. The additional and novel visual features in the new data may be beneficial for model training.\n",
        "\n",
        "3. Implement a co-training approach where two or more models are separately trained on different subsets of labelled data. Infer some unlabelled data through the partially trained models. The most confident predictions of one model are then used as labels for the other models."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cveb8vw97RTK"
      },
      "source": [
        "## The End\n",
        "\n",
        "Congratulations on reaching the end of this workshop exercise! You would by now have gained hands-on experience in applying semi-supervised deep learning techniques to training a neural network. \n",
        "\n",
        "SSDL techniques are wide and varied. Although their use in model training does not guarantee success, the theoretical concepts discussed in this workshop are widely applicable across many areas of deep learning. You would hopefully have a new tool in your arsenal of deep learning know-how to advance your deep learning research."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VbGritxP-nqJ"
      },
      "source": [
        "---\n",
        "Developed by Titus Tang, 2021\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}