DL101_part_1_OXML.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DL101 part 1-OXML.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyNHALQ7VS2yidn+SXoC8YMQ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/i-am-mel-dev/tutorials/blob/master/DL101_part_1_OXML.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zvbu6-RoTti1",
        "colab_type": "text"
      },
      "source": [
        "# Deep Learning 101 Tutorial for OxML 2020\n",
        "### Prepared by Melvin Selim Atay\n",
        "# Tutorial Objectives: \n",
        "\n",
        "*   Explore the main idea of artificial neural networks, deep feed-forward network\n",
        "*   Get intuition of why it is necessary to build deep neural networks such as loss functions \n",
        "*   Learn and compute gradient descent using automatic differentiation. \n",
        "*   Familiarize with PyTorch.\n",
        "\n",
        "\n",
        "   For convenience we will be using MNIST dataset throuhout this tutorial.\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "236YECJSeFFB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#@title Setting Up Necessary Packages { vertical-output: true, display-mode: \"form\" }\n",
        "import os\n",
        "from __future__ import print_function, division\n",
        "\n",
        "import itertools\n",
        "#import time\n",
        "from time import time\n",
        "import numpy as np \n",
        "\n",
        "# PyTorch packages\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch import optim\n",
        "import torchvision\n",
        "from torchvision import datasets, transforms\n",
        "# Plotting packages\n",
        "\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": 125,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TClEvnREeUC8",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Setting torch transform\n",
        "transform = transforms.Compose([transforms.ToTensor(),\n",
        "                              transforms.Normalize((0.5,), (0.5,)),\n",
        "                              ])"
      ],
      "execution_count": 126,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZHIEBiyQesK0",
        "colab_type": "code",
        "cellView": "form",
        "colab": {}
      },
      "source": [
        "#@title Load Data (MNIST)\n",
        "from torch.utils.data.sampler import SubsetRandomSampler\n",
        "\n",
        "n_training_samples = 60000 # \n",
        "\n",
        "n_test_samples = 10000\n",
        "\n",
        "train_sampler = SubsetRandomSampler(np.arange(n_training_samples, dtype=np.int64))\n",
        "test_sampler = SubsetRandomSampler(np.arange(n_test_samples, dtype=np.int64))\n",
        "\n",
        "train_set = datasets.MNIST(r'..\\input\\MNIST', download=True, train=True, transform=transform)\n",
        "train_loader = torch.utils.data.DataLoader(train_set, batch_size=64, sampler=train_sampler)\n",
        "\n",
        "test_set = datasets.MNIST(r'..\\input\\MNIST', download=True, train=False, transform=transform)\n",
        "test_loader = torch.utils.data.DataLoader(test_set, batch_size=64, sampler=test_sampler)"
      ],
      "execution_count": 127,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RUcL3azh3oOY",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title Enabling GPU\n",
        "if torch.cuda.is_available():\n",
        "  # Create tensors\n",
        "  x = torch.ones(1000, 1000)\n",
        "  y = 2 * x + 3\n",
        "  # Do the calculation on cpu (default)\n",
        "  # Matrix multiplication (for benchmark purpose)\n",
        "  results = torch.mm(x, y)\n",
        "  \n",
        "  # Do the same calculation but on the gpu\n",
        "  # First move tensors to gpu\n",
        "  x = x.to(\"cuda\")\n",
        "  y = y.to(\"cuda\")\n",
        "  # Matrix multiplication (for benchmark purpose)\n",
        "  results = torch.mm(x, y)\n",
        "\n",
        "seed = 42\n",
        "np.random.seed(seed)\n",
        "torch.manual_seed(seed)\n",
        "\n",
        "if torch.cuda.is_available():\n",
        "  # Make CuDNN Determinist\n",
        "  torch.backends.cudnn.deterministic = True\n",
        "  torch.cuda.manual_seed(seed)\n",
        "\n",
        "# Define default device, we should use the GPU (cuda) if available\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "execution_count": 128,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KX7dyatJ9b2a",
        "colab_type": "text"
      },
      "source": [
        "### [MNIST Dataset](http://yann.lecun.com/exdb/mnist/) is the dataset of handwritten digits, it has a training set of 60,000 examples and test set of 10,000 examples. \n",
        "![picture](https://upload.wikimedia.org/wikipedia/commons/2/27/MnistExamples.png)\n",
        "##### https://en.wikipedia.org/wiki/MNIST_database#cite_note-10"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FxSVXi6gkN6-",
        "colab_type": "code",
        "cellView": "form",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 280
        },
        "outputId": "fa744807-ef5c-4d43-a551-f50fadb9d2a0"
      },
      "source": [
        "#@title Loading and visualizing data\n",
        "dataiter = iter(train_loader) # creating a iterator\n",
        "images, labels = dataiter.next() # creating images for image and lables for image number (0 to 9) \n",
        "\n",
        "print(images.shape)\n",
        "print(labels.shape)\n",
        "\n",
        "figure = plt.figure()\n",
        "num_of_images = 60\n",
        "for index in range(1, num_of_images + 1):\n",
        "    plt.subplot(6, 10, index)\n",
        "    plt.axis('off')\n",
        "    plt.imshow(images[index].numpy().squeeze(), cmap='gray')"
      ],
      "execution_count": 129,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "torch.Size([64, 1, 28, 28])\n",
            "torch.Size([64])\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAV0AAADlCAYAAADwZiQbAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydeVhUV5r/P7VRVVDFWgsU+w6yCoooIAIKblGzaUzilpiJSSbdk5nJdPr5PZ309DPdM72nO9PZu9OazkSN0SQmMZJEUeOOgoCiAsoi+1psBQVV9/eHQ43EZYxUVdI99Xme+4dVxX2/3uV73/Oec+4RCYKACxcuXLhwDuJvW4ALFy5c/F/CZbouXLhw4URcpuvChQsXTsRlui5cuHDhRFym68KFCxdOxGW6Lly4cOFEpLf6UiQSOX08mSAIIpcOlw6XjjvX8V3S4tJxPa5M1848/fTT7Nixg+zs7G9bigsXLr6DuEzXzkilUmJjY8nJySEgIODbluPirwwvLy+KiorYsmULJpMJo9HIgQMHiIyM/LalOQ2ZTEZcXBwvvPAC58+fZ3h4mMOHD5Obm/ttS7MPgiDcdAOEqWx5eXlCcXGx8N577wkRERG39TeO0HHtFhgYKDz//PPC66+/bncd3t7ewvbt24WBgQHh+PHjQkFBwZT1Ovp4/F/WMXfuXOHtt98Wfv3rXwtarfZbPR4+Pj7C8uXLhXfffVeor68XRkdHBavVKlgsFqG9vV1YtWrVbev4az43qampwnPPPSccPnxYGBoaEsbGxgSLxSL09vYKb7755l/9tSoIwq1rulNBKpUil8tRKBQIgoDFYnFUqG+EXC5Hr9c7ZN8ikQiFQoFUKkWv16PRaJDJZIyNjTkk3l8zhYWFrFu3jvLycn75y186Pb6Pjw9z585lwYIFnDx5Er1eT2dnp9N1AISFhfHEE0+wfv16PD09kclkiESiCbNALBYjFv/tN0ozMjL4p3/6J/Lz81Gr1UgkEkQiEaOjoxw7doxXXnnlW9GlUqlYvHgxOTk57Ny5k/37909pfw4zXR8fH5KTkwkICGDPnj00Nzc7KtQ3wtPTE4PBwGuvvWb3fS9cuJC4uDjkcjlDQ0MMDw87/WETGBiIXq8nNzcXX19fvL29CQwMJCIigp6eHr788kuOHDnCxYsXaW9vZ3x83Kn6JhgcHGR0dBRvb28kEolTj5Ofnx9PPvkkGzduRKvVEhoaSlZWFu3t7U433pUrV/Lkk0+SmpqKWq12eLzMzEyef/55amtrefXVV7l48eK3dg18nccff5zFixejUChsnw0NDfHll1/yq1/9ijNnzjg0/sKFC6mtraW+vn7SMVGr1SQlJXHvvfdiMpm+u6YbGBhISkoKcrmcrq6u78SJ9ff3p7CwELVazYkTJxwSQyKRAODm5oabmxtisRir1eqQWNcSFhZGZmYmTz75JEFBQahUKiQSCWKx2NbqsFgsJCUl8fjjj9Pe3s6nn37Kyy+/THt7u8P1GQwG3N3d6ezsxGg0Mj4+jlqtxt3dHbVaTV9fn8M1TJCRkUFBQQGBgYGIRCJiYmL4wQ9+wMyZM3nzzTc5duyY07So1Wr0er3NcAcHB+np6UGlUuHr62v3eM8//zw5OTnMmTOHgoICGhoaOHr0KEePHqW2thar1Yq7uztubm54e3szZ84cZs2axd69e3n11VftrmeC5cuXM336dJRKpS3DBzh8+DA/+9nPqKiocKiHLFiwgB/96EcMDw/zs5/9jK+++srWQhWLxbi5uaFQKCY9EO4Uh5iuUqkkOTmZ+Ph4zp07R3FxsSPCfGMCAwPJzs7m0qVL9Pb22n3/IyMjtoxNLBbbmkfOoLCwkKeffpqoqCibjtHRUU6cOEFLSwsikYiwsDASExPR6XTo9XqCgoJITk7m1VdfZe/evQ7Vt3jxYtLT09m+fbstU5BIJLbNWSxYsIAnn3yS9PR0xGIxzc3NXL58GR8fH5YsWcLAwIBTTReulqUGBgYoLS1l+/btVFRUcNddd/Hcc8/ZPdZLL72Et7c3QUFBhIWFER4ezowZM1i3bh19fX1YLBaUSiVubm5IJBLUajVKpRLAYaZbUFDA9773PeLj4xGLxZjNZg4dOsTbb7/NwYMHaWlpwWw2OyT2BN7e3uh0OoKCgoiNjaW0tNRmul1dXZSXl3Pp0iW7xHKI6fr4+JCQkIC/vz8lJSUObxbcjKysLHQ6HUeOHKGjowONRkNAQAC7du2a9DS1F/7+/sjlcgBKSko4e/as0zL80dFRampq+Oijj9i/fz8DAwNYLBb6+vpsF6xSqSQoKIjExERycnLIzs4mNzcXo9HIwYMHMZlMDtPn7e2Nj48Pbm5uDotxO+Tk5DBjxgw8PDzYt28ff/zjH6murmbmzJk8/fTThIeHo9Fo6OrqcoqevXv3cuHCBWQyGd3d3TQ2NiIWi2lra3NIvAMHDtDU1IRKpSI2NpbFixeTmZlJYGAgISEhAFitVsxmMyKRCDc3N/r7+2loaHCIHvgfw3Nzc0MQBAYGBti3bx+7du1iYGDAYXGvJT4+HpVKxeDg4HVlQZVKRXBwMAaDAR8fH3x9fenp6bnjWA4x3YiICBITE7FYLHR0dDj0Zr4ZYrEYPz8/PD09GR0dxWAwMGfOHABOnjzpkJh9fX2TOs0sFotDzP1GFBcXc+rUKXp7e29Zq7106RLl5eWcPXsWuNqsi42NRS6XO/Q8+fv7MzQ0dF0Lw1ktgQk8PT1tmZter8fLy4v6+nqkUin19fWkpqby+OOP89Of/tQpepqbm2ltbQWump1YLCYlJcU2PGp8fJz6+noqKirsEm94eJiqqipEIhFnz57l+PHjaDQaYmNj0el0SCQSuru7uXLlCoGBgaxZs4aQkBCbRnvj6+vLwoULbcMrBUFgfHycvr4+pxku/I/pvvHGGxw5coSRkRHbd3K5HB8fH7RaLfHx8WRlZbF79+47jmV30/Xy8mL27NkkJyfT2dnJ5cuX7R3if0UikTB//nxmz57NqVOnGBgYICQkhLi4OEZGRmhpaXFIXLFYbDMRZ9Rxr6W1tfW2bozx8XH8/PzIzc0lJSXF1kvuyIeDTqcjNDSU2tpaWwY30SM/NjbmtE40tVptq3W3t7dz+vRpLly4wPDwMC0tLZSVlVFQUMD06dOdoge4bmRPQEAAS5YssZmuxWKhra2Nuro6u8cdGBjg/PnzAJw5cwZ3d3dEIhEjIyOMjo5OMkNHtdjc3d2Ji4vD09PTIfu/HZYtW0ZycjIKhYKzZ8/S1tY26f4dGxtjcHAQs9lMSEgIubm53y3T9fPzIyYmBp1OR3V1tUObJTdCqVSyfPlyVq5cyYULFygvL8diseDj44OPjw9VVVX09/c7JPZExxlcPVE3Ml6JREJwcDA5OTno9XoqKyupqqpyyugOjUbDzJkzWbFiBUVFRQQHB9PS0sIHH3zA8PCww+ImJCSg0+n46quvbM0yHx8fPD09OX/+PENDQw6LfS3p6elER0cjCAJ79+7lzTff5MyZM5jNZoaHh+ns7EQsFiOTyZyi5+tIJBKSkpJYuXIlvr6+jI+PU1dXx9atWydlXo5gYGBgUmYZFhZGTk4O/v7+nDlzxmF17vz8fPz9/Se1eJRKJXl5ebZS1MSwsUuXLtm970Gj0bBx40ZCQkKQSCSMj49fd98ajUYuXbpEW1sbOp0OnU43pZh2NV2JREJ4eDihoaGMjIxw4cIFamtr7RnilrFnzZrFggULyMjIoKmpiU8++YT6+nrUajWxsbEolUqKi4sd9tRua2uju7ubwMBADAYDvr6+1w2H8vDwIDMzk2effdbWtK2oqOAvf/kLx48ft7smmUxGZGQkycnJpKenk5mZSXx8PFKplCNHjvDpp5+yfft2h44l1uv1tLW1cfnyZVsJw8PDA3d3d5RKJSkpKSgUCq5cuUJ9fb1DNHh4eLBo0SJiY2Pp7OzkwIEDlJaWOtzMbheRSERoaCj5+fnExsYCV41w//79fPrpp07XYzAYmDZtGkajkUOHDlFWVmb3GAsWLGDdunX4+/tPaiW6u7uTl5dHeno6IpEIkUjE2NgYra2tZGZmUlJSQk1NjV1arDNnziQ1NRUPDw8AW3Y/Ojo66Xfx8fF2GbkAdjZdf39/8vPzSUhIoK6ujiNHjjilQ0Kn01FYWMiyZctsw8E++ugjqqqqGB8fJzAwkBkzZjAyMuKwei5AQ0MDJpMJkUhEQEAAXl5ek5rtcrmc1NRUVq9eTWJiIgDBwcEkJiYyODhod9ONi4sjOzubrKwskpKSCAsLw8vLi8HBQfbv388bb7xBWVmZw4eM+fv709fXR1dXF1arlcDAQKKjo/Hz8yMzMxOlUklzczPvvPOOw0w3OjqajIwM/Pz8aGlpoaur6ztjuADh4eGsWrWKJUuWIJFIGB4e5vjx4/zlL39xyEibWyESidDr9URERNDf328rwVyLVColLi4Os9nMxYsX7yhORkYGCQkJtmFiE6YrFottLdMJ07VarURERJCSkkJGRgbHjx/nT3/6E1euXJnS/9XX1xep9H9scOHCheTm5l5XbpPL5Xh6ejI4ODjlWrPdTTc5ORlfX1/27NnD0aNH7bn7GxIQEMCDDz7IunXr6O7uZsuWLezdu5fW1lYEQUAmk5GQkEB8fDwnT550WIcAXK2r1tfXk5ycjKenp20kwwQeHh6kp6eTn58/6XO1Wk1BQQGZmZl2bcYtX76cdevWER4ebmuqCYKA1WrFZDLR2dnpsF7yawkLCyM4OJjQ0FDUajW5ubkUFBTg5+dHc3MznZ2dnDlzxqEllsDAQPz8/JBIJDQ3N9Pd3X3D3w0NDdltaNAEaWlpJCYm0tbWRkNDw3Vm7+Pjw6JFi1i3bh3R0dEMDAxw5MgRXnvtNYeMJ5fL5SQmJmIwGCYZzrXfFxQUEBwcTFdXFykpKdx9992TfuPh4UFRURFtbW08++yzd6TDarVO6k8QBIHR0VHOnDnD+fPnGRsbIzw83FZjl8lkeHh4MH/+fKKjozl16tSUTbesrIy6ujrc3NxsSZIgCEil0hvewz09PVPu1LSb6bq5uRETE4PBYKCnp4fq6mqamprstfubcv/99/PEE08QFhbG3r178fb2Ji8vj66uLpqbmxEEgbS0NAwGA62trQ7ttOnr6+PMmTPMnTsXpVKJUqlEIpHYakS+vr5ERETYmjITTGQNRUVFdjVdpVJJa2srYrEYhUJhG3epUCjIz8/HYrHw+9//nrKyMoceF4lEQnR0NA8++CBSqRQ/Pz80Gg0jIyN8/PHH/OY3v3HqTLDz589fd22qVCp0Oh0NDQ3s2bPHLnFEIhHR0dF873vfo6CggMuXL1NdXX1d1qjT6UhPTycqKorBwUGOHTvGyy+/7LCx01qtlscff5y0tDTc3NxsGeaE+cnlcjQaDQqFAo1Gw1133UVqaqotwxseHqatrY2hoaEpjXjp7u5meHh4UpZbU1PDH//4Rz777DNGR0dJSEggPz/f1hdSWFiIRqNBq9WSlJREcXHxlEpj586d48033yQhIYHIyEgaGhro6OjA3d2dsLAwNBoNcrncNqa5r6+PCxcu3HE8sKPp6vV6CgsLiYmJ4ezZs9TV1TmlVzonJweRSER5eTlWq5V58+ahVCrp6+uzZbtpaWlotVqioqIcqmUis56YwaLT6fD29qa/vx+JREJcXBwzZsy44d+OjY1N+an9dXbs2MGhQ4eIjo5GrVYjl8sxGAwkJSUxY8YMli5disVi4bnnnnNoiaGiooKwsDDkcjkNDQ0UFxczY8YM4uPjKSsrc/rUW5PJNGmwvUgkIigoiKSkJC5fvmy3B19sbCxPPfUUd999NyqVCoPBQFZW1k1/LwgCly9f5r/+67+mbCa3QqFQEB8fj4eHh830JuJPNO19fX0ZHh7mypUrNDQ0UFVVRVtbG4Ig0Nvby8mTJ6c8/v7YsWO0tLQQEBBg67ysqKjg5MmTtnuhpKSEkpISpFIpycnJtlbBROJw9OhRDhw4MCUdmzdvxsvLi4CAANrb2+nr60Mul+Pv7297bcBDDz1EeHj4lOJMYDfTjYuLs5na6dOnqaqqsteub8lHH31EaWkp586do7q6Grg6TjgmJobY2FgyMzNtHQKDg4MO19PW1kZzczMpKSm2+lx7ezsKhYI5c+bYOkkmsFqt9PX18dVXX9k9s6mqqqKqqoovv/zS9plarWbJkiU8++yzpKamkpGRQWpqqkNnpH3yySdUVlYyPDxMfX09AQEBJCUl0d/f75QpyF/n2llWIpEIjUbD9OnTiYqKoqyszC6jW+RyOY888giPPPLIbXfATDSvR0ZGUCqVDpuF1dHRwfbt2zGZTNdN0ffy8uKee+7BYDDYss6jR49y5swZuz8EKioq2L9/PwaDgaCgIMRiMYGBgYSHh1NXVzfpfh0fH6enp8dW85fL5WRlZfHkk09O2XTh6ggFo9Fo+/fIyIgtlkwms02jHxoa+m7UdFUqFZmZmYSEhHDp0iUOHTrksA6Rr/P2229f91ltbS3FxcVIJBI2bdrEM888w8GDB2/4W3uza9cu9Hq9rXOvsLDwlr8fHR3l0KFDvPDCC3bPdG/ERL3wyy+/JDU1FaVSSWhoqENjtrS0TOppnjZtGn5+fgwMDDht5te1SKVS29RjpVLJ3Llz2bBhAwaDgZKSErvEiIyMZMmSJd9oBp5YLCY1NZVnnnmG4eFhPv74Y7to+Tr9/f289NJLN/wuIyMDuHqdHD58mM2bNztsiCXAb37zG3x8fFi1ahUajYZ58+ZhMpkQBIGqqiqGh4cxm81oNBri4uJITEy0da6JxWLc3d0dpm2C8PBwEhISbEY81RFZdjHduLg48vPzCQwMpKKigsbGxu/EC27c3d3R6XQYjUY++eQTysvLHR6zr6+P8vJyqqur0Wg0qFSqm/7WbDZTU1Njm2/vLLy9vfH390cQBLq6upz+rgGpVIpUKmVgYOCmHVqOpKurC6PRiEwmY9q0aaxYsYK0tDSqqqo4dOiQXWLMnz+f8PDwW76ScXx8nKGhIcxmM2KxGKVSaZsssGrVKsrKypz6dj6JRMLMmTPJyMigrq6OAwcOONRw4er98uGHH5KcnExGRgYymYx58+ah1Wqpra2lrq6Ozs5OsrOzCQwMJCMjA6vVytjYGLW1tWzZssWh+gB6e3vp7OwkMDCQ0dHR64aTfVPsYrre3t54eHggkUiQy+Xf+vz6CaZNm0Z8fDw1NTUOGWd4MyY6AaxWK7NmzcLDwwOFQmHrLJh4J8K5c+d45ZVX2LVrl9O0qdVqMjMzbRlFc3MzHR0dTosPVzPfpqYmwsPDCQoKcsoIimvRaDQEBQURGhrKunXrWLZsGd3d3Rw4cMBumW5WVtYNX+QzYRjDw8M0NjZy4sQJGhoaUKvVJCYmMmvWLDQaDYWFhbS0tPCDH/zALnpuB4PBQFxcHBqNhqqqqjseCvZNOXz4MO+88w7u7u7ExMQgl8uZPn0606dPZ2RkhPHxcVvyYrVaGRgYoKGhgZ///Ofs3LnT4fqqq6s5duwYs2fPJioqisTEREpLS4GrDyqlUvmNSpdTNl03NzfS0tLQ6XSMjY0xNjY2aYzb13tGnYVcLic7O5uAgADeeecduw8D+t84evQoP/3pT5k1axbZ2dkkJSXZanudnZ3s3r2bLVu2OKwMI5PJkEqlmM1mLBYLIpEId3d35s+fz5o1awgMDKSuro5PP/3U6aYHzr8eBgcHba+UXL16NXFxcSgUClJSUnBzc6OkpIQ33njDbg8gf3//STObJsyip6eHS5cuceTIEYqLi6moqGBwcBCZTIZOp+OBBx7ghz/8ITKZjLS0NLtouR3EYjErVqxg6dKlwNURHs5oGcLVjs3XX3+dnp4eli1bxvTp0wkNDbWNABofH8doNGI2m+nr66O4uJj9+/fz4YcfOkXfxDRgi8VCXFwcy5Yto6KiArFYTFRUFDNmzODPf/7z7e9wqsv1ZGRkCIcPHxbGx8eFyspKYdWqVYJMJhMAQSKRCCqVSlCpVE5d8gMQYmJihM2bNwvvvfeekJCQ8K0sPWKP7U51ZGZmCmvWrBEiIiIENzc3wd/fX9i0aZNw+vRpYWRkRBgYGBDee+89wdfX1+nHIyIiQnjllVeEvXv3CjNmzHDaedmwYYPt/2+1WgVBEITR0VHhwIEDwtKlS+2qIzIyUigpKRGampqElpYWobKyUnjmmWeEgoICwcfH54b7FIvFQmRkpLBx40bhhRdeEJ5++uk70nEn5yYkJET4wx/+IBiNRmHfvn3C/PnznXatXrvp9XohLy9PeOutt4Tm5mahublZOHz4sPDMM88I8+bNE2bNmiV4eXk57Vqd2ObNmyccPHhQMJlMwtGjR4WioiJh0aJFwsWLF4Wamppvdm6maroJCQnC559/LjQ1NQkvvfSSkJiYOKX/nL0OWF5enrBz507h+eef/1Z1THW7Ux2/+93vhAsXLgiPPvqosGrVKqG4uFgYHh4WxsbGhJaWFuG3v/2tEBAQ8K0cjwnT/fTTT4WUlBSnnpe8vDxhx44dQnNzs9Db2ysUFxcLCxcu/Ju6Pu5Eyy9+8QvBaDQKFotF2LFjx//JROVWm1KpFO677z7h7NmztvXrzGazMDAwIJw+ffobnZsplxfOnj3LggULprobu7N///4pL6vx18zY2Bienp62ZYlEIhHj4+PU1tby1ltv8cYbbzh9eukEVquVoaEhLly4YBvm5yz+r18XN8PNzQ2pVIogCE59JelfCyaTid27d3PlyhWeeuop8vPzKS4upry8/Jv3A0w107X35tJhHx3+/v7Cb3/7W6GpqUkwmUxCb2+v8NJLLwkxMTGCWCz+P3c8/i/puBMtP/zhD4WmpiahoqJCePzxx79RSfCv4Zh8V3QIgoDoVk80kUh08y8dhCAI173V2qXDpcOl4/Z1fJe0uHRcz9/+us4uXLhw8R3ilpmuCxcuXLiwL65M14ULFy6ciMt0Xbhw4cKJuEzXhQsXLpyIy3RduHDhwom4TNeFCxcunIjLdF24cOHCibhM14ULFy6ciMt0Xbhw4cKJ3PKFN9+VqXMuHS4dLh23r+O7pMWl43pcma4LFy5cOBGX6br41gkICOD555+nsrKSlJSUb1uOCxcOxSGm6+bmxqOPPkpdXR1nz57ln/7pn/D29nZEKBd/A6SmpvLss8/S1NTk1AU6Xbj4NnCI6cpkMgwGAyEhIcTExPAv//Iv/OpXv3KI8S5btoz333+f5uZmjEYjXV1dbN68mX/+539m7dq1xMXF2T3mNyE3N5cPP/yQixcv8s///M9Oi7tmzRpOnz5NR0cHn376KU8//TRz584lMDDQaRpuh7y8PH72s5/R2dnJ9u3bHfby7NWrV1NaWsqHH35IaWkpvb29tm337t0UFBTg5eXlkNi3g0wm46677uLIkSNcvHiR559//lvT8l2hqKiIffv20dfXR19fHydOnGD+/Pnftqyp44iXmMvlcmHTpk1CS0uLMD4+LoyNjQl1dXXC3//939v1BcSbNm0SKisrheHhYcFisQiCIAhWq1UwmUzCwMCAcPnyZeHtt98WioqKhP8upDv9RcgRERHCr3/9a6GtrU14+eWXnfZC5hdffFHo6uoSrFarMDo6KgwMDAjd3d1Ca2urcOjQIeGXv/yl8NRTTwnp6emCXC532vG4dktPTxd27NghDAwMCDt37nSIjoiICOGll14SamtrBZPJJJhMJmF0dFQYHx+3bSaTyXat3HXXXbf1knd7H485c+YIH374oWA2mwWj0Sj86U9/uuPr47v00u472U9UVJTwk5/8RLhw4YJgNpsFq9Vqu6/Pnz8v/OY3v3H6tWqv4yHYY7mer6PValm7di2PPfYYGo0GuLpUTFBQEOvWraO0tJRjx45NOY5CoWDJkiVEREQgFos5cODApOVnEhISiIiI4N5778XT05Pu7m7bssnOxN3dHV9fXzw8PBCLnVdC37JlC7Nnz8bX1xeZTIZMJsPDwwMAPz8/UlNTGR8fZ3h4mL6+Pjo6Ovjtb3/LRx995BR9Hh4eZGdns2TJEkpLS3nuuecYHR21a4yNGzfywAMPkJqailqtRiwWIxaLr70RgatZZlBQECtWrECr1XL8+HGnLkuvVqvJyMhg5syZSCQSRkdH6enpcVr8CQoLC4mPj+fw4cOT7hWlUklubi5/93d/x2OPPUZ3d7fDNNx///2sX7+e9PR0fHx8KC8vZ/fu3QQHB7N27VpCQkJISkrCzc0Ns9nsMB2OxK6mq9PpeOqpp1i/fj0BAQGTTEYqlRIZGcmaNWvsYrpisZjOzk62bdvG9u3bqa6unnTTTiyv/eijj5KQkEBOTg6VlZV2v7H/NxISEiYtv+4svLy8kMlkWCwWPvjgA95//308PDyIiooCri4RnpGRgUajQalU0t3dzblz55ymr6CggHXr1nH58mX++Mc/UldXZ/cYf//3f090dDRubm62z2pqarhy5Qp79uxBoVBw9913ExUVhVqtRqFQ2MzZmWRkZLBo0SK0Wi2Dg4N8/vnnvPTSS07V4OnpSX5+PoWFhZhMpkmmGxMTw09+8hO8vLzw9vZ2mOlGRUWxbNky5s6di1Kp5MKFC7z11lu8++67aDQaWltb+cd//EdiY2O59957effddx2iYwJ3d3dmz57N2rVriYiI4PDhw+zdu5ejR48yMjJyx/u1m+kaDAa+973v8eCDD2IwGBgaGmLfvn3U19fz4IMPotVqkcvl6PV6u8QbGRnhhRdewGq10tXVdUMz7e3tZcaMGRQVFZGVlcXOnTtpaGiwS/zbxWg0Mjg4iFgsRiS64bA9u+Pr68vixYsxGAxUVlbyX//1XxQXFyMWi1EqlcDV7M7HxwcvLy9GR0fp7OykpaXFKfqSk5O59957CQgIYNu2bezatQuLxWL3OD4+PsjlcgRBoK2tjY8//pjt27fT0NBAb28vYrGYkpISNmzYwIYNG+we/3aQy+UkJCQwbdo0RCIRFRUV/PnPf6apqcmpOsLDw4mNjUWj0UxKENzd3YmLiyMhIYHW1laHXcNxcXE8+z91TG0AACAASURBVOyzLFy4EHd3dwBGR0dt9dyBgQFKSkrYtGkTWq2Wxx57jF27dk3J/G6Fu7s7S5YsYePGjcyePRuFQkF8fDzz589n69at/P73v7/jTNsupuvt7c0TTzzBAw88QEBAAAAnTpzgtddeo62tDYVCwcaNG5FKpRgMBoKDg6d8UVmt1v91Hz4+Pvj5+dkulPHx8SnFvBO8vb3x9PQEcFgn0ddJTk4mNzcXpVLJ9u3bOXLkCENDQwAMDAzYftfS0oJEIkH47xVgnYFKpeKuu+5i6dKlHD9+nDfffBOj0eiQWD/60Y9YsmQJFy9eZN++fdTV1dHe3j7pAX3y5Ek0Gg3Tp08nNTWVkJAQNmzYwL//+787RNPXSUlJYe7cuej1ejo6Oti/fz9HjhzBarU6Jf4E0dHRBAUF0dbWNum+8vPzIz09naGhId59912uXLnikPiFhYXk5OSgUqno7+/HarVy9uxZysrKALBYLAwPD2OxWHBzcyM0NBSp1O7VUQACAwNZuXIlDzzwAFFRUbS2tmKxWIiLi8PT0xOTycTBgwc5ceLEHe1/yqqVSiVr167l3nvvxWAwIBKJOH36NG+//TaHDx/GbDazb98+Vq9ejUqlwt/fn8TERIc/yVNTU3n00UdZunQp586d48MPP6Srq8uhMW+EXC5HJpM5zXClUin33HMPkZGRjIyMUF1dTWdn5w1/KwiC0x9ECxcuZMWKFQwPD3Po0CEuXLjgsFgffvghJ06cYGBggI6ODsbGxiZ9LxaLiYmJoaCggJCQEARBQKlUEhwc7DBNXycrK4tZs2YxOjrKwYMH2blzp+0B6Sz8/f1JT0/H39+fQ4cO2Uo9MpmMmJgYCgsLGRwc5OTJkw7LLCf6PYqLi/nkk08YHh6msbGRy5cv3/D3EonEIToA7rnnHtatW4dYLObll1/m888/RywWs3z5cjZt2oSvr68tkboTpmy6S5cuZfXq1YSFhSEWizEajXz22Wfs2bPHllX19vYyPj6OSCTCy8uL3Nxc9uzZM9XQN0Qul1NUVMTDDz9MTk4Og4ODbNu2jeLiYqfXc78NoqKimD17Np6enpw/fx6j0ej0rOlm6PV68vLyCA8PZ9euXWzbtu06I7QnRqPxlll0cnIyjz32GMuWLcPX1xer1crAwIDTats6nY7Y2Fi0Wi0VFRV88MEHVFdXOyX2BBNmsnDhQtRqNRcvXrSZrkqlIi4uDoPBQElJCadPn7Z7fL1ez/r161myZAmenp4cPXqUnTt3MjIygsVicfo9m5mZydKlSxkdHeWdd95h165ddHV1kZmZaeu07+np4eLFi3ccY0qmO2PGDNavX09iYiJubm4YjUbee+89duzYcdPsSiqV2koQ9iYxMZH58+ezYsUK0tLSqKurY8uWLezatcupvdE3QiQSOaWmO3fuXAwGAxKJhOHhYSIjIycZm0gkwmw2097eTnNzs1Mz3aVLl5KXl8eFCxcoLi7G3d2d73//+yQnJ9Pb28vevXv5/PPPnaIlJiaGhx56iGXLlk26Hru7u/nqq6+coiEuLo7w8HCkUint7e3U1tY63WSmTZvGkiVLiIuL48yZM5w6dYqhoSFEIhHh4eHk5+fb+mccUfNPS0tj1apVJCQkYDab6ezsxGg03vJh3N/fzxdffOGQB/bs2bOJjo7mzJkztLa2Eh8fz/Tp05k/fz4zZ85keHiYc+fOTelY3LHpSqVSHn74YWbNmoWHhwcWi4XS0lLef//9G2YKE6bjKONZtGgRDzzwAHPmzCEoKIjGxka2b9/Ozp07aW5uRiaTOX2IiVgsRi6X23rPHV1iUKlULFq0yDYJJTg4mEceeYTBwUHbb0QiEWNjY/T19XHp0iW++uor9u3b59CME64OJczNzSUwMJBDhw6h1+t56KGHiI+Px8fHB6PRiF6vd5rpBgcHk5KSgr+//6TPR0dHaWtrc4qG8PBwAgICEIlEtLW10djY6JS417J8+XKmT59OX18fu3fvto1a0Ov1zJ8/n+zsbBobG9m3b5/dW0xSqZTs7GwMBgNSqZQTJ05QX19/w/4FlUple0BdvnyZ//zP/3TI/axQKJDJZEybNo3HHnsMuVyOwWDA398fDw8POjo66OnpmVKycsemW1RUxIIFC/D29kYQBMrLy9m6dStlZWW3FGS1WjGZTHca9oYsXLiQJ598krlz56JWq4GrhXeDwcDy5csRi8VIpdJJ5tPb28t7773n0Kb3RH1Qo9HQ2dlJc3Ozw2IBhISEkJiYaOt99vf3R61W09XVZTvmbm5u+Pr64u3tzeDgIFlZWcTGxvLJJ584ZNjWBIsXL2b69OlYrVZiYmKYPn06fn5+fPbZZzQ1NZGWlkZ8fLzD4k+gUqkIDQ0lPz+fqKgo2/Cwvr4+jh07xo4dOyZdJ45ErVbj7u6OSCSyjZV2FiKRiLS0NObPn49er6e6upqLFy/a+j38/f1JS0tDoVBQWVnJpUuX7K4hLCyMrKwsvL29GR4eZvfu3VRWVl53T4pEIgIDAykoKECpVNLT00N5ebnd9QAcPnyYxMRE0tLS0Gq1tplwfn5+FBYW0tPTM+Xy0x2Zrkaj4ZFHHiE4OBiRSERnZyfvv/8+e/fuvWln1cSAdJPJxJkzZ6Yk+uukpqbi6+tLY2Mj7e3tWK1WhoaGUCgUxMXFIZVKUSgUeHt7ExoaavtteXm5QztyNBoNYWFheHl5UVVVddNOAXuh1WpRKBSIRCKam5upqamhrKyM2tpam+nK5XK0Wi2xsbHMmjWL7OxsoqOj8ff358UXX3TIje/r68uiRYuIiYnBarUye/Zsmpub2bJlC2+99RZisZjAwEAiIiLsHnuCieFyCxYsYOHCheTn5xMaGgpcNdz9+/fzyiuv8OWXXzpMw7XIZDK0Wi1qtZru7m6nd/J6e3uzfv16EhISkEql+Pr6smTJElQqFW1tbSQnJ5OQkEBbWxuHDh1ySBkqNzeXyMhI3NzcOH78OAcOHLjh9ScWiwkNDaWgoACz2Wx3/7iWo0ePYrFYiImJwWQy0dvbi0wmY9myZQwODnLq1KkpXyN3ZLpJSUmkpKSgVCqxWq0cOHCAL7744oYHTCqV4uHhgUQiwWKx0NbWxv79+6ck+uuUlJTQ3t4OwOXLl7FarbbZVhMoFArCw8OZN28e9913H2FhYSQnJzvUdKOioggMDEQkEtHb22vT6CiMRiO1tbV0dnbyySefUFJSQllZGb29vZNKGxKJhODgYO6++25WrlxJeno6a9asoa2tjVdeecXupYa5c+cSFxdnK7NcunSJLVu28Oc//5nGxkZmzZpFRkaG3VtA15KUlMSsWbN46KGHbBmc1Wqlv7+fffv28fLLL3P48OGb/r2npycRERF2y7DkcjnBwcH4+Phw7tw5p5cWfH19mT17tm3WaGBgIGvWrCEvL4+Wlha0Wi3h4eGUl5dTX1/vEA1xcXEolUoEQeDo0aM3rZN6eXmRkJBAaGgoly5d4osvvnCIHoCxsTEOHz5suxaUSiX33Xcfs2fPpq6ujp07d065tv2NTVcul5Ofn4+Pjw8ikQiTycSnn37K+fPnr3saKhQKEhISKCgoQC6XYzKZHJLxHTt27LZmuVVWVjIyMsKiRYtQKpWEh4fbVcfXCQ8Pt00GaW1tdXimW19fz+uvv47FYmH//v037cy0WCzU19fzpz/9ia6uLn784x8TGhrKY489xs6dO+0+FnPp0qVERkYC0NHRwfbt23nrrbdobGxEo9Ewd+5cIiMj2bx5s13jwtVrMDMzk4ceeoilS5ei0+kQBME21fbYsWO88cYbN00EpFIpOp2OoqIiCgsLWb16tV10iUQiZDKZbdqvo4Zi3Qyz2UxzczNRUVHI5XLb5zqdjoCAAKRSKRaLhYGBAfr7+x2uZ2ho6IYPe7VazaxZs8jLy2NgYIDTp087tAz2dby8vEhNTWXatGns2rWLffv2TXmf39h0NRoNixYtQqVSIRKJGB0dpbm5+brOGh8fH9LT021jeEUiEfX19Rw8ePBbG7rl7e1NWloaQUFBDA4OEhMT49B4EokEsVjM8PAwTU1NDs90+/r6vtHUSKPRyP79+0lJSeHpp59Gr9czY8YMu5qum5sbwcHBqFQqAPbv38+2bdtobGxErVazYMEC1q5dS11dHW+//bbd4k7g7e3Nj3/8Y2bOnDlpOnBzczN79+5l27ZtHD9+/IZ/K5PJSE5OZsmSJWzYsMGuLQAvLy/b7MDe3l6nv2uht7eXrVu3cuXKlUljTiUSCRERESQlJWE0Gjlx4gS1tbUO0XDp0qVbeoFKpSI7O5tHHnmEGTNm0NTUxOuvvz6l4VrfFI1Gg06nY3h4mO7ubrt41zc2XbVajV6vt80GcXNzIzY2lra2Nlum6+HhQXp6Og8++CA5OTkIgkBjYyNbt27lk08+mbLob4pYLMbDw4Np06aRkZEBXJ1GfPbsWYfGNZlMmM1mWlpaqKurc/qgd5lMhq+vLyKRCKPReMPme1tbG7t27eKhhx5CpVKxePFiPvjgA7tpMBgMtmmdIyMjHDhwgPLyclQqFbm5uaxbtw5PT0/ef/99h3SOSCQSfH19J2VzIpGI7u5uampqMJlMBAYGYjKZcHNzw8/Pz/aKR5VKxbp167j77rvp6urid7/7nV00icVi5syZQ3h4uK3+7qiZXjdjcHCQd99997qHtE6nY9OmTQQHB3P69Gl27NjhsBmDO3fu5IknnsDf39+WoFxLWFgYq1atYsWKFYyPj1NWVuawDrQb4e7uTm5uLrNnz+by5cscPnzYLi2Sb2y6AwMDXLhwAW9vb5RKJR4eHjzzzDPcfffdtptaq9USGRmJj48PVquV7u5uvvjiCzZv3uy0+f3X4ufnR15eHg899BB5eXmMjY1RU1PD+++/79C4YWFh6HQ6Ojs7GR4edvokhZCQEO655x48PT0pKSmhtLSUgYGB63RMDEL38fGx+2ys8PBwW5ZbU1NDX18foaGhzJkzh9WrVxMSEsKf/vQn/vVf/9WucScQBMH2/712yGJGRgYZGRnU1tZy8OBBmpub8fX1JTs727Z6hVgstk0/3bNnDy+++KJdNHl6etrGxsLVB1NERAStra2T+iGcjUgkYu7cuSxfvpyxsTEOHDhwx1Ndb4fBwUGMRiMWi4XQ0FB0Oh0dHR1YrVY8PT2Jjo4mNDTU9nKrI0eOOKXUMUFMTAy5ubkYDAbeffddPvvsM7vs9xubbnNzMz/+8Y958cUXSUxMRCaTER4eTlhYmO03177roKOjg48++oif//znDjNcqVRqK8hbrVZkMhlSqdSW4RYWFvL973+f+Ph4TCYT1dXVvP766w7tvBCLxYSFhaHX629aW3U006dP5+GHHyYhIYH8/Hw2b97MwYMH6enpsY1x1Ov1rFixAn9/f/r7+9myZYtdNQwPD9taQFeuXCEiIoL77ruPrKwsqqqq+MEPfsDu3bvtGvNazGYzdXV1hIaGolKprnu1Y0REBJGRkYhEIps5W61WRCIRFouF/v5+Dh06xMsvv2w3TRqNBr1eb3sZz7x58xgeHqa/v/+mpQ5noFAoSE9PJzU1lePHjzu8JWg2mzl+/DixsbGsXLmS3t5e3n//fYaGhmxv95o1a5bNRy5evOjw8eQTiEQiQkNDCQwMpKOjg8rKSruN7Lmj0QtfffUVH3zwAR4eHgQFBdmGKU0wOjrK0NAQV65cYfv27bz66quT3nVrb4KCgliwYAFwNRMPCAggICAAuVxOWloaWVlZWCwWurq6OHz4ML/+9a8dPutIp9OhUqmc9s6FG9HU1MSVK1cICgoiOTmZX/7yl9TX13Py5Elbj3RQUBAPPPAAYrGY+vp6Dh06ZFcNEzevVqtl2rRpTJs2Dbg64uQ///M/HX4eurq6eOGFFxCJRKSkpODr64ubm5vNYCUSie3dGGazeZLx9vT0UFVVxfLly+2qqaOjg5aWFkwmE+7u7ri5uaHVam0jCb4tJo6F1Wrl8uXLVFZWOjTe2NgYlZWV9PX14evry1NPPcVjjz0GXC1bSqVSxsbGqK+v57333rNr2et/w9PTk5ycHBISEjh69Khdp0Df8eSIX/ziF3zxxRds3LiR7Oxs26QEuPrmpp07d3LixAkaGhocOhQIro7TfeKJJ4iNjbVdNBKJBIlEgtVqZXBwkNraWrZu3corr7wy6U1bjtQUFBQEXL24nPWEvpaysjJ+8IMfkJmZSXp6OjNnziQyMpL4+HhbTV4QBEZGRqitreWnP/2pQ2qL69evt/s+vwlVVVVs3LgRpVLJhg0bJi1+6efnR3h4OFarlYsXL9o6hNva2viP//gPh5w3uVyOUqm0XZ9Go5HS0lJOnTpl91jfBH9/f3Q6HbW1tZSUlDh8Mg9Ae3s71dXVtlr6RP0frk73raqqYtu2bWzevNmp99DMmTOZMWMGIyMj7Nu3z77JgSOW65nKdic6pFKpsGDBAuHFF18Uqqurhd7eXttWWVkp/PjHPxaioqKcuvRIbGyssG3bNqG7u1v4wx/+IERGRjrteNxsMxgMwoYNG4QvvvhC6O7uFnp7e4W6ujrhD3/4g5CXl+c0Hc6+Pm61BQQECAsWLBDy8vIELy8vp+hQKpXCiy++KDQ1NQkVFRXCL37xCyE1NdVux+NOj0lhYaGwf/9+4Z133hHCwsKcdm50Op3whz/8Qaivrxd6enps2x//+Mc7um+mejzEYrHw6KOPChUVFcKlS5eEJ554wr7n5m/BdF06XDpcOuxjuvv27RO2bNkihIaG/k0dk2/y99OmTRN27NghXLhwQfiXf/kXQavV2vXcOHddEhcuXHxnEQSB5uZmzpw545TSwneVgoICkpOT6enpueX7qO8UkXCLjp7/XkHXqQiCcN1ryFw6XDpcOm5fx3dJy1+bDr1ez2uvvYafnx8vvvgin3766R33Sd303LhM16XDpeNvS8d3SYtLx/Xc0nRduHDhwoV9cdV0Xbhw4cKJuEzXhQsXLpyIy3RduHDhwom4TNeFCxcunIjLdF24cOHCibhM14ULFy6ciMt0Xbhw4cKJuEzXhQsXLpzILV/t+F2ZxeHS4dLh0nH7Or5LWlw6rseV6bpw4cKFE3GZrgsXLlw4EZfpunDhwsUN0Ol0/PrXv+bzzz9n0aJFk5Ykmwp3vFzPdxWRSIRSqcTNzQ2AadOmMXv2bEJCQpg+fTpGo5Ef/vCHVFdXY7FYnKJHoVCQlJTEwoULyc3NZXx8nP/3//4fpaWlDo/v4ua4u7ujVquRSCTA1eVq7r//flavXo0gCHz88cf87Gc/o7W11aE67r77btasWcPJkyf53e9+962uCOzif5i4d+fOnYtIJMJsNvPll19Oeb9TMt05c+bwk5/8hOjoaI4fP865c+dISUlhYGCA48ePc/78eUpKSpxibu7u7mRkZJCTk0NhYaFtdWI3NzfkcjkSiQS5XI7VasXPz4/vfe97TjG9oKAg/uEf/oEHH3wQpVKJQqHAYrHw3HPPsXbtWqfdYBKJhIULF7Jp0ybkcjmff/45NTU1HDhwgP7+fqeco+8KPj4+ZGdns3LlSmbNmoVSqQSuHiMPDw88PDywWq3k5ubyu9/9zuF69Ho96enpjI6O4uvr+50yXalUiru7O2azmZGREYfGSk5OpqCggOHhYb788ktqa2tt30VGRvL973+fpUuXcubMGe6++26HarkWqVSKwWCYtOL5lPY3lT+Wy+Xo9XqCgoLw9fUlPz/ftqx0UVERQ0ND9Pb2Mjg4yMGDB/nVr36F0Wi0i/BrycrKYu3ateTm5qLValGpVMhkspv+PjIykqVLl1JVVeXQCyktLY0NGzawfPlydDqd7XOZTEZOTg6///3v+fOf/+zwFXEBQkNDWbp0KQUFBcDVh8GRI0dQKpWUlJQ4PJu7lolzc7OFBhUKBdOmTaOoqIjm5ma7LgtfVFTEww8/TGZmJnq9HqVSiVh8tco28ZpTkUjE2NgYDQ0NNDQ02C12YGAgWVlZHD16lKamJtvnWq2WgIAAZDKZTcu3TXBwMPPnz2f27NlERUUxNjZGX18fx44do6mpiZGRET7++GO7xXNzc+OBBx5g9erVWK1WPD09ef311zEajbi5uZGcnMwDDzyAt7e3U6/VCcRisW0x16kypb2cOXOGH/3oRyxevJjZs2cTGBjI4OAgbm5uaDQa/Pz8CAoKwmq1EhwcjEQi4Ze//KVdl2PfsGEDGzZsIDExEU9Pz9u6aH19fVm2bBnvvvsuFy5csJuWa9FoNNx///2sWLECf3//G2q45557mDZtGu+99x6vv/46Q0NDDtEyoWdiWfrR0VHKysr4t3/7NwYGBujv73dY3K+j1+t58MEHCQ8P58MPP6SyshJPT0+Cg4NJSUkhJiaGxMRE9Ho97u7ubN261W6xFy5cyDPPPMOsWbPw8PC4ZY1OIpHg6emJWq2mp6fHLvG9vLxYv3497u7u7Nq1y5aASKVSu93Qd4pEIsHd3R2ZTEZ2djYbN24kNTUVhUJhezharVYyMzMRBAGr1cqzzz7LO++8w44dO6Z8jBQKBdHR0QQFBTE2NoZWq7WVCCMiIrjrrrvw9fXFYrE4rVWWmJhITEyM3fc7pTPd29vL559/TllZGcHBwWg0GsbGxpBKpYSHhzNnzhyys7PR6XQEBgYyf/58tm3bZjfTXbduHU888QSJiYkoFIpJ342MjDA6OopcLr/uO4lEQlBQEAsXLnSY6T766KPcf//9+Pv7X/cg6O7u5sKFC0RGRpKWloZaraayspIvvvjCIVoAYmNjiYiIAK4ubV1XV0d9fb3D4t0IrVbLk08+yfr161GpVMybN4++vj7c3d1tdfiBgQFaW1upqamhuLiY8vJyu8Q2GAzcd999zJgxA09PT+DqMuunTp2ivLwcd3d3Fi1aRHBwsM18goODMRgMdjPdpqYmvLy82LRpEzU1NRw+fNgu+50qs2bNorCwkJkzZ6LVavHx8UGj0XDu3Dm2bNlCZWUlcLVlm5CQQHBwMEVFRXh5ebFixQoefPBB5s2bNyUN6enpBAcHIxaLsVgstvvX29ubnJwc8vPzEYlE9PT02DXDvhUajQZfX99Jn4lEIvz8/IiJieHIkSN3tN8pma4gCAwNDTE0NERLSwtSqdTWRPPw8KChoQG9Xo9Op8NsNtPa2kp7e/tUQtrQarU8+uijkwy3ubmZ6upqDhw4QG1tLePj48hkMmQyGT4+PsybN48VK1YAV5szEyZkb1JTU5k3bx5BQUGTDLe9vZ3PPvuMjz76iJaWFmJiYli5ciV5eXk89NBDHDlyxCH1vGnTppGfn2+rSZ0/f559+/bZPc6tUCqVPPzww6xatYr6+noaGxvx9vamvb2d4eFhzGYzp06doqmpif7+foaHh2lpabnj9amuJSMjgzVr1rBgwQLUajUWi4XS0lK2b9/OoUOHbNfu/v37uffee1m7di3j4+N0dXXZzXABBgcHqaurY9GiRaxfv57e3l66urrw9fW1Zd326iG/Hfz8/Fi5ciWLFy+2tS7kcjmdnZ3s2bOHv/zlLxw7dsyWkYvFYiorK3F3d+eDDz6wXdupqalT0uHp6cny5ctt92NFRQWlpaUMDQ0RFxdHZmYmBoOB0dFRzpw5w7vvvju1//htIpVKbZ2sE4hEIiIiImz36x3t1x7iJozNbDYzPj4OYKuVhoSEANDR0cHu3bvttrJmYmIi0dHRNsNtbW3l97//PQcPHqS+vh6j0YggCIhEIkQiER4eHoyOjrJ8+XLbZ3K53C5avq7r+9//Punp6bS1tVFZWcmJEye4fPkyg4OD1NTUUFtbi9ls5uLFi4yNjZGRkUFWVhbe3t4OMV0fHx90Oh1KpRKRSERjYyOnTp2ye5ybIRKJWLp0KatWraKtrY3XXnuNM2fOoFQqGRoawmw2Y7VaaW9vt4vJXsvEw3np0qVotVrEYjHl5eW8+eabfPzxx3R1ddmaqyqVCg8PDwDGx8dpb2+360qwgiCwa9cupk+fTlFREV999RVlZWV4enoiCAJKpRIvLy+7xbsZEomEnJwc7r//fgoKClCr1fT19dHf38/g4CDbt29n69at1NTUTDofVquV3t5eent7J60WfP78+SnpiYmJIS0tDW9vbwBOnz5NRUUFFosFjUZDZGQkEokEk8lER0cHV65cmVK82yEqKors7Gybf00gEolwd3e/7vNvwh2brlwup6ioiICAADw9PW03UF1dHRcvXgSuNlsCAgIAMBqNlJeX260eMzAwMCkr8PDw4Ny5c5SWlt4whtVqpaOjg66uLrRaLVKpFI1Gg1KptNuNLpFIuOeee5g/fz6enp589NFHbN26lcrKSrq7u221sInWgNFo5Pjx4xw4cID58+eTlJRES0uLXbRcS01NDcePHycpKYmgoCBMJhMDAwN2j3MzEhISWLVqFT4+Prz66qsUFxfT3d3tlNhLly4lPz8fnU6HSCSiurqaLVu2sGfPnkmtrolyQlJSEnD1ehkdHb1pZ9+dcvDgQUpLS1m2bBmZmZmcP3/ediw8PT0JCAigoqLCrjG/zn333ce6devIzMyksbGRL7/80taR9/nnn/PRRx/R0NDA7a6f2NfXNyU9RUVFhIeHI5FIGB0dpaWlhZ6eHtRqNfHx8ba6and3N4cOHbptXVMhNjaWxMRE20Owp6eHxsZGBEGgra2N4uLiO973HZtudHQ0jz76KDExMSgUCqRSKWNjY7S3t9uegvHx8ZPSc7FYjJeXF4IgYDKZpnRBd3R0YDKZbNmsp6cnWq0WvV5vM92xsTEEQSAkJIS5c+eyYMECW3HearViNpvtWpTPy8tj0aJFaLVa9uzZwzvvvMOxY8duauqCINDT08NXX33FokWLSEtLY+/evXbTM0FHRwc1NTV0d3ej0Wgc2mF3I4qKipgxBvJIRQAAIABJREFUYwbl5eUcPXoUQRBsZaGenh5b9mRvpFIpaWlp+Pn5IRaLGR8f59ixY+zbt4+2trbrfi8Wi22tH7PZTF1dnd019fT0cPz4cbKyskhPT+fAgQM2E1EoFKhUKrvH/Drr1q1j3rx5dHV1sXnzZioqKggJCaG7u5tjx47R0dHhcA0TyOVy0tPTbbXT1tZWmpubMZlMREZGMn36dLRaLQBdXV3s37/f4ZpEIhFxcXEEBwcDV1s9jY2NXLx4EavVSmNjIx988MEd7/+OTddkMnH58mVUKhXT/j97Zx4V9Xnv/9eszAwDDPu+yL6pICoYBUUigrvRaFwSl5ilaZo2SW/ae9Kmye29t2162iZNk5rFaNQYTeKucY0iouKOssimIJsssg4MMMzw/f3hjzkSlyjMkLR3Xud8zxHmy3w+fpf383k+z+d5nshInJyckEgk+Pn5MWbMGFMU2vdAeXp68sILL9DY2IhOp2P79u1cuHBhwI739PTQ3t5uEl2AhQsXEh8fT29vLyKRiLa2NgRBIDg4mFGjRjFs2DBEIhFdXV3k5+dz6NAh9Hr9gH24HZlMxvTp003lNd988w1nz5793ii6L3coEon6lZWZG7lcjlQq5erVq1y6dMlidr5LQEAAEydOxN3dHS8vL5YsWUJvb68pf9jU1MS5c+fYsGED7e3tZrWdlpbGqFGjUCqVtLW1ceTIEbZt20ZFRQW9vb39znV2diY6Oho7OztT6mfr1q1m9QcwRUodHR2EhISwbNkynJ2dgVuTM9LT0xGLxeTn51NQUGB2+3CrGkChUFBQUEBtbS2tra1cvHiRqqoqbt68aRGb98LX19dUVQNw9epVrl+/Tk9PD+7u7gQEBCAWi9HpdFy7dm1IUgv29vaEh4eb3sfi4mKOHDliKlXT6XSDSgMOWHSvXr3K6tWr8fPzY/jw4URHRxMbG0tQUJCp2Px2XF1dWbp0KQAdHR3Y2toOSnR7e3sxGAz9uhpTp041/VskEqHT6ejt7UWtVvcT57a2Ng4cOMD+/fsHbP92pFIpkydPJiEhAbVaTXFxMdeuXXvoiNISNcx9KJVKlEol586d49SpUxaz813kcjnNzc3U1NTg6enJo48+SmVlJRUVFbS1tREREUFISAg3btwYVPTwXZycnFi1ahUjRoxALpeTn5/P2rVrOXbs2B0vjEgkIigoiKlTp+Lo6EhTUxMZGRmcOXPGbP7cTl8e297enilTppieS09PT+bPn09UVBSbN2+2mOhevHgRJycnnJycWLx4MRUVFeh0OrRaLdXV1dTW1qLT6WhqaqK4uNjsefbb8fDwQK1Wm66BRqMhISEBo9FIWFiYKcrV6XRUVFTQ3d1tMV/6ePTRR4mNjTU1BEVFRWYd5B7UQFphYSGFhYUcPHiQgIAApkyZwtNPP90v0r0bcrmc8PDwwZhGp9ORkZGBp6cnrq6ud7WnUqn6/dwn0BKJBLVabUo1DBZ7e3uef/55oqOjMRgMHDlyhLKysu/NPfUl5T08POjp6eHixYtm8ed+iMViixbgazQaoqKiALhy5Qrl5eWsXbuWs2fPolKpaG5upqioiCtXriCVSnnqqad46qmnSE5ONqvoKpVKvL29TRMxWltbqaqquuuL05fPjYiIAKC7u9tsVTZ34+rVq2RnZ+Pm5oanpydyuRyRSGRKb/TVUluKDz/8kNraWiIiIlAoFIwdOxZnZ2dkMhk3b96kqqoKo9FIS0sL27dvZ9++fWbrEX6Xnp4eU4pPJBIxYsQI3NzcmDRpEmKxGG9vb0QiEb29vaZ0oaVJS0szPQs9PT00NzebNSAyW0V2ZWUlly5dMo32dnV1cfXqVQoLC+/oynV3dw86N6PVavnb3/7GiBEjeOSRR+5aiVBfX09xcTF6vR4PDw+USiUuLi44OjryxBNPUFNTw9///vdBD5bY29szYsQIVCqVqRF6kAExqVRKQEAAqampVFZWWiwCFYlEpvKXgIAAoqKiBj3ifC+mT5/OzJkzTTOXWltbycjIICMj445zAwICcHJyore31+w1w1qtFq1WiyAIGAyGewquWCzG09OTkSNH4uHhYTrfkjMVr127xpo1a6ioqCAmJgZ/f3+CgoJwdnYmLy+PTz/9lF27dlnMfkZGBsXFxfj6+mI0GvH19cXf39/UQ1UqlcTFxfHYY4/h7OzMxYsXqaiosIgvNTU1XLhwAVdXV9OECD8/P/z8/PqlKLu6usxaSXIv+tZb6Gusa2trKSoqGvRg4e2YRXSlUilRUVE888wzxMfH09XVxYULF/joo4/Ytm3bHaIrCIJZRoUrKiqoqqqip6enX1egsrKS7u5u8vPz2bVrF+3t7cTExBAQEMCcOXOIiYnBxcWFefPmsW7dukGPpGs0GlP0WFVVRU1Nzfe+tH1F1lOmTCEhIYF3333XYvk0W1tbXF1dsbW1payszKJpjIULF+Li4sJvfvMbKisr7xmZyOVyYmNjefTRR2loaDD7AImbmxseHh5IpVIqKirYtGnTXaf0ent7M3fuXNLT04FbEfHRo0f55ptvzOrPd7l8+TKXL1/G0dGRhIQEfvKTn5CUlMSpU6dYs2aNxSJLuJXT7ezs5MKFC/T09NyxBolSqWTZsmU88sgjuLm54eXlZTHRrays5P3336eqqso0oKbRaPDw8MDe3h6RSIQgCNTV1Q3JWine3t44ODiYBL+goIAzZ86YtdrHLKLr7u7Oc889x+zZs3FycuLKlSusW7eOLVu2WPThAfot1iIIAhs3buTTTz+lvr7eVKIFtx7y8PBwIiIiiImJQSaTERAQgI+Pz6BFNzY29q557PthZ2fHpEmTWLFiBXl5eWzYsOGOxslc2Nvb4+Pjg1qtprS01GK5QriVUpg0aRJeXl7U1NTcNbqUSqWEh4eTnp6Oo6MjO3bsMLtParXalCs8f/48paWld3TZ7ezsmDFjBj//+c/x8/NDEATKysr461//ajGR+S7Nzc3cuHGDjo4ObGxs0Gg0KBQKi743q1atoqioiD179tzx7NvY2Jgib6lUilartUhlye2cP3+eixcv4u3tjbu7OxERETz++OMkJydja2tLR0cHZWVlFp9BKZFImD17NhEREYjFYrq7u03jM7cjlUqxtbUdcPAy6OSenZ0dKSkpTJgwAY1GQ01NDXv37uXYsWMWF1yAc+fO9Rv1dnV1RaVSYTQaTSLWNzlCo9H0EzaJREJkZOSgfeibcfcgoikSibCzs2PcuHE899xz9Pb28uc//9ki5Ul9tLe3U1dXR3l5OVevXrXoCPWGDRtQKBS89NJLPProowQGBuLg4IBMJkMqleLg4EB0dDQrV64kNTWVY8eOsWHDBrM/Kw0NDTQ0NGA0Gk2R0+0pKIVCwfjx45k5cybe3t7ArbRXX0/lh6BPdC1dNpaQkMDIkSNxdHQ0TchwcnLC39+fpKQkli1bxtSpU+nu7iY3N9diU+Vvp7e3l8rKSlMly9atW03VAjdu3CAzM9PilQu2trZMnDjR9Dw0NjaSn59/h103NzcmT548YDuDinRtbW2ZMWMGL730EqGhodTV1ZlWzhqq0pOioiJTlYBIJCI9PZ1Tp06Zag2VSqVpcGfJkiU89thjGAwGWlpaKCgooKSkZNA+7Ny5k1GjRpGWloZEIrnnIOLtKYUXX3yRsLAwdu/ebdH8HdzqDVRUVHDlyhWuX79u0cYwLy+PDz74gJ/+9Kf87W9/48yZM2RkZJCfn48gCAwfPpzHHnuM4OBgMjIyWLt2rUUimMbGRpPoJicn09DQwNtvv01lZSX29vZER0ezZMkSxo0bZ5pplZOTw2effWbW/N2D0NPTY0q3qdVq3NzcLCr8fct8VlZWUlZWhru7O2q12jR93cvLC51Ox5kzZzh27JjF/LgXfYOK352Ca2n61n3o6zn3Tab67jl+fn7MmjWL7du3D8jOgEW3T+B+9atfERERQXt7O9u3bzdNrRwq6uvraWxsxGAwIJVKCQwMZOHChaZi65iYGOLi4vDx8cHV1ZX29nZKS0vZunUrW7Zs6bdm50ApLCwkKyuLRx55xLRMX2lpqalkTS6Xo1QqcXR0JD09nZdffhkbGxuOHDnCli1bBm3/x8aHH37IlStXWLVqFWPHjiU5ORmj0YiNjQ0ikYibN2+yceNG/vnPf1psmb7w8HDCw8ORy+WIxWLi4+OZMWMGKpWKefPmERoaako/FRYWsm7dOjZu3GjRqoV7UV1dTXl5OUajEXt7e/z8/My20M/deOONN/jJT37CK6+8YqriMRqNpslCFRUVZGVlsXr1as6ePWsxP+6FXC7HyclpSCaK3E5LSwuXL19m/PjxqFQq8vLy7gjK+gbaXFxcBmxnQKIrk8lwc3Pj2WefJSQkBIAdO3bw4YcfWmxU/F6IxWJ6enpMXXuJRMKcOXNMC9vArVyvXq+nqqqKjIwMPv74Y7OvYbtt2zZmzZrFhAkTeO2113BzcyMrK4uuri5CQ0OZOnUqiYmJ+Pn5UVVVxdtvv83Ro0eHPKoaKjIzM8nMzMTHx4eQkBASExNxcnIiLy+P7OxsysrKLDozrqysjG+//ZbZs2ej0WgYNmwYb731FoBpcKZvptFXX33Fhg0bhnQm1u1otVoqKipMo/OWXvTm2LFj5OXl8dprr5nGN3JycsjPz6epqYkrV64M+Xt8O0ajke7u7iFJT36XnTt38sgjj6BUKqmoqLDIdPUBie64ceP4j//4DxISEpDJZFy5coU9e/ZQVlZmbv++l9LSUt555x3eeOMNwsLCsLGxQSwWm+r6+mZ8ZWRksH79ek6fPm2RRWWqq6vZuXMnw4YNIzExkdGjR9PS0kJvby8ODg4oFAoaGho4fvw4a9asMcu2H/8KVFVVUVVVNSTTN2+ntbWVV155BYA5c+aY5tD39vZiNBrp6emhtLSUjz76iPXr1/+guzUYjUZKSkrIzc3Fy8vLtPSkJWlsbORXv/qVxe0MBIPBgFarteikjHtRUFDAunXryMzM5NChQ3ftiRkMhkEFDAMSXX9/f+Li4lAqlRQWFvL6669z+PBhi2/ncS/27NnDt99+y7x583jppZcICwsjNzeXgwcPcvDgQS5dujQkLec///lPbGxsePbZZ/Hy8jKtmtTe3s7u3bv5+OOPyc7O/kFa8L6V1f4v0drays9+9jNTDtPW1pby8nKysrLYtm0bZ8+eRavV/ii2Krp+/ToXLlzA2dn5jjVc/y8iCIKp3FAQhCG9R/eb/m00GsnKyhpcT7nvP3e3AxDudkRHRwsXLlwQcnNzhZSUFMHGxuau5w3keBg/LHn8O/lhZ2cnvPnmm8KBAweEBQsW/J+/Hv/ufvyYfBnod82YMUM4fvy4oNVqha1btwrh4eH/NvdGdL9pdSKR6N4fWghBEO4Ix6x+WP2w+vHgfvyYfLH6cSc/jl3wrFixYuX/CPeNdK1YsWLFinmxRrpWrFixMoRYRdeKFStWhhCr6FqxYsXKEGIVXStWrFgZQqyia8WKFStDiFV0rVixYmUIsYquFStWrAwhVtG1YsWKlSHkvgve/Fimzln9sPph9ePB/fgx+WL1406ska4VK1asDCFW0bVixYqV+5CQkMB7773HhQsXyMjIYMSIEYP6PusqY1Y/LOaHr68vsbGxREVFERERgaenJ0ajkeLiYnbv3j2gzUvNdT18fX1JSUkhPj6eS5cu8emnnz6ULz/m+zIQX3x9fXnttddYuHAhly9f5i9/+Qv79u0btC8P64dMJiMqKoqlS5cya9YslEolOTk5bNu2jd27dz/QVmDmvDdLlizh17/+NWFhYUgkEgRB4PDhw6SlpQ3IDzDTFuw/FsLDwxk3bhwuLi4MGzaM+Ph4fH19TZ83NTXx1Vdf8dvf/vYH9PLfG1tbW379618TFxdHREQEarWapqYmcnNzOX/+PCqVivj4eOLj4/nHP/7Bhg0bfhA/J0yYwAsvvEBUVBRZWVnk5uZy4sSJH8SXHwPBwcFERETg7OxMTEwMK1euxMvLi2vXrg3Zrh/z5s1jzpw5jBkzBl9fX9Puza6uriQlJfHMM8/w+9//nszMTItu9dRHcnIyTz75JJGRkYjFt5IC586d47//+78H9b0WF90RI0bwi1/8gpCQEA4ePEhOTg5Hjx7tt236YFm6dCkvvvgiAQEBpu16pFIpNjY2/XYUdXZ25he/+AXp6em89957fPbZZ2bz4X6IRCJ8fX1JTEwkNTWVkSNHUl9fz5dffsm6deswGAxD4sdQIBKJ0Ov1pr3o6uvrqa6uRqvV0tPTg1gsJi4ujp/+9KcsWrSIkpISsrOzh9zPvs1CFQoFAQEBDB8+fMhE18fHhylTpuDu7s7Zs2f7bX7Y1dVFc3OzaXfgocDd3Z25c+cSGxsLgEajYdq0aSQkJLBjx44hEd3Q0FAWL17MlClT0Ov1lJSUcPLkSS5evIi9vT2jR49m4sSJfPDBB+zZs4fXX3+dtrY2i/mTmprKq6++ysSJE02Cm5WVxeuvv87p06cH9d1mF11nZ2fCwsIYO3Ysw4cPB27tH3bx4kVaWlp48skn0Wg0Zotw3n77bebPn4+vry9S6f3/O2Kx2LTV9FtvvUVMTAwvv/yyWfy4F56ensyZM4eFCxcSFhaGWq1GoVAQFhaGo6MjBQUFnDx50qI+9KFWq4mOjmb27NkkJyfj6OiIXq+noqKC9evXm2Vn4o6ODt5//316e3vp7u7GYDBgMBi4PY118uRJgoKCePzxx4mNjf1BRPd2JBIJMplsSGzZ2Ngwf/58VqxYgbe3N52dnXR3d5s+b2xspLCw0BTJ6fV6Ll68aLEAQS6X8/Of/5x58+aZ9pHr2/HW3t4euVxuEbvfJSoqimHDhtHV1cXq1avZvHkz9fX1dHV1IRaLOX36NPb29kydOpX58+cjCAK//e1vaW1tNbsvwcHBLF++nKSkJFO03bd3WnZ29qAbRLOJbnR0NJMmTeLRRx/F0dGRoqIiLl68SG5uLsXFxXR1dSGRSGhvb+eZZ55h586dg26pli5dyvz58/H39ze1Rg+CRCLBz8+P5cuXEx0dzZQpUwblx92QyWTMmDGDZcuWERcXh6ura78X28bGhvDwcF555RXOnz/f78UzN2q1munTpzNv3jwiIiKQyWRcv36dK1euIJFIkEgkDBs2zCy2BEGgqanpvud0dnZy6dIlJk+ezKRJk9i9ezdVVVVmsf+g3Lx5k+bmZuBWSqTv/lgywpTL5fz+979n3rx5eHt7mwTt9gbJ19eX8PBw0+7Wvb29jBs3ziKiK5FISElJITk5GTc3NwwGA2VlZYjFYoKDg7l+/TqZmZlmt/tdHBwcSE1Nxd/fnwsXLnDs2DEKCwtN10CpVOLp6UlMTAwikQhXV1fS09P5/PPPBx11fpfg4GB+97vfMX36dBQKBQCXL1/m/fffZ+vWrWbZ39AsohsTE8Mrr7zC6NGjOX/+PBs3buTy5cs0Nzej1WpNG1YqlUoEQSAsLAwHB4dBi+6LL76Ir6/vfQW3vr6ey5cvU1lZia+vL48++ihwqzXXaDSMGjWK6Oho8vLyBuXL7bi5ubFgwQKWLFlCdHS0aSfgY8eOUVpaSkpKCgkJCahUKsLCwpDJZBYT3ZEjR/Lkk08ydepUVCoVx44dY+fOnZSUlNDb24u9vT1NTU0WFf270dDQQENDA8OHD8fDw2PIRff69etUV1cjEolwcnJi2LBh2NjYWFR0U1JSmDp1Kn5+fojFYrq6uti7dy8lJSU4OTnh5uZGbGwsPj4+pme6u7vb1DiYGzc3N5YtW0Z0dDQA+/fvp7m5mUmTJmEwGCgvL+f8+fMWsX07/v7+REZGYm9vT1ZWFpcvXzYJrkqlIiUlhZdeeglbW1sOHTrEgQMHeO6555g7d65ZRbdPcGfOnGnakbm2tpZt27bx9ddf09LSYhY7gxbdmJgYXn31VYKDg/n888/Zs2cPpaWldyS61Wo1jzzyCM888wyHDh363mjoQfDy8uqXUsjIyODUqVNEREQwYcIEXFxcOHz4MG+//TYdHR3Y2tri5eWFQqFg/PjxvPrqq9jZ2fHOO++YxHiwBAQE8PzzzzN79mz8/f0RBIH9+/ezefNmSktLGTVqlGm3V4PBQFNTk8Ve9KioKF566SUmTZpEQUEB27Zt4+TJk1RUVKBQKEhLSyM1NZU1a9YMbnfTAdDa2kpDQwMSicTUhRtKurq66OrqQhAEDAYDer0eS++iYmtri1KpRCwW09HRwZkzZ/j88885e/YsNjY2KBQK4uPjef755xk9ejS9vb1UV1dbLLUwa9YsRo8ejUql4uLFi2zZsgVvb28effRR9Ho9tbW1lJeXW8T27fTl1i9fvsyZM2f6aUNsbCzPPvssYWFhZGdn88knn1BeXs5TTz3FsGHDEIlEZrlvtwtuX5rlxo0brFmzhrVr15pFr/oYlOj6+/vz0ksvERQUxBdffMG2bduoqakxtVJ9aDQaEhMTmT9/PmFhYWzcuHHQo4+zZs1CpVKZfl63bh0bN26kqKgIFxcXZsyYwciRIzlw4ACXLl0ynXfp0iWkUiklJSVEREQwbdo0xo0bx9y5c9m+ffugfAoNDeWFF15g7ty5eHp6UlRUZCp1aWhoYMKECSxevBg/Pz8AtFotp06dssiW7G5ubqxYsYLExEQOHTrEpk2buHz5Mi0tLYhEIuLj43nxxRf7XcOhpKen5wfZiv5uNDU1ce3aNVOPzFKcPXuWrKwsioqKOHnyJCdPniQ3N9f0Qtva2hIXF4e7uztwKw1z4cIFDh48aHZfJkyYwIIFC/D09ARg3759nD9/nqCgIBQKBS0tLZSVlVn8mtzO+fPnKSsrw2g0IhKJCAsLY8GCBSQmJlJcXMyf/vQncnJycHBwMPWS/P39zdIwzJ8/n5SUFJPg1tTU8Mknn/Dpp59SUVEx6O+/nQGLrkgkYubMmQQEBLBlyxa+/vprbty4cUer05eLcXFxQSwW09LSwqlTpwbt+C9+8QtTF+D69et88cUXZGVl0d3dTVVVFQ0NDbi6utLQ0HDH3xoMBioqKti/fz/Tpk1DIpEQGRk5KNH18fHhlVdeYdasWbi5uXHmzBnWrVvH3r17kclkzJ07l8WLFzN8+HBTZNfc3MyBAwcsEmGNGTOGSZMmcfXqVbZs2UJ2djZ6vR6xWMyIESN46qmniIiI4MiRIxQUFJjd/vehUqlM9++HRi6Xo1arkUgkGI1Gi9mprKzk73//O3K5nIqKCm7evGmqXBGJRISEhDBlyhS8vb0xGo2UlZWxadMmGhsbzeqHvb09jz/+OMOHD0ehUJCTk8OJEycQi8UEBgYiEonIzMxk9+7dZrV7LwIDA7Gzs6OkpIT6+npEIpFpAHrmzJnodDpOnTrF6dOn6erqwt3dHScnJwwGwx0B3kBISUnhsccew8XFxfS7L7/8krVr13L9+vV+59ra2hISEoK7uztFRUUDEvwBi66fnx+zZs3i7Nmz7Nmz5w7Blcvl+Pr6EhwcjEqlwtfXF7Vazfr166msrByoWRPDhw83DUzt2rWLgoKCfnnJ6upqqqur7/n3Op2OkydPUl5ebmrtB8PEiRNJS0szlQF9+OGHnDhxgvDwcGbMmEFaWhrBwcGIxWIEQUAQBNra2rh69eqgbd+NkJAQnJ2dOXz4MFevXkWtVhMYGMjIkSOZPHkykydPpqGhgQMHDpi16/SgyOVyFAoFXV1dZheVgdDb22vx9EJvby85OTl3/Sw0NJRFixaRnJyMWCymurqaL774gmPHjpndj4CAABISEkxRXX19PUqlkuTkZGJiYgCoqqqirKzM7LbvRnh4OBqNhra2Nrq7u7GxsSE1NZUlS5Ygk8nYvn07GzZsMA3GOzs74+npyaZNm6ivrx+0/WeffZaoqCiTnly6dIl9+/b1E1R7e3tGjBjB5MmTGT16NBqNhrKyMjIyMjh69OhDie+ARTchIYGwsDDWrl3bT3DFYjE+Pj6MGTOGESNGoNPpaGpqIikpiZKSEjZv3mz2aGLfvn13jWjvR99AwenTp5kzZ86g7NvY2PDYY4/h7OyMXq8nLy8PvV7P3LlzSUlJIS4uDkdHR9rb2zEajWg0Gjo7O6moqLCY4LW2tqLX64mNjeWZZ55BpVIRFBREaGgo/v7+GAwGjhw5woEDByxi/0EQBAGtVvvQ985ciEQiRCIRRqORnp4ei4vuvdBoNEyZMoX58+fj5eVFd3c3+fn5bN682WyDN7fj5OSEk5OTaTwkLCyMlStX4ubmRkhICBKJhNGjR7Ny5UpaW1vp7u6mvLycK1eumL021tHRkfDwcCQSCd3d3aYUyxNPPIGHhwd79+5l7dq1nDt3zuT7+PHjkUqlnDhxwizpj5iYGFOara6uzlQI0EdUVBSzZs0iKSmJUaNG4ebmBtzSwJEjR6LT6YZGdBUKBWKxGCcnJzw8PJBKpfj5+REUFERAQAAajYa6ujo6OjpISkpCLBbz2WefmaVl+i43btx46NF3pVLJ2LFj8fDwGLT9YcOGERsbi0qloqenh6ioKAIDAwkODsbDwwOxWEx5eTllZWV4e3uj0Whobm4mMzPTrJNEbufUqVMcOHCAuLg4UlNT6ezspKurC6PRiEwmo6CggC+//NLs+aoHRa1WI5fLqa+vR6vVDrl9FxcXHB0dATAajXfUEg8VNjY2JCYmMnfuXFOuv7Kyki+//NJig1g3b97k7NmzODo6otFo8Pf3x9/fv985Y8eOJTg42DTgWFJSwuHDh/n6668faCrugxISEkJgYCD19fXU1dURGRnJ8uXLGT16NDk5OWzZssU0JmNnZ0diYiLz5s2juLiYwsLCQduPi4tDqVSafs7PzyczM5Pa2lo0Gg0JCQksXryYtLQ0XF1d+/2tTCYjKCiI+Pj4h6pxH7Do5uTkUF5ezvROzwKeAAAgAElEQVTp0wkODkahUKBUKrGxseHmzZtkZmZy8eJFxowZQ0BAADt27DBrEfylS5dISkoacFG7g4MDTz75JBMnThx0uZRGo0EikSASibCxsSE+Pr7f5x0dHdTU1NDS0kJYWJhp1pYlu/WFhYV88MEHREREYGdnR1NTEzKZjAULFuDs7MzRo0c5dOiQxex/H+7u7qjValOvYCiRSqXExcURGRmJIAi0tLRQVVVl0XzuvfyIj49n+fLljBs3DolEQltbG8ePHx/0oO79uH79Oh988AFtbW2EhYXh4eGBt7c3tra2wK0JGtXV1aaUlFgsJiIiguHDhyOVStmwYYPZJiX0jfeUlZWhUChITk4mMTERvV7P8ePHTZMR5HI5kZGRrFy5EhcXF958802uXbs2aPtJSUn9BpNrampob2/H39+f1NRUVqxYwbhx44BbFS+dnZ2mxhpu9exvF+0HYcCie/XqVTZs2EB8fLxpIZMLFy6YRmfb2tqIjY1l/PjxlJeXs2fPnoGauiuff/45o0aNwtHRkfj4eMrKyh74QbCzsyM5OZkxY8YAt3Jtg+niXrt2jQsXLqDX61Gr1aaSoI6ODtrb2ykrK6O+vp4RI0bg4+ODXq+nvb3dVHxtKQoLC/tFA+np6Xh7e1NSUsK+ffvQ6XQWtX8vxGIxAQEBuLi4UFdXh1wuH1LhVSgUhIeHExAQAEBLS8t98/+WIiAggGXLlpGamopCoaCzs5OzZ8+ydetWi6QV+tBqtWRlZVFbW4uPjw+hoaGsXLmSMWPG0N7ezpEjR9i5cyfOzs5MmDCBwMBARo0aRWBgIC+88AJZWVn3zE0/LMHBwSaxT01N5ZFHHkGj0bBv3z4OHDhAc3MzMpmM8PBwli5dSkJCAhkZGXzxxRdmmT7f2NjYr7ENCAhgwoQJ+Pr68sQTTxAcHAzcGvS+du0acrm8n+h2dXU9dK3wgEW3vb2djz/+mDVr1ph+d/tghK+vL9OmTcPPz481a9ZQVFQ0UFN35dSpU9TV1eHg4MAvf/lLamtrOXTo0AMJia+vL2+99RbBwcEIgkBDQ8Og5t03NDTwl7/8heHDh+Pl5YVMJqOuro7a2lpqa2spKipCoVDwi1/8gokTJ9LQ0MA333xj9obofnh6epKYmIi/vz+7du0akqL3e2Fra4ufnx+hoaHExMTQ1dVFeXk5RUVFQ5JqEIlESKVSRKJbi0BJJJLvnUL+MKjV6u9NGzk5OTF37lwSExNRqVS0trZy7tw5PvvssyFbYKa0tJTS0lIyMjKIi4sjLi6OgoICNm/ezI4dOwD47LPPSEtL4/e//z0hISE4OTn1G+U3F7a2tiQlJREQEEBFRYWpZ9zXhV+8eDFPPPEEOTk57Nq1y2zrlZw5c4bOzk7TzxMmTMDPzw87O7t+4trU1ERdXR1RUVGm33V2dnLy5MmH7sEP6kkTBOGuXTJPT08WLlzI9OnTOX36tEWmEhYWFnLixAl8fHwIDg7mueeeo66ujvPnz991soFMJsPBwcHUeoeEhAC3LtymTZvIz88fsC+CIJCVlXXPCQYikYhJkyYRHh4O3OrCfPPNN0MWXclkMqZNm8b06dOprq4mIyPDoouF3A+NRkNMTAxhYWG4uLgwdepUkpOTqa+v58MPP2T37t1D1s3vE121Wo2zs7PZvjcgIID8/Px75oiVSiVTp05l0aJFprxpVlYW//jHP8jIyBjy2YFOTk6oVCpEIhE5OTmmQSu4FRVnZ2dz6NAh0ztjCYYNG4ZUKqWrq4v9+/dz6tQp03T9OXPmMH36dAoKCvjf//1fszZKhYWFXLt2DU9PT9O07L7c+u0EBQURFBQE3BoDaGxs5Pz58/zqV7966Nyy2Re8kclkLFy4kJ/85CdcvHiRDRs2WGx0+ttvvyU5ORmVSkV6ejqlpaW0tbVRXFzcryWUyWSEhYUxbdo0EhMTmTFjBr29vbS2tnL27Fn+8z//0yL+9aFQKIiLiyMhIQGdTkdxcTG5ubkWtXk7ISEhTJs2DTc3N/7+97+zf//+IbPdh0KhwMPDgylTprBixQpCQkLIzs7m66+/RqvVsnTpUn76059SUlIyqAbwQekTXJFIxPXr1/sJzWC5XxSmUChISEjg6aefJioqiu7ubi5evMimTZv49ttvf5AV5yZMmEBoaCgikYje3t5+jZ5IJMLR0ZHQ0FDgluBYolHsa/Sys7PZv3+/afJDSkoKM2fOpKOjg7Vr15KRkWF22x9++CGenp6myo370d3dTUlJCTt27GDz5s0DqnE3q+hKJBKCgoIYOXIkNTU1bNy40aLd2C+++IKRI0fyzDPP4OTkxM9+9jNEIhEbN27k+vXryOVyVCoVbm5uLFq0iOeff970tx0dHWzevJkXXnjBYv71oVKpcHZ2Rq1Wc+XKFQ4fPmyR1ZHuhlKp5IknniAuLo7Dhw/zzTffDMlapLcjFosZO3YsTz/9NElJSbi5uXH58mU+/vhjduzYYaru+NOf/sRrr73Gyy+/bPHa4b5aabjVKPflFc3BvSKfvpmAb7zxBgkJCUgkEi5cuMA777zDzp07f7AlPpVKpWlAum9atqOjI7a2tqhUKsaPH09SUhIGg4G6ujqzViD19PT0m+Bw48YNent7mTNnDitWrCAwMJC8vDw+++wztm7dapEKky1bthAVFcWyZcv6rXtxOwaDAa1WS05ODu+9996gBjrNKrru7u789Kc/JSEhgfXr13PkyBGLl+H0LZg9ceJEZDIZL774ItOmTePs2bMEBATcUUkgCAIdHR3k5eXx0UcfWdS3Pnx8fEwlOVqtlpqamiGxKxKJSEpKIjk5mbq6Onbu3DkkUeR38fb25plnngFg69atjBo1ilOnTpmqJ3p7e8nPz+fdd9/lT3/6EwsXLmT16tVDVsI1bNgwxowZY9EAQSqV4uLiwuuvv864ceOQyWRUV1ezZs0adu3a9YNOib79Ont6epKcnExAQADjx483TWySyWQ0NDSwb98+sz5DpaWltLe3m8qxWlpaSE5OZtq0aXh6epKRkcFHH33E4cOHzWbzbrzxxhv09vayYMEC3N3dkUgkKJVK0yBvcXEx27ZtY9OmTYMfn+pr8e92AMKDHhKJRFi0aJFw5swZ4ZNPPhGCg4Mf+G9vPwbix8iRI4Xjx48LOp1O6O3tFe6FXq8Xrl27Jvz5z3+2iB/3OhYuXChcunRJMBqNwv79+4WwsDCLXo++Q6PRCH/961+F6upq4Y9//KPg4eExIP8H44dMJhPefvttYdOmTcKsWbOEP/zhD8K6deuERx555I5zvb29hffff1/IzMwU1Gq1xe6LWq0W3n33XaGzs1PQ6/VCbm6u8Nxzz1n0ekRHRwuffvqp0NjYKBiNRqG5uVn43e9+J3h5eQ34ntzLj4e9Junp6UJmZqag1+sFo9HY79Dr9UJra6tQWVkpvP/++4JGozHrM+Lh4SF89dVXQltbm2A0GgWtViu0tLQI1dXVwjvvvCMEBgYO2bMKCOHh4cKKFSuEX/7yl8LevXuF2tpaYffu3UJqaqr57o25RDcqKkrYtm2bUFhYKCxfvtysD9GD/F10dLTw7rvvCtevXxcMBoNJaHt7ewW9Xi90dHQIBw8eFKKjoy1+4757/PKXvxRaWlqE7u5uYcOGDYKdnZ3Fr4dIJBKeeOIJITc3Vzh58qQwderUAd+TwfgRGhoqbN++XZg7d64wd+5c4csvvxRefPFFQaFQ3HGujY2NkJCQIKxevVpwdXW12H0Ri8VCenq6sH//fiE3N1d48803hYCAAItejz/84Q9CQ0ODYDQahaamJuHXv/614OnpOah7ci8/BnJNVq1aJZw/f15oamoyHTdv3hROnDgh/OxnPxMWLVp0z4ZwsPcmISFBOHr0qNDU1CQ0NjYKW7ZsESZNmiTY2NgM6bNq7sPiojt37lzh7NmzwkcffTTgKNccF2zu3LnCqVOnBJ1OJ+h0OqG4uFhYt26dMGfOnB/sxr3xxhtCT0+PcOPGDeGtt94aEj+cnZ2Fjz76SGhoaBB+//vfD/oF/zE/yP8Kfnz++edCc3Oz0N3dLfzxj38U/Pz8LObHv8o1+b/ohyAI5svpenl50djYSGZm5pAtlHE3tm/fbtHZPAPh9OnTHDx4kLa2NrZu3TokNseOHUt8fDx5eXkcPnyY2traIbFr5e68++67FBYWIhKJ+PLLL4d80XYrPx6sW7D/m/qRnp7OG2+8wZEjR/joo4/uWKJuqPwwN1Y/vt+PH5MvVj/uxCq6Vj+sfvyb+fFj8sXqx53cV3StWLFixYp5efAtdK1YsWLFyqCxiq4VK1asDCFW0bVixYqVIcQqulasWLEyhFhF14oVK1aGEKvoWrFixcoQYhVdK1asWBlCrKJrxYoVK0PIfdde+LHM4rD6YfXD6seD+/Fj8sXqx51YI10rVqxYGUKsomvFyv9HLBbj6enJSy+9hFarZd++fabNCq1YMRdm35jSipV/Rfq2AH/11VeZMGECUqmUkJAQIiMjycnJ+aHds/JvhDXS/TdHJBKRmprK/v37aW5upqWlherqag4cOMBPfvITPDw8fhC/YmJi+OSTT6itrWX//v0EBwf/IH7ArW3h58+fz5/+9CcSExORyWQ0NTXx/vvvc/ny5R/Mr6Fg5syZREdHI5XeGX/NmTOH2tpaMjMzCQ8P/wG8+zfFXDtHAEJaWpqwb98+Yfv27cKkSZN+lKu+SyQSIS4uTrhw4YKwb9++f8nV5x/2nhw5ckTo6uoy7XtlMBiErq4uobm5WcjIyBCSk5OH7L74+voKM2bMENatW2faF6uoqEh4++23hQ8++EBYtGjRfbfOMfd9iYqKEt59912hpqZG0Ov1gsFgEOrr64W3335bcHNzM7sfwcHBwm9/+1vh2LFjwvXr14W6ujrTUV1dLezevVtIS0sb1PPxIL5IpVLhtddeEwoLC4W33npL8PHx6fe5o6Oj8PLLLwtdXV3CuXPnhKioqCF7RgZ7PIwfv/rVr4SSkhKhqKhIeOedd4SkpCTByclJkMvlgkKhEORyuSCRSASFQiHY2toKtra2gpOTk5CUlCR89tlnwo4dOwQXF5eHuzfmFN3Zs2cL2dnZQldXl3Dp0iXht7/97UNv3WPpGxceHi7s3btX0Gq1wu7duy3uR2RkpLB69Wph//79wvz58wW5XD5k1yM9PV04evSooNPpBIPBIBgMhn7CazAYhI6ODiE3N1f4wx/+IISHh1v8eqSmpgrffPON0NHRccfmh1qtVigtLRVeffXVIXk+pk6dKuzfv19oa2sTenp6hNbWVuHQoUPCggULBHt7e+H/j3ibzQ87Ozvhww8/FGpqaoTu7m7h+PHjwi9/+UthyZIlwnvvvSdcvXpVqK6uFv7rv/5rUM/Hg/gSHh4unDhxQujp6RE2bNhwx96BI0aMEDZs2CAYjUahurpa+M1vfmPRZ9Wcx8P48dZbbwlNTU2CwWAQtFqtUFdXJ1y4cEHYtWuXcPr0aWH37t3Cxx9/LGRnZwuVlZWmo66uTtDpdEJjY+M9t+C6172xSE7XxsaGyMhIUlNTycnJobS01BJmHhpHR0eSk5NJSkqiubmZ9957z2K2pFIpKSkpPP/880yaNAmFQkFVVRXHjh2joaHBYnb78PT0ZOHChYwcORKpVMr58+f5/PPPOXnyJK6ursTExDBixAhSUlIICAjgmWeeITk5mb/85S989dVXZvcnODiYFStWkJqaSlBQEEqlsu9lQCKRoFarAVCr1XR0dJjd/u14enry/PPPs3jxYry9vVEoFLS2tpKZmcnq1as5duwYOp3O7Ha7u7vJyclBLBZTWFjI8ePHKS0tpaurC61WS0xMDJGRkWg0GiQSCUaj0ew+ADg7O/Pmm28yfPhwxGIx58+fv2P7IDs7O9zc3ABoa2sjNzfXIr7ALb2YMWMG4eHhxMbG4ujoiEqlQqlUUldXh9FoJDAwkKqqKv75z3+adcur7OxsVq1ahUajITMzE7lcjoeHByEhIUilUhwcHAgICECj0WBnZ0dXVxdtbW3k5eVx7Ngxenp6uHTp0kPZNKvoZmRkMH36dOLj45FKpcTFxZGens7u3bvNaeZ7CQ8PZ8mSJSgUCrZs2cK5c+cACAsL4+mnn6apqYnVq1eTnZ1tUR+WLVuGi4sLJ0+eJDQ0FFtbW8Riy6fRlUolv/71r0lNTcXOzo6GhgY2bdrE+vXr0Wq1SKVSzpw5g0qlwtvbmwkTJrBs2TJGjBjByy+/zNWrV7lw4YLZ/JHL5UyfPp25c+cSGBh41/xhHyqVinHjxrF69Wqz2b8djUbDc889x/Lly/Hy8kIkElFbW8uOHTtYvXo1paWlFhFcAL1ez5YtW9i5cyednZ20t7cjlUpJTU3l2WefJSAggJMnT7Jz5056e3st4gPcej7CwsJQqVTk5+dTWFhIe3u76XOxWExISAgJCQkA6HS6QW/3dD88PT158sknSUhIwNbWFqlUilgsRiwWExoaiiAIKBQKfH190ev1ZGdnU11dbRbb+fn5dHV10dPTw5EjRzh8+DDd3d13vKeOjo64urrS1NREfX29SXwFQaC7u/uhbJpVdFtbW/tFKUqlEo1GY04T34uzszMrV65kyZIlnDt3DoVCAYCfnx8zZ85EqVTyxz/+kW3btqHVai3mR1NTEwUFBZw8eZLa2lqWLl2Km5sbQUFB1NXVWcyum5sbs2bNIjU1FRcXF0QikSmStLGxoaWlBaPRSHd3N42NjdTU1FBWVkZ+fj7z589nwYIF/PnPf+Yf//iH2Tb4lMvl+Pj44OzsjEwmQyTqXzOu1+vR6XRoNBqkUik+Pj6EhYVRVFRkFvt9eHl5sWTJEubNm4enpycikYhLly7xySefsHfvXmpqajAYDGa1+V2ampqQSCR4eHiQlpZGamoqY8eORaPRcPLkST766COys7NNvQBLEB8fj0ajwWg0smfPHgoKCvr9v0UiEQqFwtT7EASBnp4ei/ljY2ODs7OzKbK+HYlEYno+VCoVvr6+2NnZmc12V1cXgiBQU1PD6dOnycvL69fDsLW1xd3dncbGRkpLS2lqahr0tfi3KhlTKBQ8+eSTzJ49G6PRyIkTJ8jLy0OpVPLII48wa9Ysrl27xt69ey0qfAANDQ2sX7+e7u5uIiMjkcvlqNVqXF1dLWp35MiRLFu2DH9/f1NrbW9vT2JiItnZ2Xf8v41GI/X19Rw7doyuri6WLl3K2LFjmT9/vtlEV61WExAQgIODA3Drpdbr9VRVVXH27FkKCwvx8fFhxYoVaLVa9uzZw40bN8xiu48RI0awatUq0tLS8PHxoaOjw5Ry2bVrF42NjWa1dy9kMhmLFi1i9uzZREZG4unpiVqtpqamhs7OTry9vfH29qakpMRiwjtq1CjUajWHDx9m165dZr/WA+XixYvU1tbS1dVFaWkplZWVwK20jI+PDytXrkSr1VqkJ9L3fkqlUoxGI2q1milTpjBlyhQCAgIQiUQcO3aMXbt2UVlZOaiA7d9GdFUqFdOnT2fBggUolUq++OILtm3bRktLC3FxccyfP5+enh4+/vhjampqLO5PT08PFRUVwK3IwtHRkfr6eovnt0ePHk1ISAhyuZzOzk4MBgO5ubns3LmTwsLCu/6NIAjodDpyc3NZu3YtTz/9NOPHjzebTxKJBBsbGyQSCT09PVRWVpKVlcXhw4fJy8vDwcGBVatWAbei3vLyctra2sxm393dneeff545c+bg6upKR0cHW7du5fPPPyc3N3fIBBfA29ubuXPnMnXqVBQKhUlYHR0dmThxIiNGjGDatGl89dVXFsmt+/v7k5SUhJ2dHdXV1TQ0NNwRuYWHhzN27FjgluDV19f3Sz+Ym5qaGt544w2MRiMdHR0YDAaam5tNz0Bvby9jx45l9uzZ1NTUUF9fb3YfRCIRIpEItVpNcHAwU6ZM4bHHHiMqKsoU8YeHh5OUlGRKv3355ZcDGn8wq+jGxMQQFBRkzq98IFQqFdOmTePZZ58lOjqajo4ONBoNvr6+6HQ64uLiiI2NJSsri8zMTIt3Ib+LRCJBLBbT1NRk0Qg7OjqalJQU7OzsaGxsZPPmzVy7do3CwkIuXLjwvQN4fbOwVq1ahbOzs9n8am5u5syZM3R2dlJZWUlOTg45OTlcv37d1AuZNGkScKsBMGc+09nZmZ///OfMnDkTd3d36urqWL9+PRs3bqS4uBi9Xm82Ww+CVqulpKQEd3d3Wltbqa6uprKyks7OTnx8fIiOjmbMmDG4ublhMBjM1tvoY/78+YSFhSEWiykrK7tDTCUSCUFBQURFRQHQ3t5OcXGxRYSuD61Wy9GjR+/5uVQqRSaT0dvbS0VFBV1dXWazrdfruXnzJuHh4Tz++ONMmDCB0NBQhg0bhiAI7Ny5k6amJoKDg4mKimLKlCkkJCSQlJREWFgYBw4c4OTJkw+V1zWr6Pr7+981L2NJbG1tefzxx3nqqacYPXo0tra2yGQy0tLSCAwMpLy8nICAAG7evMm2bdvMGkE9KDKZjM7OTm7cuGGxiMHGxoYnn3zSNCK9Z88e1q5dS1lZGTqd7oHyUCKRCKVSaXbfdDodX3/9Nfv37+fmzZvU19ej0+lwdXVl5syZrFy5Em9vb9rb28nKyjLrSHlSUhKPPfYYHh4eaLVatm7dytdff01PTw+xsbFoNBpcXFzo7e1Fr9dTXV3N1atXLVZh0tTUxPr16/n222/p6Ojg5s2bNDY20t3djYuLC/7+/kyaNImlS5eycuVKcnNzzdo7GjVqFEqlEpFIREJCAo2NjRiNRuzt7ZFIJEgkEmJjYwkMDASgtraWY8eO0dnZaTYfHhY7OzsCAgKQyWT37K0NlI6ODj788ENeeeUV0tPTgVsDiWfOnGH37t2cOHGCtrY2vL29CQ4OZuTIkcTExBAXF8fTTz9NbGwsW7duZe3atQ+c6/2XTy94enqyZMkSRo0aRXV1NcXFxeh0OoYNG8aYMWNM5WFfffUVR48eteio8L1wd3fHwcGBpqYms7bSt+Pm5kZaWhqOjo60traya9cuioqKHuplsbOzY9q0aQiCYEqNmIvvvixeXl7MmDGD5cuXExcXR3d3N5cvX+b999/n2rVrZrEpk8mYPHkyrq6uiEQidu7cyddff010dDRJSUk4Oztjb2+Po6MjgiCg1+upq6ujsLCQzMxMjh49avbyNUEQyMvLIy8v747P2trauHbtGmVlZdjb27Nw4UIWLlzI//zP/5jNvl6vRxAExGIxkydPJjAwEJlMhq2tLRKJBJFIhK2tLba2tsCtnL+lntkHJSgoiFmzZqHX6zlz5oxZv7unp4evv/6aiRMnMmfOHMRiMdu3b2fz5s2cOXPGlHqqrKzk/PnzHDp0iJiYGKZNm0Z6ejqTJ0/G3d2d69evc+DAgQey+S8vugBXrlzh0qVL5Ofnc+XKFdrb2xkzZgyrVq1i3Lhx2NjYAFh0RPheaDQaAgMDEYvFFh2wGDt2LF5eXkgkEi5fvvzQgmtjY0N0dDRTpkzBaDSarWQsODiY+Ph4nJyc+v3e39+f5ORkoqOjgVuCc/z4cU6cOGEWuwAuLi7ExMSgUqkoLy9n//79ODg4sGTJEhITE5FKpaZKitufjeTkZFMefuvWrRYrIbsXFRUV7N27l6effpqUlBT+8pe/mE34Dh48iIODA+PGjUOlUhEWFoZEIrnn+e7u7iQlJZGVlWXRap97IZFICAsLIz4+niNHjpg90oVb6Y2vvvoKQRBob29n48aNXLx48Y7UU09PD3V1dRw5coTS0lKKi4t5/PHHGTlyJC+++CIZGRkPlGb4lxfdGzdu8Pe//52amhrTyyEWi/H19aWnpweRSIRGo2H8+PFMnTqVHTt2DGlONywsjKCgIG7evHlHAbo5iYmJQSaTIQgC+fn5D5XGkMlkhIeH8+STT+Lq6kpnZyfHjx8flD/e3t5ERUWRnp7O7Nmz+63x0FfG1veyG41Gbty4wenTpwdl87s4ODjg5eWFTCajra2NyMhIIiIiTBNG7oVKpSI+Ph61Wk1zczN79uwZsA9KpdJUlvSg9Pb20tLSwo0bN7Czs0OtVptNdHfu3ElZWZkp9x8VFWUq67SxscHX17dfitDJyYkxY8bg6ur6g4iuh4cHkZGRNDc3k5GRYbGoe+/evVy9ehWtVkt9ff19UwU9PT1cvXqVTz/9lGvXrvHyyy8zadIkxowZQ1ZW1vfasqjo9hUQW5KOjo47cl5ubm7MmDGDuLg4ioqKqKiowNPTk1mzZpGdnW1R8fsugYGB+Pn5UV5eTnNzs8XsBAUFIZFI6O3t5fr16w8c5drY2BAREcHy5ct57LHHMBgM5OTkcPDgwQH5YWtri4ODA4sWLWLBggWEh4ejUqn61ebeHl0ajUaqqqrYt28f58+fH5DNe6HX62ltbaW3t5fIyEgcHR1xdHS8o85Tp9PR0dGBnZ2dqa5bLpcTGhrK448/PmDR9ff3x8PD465R0/fR1dVFVVUVSqXSrDWyOp2O7Oxs08SgmJgY3NzcEIlEODk5sXjxYqZNm2Y6XyaTodFoLJLr/z5sbW2ZMGECKSkpFBYWcujQIYvae9gourOz07QYUGJiIikpKT+86NbW1lqkO3A/pFIpEydOZMyYMXR0dPDZZ5+xc+dOkpKSsLGxwd7efkj9UavVyGQyrl27RklJicXsTJo0CaVSSWVlJd9+++33CrxUKsXZ2Zno6GjmzJnDE088gUKhID8/n/fee2/AM5AiIiJIS0tj+fLl+Pn53bfrCmAwGLhy5QqbN282e2PY0NDAkSNHcHd3x83NDX9/f6B/KkEkEtHZ2UlDQwNyuRyFQmEqHxKLxYNaT7dv+ndeXt5Di65cLsfR0ZHq6mpaW1sH7MP3cfuylcHBwaSlpVnM1sMgl8uJjY1lwYlyygEAACAASURBVIIFODo68vnnn5vqdn9MiEQi0zPyoPfYYqLb3d1tKlUaSry9vZk/fz4RERFs3bqV3bt3U1BQQEFBwZD6AbfSHCqVio6ODm7cuPHQ0wUfhr5BkNOnT9PY2HjPAUOZTIa9vT3Dhg1jypQpTJs2jbCwMNRqNbm5ufz1r3/l4MGDA85/jxw5kpdffvmBZyJKJBJcXV0JDAwkNzfXrAOdWq2W1atXo1QqSU1NxdfX15SCuR1nZ2dTzrlvURKj0UhjYyO1tbUDti+Xy+no6HjoaymXy/H19SUsLMzsA5r3IzAwEF9f3yGzdz88PT2ZO3cuycnJ7Nq1izVr1lgkt25jY4ONjQ3d3d2mQcaHQaVSERoaisFgeOAA02KiW11dzd69e8nMzLSUiTuQSqVMnjyZ8PBwysvLOXDgwF1HiYcKGxsb/P39USqVNDU1DUkuWaPR4O7uTn19PXq93tSV71u8IzQ0lDFjxpCSksKjjz6KSCSiubmZ06dP8+6775q9LvS79Pb2YjQaaWtrQ6fT4eDgQFhYGI8//jgnT540ex1zaWkpr7/+OkVFRSxevJiwsLB+a2DodDqam5sxGAwolUrTmgitra2cOHGCXbt2Ddh2fn4+zz33HEeOHKGzs/OBXmiJREJgYCCpqal0dXVRVlY2YPsPS18q7IfG3t6ehIQExo4dS1NTE+fPn7eI4CoUCsaPH4+vry8FBQVcvXrVNE3+QZDL5URGRjJz5kx0Ot0PX71QV1dn0UUy7oabmxtz5swhODiYL/5fe2ce1dSZ/vFvFpIQEvbFsCNrVEAWqSKLokgRFDewdsDWKbae6WrPdDp2pmU62no6Zzq19tjW2lprtVJ3pVCXKiIqUNaIokBMlT2yBQiBEML7+8Ph/kqpC3ITbOd+zskfZuH9eu97n/u8z/vc59m3b8ILULPZbJiamhql5UtPTw+sra0RHR2N559/HpmZmWhsbKQKh1hZWSEmJgZLliyBt7c3taxWKBQ4cOAA9u3bZ/DzpdFo0NTUhKamJly4cAEKhQKLFy+m4u+RkZE4ePAg7eOqVCps3boVlZWVSE1NRXR0NJycnKi8z927d6OpqQlSqRTXr19HRUUFtFrtuJezhYWFeOuttxAREYGTJ09SBVLuxnDdieXLl2PlypW4fv06vvrqq3FpGAtDQ0OjVhpDQ0Po6+szaO2Fn8NmszFv3jy88MILmDZtGr777rsHNmZjxc/PD1u3boWPjw+OHTuGU6dO4ezZs6ivr3+gVamNjQ1WrlxJ1c14UKfqN5+98HO8vb1hb2+Pnp4e5Ofn4+rVqxMtCcCduKWhJ+3BgwepqmZpaWlIS0tDa2sr+Hw+RCIR5dkNe5mNjY0oKirC3r17Db4a0Wg06O/vx9mzZ/HNN9/gwoULaG9vh4uLC5ycnBAaGopJkyYhKSnJoNklubm5uHLlCpYtW4aUlBR4eHjg+vXrKC4uNkjFuaGhIWRnZ2Pjxo0ghODcuXNQqVQYHBwcZXx5PB4mT56M9PR0rF27Frdu3cKuXbuM2iqovb19xF7A0NAQmpubUVBQQFtVr/vh4OCApKQkzJw5E2fPnsXOnTtRU1NjkLGcnJzA4/HQ39+PpUuXIiEhAQcPHsSePXtQXV2NgYEBaLVaaDQaDAwMQK/Xg81mg8/nQywWY8aMGUhKSkJ3dzc+/fTTiY/pTgTx8fFwc3PDmTNnJtzLBf5/l76mpsbgMeWMjAzodDqkpqbC2tqa2ogZriim0+kwODgIhUKBnJwcHD9+3CAXtE6nozIBOBwONBoNjh49ivz8fJw6dQo3b96kvltfX4/PP/8cg4OD+Mtf/gJ3d3f4+PgY9Fi1trZi+/bt2L59u8HG+DmHDh1CfHw8Nm7ciOPHj+PkyZNULvmwVykUChEYGIjVq1fj8ccfR3l5ObZs2YKjR48aReMwFy5cQHh4OPz9/cHn89HS0oJDhw5h9+7dBq9xDNzZb3jqqacQExOD1tZWnDhx4p6PB4+XnJwcCIVCREVFIS4uDvb29khJScHChQuhVquh1+tRXFyM7OxsyGQyNDQ0wMrKCpGRkYiIiMDChQshEAhQWFg4pgwXWo2uXq+nligDAwNGW5IMIxaLqYuc7vSjh0EgEMDExARcLpdKRTIUWq0WmzZtwunTpzF79mwqm0Eul+P27duQyWSorKxEY2OjQVPXysrKsHnzZvzxj3+Em5sbvv32W3zwwQd3fcqspaUFBw8ehJubG8LCwgxehc3YNDU1YcWKFXj55ZeRnp6O9PR0KBQKXLlyhToPvr6+CA4Ohl6vx5EjR/Dhhx9OyCpNqVQiLy8PISEhCAgIQEVFBb7++muj7IuYmJhg7ty5SEhIgKOjIwoKCmh7MvFuEEJw4MABHD58GJaWloiJicEzzzyD6dOnU2VGFy9ejPj4eHR1daG1tRUsFgtSqRQDAwOoqqpCVlYWdu3aNbbH++ls1+Pj40P27t1LqqqqyNq1aw3eauOXr7/97W+kuLiYLF++3KgtP+72sre3J1988QWpqakhzz333D3bvxhSBx0vRsf4dLBYLDJp0iTy4osvkiNHjpD6+nrS3t5OWlpayLVr18hXX31Fli5dSqysrMat47dyTH5+bFJTU0llZSXR6/WkrKyMrFmzhlhYWBh9jvB4PCIUComPjw95+umnSUZGBsnIyCAbN24kR44cIb29veTmzZtk8+bNxNXV9eHODZ1GdyJP3KOog8PhkPXr15Ps7GySlpZG2Gz2//TxYHQYR8ejpOVBfufr60vy8/MJIYQMDg6SHTt2EF9f39/vuWGMLqOD0fH70vEoaXmQ3/n4+JC8vDzS0dFBNm7ceF8P8rd+blj/FfSr/Hc5bFQIIaxfvsfoYHQwOh5cx6OkhdExmnsaXQYGBgYGejF8a1oGBgYGBgrG6DIwMDAYEcboMjAwMBgRxugyMDAwGBHG6DIwMDAYEcboMjAwMBgRxugyMDAwGJF7Frx5VBKKGR2MDkbHg+t4lLQwOkbDeLoMDAwMRuR3VU/3UWTSpElYs2YNuru7sW3btomWw8BwT7hcLmbOnImVK1eioaEBH3zwwZibajLcG8bTNTBSqRRSqdQoLXsYGMYDn8/HypUr8eGHH+LZZ59FYmIiZs6cOdGyfncYzNN1dnaGp6cngDuV8T09PeHj4wOhUAgPDw+EhIRgaGgI1dXVmDVrlqFkTDgBAQGQSCQ4e/bsREsBAFhbWyMtLQ3p6ekA7nR6+Oc//2n0LgUM92fy5Ml477330NraihdffPGBGyY+LC4uLpg3bx78/f3R1dWFsrKyCemiPdFwOBzMmjULq1evxrx586DVarFz505s27YNfX194/77tBtdNpuNmJgYvPzyy4iIiABwp20Nh8Oh2oRzOBxwuVwMDQ3BzMyMbgmPFC4uLhAIBOjo6JhQHUKhEJGRkUhNTUVMTAzs7e2h0WjQ0dGBDRs2oK6uDmVlZQbVYG5ujueeew6rV6+Gvb09Ojs7kZubi6+++sogPcqG+fLLLxESEoI///nPOHv2LExMTGBtbU11An4UsbKyQlxcHIKCgvDmm28a3OCampoiLi4O4eHhYLPZKC8vx759+9De3m7Qce/GihUrsGbNGly9ehVbt25FQ0OD0cZevnw50tPT0dnZibfeegsdHR3o7++n2m+NG7rr6Zqbm5PNmzeT/v5+MjQ0dN9Xa2vrb6YW5lh+z+VySWJiIjl9+jR55513xtwRgC4dtra2ZP369SQ/P580NTWR3t5eMjg4SHp6esixY8dIeno6qampIR9//DHhcrkG0zFt2jTy3nvvkZqaGjIwMED0ej3R6XSkq6uL5OXlkaVLlxrkeKxZs4bcunWLlJWVkb/+9a/k8OHDRKFQkMbGRiKXy8nevXtJXFwcEQgERj0v93uFhoYSmUxGdu/eTUxNTcekY6xaeDweeemll0hVVRXRarUkMzOTTJ06lZiYmEzYMVm6dCkpKioiWVlZZNq0aUbTkZycTPLy8khGRgbx9PQkAoGA8Hg8wuPxxtz55W7nhnZP18LCAra2tlQMs7OzE7W1tVTfep1Oh/r6ely+fJlq/PZ7xMbGBsuWLaNafKtUqgnRERkZiRUrViAkJAQmJibU+0qlErm5ucjPz6ca7Tk4OBik62tYWBhef/11REVFwdzcnOpMzGazYWZmhuDgYLz66qsghNAe5nj66achkUjwxRdfICoqCpGRkVAqleju7oZEIsGSJUswbdo07Nq1C59++ikty8fxIhKJ4OfnBx6Ph3379hlcU3BwMOLj4+Hp6Ynbt2+jtLQUCoXC6D0Of05nZydUKhW8vLwwc+ZMo/RpA4CoqCjIZDIcO3YMP/3006iW9HRAu9H18/ODp6cnZDIZ9uzZgwsXLqCzs5Nqqz3ctFKj0YAQYpQuo8ZGJBJhxYoV8PT0xJ49e3Djxo3hu63RcXR0hJ2dHWVwb926haysLGRlZeHq1avg8XjQaDQwNTUFl2uYEP+iRYswc+ZMWFlZUe9pNBrodDqIxWKYmpoiODgYr7/+OrRaLU6ePEnLZF+wYAEmT54MrVaLGTNmQCgU4s0338S5c+fA4XDg4+ODlStXYsGCBVi/fj3c3NywadMmtLW1jXvs8eDu7o6UlBQMDAwYpQV7WloawsPDoVKpsHv3buzbtw/9/f0GHzckJAQLFizAoUOHIJfLR5xzhUKBmpoaREdHY9asWdi9e7fBsygee+wxBAcHY/v27aiqqjKIwQVoNroikQjh4eGYOnUqjh49il27dkGlUhk8HvWoYWVlhTlz5qChoQHl5eXUDWciqKqqQmVlJdrb21FeXo6zZ8+iqKgIt2/fxsDAAFxdXQ2uYcqUKbC1taX+XVdXh6+//hr19fV45plnMGPGDPD5fAQGBuLVV19FUVERLTHwyMhIiEQiKJVKBAQEYMuWLfj222+hVCrBYrFQU1OD+vp6dHR0ICkpCUlJSVAqldiyZYtBvUsLCwukpKSgtrYWBQUF0Gq1Iz63tbXF1KlTUVdXh+7uboPpAO7MVWdnZ4hEItTU1KCsrAxNTU1GcRICAgLw5JNPws/PD++++y6qq6upzzo6OtDY2AidTgd3d3dIpVLIZDKD6pFIJDA3N4dGozGogac1ZSwiIgJz5syBjY0NHB0d4eXlRV/w+TfEnDlz4ObmhkuXLuHatWvU++bm5nB1dYVYLDaaltLSUrz77rtYv349/vWvf+HEiROor6+nLnQbGxvMmDHDYOMvWrQIPj4+MDExAYvFgkajweHDh/HZZ5/h4MGDeOONN/Dll1+CxWJBIBDA19cX/v7+tIxtYWEBDocDNpuNoqIinD9/Hrdv38bQ0BD0ej16enpQWlqK48ePo7a2Fk5OTli2bBliY2NpGf9uBAcHIzExEd7e3iNCPsMMDg6is7MTCoXC4KGF2NhYeHh4oLOzE9nZ2SgoKDCYh/dLent7YWFhgfj4eDg6Oo74TKPRoKWlBZ2dneDxeEa5ZuRyOXp6emBra2vQDX7aPF2BQICYmBgEBgaCzWbjsccew4YNG7Bnzx4cP37cYHcOKysrODk5wc3NDQKBAGZmZnB3dx+xlP0larUalZWVkMvlqKyspDV25e7ujsWLF8Pa2hpKpRK9vb3gcDiYPXs2Vq1aBYlEgqamJpw5cwbHjh0zuBfc3d2N8vLyn28oUFhYWCA4OBi+vr5oaWmhfWyxWIzk5GS4urpSY5eUlODUqVNU7LigoABmZmaIiIigUgojIyORl5dHm47y8nJs374d1dXVo1ZdfX19+PHHH5GTk4MpU6bAz88PqampuHTpkkHCDM7Ozli1ahVMTExQX18/ahkvFArh4uICLpeLsrIygxpAOzs7LFy4EK6urpDL5SgrKzPIPLgbMpkMbW1t8Pf3R0BAAGQyGbXCGRoaglarhU6ng0AgMIrRvXnzJuRyOWJiYnDlyhUUFBQYZJVOm9F1dnaGt7c3LCwsANxZIs2bNw92dnZwcHBAZmYmbeknkyZNQlxcHBwdHeHt7Y1JkybBwsICer0eWq0Wvb29I4z88HsajQbd3d3w8PCAWCxGQkICrl27hl27dtE22WxsbODi4oLm5ma0tbWBzWYjNDQU6enpCAsLw+3btzFp0iR4eHigtbUV58+fp2Xce3G3C5fP58POzg5CodAg45qamsLDw4PyGnQ6Haqrq3Hz5k3qO/39/WhpaYFSqYSPjw+4XC4cHBxo1XHz5k1cv34darX6Vz9XKpXIyclBQEAAli9fjlmzZuHJJ5/E1q1badXBYrEQGRmJWbNmIT8/HwqFYtRN197eHqGhoeDz+WhqaqJ1/F8ynDsvFotx48YNNDQ0GM3LBe6cl4sXL8Ld3R3z589Hfn7+r4aV7OzsMG3aNJw4ccKgYY/e3l58+eWXeOWVV5Ceng6BQIDCwsK7zpuHhTaj293djWvXrkEikcDMzAwSiQTW1tbUBFKpVMjMzKTlzvHMM89gxYoVGBgYQFNTExoaGiCTydDc3AyVSoXW1tYRG3QDAwPo7++nNvCcnZ3B5/Ph5+eHlStXwsbGBq+99tq4dQF3LiwWi4X29nao1WpMmTIFqamp8PHxwfHjx3HmzBn4+vpixYoViI+PN4rRvRtmZmZwdnYGIQT9/f2039VZLBa4XC4VYmpvb8e1a9dG3eCGhoYMerE7OjqCz+ff9XO9Xo/q6mocPHiQepglOTkZOTk5kMvltOkYPu98Ph+lpaVQKpWjvmNpaQkPDw/09fWhtraWtrF/DQsLC5iamoLFYqGrq4vKMBIKhXB0dKTyy3U63YjNcLrQarX44YcfEBcXh1mzZsHJyYlalQF3btJarRYeHh4IDg6GSCRCT08PrRp+DiEEFy9ehKWlJebPn4/U1FTMmjULJSUluHz5MpRKJS3HgDaje/v2bWRmZqKgoABCoZC6Y8+fPx++vr544okncPXqVVp2Y6Ojo+Hh4YFPPvkEp06dQn19PXp6eqgJcj+GPa2KigpERUXhiSeeoM3oDnPr1i10dnZi7ty5WLBgAaqrq7F//36UlJSgqqoKNjY2mDlzJlxcXFBfX0/r2MOYm5tTKVodHR0j7thmZmYICAhAeHg4ent7UVFRQXtam7u7+wgvWi6Xo6qqyuCbQ8O0t7dDr9fD2dkZpqam9/xuT08Pzp8/jx07duCll16CVCpFcnIyNm/eTJueGTNmYPr06ZDJZCgvLx9lQNhsNuzt7ak58dNPP9E29r0YNnKTJk2Cs7MzgoKCIJVKIRQK0dnZCbVajZKSEpw5c4b2kMtwSMHLywuLFi2CWq2mYtl6vR7t7e2QSqVwd3eHp6enwbM5BgYG8P333+PWrVuIjIxEcHAwAgMDUVdXh6KiIuTn5497BUJr9sLly5dx+fJl6t++vr5QKpVITk6Gn58fpFIpLQetoqICQUFBaGhoQElJyUPf/bhcLths9q96HOOlrq4OHR0dcHNzg729PX744QfcunULANDW1oaKigosXLjQIEZXIBDAxsYGCQkJkEql4HK5aGtrg0KhwI0bN9DY2Ag3Nzc89dRT8PLyQk1NDXbt2kX7MiosLAyWlpZgsVhQq9X48ccfoVAo7vkbOjde5XI5tFrtCG/7XjQ1NeHUqVNISUlBYGAgkpKSsGvXLjQ3N49bC5fLRWRkJMzMzHD69GkoFIpR3r1EIkF4eDisrKyQnZ1NeZ7GwNvbG6tWrYK7uzv8/f1hZ2cHNpsNQgi0Wi1kMhl4PB6+++47Wp/ia2lpwaFDh2Bqaor4+Hi4u7ujtrYWfX19sLe3p7JehEIhJBKJUVLo+vv7UVZWhpqaGlRUVCA8PJxapfj6+uLw4cPjyhs2aJWxmpoafPHFFxCJRLTuCGdmZiIhIQHLly/HmTNnUFNTM+blKZ/Px5QpU+Dl5YWdO3fSpu2XDC+JFAoF5eENL+v1er1B8pRjY2MRFhaG5ORkTJ48GRwOB4QQNDU1QSaToba2Fo6OjoiPj4dKpcLJkycNUhvC2dkZAoEAwB2DVlJSMspLYLPZEAgE1Pf0ej1tHndrayv0ej11c30QOjs7UVxcjODgYGpZm52dPW4tdnZ2mDFjBlQqFS5fvjzq/8jn8xEWFoalS5dCq9Xi6tWrRs3tDgkJQXBwMAghkMvlOHPmDNhsNszNzeHt7Q1fX1+sWbMGarUaR44coW3c/v5+7NixA52dnVi7di0iIiIwf/78UTfJ4fIBxkStViMvLw+FhYXw8vJCbGwsgoKC8Ic//AFZWVkoLCx8qLDYmI0uj8fDtGnTIBAIwGKxUFVVBZVK9asThBCCtrY21NbW0mp0y8rKcPr0aSxfvhwSiQQKhWJM2RFsNhsSiQSLFi1CSUkJvvnmG9q0DcPj8ah0oMrKShQVFUGr1UIoFCIkJATx8fFoaWmh9QkwFosFqVSK9evXY/r06ejo6EBVVRUIIeBwOLC1tUVcXBwWLlwI4E5M7fLly9i/f79BL3BCCFQqFTo6OkblpJqamsLFxQVOTk4YGhpCe3s7bXUY2traoNPpqM1CNpt934ukubkZ+/fvx7Jly2BmZoaYmBhajK6npyfEYjGam5vh5OSE6dOnj/jczs4OiYmJmDp1KoqKiowSWmhubkZnZyf0ej212SmTybBt2zYcP34cAODk5ITHH38ca9aswYwZM5CSkkKr0QUAlUqFvXv3or+/H/PmzYOjoyN4PB5YLBYcHBzg6ekJU1NTSCQSWsd9UIZvgnK5HP7+/njyySexbt06NDU1jdgUflDGbHSjo6Pxpz/9CdbW1lCpVMjIyEBXV9eoi5bH48HGxgZhYWGYO3cuCCG0btQcPHgQAwMDUCqVY77bDBcTsbGxwbvvvmuQoh4eHh6YPHkyRCIRrly5goqKCgiFQgQHByMlJQUODg7YunUrrTEyMzMzvPjii7CwsMDp06dx8eJFtLS0QK/XQyAQ4LHHHkNCQgLc3d0BAF1dXSgtLTV40jlwZ1fe0dERQqFwhHcvFArh5uYGJycn9Pb2oqysjLbNxevXr6O3txdSqRRWVlYPZHR1Oh1u3ryJS5cuYdGiRYiPj8drr71Gy0Zfd3c33NzcsG7dulFxbYFAgMmTJ0OtVuP69esj8rsNxeXLl3Hx4kVIpVI4ODhQTlJ3dzcEAgF6enpQV1eHwsJCJCYmwtPTE15eXgbR0tfXhz179iAnJwf29vbUTXLevHl4+eWXYWtri5CQEOzcuXPCHrbSarUoKysDj8fDxo0bERAQYByju2nTJgQFBYHL5eLMmTMYGBgYMSG5XC6sra0xdepUREdH4/HHH0dQUBBkMhmtXt2FCxdw4cKFMf/OzMwMs2fPRkpKCjZt2jTiKRg6UKvV6OzsRGBgIJKTk+Hr64vi4mJYW1vD19cXycnJ8PT0xP79+3HgwAFax7a3t0dsbCx2796N7du3j4hV8/l8dHR0YMqUKXB3dwchBCwWC0KhEBwOx+D5wm5ubvDy8oJYLKaMLpfLhUQioUqAtrW14ZtvvqFth3q4ipper4eFhQV4PN4D/T/b2tpw4MABREVFwdLSEnZ2duOO+5eWluLAgQPUw0NisRiDg4PUtaNSqdDS0oL+/n4UFhYa7fH4o0ePIioqCvb29iCEICYmBnZ2dsjKysJPP/0EgUCAgIAABAYGQq1WGzyNraOjY0TamI2NDVpaWuDv7w9vb284OTmhrq7OoBruBZfLhUgkAoCHTjMds9H18PCAiYkJCCFoaGiAUCiEu7s7uFwuuFwubG1tER0djeTkZAQGBoIQgtu3b+PcuXMTXtyGxWLBy8sLa9euRVNTEy5dukT7GHK5HN9//z3S09OxYMECWFpaorm5GSkpKZg7dy5EIhF2796NzMxM2jdKhEIh1Go1fvjhhxEeNIfDgb+/P9LS0jB79mxotVo0NjaipaUFTk5OcHV1ve8G18Og0Wig1+vBYrFACIGlpSU1YblcLlxdXbF06VKkpKRQaXbHjh2jVUNubi6kUimioqJQXFwMhUJx31CKWq3GhQsXcO3aNUybNg3BwcH4/vvvx6Wjr68P//73v7Ft2zbY29sDuOP5DofFhEIhVq9ejaSkJKPWfigtLUVZWRn8/Pyo1cDwwwrAndCQTqejNq2//fZbo2kD7sTYGxoaEBQUBFdXV4SHh0+Y0WWxWHB1dUVsbCwGBwcfOqVvzEb35wFua2trrFu3DhKJBEKhkEqGt7OzA4vFQl9fH9rb23H+/Hls3759wtt+iEQi+Pv7w8HBAf/5z39GxRfpYHBwEPv374erqysWL14MGxsbxMXFISQkBOXl5fj444+Rm5trkJ1pgUAAuVwONpsNExMTmJqawsrKCg4ODli3bh0SEhIwODiIiooKfPbZZzh//jwWLFiAyMhI1NfX015VqrS0FO3t7XBwcACLxUJiYiJqa2uRk5MDW1tbREVFISEhAebm5tBqtWhtbaV1fAA4fPgwkpOTsWTJEhQVFaGtrQ1dXV33/d3wPB8aGqKt+ItOp4NOp/vVlDlbW1uq9rKx2bJlCzQaDRYvXkxlLQxvPmo0Gsjlcnz++efIysoyev3h6upq5ObmIjExERwO55751oaExWLBzs4OMTExCA0Nxf79+x/6WIzZ6KpUKlhaWoLD4WDRokUjPhtOcler1dBoNCguLsaOHTuQn58/4cWiBQIBQkJCEBcXh9zcXBQUFBhsrObmZrzzzjsoKChAWloaBAIBDh06hJycHIMWY66oqEB7ezuef/55fP7555g8eTJVUMTMzAxDQ0PIzc3FRx99hHPnzkGv1yM7OxtTpkyBo6MjldJGF3l5eSguLoazszPMzc3h6OiIjIwMvP766zAxMYGZmRn4fD40Gg2Kiorw3nvv0To+cOeY1NTUIDY2Fm+88Qb6+/uRnZ19z/Q4NpsNsVhMPRVmjDQlR0dHKsxibG7duoWNGzciOzsbQUFBEIvFsLGxgUgkyfib5QAAA1FJREFUQklJCfLy8n41xc0YdHd3o66uDl1dXRCJRJBKpeDz+QZxmIA7q0ITExPKlg0bektLSyQkJGD16tU4ffr0+DKexlrE/NlnnyUymYy0traSnp4e0t3dTTo6OkhTUxO5evUqKSwsJO+//z6Jjo4mdnZ2hM1mT3hxaFNTU7JkyRJy9uxZ8sknnxCxWDwhOh7mNVYdc+fOJVlZWUQul5Pe3l6i1WpJT08PKS8vJxs3biSBgYEjijGzWCxiZmZG7O3tDXI8EhMTSXFxMdFqtWRwcJB66fV6Mjg4SLq7u8nJkyfJ3LlzDTY/QkJCSH5+PhkYGCAlJSXkiSeeILa2tr9amJrNZhN3d3eyZcsWolQqyQsvvGCU+ZGUlERKS0tJUVERSUxMHNf8+K3M1Qd9hYaGkhMnThC9Xk8qKytJdHS0wXS4urqS+Ph4MnfuXBIWFkbi4+PJhg0byOnTp0llZSV5++23CZ/PH9+5GavRBUCcnZ1JREQEeeedd8grr7xC4uPjyfTp04mFhcUjd+K4XC5ZuXIlKSkpIR999NE9q/AbYwIZ+niIRCLi5+dHzp07Rzo6Okh+fj75xz/+QUJDQ8fcHYGu47FmzRpSVlZG+vr6iE6nI/39/aSnp4e0tLSQ/fv33/ciokPHqlWrSEFBAenp6SGNjY3k0KFDJCUlhTg5ORFra2tibm5OxGIxmTp1Knn//ffJzZs3yY4dO4iNjY1R5oeXlxf5+9//Tt5+++0xXUf/C0ZXIpGQt956i+j1elJXV0fWrl1rMB3Ozs4kIyOD5OTkkJqaGlJbW0sOHTpEXnrpJeLn5zfu4/HQRve3dOI4HA4JDQ0lS5cuHZPR+b0ej4nSMWfOHJKZmUlu3LhB9u3bR1544QUSGxv7wF4DHTrEYjHZsGEDqa6uJhqNhno1NzeT8+fPk1OnThGlUklaWlrI9u3bibW19W/yvDxKWuj62xEREaS0tJTs27fvvsbvUT4ehBCw/ivoV/nv0suoEEJGPa/J6GB0MDoeXMejpIXRMRpai5gzMDAwMNwbxugyMDAwGJF7hhcYGBgYGOiF8XQZGBgYjAhjdBkYGBiMCGN0GRgYGIwIY3QZGBgYjAhjdBkYGBiMCGN0GRgYGIzI/wFKi+Dd+6FA1gAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 60 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X1G0DgUkehD9",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "# Deep artificial neural network implementation using *pytorch* \n",
        "\n",
        "We are building a feed forward neural network with two hidden layers.\n",
        "\n",
        "\n",
        "Let $\\mathbf{p}^{(n)} = \\begin{bmatrix} p_1^{(n)} & p_2^{(n)} & \\ldots & p_N^{(n)} \\end{bmatrix}^T$ are pixel values of the input image of numbers. Here input image is 28x28 so we have 784 pixel values as an input per image in MNIST dataset.  The network we will use is described by the following set of equations:\n",
        "\\begin{align}\n",
        "    \\mathbf{A}^{(0)} &= \\mathbf{W}^{in} \\mathbf{p}^{(n)} + \\mathbf{b}^{in}, && [\\mathbf{W}^{in}: M \\times N], M = 784, N = 256\\\\\n",
        "    \\mathbf{A}^{(1)} &= \\mathbf{W}^{(h_1)} \\mathbf{A}^{(0)} + \\mathbf{b}^{(h_1)},  && [\\mathbf{W}^{(h_1)}: N \\times R], N = 256, R = 128 \\\\\n",
        "    \\mathbf{A}^{(2)} &= \\mathbf{W}^{(h_2)} \\mathbf{A}^{(1)} + \\mathbf{b}^{(h_2)},  && [\\mathbf{W}^{(h_2)}: R \\times S], R = 128, S = 64 \\\\\n",
        "   y^{(n)} &= \\mathbf{W}^{out} \\mathbf{A}^{(2)} + \\mathbf{b}^{out},  && [\\mathbf{W}^{out}: 10 \\times S], \n",
        "\\end{align}\n",
        "where $y^{(n)}$ denotes the output of the network \n",
        "\n",
        "\n",
        "The $M$-dimensional vectors $\\mathbf{A}^{(1)}$ and $\\mathbf{A}^{(2)}$    denotes the activations of the **hidden layers** of the network. \n",
        "\n",
        "\n",
        "Parameters of the network are consists of the weights and biases $\\mathbf{W}^{in}, \\mathbf{b}^{in}, \\mathbf{W}^{(h_1)}, \\mathbf{b}^{(h_1)}, \\mathbf{W}^{(h_2)}, \\mathbf{b}^{(h_2)}, \\mathbf{W}^{out}, \\mathbf{b}^{out}$.\n",
        "\n",
        " Parameters of the network will be optimized by gradient descent.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0WoKmKRdMoCR",
        "colab_type": "text"
      },
      "source": [
        "![picture](https://i-am-mel-dev.github.io/images/ann.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "REsJDTsVHoia",
        "colab_type": "text"
      },
      "source": [
        "If we do not add any activation functions,  that the deep network we constructed above comprises solely **linear** operations on each layer: each layer is just a weighted sum of the elements in the previous layer. It turns out that linear hidden layers like this are not particularly useful, since a sequence of linear transformations is actually essentially the same as a single linear transformation. \n",
        "In other words, the output is still just a weighted sum of elements in the input -- the hidden layer has done nothing to change this.\n",
        "\n",
        "To extend the set of computable input/output transformations to more than just weighted sums, we'll incorporate a **non-linear activation function** in the hidden units. This is done by simply modifying the equation for the hidden layer activations to be\n",
        "\\begin{equation}\n",
        "    \\mathbf{h}^{(n)} = \\phi(\\mathbf{W}^{in} \\mathbf{r}^{(n)} + \\mathbf{b}^{in})\n",
        "\\end{equation}\n",
        "where $\\phi$ is referred to as the activation function. Using a non-linear activation function will ensure that the hidden layer performs a non-linear transformation of the input, which will make our network much more powerful. In practice, deep networks *always* use non-linear activation functions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cQLeBn6Rqkzj",
        "colab_type": "text"
      },
      "source": [
        "**Re**ctified **L**inear **U**nits\", or **ReLU**'s\n",
        "\\begin{equation}\n",
        "  \\phi(x) = \n",
        "  \\begin{cases}\n",
        "    x & \\text{if } x > 0 \\\\\n",
        "    0 & \\text{else}\n",
        "  \\end{cases}\n",
        "\\end{equation}\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "92TvpMeG1wTm",
        "colab_type": "text"
      },
      "source": [
        "## There are three ways of implementing neural networks in PyTorch:\n",
        "\n",
        "    nn.Module\n",
        "    nn.Sequential\n",
        "    nn.ModuleList"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Of52TaLk16mp",
        "colab_type": "text"
      },
      "source": [
        "### `nn.Module`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zkrAn0yyoxZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self, n_inputs, n_unit1, n_unit2, n_unit3, n_output):\n",
        "        super().__init__()\n",
        "        \n",
        "        # Inputs to 1st hidden layer linear transformation \n",
        "        self.hidden1 = nn.Linear(n_inputs, n_unit1) # 1 layer:- input numbers, hidden unit numbers \n",
        "        \n",
        "        self.relu = nn.ReLU() # Defining Rectified linear unit as activation\n",
        "\n",
        "        self.hidden2 =  nn.Linear(n_unit1, n_unit2) #2 layer:- hidden layer \n",
        "        \n",
        "        self.tanh = nn.Tanh()\n",
        "        self.hidden3 =  nn.Linear(n_unit2, n_unit3) #3 layer:- hidden layer \n",
        "        \n",
        "        self.tanh2 = nn.Tanh()\n",
        "\n",
        "        # Output layer \n",
        "        self.output = nn.Linear(n_unit3, n_output) # output layer\n",
        "        self.softmax = nn.LogSoftmax(dim=1) # softmax\n",
        "\n",
        "        \n",
        "    def forward(self, x):\n",
        "        x = self.hidden1(x) \n",
        "        x = self.relu(x)\n",
        "        x = self.hidden2(x)\n",
        "        x = self.tanh2(x)\n",
        "        x = self.hidden3(x)\n",
        "        x = self.tanh2(x)\n",
        "        x = self.output(x)\n",
        "        x = self.softmax(x)\n",
        "        \n",
        "        return x"
      ],
      "execution_count": 130,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yOlCmZZKy2c3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "1e0f8112-2583-48d2-8e3e-7460611bce4e"
      },
      "source": [
        "model = NeuralNetwork(n_inputs=784, n_unit1=256, n_unit2=128, n_unit3= 64, n_output=10)\n",
        "model"
      ],
      "execution_count": 131,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NeuralNetwork(\n",
              "  (hidden1): Linear(in_features=784, out_features=256, bias=True)\n",
              "  (relu): ReLU()\n",
              "  (hidden2): Linear(in_features=256, out_features=128, bias=True)\n",
              "  (tanh): Tanh()\n",
              "  (hidden3): Linear(in_features=128, out_features=64, bias=True)\n",
              "  (tanh2): Tanh()\n",
              "  (output): Linear(in_features=64, out_features=10, bias=True)\n",
              "  (softmax): LogSoftmax(dim=1)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 131
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGrLkNy72dk7",
        "colab_type": "text"
      },
      "source": [
        "### `nn.ModuleList`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "44PjCD9q2jO6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class NeuralNetwork(nn.Module):\n",
        "    def __init__(self, n_inputs, n_unit1, n_unit2, n_unit3, n_output):\n",
        "        super().__init__()\n",
        "        l1 = nn.Linear(n_inputs, n_unit1)\n",
        "        a1 = nn.ReLU()\n",
        "        l2 = nn.Linear(n_unit1, n_unit2)\n",
        "        a2 = nn.Tanh()\n",
        "        l3 = nn.Linear(n_unit2, n_unit3)\n",
        "        a3 = nn.Tanh()\n",
        "        l4 = nn.Linear(n_unit3, n_output)\n",
        "        s = nn.LogSoftmax(dim=1)\n",
        "        \n",
        "        l = [l1, a1, l2, a2, l3, a3, l4, s]\n",
        "        self.module_list = nn.ModuleList(l)\n",
        "\n",
        "    def forward(self, x):\n",
        "        for f in self.module_list:\n",
        "            x = f(x)\n",
        "        return x"
      ],
      "execution_count": 132,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3jZ-3Jmi25py",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        },
        "outputId": "b3f7170d-05ce-4422-9563-361cf4124f1a"
      },
      "source": [
        "model = NeuralNetwork(n_inputs=784, n_unit1=256, n_unit2= 128, n_unit3 = 64, n_output=10)\n",
        "model"
      ],
      "execution_count": 133,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "NeuralNetwork(\n",
              "  (module_list): ModuleList(\n",
              "    (0): Linear(in_features=784, out_features=256, bias=True)\n",
              "    (1): ReLU()\n",
              "    (2): Linear(in_features=256, out_features=128, bias=True)\n",
              "    (3): Tanh()\n",
              "    (4): Linear(in_features=128, out_features=64, bias=True)\n",
              "    (5): Tanh()\n",
              "    (6): Linear(in_features=64, out_features=10, bias=True)\n",
              "    (7): LogSoftmax(dim=1)\n",
              "  )\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 133
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXkQ0flw2UGy",
        "colab_type": "text"
      },
      "source": [
        "### `nn.Sequential`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lr9a_HotkOKX",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "c66bf95f-5141-4f41-ee83-c2b6ff215b44"
      },
      "source": [
        "# Model creation with neural net Sequential model\n",
        "model=nn.Sequential(nn.Linear(784,256), # 1 Layer:- 784 input 256 output\n",
        "                    nn.ReLU(),          \n",
        "                    nn.Linear(256,128),  # 2 Layer:- 256 Input and 128 output\n",
        "                    nn.Tanh(),\n",
        "                    nn.Linear(128,64),  # 3 Layer:- 128 Input and 64 output\n",
        "                    nn.Tanh(),           \n",
        "                    nn.Linear(64,10),   # 3 Layer:- 64 Input and 10 output\n",
        "                    nn.LogSoftmax(dim=1) # Defining the softmax to find the probablities for the last output \n",
        "                  ) \n",
        "\n",
        "print(model)"
      ],
      "execution_count": 134,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sequential(\n",
            "  (0): Linear(in_features=784, out_features=256, bias=True)\n",
            "  (1): ReLU()\n",
            "  (2): Linear(in_features=256, out_features=128, bias=True)\n",
            "  (3): Tanh()\n",
            "  (4): Linear(in_features=128, out_features=64, bias=True)\n",
            "  (5): Tanh()\n",
            "  (6): Linear(in_features=64, out_features=10, bias=True)\n",
            "  (7): LogSoftmax(dim=1)\n",
            ")\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9L9qE7ysKkaT",
        "colab_type": "text"
      },
      "source": [
        "We are using negative log likelihood loss and log softmax for classification task. \n",
        "\n",
        "![picture](https://miro.medium.com/max/700/1*z5zL8kz7TPW1y3okAjuvBA.jpeg)\n",
        "\n",
        "These functions are also implemented in  `torch.nn.CrossEntropyLoss`"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uWDnz2cNk3Kj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# defining the negative log-likelihood loss for calculating loss\n",
        "criterion = nn.NLLLoss() "
      ],
      "execution_count": 135,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4auHibsfk8Vg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "images, labels = next(iter(train_loader))\n",
        "images = images.view(images.shape[0], -1)\n",
        "\n",
        "logps = model(images) #log probabilities\n",
        "loss = criterion(logps, labels) #calculate the NLL-loss"
      ],
      "execution_count": 136,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yU7Rn815LElW",
        "colab_type": "text"
      },
      "source": [
        "We are using stochastic gradient descent for optimization.\n",
        "Mathematically, Gradient Descent is a convex function whose output is the partial derivative of a set of parameters of its inputs. The greater the gradient, the steeper the slope.\n",
        "\n",
        "Starting from an initial value, Gradient Descent is run iteratively to find the optimal values of the parameters to find the minimum possible value of the given cost function.\n",
        "\n",
        "The word ‘stochastic‘ means a system or a process that is linked with a random probability. Hence, in Stochastic Gradient Descent, a few samples are selected randomly instead of the whole data set for each iteration. \n",
        "\n",
        "![picture](https://www.oreilly.com/library/view/statistics-for-machine/9781788295758/assets/0e39e950-dd75-4c2f-a4ea-aa1ccd1eb8ca.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ug25i3clChr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "c9fd537f-060b-4278-e0b2-623b9e2a88cb"
      },
      "source": [
        "# defining the optimiser with stochastic gradient descent and default parameters\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n",
        "\n",
        "print('Initial weights - ', model[0].weight)\n",
        "\n",
        "images, labels = next(iter(train_loader))\n",
        "images.resize_(64, 784)\n",
        "\n",
        "# Clear the gradients, do this because gradients are accumulated\n",
        "optimizer.zero_grad()\n",
        "\n",
        "# Forward pass\n",
        "output = model(images)\n",
        "loss = criterion(output, labels)\n",
        "# the backward pass and update weights\n",
        "loss.backward()\n",
        "print('Gradient -', model[0].weight.grad)"
      ],
      "execution_count": 137,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Initial weights -  Parameter containing:\n",
            "tensor([[-0.0240,  0.0208,  0.0216,  ...,  0.0251,  0.0122, -0.0042],\n",
            "        [-0.0312, -0.0232,  0.0135,  ..., -0.0259,  0.0095,  0.0138],\n",
            "        [-0.0053,  0.0070,  0.0048,  ...,  0.0130,  0.0272, -0.0255],\n",
            "        ...,\n",
            "        [ 0.0095,  0.0114, -0.0169,  ...,  0.0269,  0.0131, -0.0071],\n",
            "        [-0.0088, -0.0345, -0.0297,  ..., -0.0250,  0.0278,  0.0110],\n",
            "        [-0.0046,  0.0034,  0.0346,  ..., -0.0338,  0.0230, -0.0308]],\n",
            "       requires_grad=True)\n",
            "Gradient - tensor([[-6.9222e-05, -6.9222e-05, -6.9222e-05,  ..., -6.9222e-05,\n",
            "         -6.9222e-05, -6.9222e-05],\n",
            "        [ 7.1292e-04,  7.1292e-04,  7.1292e-04,  ...,  7.1292e-04,\n",
            "          7.1292e-04,  7.1292e-04],\n",
            "        [-7.4909e-04, -7.4909e-04, -7.4909e-04,  ..., -7.4909e-04,\n",
            "         -7.4909e-04, -7.4909e-04],\n",
            "        ...,\n",
            "        [-4.3261e-04, -4.3261e-04, -4.3261e-04,  ..., -4.3261e-04,\n",
            "         -4.3261e-04, -4.3261e-04],\n",
            "        [ 2.4486e-04,  2.4486e-04,  2.4486e-04,  ...,  2.4486e-04,\n",
            "          2.4486e-04,  2.4486e-04],\n",
            "        [ 0.0000e+00,  0.0000e+00,  0.0000e+00,  ...,  0.0000e+00,\n",
            "          0.0000e+00,  0.0000e+00]])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nM2dLK8ylHnk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 391
        },
        "outputId": "c107bb63-d93b-4167-f36f-7d32525ad47e"
      },
      "source": [
        "time0 = time()\n",
        "epochs = 20 # total number of iteration for training\n",
        "running_loss_list= []\n",
        "epochs_list = []\n",
        "\n",
        "for e in range(epochs):\n",
        "    running_loss = 0\n",
        "    for images, labels in train_loader:\n",
        "        # Flatenning MNIST images with size [64,784]\n",
        "        images = images.view(images.shape[0], -1) \n",
        "    \n",
        "        # defining gradient in each epoch as 0\n",
        "        optimizer.zero_grad()\n",
        "        \n",
        "        # modeling for each image batch\n",
        "        output = model(images)\n",
        "        \n",
        "        # calculating the loss\n",
        "        loss = criterion(output, labels)\n",
        "        \n",
        "        # This is where the model learns by backpropagating\n",
        "        loss.backward()\n",
        "        \n",
        "        # And optimizes its weights here\n",
        "        optimizer.step()\n",
        "        \n",
        "        # calculating the loss\n",
        "        running_loss += loss.item()\n",
        "        \n",
        "    else:\n",
        "        print(\"Epoch {} - Training loss: {}\".format(e, running_loss/len(train_loader)))\n",
        "print(\"\\nTraining Time (in minutes) =\",(time()-time0)/60)"
      ],
      "execution_count": 138,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 0 - Training loss: 0.4466558409008835\n",
            "Epoch 1 - Training loss: 0.16354922025696808\n",
            "Epoch 2 - Training loss: 0.11872489901501805\n",
            "Epoch 3 - Training loss: 0.09050703096463641\n",
            "Epoch 4 - Training loss: 0.07419387554414253\n",
            "Epoch 5 - Training loss: 0.0645542273410598\n",
            "Epoch 6 - Training loss: 0.055938178674255344\n",
            "Epoch 7 - Training loss: 0.04741653123610357\n",
            "Epoch 8 - Training loss: 0.04025967942135258\n",
            "Epoch 9 - Training loss: 0.0359757254201758\n",
            "Epoch 10 - Training loss: 0.03156802225557626\n",
            "Epoch 11 - Training loss: 0.02620987076606941\n",
            "Epoch 12 - Training loss: 0.022084646030087897\n",
            "Epoch 13 - Training loss: 0.018982726537787827\n",
            "Epoch 14 - Training loss: 0.01635531298944658\n",
            "Epoch 15 - Training loss: 0.014724115621850097\n",
            "Epoch 16 - Training loss: 0.011598301877422548\n",
            "Epoch 17 - Training loss: 0.015730660112313694\n",
            "Epoch 18 - Training loss: 0.009772086125975505\n",
            "Epoch 19 - Training loss: 0.008518094640423251\n",
            "\n",
            "Training Time (in minutes) = 3.3306051890055337\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vtlZclyflbVa",
        "colab_type": "code",
        "colab": {},
        "cellView": "form"
      },
      "source": [
        "#@title Classification predictions plot\n",
        "def classify(img, ps):\n",
        "    ''' \n",
        "    Function for viewing an image and it's predicted classes.\n",
        "    '''\n",
        "    ps = ps.data.numpy().squeeze()\n",
        "\n",
        "    fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n",
        "    ax1.imshow(img.resize_(1, 28, 28).numpy().squeeze())\n",
        "    ax1.axis('off')\n",
        "    ax2.barh(np.arange(10), ps)\n",
        "    ax2.set_aspect(0.1)\n",
        "    ax2.set_yticks(np.arange(10))\n",
        "    ax2.set_yticklabels(np.arange(10))\n",
        "    ax2.set_title('Class Probability')\n",
        "    ax2.set_xlim(0, 1.1)\n",
        "    plt.tight_layout()"
      ],
      "execution_count": 139,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I2It5Ms8OUIo",
        "colab_type": "text"
      },
      "source": [
        "### Predictions: \n",
        "Predictions below shows the output of log probabilities and predicted class of any random image from test set."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fSwxYcbVlpFY",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 270
        },
        "outputId": "b316b78b-7bb1-45f1-d1ac-84fcdc9690aa"
      },
      "source": [
        "images, labels = next(iter(test_loader))\n",
        "# replace trainloader to check training accuracy.\n",
        "\n",
        "img = images[0].view(1, 784)\n",
        "# Turn off gradients to speed up this part\n",
        "with torch.no_grad():\n",
        "    logpb = model(img)\n",
        "\n",
        "# Output of the network are log-probabilities, need to take exponential for probabilities\n",
        "pb = torch.exp(logpb)\n",
        "probab = list(pb.numpy()[0])\n",
        "print(\"Predicted Digit =\", probab.index(max(probab)))\n",
        "classify(img.view(1, 28, 28), pb)"
      ],
      "execution_count": 140,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Predicted Digit = 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAADsCAYAAAAhDDIOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAUc0lEQVR4nO3de7SddX3n8fcngYCRqyS4BILB4VKQDoXJsMB6qUUdoC6wl3FAsQNFUQS8gFra2hFrp6uOwup0YbUUUWgREeoF5SLIRdQSJAkUuS9EwASEIHKv5PadP/am68zp2cnJ8dl5nn3yfq11Fvs8v2fv53MOST7n9zy/s59UFZIkdc2MtgNIkjQRC0qS1EkWlCSpkywoSVInWVCSpE6yoCRJnWRBSRqaJKcl+ae2c6yvJPOTVJJNpvj8SrLrgLG3J7lyon2TfC7Jn08t9fRjQUn6lSR5W5JFSZ5J8nCSy5O8uqUsleTZfpZlSc5IMrONLINU1flV9aYBY++pqk8AJPmtJEs3bLpusaAkTVmSk4G/Af4KeCmwM/B3wOEtxtqnqrYADgLeBrxr/A5TnRlpw7KgJE1Jkq2BvwBOqKqvVtWzVbWyqr5ZVR8e8JyLkvwsyZNJrk/yyjFjhya5I8nT/dnPh/rb5yT5VpInkjye5HtJ1vlvV1XdBXwP2HvMKbtjkzwIXJNkRpKPJnkgyaNJzut/TWP9UZKH+jPDD43Jun+SG/qZHk5yZpJZ4557aJL7kjyW5FMvZE5ydJLvD/j+fDHJXyZ5MXA5sEN/NvhMkh2SPJdkuzH775dkeZJN1/X9GEUWlKSpOhDYHPjaejzncmA3YHtgCXD+mLHPA++uqi2BvYFr+ttPAZYCc+nN0v4UWOd7tCXZC3gNcPOYza8D9gT+G3B0/+P1wCuALYAzx73M6/t53wT8cZI39LevBj4IzKH3fTgIeO+45/4usADYj96M8o/WlfkFVfUscAjwUFVt0f94CLgOeOuYXd8BfLmqVk72tUeJBSVpqrYDHquqVZN9QlWdU1VPV9XzwGnAPmNmLSuBvZJsVVW/qKolY7a/DHh5f4b2vVr7m4guSfIL4JvA2cAXxoyd1p/p/RvwduCMqrqvqp4B/gQ4Ytzpv4/39/9R/3WO7H8di6tqYVWtqqr7gb+nV35jfbKqHq+qB+mdBj1yst+ntTgXOAqgf23tSOAfG3jdTrKgJE3Vz4E5k72ek2Rmkr9O8uMkTwH394fm9P/7+8ChwANJvpvkwP72TwH3Alf2T5mduo5D7VdV21bVf6qqj1bVmjFjPx3zeAfggTGfPwBsQm+WNtH+D/SfQ5Ld+6cdf9b/Wv5qzNex1uf+ir5Br8R3Ad4IPFlVP2zgdTvJgpI0VTcAzwNvmeT+b6N3qusNwNbA/P72AFTVTVV1OL3Tf18HvtLf/nRVnVJVrwAOA05OctAUM4+deT0EvHzM5zsDq4BHxmybN278of7jzwJ3AbtV1Vb0Tjtm3LEGPXcqWXsbqn5J7/tyFL3Te9N29gQWlKQpqqongf8FfCbJW5LMTrJpkkOS/J8JnrIlvUL7OTCb3qwDgCSz+r8ftHX/espTwJr+2JuT7JokwJP0rv+s+Q+vvv4uAD6YZJckW/TzXDjulOWf97+uVwLHABeO+VqeAp5J8mvA8RO8/oeTbJtkHvD+Mc+drEeA7SZYuHEevWtnh2FBSdLEqup04GTgo8Byeqe1TqQ3AxrvPHqnupYBdwALx42/A7i/f8rsPfSuEUFvkcJ3gGfozdr+rqqubSD+OfT+gb8e+AnwS+Ckcft8l97pxauBT1fVC79g+yF6M8KngX9g4vL5BrAYuAW4lN4ikEnrr0K8ALivv1pwh/72H9Ar6CVV9cDaXmPUxRsWStJoSXIN8KWqOrvtLMNkQUnSCEnyX4GrgHlV9XTbeYbJU3ySNCKSnEvvdOcHpns5gTMoSVJHrfX3F94447/bXtroXbXmovHLhyVtAJ7ikyR1ku/oK7Vozpw5NX/+/LZjSK1avHjxY1U1d/x2C0pq0fz581m0aFHbMaRWJZnw97k8xSdJ6iQLSpLUSRaUJKmTLChJUidZUJKkTrKgJEmdZEFJkjrJgpIkdZIFJUnqJAtKktRJFpTUsCTvT3JbktuTfKDtPNKosqCkBiXZG3gXsD+wD/DmJLu2m0oaTRaU1Kw9gRur6rmqWgV8F/i9ljNJI8mCkpp1G/CaJNslmQ0cCswbu0OS45IsSrJo+fLlrYSURoEFJTWoqu4EPglcCVwB3AKsHrfPWVW1oKoWzJ37H26BI6nPgpIaVlWfr6r/UlWvBX4B3NN2JmkUecNCqWFJtq+qR5PsTO/60wFtZ5JGkQUlNe+fk2wHrAROqKon2g4kjSILSmpYVb2m7QzSdOA1KElSJ1lQkqROsqAkSZ1kQUmSOslFEpq0TebtNHDs0hu/NeH2vRe+feBzXn7CzweOrXr4Z5MPJmlacgYlSeokC0qS1EkWlCSpkywoqWFJPti/WeFtSS5IsnnbmaRRZEFJDUqyI/A+YEFV7Q3MBI5oN5U0miwoqXmbAC9KsgkwG3io5TzSSHKZuf4/M/cYfHfy7c99ZODYylo94fYf7H/2wOe85T+fNHBs1oguM6+qZUk+DTwI/BtwZVVd2XIsaSQ5g5IalGRb4HBgF2AH4MVJjhq3j3fUlSbBgpKa9QbgJ1W1vKpWAl8FXjV2B++oK02OBSU160HggCSzkwQ4CLiz5UzSSLKgpAZV1Y3AxcAS4Ef0/o6d1WooaUS5SEJqWFV9DPhY2zmkUecMSpLUSc6gNkIz9v61gWPPnfHLgWNnzbtuvY+1eQb/EXtmh00Hjr1kvY8kabpxBiVJ6iQLSpLUSRaUJKmTLChJUidZUJKkTnIV30borvduNXDsnlee3+ixHln9/MCxl3zhhkaPJWl6cQYlSeokC0pqUJI9ktwy5uOpJB9oO5c0ijzFJzWoqu4GfgMgyUxgGfC1VkNJI8oZlDQ8BwE/rqoH2g4ijSILShqeI4ALxm/0hoXS5FhQ0hAkmQUcBlw0fswbFkqT4zWojdC5B2+42xO97tKTB47tzg83WI4WHAIsqapH2g4ijSpnUNJwHMkEp/ckTZ4FJTUsyYuBNwJfbTuLNMo8xSc1rKqeBbZrO4c06pxBSZI6yYKSJHWSBSVJ6iSvQY2wmdtuO3DsnjPnDxzbZ9ba3kV81pSy3LxizYTb9zx98C+irp7SkSRtLJxBSZI6yYKSJHWSBSVJ6iQLSpLUSRaU1LAk2yS5OMldSe5McmDbmaRR5Co+qXn/F7iiqv6g/67ms9sOJI0iC6rjZswe/G/be35448CxQ2Z/Zy2vOrWl5HeuXDlw7I+PP2niI91705SONaqSbA28FjgaoKpWACvazCSNKk/xSc3aBVgOfCHJzUnO7r95rKT1ZEFJzdoE2A/4bFXtCzwLnDp2B++oK02OBSU1aymwtKpeOP96Mb3C+nfeUVeaHAtKalBV/Qz4aZI9+psOAu5oMZI0slwkITXvJOD8/gq++4BjWs4jjSQLSmpYVd0CLGg7hzTqLKgOmLndSwaO3fm/dx04dsjs7zWeZdC7kgO888yTB4697Ip/aTyLpI2b16AkSZ1kQUmSOsmCkiR1kgUlSeokC0qS1EkWlCSpk1xmvoGsed2+A8dmnvbIwLF7dv9s41lW1uqBY0d+430Dx3Y93aXkkjYcZ1CSpE5yBiU1LMn9wNPAamBVVfmuEtIUWFDScLy+qh5rO4Q0yjzFJ0nqJAtKal4BVyZZnOS48YPesFCaHAtKat6rq2o/4BDghCSvHTvoDQulyfEaVINm7LPnwLG1LSX/xu7fbDzLrSsGLyX/yDuPHzi269ULG8+ysamqZf3/Pprka8D+wPXtppJGjzMoqUFJXpxkyxceA28Cbms3lTSanEFJzXop8LUk0Pv79aWquqLdSNJosqCkBlXVfcA+beeQpgNP8UmSOsmCkiR1kgUlSeokr0E16JFPrBk4duMQlpLfuXLlwLFTTjxp4NhmV9/UeBZJapozKElSJ1lQkqROsqAkSZ1kQUmSOsmCkiR1kgUlDUGSmUluTvKttrNIo8pl5utp1Xd2Hjh2w57/tJZnTu1ngbUtJT/5mPcOHNvsWpeSt+z9wJ3AVm0HkUaVMyipYUl2An4HOLvtLNIos6Ck5v0N8BFgwt/c9o660uRYUFKDkrwZeLSqFg/axzvqSpNjQUnN+k3gsCT3A18GfjvJ2i5OShrAgpIaVFV/UlU7VdV84Ajgmqo6quVY0kiyoCRJnbTRLjPfZKcdB45tf/FTA8c+ueMFA8dmsPmUsty6YvXAsbW+K7lLyTutqq4Drms5hjSynEFJkjrJgpIkdZIFJUnqJAtKktRJFpQkqZMsKElSJ220y8zv+LPBy8y/Pu+StTxzakvJ71m5YuDYR9554sCxza52KbmkjZMzKElSJ1lQUoOSbJ7kh0n+NcntST7ediZpVG20p/ikIXke+O2qeibJpsD3k1xeVQvbDiaNGgtKalBVFfBM/9NN+x/VXiJpdHmKT2pYkplJbgEeBa6qqhvbziSNIgtKalhVra6q3wB2AvZPsvfYce+oK03OtD7F98hJrxo4tuTNn17LMzeb0vFuXjHhHb4B+MgJ7xt8NJeST0tV9USSa4GDgdvGbD8LOAtgwYIFnv6TBnAGJTUoydwk2/Qfvwh4I3BXu6mk0TStZ1BSC14GnJtkJr0fAL9SVd9qOZM0kiwoqUFVdSuwb9s5pOnAU3ySpE6yoCRJnWRBSZI6aVpcg3r6fxww4fYPn3jhwOdsMWNqS8nXMHgp+btPf//Ase0v+5cpHU+SNlbOoCRJnWRBSZI6yYKSJHWSBSVJ6iQLSpLUSRaU1KAk85Jcm+SO/h11By/tlLRW02KZ+cp3PD7h9rdu8Wjjx3rVkrcPHNv+TJeSi1XAKVW1JMmWwOIkV1XVHW0Hk0aNMyipQVX1cFUt6T9+GrgT2LHdVNJosqCkIUkyn94bx944brs3LJQmwYKShiDJFsA/Ax+oqqfGjlXVWVW1oKoWzJ07t52A0giwoKSGJdmUXjmdX1VfbTuPNKosKKlBSQJ8Hrizqs5oO480ykZmFd8Tf3jgwLHv7/u3A0am1r/P1YqBY48v22bg2JwpHU3TzG8C7wB+lOSW/rY/rarLWswkjaSRKShpFFTV94G0nUOaDjzFJ0nqJAtKktRJFpQkqZMsKElSJ1lQkqROGplVfNucd8PAseV/+fyE218680UDn3PMAwcNHFv28V0Hjs04xE5Xc3607Enmn3pp2zGkKbn/r39nqK/vv7aSpE6yoCRJnWRBSQ1Kck6SR5Pc1nYWadRZUFKzvggc3HYIaTqwoKQGVdX1wMS3eJa0XiwoSVInjcwy86nY/Yp3Dxzb4z23DhybtXLRwLFdv/0rRZJIchxwHMDMrbxhoTSIMyhpAxt7R92Zs7duO47UWRaUJKmTLCipQUkuAG4A9kiyNMmxbWeSRtW0vgYlbWhVdWTbGaTpwhmUJKmTLChJUidNi1N8x+786gm3787g5eI1rDDSevj1Hbdm0ZDfEVoaVc6gJEmdZEFJkjrJgpIkdZIFJUnqJAtKktRJFpQkqZMsKKlhSQ5OcneSe5Oc2nYeaVRZUFKDkswEPgMcAuwFHJlkr3ZTSaPJgpKatT9wb1XdV1UrgC8Dh7ecSRpJFpTUrB2Bn475fGl/279LclySRUkWLV++fIOGk0aJBSVtYGNvWDh3rnfUlQaxoKRmLQPmjfl8p/42SevJgpKadROwW5JdkswCjgAuaTmTNJKmxbuZS11RVauSnAh8G5gJnFNVt7ccSxpJFpTUsKq6DLis7RzSqPMUnySpkywoSVInWVCSpE6yoCRJnWRBSZI6yYKSJHWSBSVJ6iQLSpLUSRaUJKmTLChJUif5VkdSixYvXvxMkrvbzjHGHOCxtkP0mWVi0zHLyyfaaEFJ7bq7qha0HeIFSRZ1JY9ZJrYxZVlrQV215qIM68CSJK2N16AkSZ1kQUntOqvtAON0KY9ZJrbRZElVDfP1JUmaEmdQkqROsqCkDSDJwUnuTnJvklMnGN8syYX98RuTzG8xy8lJ7khya5Krk0y4BHhDZBmz3+8nqSRDXb02mTxJ3tr//tye5EttZUmyc5Jrk9zc/3916JBynJPk0SS3DRhPkr/t57w1yX6NHbyq/PDDjyF+ADOBHwOvAGYB/wrsNW6f9wKf6z8+AriwxSyvB2b3Hx/fZpb+flsC1wMLgQUt/3/aDbgZ2Lb/+fYtZjkLOL7/eC/g/iFleS2wH3DbgPFDgcuBAAcANzZ1bGdQ0vDtD9xbVfdV1Qrgy8Dh4/Y5HDi3//hi4KAkw/g1j3Vmqaprq+q5/qcLgZ2GkGNSWfo+AXwS+OWQcqxPnncBn6mqXwBU1aMtZilgq/7jrYGHhhGkqq4HHl/LLocD51XPQmCbJC9r4tgWlDR8OwI/HfP50v62CfepqlXAk8B2LWUZ61h6Px0Pwzqz9E8XzauqS4eUYb3yALsDuyf5QZKFSQ5uMctpwFFJlgKXAScNKcu6rO+fqUnznSQkTSjJUcAC4HUtHX8GcAZwdBvHH2ATeqf5fovezPL6JL9eVU+0kOVI4ItVdXqSA4F/TLJ3Va1pIctQOIOShm8ZMG/M5zv1t024T5JN6J2y+XlLWUjyBuDPgMOq6vkh5JhMli2BvYHrktxP7/rGJUNcKDGZ781S4JKqWllVPwHuoVdYbWQ5FvgKQFXdAGxO773xNrRJ/ZmaCgtKGr6bgN2S7JJkFr1FEJeM2+cS4H/2H/8BcE31r0Bv6CxJ9gX+nl45DesayzqzVNWTVTWnquZX1Xx618MOq6pFbeTp+zq92RNJ5tA75XdfS1keBA7qZ9mTXkEtH0KWdbkE+MP+ar4DgCer6uEmXthTfNKQVdWqJCcC36a3Ouucqro9yV8Ai6rqEuDz9E7R3EvvgvQRLWb5FLAFcFF/ncaDVXVYS1k2mEnm+TbwpiR3AKuBD1dV4zPdSWY5BfiHJB+kt2Di6GH8UJPkAnqlPKd/vetjwKb9nJ+jd/3rUOBe4DngmMaOPZwf0iRJ+tV4ik+S1EkWlCSpkywoSVInWVCSpE6yoCRJnWRBSZI6yYKSJHWSBSVJ6qT/BymXYKq2C8mtAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x648 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dwSWM-UQOiTm",
        "colab_type": "text"
      },
      "source": [
        "### Testing and test results: \n",
        "Finally we are calculating testing accuracy of our neural network."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pmp9Vtjelt3b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        },
        "outputId": "98c96ce9-99d5-43f2-b52a-7510847d4512"
      },
      "source": [
        "correct_count, all_count = 0, 0\n",
        "for images,labels in test_loader:\n",
        "  for i in range(len(labels)):\n",
        "    img = images[i].view(1, 784)\n",
        "\n",
        "    with torch.no_grad():\n",
        "        logps = model(img)\n",
        "\n",
        "    ps = torch.exp(logps)\n",
        "    probab = list(ps.numpy()[0])\n",
        "    pred_label = probab.index(max(probab))\n",
        "    true_label = labels.numpy()[i]\n",
        "    if(true_label == pred_label):\n",
        "      correct_count += 1\n",
        "    all_count += 1\n",
        "\n",
        "print(\"Number Of Images Tested =\", all_count)\n",
        "print(\"\\nModel Accuracy =\", (correct_count/all_count))\n",
        "    "
      ],
      "execution_count": 141,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Number Of Images Tested = 10000\n",
            "\n",
            "Model Accuracy = 0.9813\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O46rkID8Oml8",
        "colab_type": "text"
      },
      "source": [
        "### How to save a model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rIrFDkM2lztL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "torch.save(model, 'nn_model_mnist.pth') # .pt or .pth extension"
      ],
      "execution_count": 142,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAK0hFE8IN2H",
        "colab_type": "text"
      },
      "source": [
        "Adapted from Neuromatch [DL Tutorial](https://github.com/NeuromatchAcademy/course-content/tree/master/tutorials/W3D4_DeepLearning1), [h1ros](h1ros.github.io) and [pytorch](https://colab.research.google.com/drive/1B5KQvPySqYEa6XicRHdOwgv8fN1BrCgQ)"
      ]
    }
  ]
}