intro-to-pytorch/Part 5 - Inference and Validation (Exercises).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inference and Validation\n",
    "\n",
    "Now that you have a trained network, you can use it for making predictions. This is typically called **inference**, a term borrowed from statistics. However, neural networks have a tendency to perform *too well* on the training data and aren't able to generalize to data that hasn't been seen before. This is called **overfitting** and it impairs inference performance. To test for overfitting while training, we measure the performance on data not in the training set called the **validation** set. We avoid overfitting through regularization such as dropout while monitoring the validation performance during training. In this notebook, I'll show you how to do this in PyTorch. \n",
    "\n",
    "As usual, let's start by loading the dataset through torchvision. You'll learn more about torchvision and loading data in a later part. This time we'll be taking advantage of the test set which you can get by setting `train=False` here:\n",
    "\n",
    "```python\n",
    "testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)\n",
    "```\n",
    "\n",
    "The test set contains images just like the training set. Typically you'll see 10-20% of the original dataset held out for testing and validation with the rest being used for training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "# Define a transform to normalize the data\n",
    "transform = transforms.Compose([transforms.ToTensor(),\n",
    "                                transforms.Normalize((0.5,), (0.5,))])\n",
    "# Download and load the training data\n",
    "trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=True, transform=transform)\n",
    "trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)\n",
    "\n",
    "# Download and load the test data\n",
    "testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/', download=True, train=False, transform=transform)\n",
    "testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here I'll create a model like normal, using the same one from my solution for part 4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import nn, optim\n",
    "import torch.nn.functional as F\n",
    "\n",
    "class Classifier(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(784, 256)\n",
    "        self.fc2 = nn.Linear(256, 128)\n",
    "        self.fc3 = nn.Linear(128, 64)\n",
    "        self.fc4 = nn.Linear(64, 10)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # make sure input tensor is flattened\n",
    "        x = x.view(x.shape[0], -1)\n",
    "        \n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = F.relu(self.fc2(x))\n",
    "        x = F.relu(self.fc3(x))\n",
    "        x = F.log_softmax(self.fc4(x), dim=1)\n",
    "        \n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of validation is to measure the model's performance on data that isn't part of the training set. Performance here is up to the developer to define though. Typically this is just accuracy, the percentage of classes the network predicted correctly. Other options are [precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall#Definition_(classification_context)) and top-5 error rate. We'll focus on accuracy here. First I'll do a forward pass with one batch from the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([64, 10])\n"
     ]
    }
   ],
   "source": [
    "model = Classifier()\n",
    "\n",
    "images, labels = next(iter(testloader))\n",
    "# Get the class probabilities\n",
    "ps = torch.exp(model(images))\n",
    "# Make sure the shape is appropriate, we should get 10 class probabilities for 64 examples\n",
    "print(ps.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the probabilities, we can get the most likely class using the `ps.topk` method. This returns the $k$ highest values. Since we just want the most likely class, we can use `ps.topk(1)`. This returns a tuple of the top-$k$ values and the top-$k$ indices. If the highest value is the fifth element, we'll get back 4 as the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1],\n",
      "        [ 1]])\n"
     ]
    }
   ],
   "source": [
    "top_p, top_class = ps.topk(1, dim=1)\n",
    "# Look at the most likely classes for the first 10 examples\n",
    "print(top_class[:10,:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can check if the predicted classes match the labels. This is simple to do by equating `top_class` and `labels`, but we have to be careful of the shapes. Here `top_class` is a 2D tensor with shape `(64, 1)` while `labels` is 1D with shape `(64)`. To get the equality to work out the way we want, `top_class` and `labels` must have the same shape.\n",
    "\n",
    "If we do\n",
    "\n",
    "```python\n",
    "equals = top_class == labels\n",
    "```\n",
    "\n",
    "`equals` will have shape `(64, 64)`, try it yourself. What it's doing is comparing the one element in each row of `top_class` with each element in `labels` which returns 64 True/False boolean values for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "equals = top_class == labels.view(*top_class.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to calculate the percentage of correct predictions. `equals` has binary values, either 0 or 1. This means that if we just sum up all the values and divide by the number of values, we get the percentage of correct predictions. This is the same operation as taking the mean, so we can get the accuracy with a call to `torch.mean`. If only it was that simple. If you try `torch.mean(equals)`, you'll get an error\n",
    "\n",
    "```\n",
    "RuntimeError: mean is not implemented for type torch.ByteTensor\n",
    "```\n",
    "\n",
    "This happens because `equals` has type `torch.ByteTensor` but `torch.mean` isn't implemented for tensors with that type. So we'll need to convert `equals` to a float tensor. Note that when we take `torch.mean` it returns a scalar tensor, to get the actual value as a float we'll need to do `accuracy.item()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 7.8125%\n"
     ]
    }
   ],
   "source": [
    "accuracy = torch.mean(equals.type(torch.FloatTensor))\n",
    "print(f'Accuracy: {accuracy.item()*100}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The network is untrained so it's making random guesses and we should see an accuracy around 10%. Now let's train our network and include our validation pass so we can measure how well the network is performing on the test set. Since we're not updating our parameters in the validation pass, we can speed up our code by turning off gradients using `torch.no_grad()`:\n",
    "\n",
    "```python\n",
    "# turn off gradients\n",
    "with torch.no_grad():\n",
    "    # validation pass here\n",
    "    for images, labels in testloader:\n",
    "        ...\n",
    "```\n",
    "\n",
    ">**Exercise:** Implement the validation loop below and print out the total accuracy after the loop. You can largely copy and paste the code from above, but I suggest typing it in because writing it out yourself is essential for building the skill. In general you'll always learn more by typing it rather than copy-pasting. You should be able to get an accuracy above 80%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Classifier()\n",
    "criterion = nn.NLLLoss()\n",
    "optimizer = optim.Adam(model.parameters(), lr=0.003)\n",
    "\n",
    "epochs = 30\n",
    "steps = 0\n",
    "\n",
    "train_losses, test_losses = [], []\n",
    "for e in range(epochs):\n",
    "    running_loss = 0\n",
    "    for images, labels in trainloader:\n",
    "        \n",
    "        optimizer.zero_grad()\n",
    "        \n",
    "        log_ps = model(images)\n",
    "        loss = criterion(log_ps, labels)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        \n",
    "        running_loss += loss.item()\n",
    "        \n",
    "    else:\n",
    "        ## TODO: Implement the validation pass and print out the validation accuracy\n",
    "        print(f'Accuracy: {accuracy.item()*100}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overfitting\n",
    "\n",
    "If we look at the training and validation losses as we train the network, we can see a phenomenon known as overfitting.\n",
    "\n",
    "<img src='assets/overfitting.png' width=450px>\n",
    "\n",
    "The network learns the training set better and better, resulting in lower training losses. However, it starts having problems generalizing to data outside the training set leading to the validation loss increasing. The ultimate goal of any deep learning model is to make predictions on new data, so we should strive to get the lowest validation loss possible. One option is to use the version of the model with the lowest validation loss, here the one around 8-10 training epochs. This strategy is called *early-stopping*. In practice, you'd save the model frequently as you're training then later choose the model with the lowest validation loss.\n",
    "\n",
    "The most common method to reduce overfitting (outside of early-stopping) is *dropout*, where we randomly drop input units. This forces the network to share information between weights, increasing it's ability to generalize to new data. Adding dropout in PyTorch is straightforward using the [`nn.Dropout`](https://pytorch.org/docs/stable/nn.html#torch.nn.Dropout) module.\n",
    "\n",
    "```python\n",
    "class Classifier(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.fc1 = nn.Linear(784, 256)\n",
    "        self.fc2 = nn.Linear(256, 128)\n",
    "        self.fc3 = nn.Linear(128, 64)\n",
    "        self.fc4 = nn.Linear(64, 10)\n",
    "        \n",
    "        # Dropout module with 0.2 drop probability\n",
    "        self.dropout = nn.Dropout(p=0.2)\n",
    "        \n",
    "    def forward(self, x):\n",
    "        # make sure input tensor is flattened\n",
    "        x = x.view(x.shape[0], -1)\n",
    "        \n",
    "        # Now with dropout\n",
    "        x = self.dropout(F.relu(self.fc1(x)))\n",
    "        x = self.dropout(F.relu(self.fc2(x)))\n",
    "        x = self.dropout(F.relu(self.fc3(x)))\n",
    "        \n",
    "        # output so no dropout here\n",
    "        x = F.log_softmax(self.fc4(x), dim=1)\n",
    "        \n",
    "        return x\n",
    "```\n",
    "\n",
    "During training we want to use dropout to prevent overfitting, but during inference we want to use the entire network. So, we need to turn off dropout during validation, testing, and whenever we're using the network to make predictions. To do this, you use `model.eval()`. This sets the model to evaluation mode where the dropout probability is 0. You can turn dropout back on by setting the model to train mode with `model.train()`. In general, the pattern for the validation loop will look like this, where you turn off gradients, set the model to evaluation mode, calculate the validation loss and metric, then set the model back to train mode.\n",
    "\n",
    "```python\n",
    "# turn off gradients\n",
    "with torch.no_grad():\n",
    "    \n",
    "    # set model to evaluation mode\n",
    "    model.eval()\n",
    "    \n",
    "    # validation pass here\n",
    "    for images, labels in testloader:\n",
    "        ...\n",
    "\n",
    "# set model back to train mode\n",
    "model.train()\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Exercise:** Add dropout to your model and train it on Fashion-MNIST again. See if you can get a lower validation loss or higher accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Define your model with dropout added"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## TODO: Train your model with dropout, and monitor the training progress with the validation loss and accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inference\n",
    "\n",
    "Now that the model is trained, we can use it for inference. We've done this before, but now we need to remember to set the model in inference mode with `model.eval()`. You'll also want to turn off autograd with the `torch.no_grad()` context."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA04AAAGdCAYAAAAllSMpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3XeYJVWd//H3hxnCgCASJPkTXF0FxdUFFcEECkZcMaKoK4iuWdaEsrqCgoppRdc1YAARM2sExCU4skpQ0iJKMIBIkDASncAM8/39UdXO5Xq7696Z7umZ5v16nvvcvlXfOnWqZwb60+fUqVQVkiRJkqTxrTHdHZAkSZKkVZ3BSZIkSZI6GJwkSZIkqYPBSZIkSZI6GJwkSZIkqYPBSZIkSZI6GJwkSZIkqYPBSZIkSZI6GJwkSZIkqYPBSZIkSZI6GJwkSZIkqYPBSZIkSZI6GJwkSZIkqYPBSZIkaRxJqn1tM919ubuYru/5ipw3ydHtsYcM226Sfdvtc5evx1rZDE6SJGnGS7Juktck+UGSK5PMT/KXJJcnOS7JS5LMme5+rixJruj5gX7sdWeSeUn+N8mbkqw73f28u2pD1SFJHj7dfdEys6e7A5IkSVMpyTOBI4HNezb/BVgKbNO+ngt8MMlLq+q0ld3HafQX4Pb267WAjYDHtq9XJNmtqq6frs6tRq4FLgVuHOGYW9pjrhywb1/gCcAVwAUr2DdNEkecJEnSjJVkX+C7NKHpUuClwCZVdY+q2gDYEHgeMBfYEnj89PR02nykqjZvXxsBmwDvAwp4ME3gVIeqOqiqtq2qT45wzHfaY/55KvumyWNwkiRJM1KSfwA+Q/PzzonAP1bVsVU1b6ymqm6pqv+uqt2AvYHbpqe3q4aqmldV7wKOajc9K8mW09knaVVhcJIkSTPV+4C1gauBfapqwUTFVfVN4D+GaTjJrCS7Jfl4knOTXJfkjiTXJPlOkidOcOwa7T0sP27vKVqc5IYkv0ryxSRPHXDM/ZJ8OsllSRa092j9IcncJAcl2WSYfo/gaz1f79DTj78ugpBk7STvTHJhktva7Rv29Xu3JN9O8qf2+/Onru9P3/HbJ/l6e9zCJJck+fcka49Tf48kz0/ylSQXJbm5/X79NsmRSf5+is477uIQE5zjbxaHGNtGM00P4Ki++9CuaOu+2H4+ruMc72nrzhi2Xxqf9zhJkqQZJ8lWwDPaj5+oqluGOa6qashTbAf03gu1CLgD2ALYC9gryTur6v0Djv0ysE/P51uADWimyT24fZ00tjPJDjRTCddvNy2muTfpvu3rCcD5vcdMgqt7vt5gwP51gNOBR7X9md9fkOQw4J3tx6K5znuz7PtzeFUdNEEfdqGZKrgecCsQ4EHAe4GnJ9mjqm7vO2Zf4D97Pt9GM1Bw//a1T5K9quqUST7vZFkAXEdzr9ma7fl7A/8N7fvngf2AZybZuHcUdUySAC9rP35xivp7t+KIkyRJmol2pfmBF+D7U9D+HcC3gGfS3D81p6ruAWwG/DtwJ3BYkp16D0ryeJrQtBR4E7BBVW1IE0S2pPnB/6d95/oITWg6G9ihqtaqqnvR/GD/SOAImlAyme7b8/XNA/a/Dngg8ELgHu01bEMT6EjyQpaFpk8C9277vCnLgs07krxkgj58Cvg18A9VdU+a78F+NEHi0QweHZzXtr8LsGF7H9s6NEH3KzTfs68mWW+SzzspquobVbU5MDZCdEDPPWibV9Uj27oz2j6uBbx4nOaeBGxN82fyjanq892JwUmSJM1E27Xvi2gWhZhUVXVZVb2gqo6vquvGRqqq6vqqOgx4D01we3XfoY9u3/+nqo6oqtva46qqrq2qL1XVW8c55oCqOr+nD/Or6pyqelNVnTnJl/jKsdMAvxiw/x7A3u0P+ne0/flDVS1uRzoObeu+XlVvqKob25p5VfVGlk0FPCzJeD+PLgKeWlW/bI+9o6qOBl7b7t8/yda9B1TV16rqjVV15tgoY/u9vYRmYZBTaMLb8ya49pHPO00+377vN87+l7fvx439PdOKMThJkqSZaOP2/aYRpt9Nph+074/p235r+37vCQJDv7FjtljhXk0gyVpJHpzk8zTLs0MTfG4YUH5hVf3POE09HHhA+/Vh49S8p33fmma63yCfqao/D9h+DHAVzc+xzx7n2L/R/j04of3Y/+cyZeedQsfQjHw+PMk/9u5Ick+W9dFpepPE4CRJkrQcksxpHxQ7N8n17SIP1d7cPzYy1L8i3Sk0P+zuAMxN8+DdrlXrTmzfj0lyeJJHJ1lzki7j4J4+LwJ+Bezf7juLZaMs/SYa4RpbTOKGqvrVoIKqupRl91HtMKiG5r6uQccuBf53vGOT3CfJB9tFO25O82DfsWv8WFs20fd8uc67srX3NX23/dg/6rQPzRTF31TV6Su1YzOYwUmSJM1EYzfL36udOjapkmxB82DS/6BZnGFTmuBxA83N/WMPQr3LvTRV9VvgNTT3yzyOZqGIq5Nc3q6ad5eRg9bbaO55WR94O01ouTXJaUlek2TOClzKX9r+XgdcA1wMfJtmWtvjqmrQ/U2wbJGCQTZt36+eoAaa0Zve+n4THT+27y7HJnkCzTUcSBNu7kmzQMTYNY6N3k10j9PI551GY9P19kmyVs/2sWl6R6FJY3CSJEkz0cXt+9o0K6JNtiNoFkf4Pc20to3ah+reu725/9HjHVhVXwTuB/wr8D2akLcNzf1Q5yb5t776ecBjgT2AT9CMZq0F7EazkMFFSe6znNfR+wDcrarqwVX13PZ5V0smOO7OIdoeuHT3JPmbMNyOwh1Lc//VKTQPM55TVRuOXSPw5vGOX97zTrNTgMtppqb+E0CShwCPoPkz+tL0dW3mMThJkqSZ6Cc0CxtA+wPlZGl/s/+s9uOLq+rbVXVTX9lmE7XRLijx8arai2b04lHAd2h+MD80zcN7e+urqk6pqgOqageapctfBfwZ+DuWTUFbFYyNRt13wioYC3vjjV5NNJ1u7H6v3mN3btv8M/CsqvrfqlrYd9yEfy7Led5p0963NXYP09h0vbGplj+qqmtWfq9mLoOTJEmacarqKpbdG/SGJIOeRfQ3hpzWtwnLRlPOH6dm92HOB38NRb8Ans+yxQce23HMTVV1JDA2OvWEiepXsvPa9/WSDFz4IckDga366vsNvKb2z+hxA44dC2KXVdXfPFeqNcyfy6jnnQpLx047RO1RNKNLT2lX+xtb4t1FISaZwUmSJM1U76K57+g+NM/uWWei4iQvYNlUroncyrLRrIcOaGcL4A3jnGOtQdsBqupOmofJQhvMkqyRZPYEfVnQW7+KuAD4bfv1v41Tc0j7fgXw83FqXpNkwwHbXwL8P5pw8e2e7WPPsvr7QX/WSZ5MM72xy6jnnQpj92IN6sddVNXVwA+BWTTPqtqUZkRsKp5fdrdmcJIkSTNSVV1A86DWAp4BnN+uYrfRWE2SeyZ5TpIf0zwkdP0h2r2dZsU5gC8meXjb1hpJnkQzTXC8kYL3JzkuyV59/dgsySdo7n0q4OR21wbAb5O8M8lDk8zqO9f72rofdX9HVo52+ti72o/PSvKfSTYGSLJxe50vave/q12tbpB1gJOSbN8eu2aSlwGfafd/oaqu7Kn/GTCf5n6fY9oAO7b64cuB/2bZoiETGfW8U2FsNcLntEuLdxlbJGJsmfVjq2rxeMVaPhP9BkOSJGm1VlVfSDIP+CywLc0qdiS5nSag9AalPwCnDdn0m4Af04w4nZ/kLzS/kJ5Dc4/Ny1m2VHSv2TSLSTy37cetNCGrtx/vqqqLej5vTfM8pMOAxUluo1ktbla7//cMN1K20lTVN5I8FHgn8HrgtUluoen32C/uD6+qr0zQzGuBzwG/bI+dQ7MoBjTB9S7XXFU3JzkI+DjNtMfnt8etR/N9v4Bm+tonOro/0nmnyJeBt9JM2bwxyfU0o5FXVdWgaZwnANey7B4sp+lNAUecJEnSjFZV36VZQOF1NPc9XUXzg/Rsmqlix9E89+ZBwz7zpqrOplmM4LvATcCawPU0Ae3hwP+Nc+jHgDfSrKZ3GU1oWhv4I82I1+Or6v099bcCe9Ks4vdzmilY69MsI/4LmmDy8PaerlVKVb0LeBLNtd5Is9rdPJopZLtX1UEdTZwB7AR8k2bKZQGXAu8Gdm1H/vrP+QngOSwbfZoNXAIcDOxCszR5l5HPO9mq6hKaVRRPopmCuDlNgB64emK7AuLYQ5d/0Re8NUkyPQ/TliRJkjRZklwG/D3wmqr6TFe9RmdwkiRJklZj7f1up9CMRG5ZVbd2HKLl4FQ9SZIkaTWVZBPgw+3HLxqapo4jTpIkSdJqJslHgBfQ3P+0Js19ZA+pquuntWMzmCNOkiRJ0upnE5rnSi0A/gd4oqFpajniJEmSJEkdHHGSJEmSpA4GJ0mSJEnqYHCSJEmSpA6zp7sDU2WPNZ7vzVuSBJy89FuZ7j5IkrS6c8RJkiRJkjrM2BEnSZLurpJcDmwAXDHNXZGk6bYNcGtV3W9FGzI4SZI082wwZ86cjbbbbruNprsjkjSdLr74YhYsWDApbRmcJEmaea7YbrvtNjr33HOnux+SNK123HFHzjvvvCsmoy3vcZIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSZIkSepgcJIkSZKkDgYnSdKkSfLGJJXkgklo6+gktw9RNzfJ3BU9X995q+e1JMkfk3w9yYMn6zzjnHvdJIck2XUqzyNJGt3s6e6AJGlGeXn7/rAkO1bVudPam+W3AHhi+/Vs4AHAu4Azkjy4qq6ZovOuCxzcfj13is4hSVoOjjhJkiZFkkcADwN+0G7afxq7s6KWVtVZ7eunVXU08C/APYFnTG/XJEnTweAkSZosY0HpQOAs4EVJ5vQWJNmmnf721iRvSXJ5ktuTnJnk0V0nSPKYJDcmOT7JehPUbZDkI237dyS5OskREx0zhFva9yV959o+yfeS3JRkYZILkrxsQJ/um+TYJNcnWZTk4vZ7sEa7fxvghrb84J6pgoesQJ8lSZPEqXqSpBXWBqQXAWdW1SVJvgB8DngO8JUBh7wOuAT41/bzocCJSe5XVbcMqCfJC4BjgC8Cb6iqO8epWxf4CXAf4P3AhcBDgPcCD02ye1XVENc09v/Isal6HwZuAk7sqXkQcAZwHfAG4M/AS4Cjk2xWVR9q6zZt69akmfL3B2BP4CPA/YHXAtcCTwVOAr4AfL49zVUT9HG8qZDbdl2fJGk0BidJ0mR4Hs00tqPaz98AjqAZhRoUnG4D9hwLP0muBc4GngZ8vb84yduB9wH/NhZGJvBG4B+AnarqnHbbqUmuBo6jCSc/7GhjPWBx37ZrgWdW1XU92w6hCUO7VdVYwDkxyYY0o0afbYPgm4GtgEf29OlHSWYBr05yRFVd1hOErqqqszr6KElaiZyqJ0maDPsD82kCE1V1G/BNYNckfzeg/oS+EaML2/et++qS5LPAe4B9hghN0IzkXARckGT22Av4EVDArkO0sQB4ZPvaiWbk7DKaULRzT90TgdN6QtOYo2kWeti5p+7XPaGpty4sW4hiJFW146AXzWieJGkSGZwkSSskyQOAx9NMYVsjyYbtiMtxNKFgvwGHzev9UFUL2y/n9NWtBewN/IruUaIxm9GMOC3ue93W9meTIdpYWlXntK+fV9V3gKfT3N/0Hz11G9OMRPW7pmf/KHWSpFWUU/UkSSvq5TSB5Hntq9++SQ6uqqXL0fYiYDea0aJTkjy1qm7qOOZGmhGjl0+wf2RVNT/J72hWDhwzD9h8QPmWfecatk6StIoyOEmSllt7j87LgN8BrxhQ8iyaBSCeTLPowciq6vwkTwBOAeYm2aOqrp/gkOOBfwPmVdXly3POQZLcg2aRiN5znwo8O8kWVdU7ovTPNFMXz+qpOyjJDlV1Xl9dAT9uPy9q3/tH3iRJ08zgJElaEU+jGTV5e1XN7d+Z5Fc0K8btz3IGJ4CqujjJ42jC0+ntynjjrTZ3BPDctu5jNPdPrQHclybAHVFVZ3Scco2e5dHXoFnY4Y3AvWgWhBjzHpp7quYmeS/NqnovpnnW04E9KwR+jCYknZDk3TSr6j2D5nvz6aq6rL3O25JcCTwzycnAzcA1U/jAXUnSkAxOkqQVsT9wB8tW07uLqrohyXdoRmWGubdoXFX1+57w9L9JnlRVvx9Q95e27h00D629H83UvStpRn6GGYWaA5w51iTNKNPFwLOr6rs957o0yS40y57/V3vcxcB+7UNzx+puaOs+0L42AH5P88yr3numoBm5+whwAs09Xu/hrmFNkjQNMsSjLFZLe6zx/Jl5YZI0opOXfivT3QetXEnO3WGHHXY499zxHvMkSXcPO+64I+edd9557YqjK8RV9SRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJGkKSfZNU3+v6JD9O8rTp7p8kaWoZnCRJGs1+wM7ALsCrgKXAiUmeMa29kiRNqdnT3QFJklYzF1XVOWMfkpwE3ATsA5wwbb2SJE0pR5wkSVoxC4E7gCVjG5IckuTnSf6c5NYk5yXZP0l6D0yydpKPJvlTkvlJfprkUUmuSHL0Sr4OSdIEHHGSJGk0s5LMBgJsBrwNWA/4Wk/N1sBngCvbz48G/hPYCnhvT91RwN7Ah4DTgAcDxwEbTGH/JUnLweAkSdJozur7vAh4fVWdNLahqvYb+zrJGsBcmqB1QJJDq6qSPBh4EfDBqjqoLT85yXXcNYSNK8m54+zadqgrkSQNzeAkSdJo/hm4uP16E+DZwH8lmVVVnwRIsgfwDuAR/O3o0b2B64AntJ+/2bf/OODLU9BvSdIKMDhJkjSai3sXhwBOSrI18KEkxwIPAn5IM8r0SuAqmnug9gLeCcxpj9u4fb+ut/GqWpJk3jAdqaodB21vR6J2GOpqJElDMThJkrTiLgSeAjwQeCGwGNizqhaOFSTZq++YsXC0GXB1T91sloUqSdIqwlX1JElacQ9v32+gea7TEuDOsZ1J5gAv7Tvm9Pb9BX3bn4e/2JSkVY7/YZYkaTTbt6NC0IwMPQfYA/hOVV2e5ATgzcDXk3ymrXkrzSISf1VVv0ryNeCtSZbSrKr3EOAtwC00AUyStIowOEmSNJqjer6+BbgceBPwaYCqOi3Jy4G3Az+gmYb3OeB64At9be0HXAvs37ZxAc0I1EnAzVN3CZKkURmcJEkaQlUdDRw9ZO1R3DVgjfliX90imhGmt4xtS7ILcE/gHCRJqwyDkyRJ0yTJ7sBOwHk0U/keRrOM+W+Ab09j1yRJfQxOkiRNn9uAp9GMOK0P3EizlPlBvSvySZKmn8FJkqRpUlVnA4+d7n5Ikrq5HLkkSZIkdTA4SZIkSVIHg5MkSZIkdTA4SZIkSVIHg5MkSZIkdTA4SZIkSVIHg5MkSZIkdTA4SZI0A1109S3T3QVJmlEMTpIkSZLUweAkSZIkSR0MTpIkSZLUweAkSZIkSR0MTpIkSZLUweAkSVolJdkpyXeTXJlkUZLrkpyZ5KPT3TeAJFckOX66+yFJWjkMTpKkVU6SPYEzgPWBA4EnAwcAPwP2nsauSZLupmZPdwckSRrgQOAK4ClVtaRn+9eTHDg9XVq5kgRYp6oWTHdfJEmOOEmSVk0bATf0hSYAqmrp2Ndj0+WSPC3JeUkWJLkkycv7j0uyeZLPJrkqyR1JLk9ycJLZfXWHJPl5kj8nubVtd/82yEwoyWuTLEnynp5tayV5V9uvRUluSHJUkk37jh27lhckuRBYCLxqqO+WJGnKOeIkSVoVnQG8MskRwJeB/xsUoloPAz4KfAC4DngF8IUkv62q06EJTcDPgaXAe4HfATsD7wK2AfbraW9r4DPAle3nRwP/CWzVHvs32lD1YeCNwCuq6uh2+xrA94DHAR9qr2tr4D3A3CSP6BtR2hF4EHAocBUwb4LvkSRpJTI4SZJWRe8AHkhzX9MBwMIkZwPHA5+qqvk9tZsAj6mqKwGSnA7sDuwDnN7WHALcC3jIWB1wapIFwEeSfLiqfg1QVX8NUW3wmQsEOCDJoVVVvR1NMocm3O0OPK2qTu3Z/QLgqcBzq+rbPcf8H/ALYF/g033X8tiq+t0w36Qk546za9thjpckDc+pepKkVU5V/bmqdgV2AN4G/AB4CM2ozi+TbNxTfkFPGKKqFgKX0YzsjNkT+DFwTZLZYy/gh+3+J4wVJtkjyalJbgHuBBbTjDRtDNy7r6sbA6cBj6IJPKf27d8TuBn4Qd95LwD+BOzaV//LYUOTJGnlcsRJkrTKqqrzgfMBkqwJHA68GXg7zQISMHg620JgTs/nzYBn0oSgQTZpz7ETTZiaC7ySZrrcHcBewDv72oRmVOxewOeq6qIB7W4GbNi2Me55e1w7Tt1AVbXjoO3tSNQOo7QlSZqYwUmStFqoqsVJ3ksTnLYf8fAbgQtpws8g17TvL6QJV3u2I1cAJNlrnOPOBL5Fc08VwGt6F69ozzuPZrreILf1fa6BVZKkaWdwkiStcpJsUVWDRl+2a9+vGbBvIscDTwd+V1U3TVC3FFhCM0VvrC9zgJeOd0BVfSnJX4CvAusleVlVjR1/PE0Ym1VVZ4/YZ0nSKsTgJElaFZ2U5Gqae5suobkn9+HAW4DbgY+P2N67gT2AM5J8ArgUWIdmRb2nA6+rqj8AJ9CMaH09yWdo7mF6K7Boosar6rgk84HjgDlJXlRVdwBfB14MnJjk4zQr+y0G7gPsBhxfVceNeC2SpGlgcJIkrYreBzwLeBOwBbA2zf0/pwAfqKqLR2msqq5N8gjg32kWm7gPzTS5y4Ef0d4nVVWntc+AejtNaLsa+BxwPfCFjnOcmOTp7XHfS/KcqlqQ5J9oVgZ8KXAQzYjWVcBPaKYPSpJWA+lbVXXG2GON58/MC5OkEZ289FudD27VzJLk3LU2u/8Oi/702+nuiiRNqx133JHzzjvvvPEW0xmFy5FLkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJM9D2W91zursgSTOKwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJElqJdkpyXeTXJlkUZLrkpyZ5KM9NVckOX6ItnZNUkl2HfLcr02y7/L3XpI0lQxOkiQBSfYEzgDWBw4EngwcAPwM2Hs5mjwP2Ll9H8ZrgX2X4zySpJVg9nR3QJKkVcSBwBXAU6pqSc/2ryc5cNTGqupW4KyuuiRzqmrBqO1LklYuR5wkSWpsBNzQF5oAqKql/duSPC3JeUkWJLkkycv79v/NVL0kc5NclGS3JGcnWQB8IMkVwEOAJ7THVJK5k3x9kqQV4IiTpLutK9+9y0j1W3/w3KFra9Gi0TqTDF9bNVrbGtYZwCuTHAF8Gfi/QSGq9TDgo8AHgOuAVwBfSPLbqjq94zxbAkcDhwOXAAuALwHHAbfQTNkDuHX5L0WSNNkMTpIkNd4BPJDmvqYDgIVJzgaOBz5VVfN7ajcBHlNVVwIkOR3YHdgH6ApO9wKeXVU/6d3Yjj7dWlWd0/t6jhkvzW87bBuSpOE4VU+SJKCq/lxVuwI7AG8DfkAzfe7DwC+TbNxTfsFYaGqPXQhcBmw9xKlu6g9NkqRVnyNOkiT1qKrzgfMBkqxJM6XuzcDbaRaQAJg34NCFwJwhTnHtJHQTgKracdD2diRqh8k6jyTJESdJksZVVYuB97Yft5+sZiepHUnSSmRwkiQJSLLFOLu2a9+vmeIuLGK4EStJ0jRwqp4kSY2TklxNc2/TJTS/XHw48BbgduDjU3z+i4C9kzwfuBy4raouneJzSpKGZHCSJKnxPuBZwJuALYC1ae5HOgX4QFVdPMXnPxjYCjgKWA/4CbDrFJ9TkjQkg5MkSUBVfRP45hB124yzfde+z3OBTFTTt+8KmiXNJUmrIO9xkiRJkqQOBidJkiRJ6jBzp+ol3TW9agpXh11j1vC1S++cun6MasTv4S377DR07Q0Dnzwyvge8+azRDtDfGuXPcyr/PYwoOz5kpPr5758/dO2+W548Utuf2WK3oWsf+Jqfj9R2Zg3/34lasmSktiVJ0opzxEmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOhicJEmSJKmDwUmSJEmSOszc5zhJknQ3dtHVt7DNO06Y7m5I0l1ccfgzprsLy80RJ0mSJEnqYHCSJEmSpA4GJ0mSJEnqMHPvcaoarT6ZuraX3jla/Qhmb7Xl0LV/eMk2I7W9YPsFI9XXrcN/X9a+9/yR2r7xX3YeunaTI88cqe1RZPZo/2TqzhH/7Ef9uzVVbY/y72HEtue9Yvg/S4DnH3DKSPVrr7F46Nr/uf7BI7W97ZsuHLp26UgtS5KkVZ0jTpIkSZLUweAkSZoRkuyU5LtJrkyyKMl1Sc5M8tFp6Ms2SSrJvstx7K7tsbtOfs8kScvL4CRJWu0l2RM4A1gfOBB4MnAA8DNg72nsmiRphpi59zhJku5ODgSuAJ5SVUt6tn89yYHT0yVJ0kziiJMkaSbYCLihLzQBUFV/XasjyQuTnJzk2iQLklyc5PAk6/Uek+ToJLcn+fskP2y//mOSjyZZu692yyTfTHJbkluSfAPYvL8fSR6R5OtJrmjPfUWSryXZevK+DZKkqWJwkiTNBGcAOyU5IsmOScabUfEA4ARgf+CpwBHAC4AfDKhdE/g+cArwLOCLwJuAt48VJJnT7n8ycFDb1vXANwa0tw1wKfCvwFM+id0hAAAgAElEQVTadrYAfpFkkxGuVZI0DZyqJ0maCd4BPJDmvqYDgIVJzgaOBz5VVfMBquqwsQOShOYeqIuBnyT5h6rqXXN+LeDdVfWt9vOpSR4J7AO8t932MmA74FlV9f1224+SrAu8vLeDVXUccFzP+We1/buubfMTo150knPH2bXtqG1JkibmiJMkabVXVX+uql2BHYC30YwgPQT4MPDLJBsDJHlAOz3uT8CdwGLgJ20z2/U3SxNsel0I9E6t2w24rSc0jTm2v49J1k/y4SS/S7IEWALcDqw34NySpFWMI06SpBmjqs4HzgdIsiZwOPBm4O1JDgVOB+YD7wIua7/+f8C3gTl9zc2vqv4ngS8E1un5vDHNiFG/awds+xpN0DoU+AVwK004O3HAuYdSVTsO2t6ORO2wPG1KkgYzOEmSZqSqWpzkvTTBaXvgiTT3FO1aVWOjTCTZcAVOMw941IDtW/R+aM/xdOA9VXV4z/a1aRa2kCSt4gxOY6qmrOk7nvKIoWv/8MxZI7V9v20H/VJzsA2X/Gmkthf+crOR6ufMy9C1S25cf6S2t3rx74euXXTkSE2PpO68c+oaB8jw38ORjfJ3fMR/D7ft/eiha5/6+p+O1Pb1d4z4d2Xtm4euXfq20X5erYXXDF2btdfuLupte9Gikep1V0m2qKpB/0EcmwJ3DTC2ut4dfTWvWoFT/xh4QZJ/6puu95K+uqVABpz7FcBo/+GXJE0Lg5MkaSY4KcnVNPc2XUJzD+/DgbfQ3Ef0cZrwdBPw2SQH09zf9GLgYStw3mNoVto7Jsk7gd8AewK79xZV1a1JTgfeluRG4HLgCTSr+w2f9iVJ08bFISRJM8H7aELRm2iWEP8h8EaapcIfVVW/rKp5wDNogtSxNMuL3w7svbwnbVfre2J7nsNpVs3bEnjhgPJ9gNOAD9LcU/UIYA/gluU9vyRp5XHESZK02quqbwLfHKLuTGCXAbvSV7cvsO+A4w8BDunbdjXwvCHaHK9um766uf3HSpKmnyNOkiRJktTB4CRJkiRJHQxOkiRJktTB4CRJkiRJHQxOkiRJktTBVfUkSZqBtt/qnpx7+DOmuxuSNGM44iRJkiRJHQxOkiRJktRhxk7Vy+zRLu3yQx45dO2df7dgpLbXXnv4+vxx/ZHa/tOp9xm6dvEGNVLbbD3adS6935Kha++4Zr2R2t5mvXlD157xyp1Hanvjz505fHGN+D1cTd3y4kePVP+0A08fuvamJeuO1PYRW5wzUv2O73nN0LWbnDPCnz1Ahn8maS0e/t8DjPbfrFoyWtuSJGnFOeIkSZIkSR0MTpIkSZLUweAkSZIkSR0MTpIkSZLUweAkSZIkSR0MTpIkSZLUweAkSZIkSR0MTpIkSZLUweAkSdIASR6a5KgklydZmOT2JOclOSzJZlN0zl2SHJJkw6loX5K0/AxOkiT1SbIfcC7wSODDwFOBZwPfAvYBPjNFp94FOBgwOEnSKmb2dHdAkqRVSZKdgM8BJwN7VdWint0nJ/kITZCSJN2NzNjgVEuWjFS/zzN/MnTtj6974EhtX3/rPYauzaaLuot6zF9nzaFr17px1khtr33RuiPVLx3hb9Nao3WFH/z8H4dve/e/jNT2xp8brS+rq+tfu8vQtfd74W9GavuMG/9u6NqlZKS2n/6UJ41Uv8mNZ45UP5KqEWrvHK3ppSP2RVPpnUABr+wLTQBU1WLgBwBJ1gDeCrwcuB9wC3AS8G9VddXYMUn2AN4A7ABsAlwFnAq8s6pubGsOoRltArg8+eu/lftV1RWTeoWSpJHN2OAkSdKokswCngSc2xt8JvBp4JXAfwInAtsAhwK7JtlhLBQB9wfOoBnJuqWtezPw0yQPbcPY54GNaALWc4Br22PH3gf199xxdm07RN8lSSMwOEmStMwmwLrA5V2FSbYF/gX4ZFUd0LP9fOBs4E00o1dU1Wd69ocmRM0F/gA8Dfh+VV2V5Mq27HxHmSRp1eLiEJIkLZ/d2vdjejdW1c+Bi2lGrgBIslmSI5NcBSwBFtOEJoDtlrcDVbXjoBdwyfK2KUkazBEnSZKWuRGYTzOVrsvG7fugqXTXAFvDX++DOhnYnGYa3y+Bv9D88vIsYM4K9ViStFIYnCRJalXVnUlOAZ6eZKuqunqC8nnt++Y0iz302pImhAE8tH3tW1VfGitI8oBJ6rYkaSVwqp4kSXf1fiDAZ5Os1b8zyZpJngmc1m56Sd/+R9BMvzu13TS2ZuIdfU29asC5x1bxcxRKklYxjjhJktSjqs5O8i/AZ4Fzknwa+DWwJvCPNAtCXFRVz05yJPDGJAX8kGWr6v0R+Fjb5CXA74DD22l7NwLPBPYYcPqL2vfXJzmW5l6oC6uqP3RJklYyg5MkSX2q6otJzqFZGe/twBY0IeYy4KvAJ9vS19CEov2B17HsOU4HVdW8tq3F7QjVx2mWL18CnALsDoytojdmLvBh4KXAq2lmhtwPuGIKLlOSNAKDkyRJA1TVhcB+HTVLgQ+1r4nqLgaePGBX+uoKOLB9SZJWId7jJEmSJEkdZuyIU+3ysJHqH7DO8UPXXrPhPUdqe6+tLhi6dqNZt4/U9gazFk5Z26Oav3TtoWvXzJKR2r5hyQZD175w/ZtGavvGP/5l6NqFVSO1vXi0ctZfI91FrU1mrTdS23MXXDh07Z0M3w+AdbJ46Nqbl647UtuHHrPnSPUP3Xj4fxOjmrdo+L7fcsdo9/bf9x7D/729+JMPGaltSZK04hxxkiRJkqQOBidJkiRJ6mBwkiRJkqQOBidJkiRJ6mBwkiRJkqQOBidJkiRJ6mBwkiRJkqQOBidJkiRJ6mBwkiRJkqQOBidJkiRJ6mBwkiRJkqQOs6e7A1NlybqjXdp7zttz6Nr6w7ojtX3qrIcPXbvG4pGaZtaiDF07e8FobS+dNVr9HRvW0LWjXue9Lhm+7Y+sM/z3BGDBpsPXz7pjpKapEX81scaS4Wvn3LB0pLbXum34+iXrjNbxpWsOX7tow9HaXrDpSOX8ZO0RDxjBKP8mavbwf2cBrthk46Fr7//7Ef8xS5KkFeaIkyRJkiR1MDhJkiRJUgeDkyTpbiPJvkmq57UwyZ+S/DjJQUnuPd19lCStmgxOkqS7o/2AnYE9gNcBFwBvBy5Osvt0dkyStGqasYtDSJI0gYuq6pyez/+d5GPA/wLfTvL3VXXdoAOTrFtV81dKLyVJqwxHnCRJAqrqSuAtwPrAqwCSHJ3k9iQPS3JaktuBr44dk2T3JKcmuTXJ/CQ/S/Kk3naTbJrkyCR/TLIoyQ1t3e49NTskOT7J9W3NNUlOSHKflXP1kqQujjhJkrTMicCdwON7tq0FfBf4FHDY2MYkLwGOAb4HvAxYTBO4fpTkKVV1alt6LPCPwDuBy4ANgR2Ajdt27gH8D3AFzbTB64DNgd1oQpwkaRVgcJIkqVVV85PcCGzZs3lN4JCq+tLYhiTrAh8Hjq+qZ/dsPxE4D3g/sFO7eRfg81X1uZ42v9fz9YNoQtT+VdW7/Ztd/U1y7ji7tu06VpI0GqfqSZJ0V4Oeiv2dvs+7ABsBX0oye+xF8//Vk4BHJlmvrf05sG+SdyV5VJL+R0b/FrgJ+GCSVyUx9EjSKsjgJElSqw07GwPX9GyeX1W39pVu1r4fRzNFr/f1dprwtVFbszfwJeAVwNnAvPbeqc0BquoW4Ak0K/t9gGZlv6uTHDIgZN1FVe046AVcsjzXL0ka34ydqrfmKePNXhjsfqdMUUckSauTZwCzgLk922pA3Y3t+xuAs8Zp6zqAqroR+FfgX5NsBewFfJDmPqantjW/BF4IkGR7YH/gYGAhcPhyX40kadLM2OAkSdIoktwX+AhwK3BkR/nPgJuBB1fVJ4c9R1VdDfxXu6LeY8apuQh4U5J9aRaRkCStAgxOkqS7o+3be5JmA/cGHkfzUNw7gb2q6vqJDq6q25O8geYep41opuxdD2wKPAzYrKpeleSewI9pljC/BLgNeCTNSNO3AZLsCbyWZuW+39NM83sOzep7J0/mRUuSlp/BSZJ0d3RU+34HzcjRxTTT5z5fVTcM00BVHZvkSuBA4LM0S4dfT3Ov0jFt2UKa+5peCmxDs0LfH2im3324rflN24cDaVbzu4MmZO3bu5KfJGl6GZwkSXcbVXU0cPQI9fsC+06w/3Tg9An2LwJe03GOS4F9hu2TJGl6uKqeJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSJElSB4OTJEmSJHUwOEmSVllJ9k1SPa+FSS5L8skkmy1He3OTzO35vE3b7r6T2W9J0swze7o7IEnSEPYDLgHmAI8HDgKenuShVfWXae2ZJOluweAkSVodXFRV57Rf/zjJLODfgb2Ar0xft6ZWknWrav5090OS5FQ9SdLq6az2feskhySp/oKeaX7bjNp4kn9KcmaS+UluS3Jykp179u/Vtv2kAce+pt33kJ5tj0jy/SR/bqcbnp/kBeP09ylJvpRkHnD1qH2XJE0NR5wkSauj+7fvNwBbTWbDSfahGcX6EfAiYG3gQGBukidV1U+BE9pz7wec2tfEvsAvqupXbXu7AScBZwOvBm4BXgh8ox1ROrrv+C8A3wP2Ae7R0ddzx9m1beeFSpJGYnCSJK0OZiWZDawDPA54F3Ab8H3gNZN1kiRrAB8GLgSeXlVL2+0nAr8DPgg8pqoWJzkWeHWSDarq1rbuwcCjgNf2NPsp4FfAE6tqSbvtR0k2Ad6f5Jix87ROrqrXTdY1SZImh1P1JEmrg7OAxTRh6UTgOuBpVXXdJJ/nQcCWwLG9Yaaqbgf+G3h0knXbzUfRLFaxd8/x+wELga8BJHkAzejPV9rPs8de7XVs0Z6z13eG7WxV7TjoRbOQhiRpEjniJElaHfwzcDGwBLiuqq6dovNs3L4Pav8aml843guYX1W/bKfK7Qt8rg1DLwW+W1U3t8eMLZn+kfY1yCZ9n6fq2iRJK8DgJElaHVzcs6pev4UASdauqkU92/sDyTDmte+bD9i3JbAUuKln21HAJ5M8kGbkaLN225gb2/cPAN8e55yX9n3+m4UuJEnTz6l6kqTV3RXt+z/0bX/mcrR1Kc1Kdi9OkrGNSdYDngOc2bc8+FeBRTSjTvsBVwGnjO2sqkuB3wAPq6pzxnndthz9lCStZI44SZJWdycCfwa+kOTdNNP59gX+36gNVdXSJAfS3JN0fJLP0qyq9zaaKXrv6Ku/Kcn3gJcDGwEf6lvoAeBVwA+T/Ag4miaYbQRsBzyyqp4zaj8lSSufI06SpNVau6LdU2kWjjgW+AxwEfC+5WzvqzQP1t0Y+AbN1Ltbgd3apcj7HUUzRW9NmmDU396PaVbauxk4gmZE6tPA7sDJy9NHSdLK54iTJGmV1T7j6Ogh6n4BPGbAri/01e3a9/kKIPSpqu/RPEtpmD6eNKiNvpoLuevqe4NqjmaIa5UkTQ9HnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJkiSpg8FJkiRJkjoYnCRJq7wkNeRr1+nuqyRpZpo93R2QJGkIO/d9fgvwvAHbf71yuiNJursxOEmSVnlVdVbv5yTXDdo+niRzgIVVVVPQvSmVZE5VLZjufkjS3Z1T9SRJM0qSp7bT9l6Y5L/akPUXYO12/8OSHJ/k5iQLkpyXZJ++Nl7dtrH5OG0/umfbo5L8MMkNSRYluTrJD3qPTbJGkgOSXJhkYZI/J/lGkq372j8ryTlJnpzkF0kWAgdPwbdJkjQiR5wkSTPVR4HTgH2BewCLkzwU+BnwR+C1wC3t/q8k2aSqPjHKCZJsCPwPcDHwauAGYAvgicB6PaVHA3sDHwPeCmxKE4h+muThVTWvp3Zr4HPA+4Df0IQ+SdI0MzhJkmaqC6vqpb0bkry3/XLXqrqu/fqEJKcChyb5QlWNElQeAtwTOKSqftSz/Rs959wVeCnwuqr6VM/2M4BLgDdy11GlTYCdh5mGmOTccXZtO+wFSJKG41Q9SdJM9Z0B254I/KgnNI35ErAB8MgRz3EJcCvw0SSvTPLAATV7AncCX00ye+xFM+r1a2DXvvprh713S5K08jjiJEmaqa7t/ZBkFk04unZA7TXt+8ajnKCq5iV5AvBO4EPAhkmuAj4LfKCq7gQ2A2YBN43TTP9KgIP6N975dxy0vR2J2mHYdiRJ3QxOkqSZ6i4r6FXVnUluBTYfULtl+35j+76wfV+7r26TvzlJ1QXA85MEeCjwL8ChwO3AEW2bS4DH0ow89etfMW+1W/lPku4OnKonSbo7ORV4SpJN+7b/M82Uu3Paz1e07//QV/dP4zVcjQur6vU0YWhsxOd4ml9UblZV5wx4/WoFrkeStJI44iRJujs5GHgyMDfJ+4CbgZcBTwIO6FkY4mfA5cDH22dA3QY8H3hEb2NJnkuzKt/32vpZwAuAOcDJAFV1apJjaFbu+yTwU2A+zSjX44DzqurIqbpgSdLkMDhJku42quqXSR5Ls9T3p2mm4v0aeElVfaWnbnGSZwD/CXyeZgTpWODN3HXRiUtolgs/iCYILaRZmvwu7dGEq58Br6BZRQ+a+6p+Bvxicq9SkjQVDE6SpNVOOx3u9ePsOwnIBMdeADxjiHNcDOw+YFd6an4FvHCItgo4sn1NVPfoifZLkqaP9zhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRJkiR1MDhJkiRJUgeDkyRpUiSpIV+7ruB5Dmvb2XCI2quSfH7Idh+b5JAkG0xQ8+Yk1yeZlWT7tv6+o/RfkrR6mj3dHZAkzRg7931+C/C8Adt/vXK6A8AzgVuGrH0scDDweeDWcWqeC3y3qu5Msn1bfwpw5Yp2VJK0ajM4SZImRVWd1fs5yXWDtq9MVXV+V02Sdatq/hB1W9KEwPdORt8kSasXp+pJklYZ7RS4dye5NMmCJDcnuTDJ6weUb5HkG0luTfKnJJ/vn2bXP1UvySvaaX57JPlyknnAH5IcBnygLftjz7TC+/Q09xya0avTkrwC+Fq7/X976h/bcx3vaK9jUZLrknypDV+9/ftpkguS7JbkF0kWJvlDkret0DdSkjTpHHGSJK1K/g14N82ozk+BtYBtgXsNqP028HXgc8DDgPcBS4F/GeI8R7XHvwi4B/Dz9hyvBf4JuKGtu77nmOcC36+qxUm+B2wOHAq8CriwrflV+34ksB/wceCHwN+1tY9PsmNV/bmn3a2ALwOHAVcAzwc+lGTtqjpsootIcu44u7ad6DhJ0ugMTpKkVckuwPlV1Tsd7qRxao+sqo+1X5+S5EHAPgwXnH5YVW/s3ZDkj/+/vbsNtqsqDzj+f5AXMSIkYooVaoRBo5VCExU1BRIQjZ0RMgg6A8zIMKIfIDg1TkW/tIz2xY6lVJ0ptkoZRbRi6/iGJmqAUlLGkhQtmBcQCa8KCQqCSSTN0w9r3/bM5t67z71373POvf3/Ztbs3P2y1trrrnuyn7P2Xrv6539m5oO1bQuBkyijTmTmYxFxT7X5x723I0bE7wIXAldm5h/1rP8RcCvwPsqzUWMOB1Zm5tqx842IFwEfiohPZOZEz1tJkgbIW/UkSQMXEfvXUlSbfgAsjYhPRcSbIuKQSbL5eu3nHwHzIuKFfVThq1Os8ipgF7Cuj31PrZaf612ZmRuAu4HTavv/oidoGnMd8DzgdZMVlJlLx0vAlj7qKUmaAgMnSdJARcT+wDO1dF61+aPAHwPLgLXAzohYFxG/P05WO2s/766WB/dRjUemWO2zgW9l5u7GPWEscBuvjId7to/5+Tj7/ayWlyRpyLxVT5I0UJm5NyJeW1t9b7XtGeDjwMcj4lDgdOAvgXURcVSfgUtf1eh3x4iYD6yg3AbYj7GA7gj+LwAa89s8O1D6rXHyOKKWlyRpyBxxkiQNXGbeXkuPj7PPE5n5FeDvKM8Bdf2i2T3Vsj5idSZlVOyGPvdfXy3P710ZEa8HjgW+X9t/fkS8pbbuXODXlFsXJUkjwBEnSdLIiIgbgDuAjZSZ7V4GrKaMSN3bcfF3VsvVEXEdsBf4IeU2vbWZ+XRt/7EZ9C6KiKeB3wBbMvOuiLgaeH/17Nba6jw+AmwHPlHLZwfw2WpK9J8C76C8uPdPnBhCkkaHgZMkaZTcSJmI4SLgBZRb3dYCH8nMvR2X/T3gCsrzVhdT7sp4KfAm4N31nTNzW0R8ELgEuKXa/yTKNOrvAe6hTEm+mvL+p28Dl40zuvYQsAb4K+DVlCnQPwR8rN3TkyTNRGT2fZv3rHL6fufMzROTpCn67r7ro3kvjScizqW882lhZj7RQf7/Bjw/M09oOd+NS5YsWbJx40SveZKk/x+WLl3Kpk2bNlUzjs6IzzhJkjSBzLwuMw/qImiSJM0uBk6SJEmS1MBnnCRJGpLM/INh10GS1B9HnCRJkiSpgYGTJEmSJDUwcJIkSZKkBgZOkiRJktTAwEmSJEmSGhg4SZIkSVIDAydJkiRJamDgJEmSJEkNDJwkSZIkqYGBkyRJkiQ1MHCSJEmSpAYGTpIkSZLUwMBJkiRJkhoYOEmSJElSg/2HXQFJktS6RZs3b2bp0qXDrockDdXmzZsBFrWRl4GTJElzz/N37dr135s2bfrhsCsy4hZXyy1DrcXos52a2Ub9GUY7LQKebCMjAydJkuaeOwEy0yGnSUTERrCdmthOzWyj/sz2dvIZJ0mSJElqYOAkSZIkSQ3m7K163913fQy7DpIkSZLmBkecJEmSJKmBgZMkSZIkNYjMHHYdJEmSJGmkOeIkSZIkSQ0MnCRJkiSpgYGTJEmSJDUwcJIkSZKkBgZOkiRJktTAwEmSJEmSGhg4SZIkSVIDAydJkiRJamDgJEnSiIiIIyPi6oh4OCL2RMR9EXFlRMyfYj4LquPuq/J5uMr3yK7LHoSZ1jUi5kXEeRFxXURsiYinI+JXEXF7RKyJiAMnOC4nSbe1e5Yz18bvNCJuajjv505w3Ksi4ssR8WhE7I6IrRFxeUQc3N4ZzlwLfWl5Q/uMpaNqx82avhQRZ0fEJyPiloh4sqrjtdPMa8rtPUp9KTJz0GVKkqSaiDgG2AAsBL4GbAFeB6wAtgLLMnNnH/m8sMrn5cB64D+AxcCZwKPAGzLz3i7KHoQ26hoRK4FvA48DNwL3AAuAtwFHVPmflpm7a8clsB24ZpxsH8zMz0z7xFrWYn+6CTgFuHyCXT6amXtrx5xI6XsHAF8BHgBOBV4D3Epp2z1TP6t2tdSXFgEXTLD5OOAs4K7MfHXtuNnUl+4AjgeeAh6kfJ58ITPPn2I+U27vketLmWkymUwmk2nICVgLJLC6tv6Kav1Vfebz6Wr/K2rrL63Wf6ersmdLOwEnAOcBB9bWHwJsrPJZM85xCdw07DYYcH+6qVwu9l3uc4AfV2Wc0bN+P8qFbwKXDbt92myjSfL/YpXPpbO8L60AjgUCWF7V/dqu23sU+5IjTpIkDVlEHA38BLgPOCYz9/VsOwR4hHLRsjAzn54kn3nAY8A+4MWZ+auebftVZSyqyri3zbIHYRB1jYhzgS8A38zMt9W2JXBzZi6f1gkMSJvtNDbilJnRZ9mnAt8H/jUzT5mgXtuBl+UQL0K77kvVyO9DlL/Fl2TmL2rbZ0VfqouI5ZRR2imNOE2nvUexL/mMkyRJw3dqtVzXe0EBUAU/twLPA17fkM8bgIOBW3uDpiqffcC66scVHZQ9CIOo6zPVcu8E2w+LiAsj4sMRcXFEjEK71LXeThHxzoi4LCLeHxFvjYiDGsr+Tn1DFaxvA14KHN1v2R3pui9dABwEXF8PmnrMhr7Ulum098j1JQMnSZKG7xXVctsE2++uli/vIJ+2yh6EQdT1wmr5rIu1yvHAZ4E/Az4F/HtE3BERx82gzLZ10U5fAv4C+GvgBuD+iDh7QGV3oet6vrtafnqSfWZDX2rLnPhsMnCSJGn4Dq2WT0ywfWz9YR3k01bZg9BpXSPiEmAlcAdw9Ti7XAEsA15EeR7qtZRnLY4H1kfES6ZTbgfabKevUSbNOJIymrmYEkAdBvxTRLy1w7K71Fk9I+IUSjvdlZkbJthttvSltsyJzyYDJ0mSRt/Y8yUzvY9/Ovm0VfYgTLuuEXEWcCXwM+DtmflMfZ/MXJOZGzJzR2Y+lZm3Z+Y5wD8DhwMfmEHdB6nvdsrMv8nMb2bmQ5m5OzO3ZuaHgTWU68g/76rsIZtJPd9TLSccbZpDfakts+KzycBJkqThG/vm9NAJtr+gtl+b+bRV9iB0UteIWEW5Fe1RYHnWpmvvw1XV8uQpHteVQfxOP0N5DuyE6uH+QZbdhq760gLg7cAu4PPTqNeo9aW2zInPJgMnSZKGb2u1nOhe/WOr5UT3+s8kn7bKHoTW6xoR5wDXAz+nzB63teGQ8TxWLedN49gudP47zfKOq7EJSHrPe7b0p67q+S7KpBBfzsxfTqNeo9aX2jInPpsMnCRJGr4bq+Wbq2nD/1f1bf4yyjfYtzXkc1u137LaKMDYdORvrpXXZtmD0Gpdq6nHvwg8TAma7m44ZCJjM4FNdaSqK53/TiPiFcB8SvC0o2fT+mq5cpxjjqZcBG9n+G3VVRtdVC3/fpr1GrW+1KeWsJ4AAALGSURBVJbptPfI9SUDJ0mShiwzf0KZKnwRcHFt8+WUb58/1/s+mYhYHBGLa/k8Rbk9aB7wp7V8LqnyX9t7K9p0yh6WttqpWv8uSlvdD5zcdHteRCyp3pNVX/97lFnRAK7t/2y601Y7RcTR401SEBGHA/9Y/filzOyduv1mYDNwckSc0XPMfsDHqh+vGuY7nKDdvtSz/STglcCdk0wKMav60lRFxAFVOx3Tu36anzMj15d8Aa4kSSOgutDYACykzGS2GTiR8s6lbcAbM3Nnz/4JUH8xafXizQ2Ub2PXAz+gXMydSXmG543VRcy0yx6mNtopIlYA36N8gXw18MA4Rf0yM6/sOeYa4CxKmz4A7KHMnLYSeA7wD8B7hx0QjGmpnS6gPMt0M+Vlo48DvwP8IeW5k9uB0+u3pEXEiZR2OoAyU9z9wGnAayjv6zktM/e0fc5T1dbfXM/2zwPnA5dm5icnKfcaZldfWgWsqn48AngLZZTnlmrdjsz8QLXvIuCnwPbMXFTLZ8qfMyPXlzLTZDKZTCbTCCTgKMo3+Y8Av6HchvK3wIJx9s3y3/i4+Syojtte5fMIJUA4so2yh51m2k6Ul5NmQ7qvdswq4F+Ae4Ane9r1G8AZw26TjtrpOOAa4L+AnZSXAz9OuWBeDRw4Sdmvojw7toMSGGyjjCwcPOx2abONerbNp9xq9mvgsIYyZ1Vfooxe9/W3QhlRetbfz3TaexT7kiNOkiRJktTAZ5wkSZIkqYGBkyRJkiQ1MHCSJEmSpAYGTpIkSZLUwMBJkiRJkhoYOEmSJElSAwMnSZIkSWpg4CRJkiRJDQycJEmSJKmBgZMkSZIkNTBwkiRJkqQGBk6SJEmS1MDASZIkSZIaGDhJkiRJUgMDJ0mSJElqYOAkSZIkSQ0MnCRJkiSpgYGTJEmSJDUwcJIkSZKkBgZOkiRJktTgfwC2XoDeAUyLnQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x648 with 2 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 206,
       "width": 423
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import helper module (should be in the repo)\n",
    "import helper\n",
    "\n",
    "# Test out your network!\n",
    "\n",
    "model.eval()\n",
    "\n",
    "dataiter = iter(testloader)\n",
    "images, labels = next(dataiter)\n",
    "img = images[0]\n",
    "# Convert 2D image to 1D vector\n",
    "img = img.view(1, 784)\n",
    "\n",
    "# Calculate the class probabilities (softmax) for img\n",
    "with torch.no_grad():\n",
    "    output = model.forward(img)\n",
    "\n",
    "ps = torch.exp(output)\n",
    "\n",
    "# Plot the image and probabilities\n",
    "helper.view_classify(img.view(1, 28, 28), ps, version='Fashion')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Up!\n",
    "\n",
    "In the next part, I'll show you how to save your trained models. In general, you won't want to train a model everytime you need it. Instead, you'll train once, save it, then load the model when you want to train more or use if for inference."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}