Chapter 3.4 电影评论分类-二分类问题.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.1 IMDB数据集\n",
    "它包含来自互联网电影数据库（IMDB）的 50,000条严重两极分化的评论。数据集被分为用于训练的25,000条评论与用于测试的25,000条评论，训练集和测试集都包含50%的正面评论和50%的负面评论。 \n",
    "\n",
    "该数据已经进行预处理，评论已经被转换成了整数序列，其中每个整数代表字典中的某个单词。\n",
    "\n",
    "我们需要关注的是模型在新数据上的性能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras import losses\n",
    "from tensorflow.keras import metrics\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 1s 0us/step\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<__array_function__ internals>:5: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "/Users/yihuawang/conda/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:159: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "/Users/yihuawang/conda/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:160: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    }
   ],
   "source": [
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
    "# num_words 仅保留训练数据中前10,000最常出现的单词。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data[0] # 代表一条由单词索引组成的评论\n",
    "train_data # 代表评论组成的列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_labels[0] # 是否为正面或负面评论\n",
    "train_labels #代表由0/1组陈的列表"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decoding reviews back to text\n",
    "将评论转化成文本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 0s 0us/step\n"
     ]
    }
   ],
   "source": [
    "word_index = imdb.get_word_index() # word_index是一个将单词映射为整数索引的字典\n",
    "\n",
    "reverse_word_index = dict( # 字典功能 \n",
    "    [(value, key) for (key, value) in word_index.items()]) # 键值颠倒，将整数索引映射为单词\n",
    "\n",
    "decoded_review = \" \".join(\n",
    "    [reverse_word_index.get(i - 3, \"?\") for i in train_data[0]]) # 对train_data[0]中的各个i, 利用reverse_word_index进行映射\n",
    "# 减去3是因为 0对应 padding; 2对应 start of sequence; 3对应 unknown. 它们通通被映射为?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reverse_word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "word_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoded_review = \" \".join(\n",
    "    [reverse_word_index.get(i - 3, \"?\") for i in train_data[i]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all\""
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "decoded_review"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.2 Preparing the data\n",
    "准备数据  \n",
    "你不能将整数序列直接输入神经网络。你需要将列表转换为张量。转换方法有以下两种。\n",
    "\n",
    "1. 填充列表，使其具有相同的长度，再将列表转换成形状为(samples,  word_indices)的整数张量，然后网络第一层使用能处理这种整数张量的层（即  Embedding层，本书后面会详细介绍）。  \n",
    "\n",
    "2. 对列表进行  one-hot编码，将其转换为  0和  1组成的向量。举个例子，序列[3,  5]将会被转换为 10  000维向量，只有索引为  3和  5的元素是   1，其余元素都是 0。然后网络第一层可以用Dense层，它能够处理浮点数向量数据。  \n",
    "\n",
    "这里，我们采用后一种方法将数据向量化。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "def vectorize_sequences(sequences, dimension=10000): # 取10,000是因为我们之前限定了10,000个最常见的单词，因此单词索引不会超过10,000\n",
    "    # 以train_data[0]为例子\n",
    "    results = np.zeros((len(sequences), dimension)) #产生一个 218* 10,000的矩阵\n",
    "    for i, sequence in enumerate(sequences): #enumerate() 会产生[(0, 1),(1, 14),...,(217, 32)], \n",
    "                                             #前一个数字代表position, 后一个数字代表单词索引\n",
    "        for j in sequence:\n",
    "            results[i, j] = 1.\n",
    "    return results\n",
    "x_train = vectorize_sequences(train_data)\n",
    "x_test = vectorize_sequences(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in enumerate(train_data[0]):\n",
    "    print(i)\n",
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0., 1., 1., ..., 0., 0., 0.],\n",
       "       [0., 1., 1., ..., 0., 0., 0.],\n",
       "       [0., 1., 1., ..., 0., 0., 0.],\n",
       "       ...,\n",
       "       [0., 1., 1., ..., 0., 0., 0.],\n",
       "       [0., 1., 1., ..., 0., 0., 0.],\n",
       "       [0., 1., 1., ..., 0., 0., 0.]])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = np.asarray(train_labels).astype(\"float32\")\n",
    "y_test = np.asarray(test_labels).astype(\"float32\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.3 Building your model\n",
    "构建网络\n",
    "输入数据是向量，而标签是标量（1和0），这是你会遇到的最简单的情况。有一类网络在这种问题上表现很好，就是带有relu激活的全连接层（Dense）的简单堆叠，比如Dense(16, activation=’relu’)。参数（16）是该层隐藏单元的个数。  \n",
    "\n",
    "一个隐藏单元（hidden unit）是该层表示空间的一个维度。每个带有relu激活的Dense层都实现了下列张量运算：  \n",
    "output = relu(dot(W, input) + b)  \n",
    "\n",
    "16个隐藏单元对应的权重矩阵W的形状为(input_dimension,16)，与W做点积相当于将输入数据投影到16维表示空间中（然后再加上偏置向量 b并应用relu运算）。  \n",
    "\n",
    "你可以将表示空间的维度直观地理解为“网络学习内部表示时所拥有的自由度”。隐藏单元越多（即更高维的表示空间），网络越能够学到更加复杂的表示，但网络的计算代价也变得更大，而且可能会导致学到不好的模式（这种模式会提高训练数据上的性能，但不会提高测试数据上的性能。  \n",
    "\n",
    "我们所要确定的是两个关键结构:  \n",
    "1) 网络到底有多少层  \n",
    "2) 每层有多少个隐藏单元  \n",
    "\n",
    "relu (rectified linear unit) 整流线型单元函数是将所有负值归零, 而sigmoid 函数将任意值压缩至[0,1]区间内。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 在此，我们构建两个中间层，每层都有16个隐藏单元；  \n",
    "# 第三层输出一个标量，预测当前评论的情感。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,))) # hidden unit=16\n",
    "model.add(layers.Dense(16, activation='relu')) #hidden unit=16\n",
    "model.add(layers.Dense(1, activation='sigmoid')) # hidden unit=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 什么是激活函数，为什么要使用激活函数？\n",
    "\n",
    "如果没有relu等激活函数（也叫非线性），Dense层将只包含两个线性运算——点积和加法：\n",
    "output = dot(W, input) + b (vs output = relu(dot(W, input) + b))  \n",
    "\n",
    "这样Dense层就只能学习输入数据的线性变换（仿射变换）：该层的假设空间是从输入数据到 16位空间所有可能的线性变换集合。这种假设空间非常有限，无法利用多个表示层的优势，因为多个线性层堆叠实现的仍是线性运算，添加层数并不会扩展假设空间。\n",
    "\n",
    "为了得到更丰富的假设空间，从而充分利用多层表示的优势，你需要添加非线性或激活函数。relu是深度学习中最常用的激活函数，但还有许多其他函数可选，它们都有类似的奇怪名称，比如prelu、elu等。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 选择损失函数和优化器\n",
    "二分类问题--binary crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(optimizer=\"rmsprop\",\n",
    "              loss=\"binary_crossentropy\",\n",
    "              metrics=[\"accuracy\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 配置优化器\n",
    "# 1. 通过向optimizer参数传入一个优化器类实例来实现\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001), # 定义lr\n",
    "        loss='binary_crossentropy',\n",
    "        metrics=['accuracy'])\n",
    "\n",
    "# 2. 通过向loss和metrics参数传入函数对象来实现\n",
    "model.compile(optimizer=optimizers.RMSprop(lr=0.001),\n",
    "        loss=losses.binary_crossentropy,\n",
    "        metrics=[metrics.binary_accuracy])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.4 Validating your approach\n",
    "验证你的方法  \n",
    "\n",
    "在原始训练数据留出10,000个样本作为验证集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_val = x_train[:10000]\n",
    "partial_x_train = x_train[10000:]\n",
    "y_val = y_train[:10000]\n",
    "partial_y_train = y_train[10000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "30/30 [==============================] - 2s 55ms/step - loss: 0.5821 - acc: 0.7138 - val_loss: 0.3573 - val_acc: 0.8726\n",
      "Epoch 2/20\n",
      "30/30 [==============================] - 1s 18ms/step - loss: 0.2954 - acc: 0.9060 - val_loss: 0.3038 - val_acc: 0.8828\n",
      "Epoch 3/20\n",
      "30/30 [==============================] - 1s 17ms/step - loss: 0.2048 - acc: 0.9330 - val_loss: 0.2809 - val_acc: 0.8890\n",
      "Epoch 4/20\n",
      "30/30 [==============================] - 1s 18ms/step - loss: 0.1620 - acc: 0.9520 - val_loss: 0.2785 - val_acc: 0.8906\n",
      "Epoch 5/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.1245 - acc: 0.9630 - val_loss: 0.3064 - val_acc: 0.8782\n",
      "Epoch 6/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.1155 - acc: 0.9654 - val_loss: 0.3019 - val_acc: 0.8874\n",
      "Epoch 7/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.0828 - acc: 0.9784 - val_loss: 0.3223 - val_acc: 0.8816\n",
      "Epoch 8/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0694 - acc: 0.9836 - val_loss: 0.3440 - val_acc: 0.8825\n",
      "Epoch 9/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.0559 - acc: 0.9866 - val_loss: 0.3716 - val_acc: 0.8749\n",
      "Epoch 10/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.0494 - acc: 0.9885 - val_loss: 0.3870 - val_acc: 0.8763\n",
      "Epoch 11/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0367 - acc: 0.9923 - val_loss: 0.4128 - val_acc: 0.8762\n",
      "Epoch 12/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0318 - acc: 0.9937 - val_loss: 0.4434 - val_acc: 0.8749\n",
      "Epoch 13/20\n",
      "30/30 [==============================] - 1s 17ms/step - loss: 0.0236 - acc: 0.9951 - val_loss: 0.4776 - val_acc: 0.8730\n",
      "Epoch 14/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0183 - acc: 0.9978 - val_loss: 0.5043 - val_acc: 0.8702\n",
      "Epoch 15/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0126 - acc: 0.9988 - val_loss: 0.5362 - val_acc: 0.8695\n",
      "Epoch 16/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0095 - acc: 0.9994 - val_loss: 0.5704 - val_acc: 0.8678\n",
      "Epoch 17/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0120 - acc: 0.9980 - val_loss: 0.6063 - val_acc: 0.8670\n",
      "Epoch 18/20\n",
      "30/30 [==============================] - 0s 16ms/step - loss: 0.0067 - acc: 0.9995 - val_loss: 0.6433 - val_acc: 0.8669\n",
      "Epoch 19/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.0059 - acc: 0.9995 - val_loss: 0.6709 - val_acc: 0.8669\n",
      "Epoch 20/20\n",
      "30/30 [==============================] - 0s 15ms/step - loss: 0.0032 - acc: 0.9999 - val_loss: 0.7011 - val_acc: 0.8658\n"
     ]
    }
   ],
   "source": [
    "#训练\n",
    "model.compile(optimizer='rmsprop',\n",
    "        loss='binary_crossentropy',\n",
    "        metrics=['acc'])\n",
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=20, # 模型训练20轮次\n",
    "                    batch_size=512, #batch_size 批量大小，即一次训练所选取的样本数，512个样本促成的小批量\n",
    "                    validation_data=(x_val, y_val)) #将验证数据传入validation_data 来监控损失和精度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['loss', 'acc', 'val_loss', 'val_acc'])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "history_dict = history.history\n",
    "history_dict.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd5hU5fn/8ffNgiKCGoqxIC2iiAEWXAFBCZYkgAQsqCABFBWxRo0KSlR+EjT5aozBYBQV6yqaqNiwBAvYpYgICIoGFEUElCZFFu7fH89ZHNbZvmdndufzuq65dk6ZM/eenT33POU8j7k7IiKSuWqkOgAREUktJQIRkQynRCAikuGUCEREMpwSgYhIhlMiEBHJcEoEUqHM7HkzG1LR+6aSmS0xs+NiOK6b2YHR8zvM7JqS7FuG9xloZi+VNc4ijtvdzJZV9HGl8tVMdQCSema2IWGxDrAF2BYtn+vuuSU9lrv3jGPf6s7dh1fEccysGfA/oJa750XHzgVK/DeUzKNEILh73fznZrYEONvdpxbcz8xq5l9cRKT6UNWQFCq/6G9mI8zsa+BeM/uZmT1rZivN7LvoeeOE17xmZmdHz88wszfM7OZo3/+ZWc8y7tvczKab2Xozm2pm483soULiLkmMY8zszeh4L5lZw4Ttg8xsqZmtNrNRRZyfzmb2tZllJaw70czmRs87mtnbZrbGzJab2T/NbJdCjnWfmf05YfmK6DVfmdnQAvseb2bvm9k6M/vCzEYnbJ4e/VxjZhvM7Ij8c5vw+i5mNsPM1kY/u5T03BTFzA6JXr/GzOabWZ+Ebb3MbEF0zC/N7PJofcPo77PGzL41s9fNTNelSqYTLsXZB6gPNAWGET4z90bLTYBNwD+LeH0nYBHQEPg/4B4zszLs+zDwHtAAGA0MKuI9SxLj6cCZwN7ALkD+hak18K/o+PtF79eYJNz9HeB74JgCx304er4NuDT6fY4AjgXOLyJuohh6RPH8GmgJFGyf+B4YDOwFHA+cZ2YnRNu6RT/3cve67v52gWPXB54DxkW/2y3Ac2bWoMDv8JNzU0zMtYBngJei110E5JrZwdEu9xCqGesBvwReidb/EVgGNAJ+DlwNaNybSqZEIMXZDlzn7lvcfZO7r3b3x919o7uvB8YCvyri9Uvd/S533wbcD+xL+Icv8b5m1gQ4HLjW3X9w9zeApwt7wxLGeK+7f+zum4DHgOxofT/gWXef7u5bgGuic1CYR4ABAGZWD+gVrcPdZ7n7O+6e5+5LgDuTxJHMqVF889z9e0LiS/z9XnP3D919u7vPjd6vJMeFkDg+cfcHo7geARYCv0vYp7BzU5TOQF3gL9Hf6BXgWaJzA2wFWpvZHu7+nbvPTli/L9DU3be6++uuAdAqnRKBFGelu2/OXzCzOmZ2Z1R1so5QFbFXYvVIAV/nP3H3jdHTuqXcdz/g24R1AF8UFnAJY/w64fnGhJj2Szx2dCFeXdh7Eb79n2RmuwInAbPdfWkUx0FRtcfXURw3EEoHxdkpBmBpgd+vk5m9GlV9rQWGl/C4+cdeWmDdUmD/hOXCzk2xMbt7YtJMPO7JhCS51MymmdkR0fqbgMXAS2b2mZmNLNmvIRVJiUCKU/Db2R+Bg4FO7r4HP1ZFFFbdUxGWA/XNrE7CugOK2L88MS5PPHb0ng0K29ndFxAueD3ZuVoIQhXTQqBlFMfVZYmBUL2V6GFCiegAd98TuCPhuMV9m/6KUGWWqAnwZQniKu64BxSo399xXHef4e59CdVGkwklDdx9vbv/0d1bEEoll5nZseWMRUpJiUBKqx6hzn1NVN98XdxvGH3DngmMNrNdom+TvyviJeWJ8T9AbzM7MmrYvZ7i/08eBi4mJJx/F4hjHbDBzFoB55UwhseAM8ysdZSICsZfj1BC2mxmHQkJKN9KQlVWi0KOPQU4yMxON7OaZnYa0JpQjVMe7xLaLq40s1pm1p3wN5oU/c0Gmtme7r6VcE62AZhZbzM7MGoLyl+/LflbSFyUCKS0bgV2A1YB7wAvVNL7DiQ0uK4G/gw8SrjfIZkyx+ju84ELCBf35cB3hMbMojwCdAdecfdVCesvJ1yk1wN3RTGXJIbno9/hFUK1ySsFdjkfuN7M1gPXEn27jl67kdAm8mbUE6dzgWOvBnoTSk2rgSuB3gXiLjV3/wHoQygZrQJuBwa7+8Jol0HAkqiKbDjw+2h9S2AqsAF4G7jd3V8rTyxSeqZ2GamKzOxRYKG7x14iEanuVCKQKsHMDjezX5hZjah7ZV9CXbOIlJPuLJaqYh/gCULD7TLgPHd/P7UhiVQPqhoSEclwqhoSEclwVa5qqGHDht6sWbNUhyEiUqXMmjVrlbs3SratyiWCZs2aMXPmzFSHISJSpZhZwTvKd1DVkIhIhlMiEBHJcEoEIiIZrsq1ESSzdetWli1bxubNm4vfWVKqdu3aNG7cmFq1aqU6FBGJVItEsGzZMurVq0ezZs0ofM4TSTV3Z/Xq1SxbtozmzZunOhwRicRaNWRmPcxskZktTjbOeDQd35zoMc/MtkWjRZbK5s2badCggZJAmjMzGjRooJKbSJqJLRFEk4CMJ4xG2BoYEE0DuIO73+Tu2e6eDVwFTHP3b8v4fuUNWSqB/k4i6SfOEkFHYLG7fxYNUTuJMFBYYQYQTfEnIiI/2rQJbroJ3nornuPHmQj2Z+fp9pax83R4O0STb/QAHo8xntisXr2a7OxssrOz2Weffdh///13LP/www9FvnbmzJlcfPHFxb5Hly5dKiTW1157jd69e1fIsUQkXnl5cPfd0LIlXHklPPNMPO8TZyJIVgdQ2Ah3vwPeLKxayMyGmdlMM5u5cuXKcgeWmwvNmkGNGuFnbm75jtegQQPmzJnDnDlzGD58OJdeeumO5V122YW8vLxCX5uTk8O4ceOKfY+34voqICJpxx0efxx++Us45xw44ACYNg1uvDGe94szESxj53lXGxPmNU2mP0VUC7n7BHfPcfecRo2SDpVRYrm5MGwYLF0aTvbSpWG5vMmgoDPOOIPLLruMo48+mhEjRvDee+/RpUsX2rdvT5cuXVi0aBGw8zf00aNHM3ToULp3706LFi12ShB169bdsX/37t3p168frVq1YuDAgeSPIDtlyhRatWrFkUceycUXX1zsN/9vv/2WE044gbZt29K5c2fmzp0LwLRp03aUaNq3b8/69etZvnw53bp1Izs7m1/+8pe8/vrrFXvCRASAV16BTp2gXz/IyoLJk0OVULduxb+2rOLsPjoDaGlmzQkTWPdn57lVATCzPYFf8ePUdbEaNQo2btx53caNYf3AgRX7Xh9//DFTp04lKyuLdevWMX36dGrWrMnUqVO5+uqrefzxn9aELVy4kFdffZX169dz8MEHc9555/2kz/3777/P/Pnz2W+//ejatStvvvkmOTk5nHvuuUyfPp3mzZszYMCAYuO77rrraN++PZMnT+aVV15h8ODBzJkzh5tvvpnx48fTtWtXNmzYQO3atZkwYQK//e1vGTVqFNu2bWNjwZMoIuUyezZcdRW89FIoAUycCIMHh2QQt9gSgbvnmdmFwItAFjDR3eeb2fBo+x3RricCL7n793HFkujzz0u3vjxOOeUUsqK/4tq1axkyZAiffPIJZsbWrVuTvub4449n1113Zdddd2XvvfdmxYoVNG7ceKd9OnbsuGNddnY2S5YsoW7durRo0WJH//wBAwYwYcKEIuN74403diSjY445htWrV7N27Vq6du3KZZddxsCBAznppJNo3Lgxhx9+OEOHDmXr1q2ccMIJZGdnl+vciEiweDH86U/w6KNQvz787W9w/vlQu3blxRDrfQTuPsXdD3L3X7j72GjdHQlJAHe/z937xxlHoiZNSre+PHbfffcdz6+55hqOPvpo5s2bxzPPPFNoX/pdd911x/OsrKyk7QvJ9inLBEPJXmNmjBw5krvvvptNmzbRuXNnFi5cSLdu3Zg+fTr7778/gwYN4oEHHij1+4nIj5YvDxf8Qw4JjcCjRsFnn8Fll1VuEoAMHGto7FioU2fndXXqhPVxWrt2LfvvHzpN3XfffRV+/FatWvHZZ5+xZMkSAB599NFiX9OtWzdyo8aR1157jYYNG7LHHnvw6aef0qZNG0aMGEFOTg4LFy5k6dKl7L333pxzzjmcddZZzJ49u8J/B5FMsHZtuOgfeCDcdVdoo/z0U/jzn2HPPVMTU7UYYqI08tsBRo0K1UFNmoQkUNHtAwVdeeWVDBkyhFtuuYVjjjmmwo+/2267cfvtt9OjRw8aNmxIx44di33N6NGjOfPMM2nbti116tTh/vvvB+DWW2/l1VdfJSsri9atW9OzZ08mTZrETTfdRK1atahbt65KBCKltHkzjB8PN9wA334L/fvDmDEhIaRalZuzOCcnxwtOTPPRRx9xyCGHpCii9LFhwwbq1q2Lu3PBBRfQsmVLLr300lSH9RP6e0km2bQp3Avw17/Cl1/Cb38buoG2b1+5cZjZLHfPSbYt46qGqrO77rqL7OxsDj30UNauXcu5556b6pBEMtb334eG3xYt4OKLoXnz0DX0hRcqPwkUJ+OqhqqzSy+9NC1LACKZZN26UAV0yy2wahUccwxMmgS/+lWqIyucEoGISAX47jsYNw7+8Y/wvGfP0C20gkaHiZUSgYhIOaxaBbfeCrfdFkoDffuGBJCTtDY+PSkRiIiUwYoVoQ3g9tvD6AT9+oXeiO3apTqy0lMiEBEphS+/DENCT5gAW7bAgAFw9dXQunXxr01X6jVUAbp3786LL76407pbb72V888/v8jX5HeD7dWrF2vWrPnJPqNHj+bmm28u8r0nT57MggULdixfe+21TJ06tTThJ6XhqkV29vnn4U7gFi1CY3D//rBwITz0UNVOAqBEUCEGDBjApEmTdlo3adKkEg38BmHU0L322qtM710wEVx//fUcd9xxZTqWiPxUXh785S9w0EHhfoAzz4RPPgmDwrVsmeroKoYSQQXo168fzz77LFu2bAFgyZIlfPXVVxx55JGcd9555OTkcOihh3LdddclfX2zZs1YtWoVAGPHjuXggw/muOOO2zFUNYR7BA4//HDatWvHySefzMaNG3nrrbd4+umnueKKK8jOzubTTz/ljDPO4D//+Q8AL7/8Mu3bt6dNmzYMHTp0R3zNmjXjuuuuo0OHDrRp04aFCxcW+ftpuGrJVB98EIaEvuoq6N07DAVxxx1hHpPqpNq1EVxyCcyZU7HHzM4OvQIK06BBAzp27MgLL7xA3759mTRpEqeddhpmxtixY6lfvz7btm3j2GOPZe7cubRt2zbpcWbNmsWkSZN4//33ycvLo0OHDhx22GEAnHTSSZxzzjkA/OlPf+Kee+7hoosuok+fPvTu3Zt+/frtdKzNmzdzxhln8PLLL3PQQQcxePBg/vWvf3HJJZcA0LBhQ2bPns3tt9/OzTffzN13313o76fhqiXTbNkShp658cYwIui//x0ag6srlQgqSGL1UGK10GOPPUaHDh1o37498+fP36kap6DXX3+dE088kTp16rDHHnvQp0+fHdvmzZvHUUcdRZs2bcjNzWX+/PlFxrNo0SKaN2/OQQcdBMCQIUOYPn36ju0nnXQSAIcddtiOgeoK88YbbzBo0CAg+XDV48aNY82aNdSsWZPDDz+ce++9l9GjR/Phhx9Sr169Io8tkm7eew8OOyyMA9S/PyxYUL2TAFTDEkFR39zjdMIJJ3DZZZcxe/ZsNm3aRIcOHfjf//7HzTffzIwZM/jZz37GGWecUejw0/nMks3wGWY8mzx5Mu3ateO+++7jtddeK/I4xY0hlT+UdWFDXRd3rPzhqo8//nimTJlC586dmTp16o7hqp977jkGDRrEFVdcweDBg4s8vkg62LgRrrsu3BG8337w3HPQq1eqo6ocKhFUkLp169K9e3eGDh26ozSwbt06dt99d/bcc09WrFjB888/X+QxunXrxpNPPsmmTZtYv349zyTMVL1+/Xr23Xdftm7dumPoaIB69eqxfv36nxyrVatWLFmyhMWLFwPw4IMP8qsy3uOu4aqlups+PfT/v/lmOPtsmDcvc5IAVMMSQSoNGDCAk046aUcVUbt27Wjfvj2HHnooLVq0oGvXrkW+vkOHDpx22mlkZ2fTtGlTjjrqqB3bxowZQ6dOnWjatClt2rTZcfHv378/55xzDuPGjdvRSAxQu3Zt7r33Xk455RTy8vI4/PDDGT58eJl+Lw1XLdXV+vUwcmS4KaxFizAo3NFHpzqqyqdhqKXS6e8l6eDFF8OkMF98AX/4Q5gYJmFSwWpHw1CLiES++y7cC9CjR5id8M034e9/r95JoDhKBCKSMZ58MtwF/OCDYViI99+HI45IdVSpF2siMLMeZrbIzBab2chC9uluZnPMbL6ZTSvre1W1Kq5Mpb+TpMKKFXDaaXDSSfDzn8OMGeE+gcqeJD5dxZYIzCwLGA/0BFoDA8ysdYF99gJuB/q4+6HAKWV5r9q1a7N69WpdZNKcu7N69Wpq679PKok73HMPtGoFkyeHewNmzEi/GcJSLc5eQx2Bxe7+GYCZTQL6Aol3VJ0OPOHunwO4+zdleaPGjRuzbNkyVq5cWc6QJW61a9emcePGqQ5DMsDHH4fG4GnT4KijwmihrVqlOqr0FGci2B/4ImF5GdCpwD4HAbXM7DWgHvAPd/9Jf0MzGwYMA2jSpMlP3qhWrVo0b968YqIWkSrthx/CMNFjxsBuu8Fdd8HQoVBDLaKFijMRJLtFtmDdTU3gMOBYYDfgbTN7x90/3ulF7hOACRC6j8YQq4hUA2+/HUoB8+bBqaeGaSP32SfVUaW/OHPkMuCAhOXGwFdJ9nnB3b9391XAdKAKzu8jIqm0bh1ceCF07Qpr1sDTT8OjjyoJlFSciWAG0NLMmpvZLkB/4OkC+zwFHGVmNc2sDqHq6KMYYxKRauapp0KX0Ntvh4suCoPE/e53qY6qaomtasjd88zsQuBFIAuY6O7zzWx4tP0Od//IzF4A5gLbgbvdfV5cMYlI9fHVV3DxxfD449CmDTzxBHTsmOqoqqZqMcSEiGSO7dtDD6ARI8K8AdddB5dfDrVqpTqy9FbUEBMadE5EqoyPPgqNwW+8AcccE2YLqy7TRaaSOlSJSNrbvBlGjw5DRS9YAPfeC1OnKglUFJUIRCStPf98aAT+9FM4/fQwQNzee6c6qupFJQIRSUuffx7GBurVC2rWhP/+F3JzlQTioEQgImnlhx/gL3+BQw4JcwbceCPMnQvHHZfqyKovVQ2JSNp4+WW44AJYtAhOPDHMQZ5kVBmpYCoRiEjKffkl9O8fvvXn5cGUKeG+ACWByqFEICIps3Ur3HLLj8NEjx4dxgnq2TPVkWUWVQ2JSEq8/jqcf3648PfqBePGwS9+keqoMpNKBCJSqVasgCFDoFu3MFjck0/Cs88qCaSSEoGIVIpt2+Cf/4SDD4ZHHoGrrgo3h51wAliyQeul0qhqSERiN20aXHppmCz+2GNDQtBsYelDJQIRic3ixeGmsO7dYeVKmDQp3BimJJBelAhEpMKtWQN//GOYJ+Cll8K0kYsWwWmnqRooHalqSEQqzNatcOedoRvot9+GuYLHjIF99011ZFIUlQhEpNzc4bnnoG3bMEBc27YwezbcfbeSQFWgRCAi5fLhh/Db30Lv3qFn0FNPhaEisrNTHZmUlBKBiJTJihVw7rnhgj9zZhgXaN486NNH7QBVjdoIRKRUNm8OF/0bboBNm8K8wddcA/XrpzoyKatYSwRm1sPMFpnZYjMbmWR7dzNba2Zzose1ccYjImXnHrp/tmoVbgY7+miYPz9MFKMkULXFlgjMLAsYD/QEWgMDzKx1kl1fd/fs6HF9HLHk5kKzZlCjRviZmxvHu4hUT+5hNNAuXWDAANhrr9AG8NRTcNBBqY5OKkKcJYKOwGJ3/8zdfwAmAX1jfL+kcnPDZNdLl4YP9NKlYVnJQKRoeXmhBNC+PRx/fBgq+u67YdasMHG8VB9xJoL9gS8SlpdF6wo6wsw+MLPnzezQZAcys2FmNtPMZq5cubJUQYwaBRs37rxu48awXkR+assWmDAhVAENGBCW77svzBl81lmQlZXqCKWixZkIkvUb8ALLs4Gm7t4OuA2YnOxA7j7B3XPcPadRo0alCuLzz0u3XiRTrV8PN98MzZuH3kD164fJYebPD6OF1qqV6gglLnEmgmXAAQnLjYGvEndw93XuviF6PgWoZWYNKzKIwmY40sxHIsGqVXDttdC0KVxxRRgW4r//hXffDdNF1lAn82ovzj/xDKClmTU3s12A/sDTiTuY2T5mocexmXWM4lldkUGMHQt16uy8rk6dsF4kk33xBfzhD+FL0ZgxYWC4d9+FqVPDlJG6FyBzxHYfgbvnmdmFwItAFjDR3eeb2fBo+x1AP+A8M8sDNgH93b1g9VG5DBwYfo4aFaqDmjQJSSB/vUimWbgQ/vpXeOihsDxwIIwYAYccktq4JHWsgq+7scvJyfGZM2emOgyRKmfWrHAT2JNPQu3acPbZYYTQpk1THZlUBjOb5e45ybbpzmKRau6bb+DKK+H++2HPPeHqq0OVUCn7XUg1pkQgUk1t2xaGhB41Cr7/HkaODHcE77FHqiOTdKNEIFINvfcenH/+jzd/jR+vWcGkcOoYJlKNrF4d7gHo3BmWLw93Bk+dqiQgRVMiEKkGtm+He+6Bgw8OPy+9NPQO0tSQUhKqGhKp4ubMCdVAb78NRx0VqoHatEl1VFKVqEQgUkWtXRvmAjjssDAO0P33w7RpSgJSeioRiFQx7mH03Msvh5Ur4bzz4M9/DsNDi5SFEoFIFTJ/PlxwQfjm36lTmCegQ4dURyVVnaqGRKqANWtCCSA7O0wWf9dd8NZbSgJSMVQiEEljmzeHxt+xY0MyOPtsuPFGaNAg1ZFJdaISgUga2rYNHnggdAe9/PJQDfT++2HCGCUBqWhKBCJpxB2efz5U+QwZEsYDevnlsK5du1RHJ9WVEoFImpgxIwwH0atXGBto0qQwVITmB5a4KRGIpNgnn8Cpp0LHjqFX0D//CQsWhLuCNTuYVAY1FoukyIoVcP31od5/113DdJGXXw716qU6Msk0SgQilWz9evjb38JE8Vu2wLBhcM01sM8+qY5MMpUSgUgl+eGH0P//+uvDZDGnnBK6hbZsmerIJNMpEYjEbNOm0BX0ppvCmEC/+hU880xoExBJB7E2RZlZDzNbZGaLzWxkEfsdbmbbzKxfnPGIVKZVq8K3/6ZNYfhw+NnP4Lnn4NVXlQQkvcRWIjCzLGA88GtgGTDDzJ529wVJ9vsr8GJcsYhUpsWL4e9/h3vvDaWB44+HK66Abt00N4CkpzirhjoCi939MwAzmwT0BRYU2O8i4HHg8BhjEYndu++G6p8nnoBateD3v4c//hFat051ZCJFizMR7A98kbC8DOiUuIOZ7Q+cCByDEoFUQdu3w7PPhgTwxhthKOiRI+Gii2DffVMdnUjJxJkIkhWCvcDyrcAId99mRZSZzWwYMAygSZMmFRagSFlt3gwPPhi6gS5aFNoBbr0VzjoL6tZNdXQipRNnIlgGHJCw3Bj4qsA+OcCkKAk0BHqZWZ67T07cyd0nABMAcnJyCiYTkUqzejX8619w222hC2iHDvDII9CvH9RUHzypouL86M4AWppZc+BLoD9weuIO7t48/7mZ3Qc8WzAJiKTapk2h2ufJJ8N0kBs3Qs+eoQG4e3c1AEvVF1sicPc8M7uQ0BsoC5jo7vPNbHi0/Y643lukPNxh3jx46aXwmD49VAXtsgucfnpoAP7lL1MdpUjFibUw6+5TgCkF1iVNAO5+RpyxiBTlm29g6tQfL/7Ll4f1hxwC554Lv/lNuBFs991TG6dIHEqUCMxsd2CTu283s4OAVsDz7r411uhEYrJlC7z55o8X/vffD+vr14df/zpc+H/9azjggKKPI1IdlLREMB04ysx+BrwMzAROAwbGFZhIRVu0KEzw8tJLYfL3jRtDA2+XLmHMn9/8Btq3h6ysVEcqUrlKmgjM3Tea2VnAbe7+f2b2fpyBiVSUt96CMWPghRfC8kEHwdCh4cLfvbuGfRYpcSIwsyMIJYCzSvlakUrnDq+9FhLAq69Cw4Zwww0wYAA0a5bq6ETSS0kv5pcAVwFPRj1/WgCvxheWSNm4h6qfMWNCG8C++8Itt4Qx/9XQK5JciRKBu08DpgGYWQ1glbtfHGdgcZg3T93+qiv3MNTDmDFh7t8DDghTPp51FtSuneroRNJbiYahNrOHzWyPqPfQAmCRmV0Rb2gV6/77oV27MC68VB/bt8N//hMaefv0CXf+3nVXGAH0gguUBERKoqTzEbR293XACYT7ApoAg2KLKgYnnwxHHw1DhsDtt6c6Gimvbdvg4YehTZsw09emTSHZL1oEZ58dbv4SkZIpaSKoZWa1CIngqej+gSo15k/duqHqoE+f8E3xL39JdURSFlu3wn33hRu9Bg4Mwzs88ggsWACDB2u8H5GyKGkiuBNYAuwOTDezpsC6uIKKS+3aoRrh9NPhqqvg6qtD3bKkvw0b4I47QtfPM88Mif2JJ2DuXOjfX33/RcqjpI3F44BxCauWmtnR8YQUr1q1QjtB3bpw442wfj384x9QI9ZJO6Us3OGdd+Cee+DRR0My6NQpNAL36qXB3kQqSkmHmNgTuA7oFq2aBlwPrI0prlhlZYVvl/XqhfHkN2wIDYyqVkgPX38dxvqfOBEWLgzdPk87LdwE1qWLEoBIRSvppW8iMA84NVoeBNwLnBRHUJXBLMwqtccecN11IRnk5qqRMVW2bg3DP0ycGNpytm2Drl1DaeCUU3T3r0icSpoIfuHuJycs/z8zmxNHQJXJDK69NlxkLrssJIPHH4c6dVIdWeZYuDBc/B94AFasgJ//PAzzPHQoHHxwqqMTyQwlTQSbzOxId38DwMy6ApviC6tyXXppSAbDhoUJR555JpQUJB7r18Njj4UE8NZboUqud+9w8e/RI7TjiEjlKWkiGA48ELUVAHwHDIknpNQ4++zQgDxoEBx7bBigrMK33EgAABHvSURBVEGDVEdVfbiHWb4mTgxJYONGaNUqVM8NGhRKAiKSGiXtNfQB0M7M9oiW15nZJcDcOIOrbP37h4bJU04Jk5D8979hrBopu0WLQtvLQw/B//4Xku3pp4ehHzp1UsOvSDooVadJd18X3WEMcFkM8aTc734Hzz0HS5ZAt26wdGmqI6p6vvkGxo2Djh3Dt/6xY+HAA8Odv19/HXpode6sJCCSLsrTe77a/hsfe2woDaxaBUceCR9/nOqI0t/GjeEO3+OPh/32gz/8IfQE+tvf4IsvwoiggwdrBFCRdFSenvPV+p7cI44I49j/5jdw1FHhQtauXaqjSi/btoVz9NBDobfVhg1h1M8rrgjDP2ikV5GqochEYGbrSX7BN2C34g5uZj2AfwBZwN3u/pcC2/sCY4DtQB5wSX7PpHSQnQ3Tp4e5a7t3D/3cO3dOdVSl5x5KOA89FIbZaNQoTNSS7FG3bvFVNh98EI718MPw1Vehh9Vpp8Hvfx+q03SXtkjVUmQicPcy38ZjZlnAeODXwDJghpk97e4LEnZ7GXja3d3M2gKPAa3K+p5xaNUKXn89VBcde2zo4njuuVXj2657SF7XXw/vvht6QdWqFaq88vKSv2bXXQtPEhDG9/nww9Dls1evcPHv3Rt2K/ZrgYikqzgHVegILHb3zwDMbBLQlzCfAQDuviFh/91J0+qmZs1C18fLL4cJE8JYN127wvDh0K9f+o157x7uhbj+epg1K8Q/YUIYgnuXXcL2detg5cqQFAo+Ete//374+e234dhHHAHjx8Opp/6YHESkajOPafhNM+sH9HD3s6PlQUAnd7+wwH4nAjcCewPHu/vbSY41DBgG0KRJk8OWprArz6pVYRjkO+8Mk5/Urx8usOeem/o7Ybdvh8mTwyxdc+ZAixYwalTop1/em7Ty8sKY/xrqQaRqMrNZ7p6TbFuctbnJapp/knXc/Ul3b0WY62BMsgO5+wR3z3H3nEaNGlVwmKXTsGEoGSxaBFOnhuqi224LVUhHHw2TJsGWLZUb0/bt8O9/hzaNk0+G77//cZKWoUMr5k7dmjWVBESqqzgTwTLggITlxsBXhe3s7tOBX5hZ2lU45OaG6pUaNcLP3Nzw/Nhjw12yX3wBN9wQ7j0YMCD0nBkxAj79NN64tm0LXTbbtAlVNVu3htg++kiTtIhIycWZCGYALc2suZntAvQHnk7cwcwONAt9VMysA7ALsDrGmEotNzeMQbR0aahbX7o0LOfm/rjPPvuEiW4+/TQMTdG1a+g/f+CBofvpE0+Ei3RFycsLwzS3bh3u0jULJZF588KyJmkRkdKIrY0AwMx6AbcSuo9OdPexZjYcwN3vMLMRwGBgK2EQuyuK6z6ak5PjM2fOjC3mgpo1S353cdOmoQRQmC+/DOPq3HVXKDHss0+oq2/cONxUlfioUyf5uoLdMLduDd02x44NSadt2zB66oknqsumiBStqDaCWBNBHCo7EdSokXw6S7NQN1+cbdtCF84774QpU0r2mny77bZzclizJvTbb98+JIA+fZQARKRkikoEqkUuRpMmyUsETZqU7PVZWaGffe/eoRF5w4bQmFuWB4T5eo8/XuP0iEjFUSIoxtixoU1g48Yf19WpE9aX1q67hoeGtxaRdKKKhWIMHBhuxmraNHwLb9o0LA8cmOrIREQqhkoEJTBwoC78IlJ9qUQgIpLhlAhERDKcEoGISIZTIhARyXBKBCIiGU6JQEQkwykRiIhkOCUCEZEMp0QgIpLhlAhERDKcEoGISIZTIhARyXBKBCIiGU6JQEQkwykRVILc3DD3cY0a4WfixPciIqkWayIwsx5mtsjMFpvZyCTbB5rZ3Ojxlpm1izOeVMjNDTOcLV0a5j5eujQsKxmISLqILRGYWRYwHugJtAYGmFnrArv9D/iVu7cFxgAT4oonVUaN2nmaSwjLo0alJh4RkYLiLBF0BBa7+2fu/gMwCeibuIO7v+Xu30WL7wCNY4wnJT7/vHTrRUQqW5yJYH/gi4TlZdG6wpwFPJ9sg5kNM7OZZjZz5cqVFRhi/Jo0Kd16EZHKFmcisCTrPOmOZkcTEsGIZNvdfYK757h7TqNGjSowxPiNHQt16uy8rk6dsF5EJB3EmQiWAQckLDcGviq4k5m1Be4G+rr76hjjSYmBA2HCBGjaFMzCzwkTwnoRkXRQM8ZjzwBamllz4EugP3B64g5m1gR4Ahjk7h/HGEtKDRyoC7+IpK/YEoG755nZhcCLQBYw0d3nm9nwaPsdwLVAA+B2MwPIc/ecuGISEZGfMvek1fZpKycnx2fOnJnqMEREqhQzm1XYF23dWSwikuGUCEREMpwSQRWgsYpEJE5x9hqSCpA/VlH+MBX5YxWBeiKJSMVQiSDNaawiEYmbEkGa01hFIhI3JYI0p7GKRCRuSgRpTmMViUjclAjSnMYqEpG4qddQFaCxikQkTioRiIhkOCWCDKAb0kSkKKoaquZ0Q5qIFEclgmpON6SJSHGUCKo53ZAmIsVRIqjmdEOaiBRHiaCa0w1pIlIcJYJqrqJuSFPPI5HqS72GMkB5b0hTzyOR6i3WEoGZ9TCzRWa22MxGJtneyszeNrMtZnZ5nLFI2annkUj1FluJwMyygPHAr4FlwAwze9rdFyTs9i1wMXBCXHFI+annkUj1FmeJoCOw2N0/c/cfgElA38Qd3P0bd58BbI0xDikn9TwSqd7iTAT7A18kLC+L1kkVUxE9j9TYLJK+4kwElmSdl+lAZsPMbKaZzVy5cmU5w5LSKm/Po/zG5qVLwf3HxmYlA5H0EGciWAYckLDcGPiqLAdy9wnunuPuOY0aNaqQ4KR0Bg6EJUtg+/bwszS9hdTYLJLe4kwEM4CWZtbczHYB+gNPx/h+kqbU2CyS3mJLBO6eB1wIvAh8BDzm7vPNbLiZDQcws33MbBlwGfAnM1tmZnvEFZOkRkU0NquNQSQ+sd5Q5u5TgCkF1t2R8PxrQpWRVGNjx+58QxqUrrFZN7SJxEtDTEjsytvYrDYGkXiZe5k68qRMTk6Oz5w5M9VhSCWqUSP0NirILDRei0jxzGyWu+ck26YSgaQ93dAmEi8lAkl7uqFNJF5KBJL2dEObSLyUCKRKSPUNbSpRSHWm+Qik2ivvDW3qvirVnUoEUu2Vt7FZ3VelulMikGqvvI3NFTFEhqqWJJ0pEUi1V97G5vKWKNRYLelON5SJFKNgGwGEEkVJk0mzZuHiX1DTpqHhW6Qy6IYykXIob4lCVUuS7pQIREqgPN1X06FqSYlEiqJEIBKz8jZWl7fXkhKJFEeJQCRmqa5aSodEIulNiUCkEqSyainViQRUokh3SgQiaa68VUupTiTpUKJQIiqaEoFImitv1VKqE0mqSxRKRCXg7lXqcdhhh7mIlM5DD7k3bepuFn4+9FDpXlunjnu4jIZHnTolP4bZzq/Nf5hVzvs3bZr8/Zs2Ldnr82NI1fmrKMBML+S6mvILe2kfSgQila88F8LyXojL+/pMT0T5UpYIgB7AImAxMDLJdgPGRdvnAh2KO6YSgUjVkuoSRaYnonxFJYLY2gjMLAsYD/QEWgMDzKx1gd16Ai2jxzDgX3HFIyKpkeqxnlI96GA6tLEUJ87G4o7AYnf/zN1/ACYBfQvs0xd4IEpY7wB7mdm+McYkIilQnu6z5b2QZ3oiKok4E8H+wBcJy8uidaXdBzMbZmYzzWzmypUrKzxQEUlf5b2Q5x8jUxNRScSZCCzJuoJDnZZkH9x9grvnuHtOo0aNKiQ4Eak6ynMhr4j3rsqJqCTinKpyGXBAwnJj4Ksy7CMiklIDB6ZuWtL89x01KlQHNWkSkkBFxhNnIpgBtDSz5sCXQH/g9AL7PA1caGaTgE7AWndfHmNMIiJVTtyJKLZE4O55ZnYh8CKQBUx09/lmNjzafgcwBehF6D66ETgzrnhERCS5OEsEuPsUwsU+cd0dCc8duCDOGEREpGgaa0hEJMMpEYiIZDglAhGRDGehmr7qMLOVwNJUx1GIhsCqVAdRhHSPD9I/RsVXPoqvfMoTX1N3T3ojVpVLBOnMzGa6e06q4yhMuscH6R+j4isfxVc+ccWnqiERkQynRCAikuGUCCrWhFQHUIx0jw/SP0bFVz6Kr3xiiU9tBCIiGU4lAhGRDKdEICKS4ZQISsnMDjCzV83sIzObb2Z/SLJPdzNba2Zzose1lRzjEjP7MHrvmUm2m5mNM7PFZjbXzDpUYmwHJ5yXOWa2zswuKbBPpZ8/M5toZt+Y2byEdfXN7L9m9kn082eFvLaHmS2KzufISozvJjNbGP0NnzSzvQp5bZGfhxjjG21mXyb8HXsV8tpUnb9HE2JbYmZzCnltrOevsGtKpX7+CpvMWI/kD2BfoEP0vB7wMdC6wD7dgWdTGOMSoGER23sBzxMmBuoMvJuiOLOArwk3uqT0/AHdgA7AvIR1/weMjJ6PBP5ayO/wKdAC2AX4oODnIcb4fgPUjJ7/NVl8Jfk8xBjfaODyEnwGUnL+Cmz/G3BtKs5fYdeUyvz8qURQSu6+3N1nR8/XAx+RZHrNNJcuc0UfC3zq7im/U9zdpwPfFljdF7g/en4/cEKSl5Zkbu5Y4nP3l9w9L1p8hzCxU0oUcv5KImXnL5+ZGXAq8EhFv29JFHFNqbTPnxJBOZhZM6A98G6SzUeY2Qdm9ryZHVqpgYXpPl8ys1lmNizJ9hLNFV0J+lP4P18qz1++n3s0UVL0c+8k+6TLuRxKKOUlU9znIU4XRlVXEwup2kiH83cUsMLdPylke6WdvwLXlEr7/CkRlJGZ1QUeBy5x93UFNs8mVHe0A24DJldyeF3dvQPQE7jAzLoV2F6iuaLjZGa7AH2AfyfZnOrzVxrpcC5HAXlAbiG7FPd5iMu/gF8A2cByQvVLQSk/f8AAii4NVMr5K+aaUujLkqwr9flTIigDM6tF+IPluvsTBbe7+zp33xA9nwLUMrOGlRWfu38V/fwGeJJQfEyUDnNF9wRmu/uKghtSff4SrMivMot+fpNkn5SeSzMbAvQGBnpUaVxQCT4PsXD3Fe6+zd23A3cV8r6pPn81gZOARwvbpzLOXyHXlEr7/CkRlFJUn3gP8JG731LIPvtE+2FmHQnneXUlxbe7mdXLf05oUJxXYLengcFR76HOpGau6EK/haXy/BXwNDAkej4EeCrJPjvm5o5KOf2j18XOzHoAI4A+7r6xkH1K8nmIK77EdqcTC3nflJ2/yHHAQndflmxjZZy/Iq4plff5i6slvLo+gCMJRa+5wJzo0QsYDgyP9rkQmE9owX8H6FKJ8bWI3veDKIZR0frE+AwYT+ht8CGQU8nnsA7hwr5nwrqUnj9CUloObCV8yzoLaAC8DHwS/awf7bsfMCXhtb0IPT0+zT/flRTfYkL9cP7n8I6C8RX2eaik+B6MPl9zCRenfdPp/EXr78v/3CXsW6nnr4hrSqV9/jTEhIhIhlPVkIhIhlMiEBHJcEoEIiIZTolARCTDKRGIiGQ4JQKRiJlts51HRq2wkTDNrFniyJci6aRmqgMQSSOb3D071UGIVDaVCESKEY1H/1czey96HBitb2pmL0eDqr1sZk2i9T+3MD/AB9GjS3SoLDO7Kxpz/iUz2y3a/2IzWxAdZ1KKfk3JYEoEIj/arUDV0GkJ29a5e0fgn8Ct0bp/EobzbksY8G1ctH4cMM3DoHkdCHekArQExrv7ocAa4ORo/UigfXSc4XH9ciKF0Z3FIhEz2+DudZOsXwIc4+6fRYODfe3uDcxsFWHYhK3R+uXu3tDMVgKN3X1LwjGaAf9195bR8giglrv/2cxeADYQRlmd7NGAeyKVRSUCkZLxQp4Xtk8yWxKeb+PHNrrjCWM/HQbMikbEFKk0SgQiJXNaws+3o+dvEUZ7BBgIvBE9fxk4D8DMssxsj8IOamY1gAPc/VXgSmAv4CelEpE46ZuHyI92s50nMH/B3fO7kO5qZu8SvjwNiNZdDEw0syuAlcCZ0fo/ABPM7CzCN//zCCNfJpMFPGRmexJGhf27u6+psN9IpATURiBSjKiNIMfdV6U6FpE4qGpIRCTDqUQgIpLhVCIQEclwSgQiIhlOiUBEJMMpEYiIZDglAhGRDPf/AcWE4yOUNJQ5AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画图 Plotting the training and validation loss\n",
    "import matplotlib.pyplot as plt\n",
    "history_dict = history.history\n",
    "loss_values = history_dict[\"loss\"] # 提取在取出10,000个样品外的train集上的loss values\n",
    "val_loss_values = history_dict[\"val_loss\"] # 提取在10,000个样品的train集上的loss values\n",
    "epochs = range(1, len(loss_values) + 1) # x axis\n",
    "plt.plot(epochs, loss_values, \"bo\", label=\"Training loss\")\n",
    "plt.plot(epochs, val_loss_values, \"b\", label=\"Validation loss\")\n",
    "plt.title(\"Training and validation loss\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Loss\")\n",
    "plt.legend()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwU1bn/8c/DgMAAgiwi+6CioFEQJqi4YaIGl2jcrhDc9Ye4RnONEk2MUfldo97oNW4Xb4hRMaA/xauJW0SjiTHKgICCoqOOOoLIIrssA8/vj1MDPU31TM9ML7N8369Xvbr2frqmp54+51SdMndHREQkWYt8ByAiIg2TEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUISZuZPW9m52R63XwyszIzOyoL+3Uz2zMaf8DMfpnOunV4n7Fm9lJd4xSpjuk+iKbNzNYmTBYCG4Et0fRF7j4l91E1HGZWBlzo7i9neL8ODHD30kyta2ZFwKdAK3evyEScItVpme8AJLvcvX3leHUnQzNrqZOONBT6PjYMqmJqpsxspJmVm9m1ZvYV8Acz28XM/mxmS83sm2i8d8I2fzOzC6Pxc83sH2Z2R7Tup2Z2bB3X7W9mr5vZGjN72czuNbNHU8SdTow3m9kb0f5eMrOuCcvPMrPPzGy5mV1fzfE5yMy+MrOChHknm9m8aHy4mb1pZivNbLGZ3WNmO6XY10NmdkvC9M+ibRaZ2flJ6x5vZu+Y2Woz+8LMbkxY/Hr0utLM1prZwZXHNmH7EWY208xWRa8j0j02tTzOnc3sD9Fn+MbMnk5YdpKZzYk+w8dmNiqaX6U6z8xurPw7m1lRVNV2gZl9DrwSzX8i+jusir4j+yZs39bM/jP6e66KvmNtzewvZnZ50ueZZ2Y/ivuskpoSRPO2G9AZ6AeMI3wf/hBN9wW+Be6pZvsDgYVAV+A24PdmZnVY9zHgbaALcCNwVjXvmU6MPwbOA3YFdgKuBjCzfYD7o/33jN6vNzHc/V/AOuB7Sft9LBrfAlwVfZ6Dge8Dl1QTN1EMo6J4jgYGAMntH+uAs4FOwPHAxQkntsOj107u3t7d30zad2fgL8Dd0Wf7LfAXM+uS9Bl2ODYxajrOjxCqLPeN9nVnFMNw4GHgZ9FnOBwoS3U8YhwBDAJ+EE0/TzhOuwKzgcQq0TuAYcAIwvf4GmAr8EfgzMqVzGww0At4rhZxCIC7a2gmA+Ef9ahofCSwCWhTzfpDgG8Spv9GqKICOBcoTVhWCDiwW23WJZx8KoDChOWPAo+m+ZniYvxFwvQlwAvR+A3A1IRl7aJjcFSKfd8CTI7GOxBO3v1SrHslMD1h2oE9o/GHgFui8cnArQnr7ZW4bsx+7wLujMaLonVbJiw/F/hHNH4W8HbS9m8C59Z0bGpznIEehBPxLjHr/XdlvNV9/6LpGyv/zgmfbfdqYugUrdORkMC+BQbHrNcaWEFo14GQSO7L9f9bUxhUgmjelrr7hsoJMys0s/+OiuyrCVUanRKrWZJ8VTni7uuj0fa1XLcnsCJhHsAXqQJOM8avEsbXJ8TUM3Hf7r4OWJ7qvQilhVPMrDVwCjDb3T+L4tgrqnb5Korj/xJKEzWpEgPwWdLnO9DMXo2qdlYB49Pcb+W+P0ua9xnh13OlVMemihqOcx/C3+ybmE37AB+nGW+cbcfGzArM7Naommo120siXaOhTdx7uftG4HHgTDNrAYwhlHiklpQgmrfkS9j+HdgbONDdd2Z7lUaqaqNMWAx0NrPChHl9qlm/PjEuTtx39J5dUq3s7gsIJ9hjqVq9BKGq6gPCr9SdgevqEgOhBJXoMeAZoI+7dwQeSNhvTZccLiJUCSXqC3yZRlzJqjvOXxD+Zp1itvsC2CPFPtcRSo+VdotZJ/Ez/hg4iVAN15FQyqiMYRmwoZr3+iMwllD1t96TquMkPUoQkqgDodi+MqrP/lW23zD6RV4C3GhmO5nZwcAPsxTj/wNOMLNDowblm6j5f+Ax4ArCCfKJpDhWA2vNbCBwcZoxPA6ca2b7RAkqOf4OhF/nG6L6/B8nLFtKqNrZPcW+nwP2MrMfm1lLMzsD2Af4c5qxJccRe5zdfTGhbeC+qDG7lZlVJpDfA+eZ2ffNrIWZ9YqOD8AcYHS0fjFwWhoxbCSU8goJpbTKGLYSqut+a2Y9o9LGwVFpjyghbAX+E5Ue6kwJQhLdBbQl/Dr7F/BCjt53LKGhdzmh3n8a4cQQp84xuvt84FLCSX8x8A1QXsNmfyK017zi7ssS5l9NOHmvAR6MYk4nhuejz/AKUBq9JroEuMnM1hDaTB5P2HY9MBF4w8LVUwcl7Xs5cALh1/9yQqPtCUlxp6um43wWsJlQivqa0AaDu79NaAS/E1gFvMb2Us0vCb/4vwF+TdUSWZyHCSW4L4EFURyJrgbeBWYS2hx+Q9Vz2sPAfoQ2LakD3SgnDY6ZTQM+cPesl2Ck6TKzs4Fx7n5ovmNprFSCkLwzs++a2R5RlcQoQr3z0zVtJ5JKVH13CTAp37E0ZkoQ0hDsRrgEcy3hGv6L3f2dvEYkjZaZ/YDQXrOEmquxpBqqYhIRkVgqQYiISKwm1Vlf165dvaioKN9hiIg0GrNmzVrm7t3iljWpBFFUVERJSUm+wxARaTTMLPnu+21UxSQiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISK2sJwswmm9nXZvZeiuVmZnebWWn0OMChCctGmdnCaNmEbMUoItKYTZkCRUXQokV4nTKlpi1qJ5sliIeAUdUsP5bwKMEBhMdd3g/hISHAvdHyfYAx0aMiRUSqyPYJsiG//5QpMG4cfPYZuIfXceMyG0PWEoS7v07ogjeVk4CHPfgX4WlVPYDhhMdTfuLum4Cp0boiIttk4gRZnxN8vt//+uth/fqq89avD/MzJZ9tEL2o+ujF8mheqvmxzGycmZWYWcnSpUuzEqiIZEc+T5D1PcHn+/0//7x28+sinwki7vGMXs38WO4+yd2L3b24W7fYu8VFJEvy+Qu8vifI+p7g8/3+fZMfVlvD/LrIZ4Iop+qzeXsTnqmbar6INCD5/gVe3xNkfU/w+X7/iROhsLDqvMLCMD9T8pkgngHOjq5mOghYFT3rdiYwwMz6R88NHh2tKyIZls8qnnyfIOt7gs/3+48dC5MmQb9+YBZeJ00K8zPG3bMyEJ7lu5jw3Npy4AJgPDA+Wm6Eq5U+JjxXtjhh2+OAD6Nl16f7nsOGDXOR5uTRR9379XM3C6+PPlq7bQsL3cPv/zAUFqa/D7Oq21YOZult369f/Pb9+tXuM+Tr8zeE988EoMRTncdTLWiMgxKENCf1PcHU9wRd3+0bwgmyPif4pvD+7koQIg1WfU4Q9T1B17cEkO9f4JIZ1SWIJvXI0eLiYtfzIKSxqGzkTazHLyxMvx65RYtwWk5mBlu31rx9UVFoWE7Wrx+UldW8PYTPcP31od2gb99Q/57ROnDJOjOb5e7FccvUF5NInuT7Kp5MXAUzdmxIJlu3hlclh6ZFCUKkHupzFVC+r+LJyVUw0qg1qUeOiuRSchVR5X0AkN5Jtm/f+Cqe2lzmCPWr4hk7VglBUlMbhEgd1bcOv75tECKZoDYIkRTyWUWkKh5p6FTFJM1WvquIKt9HCUEaKpUgpNmq71VEuegLRySflCCk2VIVkUj1VMUkzZaqiESqpxKENGr1aWRWFZFI9ZQgpNGq7/MIVEUkUj3dByGNVib6EhJp7nQfhDRJuXgmr0hzpgQhjVYunskr0pwpQUijpUZmkexSgpC8qs9VSGpkFsku3QcheVPfri4q11NCEMkOlSAkb+rb1YWIZJcShOSNrkISadiUICRvdBWSSMOmBCH1oq4uRJouJQipM3V1IdK0qasNqTN1dSHS+KmrDckKNTKLNG1KEFJnamQWadqUIKTO1Mgs0rQpQTRz6upCRFJRVxvNmLq6EJHqqATRjKmrCxGpjhJEM6arkESkOkoQzZiuQhKR6ihBNGO6CklEqqME0YzpKiQRqY6uYmrmdBWSiKSiEoSIiMTKaoIws1FmttDMSs1sQszyXcxsupnNM7O3zew7CcvKzOxdM5tjZuqBT0Qkx7JWxWRmBcC9wNFAOTDTzJ5x9wUJq10HzHH3k81sYLT+9xOWH+nuy7IVo4iIpJbNEsRwoNTdP3H3TcBU4KSkdfYBZgC4+wdAkZl1z2JMIiKSpmwmiF7AFwnT5dG8RHOBUwDMbDjQD+gdLXPgJTObZWbjUr2JmY0zsxIzK1m6dGnGgm8s6tOXkohIdbKZICxmXvLTiW4FdjGzOcDlwDtARbTsEHcfChwLXGpmh8e9ibtPcvdidy/u1q1bhkJvHOr7RDcRkepkM0GUA30SpnsDixJXcPfV7n6euw8Bzga6AZ9GyxZFr18D0wlVVpJAfSmJSDZlM0HMBAaYWX8z2wkYDTyTuIKZdYqWAVwIvO7uq82snZl1iNZpBxwDvJfFWBsl9aUkItmUtauY3L3CzC4DXgQKgMnuPt/MxkfLHwAGAQ+b2RZgAXBBtHl3YLqZVcb4mLu/kK1YG6u+feOfCa2+lEQkE7J6J7W7Pwc8lzTvgYTxN4EBMdt9AgzOZmxNwcSJVZ/nAOpLSUQyR3dSN2LqS0lEskl9MTVy6ktJRLJFJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISSwkiz9Qbq4g0VLoPIo8qe2OtvBO6sjdW0L0NIpJ/KkHkkXpjFZGGTAkij9Qbq4g0ZEoQeZSq11X1xioiDYESRB5NnBh6X02k3lhFpKFQgsgj9cYqIg2ZrmLKM/XGKiINlUoQIiISSwlCRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISSwlCRERiKUGIiEisGhOEmZ1gZkokIiLNTDon/tHAR2Z2m5kNynZAIiLSMNSYINz9TOAA4GPgD2b2ppmNM7MOWY9ORETyJq2qI3dfDTwJTAV6ACcDs83s8izGJiIieZROG8QPzWw68ArQChju7scCg4GrsxyfiIjkSToPDDoduNPdX0+c6e7rzez87IQlIiL5lk4V06+AtysnzKytmRUBuPuM7ITVeEyZAkVF0KJFeJ0yJd8RiYhkRjoJ4glga8L0lmheszdlCowbB599Bu7hddw4JQkRaRrSSRAt3X1T5UQ0vlP2Qmo8rr8e1q+vOm/9+jBfRKSxSydBLDWzEysnzOwkYFn2Qmo8Pv+8dvNFRBqTdBLEeOA6M/vczL4ArgUuSmfnZjbKzBaaWamZTYhZvouZTTezeWb2tpl9J91tG4K+fWs3X0SkMUnnRrmP3f0gYB9gH3cf4e6lNW1nZgXAvcCx0bZjzGyfpNWuA+a4+/7A2cB/1WLbvJs4EQoLq84rLAzzRUQau3Quc8XMjgf2BdqYGQDuflMNmw0HSt39k2gfU4GTgAUJ6+wD/Ee0vw/MrMjMugO7p7Ft3o0dG16vvz5UK/XtG5JD5XwRkcasxgRhZg8AhcCRwP8Ap5Fw2Ws1egFfJEyXAwcmrTMXOAX4h5kNB/oBvdPctkEYO1YJQUSapnTaIEa4+9nAN+7+a+BgoE8a21nMPE+avhXYxczmAJcD7wAVaW4b3iT0C1ViZiVLly5NIywREUlHOlVMG6LX9WbWE1gO9E9ju3KqJpLewKLEFaI+ns4DsFB39Wk0FNa0bcI+JgGTAIqLi2OTiIiI1F46JYhnzawTcDswGygD/pTGdjOBAWbW38x2InQb/kziCmbWKVoGcCHwepQ0atxWRESyq9oSRPSgoBnuvhJ40sz+DLRx91U17djdK8zsMuBFoACY7O7zzWx8tPwBYBDwsJltITRAX1DdtnX+lCIiUmvmXn2tjJm96e4H5yieeikuLvaSkpJ8hyEi0miY2Sx3L45blk4V00tmdqpVXt8qIiLNQjqN1D8F2gEVZraBcIWRu/vOWY1MRETyqsYE4e56tGgD4Q6rV8OSJfDVV2FYsgS+/hqOPhoOPzzfEYpIU5LOjXKxp53kBwhJ3W3cCF9+uf2En3jyTx7fsCF+HxMnwtVXwy23wE7qa1dEMiCdKqafJYy3IXShMQv4XlYiaiYqKuCvfw3Pjpg+fcduw82ga1fYbbcwDBiwfbx796rjbduG5HD77TBjBjz2GOy9d34+l4g0HelUMf0wcdrM+gC3ZS2iJswdZs6ERx+FadNC1VCnTqGrjhEjqp70u3WDlmn1lBU88ACMGgUXXghDh8Jdd4XxbF9asHYt/M//QEEBnHoq9OyZ3fcTkdyp8TLXHTYIVzPNc/f9shNS3TXUy1xLS0NJYcoU+OgjaN0aTjgBzjwTjj02TGfKokVw9tmhJHHyyfDgg9ClS+b2X2nTJpg0CW6+OSQ6CMno0EPh3/4tJIsePTL/viKSWdVd5prOfRC/Y3s/SC2AIUCZu5+Z0SgzoCEliKVLQynh0UfhrbfCyXPkyFBaOPXUUHLIlq1b4be/heuug113hYcfhu9lqEJwyxb405/ghhvg00/hiCPg1luhY0d44gl4/HGYPz983sMO254sdtstM+8vIplV3wRxTsJkBSE5vJHB+DIm3wli3Tp45pmQFF58MZxM998/lBTGjIHevXMbz+zZ4X0/+gh+9rPwa7+uDdju8Je/hKTz7rtwwAHwH/8BxxyzYzXWggUhWTzxxPZkcfjhIVmccoqShUhDUl2CwN2rHQj3QBQkTBcAhTVtl49h2LBhng9/+5v7WWe5t2vnDu59+rhfe637vHl5CaeKtWvd/8//CXENG+a+cGHt9/H66+4jRoR97Lmn+9Sp7lu2pLft/PnuN97ovs8+YXsz95Ej3e+7z/2rr2ofS7KtW93XrUs/HhGpCijxFOfUdEoQ/wKOcve10XR74CV3H5GxFJYhuS5BfPIJ/PSn8L//G6pYTj89lBYOOwxapHOPeg5Nnx4arTdsgP/6L7jggpobsOfODSWG554Ljc+/+hWcdx60alW3GObP314N9f774RgdcUQ4bnvtFe7xWLMmDInjydPJy7Zuhf794fLLw+faWbdwiqStvlVMc9x9SE3zGoJcJYh160K9++23hyuNfvlL+MlPoE2brL91vXz5ZWjAfuWV0C4waRJ07rzjeh9/HNoYHnsstJX8/Odw2WU7Pl61rtyrJosPPohfr2VL6NAhDDvvvH08ebqwEJ5/Hv7xjzB9/vkhWeyxR2biFWnK6lvF9AYwNGF6GPBmTdvlY8h2FdPWre7TpoUqJHAfO9b9yy+z+pYZt2WL+29+496ypXuvXu6vvLJ92aJF7hdfHJa1bev+85+7r1iR3Xi2bg3VUK+95v7OO+6lpe5Llrh/+21YVhszZ7qfeWaI38z9pJPcX3219vsRaU6opoopnQTxXeBj4O/RUAoMq2m7fAzZTBDz5oW6c3AfMsT973/P2lvlREmJ+157hRPptdeGZNC2bTi5XnJJSBaN1Zdful9/vXuXLtv/Xg895L5hQ74jE2l46pUgwva0Ar4D7Ae0SmebfAzZSBArVrhffrl7QYF7587u99/vXlGR8bfJi7Vr3S+8MHwLwP3HPw6/4JuK9evdH3zQfd99w+fbddfQYJ6JxnGRpqK6BJFOG8SlwBQPDw3CzHYBxrj7ffWo9sqKTLZBbNkCkyeHRtoVK2D8eLjppuzcdJZvr7wSuvXYf/98R5Id7uHGwbvuCpfq7rRTuB/lJz+BwYPrtr/Vq2HZsnC/y9KlofG/RYvQ8J/ua+J4YWH4G3TpEtpR1Lm+5Eo2GqnfcfcDMhhjRmQqQbz5ZmjknDUrXJF0990wpME1yUtdLFwY/p4PPRT6vzrySLjySjjwwKon/OrGly0Ld5JnS6tW4eKBLl3CUJk4EofkeZ07h+5ORGqrvgliHjA4KopgZgWErjb2zXik9VTfBLF4MUyYEO487tUrXKU0erR+zTVF33wT+pD63e/giy9Sr9epUzgZd+sWhsTxxOm2bUPJYuvW2r9u3RqS1bJlsHx5/FC5bPPm+DgLCsKlyH36hBsy44YePWrXv5c0D/VNELcDRcADhC43xgOfu/vVGY6z3uqaIDZtCr8qb7opdL397/8eqpbat89CkNKgVFSEu98XLdrxpN+1a93v+cgG99A5YlzyWLIkJLry8u1Dcg/BLVqEu9jjkkf//jBoULifR5qX+iaIFsA44CjC0+TeAXq4+6WZDrS+6pIgvvkGDj44VD2ccALceSfsuWeWAhTJEXdYubJqwkgcvvgiDGvXVt2uZ8+QKAYNgn322f7arZtK0k1VdQkine6+t0Z3U+8OnAF0Bp7MbIj5s8suoZvs3/4Wjjsu39GIZIZZ+G7vsgvsV02/y6tXh0Tx8cfh7vYFC8LrQw9VTR6dO++YNAYNClVaShxNV8oShJntBYwGxgDLgWnA1e7eL3fh1U6+O+sTaSrcQ0nj/ferJo4FC0K1VqV27bYninbtwtVYhYVVx5On48Z33rlhVec1J3UtQXxAuDHuh+5eGu3oqizEJyINjFk46ffpE3rsTbR06Y5J46OPQpvH+vWhK5p160Lje23er2vX0JAeN/TsuX28oXdp05RUlyBOJZQgXjWzF4CphDYIEWnGKhvzD499Wn3gHq64WreuauKoHE+eXrEiXEW4aFF4fe+90PBeUbHjvjt12jGBdOsWLipp1y68Jg+V8wsLG15Hmg1ZygTh7tOB6WbWDvgRcBXQ3czuB6a7+0s5ilFEGhmzcEPiTjuFdpC62Lo1XKG1ePGOQ2UieeON8LpxY/r7rUwWicmkTZva3eQYd9NjfdpiWrUKia9Tp3C8KseT57Vvn9sEl04j9TpgCjDFzDoDpwMTACUIEcmaFi3CExF33bX6O97dt5dI1q7dPiRPVzd/w4bKDmfqdi9LDReD1mjjRli1Klw0UNMx6dhxx0TSs2e4pyfTanXbjLuvAP47GkRE8s4slAbatQvJpDHbsiUkipUrtw/ffFP99AcfwOefZyce3VcpItJAFBSES4rjntOSD2quERGRWEoQIiISSwlCRERiKUGIiEgsJQgREYmlBCEiIrGUIEREJJYShIiIxFKCEBGRWEoQIiISK6sJwsxGmdlCMys1swkxyzua2bNmNtfM5pvZeQnLyszsXTObY2Z6CpCISI5lrS8mMysA7gWOBsqBmWb2jLsvSFjtUmCBu//QzLoBC81sirtvipYf6e7LshWjiIikls0SxHCg1N0/iU74U4GTktZxoIOZGdAeWAHEPCJERERyLZsJohfwRcJ0eTQv0T3AIGAR8C7wE3evfFChAy+Z2SwzG5fqTcxsnJmVmFnJ0qVLMxe9iEgzl80EEfd8peTHavwAmAP0BIYA95jZztGyQ9x9KHAscKmZxT7g0N0nuXuxuxd369YtQ6GLiEg2E0Q50CdhujehpJDoPOApD0qBT4GBAO6+KHr9GphOqLISEZEcyWaCmAkMMLP+ZrYTMBp4Jmmdz4HvA5hZd2Bv4BMza2dmHaL57YBjgPeyGKuIiCTJ2lVM7l5hZpcBLwIFwGR3n29m46PlDwA3Aw+Z2buEKqlr3X2Zme0OTA9t17QEHnP3F7IVq4iI7Mi8vk/bbkCKi4u9pES3TIiIpMvMZrl7cdwy3UktIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxlCBERCSWEoSIiMRSghARkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhIiKxWuY7ABFpGjZv3kx5eTkbNmzIdygSo02bNvTu3ZtWrVqlvY0ShIhkRHl5OR06dKCoqAgzy3c4ksDdWb58OeXl5fTv3z/t7VTFJCIZsWHDBrp06aLk0ACZGV26dKl16U4JQkQyRsmh4arL30YJQkREYilBiEheTJkCRUXQokV4nTKl7vtavnw5Q4YMYciQIey222706tVr2/SmTZuq3bakpIQrrriixvcYMWJE3QNspNRILSI5N2UKjBsH69eH6c8+C9MAY8fWfn9dunRhzpw5ANx44420b9+eq6++etvyiooKWraMP90VFxdTXFxc43v885//rH1gjZxKECKSc9dfvz05VFq/PszPlHPPPZef/vSnHHnkkVx77bW8/fbbjBgxggMOOIARI0awcOFCAP72t79xwgknACG5nH/++YwcOZLdd9+du+++e9v+2rdvv239kSNHctpppzFw4EDGjh2LuwPw3HPPMXDgQA499FCuuOKKbftNVFZWxmGHHcbQoUMZOnRolcRz2223sd9++zF48GAmTJgAQGlpKUcddRSDBw9m6NChfPzxx5k7SDVQCUJEcu7zz2s3v64+/PBDXn75ZQoKCli9ejWvv/46LVu25OWXX+a6667jySef3GGbDz74gFdffZU1a9aw9957c/HFF+9w78A777zD/Pnz6dmzJ4cccghvvPEGxcXFXHTRRbz++uv079+fMWPGxMa066678te//pU2bdrw0UcfMWbMGEpKSnj++ed5+umneeuttygsLGTFihUAjB07lgkTJnDyySezYcMGtm7dmtmDVA0lCBHJub59Q7VS3PxMOv300ykoKABg1apVnHPOOXz00UeYGZs3b47d5vjjj6d169a0bt2aXXfdlSVLltC7d+8q6wwfPnzbvCFDhlBWVkb79u3Zfffdt91nMGbMGCZNmrTD/jdv3sxll13GnDlzKCgo4MMPPwTg5Zdf5rzzzqOwsBCAzp07s2bNGr788ktOPvlkINzslkuqYhKRnJs4EaLz4DaFhWF+JrVr127b+C9/+UuOPPJI3nvvPZ599tmU9wS0bt1623hBQQEVFRVprVNZzVSTO++8k+7duzN37lxKSkq2NaK7+w6Xoqa7z2xRghCRnBs7FiZNgn79wCy8TppUtwbqdK1atYpevXoB8NBDD2V8/wMHDuSTTz6hrKwMgGnTpqWMo0ePHrRo0YJHHnmELVu2AHDMMccwefJk1keNMytWrGDnnXemd+/ePP300wBs3Lhx2/JcyGqCMLNRZrbQzErNbELM8ijQXBIAAA0NSURBVI5m9qyZzTWz+WZ2XrrbikjjNnYslJXB1q3hNZvJAeCaa67h5z//OYcccsi2k3ImtW3blvvuu49Ro0Zx6KGH0r17dzp27LjDepdccgl//OMfOeigg/jwww+3lXJGjRrFiSeeSHFxMUOGDOGOO+4A4JFHHuHuu+9m//33Z8SIEXz11VcZjz0Vy1YRxswKgA+Bo4FyYCYwxt0XJKxzHdDR3a81s27AQmA3YEtN28YpLi72kpKSbHwcEanB+++/z6BBg/IdRl6tXbuW9u3b4+5ceumlDBgwgKuuuirfYW0T9zcys1nuHnudbzZLEMOBUnf/xN03AVOBk5LWcaCDhYq39sAKoCLNbUVEGpQHH3yQIUOGsO+++7Jq1SouuuiifIdUL9m8iqkX8EXCdDlwYNI69wDPAIuADsAZ7r7VzNLZFgAzGweMA+ib6UsgRERq4aqrrmpQJYb6ymYJIq5nqOT6rB8Ac4CewBDgHjPbOc1tw0z3Se5e7O7F3bp1q0+8IiKSIJsJohzokzDdm1BSSHQe8JQHpcCnwMA0txURkSzKZoKYCQwws/5mthMwmlCdlOhz4PsAZtYd2Bv4JM1tRUQki7LWBuHuFWZ2GfAiUABMdvf5ZjY+Wv4AcDPwkJm9S6hWutbdlwHEbZutWEVEZEdZvQ/C3Z9z973cfQ93nxjNeyBKDrj7Inc/xt33c/fvuPuj1W0rIhJn5MiRvPjii1Xm3XXXXVxyySXVblN5Wfxxxx3HypUrd1jnxhtv3HY/QipPP/00CxZsvwL/hhtu4OWXX65N+A2W7qQWkUZvzJgxTJ06tcq8qVOnpuwwL9lzzz1Hp06d6vTeyQnipptu4qijjqrTvhoaddYnIhl35ZUQPZ4hY4YMgbvuil922mmn8Ytf/IKNGzfSunVrysrKWLRoEYceeigXX3wxM2fO5Ntvv+W0007j17/+9Q7bFxUVUVJSQteuXZk4cSIPP/wwffr0oVu3bgwbNgwI9zhMmjSJTZs2seeee/LII48wZ84cnnnmGV577TVuueUWnnzySW6++WZOOOEETjvtNGbMmMHVV19NRUUF3/3ud7n//vtp3bo1RUVFnHPOOTz77LNs3ryZJ554goEDB1aJqaysjLPOOot169YBcM8992x7aNFtt93GI488QosWLTj22GO59dZbKS0tZfz48SxdupSCggKeeOIJ9thjj3odc5UgRKTR69KlC8OHD+eFF14AQunhjDPOwMyYOHEiJSUlzJs3j9dee4158+al3M+sWbOYOnUq77zzDk899RQzZ87ctuyUU05h5syZzJ07l0GDBvH73/+eESNGcOKJJ3L77bczZ86cKifkDRs2cO655zJt2jTeffddKioquP/++7ct79q1K7Nnz+biiy+Orcaq7BZ89uzZTJs2bdtT7xK7BZ87dy7XXHMNELoFv/TSS5k7dy7//Oc/6dGjR/0OKipBiEgWpPqln02V1UwnnXQSU6dOZfLkyQA8/vjjTJo0iYqKChYvXsyCBQvYf//9Y/fx97//nZNPPnlbl9snnnjitmXvvfcev/jFL1i5ciVr167lBz/4QbXxLFy4kP79+7PXXnsBcM4553Dvvfdy5ZVXAiHhAAwbNoynnnpqh+0bQrfgzb4Ekcnn4opI/vzoRz9ixowZzJ49m2+//ZahQ4fy6aefcscddzBjxgzmzZvH8ccfn7Kb70rJXW5XOvfcc7nnnnt49913+dWvflXjfmrq566yy/BUXYo3hG7Bm3WCqHwu7mefgfv25+IqSYg0Pu3bt2fkyJGcf/752xqnV69eTbt27ejYsSNLlizh+eefr3Yfhx9+ONOnT+fbb79lzZo1PPvss9uWrVmzhh49erB582amJJwkOnTowJo1a3bY18CBAykrK6O0tBQIvbIeccQRaX+ehtAteLNOELl4Lq6I5M6YMWOYO3cuo0ePBmDw4MEccMAB7Lvvvpx//vkccsgh1W4/dOhQzjjjDIYMGcKpp57KYYcdtm3ZzTffzIEHHsjRRx9dpUF59OjR3H777RxwwAFVnhfdpk0b/vCHP3D66aez33770aJFC8aPH5/2Z2kI3YJnrbvvfKhtd98tWoSSQzKz0Ee9iKRP3X03fA2pu+8GL1Xnr+oUVkSkmSeIXD0XV0SkMWrWCSIfz8UVacqaUpV1U1OXv02zvw9i7FglBJFMaNOmDcuXL6dLly4pLxWV/HB3li9fXuv7I5p9ghCRzOjduzfl5eUsXbo036FIjDZt2tC7d+9abaMEISIZ0apVK/r375/vMCSDmnUbhIiIpKYEISIisZQgREQkVpO6k9rMlgKf5TuOFLoCy/IdRDUUX/0ovvpRfPVTn/j6uXu3uAVNKkE0ZGZWkup29oZA8dWP4qsfxVc/2YpPVUwiIhJLCUJERGIpQeTOpHwHUAPFVz+Kr34UX/1kJT61QYiISCyVIEREJJYShIiIxFKCyCAz62Nmr5rZ+2Y238x+ErPOSDNbZWZzouGGHMdYZmbvRu+9w+P3LLjbzErNbJ6ZDc1hbHsnHJc5ZrbazK5MWienx8/MJpvZ12b2XsK8zmb2VzP7KHrdJcW2o8xsYXQsJ+QwvtvN7IPo7zfdzDql2Lba70IW47vRzL5M+Bsel2LbfB2/aQmxlZnZnBTb5uL4xZ5TcvYddHcNGRqAHsDQaLwD8CGwT9I6I4E/5zHGMqBrNcuPA54HDDgIeCtPcRYAXxFu4snb8QMOB4YC7yXMuw2YEI1PAH6TIv6Pgd2BnYC5yd+FLMZ3DNAyGv9NXHzpfBeyGN+NwNVp/P3zcvySlv8ncEMej1/sOSVX30GVIDLI3Re7++xofA3wPtArv1HV2knAwx78C+hkZj3yEMf3gY/dPa93xrv768CKpNknAX+Mxv8I/Chm0+FAqbt/4u6bgKnRdlmPz91fcveKaPJfQO36eM6gFMcvHXk7fpUsPNTi34A/Zfp901XNOSUn30EliCwxsyLgAOCtmMUHm9lcM3vezPbNaWDgwEtmNsvMxsUs7wV8kTBdTn6S3GhS/2Pm8/gBdHf3xRD+gYFdY9ZpKMfxfEKJME5N34VsuiyqApuconqkIRy/w4Al7v5RiuU5PX5J55ScfAeVILLAzNoDTwJXuvvqpMWzCdUmg4HfAU/nOLxD3H0ocCxwqZkdnrQ87lFgOb0W2sx2Ak4EnohZnO/jl66GcByvByqAKSlWqem7kC33A3sAQ4DFhGqcZHk/fsAYqi895Oz41XBOSblZzLxaHUMliAwzs1aEP+QUd38qebm7r3b3tdH4c0ArM+uaq/jcfVH0+jUwnVAMTVQO9EmY7g0syk102xwLzHb3JckL8n38Iksqq92i169j1snrcTSzc4ATgLEeVUgnS+O7kBXuvsTdt7j7VuDBFO+b7+PXEjgFmJZqnVwdvxTnlJx8B5UgMiiqs/w98L67/zbFOrtF62Fmwwl/g+U5iq+dmXWoHCc0Zr6XtNozwNnR1UwHAasqi7I5lPKXWz6PX4JngHOi8XOA/41ZZyYwwMz6RyWi0dF2WWdmo4BrgRPdfX2KddL5LmQrvsQ2rZNTvG/ejl/kKOADdy+PW5ir41fNOSU338FstsA3twE4lFCEmwfMiYbjgPHA+Gidy4D5hCsK/gWMyGF8u0fvOzeK4fpofmJ8BtxLuPrhXaA4x8ewkHDC75gwL2/Hj5CoFgObCb/ILgC6ADOAj6LXztG6PYHnErY9jnDVyceVxzpH8ZUS6p4rv4MPJMeX6ruQo/geib5b8wgnrB4N6fhF8x+q/M4lrJuP45fqnJKT76C62hARkViqYhIRkVhKECIiEksJQkREYilBiIhILCUIERGJpQQhUgMz22JVe5nNWM+iZlaU2JOoSEPSMt8BiDQC37r7kHwHIZJrKkGI1FH0PIDfmNnb0bBnNL+fmc2IOqObYWZ9o/ndLTyfYW40jIh2VWBmD0b9/b9kZm2j9a8wswXRfqbm6WNKM6YEIVKztklVTGckLFvt7sOBe4C7onn3ELpM35/QUd7d0fy7gdc8dDQ4lHAHLsAA4F533xdYCZwazZ8AHBDtZ3y2PpxIKrqTWqQGZrbW3dvHzC8Dvufun0Qdqn3l7l3MbBmh+4jN0fzF7t7VzJYCvd19Y8I+ioC/uvuAaPpaoJW732JmLwBrCT3WPu1RJ4UiuaIShEj9eIrxVOvE2ZgwvoXtbYPHE/rFGgbMinoYFckZJQiR+jkj4fXNaPyfhJ4zAcYC/4jGZwAXA5hZgZntnGqnZtYC6OPurwLXAJ2AHUoxItmkXyQiNWtrVR9c/4K7V17q2trM3iL82BoTzbsCmGxmPwOWAudF838CTDKzCwglhYsJPYnGKQAeNbOOhB5273T3lRn7RCJpUBuESB1FbRDF7r4s37GIZIOqmEREJJZKECIiEkslCBERiaUEISIisZQgREQklhKEiIjEUoIQEZFY/x/joOTBGje39gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 画图 Plotting the training and validation accuracy\n",
    "plt.clf()\n",
    "acc = history_dict[\"acc\"]\n",
    "val_acc = history_dict[\"val_acc\"]\n",
    "plt.plot(epochs, acc, \"bo\", label=\"Training acc\")\n",
    "plt.plot(epochs, val_acc, \"b\", label=\"Validation acc\")\n",
    "plt.title(\"Training and validation accuracy\")\n",
    "plt.xlabel(\"Epochs\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如你所见，训练损失每轮都在降低，训练精度每轮都在提升。这就是梯度下降优化的预期结果——你想要最小化的量随着每次迭代越来越小。但验证损失和验证精度并非如此：它们似乎在第四轮达到最佳值。这就是我们之前警告过的一种情况：模型在训练数据上的表现越来越好，但在前所未见的数据上不一定表现得越来越好。准确地说，你看到的是过拟合（overfit）：在第二轮之后，你对训练数据过度优化，最终学到的表示仅针对于训练数据，无法泛化到训练集之外的数据。\n",
    "\n",
    "在这种情况下，为了防止过拟合，你可以在  3轮之后停止训练。通常来说，你可以使用许多方法来降低过拟合，我们将在第 4章中详细介绍。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "49/49 [==============================] - 1s 10ms/step - loss: 0.5699 - accuracy: 0.7318\n",
      "Epoch 2/4\n",
      "49/49 [==============================] - 1s 10ms/step - loss: 0.2919 - accuracy: 0.9033\n",
      "Epoch 3/4\n",
      "49/49 [==============================] - 0s 10ms/step - loss: 0.2078 - accuracy: 0.9277\n",
      "Epoch 4/4\n",
      "49/49 [==============================] - 0s 10ms/step - loss: 0.1653 - accuracy: 0.9448\n",
      "782/782 [==============================] - 1s 736us/step - loss: 0.3128 - accuracy: 0.8761\n"
     ]
    }
   ],
   "source": [
    "#### 从头训练一个模型 Retraining a model from scratch\n",
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop',\n",
    "        loss='binary_crossentropy',\n",
    "        metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.312835693359375, 0.8761199712753296]"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results #平均值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.1676541 ],\n",
       "       [0.99964523],\n",
       "       [0.6325836 ],\n",
       "       ...,\n",
       "       [0.07757184],\n",
       "       [0.04434693],\n",
       "       [0.42856777]], dtype=float32)"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Using a trained model to generate predictions on new data\n",
    "model.predict(x_test)\n",
    "#网络对某些样本的结果非常确信（大于等于 0.99，或小于等于 0.01），但对其他结果却不那么确信（0.6或  0.4）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Further experiments 进一步的实验\n",
    "前面使用了两个隐藏层。你可以尝试使用一个或三个隐藏层，然后观察对验证精度和测试精度的影响。  \n",
    "尝试使用更多或更少的隐藏单元，比如  32个、64个等。  \n",
    "尝试使用mse损失函数代替binary_crossentropy。  \n",
    "尝试使用tanh激活（这种激活在神经网络早期非常流行）代替relu。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "49/49 [==============================] - 1s 10ms/step - loss: 0.1727 - accuracy: 0.7502\n",
      "Epoch 2/4\n",
      "49/49 [==============================] - 1s 11ms/step - loss: 0.0758 - accuracy: 0.9141\n",
      "Epoch 3/4\n",
      "49/49 [==============================] - 1s 11ms/step - loss: 0.0556 - accuracy: 0.9359\n",
      "Epoch 4/4\n",
      "49/49 [==============================] - 1s 10ms/step - loss: 0.0437 - accuracy: 0.9505\n",
      "782/782 [==============================] - 1s 746us/step - loss: 0.0888 - accuracy: 0.8799\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(32, activation='tanh', input_shape=(10000,)))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop',\n",
    "        loss='mean_squared_error',\n",
    "        metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "49/49 [==============================] - 1s 10ms/step - loss: 0.1697 - accuracy: 0.7428\n",
      "Epoch 2/4\n",
      "49/49 [==============================] - 0s 10ms/step - loss: 0.0675 - accuracy: 0.9101\n",
      "Epoch 3/4\n",
      "49/49 [==============================] - 0s 10ms/step - loss: 0.0497 - accuracy: 0.9362\n",
      "Epoch 4/4\n",
      "49/49 [==============================] - 0s 10ms/step - loss: 0.0418 - accuracy: 0.9491\n",
      "782/782 [==============================] - 1s 775us/step - loss: 0.1063 - accuracy: 0.8638\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(16, activation='tanh', input_shape=(10000,)))\n",
    "model.add(layers.Dense(16, activation='tanh'))\n",
    "model.add(layers.Dense(16, activation='tanh'))\n",
    "model.add(layers.Dense(16, activation='tanh'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop',\n",
    "        loss='mean_squared_error',\n",
    "        metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/4\n",
      "49/49 [==============================] - 3s 14ms/step - loss: 0.1828 - accuracy: 0.7188\n",
      "Epoch 2/4\n",
      "49/49 [==============================] - 1s 14ms/step - loss: 0.0713 - accuracy: 0.9069\n",
      "Epoch 3/4\n",
      "49/49 [==============================] - 1s 14ms/step - loss: 0.0496 - accuracy: 0.9361\n",
      "Epoch 4/4\n",
      "49/49 [==============================] - 1s 17ms/step - loss: 0.0380 - accuracy: 0.9520\n",
      "782/782 [==============================] - 2s 2ms/step - loss: 0.0974 - accuracy: 0.8734\n"
     ]
    }
   ],
   "source": [
    "model = models.Sequential()\n",
    "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(64, activation='relu'))\n",
    "model.add(layers.Dense(1, activation='sigmoid'))\n",
    "model.compile(optimizer='rmsprop',\n",
    "        loss='mean_squared_error',\n",
    "        metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, epochs=4, batch_size=512)\n",
    "results = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 下面是你应该从这个例子中学到的要点。\n",
    "\n",
    "通常需要对原始数据进行大量预处理，以便将其转换为张量输入到神经网络中。单词序列可以编码为二进制向量，但也有其他编码方式。  \n",
    "带有relu激活的Dense层堆叠，可以解决很多种问题（包括情感分类），你可能会经常用到这种模型。  \n",
    "对于二分类问题（两个输出类别），网络的最后一层应该是只有一个单元并使用sigmoid激活的Dense层，网络输出应该是0~1范围内的标量，表示概率值。  \n",
    "对于二分类问题的  sigmoid标量输出，你应该使用binary_crossentropy损失函数。  \n",
    "无论你的问题是什么，rmsprop优化器通常都是足够好的选择。这一点你无须担心。  \n",
    "随着神经网络在训练数据上的表现越来越好，模型最终会过拟合，并在前所未见的数据上得到越来越差的结果。一定要一直监控模型在训练集之外的数据上的性能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}