Chapter 6.4 用卷积神经网络处理序列.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第 5章我们学习了卷积神经网络（convnet），并知道它在计算机视觉问题上表现出色，原因在于它能够进行卷积运算，从局部输入图块中提取特征，并能够将表示模块化，同时可以高效地利用数据。这些性质让卷积神经网络在计算机视觉领域表现优异，同样也让它对序列处理特别有效。时间可以被看作一个空间维度，就像二维图像的高度或宽度。\n",
    "\n",
    "对于某些序列处理问题，这种一维卷积神经网络的效果可以媲美  RNN，而且计算代价通常要小很多。最近，一维卷积神经网络［通常与空洞卷积核（dilated  kernel）一起使用］已经在音频生成和机器翻译领域取得了巨大成功。除了这些具体的成就，人们还早已知道，对于文本分类和时间序列预测等简单任务，小型的一维卷积神经网络可以替代 RNN，而且速度更快"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.1　理解序列数据的一维卷积\n",
    "这种一维卷积层可以识别序列中的局部模式。因为对每个序列段执行相同的输入变换，所以在句子中某个位置学到的模式稍后可以在其他位置被识别，这使得一维卷积神经网络具有平移不变性（对于时间平移而言）。举个例子，使用大小为 5的卷积窗口处理字符序列的一维卷积神经网络，应该能够学习长度不大于  5的单词或单词片段，并且应该能够在输入句子中的任何位置识别这些单词或单词段。因此，字符级的一维卷积神经网络能够学会单词构词法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.2　序列数据的一维池化\n",
    "你已经学过二维池化运算，比如二维平均池化和二维最大池化，在卷积神经网络中用于对图像张量进行空间下采样。一维也可以做相同的池化运算：从输入中提取一维序列段（即子序列），然后输出其最大值（最大池化）或平均值（平均池化）。与二维卷积神经网络一样，该运算也是用于降低一维输入的长度（子采样）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.3　实现一维卷积神经网络\n",
    "Keras中的一维卷积神经网络是 Conv1D层，其接口类似于 Conv2D。它接收的输入是形状为(samples,  time, features)的三维张量，并返回类似形状的三维张量。卷积窗口是时间轴上的一维窗口（时间轴是输入张量的第二个轴）。\n",
    "\n",
    "我们来构建一个简单的两层一维卷积神经网络，并将其应用于我们熟悉的   IMDB情感分类任务。提醒一下，获取数据并预处理的代码如下所示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import imdb\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "from tensorflow.keras import layers\n",
    "from tensorflow.keras.models import Sequential\n",
    "import os, shutil\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras.optimizers import RMSprop\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading data...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<__array_function__ internals>:5: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "/Users/yihuawang/conda/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:159: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
      "/Users/yihuawang/conda/lib/python3.8/site-packages/tensorflow/python/keras/datasets/imdb.py:160: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.\n",
      "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000 train sequences\n",
      "25000 test sequences\n",
      "Pad sequences (samples x time)\n",
      "x_train shape: (25000, 500)\n",
      "x_test shape: (25000, 500)\n"
     ]
    }
   ],
   "source": [
    "# 准备IMDB数据\n",
    "max_features = 10000\n",
    "max_len = 500\n",
    "print('Loading data...')\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "print('Pad sequences (samples x time)')\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=max_len)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=max_len)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一维卷积神经网络的架构与第 5章的二维卷积神经网络相同，它是Conv1D层和MaxPooling1D层的堆叠，最后是一个全局池化层或Flatten层，将三维输出转换为二维输出，让你可以向模型中添加一个或多个Dense层，用于分类或回归。\n",
    "\n",
    "不过二者有一点不同：一维卷积神经网络可以使用更大的卷积窗口。对于二维卷积层，3×3的卷积窗口包含  3×3=9个特征向量；但对于一位卷积层，大小为    3的卷积窗口只包含 3个卷积向量。因此，你可以轻松使用大小等于 7或 9的一维卷积窗口。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, 500, 128)          1280000   \n",
      "_________________________________________________________________\n",
      "conv1d (Conv1D)              (None, 494, 32)           28704     \n",
      "_________________________________________________________________\n",
      "max_pooling1d (MaxPooling1D) (None, 98, 32)            0         \n",
      "_________________________________________________________________\n",
      "conv1d_1 (Conv1D)            (None, 92, 32)            7200      \n",
      "_________________________________________________________________\n",
      "global_max_pooling1d (Global (None, 32)                0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 1,315,937\n",
      "Trainable params: 1,315,937\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "157/157 [==============================] - 24s 151ms/step - loss: 1.5239 - acc: 0.4943 - val_loss: 0.6894 - val_acc: 0.5398\n",
      "Epoch 2/10\n",
      "157/157 [==============================] - 23s 149ms/step - loss: 0.6808 - acc: 0.5989 - val_loss: 0.6721 - val_acc: 0.6214\n",
      "Epoch 3/10\n",
      "157/157 [==============================] - 23s 145ms/step - loss: 0.6456 - acc: 0.7331 - val_loss: 0.6236 - val_acc: 0.7378\n",
      "Epoch 4/10\n",
      "157/157 [==============================] - 23s 144ms/step - loss: 0.5786 - acc: 0.7962 - val_loss: 0.5220 - val_acc: 0.7994\n",
      "Epoch 5/10\n",
      "157/157 [==============================] - 23s 146ms/step - loss: 0.4605 - acc: 0.8367 - val_loss: 0.4179 - val_acc: 0.8308\n",
      "Epoch 6/10\n",
      "157/157 [==============================] - 23s 148ms/step - loss: 0.3596 - acc: 0.8689 - val_loss: 0.4021 - val_acc: 0.8504\n",
      "Epoch 7/10\n",
      "157/157 [==============================] - 25s 161ms/step - loss: 0.3102 - acc: 0.8916 - val_loss: 0.3934 - val_acc: 0.8572\n",
      "Epoch 8/10\n",
      "157/157 [==============================] - 24s 151ms/step - loss: 0.2788 - acc: 0.9043 - val_loss: 0.4035 - val_acc: 0.8638\n",
      "Epoch 9/10\n",
      "157/157 [==============================] - 24s 153ms/step - loss: 0.2562 - acc: 0.9138 - val_loss: 0.4098 - val_acc: 0.8682\n",
      "Epoch 10/10\n",
      "157/157 [==============================] - 24s 153ms/step - loss: 0.2309 - acc: 0.9239 - val_loss: 0.4395 - val_acc: 0.8722\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Embedding(max_features, 128, input_length=max_len))\n",
    "\n",
    "model.add(layers.Conv1D(32, 7, activation='relu')) # Conv1D\n",
    "\n",
    "model.add(layers.MaxPooling1D(5)) # pooling\n",
    "model.add(layers.Conv1D(32, 7, activation='relu')) # Conv1D\n",
    "model.add(layers.GlobalMaxPooling1D()) # 最后一个全局池化\n",
    "model.add(layers.Dense(1))\n",
    "model.summary()\n",
    "model.compile(optimizer=RMSprop(lr=1e-4),\n",
    "    loss='binary_crossentropy',\n",
    "    metrics=['acc'])\n",
    "history = model.fit(x_train, y_train,\n",
    "    epochs=10,\n",
    "    batch_size=128,\n",
    "    validation_split=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU1f3/8dcHZAsICKioQYLfIigii5EqLkUFpYWKuIJRQVopblT81V2rVWmt0rrUragUl1jqVqoWXKBarLaWgLiASFFZIi4YNewQ4PP740zCJGSZhCR3ZvJ+Ph7zmLucufOZO+TDmXPPPcfcHRERSX2Nog5ARERqhxK6iEiaUEIXEUkTSugiImlCCV1EJE0ooYuIpAkl9DRmZjPNbFRtl42SmS0zs4F1cFw3s+/Flh80sxsSKVuD98kxs1dqGqdIZUz90JOLma2LW80ANgPbYus/c/fc+o8qeZjZMuCn7j6rlo/rQFd3X1pbZc0sC/gUaOLuW2sjTpHK7BZ1AFKau7cqXq4seZnZbkoSkiz07zE5qMklRZjZADPLN7OrzOwL4E9mtoeZvWhmq83s29hyZtxrXjezn8aWR5vZv8xsUqzsp2b2wxqW7WJmc8xsrZnNMrP7zOyJCuJOJMZbzOzN2PFeMbMOcfvPNbPlZlZgZtdVcn6OMLMvzKxx3LbhZvZebLmfmf3bzL4zs8/N7F4za1rBsaaa2a1x61fEXrPKzMaUKTvEzN4xszVmttLMborbPSf2/J2ZrTOzI4vPbdzr+5vZXDMrjD33T/TcVPM8tzOzP8U+w7dmNj1u3zAzWxD7DB+b2eDY9lLNW2Z2U/H3bGZZsaann5jZCuAfse1Px76Hwti/kR5xr29hZr+LfZ+FsX9jLczs72Z2aZnP856ZnVLeZ5WKKaGnlo5AO6AzMJbw/f0ptr4/sBG4t5LXfx/4COgA3A48YmZWg7JPAv8F2gM3AedW8p6JxHg2cD6wF9AU+AWAmR0MPBA7/r6x98ukHO7+H2A9cHyZ4z4ZW94GTIh9niOBE4CLKombWAyDY/EMAroCZdvv1wPnAW2BIcCFcYno2NhzW3dv5e7/LnPsdsDfgXtin+33wN/NrH2Zz7DTuSlHVef5cUITXo/Yse6MxdAPeAy4IvYZjgWWVXQ+yvED4CDgpNj6TMJ52guYD8Q3EU4CDgP6E/4dXwlsBx4FzikuZGa9gP2AGdWIQwDcXY8kfRD+sAbGlgcAW4DmlZTvDXwbt/46ockGYDSwNG5fBuBAx+qUJSSLrUBG3P4ngCcS/EzlxXh93PpFwEux5V8C0+L2tYydg4EVHPtWYEpseXdCsu1cQdnLgL/GrTvwvdjyVODW2PIU4La4cgfGly3nuHcBd8aWs2Jld4vbPxr4V2z5XOC/ZV7/b2B0VeemOucZ2IeQOPcop9wfi+Ot7N9fbP2m4u857rMdUEkMbWNl2hD+w9kI9CqnXDPgG8J1CQiJ//76/ntLh4dq6KlltbtvKl4xswwz+2PsJ+wawk/8tvHNDmV8Ubzg7htii62qWXZf4Ju4bQArKwo4wRi/iFveEBfTvvHHdvf1QEFF70WojZ9qZs2AU4H57r48FseBsWaIL2Jx/JpQW69KqRiA5WU+3/fN7LVYU0chMC7B4xYfe3mZbcsJtdNiFZ2bUqo4z50I39m35by0E/BxgvGWp+TcmFljM7st1myzhh01/Q6xR/Py3svdNwNPAeeYWSNgJOEXhVSTEnpqKdsl6f8B3YDvu3trdvzEr6gZpTZ8DrQzs4y4bZ0qKb8rMX4ef+zYe7avqLC7LyIkxB9SurkFQtPNYkItsDVwbU1iIPxCifck8DzQyd3bAA/GHbeqLmSrCE0k8fYHPksgrrIqO88rCd9Z23JetxL4vwqOuZ7w66xYx3LKxH/Gs4FhhGapNoRafHEMXwObKnmvR4EcQlPYBi/TPCWJUUJPbbsTfsZ+F2uPvbGu3zBW480DbjKzpmZ2JPDjOorxGWComR0du4B5M1X/m30SGE9IaE+XiWMNsM7MugMXJhjDU8BoMzs49h9K2fh3J9R+N8Xao8+O27ea0NRxQAXHngEcaGZnm9luZnYWcDDwYoKxlY2j3PPs7p8T2rbvj108bWJmxQn/EeB8MzvBzBqZ2X6x8wOwABgRK58NnJ5ADJsJv6IyCL+CimPYTmi++r2Z7RurzR8Z+zVFLIFvB36Hauc1poSe2u4CWhBqP/8BXqqn980hXFgsILRb/4Xwh1yeGsfo7guBiwlJ+nPgWyC/ipf9mXC94R/u/nXc9l8Qku1a4KFYzInEMDP2Gf4BLI09x7sIuNnM1hLa/J+Ke+0GYCLwpoXeNUeUOXYBMJRQuy4gXCQcWibuRFV1ns8Figi/Ur4iXEPA3f9LuOh6J1AI/JMdvxpuINSovwV+RelfPOV5jPAL6TNgUSyOeL8A3gfmEtrMf0vpHPQY0JNwTUZqQDcWyS4zs78Ai929zn8hSPoys/OAse5+dNSxpCrV0KXazOxwM/u/2E/0wYR20+lVvU6kIrHmrIuAyVHHksqU0KUmOhK61K0j9KG+0N3fiTQiSVlmdhLhesOXVN2sI5VQk4uISJpQDV1EJE1ENjhXhw4dPCsrK6q3FxFJSfPmzfva3fcsb19kCT0rK4u8vLyo3l5EJCWZWdm7i0uoyUVEJE0ooYuIpAkldBGRNJFUMxYVFRWRn5/Ppk2bqi4skWjevDmZmZk0adIk6lBEpIykSuj5+fnsvvvuZGVlUfG8CxIVd6egoID8/Hy6dOkSdTgiUkZSNbls2rSJ9u3bK5knKTOjffv2+gUlUkO5uZCVBY0ahefcWp7yPalq6ICSeZLT9yNSM7m5MHYsbIhNDbN8eVgHyMmpnfdIqhq6iEi6uu66Hcm82IYNYXttUUKPU1BQQO/evenduzcdO3Zkv/32K1nfsmVLpa/Ny8tj/PjxVb5H//79qywjIulnxYrqba+JlE7otd0e1b59exYsWMCCBQsYN24cEyZMKFlv2rQpW7durfC12dnZ3HPPPVW+x1tvvbVrQYpItdV123Ui9i87eWEV22siZRN6cXvU8uXgvqM9qra/qNGjR3P55Zdz3HHHcdVVV/Hf//6X/v3706dPH/r3789HH30EwOuvv87QoUMBuOmmmxgzZgwDBgzggAMOKJXoW7VqVVJ+wIABnH766XTv3p2cnJziGdCZMWMG3bt35+ijj2b8+PElx423bNkyjjnmGPr27Uvfvn1L/Udx++2307NnT3r16sXVV18NwNKlSxk4cCC9evWib9++fPzxrswLLJI66itXVGXiRMjIKL0tIyNsrzXuHsnjsMMO87IWLVq007aKdO7sHr6e0o/OnRM+RKVuvPFGv+OOO3zUqFE+ZMgQ37p1q7u7FxYWelFRkbu7v/rqq37qqae6u/trr73mQ4YMKXntkUce6Zs2bfLVq1d7u3btfMuWLe7u3rJly5LyrVu39pUrV/q2bdv8iCOO8DfeeMM3btzomZmZ/sknn7i7+4gRI0qOG2/9+vW+ceNGd3dfsmSJF5/PGTNm+JFHHunr1693d/eCggJ3d+/Xr58/99xz7u6+cePGkv01UZ3vSSRqdZ0rquOJJ8L7moXnJ56o/jGAPK8gryZdL5dE1Ud7VLEzzjiDxo0bA1BYWMioUaP43//+h5lRVFRU7muGDBlCs2bNaNasGXvttRdffvklmZmZpcr069evZFvv3r1ZtmwZrVq14oADDijp5z1y5EgmT955EpeioiIuueQSFixYQOPGjVmyZAkAs2bN4vzzzycjVhVo164da9eu5bPPPmP48OFAuDlIpKGoz1xRlZyc2uvRUp6UbXKpj/aoYi1btixZvuGGGzjuuOP44IMPeOGFFyrsk92sWbOS5caNG5fb/l5eGU9wwpE777yTvffem3fffZe8vLySi7buvlPXwkSPKZKO6jNXRC1lE3q9tEeVo7CwkP322w+AqVOn1vrxu3fvzieffMKyZcsA+Mtfyp+cvrCwkH322YdGjRrx+OOPs23bNgBOPPFEpkyZwoZY/6hvvvmG1q1bk5mZyfTpYdrPzZs3l+wXqUvJcDEyqlwRhZRN6Dk5MHkydO4MZuF58uS6/TkDcOWVV3LNNddw1FFHlSTR2tSiRQvuv/9+Bg8ezNFHH83ee+9NmzZtdip30UUX8eijj3LEEUewZMmSkl8RgwcP5uSTTyY7O5vevXszadIkAB5//HHuueceDj30UPr3788XX3xR67GLxEuWi5FR5YooJDSnaGxm97uBxsDD7n5bmf17AFOA/wM2AWPc/YPKjpmdne1lJ7j48MMPOeigg6r1AdLRunXraNWqFe7OxRdfTNeuXZkwYULUYZXQ9ySJyMoKSbyszp0h9gNUasDM5rl7dnn7qqyhm1lj4D7gh8DBwEgzO7hMsWuBBe5+KHAeIflLDT300EP07t2bHj16UFhYyM9+9rOoQxKptmS6GNlQJNLLpR+w1N0/ATCzacAwYFFcmYOB3wC4+2IzyzKzvd39y9oOuCGYMGFCUtXIRWpi//3Lr6Gn48XIZJFIG/p+wMq49fzYtnjvAqcCmFk/oDOQWaYMZjbWzPLMLG/16tU1i1hEUkJDuhiZLBJJ6OUNr1e24f02YA8zWwBcCrwD7NRPz90nu3u2u2fvuWe5k1aLSC1Iht4lDeliZLJIpMklH+gUt54JrIov4O5rgPMBLHSC/jT2EJF6Vh/DtCaqrm+kkdISqaHPBbqaWRczawqMAJ6PL2BmbWP7AH4KzIkleRGpZ/UxTKskpyoTurtvBS4BXgY+BJ5y94VmNs7MxsWKHQQsNLPFhN4wP6+rgOvSgAEDePnll0ttu+uuu7jooosqfU1x98sf/ehHfPfddzuVuemmm0r6g1dk+vTpLFq04zrzL3/5S2bNmlWd8EUA9S5pyBIay8XdZwAzymx7MG7530DX2g2t/o0cOZJp06Zx0kknlWybNm0ad9xxR0KvnzFjRtWFKjB9+nSGDh3KwQeHHqE333xzjY8lDZt6lzRcKXunaF04/fTTefHFF9m8eTMQhqhdtWoVRx99NBdeeCHZ2dn06NGDG2+8sdzXZ2Vl8fXXXwMwceJEunXrxsCBA0uG2IXQx/zwww+nV69enHbaaWzYsIG33nqL559/niuuuILevXvz8ccfM3r0aJ555hkAZs+eTZ8+fejZsydjxowpiS8rK4sbb7yRvn370rNnTxYvXrxTTBpmt+FR75KGK2lHW7zsMliwoHaP2bs33HVXxfvbt29Pv379eOmllxg2bBjTpk3jrLPOwsyYOHEi7dq1Y9u2bZxwwgm89957HHrooeUeZ968eUybNo133nmHrVu30rdvXw477DAATj31VC644AIArr/+eh555BEuvfRSTj75ZIYOHcrpp59e6libNm1i9OjRzJ49mwMPPJDzzjuPBx54gMsuuwyADh06MH/+fO6//34mTZrEww8/XOr1e+21F6+++irNmzfnf//7HyNHjiQvL4+ZM2cyffp03n77bTIyMvjmm28AyMnJ4eqrr2b48OFs2rSJ7du31+hcS3SKL0Jed11oZtl//5DMdXEy/amGXkZxswuE5paRI0cC8NRTT9G3b1/69OnDwoULS7V3l/XGG28wfPhwMjIyaN26NSeffHLJvg8++IBjjjmGnj17kpuby8KFCyuN56OPPqJLly4ceOCBAIwaNYo5c+aU7D/11FMBOOyww0oG9IpXVFTEBRdcQM+ePTnjjDNK4k50mN2MslU9SQk5OeH2+u3bw7OSecOQtDX0ymrSdemUU07h8ssvZ/78+WzcuJG+ffvy6aefMmnSJObOncsee+zB6NGjKxw2t1jZIWyLjR49munTp9OrVy+mTp3K66+/Xulxqhprp3gI3oqG6I0fZnf79u0lY6FrmF2R9KMaehmtWrViwIABjBkzpqR2vmbNGlq2bEmbNm348ssvmTlzZqXHOPbYY/nrX//Kxo0bWbt2LS+88ELJvrVr17LPPvtQVFREbtzdHrvvvjtr167d6Vjdu3dn2bJlLF26FAijJv7gBz9I+PNomN36lww39UjDpIRejpEjR/Luu+8yYsQIAHr16kWfPn3o0aMHY8aM4aijjqr09X379uWss86id+/enHbaaRxzzDEl+2655Ra+//3vM2jQILp3716yfcSIEdxxxx306dOn1IXI5s2b86c//YkzzjiDnj170qhRI8aNG0eiNMxu/UqWIWOlYUpo+Ny6oOFzU5e+p4ppyNiGY+tWWLMmPAoLq7d83nkwfnzN3rey4XOTtg1dJBXppp7kt20brF1bedJNJCkn0hrZuDG0bg1t2ux47tgR2ratm8+mhC5Si3RTT93bvDkk1u++2/lR3vb4bYWFsG5d1e9hFhJw8aNNG2jfHrp0KZ2c4/eXt61Fi3Cs+pJ0Cb283heSPNQTpnITJ5YeGAt0U088d9i4serEW9n2KjqY0bhxSKZt2+54dOsWthU/KkrIxcstW4aL2qkmqRJ68+bNKSgooH379krqScjdKSgoKOn6KDtriDf1bNwIX34JX3yx47ns8tdf70jIRUWVH69JE9hjj9JJuVOn8Fw2UZe3rWXL+q0VJ5OkuihaVFREfn5+lX28JTrNmzcnMzOTJk2aRB2K1KGiIvjqq50Tc3lJu7Cw/GO0bx/aizt2hA4dQpJOJCk3b95wE3IiUuaiaJMmTejSpUvUYYikpW3boKCg4sQcn7QLCso/Rps2sPfeIUn37r1jufhRvL7XXqGmLfUrqRK6iFTf2rUhCX/++Y6EHL9c/PjqqzAUQFktWsA++4RkfOCBcOyxpZNz8fLee4eykryU0EWS0LZtsHr1jsRc9jl+ef36nV+/2247knFmJmRn71yLLl5u1UpNHOlCCV2kHq1bV3Fijn9evbr82nTbtiER77MP9OsXnouTc/HyPvuE9upU7KUhu0YJXaQWbd0Kc+fCG29Afv7Oybu8PtDxtelOneDww3dO0MW1aTV5SGWU0CVt5OZG011w+XJ4+WV45RWYPTt0zYNwAbE4GWdn75ygi5fbtVNtWmqHErqkhfqc6X7dOnj99R1JfMmSsD0zE047DU46CY4/PnTbE6lPSdUPXaSm6nJQrO3b4Z13QvJ++WV4663QTzsjAwYMgBNPDEm8WzddXJS6lzL90EVqqrYHxVq1KiTwV16BV18NdzoC9OkDl18ekvhRR0FsfhGRpKCELmlhVwfF2rgxXMgsroV/8EHYvvfe8MMfhhr4wIFhXSRZKaFLWqjuoFjuIWkX18LnzAmDPjVrBsccE8arPukk6NlTzSiSOpTQJS0kMijW6tWh+aQ4iX/+edh+8MFw4YWhGeXYY8N/BCKpSAld0kZOTukEvmVL6I1S3Iwyf37Y3q4dDBoUauCDBoXeKSLpQAld0srHH8OMGSGJv/ZauC1+t92gf3+49daQxPv0CWNmi6QbJXRJee6hBn7nnSGRA3zvezB6dGhGGTAgTFwgku6U0CVlbdgAjz8Od98NH34Y7rq89VYYORIOOCDq6ETqnxK6pJzPPoP77oM//hG++Qb69g2J/cwzoWnTqKMTiY4SuqSMefNCs8pf/hKGlz3lFLjsstDNUF0LRZTQJclt2wZ/+xvcdVe48adVK7j4Yhg/Xs0qImUpoUtSWrMGpkyBe+6BTz8NY7X8/vcwZkwYxVBEdqaELknl00/hD3+Ahx8OU6sddRTccQcMGxa6H4pIxfQnIpFzhzffDO3j06eHscHPPDO0jx9+eNTRiaQOJXSJzJYt8PTToX08Ly9Mm3bllaGNXHdvilSfErrUu4ICmDwZ7r03DFPbrRs88ACcey60bBl1dCKpSwld6s3ixaE2/thjYbjaQYNCW/lJJ2kKNpHaoIQudcodZs0K7eMzZ4bhac85J7SPH3JI1NGJpJeE6kVmNtjMPjKzpWZ2dTn725jZC2b2rpktNLPzaz9USVa5uaFbYaNG4Tk3N9TAH344jCd+4olhpMObbw5D2z78sJK5SF2ock5RM2sMLAEGAfnAXGCkuy+KK3Mt0MbdrzKzPYGPgI7uvqWi42pO0fRQdnJmCN0LW7QI3Q579YIJE2DECE3XJlIbdnVO0X7AUnf/JHawacAwYFFcGQd2NzMDWgHfAFt3KWpJCdddVzqZA2zdGh7/+EcY6VC35YvUj0SaXPYDVsat58e2xbsXOAhYBbwP/Nzdt5c9kJmNNbM8M8tbvXp1DUOWZFLRJMybNsFxxymZi9SnRBJ6eX+SZdtpTgIWAPsCvYF7zWynEajdfbK7Z7t79p577lntYCW5uFd8G36ikzOLSO1JJKHnA53i1jMJNfF45wPPebAU+BToXjshSjIqKoILLoDvvtt59p/KJmcWkbqTSEKfC3Q1sy5m1hQYATxfpswK4AQAM9sb6AZ8UpuBSvIoLIQhQ+CRR+D662HqVOjcOTSvdO4cbhqKn9tTROpHlRdF3X2rmV0CvAw0Bqa4+0IzGxfb/yBwCzDVzN4nNNFc5e5f12HcEpEVK0IyX7w4jIZ4fqyD6jnnRBuXiCR4Y5G7zwBmlNn2YNzyKuDE2g1Nks28eTB0aOjV8tJLcMIJUUckIvF0w7Uk5IUX4NhjwxRvb72lZC6SjJTQpUr33humezv4YHj7bejRI+qIRKQ8SuhSoW3bwl2el14KP/4xvP46dOwYdVQiUhEldCnX+vVw2mlhdMTLLoNnn9XQtiLJTqMtyk6++CLUyOfPD3N6Xnpp1BGJSCKU0KWUhQtDt8TVq8N0cD/+cdQRiUii1OQiJWbPDpMyb94Mc+YomYukGiV0AcLdnoMHh7k8334bDjss6ohEpLqU0Bs4d7jhhnDH54AB8OabGlhLJFWpDb0B27wZxoyBJ5+En/wkTNTcpEnUUYlITSmhN1AFBTB8OLzxBvz613D11Rq7XCTVKaE3QB9/DD/6ESxbBn/+c5geTkRSnxJ6A/PWWzBsGGzfHnq1HH101BGJSG3RRdEG5Omn4fjjoW1b+M9/lMxF0o0SegPgDrffDmeeCdnZ8O9/Q9euUUclIrVNCT3Nbd0K48bBVVeFtvJZs6BDh6ijEpG6oISextasCRNSTJ4M11wDubnQvHnUUYlIXdFF0TS1cmUYk2XRInjoIfjpT6OOSETqmmroKSw3F7KyoFGj8JybG7a/8w4ccQQsXw4zZyqZizQUqqGnqNxcGDs2zO8JIXmPHQsLFoQ7Ptu1g3/9C3r2jDZOEak/qqGnqOuu25HMi23YAJMmQbduoVuikrlIw6IaeopasaLiff/8J7RqVX+xiEhyUA09RVU0IuL++yuZizRUSugpauJEaNGi9LaMjDDQlog0TEroKSonJ1wELda5c+hvnpMTXUwiEi21oaeodevg2WehVy/Iy4Pd9E2KNHhKAynqppsgPx+eekrJXEQCNbmkoPfeg7vuCjcMHXlk1NGISLJQQk8x27fDhRfCHnvAbbdFHY2IJBP9WE8xU6eGSSqmTIH27aOORkSSiWroKaSgAK68MkxMMWpU1NGISLJRQk8hV10F330XxmpppG9ORMpQWkgRb74JjzwCEybAIYdEHY2IJCMl9BRQVBQuhHbqBDfeGHU0IpKsdFE0BfzhD/D++/DccxqnRUQqphp6ksvPD7XyIUPglFOijkZEkpkSepK77LIw0fMf/gBmUUcjIslMCT2JzZwZxmu5/nro0iXqaEQk2SWU0M1ssJl9ZGZLzezqcvZfYWYLYo8PzGybmbWr/XAbjo0b4ZJLwuxDv/hF1NGISCqo8qKomTUG7gMGAfnAXDN73t0XFZdx9zuAO2LlfwxMcPdv6ibkhuE3v4FPPoHZs6FZs6ijEZFUkEgNvR+w1N0/cfctwDRgWCXlRwJ/ro3gGqolS+C3vw1jmx9/fNTRiEiqSCSh7wesjFvPj23biZllAIOBZ3c9tIbJHS66KMxGNGlS1NGISCpJpB96eX0rvIKyPwberKi5xczGAmMB9q9oUswGbtq00Mxy773QsWPU0YhIKkmkhp4PdIpbzwRWVVB2BJU0t7j7ZHfPdvfsPffcM/EoG4jCQrj8csjOhnHjoo5GRFJNIjX0uUBXM+sCfEZI2meXLWRmbYAfAOfUaoQNyA03wJdfwgsvQOPGUUcjIqmmyoTu7lvN7BLgZaAxMMXdF5rZuNj+B2NFhwOvuPv6Oos2jc2fD/fdF9rPs7OjjkZEUpG5V9QcXreys7M9Ly8vkvdONtu2hankVqyAxYuhbduoIxKRZGVm89y93GqfBudKApMnw9y58MQTSuYiUnO69T9iX34J11wT+pufvdOVCRGRxCmhR+yKK2DDhtB+rsG3RGRXKKFH6PXX4fHHwzyh3btHHY2IpDol9Ihs2RJ6tHTpAtddF3U0IpIOdFE0Ir/7HXz4Ibz4YrjNX0RkV6mGHoFly+CWW2D48DATkYhIbVBCj8D48dCoEdx9d9SRiEg6UZNLPfvb38Kt/XfcAZ06VV1eRCRRqqHXo/XrQ+38kEPg5z+POhoRSTeqodejm28Ot/e/8QY0aRJ1NCKSblRDrycLF8Lvfw/nnw9HHx11NCKSjpTQ60HxLEStW8Ptt0cdjYikKzW51IPHHoM5c+Chh6BDh6ijEZF0pRp6HfvmG/jFL8LwuGPGRB2NiKQzJfQ6ds018O238MADoe+5iEhdUYqpQ//5T2hmGT8eevWKOhoRSXdK6HVk61a48ELYd1/41a+ijkZEGgJdFK0j990HCxbA00/D7rtHHY2INASqodeBVavghhvgpJPgtNOijkZEGgol9DowYUIY7/zeezULkYjUHyX0WvbKK/DUU3DttfC970UdjYg0JErotWjTJrj4YujaFa66KupoRKSh0UXRWvTb38LSpaGW3qxZ1NGISEOjGnotWboUfvMbOOssGDQo6mhEpCFSQq8F7qGppWnTMKKiiEgU1ORSC555JjSz3H13uJFIRCQKqqHvojVr4LLLoE+fMESuiEhUlNBrIDcXsrLCYFuZmfD552Hwrd30e0dEIqSEXk25uTB2LCxfHtrO166Fxo3DRVERkSgpoVfTddfBhg2lt23dGraLiERJCb2aVqyo3nYRkfqihF5N+12oGEkAAAkpSURBVO9fve0iIvVFCb2afvrTnbdlZMDEifUfi4hIPCX0ali2LIxz3qFD6N1iBp07w+TJkJMTdXQi0tCpo12Cvv0WfvQj2LgR3noLDj446ohEREpTQk/A5s0wfPiOgbeUzEUkGSmhV8EdxoyBf/4z9EEfMCDqiEREyqc29Cpcfz08+WS46Hn22VFHIyJSsYQSupkNNrOPzGypmV1dQZkBZrbAzBaa2T9rN8xoTJ4Mv/41XHABXHNN1NGIiFSuyiYXM2sM3AcMAvKBuWb2vLsviivTFrgfGOzuK8xsr7oKuL7MnBkG2xo8GO6/X3ODikjyS6SG3g9Y6u6fuPsWYBowrEyZs4Hn3H0FgLt/Vbth1q/58+GMM+DQQ8P8oBp0S0RSQSIJfT9gZdx6fmxbvAOBPczsdTObZ2bnlXcgMxtrZnlmlrd69eqaRVzHVqyAIUOgXTt48UXYffeoIxIRSUwidc/yGhu8nOMcBpwAtAD+bWb/cfclpV7kPhmYDJCdnV32GJH77rsdfc1nzdJkFSKSWhJJ6PlAp7j1TGBVOWW+dvf1wHozmwP0ApaQIrZsgVNPhSVL4KWXoEePqCMSEameRJpc5gJdzayLmTUFRgDPlynzN+AYM9vNzDKA7wMf1m6odcc9jNHy2mvwyCNw/PFRRyQiUn1V1tDdfauZXQK8DDQGprj7QjMbF9v/oLt/aGYvAe8B24GH3f2Dugy8Nt14Izz+ONxyC5x7btTRiIjUjLlH05SdnZ3teXl5kbx3vEceCbXzn/wEHnpI3RNFJLmZ2Tx3zy5vX4O+U/Tll+FnP4MTTwxzgiqZi0gqa7AJfcECOP10OOQQePppaNIk6ohERHZNg0zoK1eGvuZt28Lf/w6tW0cdkYjIrmtw90AWFoa+5uvWwb/+BfuVvUVKRCRFNaiEvmVLaGZZvDiM1dKzZ9QRiYjUngaT0N1h7NhwB+jUqTBwYNQRiYjUrgbThn7zzfDoo3DTTTBqVNTRiIjUvgaR0KdODYl89Gj45S8jDkZEpI6kfUKfNStMUDFwYJiwQn3NRSRdpXVCf++9MODWQQfBM8+or7mIpLe0Tej5+aF7YuvWMGMGtGkTdUQiInUrLXu5rFkTbhxaswbeeAMyM6OOSESk7qVdQi8qCtPHLVwYaua9ekUdkYhI/UirhO4O48bBK6+EURRPPDHqiERE6k9ataHfeitMmQI33ABjxkQdjYhI/UqbhP7YY6GP+bnnwq9+FXU0IiL1Ly0S+uzZYYKK44+Hhx9WX3MRaZhSPqF/8EHoa96tGzz7LDRtGnVEIiLRSOmEvmpV6GvesmXo0dK2bdQRiYhEJ2V7uaxdG/qaf/stzJkD++8fdUQiItFKyYReVARnngnvvw8vvAB9+kQdkYhI9FIuobvDRRfBSy+FwbZ++MOoIxIRSQ4p14Y+dWroyXLttWEURRERCVKuhn7WWWE+0EsuiToSEZHkknIJPSMDLr006ihERJJPyjW5iIhI+ZTQRUTShBK6iEiaUEIXEUkTSugiImlCCV1EJE0ooYuIpAkldBGRNKGELiKSJpTQRUTShBK6iEiaSKmEnpsLWVnQqFF4zs2NOiIRkeSRMoNz5ebC2LGwYUNYX748rAPk5EQXl4hIskiohm5mg83sIzNbamZXl7N/gJkVmtmC2OOXtR3oddftSObFNmwI20VEJIEaupk1Bu4DBgH5wFwze97dF5Up+oa7D62DGAFYsaJ620VEGppEauj9gKXu/om7bwGmAcPqNqydVTQJtCaHFhEJEkno+wEr49bzY9vKOtLM3jWzmWbWo1aiizNxYpjcIl5GRtguIiKJJXQrZ5uXWZ8PdHb3XsAfgOnlHshsrJnlmVne6tWrqxVoTk6YFLpzZzALz5Mn64KoiEixRBJ6PtApbj0TWBVfwN3XuPu62PIMoImZdSh7IHef7O7Z7p695557VjvYnBxYtgy2bw/PSuYiIjskktDnAl3NrIuZNQVGAM/HFzCzjmZmseV+seMW1HawIiJSsSp7ubj7VjO7BHgZaAxMcfeFZjYutv9B4HTgQjPbCmwERrh72WYZERGpQxZV3s3Ozva8vLxI3ltEJFWZ2Tx3zy5vX0rd+i8iIhVTQhcRSRORNbmY2WpgeSRvXns6AF9HHUQS0fkoTedjB52L0nblfHR293K7CUaW0NOBmeVV1JbVEOl8lKbzsYPORWl1dT7U5CIikiaU0EVE0oQS+q6ZHHUASUbnozSdjx10Lkqrk/OhNnQRkTShGrqISJpQQhcRSRNK6DVgZp3M7DUz+9DMFprZz6OOKWpm1tjM3jGzF6OOJWpm1tbMnjGzxbF/I0dGHVOUzGxC7O/kAzP7s5k1jzqm+mRmU8zsKzP7IG5bOzN71cz+F3veozbeSwm9ZrYC/8/dDwKOAC42s4MjjilqPwc+jDqIJHE38JK7dwd60YDPi5ntB4wHst39EMIAfyOijareTQUGl9l2NTDb3bsCs2Pru0wJvQbc/XN3nx9bXkv4gy1vFqcGwcwygSHAw1HHEjUzaw0cCzwC4O5b3P27aKOK3G5ACzPbDcigzHwK6c7d5wDflNk8DHg0tvwocEptvJcS+i4ysyygD/B2tJFE6i7gSmB71IEkgQOA1cCfYk1QD5tZy6iDioq7fwZMAlYAnwOF7v5KtFElhb3d/XMIFURgr9o4qBL6LjCzVsCzwGXuvibqeKJgZkOBr9x9XtSxJIndgL7AA+7eB1hPLf2cTkWxtuFhQBdgX6ClmZ0TbVTpSwm9hsysCSGZ57r7c1HHE6GjgJPNbBkwDTjezJ6INqRI5QP57l78i+0ZQoJvqAYCn7r7ancvAp4D+kccUzL40sz2AYg9f1UbB1VCr4HYdHuPAB+6+++jjidK7n6Nu2e6exbhYtc/3L3B1sDc/QtgpZl1i206AVgUYUhRWwEcYWYZsb+bE2jAF4njPA+Mii2PAv5WGwetcgo6KddRwLnA+2a2ILbt2tgE2SKXArmxOXg/Ac6POJ7IuPvbZvYMMJ/QO+wdGtgwAGb2Z2AA0MHM8oEbgduAp8zsJ4T/9M6olffSrf8iIulBTS4iImlCCV1EJE0ooYuIpAkldBGRNKGELiKSJpTQRUTShBK6iEia+P/Or9ZLMQmAOwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxU5d338c/PsIZFBOJGJIEWRBAINCAKIm4VqgWlegtGEalSXKu2VlpvK9Vyv2zr09v6uJXiVptKfaSlqFQtIOLWSgSKoqjIohGtAcoiiyz+nj+uCUyGSTIJk0wy832/XvPKzDnXnPnNBL45c53rXMfcHRERafwOSXUBIiKSHAp0EZE0oUAXEUkTCnQRkTShQBcRSRMKdBGRNKFAl7jM7G9mdmmy26aSma0xszPqYLtuZl+P3H/QzG5NpG0tXqfIzF6obZ1VbHeYmZUme7tS/5qkugBJHjP7IuphNvAlsDfy+HvuXpzottx9RF20TXfuPikZ2zGzfGA10NTd90S2XQwk/DuUzKNATyPu3rr8vpmtAS5397mx7cysSXlIiEj6UJdLBij/Sm1mN5vZZ8AjZnaYmT1jZmVm9p/I/dyo5ywws8sj98eb2Stmdlek7WozG1HLtl3MbKGZbTWzuWZ2n5n9oZK6E6nxDjN7NbK9F8ysY9T6S8xsrZltMLNbqvh8BpnZZ2aWFbXsPDNbFrk/0MxeN7NNZvapmd1rZs0q2dajZvbzqMc3RZ6zzswmxLQ928yWmNkWM/vYzKZErV4Y+bnJzL4wsxPLP9uo559kZovMbHPk50mJfjZVMbPjIs/fZGbLzWxk1Lpvmdk7kW1+YmY/jCzvGPn9bDKzjWb2spkpX+qZPvDMcSTQHsgDJhJ+949EHncGdgD3VvH8E4D3gI7AL4GHzMxq0faPwBtAB2AKcEkVr5lIjRcBlwGHA82A8oDpCTwQ2f7RkdfLJQ53/wewDTgtZrt/jNzfC9wQeT8nAqcDV1VRN5EahkfqORPoBsT2328DxgHtgLOBK83s3Mi6oZGf7dy9tbu/HrPt9sCzwD2R9/Zr4Fkz6xDzHg74bKqpuSnwNPBC5HnXAsVmdmykyUOE7rs2wPHA/MjyHwClQA5wBPATQPOK1DMFeub4CrjN3b909x3uvsHdZ7r7dnffCkwFTqni+Wvd/Xfuvhd4DDiK8B834bZm1hkYAPzU3Xe5+yvA7MpeMMEaH3H39919B/AkUBBZfj7wjLsvdPcvgVsjn0FlngDGAphZG+BbkWW4+5vu/g933+Pua4Dfxqkjnv+K1Pe2u28j/AGLfn8L3P0td//K3ZdFXi+R7UL4A/CBuz8eqesJYAXw7ag2lX02VRkEtAbujPyO5gPPEPlsgN1ATzNr6+7/cffFUcuPAvLcfbe7v+yaKKreKdAzR5m77yx/YGbZZvbbSJfEFsJX/HbR3Q4xPiu/4+7bI3db17Dt0cDGqGUAH1dWcII1fhZ1f3tUTUdHbzsSqBsqey3C3vhoM2sOjAYWu/vaSB3dI90Jn0Xq+B/C3np1KtQArI15fyeY2YuRLqXNwKQEt1u+7bUxy9YCnaIeV/bZVFuzu0f/8Yve7ncIf+zWmtlLZnZiZPmvgJXAC2a2yswmJ/Y2JJkU6Jkjdm/pB8CxwAnu3pb9X/Er60ZJhk+B9maWHbXsmCraH0yNn0ZvO/KaHSpr7O7vEIJrBBW7WyB03awAukXq+EltaiB0G0X7I+EbyjHufijwYNR2q9u7XUfoiorWGfgkgbqq2+4xMf3f+7br7ovcfRShO2YWYc8fd9/q7j9w966Ebwk3mtnpB1mL1JACPXO1IfRJb4r0x95W1y8Y2eMtAaaYWbPI3t23q3jKwdT4FHCOmQ2JHMC8ner/vf8RuI7wh+P/xdSxBfjCzHoAVyZYw5PAeDPrGfmDElt/G8I3lp1mNpDwh6RcGaGLqGsl254DdDezi8ysiZldCPQkdI8cjH8S+vZ/ZGZNzWwY4Xc0I/I7KzKzQ919N+Ez2QtgZueY2dcjx0rKl++N/xJSVxTometuoCWwHvgH8Fw9vW4R4cDiBuDnwJ8I4+XjqXWN7r4cuJoQ0p8C/yEctKvKE8AwYL67r49a/kNC2G4FfhepOZEa/hZ5D/MJ3RHzY5pcBdxuZluBnxLZ2408dzvhmMGrkZEjg2K2vQE4h/AtZgPwI+CcmLprzN13ASMJ31TWA/cD49x9RaTJJcCaSNfTJODiyPJuwFzgC+B14H53X3AwtUjNmY5bSCqZ2Z+AFe5e598QRNKd9tClXpnZADP7mpkdEhnWN4rQFysiB0lnikp9OxL4M+EAZSlwpbsvSW1JIulBXS4iImlCXS4iImkiZV0uHTt29Pz8/FS9vIhIo/Tmm2+ud/eceOtSFuj5+fmUlJSk6uVFRBolM4s9Q3gfdbmIiKQJBbqISJpQoIuIpAmNQxfJILt376a0tJSdO3dW31hSqkWLFuTm5tK0adOEn6NAF8kgpaWltGnThvz8fCq/PomkmruzYcMGSktL6dKlS8LPa1RdLsXFkJ8PhxwSfhbrcrkiNbJz5046dOigMG/gzIwOHTrU+JtUo9lDLy6GiRNhe+TSCGvXhscARUWpq0uksVGYNw61+T01mj30W27ZH+bltm8Py0VEpBEF+kcf1Wy5iDQ8GzZsoKCggIKCAo488kg6deq07/GuXbuqfG5JSQnXXXddta9x0kknJaXWBQsWcM455yRlW/Wl0QR659iLd1WzXEQOXrKPW3Xo0IGlS5eydOlSJk2axA033LDvcbNmzdizZ0+lzy0sLOSee+6p9jVee+21gyuyEWs0gT51KmRnV1yWnR2Wi0jylR+3WrsW3Pcft0r2YITx48dz4403cuqpp3LzzTfzxhtvcNJJJ9GvXz9OOukk3nvvPaDiHvOUKVOYMGECw4YNo2vXrhWCvnXr1vvaDxs2jPPPP58ePXpQVFRE+eyyc+bMoUePHgwZMoTrrruu2j3xjRs3cu6559KnTx8GDRrEsmXLAHjppZf2fcPo168fW7du5dNPP2Xo0KEUFBRw/PHH8/LLLyf3A6tCozkoWn7g85ZbQjdL584hzHVAVKRuVHXcKtn/795//33mzp1LVlYWW7ZsYeHChTRp0oS5c+fyk5/8hJkzZx7wnBUrVvDiiy+ydetWjj32WK688soDxmwvWbKE5cuXc/TRRzN48GBeffVVCgsL+d73vsfChQvp0qULY8eOrba+2267jX79+jFr1izmz5/PuHHjWLp0KXfddRf33XcfgwcP5osvvqBFixZMmzaNs846i1tuuYW9e/eyPfZDrEONJtAh/CNSgIvUj/o8bnXBBReQlZUFwObNm7n00kv54IMPMDN2794d9zlnn302zZs3p3nz5hx++OH8+9//Jjc3t0KbgQMH7ltWUFDAmjVraN26NV27dt03vnvs2LFMmzatyvpeeeWVfX9UTjvtNDZs2MDmzZsZPHgwN954I0VFRYwePZrc3FwGDBjAhAkT2L17N+eeey4FBQUH9dnURKPpchGR+lWfx61atWq17/6tt97Kqaeeyttvv83TTz9d6Vjs5s2b77uflZUVt/89XpvaXNQn3nPMjMmTJzN9+nR27NjBoEGDWLFiBUOHDmXhwoV06tSJSy65hN///vc1fr3aUqCLSFypOm61efNmOnXqBMCjjz6a9O336NGDVatWsWbNGgD+9Kc/VfucoUOHUhw5eLBgwQI6duxI27Zt+fDDD+nduzc333wzhYWFrFixgrVr13L44YdzxRVX8N3vfpfFixcn/T1URoEuInEVFcG0aZCXB2bh57Rpdd/t+aMf/Ygf//jHDB48mL179yZ9+y1btuT+++9n+PDhDBkyhCOOOIJDDz20yudMmTKFkpIS+vTpw+TJk3nssccAuPvuuzn++OPp27cvLVu2ZMSIESxYsGDfQdKZM2fy/e9/P+nvoTIpu6ZoYWGh6wIXIvXr3Xff5bjjjkt1GSn3xRdf0Lp1a9ydq6++mm7dunHDDTekuqwDxPt9mdmb7l4Yr7320EUk4/zud7+joKCAXr16sXnzZr73ve+luqSkaFSjXEREkuGGG25okHvkByuhPXQzG25m75nZSjObHGf9YWb2FzNbZmZvmNnxyS9VRESqUm2gm1kWcB8wAugJjDWznjHNfgIsdfc+wDjgN8kuVEREqpbIHvpAYKW7r3L3XcAMYFRMm57APAB3XwHkm9kRSa1URESqlEigdwI+jnpcGlkW7V/AaAAzGwjkAbkxbTCziWZWYmYlZWVltatYRETiSiTQ482yHjvW8U7gMDNbClwLLAEOOG3L3ae5e6G7F+bk5NS4WBFp3IYNG8bzzz9fYdndd9/NVVddVeVzyoc4f+tb32LTpk0HtJkyZQp33XVXla89a9Ys3nnnnX2Pf/rTnzJ37tyalB9XQ5pmN5FALwWOiXqcC6yLbuDuW9z9MncvIPSh5wCrk1aliKSFsWPHMmPGjArLZsyYkdAEWRBmSWzXrl2tXjs20G+//XbOOOOMWm2roUok0BcB3cysi5k1A8YAs6MbmFm7yDqAy4GF7r4luaWKSGN3/vnn88wzz/Dll18CsGbNGtatW8eQIUO48sorKSwspFevXtx2221xn5+fn8/69esBmDp1KsceeyxnnHHGvil2IYwxHzBgAH379uU73/kO27dv57XXXmP27NncdNNNFBQU8OGHHzJ+/HieeuopAObNm0e/fv3o3bs3EyZM2Fdffn4+t912G/3796d3796sWLGiyveX6ml2qx2H7u57zOwa4HkgC3jY3Zeb2aTI+geB44Dfm9le4B3guwddmYjUqeuvh6VLk7vNggK4++7K13fo0IGBAwfy3HPPMWrUKGbMmMGFF16ImTF16lTat2/P3r17Of3001m2bBl9+vSJu50333yTGTNmsGTJEvbs2UP//v35xje+AcDo0aO54oorAPjv//5vHnroIa699lpGjhzJOeecw/nnn19hWzt37mT8+PHMmzeP7t27M27cOB544AGuv/56ADp27MjixYu5//77ueuuu5g+fXql7y/V0+wmNA7d3ee4e3d3/5q7T40sezAS5rj76+7ezd17uPtod//PQVcmImkputslurvlySefpH///vTr14/ly5dX6B6J9fLLL3PeeeeRnZ1N27ZtGTly5L51b7/9NieffDK9e/emuLiY5cuXV1nPe++9R5cuXejevTsAl156KQsXLty3fvTo0QB84xvf2DehV2VeeeUVLrnkEiD+NLv33HMPmzZtokmTJgwYMIBHHnmEKVOm8NZbb9GmTZsqt50InSkqkqGq2pOuS+eeey433ngjixcvZseOHfTv35/Vq1dz1113sWjRIg477DDGjx9f6bS55czijdcIV0CaNWsWffv25dFHH2XBggVVbqe6+azKp+CtbIre6rZVPs3u2WefzZw5cxg0aBBz587dN83us88+yyWXXMJNN93EuHHjqtx+dTSXi4jUq9atWzNs2DAmTJiwb+98y5YttGrVikMPPZR///vf/O1vf6tyG0OHDuUvf/kLO3bsYOvWrTz99NP71m3dupWjjjqK3bt375vyFqBNmzZs3br1gG316NGDNWvWsHLlSgAef/xxTjnllFq9t1RPs6s9dBGpd2PHjmX06NH7ul769u1Lv3796NWrF127dmXw4MFVPr9///5ceOGFFBQUkJeXx8knn7xv3R133MEJJ5xAXl4evXv33hfiY8aM4YorruCee+7ZdzAUoEWLFjzyyCNccMEF7NmzhwEDBjBp0qRava8pU6Zw2WWX0adPH7KzsytMs/viiy+SlZVFz549GTFiBDNmzOBXv/oVTZs2pXXr1km5EIamzxXJIJo+t3HR9LkiIhlKgS4ikiYU6CIZJlXdrFIztfk9KdBFMkiLFi3YsGGDQr2Bc3c2bNhAixYtavQ8jXIRySC5ubmUlpai2U4bvhYtWpCbe8CktVVSoItkkKZNm9KlS5dUlyF1RF0uIiJpQoEuIpImFOgiImlCgS4ikiYU6CIiaUKBLiKSJhToIiJpQoEuIpImEgp0MxtuZu+Z2Uozmxxn/aFm9rSZ/cvMlpvZZckvVUREqlJtoJtZFnAfMALoCYw1s54xza4G3nH3vsAw4P+YWbMk1yoiIlVIZA99ILDS3Ve5+y5gBjAqpo0DbSxc5K81sBGo+uJ7IiKSVIkEeifg46jHpZFl0e4FjgPWAW8B33f3r2I3ZGYTzazEzEo0OZCISHIlEujxLq0dO/fmWcBS4GigALjXzNoe8CT3ae5e6O6FOTk5NS5WREQql0iglwLHRD3OJeyJR7sM+LMHK4HVQI/klCgiIolIJNAXAd3MrEvkQOcYYHZMm4+A0wHM7AjgWGBVMgsVEZGqVTsfurvvMbNrgOeBLOBhd19uZpMi6x8E7gAeNbO3CF00N7v7+jqsW0REYiR0gQt3nwPMiVn2YNT9dcA3k1uaiIjUhM4UFRFJEwp0EZE0oUAXEUkTCnQRkTShQBcRSRMKdBGRNKFAFxFJEwp0EZE0oUAXEUkTCnQRkTShQBcRSRMKdBGRNKFAFxFJEwp0EZE0oUAXEUkTCnQRkTShQBcRSRMKdBGRNKFAFxFJEwkFupkNN7P3zGylmU2Os/4mM1saub1tZnvNrH3yyxURkcpUG+hmlgXcB4wAegJjzaxndBt3/5W7F7h7AfBj4CV331gXBYuISHyJ7KEPBFa6+yp33wXMAEZV0X4s8EQyihMRkcQlEuidgI+jHpdGlh3AzLKB4cDMStZPNLMSMyspKyuraa0iIlKFRALd4izzStp+G3i1su4Wd5/m7oXuXpiTk5NojSIikoAmCbQpBY6JepwLrKuk7RjquLtl4UK45Rb4+tehW7fw8+tfh699DQ49tC5fWUSkYUsk0BcB3cysC/AJIbQvim1kZocCpwAXJ7XCGHv2wCGHwAsvwKOPVlyXk7M/4GNv7TXmRkTSXLWB7u57zOwa4HkgC3jY3Zeb2aTI+gcjTc8DXnD3bXVWLXDaaeEGsG0brFoFK1dWvC1YAI8/XvF5hx1Wedjn5IDF61iqRHFx+Jbw0UfQuTNMnQpFRUl7iyIitWLulXWH163CwkIvKSmps+3v2AGrVx8Y9itXwtq18NVX+9u2bVt52B95ZMWwLy6GiRNh+/b9y7KzYdo0hbqI1D0ze9PdC+OuS9dAr8quXbBmTfywX706dOuUy86uGPDTp8PGOId88/LCNkVE6lJVgZ5IH3raadYMuncPt1h79oSulA8+qBj077wDzzwT/hjE89FHdVuziEh1MjLQq9KkCXTtGm5nnVVx3d69YU/8k08OfF6zZuEg7X/9V9irFxGpb5qcqwaysuAXvzgwsJs2DQddL7sMjjoKrroKli5NTY0ikrkU6DVUVBQOgOblhYOleXnwyCOwbh289BKMHAkPPwz9+sGAAaHt1q2prlpEMkFGHhStaxs3wh/+AL/7Hbz9NrRqBWPGhNExAwbUbIikiEi0qg6Kag+9DrRvD9ddB8uWweuvw4UXwhNPwAknQEEB3HsvbNqU6ipFJN0o0OuQGQwaBA89BJ9+Cg88EA66Xntt6Gu/9FJ45RVI0ZckEUkzCvR60rYtTJoEb74ZbuPHw1/+AiefDL16wa9/DevXp7pKEWnMFOgp0L9/2Ftfty7svR96KPzgB9CpE4wdC/PnVzyTtTLFxZCfH+a2yc8Pj0UkcynQU6h1a5gwIfSzL1sW9uCfew5OPz2c9HTnnfDZZ/GfWz4Fwdq1octm7drwWKEukrk0yqWB2bEDZs4MI2QWLgx97iNHwhVXwJlnhrHwEPbI16498PmagkAkvWmUSyPSsiVcfHEY075iBVx/fQj2ESPC2au33w6lpZVPNaApCEQylwK9ATv2WPjVr0KA/+lPoRvmttvCXniLFvGf07lz/dYoIg2HAr0RaN48zBHz97/Dhx/C5Mlh7phY2dlhbnYRyUwK9Eama9cQ2mVloTumfE89OzucsKQ52UUylwK9kWraFP73f8OFNu68MxxMvf/++DNBikhmUKA3cmZw880wa1Y4iDpgACxalOqqRCQVEgp0MxtuZu+Z2Uozm1xJm2FmttTMlpvZS8ktU6ozciS89lrobx86FGbMSHVFIlLfqg10M8sC7gNGAD2BsWbWM6ZNO+B+YKS79wIuqINapRq9e8Mbb0BhYTjj9NZbEzvjVETSQyJ76AOBle6+yt13ATOAUTFtLgL+7O4fAbj758ktUxKVkwPz5oUzUH/+c7jgAti2LdVViUh9SCTQOwEfRz0ujSyL1h04zMwWmNmbZjYuWQVKzTVrFi5m/etfh771IUN0wpFIJkgk0ONdjiF2voAmwDeAs4GzgFvN7IBLMJvZRDMrMbOSsrKyGhcriTODG24IF7ZetQoGDgxzxohI+kok0EuBY6Ie5wLr4rR5zt23uft6YCHQN3ZD7j7N3QvdvTAnJ6e2NUsNjBgB//hHmAhs2DB4/PFUVyQidSWRQF8EdDOzLmbWDBgDzI5p81fgZDNrYmbZwAnAu8ktVWrruOPgn/+EwYNh3LhwpunevamuSkSSrdpAd/c9wDXA84SQftLdl5vZJDObFGnzLvAcsAx4A5ju7m/XXdlSUx06wPPPhyl6f/ELOO88XbxaJN1o+twMdP/94Zqnxx0Hs2dDly6prkhEEqXpc6WCq64KF9IoLQ0HSxcuTHVFIpIMCvQMdcYZoV+9Q4dw/6GHUl2RiBwsBXoG6949jIA59VS4/PIwzHHPnlRXJSK1pUDPcO3awbPPhj71u++Gb38bNm+u2TZ0sWqRhkGBLjRpAr/5Dfz2tzB3LgwaBB98kNhzdbFqkYZDgS77TJwYAr2sDE44AebPr/45t9wS5mSPtn17WC4i9UuBLhWcckqYsfGoo+Cb34QHHqi6vS5WLdJwKNDlAF27hnlfhg8PQxyvvhp2747ftrKLUuti1SL1T4EucbVtC3/9K9x0UzgRacQI2LjxwHZTp4brmUbTxapFUkOBLpXKyoJf/hIeeQRefjn0q69YUbFNURFMmwZ5eWGGx7y88FgXqxapfzr1XxLy6qswejR8+WW4vN3w4amuSCQz6dR/OWiDB4eDpfn5cPbZYZhjivYFRKQSCnRJWF4evPJKuCD19deHYY67dqW6KhEpp0CXGmndGmbODOPMp0+HM8+E9etTXZWIgAJdauGQQ8IFqIuLwwRfAwbA25r9XiTlFOhSaxddFKbe3bkTTjwxXL9URFJHgS4HZeBAWLQozNw4alSY4EsHS0VSQ4EuBy03N+ypjxoVpuC9+mpNwyuSCgp0SYpWreCpp+Dmm8P8L2efXfNpeEXk4CQU6GY23MzeM7OVZjY5zvphZrbZzJZGbj9NfqnS0B1yCNx5Z7j60fz5cNJJsHp1qqsSyRzVBrqZZQH3ASOAnsBYM+sZp+nL7l4Qud2e5DqlEZkwAV54AT79NEwX8Nprqa5IJDMksoc+EFjp7qvcfRcwAxhVt2VJY3fqqWHGxrZt4bTT4IknUl2RSPpLJNA7AR9HPS6NLIt1opn9y8z+Zma94m3IzCaaWYmZlZSVldWiXGlMjj02jFM/4YQwxPFnP9MIGJG6lEigW5xlsf8tFwN57t4X+L/ArHgbcvdp7l7o7oU5OTk1q1QapQ4dQvfLpZfClClw8cVh3LqIJF8igV4KHBP1OBdYF93A3be4+xeR+3OApmbWMWlVSqPWvHmYgvd//gf++Ec4/fRwmTsRSa5EAn0R0M3MuphZM2AMMDu6gZkdaWYWuT8wst0NyS5WGi8z+PGP4cknYfHi0A3zzjuprkokvVQb6O6+B7gGeB54F3jS3Zeb2SQzmxRpdj7wtpn9C7gHGOOpmmhdGrQLLoCXXgoXkj7xRPj731NdkUj60AUuJCU++gjOOSfspd97L0yaVP1zREQXuJAGqHPnMLf6WWfBlVeGKQP27k11VSKNmwJdUqb8QtTXXRcm9Tr3XPjii1RXJdJ4KdAlpZo0CZezu/demDMHhgyBjz+u/nkiciAFujQIV18Nzz4Lq1aFETA6vCJScwp0aTCGDw/zvjRrBkOHwp//nOqKRBoXBbo0KMcfH6YL6NsXvvMd+MUvNF2ASKIU6NLgHHFEmH73wgth8mS4/HLYtSvVVYk0fE1SXYBIPC1bhmkCuneHO+4IfeszZ0L79qmuTKTh0h66NFiHHAK33w6PPx761k88ET74INVViTRcCnRp8C6+GObNgw0bYNCgMHWAiBxIgS6NwpAh4WDp4YfDmWfCo48e2Ka4GPLzw559fn54LNIQbNwIs2fDD38IAweGcy/qgvrQpdH42tdC18sFF8Bll8H778PPfx4CvLgYJk4Mk34BrF0bHgMUFaWuZslMn30GCxfuv731VljevHk4z6KuLgehybmk0dm9G665BqZNg/PPh8ceg549Q4jHysuDNWvqvUTJMGvXVgzw998Py1u1gsGDw3kVQ4fCgAHQosXBvVZVk3NpD10anaZN4cEHwyXufvjD8J8pXphDmNVRJJncQ2BHB3j5v7N27eDkk+GKK0KA9+sX/r3WFwW6NEpmcOONoRvmoosgKyv+bI2dO9d/bZJevvoqdJlEB/jnn4d1RxwRgvumm8LP448PXYCpokCXRm3UqDAN7+mnw3/+U3FddjZMnZqauqTx2r0blizZH94vvwybNoV1xxwD3/xmCO9TToFu3cLORUOhQJdGr1+/sAc1ePD+rpfsbDjttDAd7wsvQJcuoT+9WbPU1ioNz86d8MYb+wP8tddg27awrnv3cJymvA88Ly+1tVZHgS5poVMnWL4cfvazsHe1ejU89xw888z+NmahXZcu8W9HHx26biS9bd0Kr7++P8D/+c/9U0v06RNGUA0dGvrCjzwytbXWlAJd0karVvDLX+5/vHcvfPJJCPfY27x5sG5dxYm/mjYNe2CVBX7Hjg3r63Wm++qrsCe9dWvF25YtBy4rv731VrhI+d694Y93//5w7bUhwIcMafxTSyQU6GY2HPgNkAVMd/c7K2k3APgHcKG7P5W0KkVqISsrHBTt3Dn0d8b68svQRRMv8BcvDmemRmvdOpywVFngt2lTL+0XsOgAAAkrSURBVG+rUfvyy8rDtrowjm2zbVtiM3Gahd9dmzbQtWuY8G3o0DCVRLr9zqoNdDPLAu4DzgRKgUVmNtvd34nT7hfA83VRqEiyNW8e+ki7d4+/fsuWMIY9XuDPn7+/n7Vchw77wz0/P5w80r59WN6+fcXbwY5Fbgh27oSyssRuGzeGEE501szmzUPYtm0bfrZpEz7Prl33P46+RbeLvWVnp3bkSX1KZA99ILDS3VcBmNkMYBTwTky7a4GZwICkViiSIm3bhj7VPn0OXOcO69fHD/slS2DWrDBaojItW8YP+uqWtWxZd+9327aKIfz551WHdGXXf23SJHRP5eSEW//+4T3UJITrc+x2Okkk0DsB0Vd5LAVOiG5gZp2A84DTqCLQzWwiMBGgswYISyNmtj+wBg48cL17CLyNG8Ntw4b99+MtW7Fi/7Kq/hC0aJFY8Ef/AdiwIbG96B074r9ms2bhfR5+ePjZrdv+9x7v1q6djjWkSiKBHu9XE9tzdTdws7vvtSp+k+4+DZgG4dT/RIsUaWzM9u9t1mSom3vYU070D8H77+9flmh3Rnb2/vA9/HDo1avqgG7TRgHdWCQS6KXAMVGPc4F1MW0KgRmRMO8IfMvM9rj7rKRUKZIhyg/gtW5ds7Nc3cPEZLHBv2NH2HMv37vOyQmBLukpkUBfBHQzsy7AJ8AY4KLoBu7epfy+mT0KPKMwF6k/ZmHYZqtW4WxGyUzVBrq77zGzawijV7KAh919uZlNiqx/sI5rFBGRBCQ0Dt3d5wBzYpbFDXJ3H3/wZYmISE1lyOhMEZH0p0AXEUkTCnQRkTShQBcRSRMKdJEkKy4Oc7kcckj4WVyc6ookU2j6XJEkKi6GiRPDST4QZnOcODHcLypKXV2SGbSHLpJEt9yyP8zLbd8elovUNQW6SBKVX/090eUiyaRAF0miyuZf0eSiUh8U6CJJNHXqgZNfZWeH5SJ1TYEukkRFRTBtWpgy1yz8nDZNB0SlfmiUi0iSFRUpwCU1tIcuIpImFOgiImlCgS4ikiYU6CJpSNMPZCYdFBVJM5p+IHNpD10kzWj6gcyVUKCb2XAze8/MVprZ5DjrR5nZMjNbamYlZjYk+aWKSCI0/UDmqjbQzSwLuA8YAfQExppZz5hm84C+7l4ATACmJ7tQEUmMph/IXInsoQ8EVrr7KnffBcwARkU3cPcv3N0jD1sBjoikhKYfyFyJBHon4OOox6WRZRWY2XlmtgJ4lrCXfgAzmxjpkikpKyurTb0iUg1NP5C5Egl0i7PsgD1wd/+Lu/cAzgXuiLchd5/m7oXuXpiTk1OzSkUkYUVFsGYNfPVV+KkwzwyJBHopcEzU41xgXWWN3X0h8DUz63iQtYmISA0kEuiLgG5m1sXMmgFjgNnRDczs62Zmkfv9gWbAhmQXKyKNi05wql/Vnljk7nvM7BrgeSALeNjdl5vZpMj6B4HvAOPMbDewA7gw6iCpiGQgneBU/yxVuVtYWOglJSUpeW0RqXv5+SHEY+XlhX59qR0ze9PdC+Ot05miIlIndIJT/VOgi0id0AlO9U+BLiJ1Qic41T8FuojUCZ3gVP8U6CJSZxrKCU6ZMnxS86GLSFrLpOGT2kMXkbSWSfPDK9BFJK1l0vBJBbqIpLVMGj6pQBeRtJZJwycV6CKS1hrS8Mm6Hm2jUS4ikvaKilI/oqU+RttoD11EpB7Ux2gbBbqISD2oj9E2CnQRkXpQH6NtFOgiIvWgPkbbKNBFROpBfYy20SgXEZF6UtejbbSHLiKSJhIKdDMbbmbvmdlKM5scZ32RmS2L3F4zs77JL1VERKpSbaCbWRZwHzAC6AmMNbOeMc1WA6e4ex/gDmBasgsVEZGqJbKHPhBY6e6r3H0XMAMYFd3A3V9z9/9EHv4DyE1umSIiUp1EAr0T8HHU49LIssp8F/hbvBVmNtHMSsyspKysLPEqRUSkWomMcrE4yzxuQ7NTCYE+JN56d59GpDvGzMrMbG2CdTZUHYH1qS6iAdHnUZE+j/30WVR0MJ9HXmUrEgn0UuCYqMe5wLrYRmbWB5gOjHD3DdVt1N1zEnjtBs3MSty9MNV1NBT6PCrS57GfPouK6urzSKTLZRHQzcy6mFkzYAwwO6a4zsCfgUvc/f1kFykiItWrdg/d3feY2TXA80AW8LC7LzezSZH1DwI/BToA95sZwB79NRYRqV8JnSnq7nOAOTHLHoy6fzlweXJLaxQ0PLMifR4V6fPYT59FRXXyeZh73OObIiLSyOjUfxGRNKFAFxFJEwr0WjCzY8zsRTN718yWm9n3U11TqplZlpktMbNnUl1LqplZOzN7ysxWRP6NnJjqmlLJzG6I/D9528yeMLMWqa6pPpnZw2b2uZm9HbWsvZn93cw+iPw8LBmvpUCvnT3AD9z9OGAQcHWc+W0yzfeBd1NdRAPxG+A5d+8B9CWDPxcz6wRcBxS6+/GEkXJjUltVvXsUGB6zbDIwz927AfMijw+aAr0W3P1Td18cub+V8B+2qukQ0pqZ5QJnE04sy2hm1hYYCjwE4O673H1TaqtKuSZASzNrAmQT58TEdObuC4GNMYtHAY9F7j8GnJuM11KgHyQzywf6Af9MbSUpdTfwI+CrVBfSAHQFyoBHIl1Q082sVaqLShV3/wS4C/gI+BTY7O4vpLaqBuEId/8Uwg4icHgyNqpAPwhm1hqYCVzv7ltSXU8qmNk5wOfu/maqa2kgmgD9gQfcvR+wjSR9nW6MIn3Do4AuwNFAKzO7OLVVpS8Fei2ZWVNCmBe7+59TXU8KDQZGmtkawtTKp5nZH1JbUkqVAqXuXv6N7SlCwGeqM4DV7l7m7rsJU4SclOKaGoJ/m9lRAJGfnydjowr0WrAwv8FDwLvu/utU15NK7v5jd89193zCwa757p6xe2Du/hnwsZkdG1l0OvBOCktKtY+AQWaWHfl/czoZfJA4ymzg0sj9S4G/JmOjukh07QwGLgHeMrOlkWU/iUyRIHItUByZzG4VcFmK60kZd/+nmT0FLCaMDltChk0DYGZPAMOAjmZWCtwG3Ak8aWbfJfzRuyApr6VT/0VE0oO6XERE0oQCXUQkTSjQRUTShAJdRCRNKNBFRNKEAl1EJE0o0EVE0sT/B3EPovf2gmL2AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(acc) + 1)\n",
    "plt.plot(epochs, acc, 'bo', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.legend()\n",
    "plt.figure()\n",
    "plt.plot(epochs, loss, 'bo', label='Training loss')\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "出了模型的训练结果和验证结果。验证精度略低于 LSTM，但在 CPU和GPU上的运行速度都要更快（速度提高多少取决于具体配置，会有很大差异）。现在，你可以使用正确的轮数（4轮）重新训练这个模型，然后在测试集上运行。这个结果可以让我们确信，在单词级的情感分类任务上，一维卷积神经网络可以替代循环网络，并且速度更快、计算代价更低"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.4　结合 CNN和 RNN来处理长序列\n",
    "一维卷积神经网络分别处理每个输入序列段，所以它对时间步的顺序不敏感（这里所说顺序的范围要大于局部尺度，即大于卷积窗口的大小），这一点与 RNN不同。当然，为了识别更长期的模式，你可以将许多卷积层和池化层堆叠在一起，这样上面的层能够观察到原始输入中更长的序列段，但这仍然不是一种引入顺序敏感性的好方法。想要证明这种方法的不足，一种方法是在温度预测问题上使用一维卷积神经网络，在这个问题中顺序敏感性对良好的预测结果非常关键。以下示例复用了前面定义的这些变量：   float_data、train_gen、val_gen和val_steps。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\"Date Time\"', '\"p (mbar)\"', '\"T (degC)\"', '\"Tpot (K)\"', '\"Tdew (degC)\"', '\"rh (%)\"', '\"VPmax (mbar)\"', '\"VPact (mbar)\"', '\"VPdef (mbar)\"', '\"sh (g/kg)\"', '\"H2OC (mmol/mol)\"', '\"rho (g/m**3)\"', '\"wv (m/s)\"', '\"max. wv (m/s)\"', '\"wd (deg)\"']\n",
      "420451\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "data_dir = '/Users/yihuawang/Deeplearning/'\n",
    "fname = os.path.join(data_dir, 'jena_climate_2009_2016.csv')\n",
    "f = open(fname) # 读取文件\n",
    "data = f.read()\n",
    "f.close()\n",
    "\n",
    "lines = data.split('\\n')\n",
    "header = lines[0].split(',')\n",
    "lines = lines[1:]\n",
    "print(header)\n",
    "print(len(lines))\n",
    "# 共有 420551行数据（每行是一个时间步，记录了一个日期和14个与天气有关的值），还输出了下列表头。\n",
    "float_data = np.zeros((len(lines), len(header) - 1))\n",
    "for i, line in enumerate(lines):\n",
    "    values = [float(x) for x in line.split(',')[1:]]\n",
    "    float_data[i, :] = values\n",
    "\n",
    "def generator(data, lookback, delay, min_index, max_index,\n",
    "              shuffle=False, batch_size=128, step=6):\n",
    "    \n",
    "    if max_index is None:\n",
    "        max_index = len(data) - delay - 1\n",
    "    i = min_index + lookback\n",
    "    while 1:\n",
    "        if shuffle:\n",
    "            rows = np.random.randint(\n",
    "                min_index + lookback, max_index, size=batch_size)\n",
    "    else:\n",
    "        if i + batch_size >= max_index:\n",
    "            i = min_index + lookback\n",
    "        rows = np.arange(i, min(i + batch_size, max_index))\n",
    "        i += len(rows)\n",
    "        \n",
    "    samples = np.zeros((len(rows),\n",
    "                    lookback // step,\n",
    "                    data.shape[-1]))\n",
    "    targets = np.zeros((len(rows),))\n",
    "    for j, row in enumerate(rows):\n",
    "        indices = range(rows[j] - lookback, rows[j], step)\n",
    "        samples[j] = data[indices]\n",
    "        targets[j] = data[rows[j] + delay][1]\n",
    "    yield samples, targets\n",
    "\n",
    "# 准备训练生成器、验证生成器和测试生成器\n",
    "lookback = 1440\n",
    "step = 6\n",
    "delay = 144\n",
    "batch_size = 128\n",
    "\n",
    "train_gen = generator(float_data,\n",
    "                      lookback=lookback,\n",
    "                      delay=delay,\n",
    "                      min_index=0,\n",
    "                      max_index=200000, # 前200,000个\n",
    "                      shuffle=True,\n",
    "                      step=step, \n",
    "                      batch_size=batch_size)\n",
    "val_gen = generator(float_data,\n",
    "                    lookback=lookback,\n",
    "                    delay=delay,\n",
    "                    min_index=200001, #前200,001个\n",
    "                    max_index=300000,\n",
    "                    step=step,\n",
    "                    batch_size=batch_size)\n",
    "test_gen = generator(float_data,\n",
    "                     lookback=lookback,\n",
    "                     delay=delay,\n",
    "                     min_index=300001,\n",
    "                     max_index=None,\n",
    "                     step=step,\n",
    "                     batch_size=batch_size)\n",
    "\n",
    "# This is how many steps to draw from `val_gen`\n",
    "# in order to see the whole validation set:\n",
    "val_steps = (300000 - 200001 - lookback) // batch_size # 为了查看在验证集中，需要从val_gen中抽取多少次\n",
    "\n",
    "# This is how many steps to draw from `test_gen`\n",
    "# in order to see the whole test set:\n",
    "test_steps = (len(float_data) - 300001 - lookback) // batch_size # 为了查看整个测试集中，需要从test_gen中抽取多少次"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Conv1D(32, 5, activation='relu',\n",
    "     input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.GlobalMaxPooling1D())\n",
    "model.add(layers.Dense(1))\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "\n",
    "history = model.fit_generator(train_gen,\n",
    "    steps_per_epoch=500,\n",
    "    epochs=20,\n",
    "    validation_data=val_gen,\n",
    "    validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "验证 MAE停留在 0.4~0.5，使用小型卷积神经网络甚至无法击败基于常识的基准方法。同样，这是因为卷积神经网络在输入时间序列的所有位置寻找模式，它并不知道所看到某个模式的时间位置（距开始多长时间，距结束多长时间等）。对于这个具体的预测问题，对最新数据点的解释与对较早数据点的解释应该并不相同，所以卷积神经网络无法得到有意义的结果。卷积神经网络的这种限制对于 IMDB数据来说并不是问题，因为对于与正面情绪或负面情绪相关联的关键词模式，无论出现在输入句子中的什么位置，它所包含的信息量是一样的"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要想结合卷积神经网络的速度和轻量与  RNN的顺序敏感性，一种方法是在   RNN前面使用一维卷积神经网络作为预处理步骤（见图 6-30）。对于那些非常长，以至于 RNN无法处理的序列（比如包含上千个时间步的序列），这种方法尤其有用。卷积神经网络可以将长的输入序列转换为高级特征组成的更短序列（下采样）。然后，提取的特征组成的这些序列成为网络中 RNN的输入。   \n",
    "\n",
    "种方法在研究论文和实际应用中并不多见，可能是因为很多人并不知道。这种方法非常有效，应该被更多人使用。我们尝试将其应用于温度预测数据集。因为这种方法允许操作更长的序列，所以我们可以查看更早的数据（通过增大数据生成器的 lookback参数）或查看分辨率更高的时间序列（通过减小生成器的 step参数）。这里我们任意地将step减半，得到时间序列的长度变为之前的两倍，温度数据的采样频率变为每 30分钟一个数据点。本示例复用了之前定义的generator 函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This was previously set to 6 (one point per hour).\n",
    "# Now 3 (one point per 30 min).\n",
    "step = 3\n",
    "lookback = 720  # Unchanged\n",
    "delay = 144 # Unchanged\n",
    "\n",
    "train_gen = generator(float_data,\n",
    "                      lookback=lookback,\n",
    "                      delay=delay,\n",
    "                      min_index=0,\n",
    "                      max_index=200000,\n",
    "                      shuffle=True,\n",
    "                      step=step)\n",
    "val_gen = generator(float_data,\n",
    "                    lookback=lookback,\n",
    "                    delay=delay,\n",
    "                    min_index=200001,\n",
    "                    max_index=300000,\n",
    "                    step=step)\n",
    "test_gen = generator(float_data,\n",
    "                     lookback=lookback,\n",
    "                     delay=delay,\n",
    "                     min_index=300001,\n",
    "                     max_index=None,\n",
    "                     step=step)\n",
    "val_steps = (300000 - 200001 - lookback) // 128\n",
    "test_steps = (len(float_data) - 300001 - lookback) // 128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv1d_8 (Conv1D)            (None, None, 32)          2272      \n",
      "_________________________________________________________________\n",
      "max_pooling1d_5 (MaxPooling1 (None, None, 32)          0         \n",
      "_________________________________________________________________\n",
      "conv1d_9 (Conv1D)            (None, None, 32)          5152      \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, 32)                6336      \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 13,793\n",
      "Trainable params: 13,793\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(layers.Conv1D(32, 5, activation='relu',\n",
    "                        input_shape=(None, float_data.shape[-1])))\n",
    "model.add(layers.MaxPooling1D(3))\n",
    "model.add(layers.Conv1D(32, 5, activation='relu'))\n",
    "model.add(layers.GRU(32, dropout=0.1, recurrent_dropout=0.5))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "model.summary()\n",
    "\n",
    "model.compile(optimizer=RMSprop(), loss='mae')\n",
    "history = model.fit_generator(train_gen,\n",
    "                              steps_per_epoch=500,\n",
    "                              epochs=20,\n",
    "                              validation_data=val_gen,\n",
    "                              validation_steps=val_steps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从验证损失来看，这种架构的效果不如只用正则化 GRU，但速度要快很多。它查看了两倍的数据量，在本例中可能不是非常有用，但对于其他数据集可能非常重要。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4.5　小结\n",
    "下面是你应该从本节中学到的要点。\n",
    "\n",
    "1. 二维卷积神经网络在二维空间中处理视觉模式时表现很好，与此相同，一维卷积神经网络在处理时间模式时表现也很好。对于某些问题，特别是自然语言处理任务，它可以替代 RNN，并且速度更快。   \n",
    "\n",
    "2. 通常情况下，一维卷积神经网络的架构与计算机视觉领域的二维卷积神经网络很相似，它将Conv1D层和MaxPooling1D层堆叠在一起，最后是一个全局池化运算或展平操作。  \n",
    "\n",
    "3. 因为 RNN在处理非常长的序列时计算代价很大，但一维卷积神经网络的计算代价很小，所以在 RNN之前使用一维卷积神经网络作为预处理步骤是一个好主意，这样可以使序列变短，并提取出有用的表示交给 RNN来处理。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 本章总结\n",
    "\n",
    "你在本章学到了以下技术，它们广泛应用于序列数据（从文本到时间序列）组成的数据集。   \n",
    "如何对文本分词。   \n",
    "什么是词嵌入，如何使用词嵌入。    \n",
    "\n",
    "什么是循环网络，如何使用循环网络。    \n",
    "\n",
    "如何堆叠    RNN层和使用双向  RNN，以构建更加强大的序列处理模型。    \n",
    "\n",
    "如何使用一维卷积神经网络来处理序列。    \n",
    "\n",
    "如何结合一维卷积神经网络和    RNN来处理长序列。    \n",
    "\n",
    "你可以用  RNN进行时间序列回归（“预测未来”）、时间序列分类、时间序列异常检测和序列标记（比如找出句子中的人名或日期）。    \n",
    "\n",
    "同样，你可以将一维卷积神经网络用于机器翻译（序列到序列的卷积模型，比如SliceNet）、文档分类和拼写校正。    \n",
    "\n",
    "如果序列数据的整体顺序很重要，那么最好使用循环网络来处理。时间序列通常都是这样，最近的数据可能比久远的数据包含更多的信息量。  \n",
    "\n",
    "如果整体顺序没有意义，那么一维卷积神经网络可以实现同样好的效果，而且计算代价更小。文本数据通常都是这样，在句首发现关键词和在句尾发现关键词一样都很有意义。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}