lm2-french-classifier-amazon.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ULMFiT on French Amazon Customer Reviews\n",
    "### (architecture QRNN, SentenPiece tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: September 2019\n",
    "- Post in medium: [link](https://medium.com/@pierre_guillou/nlp-fastai-french-language-model-d0e2a9e12cab)\n",
    "- Ref: [Fastai v1](https://docs.fast.ai/) (Deep Learning library on PyTorch)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Information**\n",
    "\n",
    "According to this new article \"[MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://arxiv.org/abs/1909.04761)\" (September 10, 2019), the QRNN architecture and the SentencePiece tokenizer give better results than AWD-LSTM and the spaCy tokenizer respectively. \n",
    "\n",
    "Therefore, they have been used in this notebook to **fine-tune a French bidirectional Language Model** by Transfer Learning of a French bidirectional Language Model (with the QRNN architecture and the SentencePiece tokenizer, too) trained on a Wikipedia corpus of 100 millions tokens ([lm2-french.ipynb](https://github.com/piegu/language-models/blob/master/lm2-french.ipynb)). \n",
    "\n",
    "This French bidirectional Language Model has been **fine-tuned on \"[French Amazon Customer Reviews](https://s3.amazonaws.com/amazon-reviews-pds/readme.html)\"** and **its encoder part has been transfered to a sentiment classifier which has been finally trained on this amazon corpus**.\n",
    "\n",
    "This process **LM General --> LM fine-tuned --> Classifier fine-tuned** is called [ULMFiT](http://nlp.fast.ai/category/classification.html).\n",
    "\n",
    "More, the following hyperparameters values given at the end of the MultiFiT article have been used:\n",
    "- Language Model\n",
    "    - (batch size) bs = 50\n",
    "    - (QRNN) 3 QRNN (default: 3) with 1152 hidden parameters each one (default: 1152) (note: it would have been better to increae to 4 QRNN with 1550 hidden parameters like described in the article)\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 0\n",
    "    - (weight decay) wd = 0.01\n",
    "    - (number of training epochs) 10 epochs\n",
    "    \n",
    "\n",
    "- Sentiment Classifier\n",
    "    - (batch size) bs = 18\n",
    "    - (SentencePiece) vocab of 15000 tokens\n",
    "    - (dropout) mult_drop = 0.5\n",
    "    - (weight decay) wd = 0.01\n",
    "    - (number of training epochs) 20 epochs\n",
    "    - (loss) FlattenedLoss of weighted CrossEntropyLoss (the FlattenedLoss of LabelSmoothing CrossEntropy has been tested but was not kept because of a lower accuracy that could be a consequence of the fact that the dataset is unbalanced)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our Bidirectional French LM ([lm-french.ipynb](https://github.com/piegu/language-models/blob/master/lm-french.ipynb)) and Sentiment Classifier with a AWD-LSTM architecture and using the spaCy tokenizer ([lm-french-classifier-amazon.ipynb](https://github.com/piegu/language-models/blob/master/lm-french-classifier-amazon.ipynb)) have better results (accuracy, perplexity and f1) than the Bidirectional French LM ([lm2-french.ipynb](https://github.com/piegu/language-models/blob/master/lm2-french.ipynb)) and Sentiment Classifier with a QRNN architecture and using the SentencePiecce tokenizer ([lm2-french-classifier-amazon.ipynb](https://github.com/piegu/language-models/blob/master/lm2-french-classifier-amazon.ipynb)). \n",
    "\n",
    "But because of the \"To be improved\" paragraph, we should retrain all in order to get a final comparaison."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### French Bidirectional LM (QRNN, SentencePiece)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **About the data**: the dataset \"French Amazon Customer Reviews\" is unbalanced. Therefore, we used a weighted loss function (FlattenedLoss of weighted CrossEntropyLoss).\n",
    "    - neg: 25637 (11.1%)\n",
    "    - pos: 205047 (88.9%)\n",
    "\n",
    "\n",
    "- **Accuracy and Perplexity** of the fine-tuned Language Model:    \n",
    "    - forward : (accuracy) 35.79% | (perplexity) 28.97\n",
    "    - backward: (global) 35.22% | (perplexity) 30.27\n",
    "    \n",
    "\n",
    "- **Accuracy** of the sentiment classifier:\n",
    "    - forward : (global) 93.51% | **(neg) 93.69%** | (pos) 93.48%\n",
    "    - backward: (global) 93.18% | (neg) 90.34% | (pos) 93.54%\n",
    "    - ensemble: **(global) 93.70%** | (neg) 92.36% | **(pos) 93.86%**\n",
    "\n",
    "\n",
    "- **f1 score** of the sentiment classifier:\n",
    "    - forward: 0.9624\n",
    "    - backward: 0.9606\n",
    "    - ensemble: **0.9636**\n",
    "    \n",
    "\n",
    "(neg = negative reviews | pos = positive reviews)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To be improved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Out of the 230 684 reviews of our file amazon_reviews_fr.csv, we found (but after the training of our models) that **11 098 reviews are not in French: almost 5%** (4.8%)!\n",
    "\n",
    "We should delete these 11 098 review and re-fine-tune our LM and after our sentiment classifier on the only-French reviews dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "\n",
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "@np_func\n",
    "def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# bs=48\n",
    "# bs=24\n",
    "bs=50"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "torch.cuda.set_device(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_path = Config.data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lang = 'fr'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "\n",
    "lm_fns2 = [f'{lang}_wt_sp15', f'{lang}_wt_vocab_sp15']\n",
    "lm_fns2_bwd = [f'{lang}_wt_sp15_bwd', f'{lang}_wt_vocab_sp15_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- [French Amazon Customer Reviews](https://s3.amazonaws.com/amazon-reviews-pds/readme.html)\n",
    "- [Guide on how to download the French Amazon Customer Reviews](https://forums.fast.ai/t/ulmfit-french/29379/36)\n",
    "- File: amazon_reviews_multilingual_FR_v1_00.tsv.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/jupyter/.fastai/data/amazon_reviews_fr/amazon_reviews_multilingual_FR_v1_00.tsv'),\n",
       " PosixPath('/home/jupyter/.fastai/data/amazon_reviews_fr/amazon_reviews_fr.csv')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "name = 'amazon_reviews_fr'\n",
    "path_data = data_path/name\n",
    "path_data.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Run this code the first time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# to solve display error of pandas dataframe\n",
    "get_ipython().config.get('IPKernelApp', {})['parent_appname'] = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_body</th>\n",
       "      <th>star_rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>R32VYUWDIB5LKE</td>\n",
       "      <td>je conseille fortement ce bouquin à ceux qui s...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>R3CCMP4EV6HAVL</td>\n",
       "      <td>ce magnifique est livre , les personnages sont...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>R14NAE6UGTVTA2</td>\n",
       "      <td>Je dirais qu'il a un défaut :&lt;br /&gt;On ne peut ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>R2E7QEWSC6EWFA</td>\n",
       "      <td>Je l'ai depuis quelques jours et j'en suis trè...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>R26E6I47GQRYKR</td>\n",
       "      <td>je m'attendait à un bon film, car j'aime beauc...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        review_id                                        review_body  \\\n",
       "0  R32VYUWDIB5LKE  je conseille fortement ce bouquin à ceux qui s...   \n",
       "1  R3CCMP4EV6HAVL  ce magnifique est livre , les personnages sont...   \n",
       "2  R14NAE6UGTVTA2  Je dirais qu'il a un défaut :<br />On ne peut ...   \n",
       "3  R2E7QEWSC6EWFA  Je l'ai depuis quelques jours et j'en suis trè...   \n",
       "4  R26E6I47GQRYKR  je m'attendait à un bon film, car j'aime beauc...   \n",
       "\n",
       "   star_rating  \n",
       "0            5  \n",
       "1            5  \n",
       "2            3  \n",
       "3            4  \n",
       "4            2  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fields = ['review_id', 'review_body', 'star_rating']\n",
    "df = pd.read_csv(path_data/'amazon_reviews_multilingual_FR_v1_00.tsv', delimiter='\\t',encoding='utf-8', usecols=fields)\n",
    "df = df[fields]\n",
    "df.loc[pd.isna(df.review_body),'review_body']='NA'\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of reviews: 253961\n",
      "number of identical reviews: 0\n",
      "number of reviews neg + pos (rating != 3): 230684\n"
     ]
    }
   ],
   "source": [
    "# number of reviews\n",
    "print(f'number of reviews: {len(df)}')\n",
    "\n",
    "# check that there is no twice the same review\n",
    "same = len(df) - len(df['review_id'].unique())\n",
    "print(f'number of identical reviews: {same}')\n",
    "\n",
    "# number of reviews neg or pos\n",
    "num_neg_pos = len(df[df['star_rating'] != 3])\n",
    "print(f'number of reviews neg + pos (rating != 3): {num_neg_pos}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[205047, 25637]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(df_trn_val['label'].value_counts().array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pos    205047\n",
      "neg     25637\n",
      "Name: label, dtype: int64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD4CAYAAADy46FuAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAV0ElEQVR4nO3dcbDdZX3n8fdnk+LoWgrIhWFI2FCbugW2jZLFdB0dV1YIdKfBrmyT6ZisZSbKwk5duzPG7h84KjO4u61TpoobS4awqyCKLhkbi1nW0e0OKBehQESaS6ByTQYiQaSDxQl+94/zXD2Ek+de7g33puT9mjlzfuf7e57n95yZaz78nt/v+EtVIUnSofyjhZ6AJOnIZlBIkroMCklSl0EhSeoyKCRJXYsXegKH24knnljLli1b6GlI0j8od9111w+qamzUvpddUCxbtozx8fGFnoYk/YOS5G8Ptc+lJ0lSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUtfL7pfZc7Fs018s9BR0BHvkqt9a6ClIC8IzCklS17RBkWRpkq8leSDJziR/0OonJNmRZFd7P77Vk+TqJBNJ7k3yhqGxNrT2u5JsGKqfneS+1ufqJOkdQ5I0f2ZyRnEA+MOq+jVgFXBZkjOATcBtVbUcuK19BrgAWN5eG4FrYPCPPnAF8EbgHOCKoX/4r2ltp/qtbvVDHUOSNE+mDYqq2ltV327bTwMPAKcCa4CtrdlW4KK2vQa4vgbuAI5LcgpwPrCjqvZX1ZPADmB123dsVd1eVQVcf9BYo44hSZonL+oaRZJlwOuBbwInV9VeGIQJcFJrdirw6FC3yVbr1SdH1Okc4+B5bUwynmR83759L+YrSZKmMeOgSPJq4GbgfVX1o17TEbWaRX3GqmpzVa2sqpVjYyOfuyFJmqUZBUWSX2AQEp+pqi+28mNt2Yj2/nirTwJLh7ovAfZMU18yot47hiRpnszkrqcA1wIPVNWfDO3aBkzdubQBuGWovr7d/bQKeKotG90KnJfk+HYR+zzg1rbv6SSr2rHWHzTWqGNIkubJTH5w9ybgXcB9Se5ptT8CrgJuSnIJ8D3g4rZvO3AhMAE8A7wboKr2J/kIcGdr9+Gq2t+2LwWuA14JfKW96BxDkjRPpg2KqvorRl9HADh3RPsCLjvEWFuALSPq48BZI+pPjDqGJGn++MtsSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6ZvIo1C1JHk9y/1Dtc0nuaa9Hpp58l2RZkh8P7fvUUJ+zk9yXZCLJ1e2xpyQ5IcmOJLva+/GtntZuIsm9Sd5w+L++JGk6MzmjuA5YPVyoqt+tqhVVtQK4Gfji0O6HpvZV1XuH6tcAG4Hl7TU15ibgtqpaDtzWPgNcMNR2Y+svSZpn0wZFVX0D2D9qXzsr+LfADb0xkpwCHFtVt7dHpV4PXNR2rwG2tu2tB9Wvr4E7gOPaOJKkeTTXaxRvBh6rql1DtdOT3J3k60ne3GqnApNDbSZbDeDkqtoL0N5PGurz6CH6PE+SjUnGk4zv27dvbt9IkvQ8cw2KdTz/bGIvcFpVvR54P/DZJMcCGdG3phl7xn2qanNVrayqlWNjYzOYtiRpphbPtmOSxcDvAGdP1arqWeDZtn1XkoeAX2VwNrBkqPsSYE/bfizJKVW1ty0tPd7qk8DSQ/SRJM2TuZxR/Cvgu1X1syWlJGNJFrXtX2ZwIXp3W1J6Osmqdl1jPXBL67YN2NC2NxxUX9/ufloFPDW1RCVJmj8zuT32BuB24HVJJpNc0nat5YUXsd8C3Jvkr4EvAO+tqqkL4ZcCfw5MAA8BX2n1q4C3J9kFvL19BtgO7G7tPw38+xf/9SRJczXt0lNVrTtE/d+NqN3M4HbZUe3HgbNG1J8Azh1RL+Cy6eYnSXpp+ctsSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6ZvKEuy1JHk9y/1DtQ0m+n+Se9rpwaN8Hk0wkeTDJ+UP11a02kWTTUP30JN9MsivJ55Ic0+qvaJ8n2v5lh+tLS5JmbiZnFNcBq0fUP15VK9prO0CSMxg8IvXM1ueTSRa152h/ArgAOANY19oCfKyNtRx4Eph61OolwJNV9SvAx1s7SdI8mzYoquobwP7p2jVrgBur6tmqepjB867Paa+JqtpdVT8BbgTWJAnwNgbP1wbYClw0NNbWtv0F4NzWXpI0j+ZyjeLyJPe2panjW+1U4NGhNpOtdqj6a4AfVtWBg+rPG6vtf6q1f4EkG5OMJxnft2/fHL6SJOlgsw2Ka4DXAiuAvcAft/qo/+KvWdR7Y72wWLW5qlZW1cqxsbHevCVJL9KsgqKqHquq56rqp8CnGSwtweCMYOlQ0yXAnk79B8BxSRYfVH/eWG3/LzHzJTBJ0mEyq6BIcsrQx3cAU3dEbQPWtjuWTgeWA98C7gSWtzucjmFwwXtbVRXwNeCdrf8G4JahsTa07XcC/6e1lyTNo8XTNUhyA/BW4MQkk8AVwFuTrGCwFPQI8B6AqtqZ5CbgO8AB4LKqeq6NczlwK7AI2FJVO9shPgDcmOSjwN3Ata1+LfA/kkwwOJNYO+dvK0l60aYNiqpaN6J87YjaVPsrgStH1LcD20fUd/Pzpavh+t8DF083P0nSS8tfZkuSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1DVtUCTZkuTxJPcP1f5rku8muTfJl5Ic1+rLkvw4yT3t9amhPmcnuS/JRJKrk6TVT0iyI8mu9n58q6e1m2jHecPh//qSpOnM5IziOmD1QbUdwFlV9evA3wAfHNr3UFWtaK/3DtWvATYyeI728qExNwG3VdVy4Lb2GeCCobYbW39J0jybNiiq6hsMnlk9XPtqVR1oH+8AlvTGSHIKcGxV3V5VBVwPXNR2rwG2tu2tB9Wvr4E7gOPaOJKkeXQ4rlH8PvCVoc+nJ7k7ydeTvLnVTgUmh9pMthrAyVW1F6C9nzTU59FD9JEkzZPFc+mc5D8DB4DPtNJe4LSqeiLJ2cD/SnImkBHda7rhZ9onyUYGy1OcdtppM5m6JGmGZn1GkWQD8K+B32vLSVTVs1X1RNu+C3gI+FUGZwPDy1NLgD1t+7GpJaX2/nirTwJLD9Hneapqc1WtrKqVY2Njs/1KkqQRZhUUSVYDHwB+u6qeGaqPJVnUtn+ZwYXo3W1J6ekkq9rdTuuBW1q3bcCGtr3hoPr6dvfTKuCpqSUqSdL8mXbpKckNwFuBE5NMAlcwuMvpFcCOdpfrHe0Op7cAH05yAHgOeG9VTV0Iv5TBHVSvZHBNY+q6xlXATUkuAb4HXNzq24ELgQngGeDdc/mikqTZmTYoqmrdiPK1h2h7M3DzIfaNA2eNqD8BnDuiXsBl081PkvTS8pfZkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqcugkCR1GRSSpK4ZBUWSLUkeT3L/UO2EJDuS7Grvx7d6klydZCLJvUneMNRnQ2u/K8mGofrZSe5rfa5uz9U+5DEkSfNnpmcU1wGrD6ptAm6rquXAbe0zwAXA8vbaCFwDg3/0GTxv+43AOcAVQ//wX9PaTvVbPc0xJEnzZEZBUVXfAPYfVF4DbG3bW4GLhurX18AdwHFJTgHOB3ZU1f6qehLYAaxu+46tqtvbc7KvP2isUceQJM2TuVyjOLmq9gK095Na/VTg0aF2k63Wq0+OqPeO8TxJNiYZTzK+b9++OXwlSdLBXoqL2RlRq1nUZ6yqNlfVyqpaOTY29mK6SpKmMZegeKwtG9HeH2/1SWDpULslwJ5p6ktG1HvHkCTNk7kExTZg6s6lDcAtQ/X17e6nVcBTbdnoVuC8JMe3i9jnAbe2fU8nWdXudlp/0FijjiFJmieLZ9IoyQ3AW4ETk0wyuHvpKuCmJJcA3wMubs23AxcCE8AzwLsBqmp/ko8Ad7Z2H66qqQvklzK4s+qVwFfai84xJEnzZEZBUVXrDrHr3BFtC7jsEONsAbaMqI8DZ42oPzHqGJKk+eMvsyVJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6pp1UCR5XZJ7hl4/SvK+JB9K8v2h+oVDfT6YZCLJg0nOH6qvbrWJJJuG6qcn+WaSXUk+l+SY2X9VSdJszDooqurBqlpRVSuAsxk89vRLbffHp/ZV1XaAJGcAa4EzgdXAJ5MsSrII+ARwAXAGsK61BfhYG2s58CRwyWznK0mancO19HQu8FBV/W2nzRrgxqp6tqoeZvBM7XPaa6KqdlfVT4AbgTVJArwN+ELrvxW46DDNV5I0Q4crKNYCNwx9vjzJvUm2JDm+1U4FHh1qM9lqh6q/BvhhVR04qP4CSTYmGU8yvm/fvrl/G0nSz8w5KNp1g98GPt9K1wCvBVYAe4E/nmo6onvNov7CYtXmqlpZVSvHxsZexOwlSdNZfBjGuAD4dlU9BjD1DpDk08CX28dJYOlQvyXAnrY9qv4D4Lgki9tZxXB7SdI8ORxLT+sYWnZKcsrQvncA97ftbcDaJK9IcjqwHPgWcCewvN3hdAyDZaxtVVXA14B3tv4bgFsOw3wlSS/CnM4okrwKeDvwnqHyf0mygsEy0SNT+6pqZ5KbgO8AB4DLquq5Ns7lwK3AImBLVe1sY30AuDHJR4G7gWvnMl9J0os3p6CoqmcYXHQerr2r0/5K4MoR9e3A9hH13QzuipIkLRB/mS1J6jIoJEldBoUkqcugkCR1GRSSpC6DQpLUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUtecgyLJI0nuS3JPkvFWOyHJjiS72vvxrZ4kVyeZSHJvkjcMjbOhtd+VZMNQ/ew2/kTrm7nOWZI0c4frjOJfVtWKqlrZPm8Cbquq5cBt7TPABQyelb0c2AhcA4NgAa4A3sjgiXZXTIVLa7NxqN/qwzRnSdIMvFRLT2uArW17K3DRUP36GrgDOC7JKcD5wI6q2l9VTwI7gNVt37FVdXtVFXD90FiSpHlwOIKigK8muSvJxlY7uar2ArT3k1r9VODRob6TrdarT46oP0+SjUnGk4zv27fvMHwlSdKUxYdhjDdV1Z4kJwE7kny303bU9YWaRf35harNwGaAlStXvmC/JGn25nxGUVV72vvjwJcYXGN4rC0b0d4fb80ngaVD3ZcAe6apLxlRlyTNkzkFRZJ/nOQXp7aB84D7gW3A1J1LG4Bb2vY2YH27+2kV8FRbmroVOC/J8e0i9nnArW3f00lWtbud1g+NJUmaB3NdejoZ+FK7Y3Ux8Nmq+sskdwI3JbkE+B5wcWu/HbgQmACeAd4NUFX7k3wEuLO1+3BV7W/blwLXAa8EvtJekqR5MqegqKrdwG+MqD8BnDuiXsBlhxhrC7BlRH0cOGsu85QkzZ6/zJYkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktRlUEiSugwKSVKXQSFJ6jIoJEldBoUkqWvWQZFkaZKvJXkgyc4kf9DqH0ry/ST3tNeFQ30+mGQiyYNJzh+qr261iSSbhuqnJ/lmkl1JPpfkmNnOV5I0O3M5ozgA/GFV/RqwCrgsyRlt38erakV7bQdo+9YCZwKrgU8mWZRkEfAJ4ALgDGDd0Dgfa2MtB54ELpnDfCVJszDroKiqvVX17bb9NPAAcGqnyxrgxqp6tqoeZvDc7HPaa6KqdlfVT4AbgTUZPIj7bcAXWv+twEWzna8kaXbm9MzsKUmWAa8Hvgm8Cbg8yXpgnMFZx5MMQuSOoW6T/DxYHj2o/kbgNcAPq+rAiPbSUWnZpr9Y6CnoCPbIVb/1kow754vZSV4N3Ay8r6p+BFwDvBZYAewF/niq6YjuNYv6qDlsTDKeZHzfvn0v8htIknrmFBRJfoFBSHymqr4IUFWPVdVzVfVT4NMMlpZgcEawdKj7EmBPp/4D4Lgkiw+qv0BVba6qlVW1cmxsbC5fSZJ0kLnc9RTgWuCBqvqTofopQ83eAdzftrcBa5O8IsnpwHLgW8CdwPJ2h9MxDC54b6uqAr4GvLP13wDcMtv5SpJmZy7XKN4EvAu4L8k9rfZHDO5aWsFgmegR4D0AVbUzyU3AdxjcMXVZVT0HkORy4FZgEbClqna28T4A3Jjko8DdDIJJkjSPZh0UVfVXjL6OsL3T50rgyhH17aP6VdVufr50JUlaAP4yW5LUZVBIkroMCklSl0EhSeoyKCRJXQaFJKnLoJAkdRkUkqQug0KS1GVQSJK6DApJUpdBIUnqMigkSV0GhSSpy6CQJHUZFJKkLoNCktR1xAdFktVJHkwykWTTQs9Hko42R3RQJFkEfAK4ADiDwfO4z1jYWUnS0eWIDgoGz8ueqKrdVfUT4EZgzQLPSZKOKosXegLTOBV4dOjzJPDGgxsl2QhsbB//LsmD8zC3o8GJwA8WehJHinxsoWegEfwbHTLHv9F/cqgdR3pQZEStXlCo2gxsfumnc3RJMl5VKxd6HtKh+Dc6P470padJYOnQ5yXAngWaiyQdlY70oLgTWJ7k9CTHAGuBbQs8J0k6qhzRS09VdSDJ5cCtwCJgS1XtXOBpHU1cztORzr/ReZCqFyz5S5L0M0f60pMkaYEZFJKkLoNCktRlUEiSugyKo1iSZUm+m2RrknuTfCHJq5Kcm+TuJPcl2ZLkFa39VUm+09r+t4Wev17e2t/nA0k+nWRnkq8meWWS1yb5yyR3Jfm/Sf5pa//aJHckuTPJh5P83UJ/h5cLg0KvAzZX1a8DPwLeD1wH/G5V/TMGt1BfmuQE4B3Ama3tRxdovjq6LAc+UVVnAj8E/g2DW2L/Q1WdDfwn4JOt7Z8Cf1pV/xx/mHtYGRR6tKr+X9v+n8C5wMNV9TetthV4C4MQ+Xvgz5P8DvDMvM9UR6OHq+qetn0XsAz4F8Dnk9wD/HfglLb/N4HPt+3PzuckX+6O6B/caV7M6Ic07ceP5zAIkrXA5cDbXsqJScCzQ9vPAScDP6yqFQs0n6OSZxQ6Lclvtu11wP8GliX5lVZ7F/D1JK8GfqmqtgPvA/wfqhbCj4CHk1wMkIHfaPvuYLA0BYP/mNFhYlDoAWBDknuBE4CPA+9mcGp/H/BT4FPALwJfbu2+DvzHBZqv9HvAJUn+GtjJz59R8z7g/Um+xWA56qkFmt/Ljv8XHkexJMuAL1fVWQs8FWnOkrwK+HFVVZK1wLqq8kFnh4HXKCS9XJwN/FmSMLhD6vcXeD4vG55RSJK6vEYhSeoyKCRJXQaFJKnLoJAkdRkUkqSu/w888NUN9lSZHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# categorify reviews in 2 classes neg, pos in the label column (rating != 3)\n",
    "df_trn_val = df[df['star_rating'] != 3].copy()\n",
    "df_trn_val['label'] = 'neg'\n",
    "df_trn_val.loc[df_trn_val['star_rating'] > 3, 'label'] = 'pos'\n",
    "\n",
    "# plot histogram\n",
    "x= [1,2]\n",
    "keys = list(df_trn_val['label'].value_counts().keys())\n",
    "values = list(df_trn_val['label'].value_counts().array)\n",
    "plt.bar(x, values) \n",
    "plt.xticks(x, keys)\n",
    "print(df_trn_val['label'].value_counts())\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_trn_val.to_csv (path_data/'amazon_reviews_fr.csv', index = None, header=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Get the csv of pre-processed data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_trn_val = pd.read_csv(path_data/'amazon_reviews_fr.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Check that the text language of each review is French and delete the non French ones"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from langdetect import detect, detect_langs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_csv(path_data/'amazon_reviews_fr.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "# to solve display error of pandas dataframe\n",
    "get_ipython().config.get('IPKernelApp', {})['parent_appname'] = \"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review_id</th>\n",
       "      <th>review_body</th>\n",
       "      <th>star_rating</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>R32VYUWDIB5LKE</td>\n",
       "      <td>je conseille fortement ce bouquin à ceux qui s...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>R3CCMP4EV6HAVL</td>\n",
       "      <td>ce magnifique est livre , les personnages sont...</td>\n",
       "      <td>5</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>R2E7QEWSC6EWFA</td>\n",
       "      <td>Je l'ai depuis quelques jours et j'en suis trè...</td>\n",
       "      <td>4</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>R26E6I47GQRYKR</td>\n",
       "      <td>je m'attendait à un bon film, car j'aime beauc...</td>\n",
       "      <td>2</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>R1RJMTSNCKB9LP</td>\n",
       "      <td>Ne disait pas sur l'annonce que c'était un 10'...</td>\n",
       "      <td>2</td>\n",
       "      <td>neg</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        review_id                                        review_body  \\\n",
       "0  R32VYUWDIB5LKE  je conseille fortement ce bouquin à ceux qui s...   \n",
       "1  R3CCMP4EV6HAVL  ce magnifique est livre , les personnages sont...   \n",
       "2  R2E7QEWSC6EWFA  Je l'ai depuis quelques jours et j'en suis trè...   \n",
       "3  R26E6I47GQRYKR  je m'attendait à un bon film, car j'aime beauc...   \n",
       "4  R1RJMTSNCKB9LP  Ne disait pas sur l'annonce que c'était un 10'...   \n",
       "\n",
       "   star_rating label  \n",
       "0            5   pos  \n",
       "1            5   pos  \n",
       "2            4   pos  \n",
       "3            2   neg  \n",
       "4            2   neg  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 18min 31s, sys: 11 s, total: 18min 42s\n",
      "Wall time: 18min 43s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "list_idx = []\n",
    "for idx, row in df.iterrows():\n",
    "    try:\n",
    "        language = detect(row['review_body'])\n",
    "    except:\n",
    "        language = \"error\"   \n",
    "    if language != 'fr':\n",
    "        list_idx.append(idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(230684, 11098)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df), len(list_idx)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15     Just great complination, there are 48 cds insi...\n",
       "18     I know it's a classic but really it is a marve...\n",
       "19     Waiting for so long to get a sequel of Bridget...\n",
       "66     Not one of his best science fiction novels but...\n",
       "117    Für die  Liebhaber von Schwarzer Humor  ist di...\n",
       "151    A great alternate look into the world of cats,...\n",
       "175    It's the perfect book for a screenwriter and a...\n",
       "214    Good delivery, on time. However the image of t...\n",
       "324    This is Frank Herbert's masterpiece and should...\n",
       "327    It was by sheer chance that I came across this...\n",
       "Name: review_body, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"review_body\"][list_idx][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df2 = df.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df2 = df.copy()\n",
    "df2.drop(list_idx, axis=0, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "219586"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Fine-tuning \"forward LM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7min 1s, sys: 3.93 s, total: 7min 5s\n",
      "Wall time: 3min 27s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols='review_body', \n",
    "                            processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_aws_sp15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4.04 s, sys: 1.71 s, total: 5.75 s\n",
      "Wall time: 6.54 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns2, drop_mult=0.3, \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZRc5Xnn8e9TS++r1K1dQiAJAcEGQ0PAPmZwiB1DPBBsMoPPeGLjjBkc7z5OjnN8BmecOIkTZxITzhgTO4w9SYhjsH2wjW0cT7AxMYsQEohVEhJIraVb6rW6q6treeaPe7tVNCWpJfWtRfX7nHNP37r3rXufrq6up97lvtfcHRERqV+xSgcgIiKVpUQgIlLnlAhEROqcEoGISJ1TIhARqXNKBCIidS7SRGBmHzOzbWb2jJl9vMT+K81s1My2hMutUcYjIiKvlYjqwGZ2PvAB4FJgGviRmf3A3bfPKfqQu78jqjhEROTYoqwRnAs84u6T7p4DfgZcH+H5RETkJERWIwC2AZ83s8VAGrgG2FSi3OVmthXYB3zK3Z851kF7enp87dq1Cx2riMhp7Yknnjjk7r2l9kWWCNz9OTP7AvATIAVsBXJzim0GznD3lJldA3wX2DD3WGZ2M3AzwJo1a9i0qVQ+ERGRozGzl4+2L9LOYnf/mrtf5O5XAEPA9jn7x9w9Fa7fDyTNrKfEce509z537+vtLZnQRETkJEU9amhJ+HMN8E7g7jn7l5mZheuXhvEcjjImERF5tSj7CADuDfsIssCH3H3YzG4BcPc7gBuAD5pZjqAf4UbXdKgiImUVaSJw9zeX2HZH0frtwO1RxiAiIsemK4tFROqcEoGISJ1TIhARqXNKBCIiNeBv/vVFHto+GMmxlQhERKqcu3PbT7fz6EtDkRxfiUBEpMpNTOcpOLQ3RTPQU4lARKTKjaWzAHQ0JyM5vhKBiEiVG5sKE0GTEoGISF0anwrm6+xoVtOQiEhdmm0aUo1ARKQ+zTYNqY9ARKQ+jaWDpiGNGhIRqVMzTUNKBCIidWpsKktTMkZjIh7J8ZUIRESq3PhULrKOYlAiEBGpemNT2cg6ikGJQESk6o2lc3RE1D8ASgQiIlVvbCpLu5qGRETq11i6hpuGzOxjZrbNzJ4xs4+X2G9mdpuZ7TCzp8zsoijjERGpRUFncQ02DZnZ+cAHgEuBC4B3mNmGOcWuBjaEy83Al6OKR0SkFrl7TXcWnws84u6T7p4DfgZcP6fMdcA3PPAI0GVmyyOMSUSkpkxlC2TzXrPDR7cBV5jZYjNrAa4BVs8psxLYU/R4b7hNREQ4Ms9QVFcVA0R2ZHd/zsy+APwESAFbgdycYlbqqXM3mNnNBE1HrFmzZoEjFRGpXlHflAYi7ix296+5+0XufgUwBGyfU2Qvr64lrAL2lTjOne7e5+59vb290QUsIlJljtyUpgY7iwHMbEn4cw3wTuDuOUXuA34nHD10GTDq7vujjElEpJaMzd6UJroaQXQpJnCvmS0GssCH3H3YzG4BcPc7gPsJ+g52AJPATRHHIyJSU6K+KQ1EnAjc/c0ltt1RtO7Ah6KMQUSklo1FfJtK0JXFIiJVrRw1AiUCEZEqNjaVpSEeozER3ce1EoGISBUbn8rR0ZzArNRo+4WhRCAiUsXG0tlIm4VAiUBEpKqNTeVoj3DoKCgRiIhUtaBGEO1IfyUCEZEqNjalpiERkbo2ls5Feg0BKBGIiFS1cdUIRETq11Q2TyZXiHSeIVAiEBGpWuMz00uos1hEpD7NTkGtGoGISH2amWcoyruTgRKBiEjVOtI0pBqBiEhdUtOQiEidG0urRiAiUteO1AjURyAiUpfG0lkSMaM5GY/0PEoEIiJVanwqR3tTtPciACUCEZGqNTaVjbyjGCJOBGb2CTN7xsy2mdndZtY0Z//7zGzQzLaEy3+LMh4RkVpSjpvSQISJwMxWAh8F+tz9fCAO3Fii6Dfd/cJw+WpU8YiI1JqxqehnHoXom4YSQLOZJYAWYF/E5xMROW3UfI3A3fuBLwKvAPuBUXd/oETRd5nZU2Z2j5mtLnUsM7vZzDaZ2abBwcGoQhYRqSrluCkNRNs01A1cB5wJrABazew9c4p9D1jr7q8H/hX4eqljufud7t7n7n29vb1RhSwiUlVmRg1FLcqmoV8Hdrn7oLtngW8Dbywu4O6H3T0TPvw74OII4xERqRnZfIHJ6XzNjxp6BbjMzFosGAR7FfBccQEzW1708Nq5+0VE6lW57kUAQWduJNz9UTO7B9gM5IAngTvN7HPAJne/D/iomV0b7h8C3hdVPCIitWRmCupy1AgiTTXu/lngs3M231q0/w+BP4wyBhGRWjQ7z1AtdxaLiMjJm20aqvE+AhEROUnlujsZKBGIiFSlct2UBpQIRESq0pGb0qhGICJSl8amssQMWhuUCERE6tJYOktbY4JYLNp7EYASgYhIVRqezNLd2lCWcykRiIhUoaGJaRYpEYiI1K/DE9MsViIQEalfQxMZ1QhEROqVu4dNQ41lOZ8SgYhIlRnP5MjmnUWt0V9MBkoEIiJVZ3hiGkA1AhGRenU4TATqLBYRqVNDqZkagRKBiEhdGppQIhARqWuzTUNtSgQiInVpaCJDUzJGSxkmnAMlAhGRqhNcVVyeEUMQcSIws0+Y2TNmts3M7jazpjn7G83sm2a2w8weNbO1UcYjIlILhiam6S7TNQQQYSIws5XAR4E+dz8fiAM3zin2u8Cwu68H/hr4QlTxiIjUiuEyXlUM0TcNJYBmM0sALcC+OfuvA74ert8DXGVm0U++LSJSxco54RxEmAjcvR/4IvAKsB8YdfcH5hRbCewJy+eAUWBxVDGJiNSCck5BDdE2DXUTfOM/E1gBtJrZe+YWK/FUL3Gsm81sk5ltGhwcXPhgRUSqxFQ2z+R0/vRIBMCvA7vcfdDds8C3gTfOKbMXWA0QNh91AkNzD+Tud7p7n7v39fb2RhiyiEhllXt6CYg2EbwCXGZmLWG7/1XAc3PK3Ae8N1y/Afh/7v6aGoGISL0o9/QSEG0fwaMEHcCbgafDc91pZp8zs2vDYl8DFpvZDuCTwKejikdEpBYcnsgA5buqGIJRPZFx988Cn52z+dai/VPAb0cZg4hILRkq8xTUoCuLRUSqymwiaDkNmoZEROTEDU1Mk4gZHc3lmWcIlAhERKpKML1EA+W8tlaJQESkipT7qmJQIhARqSrlvqoYlAhERKqKEoGISJ07nMqoaUhEpF5l8wXGpnJlvYYAlAhERKrG8OTMxWTluykNzDMRmNk6M2sM1680s4+aWVe0oYmI1JdKXFUM868R3AvkzWw9wfxAZwL/FFlUIiJ1qBITzsH8E0EhvHHM9cDfuPsngOXRhSUiUn9mp6Au44RzMP9EkDWzdxNMGf39cFt5G7FERE5zR5qGqjMR3ARcDnze3XeZ2ZnAP0QXlohI/Tk8MY0ZdJdxwjmY5zTU7v4s8FGYvQVlu7v/eZSBiYjUm6GJDF3NSeKx8s0zBPMfNfSgmXWY2SJgK3CXmf2vaEMTEakvlbiqGObfNNTp7mPAO4G73P1ignsSi4jIAqn2RJAws+XAf+JIZ7GIiCygak8EnwN+DOx098fN7Cxge3RhiYjUnyARlPdiMphnInD3b7n76939g+Hjl9z9Xcd6jpltNLMtRcuYmX18TpkrzWy0qMytRzueiMjprFBwhiezZZ9wDuY5asjMVgF/C7wJcOAXwMfcfe/RnuPuLwAXhs+PA/3Ad0oUfcjd33GCcYuInFZG01nyBa/qpqG7gPuAFcBK4Hvhtvm6iqBZ6eUTC09EpD5U6qpimH8i6HX3u9w9Fy7/B+g9gfPcCNx9lH2Xm9lWM/uhmf3KCRxTROS0cSiVAWBxtfYRAIfM7D1mFg+X9wCH5/NEM2sArgW+VWL3ZuAMd7+AoOnpu0c5xs1mtsnMNg0ODs4zZBGR2rFvJA3Ayu7msp97vong/QRDRw8A+4EbCKadmI+rgc3ufnDuDncfc/dUuH4/kDSznhLl7nT3Pnfv6+09kYqIiEht6B8OEsHyzqayn3u+o4Zecfdr3b3X3Ze4+28RXFw2H+/mKM1CZrbMzCxcvzSMZ141DRGR08m+0TQ9bQ00JeNlP/ep3KHsk8crYGYtwFuBbxdtu8XMbgkf3gBsM7OtwG3Aje7upxCTiEhN6h+ZYkVX+ZuFYJ7DR4/iuLMiufsksHjOtjuK1m8Hbj+FGERETgv7RtKs722ryLlPpUagb+4iIgvA3ekfTlekoxiOUyMws3FKf+AbUJmIRUROMyOTWdLZfHU2Dbl7e7kCERGpV/0zQ0e7yj9iCE6taUhERBbAzDUElaoRKBGIiFRYvxKBiEh92zeSpjERq8jMo6BEICJScftGpljZ1Ux4fW3ZKRGIiFRY/0i6Ys1CoEQgIlJxQSKozIghUCIQEamoTC7P4HiGlV0tFYtBiUBEpIIOjE4BqEYgIlKvjlxMpj4CEZG6NHMfAnUWi4jUqX0jQdPQsgrckGaGEoGISAXtG0nT295YkRvSzFAiEBGpoH2jlb2GAJQIREQqqn84XbFZR2coEYiIVIi7BxeTdapGICJSl4YmpsnkChW7M9mMyBKBmW00sy1Fy5iZfXxOGTOz28xsh5k9ZWYXRRWPiEi1mRkxVOk+glO5ef0xufsLwIUAZhYH+oHvzCl2NbAhXH4V+HL4U0TktNc/MglU9mIyKF/T0FXATnd/ec7264BveOARoMvMlpcpJhGRiuqvkhpBuRLBjcDdJbavBPYUPd4bbhMROe3tG0nTlIzR3ZKsaByRJwIzawCuBb5VaneJbV7iGDeb2SYz2zQ4OLjQIYqIVMS+kXRFb0gzoxw1gquBze5+sMS+vcDqosergH1zC7n7ne7e5+59vb29EYUpIlJelb4hzYxyJIJ3U7pZCOA+4HfC0UOXAaPuvr8MMYmIVJS7s+vQBGsWVe4+BDMiGzUEYGYtwFuB/1607RYAd78DuB+4BtgBTAI3RRmPiEi1GBjPMD6V4+yl7ZUOJdpE4O6TwOI52+4oWnfgQ1HGICJSjXYMpABYv6StwpHoymIRkYrYfnAcgA1KBCIi9Wn7QIqOpgS97Y2VDkWJQESkEnYMpFi/pK3iQ0dBiUBEpCJ2DKTYsKTyHcWgRCAiUnZDE9Mcnphmw9LK9w+AEoGISNnNjBhaVwUdxaBEICJSdtsHqmfEECgRiIiU3Y6BFM3JeMXvTDZDiUBEpMxmRgzFYpUfMQRKBCIiZbf9YKpqmoVAiUBEpKzGp7IcGJuqmo5iUCIQESmrmRFDqhGIiNSp7TOJoApmHZ2hRCAiUkY7B1I0xGOs7q6OEUOgRCAiUlbbB1Kc1dtKIl49H7/VE4mISB2YGTpaTZQIRETKJD2dZ8/wpBKBiEi92jmYwp2qmXV0hhKBiEiZ7BycGTFURzUCM+sys3vM7Hkze87MLp+z/0ozGzWzLeFya5TxiIhU0rb+URriMdYubq10KK8S6c3rgS8BP3L3G8ysAWgpUeYhd39HxHGIiFTcY7uHuWB1Jw2J6mqMiSwaM+sArgC+BuDu0+4+EtX5RESq2eR0jmf6R7lk7aJKh/IaUaals4BB4C4ze9LMvmpmpepDl5vZVjP7oZn9SoTxiIhUzJOvjJArOJecWV+JIAFcBHzZ3d8ATACfnlNmM3CGu18A/C3w3VIHMrObzWyTmW0aHByMMGQRkWg8vnsIM7j4jO5Kh/IaUSaCvcBed380fHwPQWKY5e5j7p4K1+8HkmbWM/dA7n6nu/e5e19vb2+EIYuIROPx3UOcu6yDjqZkpUN5jcgSgbsfAPaY2cZw01XAs8VlzGyZmVm4fmkYz+GoYhIRqYRsvsDml0e4tAqbhSD6UUMfAf4xHDH0EnCTmd0C4O53ADcAHzSzHJAGbnR3jzgmEZGyembfGOlsvio7iiHiRODuW4C+OZvvKNp/O3B7lDGIiFTa47uGALhkbfX1D4CuLBYRidxju4c4Y3ELSzqaKh1KSUoEIiIRKhScTbuHqrZZCJQIREQitXMwxfBklkuVCERE6tNju8P+gSodMQRKBCIikXp81xA9bY2sXVxqqrXqoEQgIhKhx3cPc+mZ3YSXTFUlJQIRkYjsHEzRP5Ku6v4BUCIQEYnMdzb3EzO45nXLKx3KMSkRiIhEoFBwvvNkP2/e0Fu11w/MUCIQEYnAo7uG6B9J886LVlY6lONSIhARicC9m/fS1pjgbectq3Qox6VEICKywCanc/zw6f385uuW09wQr3Q4x6VEICKywB545iAT0/maaBYCJQIRkQV37+a9rOpurur5hYopEYiILKADo1P8Ysch3vmGlcRi1XsRWTElAhGRBfTdLf24wzsvWlXpUOZNiUBEZIFMTue46+FdXLp2EWt7WisdzrwpEYiILJCv/OwlDo5l+IO3bzx+4SoS9T2LZQHkC87hVIa2pgQtDUf+ZNl8gf0jU+wdnuTwxDSj6Syj6SyZXIGlHY2s6GxmRVcz8ZgxMD7F4HiGwfHMbLnRdBZ32LisnfOWd3Du8g6WdjRW9eRYItVq/2iar/x8J7/5+uX01Ugn8YxIE4GZdQFfBc4HHHi/u/+yaL8BXwKuASaB97n75ihjqhR3n/3g7WpJlvywHZ/KsmMgxfaBFDsHUuwcnGD34QleOTzJdL4AQHMyzuK2BgD2j06RL/gJx2IGHU1JOpuT5AvOfVv3ze5rTMToaWukt72RnrZGulqSs2WTCWMqW2Aqm2cqmycZj9HWmKCtMUFrY4LWxvjselMyTiJmJOMxknGjszlJV0sD8RrpPBM5UX/xoxcoOHz67edUOpQTFnWN4EvAj9z9BjNrAOZOyH01sCFcfhX4cvizpmRyebYfTPH8gXFeODDG8GR29sMylclxcCzD/tE0U9ngw7ylIc6q7maWdTaTns5xKDXNofEM45nc7DEbEjHWLm5hXW8rV527hFVdzaQyeQ6nMhyemCZfcNYsamHNohZWLWqmt62RzuYkHc1JkvEYg+MZ+kfS7BtJU3Cnt72RJeGHe0dT8lWjGcamsjy/f5zn9o+xbyQd1BxSGfYOT/LsvqDmMDGdny3flIzRlIwznSswWbT9eMygsznJopYGOluSdDUHCWZpRxOruptZtaiFpe1NpLM5RiaD86azeRIxI2ZGPGY0JGI0JuI0JGK0NMRnk1Zb46vfyvmC0z+c5qVDKV4anKB/JM3wxDTDk9MMT2aJGbQ3JWlrStAeJq5giVFwGCuqNU1O50hnC2TCv2k6m2dyOlgHaErEaQyf25CIkYzFSCaMuBl5d/KF4ItAUzJOV0uS7pYGOpuTtDcdSaJdLcnZ36W3vZH2xoRqZjVky54RvvNkP7935TpWL6re+w4cjbmf+DfKeR3YrAPYCpzlRzmJmX0FeNDd7w4fvwBc6e77j3bcvr4+37RpUxQhH1Oh4GzbN8pD2w/x2K4hRianGc/kSE3lGJqYJhd+M29MxFjc2kBTQ5ymRJzWxjhLO5pY3tnEss5m3J3+kTT9w2kOjE3Nfpj1tDWyrLOJdb1tbFjSxupFLVX17TmXL5DNO03J2Ks+oAoFZ2I6x0QmSHoTmRypTI6pbJ5s3skVCmTzBUYnswxNZhmemGZocpqxdJaRySzDk9MMjGVmazwnq6UhTnMyznQ+ON90rkBxZak5GWdRawOLWhvoakniHtTAxqdyjIfxZrKF2TjamxJ0homqtSFBYzJGc5gsWhqCnzNXjAZJP0gUM+fP5Z1cwYmHSSxmkM7mGU0Hv/PIZJZUJsfR/v2ScaO7JYi3rTFBKpNjfCrHWDpLLGZ0twQ1rM7mJGZB4iu4E4/F6GlrCBN/E4tak+HvERxrdXczibi6BheSu3PDHb/k5cOTPPj7V77mS0m1MLMn3L2v1L4oIz4LGATuMrMLgCeAj7n7RFGZlcCeosd7w21HTQQn68lXhvmz+5/nirN7ePOGXl63snP2W/FUNs+B0Sl2DgbNMtsPpjgwlp79Jy2488KBcYYnswCcs6ydJR1NrFrUQntjgp62Rs5Z3s65yztYu7i1qj7AF0oiHiNR4kr5WMxob0rS3pQ86WMXCs5gKsOeoUkGxjO0NMTpammgqzlJc0OcfMHJF4IP1pkP+Uwuz0Qmz6FU0O8xMJ5hKpunIRGjIR58M1/Z1cxZvW2c2dNKT1vDvL5hzzS1leNv6O6ks3lSUzmGJ7Ozv8vgeFDrG56Y5vDENBOZHKsXtdDRFNQiCu4MT2YZmQxqODPxxs3I5nPsODjOYCpDNv/aLNOQiLG+t41zlrWzbkkbK7uaWd7ZxIquI/1JcmK+9otdPPHyMF941+uqNgkcT5RRJ4CLgI+4+6Nm9iXg08D/KCpT6l33mnevmd0M3AywZs2akwpmIpNnMpvjiw+8yBcfeJHuliTLOps5ODbF0MT0q8ouaW9kVfer/ynecs4S/sPZvbxpfQ89bY0nFYOUFosZSzuaWFoFU/WW84PQzGhpCAYALOloYiPtC3bsmT6pmYQxks5yaDzD9oGgCfPhnYf49pP9r3pOUzLGxqXtbFzWzsZlHaxf0sa63lZWdDbXzIVR5fbdJ/v5kx88x2/8ylJuuHh1pcM5aVE2DS0DHnH3teHjNwOfdvffLCpT9qahQ6kMD+84xM9fPMTw5DTLOptY3tHEss4mzuptZX1vO50tJ//tVqRWpKfz7BsN+pH2DqfZfjDFCwfHeH7/OIeLvhw1JWOsX9I2O7Ls3OUdbFzaTndrQwWjr7x/e2GAD3x9E5esXcRdN11CU7K6J5erSNOQux8wsz1mttHdXwCuAp6dU+w+4MNm9s8EncSjx0oCC6GnrZHrLlzJdRfWxmRQIlFpboizrreNdb1tr9l3KJXhpcEJdg6m2DGQ4sWD4/z0uQH+ZdPe2TK97Y2cvbSNc5Z1cNGabi46o4vlnc3l/BUq5omXh/jgPzzBOcvbufN3Lq76JHA8kdUIAMzsQoLhow3AS8BNwH8GcPc7wuGjtwNvJxg+epO7H/PrfqU6i0XqnbszOJ7hmf1jbD84zosHU2w/OM7zB8bJ5IJO9hWdTVx65iLeuK6HN65fzKru2htBczw/efYgn/zmFnraG/nWLZfXTFPxsWoEkSaCKCgRiFSX6VyB5/aPsfmVYTa9PMyjLx3mUCpoWlq9qJmNSztYt6SVdb1tnLusg3OXt9fkyKVCwfmbn27ntp9u53UrO/nKf72YFV21UwNSIhCRsnF3XjyY4uEdh9j08hA7BlLsPnTkosiWhjgXru6i74xu3rCmm9ev6mRxlX+rHp3M8sl/2cJPnx/ghotX8Se/dX7NNQcpEYhIReXyBfYMp3m6f5Qndg+x6eVhnts/Nnutx6ruZi5Y3cUFqzp5/aouzl/ZWRVDMQ+nMvz9w7v4xr+/TDqb57P/8Tzec9kZNXmxX6WuIxARAYLrUM7saeXMnlauvWAFAKlMjm39o2zdM8JTe0fZ8soIP3gqGCtiBut723j9qi4uWN3J61Z2cvbSdlrLkBxy+QJb9ozw/af2883H9zCVy3P1+cv48Fs2cN6KjsjPXwlKBCJSEW2NCS47azGXnbV4dtuhVIan946ydW+QHH724gD3bj4yUmlFZxPrlrSxdnEri1obWNwWXDHdUTRdSFtTgu6WhuM23bg7E9N5+ofT9I9Msnc4zWO7hnho+yFG01kSMePaC1fwe1euY/2ShbvGoxopEYhI1ehpa+Qt5yzhLecsAYIP632jUzy9d5QdA+PsGEixYzDF0/2jjIRX+h9NczJOd0uSpmScbCGY9iObd6ZzeTK5wuxIp2K97Y289bylXLmxlzev762ba4qUCESkapkZK7uaWdnVDCx71b5cvsDwZJahielg3qhwrquxdC6czymYoiOTK9AQj5GIGYl4jMZEjMZkMHlha0OcFV3NrOxuZlVXM73t9TkNuxKBiNSkRDw2O1urnJraG8wrIiILSolARKTOKRGIiNQ5JQIRkTqnRCAiUueUCERE6pwSgYhInVMiEBGpczU3+6iZDQIjwOicXZ3H2Xa89ZmfPcChkwit1Pnns3/u9mM9nhtr8baTibucMRevV+K11vtD749j7a/F98eJxAywwd07Sx7d3WtuAe480W3HWy/6uWmhYprP/rnbj/V4bqynGnc5Y670a633h94fp9v740RiPt45arVp6Hsnse1466Wef6oxzWf/3O3Helwq1lOJu5wxF69X4rXW++PE6f0x//Vqj/mY56i5pqGomdkmP8rNG6pZLcatmMunFuNWzOVTqzWCKN1Z6QBOUi3GrZjLpxbjVsxlohqBiEidU41ARKTOndaJwMz+3swGzGzbSTz3YjN72sx2mNltVnS3CjP7iJm9YGbPmNlfLGzU0cRtZn9kZv1mtiVcrqn2mIv2f8rM3Mx6Fi7iyF7nPzazp8LX+AEzW1EDMf+lmT0fxv0dM+tayJgjjPu3w//BgpktWLv8qcR6lOO918y2h8t7i7Yf831fVicz1KlWFuAK4CJg20k89zHgcsCAHwJXh9vfAvwr0Bg+XlIjcf8R8Klaeq3DfauBHwMvAz3VHjPQUVTmo8AdNRDz24BEuP4F4Au18P4AzgU2Ag8CfZWONYxj7Zxti4CXwp/d4Xr3sX6vSiyndY3A3X8ODBVvM7N1ZvYjM3vCzB4ys3PmPs/MlhP8Q//Sg7/YN4DfCnd/EPhzd8+E5xiokbgjFWHMfw38AbDgnVlRxOzuY0VFWxc67ohifsDdc2HRR4BVCxlzhHE/5+4vVEusR/EbwE/cfcjdh4GfAG+v5P9qKad1IjiKO4GPuPvFwKeA/12izEpgb9HjveE2gLOBN5vZo2b2MzO7JNJojzjVuAE+HFb//97MuqMLddYpxWxm1wL97r416kCLnPLrbGafN7M9wH8Bbo0w1hkL8d6Y8X6Cb6flsJBxR20+sZayEthT9Hgm/mr5vYA6u2exmbUBbwS+VdQcV+qGp6Xa6ma+2SUIqniXAZcA/2JmZ4VZPRILFPeXgT8OH/8x8FcE//SRONWYzawF+AxBs0VZLNDrjLt/BviMmf0h8GHgswsc6pFAFijm8FifAXLAPy5kjKUsZNxRO1asZqRFkOYAAAS1SURBVHYT8LFw23rgfjObBna5+/UcPf6K/17F6ioRENSARtz9wuKNZhYHnggf3kfwoVlcPV4F7AvX9wLfDj/4HzOzAsH8IoPVHLe7Hyx63t8B348wXjj1mNcBZwJbw3++VcBmM7vU3Q9Uacxz/RPwAyJMBCxQzGEn5juAq6L8UlNkoV/rKJWMFcDd7wLuAjCzB4H3ufvuoiJ7gSuLHq8i6EvYS+V/ryMq1TlRrgVYS1GnD/DvwG+H6wZccJTnPU7wrX+mI+eacPstwOfC9bMJqn1WA3EvLyrzCeCfqz3mOWV2s8CdxRG9zhuKynwEuKcGYn478CzQu9CxluP9wQJ3Fp9srBy9s3gXQStCd7i+aL7v+3ItFTlp2X45uBvYD2QJMvDvEnzL/BGwNXzz33qU5/YB24CdwO0cufiuAfiHcN9m4NdqJO7/CzwNPEXwTWt5tcc8p8xuFn7UUBSv873h9qcI5nZZWQMx7yD4QrMlXBZ0pFOEcV8fHisDHAR+XMlYKZEIwu3vD1/jHcBNJ/K+L9eiK4tFROpcPY4aEhGRIkoEIiJ1TolARKTOKRGIiNQ5JQIRkTqnRCCnBTNLlfl8XzWz8xboWHkLZivdZmbfO97sn2bWZWa/txDnFgHdmEZOE2aWcve2BTxewo9MxBap4tjN7OvAi+7++WOUXwt8393PL0d8cvpTjUBOW2bWa2b3mtnj4fKmcPulZvbvZvZk+HNjuP19ZvYtM/se8ICZXWlmD5rZPRbM1/+PM3PGh9v7wvVUONHcVjN7xMyWhtvXhY8fN7PPzbPW8kuOTLrXZmY/NbPNFsxbf11Y5s+BdWEt4i/Dsr8fnucpM/ufC/gySh1QIpDT2ZeAv3b3S4B3AV8Ntz8PXOHubyCYHfRPi55zOfBed/+18PEbgI8D5wFnAW8qcZ5W4BF3vwD4OfCBovN/KTz/ceeRCefZuYrgym+AKeB6d7+I4D4YfxUmok8DO939Qnf/fTN7G7ABuBS4ELjYzK443vlEZtTbpHNSX34dOK9oxsgOM2sHOoGvm9kGghkfk0XP+Ym7F89F/5i77wUwsy0Ec9D8Ys55pjkyid8TwFvD9cs5Msf8PwFfPEqczUXHfoJgznoI5qD50/BDvUBQU1ha4vlvC5cnw8dtBInh50c5n8irKBHI6SwGXO7u6eKNZva3wL+5+/Vhe/uDRbsn5hwjU7Sep/T/TNaPdLYdrcyxpN39QjPrJEgoHwJuI7ifQS9wsbtnzWw30FTi+Qb8mbt/5QTPKwKoaUhObw8Q3A8AADObmUa4E+gP198X4fkfIWiSArjxeIXdfZTg9pafMrMkQZwDYRJ4C3BGWHQcaC966o+B94fz5mNmK81syQL9DlIHlAjkdNFiZnuLlk8SfKj2hR2ozxJMIQ7wF8CfmdnDQDzCmD4OfNLMHgOWA6PHe4K7P0kww+WNBDeI6TOzTQS1g+fDMoeBh8Phpn/p7g8QND390syeBu7h1YlC5Jg0fFQkIuFd1tLu7mZ2I/Bud7/ueM8TKTf1EYhE52Lg9nCkzwgR3hpU5FSoRiAiUufURyAiUueUCERE6pwSgYhInVMiEBGpc0oEIiJ1TolARKTO/X97s7q/4tClEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.364937</td>\n",
       "      <td>4.127997</td>\n",
       "      <td>0.728301</td>\n",
       "      <td>0.271699</td>\n",
       "      <td>62.053352</td>\n",
       "      <td>02:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.188382</td>\n",
       "      <td>3.942470</td>\n",
       "      <td>0.710881</td>\n",
       "      <td>0.289120</td>\n",
       "      <td>51.545666</td>\n",
       "      <td>02:20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_sp15')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.975761</td>\n",
       "      <td>3.833473</td>\n",
       "      <td>0.699623</td>\n",
       "      <td>0.300377</td>\n",
       "      <td>46.222725</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.848893</td>\n",
       "      <td>3.728329</td>\n",
       "      <td>0.687341</td>\n",
       "      <td>0.312659</td>\n",
       "      <td>41.609566</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.721791</td>\n",
       "      <td>3.643100</td>\n",
       "      <td>0.677313</td>\n",
       "      <td>0.322686</td>\n",
       "      <td>38.210094</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.683572</td>\n",
       "      <td>3.585832</td>\n",
       "      <td>0.670209</td>\n",
       "      <td>0.329791</td>\n",
       "      <td>36.083378</td>\n",
       "      <td>03:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.627503</td>\n",
       "      <td>3.545522</td>\n",
       "      <td>0.665744</td>\n",
       "      <td>0.334256</td>\n",
       "      <td>34.657887</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.588596</td>\n",
       "      <td>3.513960</td>\n",
       "      <td>0.661785</td>\n",
       "      <td>0.338215</td>\n",
       "      <td>33.581036</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.559013</td>\n",
       "      <td>3.486362</td>\n",
       "      <td>0.658416</td>\n",
       "      <td>0.341584</td>\n",
       "      <td>32.666874</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.530540</td>\n",
       "      <td>3.464996</td>\n",
       "      <td>0.655928</td>\n",
       "      <td>0.344072</td>\n",
       "      <td>31.976351</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.486778</td>\n",
       "      <td>3.447401</td>\n",
       "      <td>0.653625</td>\n",
       "      <td>0.346374</td>\n",
       "      <td>31.418600</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.500075</td>\n",
       "      <td>3.430552</td>\n",
       "      <td>0.651237</td>\n",
       "      <td>0.348763</td>\n",
       "      <td>30.893700</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.474257</td>\n",
       "      <td>3.416125</td>\n",
       "      <td>0.649317</td>\n",
       "      <td>0.350683</td>\n",
       "      <td>30.451197</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.425713</td>\n",
       "      <td>3.401726</td>\n",
       "      <td>0.647419</td>\n",
       "      <td>0.352581</td>\n",
       "      <td>30.015905</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.415072</td>\n",
       "      <td>3.389692</td>\n",
       "      <td>0.645582</td>\n",
       "      <td>0.354418</td>\n",
       "      <td>29.656876</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.381176</td>\n",
       "      <td>3.380713</td>\n",
       "      <td>0.644282</td>\n",
       "      <td>0.355718</td>\n",
       "      <td>29.391685</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.401082</td>\n",
       "      <td>3.372867</td>\n",
       "      <td>0.643033</td>\n",
       "      <td>0.356966</td>\n",
       "      <td>29.162067</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3.353319</td>\n",
       "      <td>3.368499</td>\n",
       "      <td>0.642499</td>\n",
       "      <td>0.357501</td>\n",
       "      <td>29.034859</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>3.384467</td>\n",
       "      <td>3.366534</td>\n",
       "      <td>0.642141</td>\n",
       "      <td>0.357859</td>\n",
       "      <td>28.977922</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>3.331605</td>\n",
       "      <td>3.366147</td>\n",
       "      <td>0.642058</td>\n",
       "      <td>0.357942</td>\n",
       "      <td>28.966690</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(18, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "28.966703060315766"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# perplexity\n",
    "val_loss = 3.366147\n",
    "np.exp(val_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_sp15')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_sp15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_sp15')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_sp15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"backward LM\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7min 6s, sys: 3.71 s, total: 7min 10s\n",
      "Wall time: 3min 37s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = (TextList.from_df(df_trn_val, path, cols='review_body', \n",
    "                            processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_for_lm()           \n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm.save(f'{path}/{lang}_databunch_lm_aws_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.41 s, sys: 196 ms, total: 2.61 s\n",
      "Wall time: 2.61 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_lm_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 928 ms, sys: 196 ms, total: 1.12 s\n",
      "Wall time: 544 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "perplexity = Perplexity()\n",
    "learn_lm = language_model_learner(data_lm, AWD_LSTM, config=config, pretrained_fnames=lm_fns2_bwd, drop_mult=0.3, \n",
    "                                  metrics=[error_rate, accuracy, perplexity]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_lm.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5hcVZnv8e9bVX3vpNNJOqFzIyQEJCKX0KBBDQiIwKAIigfUM6AoowOieNTR8XnGUeeiozIzyowcVARHxOE6gqAQOQKi4dIBQhJAE8IlTXfSnXSn75eqrvf8sXcnlaY6aZPeden+fZ5nP1V71dp7v12p1Ftrr73XMndHRERktFi+AxARkcKkBCEiIlkpQYiISFZKECIikpUShIiIZJXIdwATafbs2b548eJ8hyEiUjTWrl27w93rsr02qRLE4sWLaWxszHcYIiJFw8xeGes1nWISEZGslCBERCQrJQgREclKCUJERLJSghARkayUIEREJCslCBERyUoJQkSkiK1+bjvXPfxiJPtWghARKWK/eW47N/7+5Uj2rQQhIlLEegZTVJdHMyiGEoSISBHrGkhSXaYEISIio/QMppimFoSIiIzWM1CECcLMbjCzVjPbkFH2LTN7wcyeNbO7zGzGGNueZWZ/NLPNZvbFqGIUESl2PYOpojzFdCNw1qiy1cDR7n4M8CfgS6M3MrM48B/A2cBy4GIzWx5hnCIiRatnIEV1WUkk+44sQbj7I0D7qLIH3D0Vrj4GLMiy6UnAZnff4u5DwM+B86KKU0SkWKXTTs/Q5LyK6aPAr7KUzwe2Zqw3hWVZmdnlZtZoZo1tbW0THKKISOHqSw7jDtOK8BTTmMzsy0AKuDnby1nKfKx9ufv17t7g7g11dVlnzRMRmZR6BoITMlG1IHI+5aiZXQKcC5zu7tm++JuAhRnrC4DmXMQmIlJMugeSAEXZSf06ZnYW8DfAe9y9b4xqTwLLzOwwMysFLgLuzlWMIiLFonsw2hZElJe53gKsAY40syYzuwy4FpgGrDazZ8zsurDuPDO7DyDsxL4SuB94HrjV3TdGFaeISLEaOcUUVR9EZKeY3P3iLMU/GqNuM3BOxvp9wH0RhSYiMin0hC2IaeVFdpmriIhEK+pOaiUIEZEitbsPYjJ0UouIyMTZ3YJQghARkUw9g0kqS+PEY9luHzt4ShAiIkWqeyC6gfpACUJEpGh1RzibHChBiIgUrZ6BVGT3QIAShIhI0YpyPmpQghARKVpBCyKam+RACUJEpGipBSEiIll1DyR1FZOIiOzN3ekZTDFNLQgREcnUnxwm7dHdRQ1KECIiRak74oH6QAlCRKQodUc8DhMoQYiIFKU9c0EUYYIwsxvMrNXMNmSUXWhmG80sbWYN+9j2ZTNbH8461xhVjCIixWr3bHIRTRYE0bYgbgTOGlW2AbgAeGQc27/D3Y9z9zETiYjIVNUzmASiPcUU5ZSjj5jZ4lFlzwOYRTM0rYjIVDGV+yAceMDM1prZ5fuqaGaXm1mjmTW2tbXlKDwRkfwq6j6Ig/RWd18BnA1cYWarxqro7te7e4O7N9TV1eUuQhGRPBrpg6iaai0Id28OH1uBu4CT8huRiEhh6R5MUV4SoyQe3dd4wSUIM6sys2kjz4EzCTq3RUQkFMwmF90VTBDtZa63AGuAI82sycwuM7PzzawJWAnca2b3h3Xnmdl94aZzgUfNbB3wBHCvu/86qjhFRIpR1OMwQbRXMV08xkt3ZanbDJwTPt8CHBtVXCIik0FPxCO5QgGeYhIRkf3LRQtCCUJEpAgFfRBKECIiMkrUs8mBEoSISFHqGUwxTS0IERHJ5O7BKSa1IEREJNNAMs1w2ov3PggREYlG98hIrmpBiIhIpt1zQagPQkREMuViJFdQghARKTo9OZgLApQgRESKTnfYglAfhIiI7GVPH4SuYhIRkQw9akGIiEg23QPBZa5VZfFIj6MEISJSZLoHU5QmYpQllCBERCRDz0D04zBBtDPK3WBmrWa2IaPsQjPbaGZpM2vYx7ZnmdkfzWyzmX0xqhhFRIpRLkZyhWhbEDcCZ40q2wBcADwy1kZmFgf+AzgbWA5cbGbLI4pRRKTo9AxEP1kQRJgg3P0RoH1U2fPu/sf9bHoSsNndt7j7EPBz4LyIwhQRKTrdg9FPFgSF2QcxH9iasd4UlmVlZpebWaOZNba1tUUenIhIvvUMpCIfyRUKM0FYljIfq7K7X+/uDe7eUFdXF2FYIiKFIRfzUUNhJogmYGHG+gKgOU+xiIgUnO6B5JQ9xfQksMzMDjOzUuAi4O48xyQiUhDcvfivYjKzW4A1wJFm1mRml5nZ+WbWBKwE7jWz+8O688zsPgB3TwFXAvcDzwO3uvvGqOIUESkmg6k0yWHPSQsisiO4+8VjvHRXlrrNwDkZ6/cB90UUmohI0crVXBBQmKeYRERkDLmaCwKUIEREisrukVyVIEREJNOuvmAk1xmVpZEfSwlCRKSItPcNATCzamreKCciImPo6A0SRK1aECIikqm9dwgzqKlQC0JERDJ09A1RU1FCIh7917cShIhIEWnvHWJmDk4vgRKEiEhR6egborZKCUJEREZp703mpIMalCBERIpKR+9QTi5xBSUIEZGi4e606xSTiIiM1jc0zFAqrU5qERHZW/vITXJqQYiISKaOkWE21IIQEZFMk6YFYWY3mFmrmW3IKJtpZqvNbFP4WDvGtsNm9ky4aLpRERH2jORaW1n8VzHdCJw1quyLwIPuvgx4MFzPpt/djwuX90QYo4hI0RhpQcws9haEuz8CtI8qPg+4KXx+E/DeqI4vIjLZdPQNETOYXl78LYhs5rp7C0D4OGeMeuVm1mhmj5nZPpOImV0e1m1sa2ub6HhFRApGe+8QtZWlxGKWk+MVaif1IndvAD4I/JuZLR2rortf7+4N7t5QV1eXuwhFRHIsl+MwQe4TxHYzqwcIH1uzVXL35vBxC/AQcHyuAhQRKVS5HMkVcp8g7gYuCZ9fAvxidAUzqzWzsvD5bOCtwHM5i1BEpEB19CapzdE4TBDtZa63AGuAI82sycwuA74BvNPMNgHvDNcxswYz+2G46VFAo5mtA34LfMPdlSBEZMpr7xvK2RVMAImoduzuF4/x0ulZ6jYCHwuf/wF4U1RxiYgUI3enI+ykzpVC7aQWEZEM3YMpUmnPaQtCCUJEpAh0jAyzUWgtCDNbmtFxfKqZXWVmM6INTURERuT6LmoYfwviDmDYzA4HfgQcBvwssqhERGQvIyO5FuJ9EGl3TwHnA//m7lcD9dGFJSIimdp7g4H6CvE+iKSZXUxw78Ivw7LcXYwrIjLF7e6DKMD7ID4CrAT+0d1fMrPDgJ9GF5aIiGRq7xuiJG5Ul0V2d8LrjOtI4Y1qV0FwpzMwzd2/EWVgIiKyx8g9EGa5GagPxn8V00NmNt3MZgLrgB+b2TXRhiYiIiPae3N7FzWM/xRTjbt3ARcAP3b3E4AzogtLREQydfTl9i5qGH+CSISjr36APZ3UIiKSI4XcgvgacD/wors/aWZLgE3RhSUiIpk6+nI7kiuMv5P6NuC2jPUtwPuiCkpERPYYTju7+nI7FwSMv5N6gZndZWatZrbdzO4wswVRByciItDVnyTtub2LGsZ/iunHBJP9zAPmA/eEZSIiErH2vtyPwwTjTxB17v5jd0+Fy42AJoAWEcmBfIzkCuNPEDvM7MNmFg+XDwM797eRmd0QnpbakFE208xWm9mm8LF2jG0vCetsMrNLstUREZkK8jGSK4w/QXyU4BLXbUAL8H6C4Tf250bgrFFlXwQedPdlwIPh+l7CG/K+ArwZOAn4yliJRERkssvHSK4wzgTh7q+6+3vcvc7d57j7ewlumtvfdo8A7aOKzwNuCp/fBLw3y6bvAla7e7u7dwCreX2iERGZEvIxkisc3Ixynz3A7ea6ewtA+DgnS535wNaM9aaw7HXM7HIzazSzxra2tgMMSUSkcHX0DVFeEqOiNJ7T4x5MgohyxKhs+/ZsFd39endvcPeGujr1m4vI5NPem/t7IODgEkTWL+xx2B4O20H42JqlThOwMGN9AdB8gMcTESlqHb1DOe9/gP0kCDPrNrOuLEs3wT0RB+JugomHCB9/kaXO/cCZZlYbdk6fGZaJiEw57X25H4cJ9jPUhrtPO5idm9ktwKnAbDNrIrgy6RvArWZ2GfAqcGFYtwH4hLt/zN3bzezrwJPhrr7m7qM7u0VEpoSdPUMsmlmZ8+NGOjWRu188xkunZ6nbCHwsY/0G4IaIQhMRKQrptLOtc4BDaspzfuyD6YMQEZGItfcNMTScpn66EoSIiGRo2TUAQP2MipwfWwlCRKSANXf2AzCvRglCREQybOscaUHoFJOIiGRo7uynNB4ruhvlREQkYi27giuYYrEoB6/ITglCRKSAtXT2U5+HS1xBCUJEpKC1dA4oQYiIyN7SaWd710BeLnEFJQgRkYK1o2eQ5LAzTy0IERHJ1BJe4npIHu6BACUIEZGC1RLeJKc+CBER2UtzOMzGPPVBiIhIppbOfsoSMWorS/JyfCUIEZECNXKJq1nub5KDPCUIM/u0mW0ws41m9pksr59qZp1m9ky4/F0+4hQRyacgQeTn9BJEPGFQNmZ2NPBx4CRgCPi1md3r7ptGVf2du5+b6/hERApFy65+3rJ0Vt6On48WxFHAY+7e5+4p4GHg/DzEISJSsIbTzvbuwbxdwQT5SRAbgFVmNsvMKoFzgIVZ6q00s3Vm9isze2NuQxQRya+27kGG0z61TjG5+/Nm9k1gNdADrANSo6o9BRzq7j1mdg7wP8CybPszs8uBywEWLVoUWdwiIrm0e6KgPMwDMSIvndTu/iN3X+Huq4B2YNOo17vcvSd8fh9QYmazx9jX9e7e4O4NdXV1kccuIpILI1ONHjI9fy2IfF3FNCd8XARcANwy6vVDLLyuy8xOIohzZ67jFBHJl5YCaEHk/BRT6A4zmwUkgSvcvcPMPgHg7tcB7wc+aWYpoB+4yN09T7GKiORcS+cAFSVxairyc5Mc5ClBuPvbs5Rdl/H8WuDanAYlIlJARiYKytdNcqA7qUVEClJL5wD1eTy9BEoQIiIFqWVXfu+iBiUIEZGCkxpO09qdv6lGRyhBiIgUmO3dg6QdtSBERGRv20YmClIfhIiIZNo9UZBaECIikql5V9CCOER9ECIikmlLWy+zq0vzepMcKEGIiBScTa3dHD6nOt9hKEGIiBQSd2dTa48ShIiI7K21e5DugRTL5kzLdyhKECIihWTT9h4AlqkFISIimTa3dgNw+FwlCBERybCptYeaihLqqsvyHYoShIhIIdnU2sOyOdV5HeZ7hBKEiEgB2dzaw7ICOL0E+Zty9NNmtsHMNprZZ7K8bmb2XTPbbGbPmtmKfMQpIpJLO3sGae8d4vACuIIJ8pAgzOxo4OPAScCxwLlmtmxUtbOBZeFyOfD9nAYpIpIHm1qDK5gK4R4IyE8L4ijgMXfvc/cU8DBw/qg65wE/8cBjwAwzq891oCIiubS5tXAucYX8JIgNwCozm2VmlcA5wMJRdeYDWzPWm8IyEZFJa3NrD1Wl8bxPFDQikesDuvvzZvZNYDXQA6wDUqOqZeu+92z7M7PLCU5DsWjRogmMVEQktza1dnP43GkFcQUT5KmT2t1/5O4r3H0V0A5sGlWlib1bFQuA5jH2db27N7h7Q11dXTQBi4jkwKbtPQVzegny0IIAMLM57t5qZouAC4CVo6rcDVxpZj8H3gx0untLLmNMDqfZ1jnAa7v6aensZ1vnINu7BmjtHiARi1FVlqC6LM708hJmVZcxu7qUumll1NdUMGdaGbFYYfwCEJHi0NmfpLV7UAkCuMPMZgFJ4Ap37zCzTwC4+3XAfQR9E5uBPuAjUQZzxc1P0dmfpHcoRe9giq7+FK3dA6RHndSaVp5gzrQyhtNOz+AwvYMp+pPDr9tfaSLGgtoK6mvK6RsaZldfko6+IYZSaarLEkwrT1BdXsKMihJqK0uYUVnKjMoSqkoTlJfGqSyJE4tB/1Ca/uQwA8ngWL2DKXoGh+kbStEzmKJ7IHgcTA2TTkMqncY9iLO2snT3fqeVJ5heHjwCJIed1HCaYXcqS+NUliaoKoszo7KUeTUV1M8oZ3r5nnHoh9NO2p2SuG6bEYnK7g7qArkHAvKUINz97VnKrst47sAVuYqnJZz/tboswdxp5VSXJ5hXU8782grmz6ikfkY5h0wvp6rs9W/XUCrNzt5BdnQP0dYzQPOuAba29/Fqex8tnQNUlyWYP6OC2spSShMxesMv9q6BIGls2dHDrt4k3YOju2H2lohZ2GpJUFkaD770K0qYN6OcskSceMxIhK2WroEkHb1Jmjr62Nic3J1I/hyVpXEMGEylSYWZsjQeo7IsTlVpgtJELOgoMoibMT0j2U0vL6GyNE5FaZyyRJBUUukgKY0ksOkVJUwrLyERs91JcCCZJh6DRCxGIm6UxmOUxGOUJkYejdJ4nNJEjHjMGErtSaCptFMSs+B9iMcoSwTblYXbZp7S7RsapqN3iPbeIXb1JYP3Nx5sGzMj7c5w2kmlnXT4OBwuJfFg/yXxGDGDtEM6TKCVZQlqKkp2LzMqSpheUUJcrUkZh91jMNUVxj0QkL8WREG586/fesDbliZi1NdUUF9TAdQc8H5Sw8GXXf/QMP3JYdIefEmXl8QpL4lRlogf8L4haAX0DKTAoCRu4Rdc8OXcNxgkkPbeIZo7B2jZ1c+2rgFiZpQlgmObBV+sIy2ZZNpxdxwYHna6BpK8tmuAjc1ddPUnd/8NAtPDhDjSiptWXkJZSYzSeLBUlMaZVVW6+1TlnOnlzJ1exuzqMrXappBN23soL4kxvza/81BnUoIoEIl4jGnxGNPKo5liMB4zaipfv+/qsFUyB1gygX387k5y2OlPDmMWtIBGfkn3Dg7T1Z+kayBJKh2c5ipPxCkrieEe9P8khz18DJbBVFA2lNpTVpYIWinliaDFkRoOfu0nh9MMpdIMjTym0nvFVl4SZ2ZVKTOrgikdzdirxRALW2Mxs90ti0QsaDGM7D81HLQo4jHDDGJm9A2l6OxP0tmfZFffnsddfUN0DaToHkjSNZDitV39DKWGd/+NI6cLRzODWVWlVJYmqCiJ7z79ONL/VVWW4JDp5SycWcnCmRUsrK1kdrX6v4rVptYeltZVF1SLUwlCImFmwSmhxOt/AZclgi9o2WMwNUx77xA7uodo7R5ge9cg27oGaOsepH8o6OvqT6bpGwwSTN9QkFTae4f22k9J3Dikppx5NRUsnFnJYbOrdi+HzqqkslT/5QvV5tYeTlxcm+8w9qJPi0gBKEvED+hU5UBymKaOPra297O1o4/mXQO0dPbzWkc/j/ypjdvXNu1Vf3Z1GYfOqmTJ7CqWz5vO0fNrOKp+OtVZ+tckd3rCxH/xnNH3DOeXPhUiRay8JM7hc6aNObhbz2CKl3f08tKOXl5t7+OVncHjb//Yym1h8jCDpXXVHL9wBscvqmXFoTM4Ys40narKocaX2wE4ZsGMPEeyNyUIkUmsuizB0fNrOHr+3q0Sd6e1e5CNzZ2sb+piXdMufvP89t1JY0ZlCScunsmbD5vJW5bMYnn9dCWMCK3ZspOSuHHi4pn5DmUvShAiU5CZMXd6OXOnl3PaG+YCQdJ4ZWcfa1/p4PGXdvL4S+2sfm47ADUVJbz5sJmsXDqLdy6fy4LaynyGP+mseXEnxy+spaL04K5WnGhKECICBElj8ewqFs+u4n0nLACCe4Qe39LOmhd38octO3jgue189Z7nOGnxTN57/Hz+4k31Wa+Ok/Hr7E+y4bVOPnXa6FkP8k8JQkTGVF9TwXuPn897jw8GU35lZy/3rGvmrqdf42/vWs9X7t7AKUfM4d3H1vPO5XN1ldQBeOKldtIOJy+dle9QXkf/miIybofOquLK05ZxxTsOZ2NzF//z9Gv88tkWfvP8dipK4px21BzOPvoQ3nHknKwjD8jr/eHFHZQlYhy3qLA6qEEJQkQOgJnt7vz+23OO4smX27l7XTP3b9zGvc+2UJqIccoRdVx68mJOXjqrYIavLkRrXtzJiYtnHvRoCVFQghCRgxKLGW9eMos3L5nF1847msaX2/n1xm388tkWPvTDx1mxaAafOm0Zpx5Zp0Qxys6eQV7Y1s3n3zUv36FkpQQhIhMmnpEs/uasN3D72ia+/9CLfOTGJ3nT/BquOn0ZZxw1R4ki9NiW4P6HlQXY/wB5mjBIRCa/8pI4H37LoTz0+VP5l/cdQ9dAko//pJFzv/coD2zcRjBo89S2ZssOqssSHDP/wAf6jJIShIhEqiQe4wMnLuTBz57Cty88lp7BFJf/11rO/vffcc+6Zoan8LC/f3hxJycdNpNEgY7aW5hRicikk4jHeP8JC3jws6fwnQuPJTmc5lO3PM0Z1zzMrU9uJTmc3v9OJpHtXQNsaetl5ZLCPL0EeUoQZna1mW00sw1mdouZlY96/VIzazOzZ8LlY/mIU0QmXiIe430nLGD11afw/Q+toKoszhfueJbTvvPQlEoUa17cCRRu/wPkIUGY2XzgKqDB3Y8G4sBFWar+t7sfFy4/zGmQIhK5WMw4+0313HPl2/jxpSdSW1nKF+54ltO/8zC3Nm4lNckTxaObd1BTUcLy+un5DmVM+TrFlAAqzCwBVALNeYpDRPLMzHjHG+bwiyveyo8uaWB6RYIv3P4sZ1zzMHc+1TQp+yg6eoe499kWzlw+t6AHQcx5gnD314BvA68CLUCnuz+Qper7zOxZM7vdzMYcJN3MLjezRjNrbGtriyhqEYmamXH6UXO558q38YO/bKCiNMFnb13HO695mF9vaJlUVz39ZM0r9CeHuXzVknyHsk/5OMVUC5wHHAbMA6rM7MOjqt0DLHb3Y4DfADeNtT93v97dG9y9oa5uAufMFJG8MDPeuXwu937qbVz34RXEY8YnfvoUF//gMZ5r7sp3eAetf2iYm9a8zBlHzWHZ3OzzeBSKfJxiOgN4yd3b3D0J3AmcnFnB3Xe6+2C4+gPghBzHKCJ5FosZZx1dz68+/Xa+ft4b+eO2bs793u/40p3r2dEzuP8dFKjb1m6lvXeIvzplab5D2a98JIhXgbeYWaUFt1OeDjyfWcHM6jNW3zP6dRGZOhLxGP975WIe+tw7uOTkxdzWuJVTv/UQ33/oRQaSw/kO78+SGk7zg99tYcWiGTQcWljzT2eTjz6Ix4HbgaeA9WEM15vZ18zsPWG1q8LLYNcRXPF0aa7jFJHCUlNZwlfe/Ubuv3oVb1kyk2/++gXOuOZhfvHMa6SLpCP7vg3b2NrezydOWVoUw43YZOr4aWho8MbGxnyHISI58OimHfzDvc/xwrZujphbzdVnHMG73nhIwV4V5O6c+71H6U8O85urTymYOM1srbs3ZHtNd1KLSFF627LZ3HfV2/nuxceTSjufvPkpzv3eo/xqfUtBXhr7m+db2djcxV+tWlIwyWF/lCBEpGjFYsZ7jp3HA59ZxTUfOJa+oRSfvPkpzrjmYX7+xKsMpgqjj2JnzyBfunM9R86dtnt2vmKgBCEiRS8Rj3HBigU8+H9O5T8+GAzf8cU71/O2b/6W7z24ifbeobzF5u586c71dPUn+beLjivIiYHGovkgRGTSiMeMvzimnnPedAiPbt7BD373Et9Z/Seu/e1mzj9+PpecvJijcjy0xa2NW3ngue18+Zyjcn7sg6UEISKTjpnx9mV1vH1ZHZu2d3PD71/mzqea+PmTW1mxaAYfevOh/MUx9ZSXRPtr/pWdvXz1nudYuWQWl73tsEiPFQVdxSQiU8KuviFuX9vEzx5/lS07eqmpKOGCFfP54EmLIrmjeUfPIB+98Ule2tHL/Z9ZxbwZFRN+jImwr6uYlCBEZEpxdx7b0s7Nj7/C/Ru3kRx2TlxcywcaFvKuow9hennJQR/jsS07ueqWp9nVn+Tai4/nzDceMgGRR0MJQkQkix09g9yxtolbnniVl3f2URqPseqIOt59bD1vO3w2s6rL/qz9pdPOfz60mWtW/4nFs6q49oMrWD6vsPsdlCBERPbB3Xl66y5+ua6Fe9c3s70rGOtpyewqVhxay/GLZrC0rpols6uom1a2113Q7s7G5i7uXd/CfetbeGVnH+8+dh7/fMGbqC4r/G5eJQgRkXFKp52nt3bwxEsdrH2lnbWvdNDRl9z9enVZghmVJcTMiMeMvqEU27sGiceMk5fO4sKGhbz7mPqiGEoD9p0gCj+9iYjkUCxmnHDoTE44dCawFHenqaOfl3b07l66+pMMuzOcduIxY+WSWZz5xkOYWVWa7/AnlBKEiMg+mBkLZ1aycGYlq46YWnPO6E5qERHJSglCRESyUoIQEZGslCBERCSrvCQIM7s6nDFug5ndYmblo14vM7P/NrPNZva4mS3OR5wiIlNZzhOEmc0nmEa0wd2PBuLARaOqXQZ0uPvhwL8C38xtlCIikq9TTAmgwswSQCXQPOr184Cbwue3A6dbsdx1IiIySeQ8Qbj7a8C3gVeBFqDT3R8YVW0+sDWsnwI6gVnZ9mdml5tZo5k1trW1RRe4iMgUk/Mb5cyslqCFcBiwC7jNzD7s7j/NrJZl06xjgrj79cD14b47zWxTlmo1BElmPOsjz7OVzQZ2jPnHZTf6WON9PVt5tpjGen4wMe8rrvHGVywxZysvxs/HeGLOfK7Px/hfn+yfj2Vj7t3dc7oAFwI/ylj/S+A/R9W5H1gZPk8QvKk2jn1fP57yfa2PPB+jrPEA/t6sMf25MY8V0/7iP5CYDzTuYox5snw+xhNzvt9rfT4K//MxeslHH8SrwFvMrDLsVzgdeH5UnbuBS8Ln7wf+n4d/yX7cM87yfa3fs4+yA7G/bccb8+iy/T0/mJjHs/3+4stWVogxZysvxs/HeGLOfK7Px/hfn0qfj73kZTRXM/sq8L+AFPA08DHgywQZ9u7wstf/Ao4H2oGL3H1LzgMdxcwafYxRDwuVYs6dYoxbMedOMcadl8H63P0rwFdGFf9dxusDBKeiCs31+Q7gACjm3CnGuBVz7hRd3Pr4JBAAAAcASURBVJNqPggREZk4GmpDRESyUoIQEZGspmyCMLMbzKzVzDYcwLYnmNn6cKyo72be5W1mnzKzP4ZjTf1LocdsZn9vZq+Z2TPhck6hx5zx+ufMzM1s9sRFvHvfUbzXXzezZ8P3+QEzm1cEMX/LzF4I477LzGYUQcwXhv//0mY2YZ3CBxPrGPu7xMw2hcslGeX7/Nzn1IFcTzwZFmAVsALYcADbPgGsJLih71fA2WH5O4DfAGXh+pwiiPnvgc8V0/scvraQ4H6ZV4DZxRA3MD2jzlXAdUUQ85lAInz+TeCbRRDzUcCRwEMEY77lNdYwjsWjymYCW8LH2vB57b7+rnwsU7YF4e6PEFxCu5uZLTWzX5vZWjP7nZm9YfR2ZlZP8B99jQf/mj8B3hu+/EngG+4+GB6jtQhijlSEMf8r8AXGuMO+EON2966MqlUTHXtEMT/gwXA3AI8BC4og5ufd/Y8TGefBxDqGdwGr3b3d3TuA1cBZ+fy/ms2UTRBjuB74lLufAHwO+M8sdeYDTRnrTWEZwBHA2y0YovxhMzsx0mgDBxszwJXhKYQbLBgKJWoHFbOZvQd4zd3XRR3oKAf9XpvZP5rZVuBDZFzaHaGJ+HyM+CjBL9qoTWTMURtPrNnsHm8uNBJ/ofxdQJ7ugyhEZlYNnEwwNtRIcVm2qlnKRn4JJgiai28BTgRuNbMl4S+BCTdBMX8f+Hq4/nXgOwRfBJE42JjNrJLgpsozo4kwuwl6r3H3LwNfNrMvAVfy+vuBJsxExRzu68sEN7bePJExvi6QCYw5avuK1cw+Anw6LDscuM/MhoCX3P18xo4/739XJiWIPWLALnc/LrPQzOLA2nD1boIv1Mxm9gL2DFfeBNwZJoQnzCxNMEBXVMPMHnTM7r49Y7sfAL+MKNYRBxvzUoKBHteF/ykXAE+Z2Unuvq2A4x7tZ8C9RJggmKCYww7Uc4HTo/qxk2Gi3+coZY0VwN1/DPwYwMweAi5195czqjQBp2asLyDoq2gi/3/XHvnq/CiEBVhMRocT8AfgwvC5AceOsd2TBK2EkU6kc8LyTwBfC58fQdCE3O8gg3mOuT6jztXAzwv9fR5V52Ui6KSO6L1ellHnU8DtRRDzWcBzQF0U73GUnw8muJP6QGNl7E7qlwjOONSGz2eO93OfqyUvBy2EBbiFYD6KJEHWvozgl+mvgXXhf4q/G2PbBmAD8CJwLXvuSC8Ffhq+9hRwWhHE/F/AeuBZgl9m9YUe86g6LxPNVUxRvNd3hOXPEgyQNr8IYt5M8EPnmXCZ6Cuvooj5/HBfg8B24P58xkqWBBGWfzR8fzcDH/lzPve5WjTUhoiIZKWrmEREJCslCBERyUoJQkREslKCEBGRrJQgREQkKyUImdTMrCfHx/uhmS2foH0NWzDy6wYzu2d/I6ma2Qwz++uJOLYIaEY5meTMrMfdqydwfwnfM3hdpDJjN7ObgD+5+z/uo/5i4JfufnQu4pPJTy0ImXLMrM7M7jCzJ8PlrWH5SWb2BzN7Onw8Miy/1MxuM7N7gAfM7FQze8jMbrdgroSbR8bsD8sbwuc94eB868zsMTObG5YvDdefNLOvjbOVs4Y9gxVWm9mDZvaUBfMGnBfW+QawNGx1fCus+/nwOM+a2Vcn8G2UKUAJQqaifwf+1d1PBN4H/DAsfwFY5e7HE4y0+k8Z26wELnH308L144HPAMuBJcBbsxynCnjM3Y8FHgE+nnH8fw+Pv99xdsJxiE4nuNMdYAA4391XEMxB8p0wQX0ReNHdj3P3z5vZmcAy4CTgOOAEM1u1v+OJjNBgfTIVnQEszxiBc7qZTQNqgJvMbBnBCJolGdusdvfMuQCecPcmADN7hmCMnkdHHWeIPYMfrgXeGT5fyZ4x/n8GfHuMOCsy9r2WYM4ACMbo+afwyz5N0LKYm2X7M8Pl6XC9miBhPDLG8UT2ogQhU1EMWOnu/ZmFZvY94Lfufn54Pv+hjJd7R+1jMOP5MNn/LyV9TyffWHX2pd/djzOzGoJEcwXwXYK5JOqAE9w9aWYvA+VZtjfgn939//6ZxxUBdIpJpqYHCOZiAMDMRoZrrgFeC59fGuHxHyM4tQVw0f4qu3snwRSlnzOzEoI4W8Pk8A7g0LBqNzAtY9P7gY+G8xZgZvPNbM4E/Q0yBShByGRXaWZNGctnCb5sG8KO2+cIhmkH+Bfgn83s90A8wpg+A3zWzJ4A6oHO/W3g7k8TjBh6EcGkPQ1m1kjQmnghrLMT+H14Wey33P0BglNYa8xsPXA7eycQkX3SZa4iORbOitfv7m5mFwEXu/t5+9tOJNfUByGSeycA14ZXHu0iwileRQ6GWhAiIpKV+iBERCQrJQgREclKCUJERLJSghARkayUIEREJKv/D2Mcd5t+I+3AAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.793482</td>\n",
       "      <td>4.527842</td>\n",
       "      <td>0.768429</td>\n",
       "      <td>0.231571</td>\n",
       "      <td>92.558655</td>\n",
       "      <td>02:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.632069</td>\n",
       "      <td>4.341979</td>\n",
       "      <td>0.753315</td>\n",
       "      <td>0.246686</td>\n",
       "      <td>76.859566</td>\n",
       "      <td>02:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned1_sp15_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned1_enc_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>error_rate</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.296514</td>\n",
       "      <td>4.161376</td>\n",
       "      <td>0.736233</td>\n",
       "      <td>0.263767</td>\n",
       "      <td>64.159561</td>\n",
       "      <td>03:23</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.130993</td>\n",
       "      <td>3.987864</td>\n",
       "      <td>0.716614</td>\n",
       "      <td>0.283386</td>\n",
       "      <td>53.939629</td>\n",
       "      <td>03:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.956398</td>\n",
       "      <td>3.833526</td>\n",
       "      <td>0.698954</td>\n",
       "      <td>0.301046</td>\n",
       "      <td>46.225235</td>\n",
       "      <td>03:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.807486</td>\n",
       "      <td>3.726387</td>\n",
       "      <td>0.686238</td>\n",
       "      <td>0.313763</td>\n",
       "      <td>41.528687</td>\n",
       "      <td>03:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.754490</td>\n",
       "      <td>3.651143</td>\n",
       "      <td>0.678229</td>\n",
       "      <td>0.321771</td>\n",
       "      <td>38.518734</td>\n",
       "      <td>03:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.655463</td>\n",
       "      <td>3.600486</td>\n",
       "      <td>0.672112</td>\n",
       "      <td>0.327887</td>\n",
       "      <td>36.615993</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.633501</td>\n",
       "      <td>3.560688</td>\n",
       "      <td>0.667255</td>\n",
       "      <td>0.332745</td>\n",
       "      <td>35.187481</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.603338</td>\n",
       "      <td>3.529700</td>\n",
       "      <td>0.663621</td>\n",
       "      <td>0.336379</td>\n",
       "      <td>34.113655</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.567799</td>\n",
       "      <td>3.505535</td>\n",
       "      <td>0.660722</td>\n",
       "      <td>0.339277</td>\n",
       "      <td>33.299282</td>\n",
       "      <td>03:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.557265</td>\n",
       "      <td>3.483793</td>\n",
       "      <td>0.658104</td>\n",
       "      <td>0.341895</td>\n",
       "      <td>32.583061</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.523547</td>\n",
       "      <td>3.464284</td>\n",
       "      <td>0.655464</td>\n",
       "      <td>0.344535</td>\n",
       "      <td>31.953585</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.481527</td>\n",
       "      <td>3.450032</td>\n",
       "      <td>0.653492</td>\n",
       "      <td>0.346508</td>\n",
       "      <td>31.501377</td>\n",
       "      <td>03:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.483834</td>\n",
       "      <td>3.436066</td>\n",
       "      <td>0.651271</td>\n",
       "      <td>0.348730</td>\n",
       "      <td>31.064537</td>\n",
       "      <td>03:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.466180</td>\n",
       "      <td>3.425846</td>\n",
       "      <td>0.650107</td>\n",
       "      <td>0.349893</td>\n",
       "      <td>30.748672</td>\n",
       "      <td>03:21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.438890</td>\n",
       "      <td>3.417838</td>\n",
       "      <td>0.648938</td>\n",
       "      <td>0.351062</td>\n",
       "      <td>30.503456</td>\n",
       "      <td>03:19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>3.406104</td>\n",
       "      <td>3.412687</td>\n",
       "      <td>0.648142</td>\n",
       "      <td>0.351857</td>\n",
       "      <td>30.346628</td>\n",
       "      <td>03:10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>3.402734</td>\n",
       "      <td>3.410627</td>\n",
       "      <td>0.647868</td>\n",
       "      <td>0.352132</td>\n",
       "      <td>30.284266</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>3.415850</td>\n",
       "      <td>3.410233</td>\n",
       "      <td>0.647768</td>\n",
       "      <td>0.352232</td>\n",
       "      <td>30.272303</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(18, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30.27229688291125"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# perplexity\n",
    "val_loss = 3.410233\n",
    "np.exp(val_loss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned2_sp15_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned2_enc_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save best LM learner and its encoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_sp15_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## Fine-tuning \"forward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.16 s, sys: 408 ms, total: 2.56 s\n",
      "Wall time: 2.56 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7min 10s, sys: 4.84 s, total: 7min 15s\n",
      "Wall time: 4min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, vocab=data_lm.vocab, cols='review_body', \n",
    "                              processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols='label')\n",
    "    .databunch(bs=bs, num_workers=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.52 s, sys: 1.03 s, total: 6.55 s\n",
      "Wall time: 6.06 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_sp15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.9 s, sys: 832 ms, total: 12.7 s\n",
      "Wall time: 12.5 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(207616, 23068, 230684)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 23071, 184545]), array([ 2566, 20502]))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.888876579839704, 0.11112342016029597],\n",
       " [0.8887636552800416, 0.11123634471995836])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Training (Loss = FlattenedLoss of weighted CrossEntropyLoss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12.6 s, sys: 284 ms, total: 12.9 s\n",
      "Wall time: 12.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=0.5, \n",
    "                                  metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "functools.partial(<function cross_entropy at 0x7f30905a4200>, weight=tensor([0.8889, 0.1111], device='cuda:0'))"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU5dXA8d+ZrGQBQgiIhCUgq6AskUVw3xDfikttwQ23qrUutdW+Wqu1+lq1tdW2Yi0KWutCFZeiVRFFRBSEIHsgkIQtIBASEshCtjnvH3MDQ5iEIHOTzOR8P5/5MHPvc2fOJBnO3Oc+z3lEVTHGGGPq8jR3AMYYY1omSxDGGGMCsgRhjDEmIEsQxhhjArIEYYwxJqDI5g4gWDp27Kg9e/Zs7jCMMSakLF26dLeqpgTaFzYJomfPnmRkZDR3GMYYE1JEZHN9+6yLyRhjTECWIIwxxgRkCcIYY0xAliCMMcYEZAnCGGNMQJYgjDHGBGQJwhhjTECWIIwxJoTNXJrH699sceW5LUEYY0wIe3tpHu8uy3PluS1BGGNMCCsqr6Jdm2hXntsShDHGhLDiskrax0W58tyWIIwxJoTtKauifRtLEMYYY/zsr6qhvKqGpPgQ7GISkXEikiUi2SJyX4D9PUTkMxFZKSLzRCTVb1+NiCx3brPcjNMYY0LR3vIqANq5dAbhWrlvEYkApgDnAXnAEhGZpaqZfs2eAl5R1X+KyNnA48A1zr5yVR3iVnzGGBPqipwEEYrXIEYA2aqaq6qVwAxgQp02A4HPnPufB9hvjDGmHkVlToIIwVFMXYGtfo/znG3+VgCXO/cvBRJFJNl5HCsiGSKySEQuCfQCInKz0yYjPz8/mLEbY0yLt6esEgjNMwgJsE3rPL4HOENElgFnANuAamdfd1VNB64EnhGR3oc9mepUVU1X1fSUlIAr5hljTNgqLnO3i8nNJUfzgG5+j1OB7f4NVHU7cBmAiCQAl6tqsd8+VDVXROYBQ4EcF+M1xpiQUlReewYRel1MS4A+IpImItHAROCQ0Ugi0lFEamO4H5jubE8SkZjaNsAYwP/itjHGtHpFZVVEeoT46AhXnt+1BKGq1cDtwGxgLfCmqq4RkUdE5GKn2ZlAloisBzoDjznbBwAZIrIC38XrJ+qMfjLGmFavqLyK9nFRiATq0T92bnYxoaofAh/W2faQ3/2ZwMwAx30NDHYzNmOMCXVFZZWudS+BzaQ2xpiQVeRimQ2wBGGMMSGrqKzKtRFMYAnCGGNCVrGLpb7BEoQxxoSsPS6W+gZLEMYYE5Iqqmsoq6whyRKEMcYYf8W1lVxtFJMxxhh/B8ps2CgmY4wx/twu9Q2WIIwxJiTtKXXqMNkoJmOMMf7sDMIYY0xAbpf6BksQxhgTkorKK4nwCAkx7pXUswRhjDEhqLYOk1uVXMEShDHGhKSisiraudi9BJYgjDEmJBWVV5Lk4iQ5sARhjDEhye1S32AJwhhjQpJ1MRljjAmoqKzS1UlyYAnCGGNCTmW1l1KXK7mCJQhjjAk5xU0wixosQRhjTMgpLvfVYXKz1DdYgjDGmJBT1ASlvsEShDHGhJw9TVCHCVxOECIyTkSyRCRbRO4LsL+HiHwmIitFZJ6IpPrtmywiG5zbZDfjNMaYUFJU5utiCtmJciISAUwBLgQGApNEZGCdZk8Br6jqScAjwOPOsR2A3wIjgRHAb0Ukya1YjTEmlBxcbjR0zyBGANmqmquqlcAMYEKdNgOBz5z7n/vtvwCYo6qFqroHmAOMczFWY4wJGUVlVUR4hEQXK7mCuwmiK7DV73Ges83fCuBy5/6lQKKIJDfyWETkZhHJEJGM/Pz8oAVujDEt2Z6yStq5XMkV3E0QgSLXOo/vAc4QkWXAGcA2oLqRx6KqU1U1XVXTU1JSjjVeY4wJCUXl7tdhAnDz/CQP6Ob3OBXY7t9AVbcDlwGISAJwuaoWi0gecGadY+e5GKsxxoSM4rIq10cwgbtnEEuAPiKSJiLRwERgln8DEekoIrUx3A9Md+7PBs4XkSTn4vT5zjZjjGn1isorae/yCCZwMUGoajVwO77/2NcCb6rqGhF5REQudpqdCWSJyHqgM/CYc2wh8Ci+JLMEeMTZZowxrV5TlPoGd7uYUNUPgQ/rbHvI7/5MYGY9x07n4BmFMcYYR1OU+gabSW2MMSGlqsZLSUW165PkwBKEMcaElKaq5AqWIIwxJqTUFupr1wTXICxBGGNMCKkt9R3So5iMMcYE357Spin1DZYgjDEmpBQ51yDsIrUxxphD1Jb6tmGuxhhjDlFcXoVHcL2SK1iCMMaYkFJbydXjcbeSK1iCMMaYkFJYWkmHePevP4AlCGOMCSkFJZUkx8c0yWtZgjDGmBBiZxDGGGMCKiytJMkShDHGGH9er7KnrJJkSxDGGGP8FZdX4VWsi8kYY8yhCkp9k+SSEyxBGGOM8VPoJAg7gzDGGHOIwtIKoGnqMIElCGOMCRmFTiVX62IyxhhziNozCOtiMsYYc4iC0koSYiKJiYxoktezBGGMMSGiKWdRgyUIY4wJGZYgjDHGBFRQEkYJQkTGiUiWiGSLyH0B9ncXkc9FZJmIrBSR8c72niJSLiLLndvzbsZpjDGhYE9Z0yYI15YkEpEIYApwHpAHLBGRWaqa6dfsN8Cbqvp3ERkIfAj0dPblqOoQt+IzxphQoqoUlDZdHSZw9wxiBJCtqrmqWgnMACbUaaNAW+d+O2C7i/EYY0zIKq2sobLaGzZdTF2BrX6P85xt/h4GrhaRPHxnD3f47Utzup6+EJHTAr2AiNwsIhkikpGfnx/E0I0xpmUpLGnaMhvgboIItGCq1nk8CXhZVVOB8cC/RMQDfAd0V9WhwC+A10WkbZ1jUdWpqpququkpKSlBDt8YY1qOgiaeJAfuJog8oJvf41QO70K6EXgTQFUXArFAR1WtUNUCZ/tSIAfo62KsxhjTou0pC68ziCVAHxFJE5FoYCIwq06bLcA5ACIyAF+CyBeRFOciNyLSC+gD5LoYqzHGtGgFThdTU61HDS6OYlLVahG5HZgNRADTVXWNiDwCZKjqLOCXwAsicje+7qfrVFVF5HTgERGpBmqAW1W10K1YjTGmpTtQ6ruJCvWBiwkCQFU/xHfx2X/bQ373M4ExAY57G3jbzdiMMSaUFJZWEh3pIT66aeowgc2kNsaYkFA7B0Ik0Pgfd1iCMMaYEFBYWtlkCwXVsgRhjDEhoLC0sskWCqplCcIYY0JAU1dyBUsQxhgTEixBGGOMOUxFdQ0lFdVNWqgPLEEYY0yLVzsHIskShDHGGH+1CcLOIIwxxhziwCzqJiyzAZYgjDGmxTuYIOwMwhhjjJ+DhfosQRhjjPFTWFqJR6Bdm6gmfd1GJQgR6S0iMc79M0XkThFp725oxhhjwFeHKSkuGo+n6eowQePPIN4GakTkBGAakAa87lpUxhhjDtjTDJPkoPEJwquq1cClwDOqejfQxb2wjDHG1GqOWdTQ+ARRJSKTgMnAB862pu0MM8aYVqqgtKLJC/VB4xPE9cBo4DFV3SgiacCr7oVljDGmVnOdQTRqRTln5bc7AUQkCUhU1SfcDMwYYwzUeJWi8io6NPFaEND4UUzzRKStiHQAVgAvicif3Q3NGGNMUVklqk0/SQ4a38XUTlX3ApcBL6nqcOBc98IyxhgDfrOoE5q2zAY0PkFEikgX4EccvEhtjDHGZQXNVKgPGp8gHgFmAzmqukREegEb3AvLGGMMNF8dJmj8Req3gLf8HucCl7sVlDHGGJ/tReVACz6DEJFUEXlXRHaJyE4ReVtEUhtx3DgRyRKRbBG5L8D+7iLyuYgsE5GVIjLeb9/9znFZInLB0b0tY4wJfTVe5fXFWzjx+LakJLbcaxAvAbOA44GuwPvOtnqJSAQwBbgQGAhMEpGBdZr9BnhTVYcCE4HnnGMHOo9PBMYBzznPZ4wxrcaczB3k5pfy0zN7I9K0dZig8QkiRVVfUtVq5/YykHKEY0YA2aqaq6qVwAxgQp02CrR17rcDtjv3JwAzVLVCVTcC2c7zGWNMq6Cq/H1eDj2S47hwUPNUNmpsgtgtIleLSIRzuxooOMIxXYGtfo/znG3+HgauFpE84EPgjqM4FhG5WUQyRCQjPz+/kW/FGGNavoU5BazIK+aW03sT0cRVXGs1NkHcgG+I6w7gO+CH+MpvNCTQO9I6jycBL6tqKjAe+JeIeBp5LKo6VVXTVTU9JeVIJzTGGBM6/v5FDimJMVw27LDvxk2mUQlCVbeo6sWqmqKqnVT1EnyT5hqSB3Tze5zKwS6kWjcCbzqvsRCIBTo28lhjjAlLq/KK+XLDbm4cm0ZsVPNdfj2WFeV+cYT9S4A+IpImItH4LjrPqtNmC3AOgIgMwJcg8p12E0UkxikM2AdYfAyxGmNMyHj+ixwSYyO5amT3Zo2jUfMg6tFgp5iqVovI7fgm2EUA01V1jYg8AmSo6izgl8ALInI3vi6k61RVgTUi8iaQCVQDP1PVmmOI1RhjQsLmglI+XP0dPz2jN4mxzbuqwrEkiMOuCRzWQPVDfBef/bc95Hc/ExhTz7GPAY8dQ3zGGBNyvs4pQBV+lN7tyI1d1mCCEJF9BE4EArRxJSJjjGnFVm0rpm1sJD2S45o7lIYThKomNlUgxhhjYM22YgZ1bdcsE+PqOpaL1MYYY4KoqsbL2h37GNS1XXOHAliCMMaYFmPDzhIqq72WIIwxxhxq9fZiAAYd3/YILZuGJQhjjGkhVm8rJiEmkp7J8c0dCmAJwhhjWozV24oZeHxbPM1Ue6kuSxDGGNMCVNd4yfxuL4OObxnXH8AShDHGtAi5u0vZX+VlcGrLuP4AliCMMaZFWL2t9gK1nUEYY4zxs2pbMW2iIuiVktDcoRxgCcIYY1qANdv2MvD4ts22OFAgliCMMaaZeb3Kmu3FLWb+Qy1LECbs+CrGGxM6NhaUUlpZ02JmUNeyBGHCyp8/ySL9/z5lxuIteL2HJgpVZVVeMVsLy5opOmN8nv8ihwUbdh94fOACdQtLEMeyHoQxLcpHq77jr3OzSUmM4b53VvH2t3k8dulgenWM56PVO3hxwUZWbC0iOtLDby4awDWjerSIipmmddlcUMoTH60jwiM8cdlgrkjvxuptxcREeujTqeVcoAZLECZMZO8q4Z63VjCkW3tm3DyKWSu28/iHaxn/ly/pmBDDjr37SesYz8M/GMi89fk89J81zF+/mz/+8CSS4qObO3zTiszLygd8Zwv3zlzJrn0VrNpWTP8ubYmMaFmdOpYgTMgrqajmln9lEBsVwd+vHkZsVAQ/Su/GuQM688fZWWwvKufRSwZxTv9OeDzCtaN78tLXm3jyo3WM+8t8LhzU5cBzRXqE8088jhFpHZrxHZlwNi9rFz2T43jrltH8auYK/jg7C6DZ158OxBKECWmqyr1vrWDj7lJevWkkXdodXOiwQ3w0j182+LBjPB7hxrFpjEzrwH3vrOSdb/MO7Ntf7eXFBRsZ3SuZO8/pw+jeyU3yPkzrsL+qhq9zCpg0ojvRkR7+/KMhdGoby9T5uaT3TGru8A5jCcKELFXl9x+u5aPVO7j/wv6c2rvjUR0/qGs7PrjjtEO2lVfW8PriLTz/RQ6TXljEyLQOPP3jIRzf3lbYNcduUW4BFdVezuiXAvi+rPx6/ACuGJ7KCS3s+gPYKCYTorxe5eFZa3jhy41cO7oHN5/eKyjP2yY6ghvHpvHlr87itz8YSOb2vVz5wiJ27d0flOc3rdu8rHxiIj2M7nXomWmfzoktcsCEJQgTcrxe5YH3VvHPhZu5aWwav7v4xKB/uGKjIrh+TBov33AKu/ZVcNWL31BQUhHU1zCtzxfr8xndO5nYqIjmDqVRLEGYkOL1Kr96eyVvLN7K7WedwAMXDXD1m9fwHh2YNvkUthSWcfW0xRSVVbr2Wia8bdpdysbdpZzVr1Nzh9JoriYIERknIlkiki0i9wXY/7SILHdu60WkyG9fjd++WW7G2Ros3byHS5/7iuVbi47cuAWb+mUuM5fmcfe5fbnngn5Nclo+uncyL1ybTs6uEq54fiEPvreaP3+SxfQFG/kmt8D11zfhYV7WLgDOdK4/hALXLlKLSAQwBTgPyAOWiMgsVc2sbaOqd/u1vwMY6vcU5ao6xK34wo2q8vriLQzvkUT/4w6t57K7pILbXlvKzr0VXDPtG16/aRSDU1vWjM3GyNy+lz99ksX4wcdx5zknNOlrn943heevGcYTH63jg5XbKSqvoraix2OXDuKqkT2aNB4TeuatzyetYzw9Wshyoo3h5iimEUC2quYCiMgMYAKQWU/7ScBvXYwnrM1YspUH3l1NYmwkL18/guE9fEPmarzKXTOWUVRWxYvXpvPw+2u4eto3vHbTyBY3rb8h+6tquPvfy0mKi+axSwY3ywW9s/t35uz+nQFfV1dReRX3vrWCB95djSBc2QLHsZuWYX9VDQtzCkLub8TNLqauwFa/x3nOtsOISA8gDZjrtzlWRDJEZJGIXFLPcTc7bTLy8/ODFXfIyd5Vwu/eX8OItA4kx0dzzbRvWJjj6/p45tP1fJVdwKOXDOLcgZ154yejiI+O4Jpp37Bux95mjrzx/vRJFlk79/GHFjLz2eMROsRH89zVwzi7fyd+/e4qXv9mS3OHZVqohc7w1jND6PoDuJsgAn3Fq6/M5kRgpqrW+G3rrqrpwJXAMyLS+7AnU52qqumqmp6SEjr9esFUUV3DnW8sIy46kmcnDeXNW0bTtX0brntpMU/NzuJvc7O5YngqP0rvBkC3DnG8cfMoYiIj+NHzC/nzJ1ns2hf8IZx1C+Udi69zdvPigo1cM6pHi/uAxUT6Zm+f1S+FX7+7ijcWW5Iwh/siK5/YKA8jQ2yGvptdTHlAN7/HqcD2etpOBH7mv0FVtzv/5orIPHzXJ3KCH2Zo++PHWWR+t5dpk9Pp1DYWgBk3j+KaaYt59vNs+h+XyCMTBh1yTI/keP59yyge/SCTv32ezfNf5DJhyPGcM6AzBaUV7Cjez3fF++nXOZEbxqYd1QIm63bs5Zk5G5i/IZ/XbhrJ0O5Hnh367NwNLN28h94pCfTulECvjvHs219N1s59ZO3Yx5cb8klLjuf+8f2P7ofTRHxJYji3vrqUX7+7iqgIDz8cntrcYZkWQlWZu24Xo3uFzvDWWuJW7XwRiQTWA+cA24AlwJWquqZOu37AbCBNnWBEJAkoU9UKEekILAQm+F/gris9PV0zMjJceS/BUFhayRuLtzBr+XbuvaAf5w7sfMzP+cX6fCZPX8y1o3sclgSKy6r4+xc5XDWyO906xNX7HLn5Jbz01SbeWrqV/VVeADwCHeJj2F1SwaheHfjLxKF0dpJPfbJ3lfDMp+v576rvSIiOJDrSQ/u4KP5752kNfigW5hQw6YVFdGkXS2FpJRXV3kP2d23fhgFdErnngn6HXXxvafZX1XDTPzP4Omc3T/94CBOGBOxRNa1M1o59XPDM/BY7mEFEljq9NYfvc3NxFREZDzwDRADTVfUxEXkEyFDVWU6bh4FYVb3P77hTgX8AXnzdYM+o6rSGXqulJIhH3s9k2dY99O2USJ/OCfRIjmfuup288+02Kqq9tI+Losar/PeO0+ieXP9/3EdSVePlzD/OIz4mglm3jz3mbyZFZZVs3F3Kce1iSUmIITLCw8yleTz43mrioiP484+HcEbfw7vxdu3dz9OfruffS7Y6k8t68pPTerEyr5hrpy/mltN7cf/4AQFfc39VDRf+5UtqvMrsn59OdKSH7UXl5OSXkBgbRd/OCSTGRh3T+2pq5ZU1TH5pMUs372HKlUMZ51cI0LROz87dwFOfrOebX59zxC9azaHZEkRTagkJYk9pJemPfUqXdrGUV9ZQUOqbVBUT6eGyYV25fkwabaIiuOivX9IjOZ6ZPx1NTOT3+4/9P8u3cdeM5UybnM45A479bKQ+2bv28bPXlpG1cx/n9O/E8J5JDO2WRJ/OCby6aDNT5+dSVePlmlE9+dlZvUlOiDlw7P3vrGLGki3MvHU0w3sc3vf6h4/X8dy8HF67aSRjTji6OkotWUlFNddO+4ZV24qZek06Z/VvWddNTNOa8OwCEOE/PxvT3KEE1FCCsGJ9QfTp2p3UeJXnrhrGSantKSipYOPuUnqlJNDBb+TNU1eczM3/Wspj/117WNdQY6gqL3yZS6+UeNdnZZ7QKZH/3D6GP89Zz6eZO/ls3a5D9l80uAu/Gtcv4NjuX4/vz/z1+dz71ko+vOvQrqbM7Xv5x/xcrhieGlbJASAhJpKXbxjBpKmLuGvGMmbfffohVWZN67Fz735W5BVz7wX9mjuU78VKbRzBqrxiHnxvNaUV1UdsO3vNDrq2b8NgZ35BckIM6T07HJIcAM4/8ThuGpvGKws388HK+q7b129RbiGrt+3lprG98BzFBeTvKzYqgl+PH8Dce85k2YPnMf26dO45vy9v//RUplw1rN6JP4mxUTx5+Unk7i7ld+9nsnF3KVU1XqprvNz3zkqS4qJ44KLA3U+hrm1sFM9eOYyqGuWet1YEdVSXCR1zMncCcF4Qrjk2BzuDaED2rn1cM/0bisqqiI+J5L4L6x9FU1JRzfwNu7l6ZOOWsfzfC/vz7ZY9/O/Mlcxdu4v+XRLpf1xbBndtd8Rx/tMW5JIcH81lw5r+ImhSfPQhE8aOZGyfjlw7ugevLNzMG4u3EOERkuOj2bWvgmevHEr7uOaf0+CWtI7xPPg/A/n1u6t4+etN3DA27ZD9NV49qhFiJvTMydxJj+S4FreUaGNZgqjHtqJyrpm2mEiPh3P6d+LFL3P54fCunNApMWD7z9ftorLay7hBxzXq+aMiPEy5ahgP/WcNX+cU8M6ybQDERUfw3s/G0Ldz4NfJyS/h07W7uOucPiEzZO7hH5zIhCFd2bi7lE27S9lUUErXpDZcNDj8L+BOGtGNz9bu5ImP1zG2T0f6dk6koKSCKZ/n8No3m5kw5Hge+sGJJMTYRzHclFRUszCngGtHh+7a5/ZXGUBBia9mUcn+ambcMorObWM5+6l5PPjeGl7/yciAv+yP1+ygY0L0gRIXjdGlXRteuNZ3bWhPaSWZ3+3l9te/5b63VzLz1lMDdh9NW7CR6EgP14xuecPl6uPxCMN7JB3VzyZciAhPXH4S456Zz89nLOfcgZ2Z9mUu5VU1jDmhIzOX5rEwt4A/XTHEljkNM19k5VNZ4w3Z7iWwBEFpRTW//3AtkR7B4xEiPcKC7AK27SnnlRtGcOLxvusJ947rz4PvrWbWiu2HjW/fX1XD5+t2MWFI1+/dZZAUH82YEzry0A8Gcve/V/DqN5u5dnTPQ9oUlFTw9tI8Lh/WlY5+o4VMy5aSGMMTl5/ET17JIPO7vYwffBy/OK8fJ3RKYOnmQu7+9wp+PHUhN5/ei3vP7xdw4frc/BLW7djH+HrOumpHI4bqN9VwNCdzB0lxUSH9xajVJ4j9VTV8vHoH1V7F61WqvUpslIcpVw5jpN+qT1eO6M5bGVt57L9rObt/p0PG5y/YsJuyyhoubGT3UkMuGdKVd5dt58mP1nHugM4Hlrr0epUpn+dQUe3lxjp92ablO29gZ56/ehhd28cdUkl3eI8OfHTXafzffzP5xxe5oBw2b6SwtJKrX/yG7cX7efB/Bh72+/+uuJwbXs7glJ5J32tUnAm+qhovc9ft4ryBxwVM+KGi1SeI5IQYlj543hHbRXiERycM4pLnvuJPn6zn4YtPPLDvo9U7aBsbyahex77AvYjw2CWDOP/p+Tz43mpenJzOlsIy7nt7FQtzC7hsWP3XQUzLVt+kufiYSB6/7CQiPR7+MT+Xk1Lbc9FJvrZer/Lzfy9nd0klo3sl8+gHmSTHR3PJUN9Z7NbCMq58cRFbC8tZ+91eLh3atVHlTYy7lmwsZO/+6pDuXgIb5npUTu7WnqtGduflrzfx01eXsmvffqpqvHy6difnDuhMdGRwfpzdOsTxy/P78tm6XfzyrRWMe+ZLVm0r5vHLBvOnK04OymuYlufB/xnIsO7tuXfmCjbs3AfA3+ZmM399Pr+9eCAvXX8Ko3p14J63VjAvaxebdpcyceoiisuqeP0nI+mUGMPD72fakNoWYM7ancREeji9b2jP8bEEcZR++4MT+dW4fny2bhfn/ukLHv0gk+LyKi4IQveSv+vHpHFyajve+XYbo3p1YM4vTmfSiO7WxxzGoiM9PHfVcOKiI7nlX0v5cNV3PPPZei4b2pUrR3QnNiqCF65Np99xifz01W+54h8LKaus5o2bR3Fq747cd2F/VmwtOjAizjSfFVuLOLlbe+KiQ7uTxhLEUYqK8HDbmSfw0V2n0f+4tryycDNtoiIC1ik6FhEe4YVr0/nnDSOYft0pNhO3lTiuXSzPXTWMLYVl3Pbat/TplMD/XTrowBeDxNgoXr5+BJ3bxqAKM24efWAgxSVDujKkW3ue/HgdJY2Y2Gnck7u7lN4poTn3wV9op7dm1DslgRk3j2Lmt3lERYgrcxI6tY09UMLbtB4j0jrw8MUn8vd5Ofz96uGHfQtNSYzhv3eeRrVXadfm4GAJj0d4+OITuWTKVzw7N7vBiZ3GPYWllRSVVdE7JXSWFq2PJYhj4PHIgYV4jAmmq0f14KqR9XcpxtczsW5It/ZcPiyV6Qs28uNTupHWMfT/kwo1OfklAGFxBmFdTMa0UN/3etP/jutHTJSHiVMXsmJrUZCjMkeS6ySIXmFwBmEJwpgw06ltLG/eMppIj4cf/WMh/1luF62bUm5+KdERHlKTvv96Ly2FJQhjwtCALm2ZdfsYTu7WnrtmLOfJj9fZ8NcmkpNfSo/kuLAoxGgJwpgwlZwQw6s3juTKkd35+7wcpnye3dwhtQq5u0vConsJLEEYE9aiIz38/tLB/ODk4/nr3A1kbt/b3CGFtaoaL1sKysLiAjVYgjCmVXjk4hNpHxfNL95cTmW1t7nDCVtbCsuo9iq9LEEYY0JFUnw0j186mHU79vG3uRuaO5ywlZtfCoTHCCawBGFMq3HuwM5cPiyV5+blsDKviJKKat5YvIWLn13A8DTxrNYAABKcSURBVEfnUFRW2dwhhrzaIa69O4bHGYRNlDOmFXnoBwP5Kns3N/0zg9KKakora+iVEk9BaSUzl+Zx02m9mjvEkJabX0pyfDTt4qKO3DgE2BmEMa1IuzZRPHXFySgwfnAX3rntVD77xRkM696e17/ZcmDhIfP9hNMIJrAEYUyrM7ZPR5Y8cC5/vOJkhnVPQkS4elQPcneX8nVOQXOHF9Jy80vpFSbdS+ByghCRcSKSJSLZInJfgP1Pi8hy57ZeRIr89k0WkQ3ObbKbcRrT2o0f3IWkuCheXbS5uUMJWUVllRSUVtK7U/icQbh2DUJEIoApwHlAHrBERGapamZtG1W926/9HcBQ534H4LdAOqDAUufYPW7Fa0xrFhsVwRXp3Zi2YCM79+6ns1URPmo5tSOY7AyiUUYA2aqaq6qVwAxgQgPtJwFvOPcvAOaoaqGTFOYA41yM1ZhW78oR3anxKjMWbz2m56lppSU9wqlIXy03E0RXwP8vLc/ZdhgR6QGkAXOP5lgRuVlEMkQkIz8/PyhBG9Na9ewYz2l9OvLG4i1U1/gm0+2vquH1b7bw8eodDR5bXF7Fm0u2ctWLi+j3m4/4YOX2pgi5RcndXUqkR+jWIfSL9NVyc5hroEpV9X21mAjMVNWaozlWVacCUwHS09Nb59cWY4LoqpE9uPXVpXy0egd7yip57vMcduzdT4RHeOWGEYw54dA1lksrqnng3VV8uGoHlTVeeiTH0SM5jl+/s4rhPZJa1UqIufkldE+OIyoifMb+uPlO8gD/1XRSgfq+VkzkYPfS0R5rjAmScwd04ri2sdzxxjIe+s8aUpPaMG1yOr1T4vnZ69+ypaDsQNvSimquf2kJ76/8jitHdue9n41h3j1nMm3yKVR7lXvfWtmqKsiG2wgmcDdBLAH6iEiaiETjSwKz6jYSkX5AErDQb/Ns4HwRSRKRJOB8Z5sxxkWRER5+eX5fzuibwqs3juStW0dzzoDOTL0mHa9X+ckrzgS7imquf3kJS7fs4ZkfD+Hhi09kSLf2iAg9O8bzwEUDWJC9m1cWbmrut9Qkqmu8bC4oC6sRTOBiF5OqVovI7fj+Y48ApqvqGhF5BMhQ1dpkMQmYoX4zdFS1UEQexZdkAB5R1UK3YjXGHHRFejeuqLOUbs+O8Tx75TCue2kxd/97OUXlVWRsKuSZiUP5wcnHH/YcV47ozqeZO3n8o3WM7ZPCCZ3C65t1XXl7yqms8YZNiY1aEi4zJ9PT0zUjI6O5wzAmrL34ZS7/99+1eASe/vEQJgwJOO4EgF1793PBM/NJTYrjndtODau++brmrtvJDS9nMPPW0aT37NDc4RwVEVmqqumB9lktJmNMo904No1qr9I7JYHzBnZusG2ntrE8dulgbnvtW6Z8ns3Pz+3bRFE2vYNVXMPrDMIShDGm0USEW8/o3ej24wd3YcKQ43l2bjbn9O/M4NR2LkbXfJZu3kPntjF0iI9u7lCCKnzP+YwxLcIjFw8iOcG3WNH+qpojHxBi9lfVMC8rn3MHNHxGFYosQRhjXNUuLoonLz+JDbtK+POc9c0dTtAt2LCb8qoaLjjxuOYOJegsQRhjXHdmv05MGtGdF77MZcmm8BqQOHvNDhJjIxnVK7m5Qwk6uwZhjGkSvrkR+dz1xjIeu3QwZ/ZLQSRQ0YT6bSsq57uicvaUVVFUVklJRTUJMZF0iI8mKT6a1KQ2dEoMfqHBDTv3cccby/jrpKH07Zx4YHt1jZdP1+7k7P6diI4Mv+/bliCMMU0iISaSZycN484Zy7j+5SWMSOvAfRf2Z1j3pAaPq6rx8smanfxr0SYW5TZ89hHpEV6cnM6Z/ToFM3T+MDuLdTv28dTsLKZee3BEaMbmPewpqwrL7iWwBGGMaUInd2vPnLvPYMaSLfz1sw1c9tzX9D8ukaoaL+WVNZRV1RAV4aFTYgwpiTEkxUWzIHs3+fsqSE1qw70X9GNw13a0j4siKS6a+JhISvZXU1hWyZ6ySv7wcRZ3vLGMWbePJa1jcGY1r9haxJzMnXTvEMcnmTtZva2YQV19o7Fmr9lBdKSHM/qmBOW1WhqbKGeMaRalFdW89NVGvt1SRJuoCNpERxAXHUFFlZfdJRXkl1Swe18F/bu05ZpRPTi9bwoRnoa7pLYWlnHxswtITojh3dtOJTH22NeGvnb6YlblFfHhXadxwdPzGZHWgRcnn4KqMvbJz+l/XCLTrjvlmF+nudhEOWNMixMfE8ntZ/cJ6nN26xDHlKuGcc00X0mQqdek4zlCUmnI4o2FzF+fz/0X9qdLuzbcfHovnvpkPSu2FhHhEbYVlXPnOScE8R20LOF3VcUY06qd2rsjD140gE/X7uKZT7//sFpV5anZWaQkxnDt6J4ATD61J+3jonjm0/V8krkTjxCW8x9qWYIwxoSdyaf25PJhqfzt8+xDSpQfjQXZu1m8qZDbzzqBNtERACTGRnHz6b34PCufVxdtJr1HB5ITYoIZeotiCcIYE3ZEhHsv6EeECP9cuOmoj/d6lac+Wc/x7WKZOOLQyraTR/ekQ3w0haWVnH9i+J49gCUIY0yYOq5dLBed1IV/L9nKvv1VjT5OVXngvdWs2FrE3ef1JSYy4pD98TGR3HZmb6IiJGyHt9ayBGGMCVvXj0mjpKKamUvzGtVeVfnd+5m8sXgLt53Zmx8OTw3Y7saxaXx139lhtf50IJYgjDFha0i39gzvkcTLX2+i5gjLn6oqj3+0jpe/3sSNY9O494J+9c70FhFXZmy3NJYgjDFh7YYxaWwuKGPuul0Ntnt6znqmzs/lmlE9+M1FA466DEg4sgRhjAlrF5zYmePbxTJ9wcZ627yZsZW/zs3mR+mp/O7iEy05OCxBGGPCWmSEh8mn9mRhbgGZ2/cetj9jUyEPvLuKsSd05PeXDj6miXXhxhKEMSbsTTylO22iIvjznPUUlx0c0ZS3p4xb/rWU1KQ4plw5jMgwXjf7+7CfhjEm7LWLi+LWM3rz6dqdjH7iM373/hqyduzjJ68spbLGywvXptMu7tjrNoUbK9ZnjGk11n63lxe+zGXW8u1UexWPwEvXjwjbaqyN0VCxPksQxphW57vicl5btIU+nROYMKRrc4fTrBpKEK52MYnIOBHJEpFsEbmvnjY/EpFMEVkjIq/7ba8RkeXObZabcRpjWpcu7dpwzwX9Wn1yOBLXyn2LSAQwBTgPyAOWiMgsVc30a9MHuB8Yo6p7RMR/GahyVR3iVnzGGGMa5uYZxAggW1VzVbUSmAFMqNPmJ8AUVd0DoKoNz2QxxhjTZNxMEF2BrX6P85xt/voCfUXkKxFZJCLj/PbFikiGs/2SQC8gIjc7bTLy8/ODG70xxrRybq4oF2i2Sd0r4pFAH+BMIBX4UkQGqWoR0F1Vt4tIL2CuiKxS1ZxDnkx1KjAVfBepg/0GjDGmNXPzDCIP8C+kngpsD9DmP6papaobgSx8CQNV3e78mwvMA4a6GKsxxpg63EwQS4A+IpImItHARKDuaKT3gLMARKQjvi6nXBFJEpEYv+1jgEyMMcY0Gde6mFS1WkRuB2YDEcB0VV0jIo8AGao6y9l3vohkAjXAvapaICKnAv8QES++JPaE/+gnY4wx7rOJcsYY04q1ipnUIpIPbA6wqx1QfIRt/o8D3a/9tyOw+3uGGCiOxuy3+A/d9n3fw5Hib6hNQ/HWfXyk+xb/0bc50t9Qfe8nmPE3FN+R9rf0z3APVQ1ca0RVw/oGTD3SNv/Hge77/ZsRzDgas9/iP2zb93oPR4r/aN7D0cYfjN+BxV//tvreTzDjb8x7CPXPcKBba6jm+n4jtr1/hPuBniMYcTRmv8XfNPE31KaheOs+bsz978Pir39bfe8nmPE35jnC4TNwiLDpYmoKIpKh9fTVhYJQjx9C/z1Y/M3L4j86reEMIpimNncAxyjU44fQfw8Wf/Oy+I+CnUEYY4wJyM4gjDHGBGQJwhhjTECtNkGIyHQR2SUiq7/HscNFZJWzENJfRUT89t3hLJK0RkT+ENyoD4kh6PGLyMMiss1voabxwY/8QAyu/Pyd/feIiDplWlzj0u/gURFZ6fz8PxGR44Mf+YEY3Ij/jyKyznkP74pI++BHfiAGN+K/wvnsekXElYvBxxJ3Pc83WUQ2OLfJftsb/Jw0yvcdUxvqN+B0YBiw+nscuxgYja9i7UfAhc72s4BPgRjncacQi/9h4J5Q/fk7+7rhK+GyGegYau8BaOvX5k7g+RCL/3wg0rn/JPBkiMU/AOiHr0BoekuK24mpZ51tHYBc598k535SQ+/xaG6t9gxCVecDhf7bRKS3iHwsIktF5EsR6V/3OBHpgu9DvFB9v4VXgNr1Kn6Kr25UhfMari2A5FL8TcbF+J8GfsXhpeWDzo33oKp7/ZrG4+L7cCn+T1S12mm6CF8V51CKf62qZrkV87HEXY8LgDmqWqi+hdfmAOOC9TlvtQmiHlOBO1R1OHAP8FyANl3xlSmv5b8QUl/gNBH5RkS+EJFTXI32cMcaP8DtTvfAdBFJci/UgI4pfhG5GNimqivcDrQBx/w7EJHHRGQrcBXwkIuxBhKMv6FaN+D75tqUghl/U2pM3IHUtzBbUN6jmwsGhRQRSQBOBd7y66qLCdQ0wLbab3mR+E7zRgGnAG+KSC8ng7sqSPH/HXjUefwo8Cd8H3LXHWv8IhIHPICvi6NZBOl3gKo+ADwgIvcDtwO/DXKoAQUrfue5HgCqgdeCGWNDghl/U2oobhG5HrjL2XYC8KGIVAIbVfVS6n8vQXmPliAO8gBFqjrEf6OIRABLnYez8P0n6n/a7L8QUh7wjpMQFouvXHlHoCnWQz3m+FV1p99xLwAfuBlwHccaf28gDVjhfMhSgW9FZISq7nA59lrB+Bvy9zrwX5ooQRCk+J0Lpf8DnNMUX478BPvn31QCxg2gqi8BLwGIyDzgOlXd5NckD9+KnLVS8V2ryCMY79GNizChcgN64nehCPgauMK5L8DJ9Ry3BN9ZQu3Fn/HO9luBR5z7ffGd+kkIxd/Fr83dwIxQ+vnXabMJly9Su/Q76OPX5g5gZojFPw7f4l4pbv/s3fwbwsWL1N83buq/SL0RX89FknO/Q2PeY6PibIpfYku8AW8A3wFV+LLtjfi+gX4MrHD+yB+q59h0YDWQAzzLwRnp0cCrzr5vgbNDLP5/AauAlfi+aXUJpfjrtNmE+6OY3PgdvO1sX4mvuFrXEIs/G98Xo+XOzc1RWG7Ef6nzXBXATmB2S4mbAAnC2X6D83PPBq4/ms/JkW5WasMYY0xANorJGGNMQJYgjDHGBGQJwhhjTECWIIwxxgRkCcIYY0xAliBMWBORkiZ+vRdFZGCQnqtGfFVdV4vI+0eqjCoi7UXktmC8tjFgK8qZMCciJaqaEMTni9SDxehc5R+7iPwTWK+qjzXQvifwgaoOaor4TPizMwjT6ohIioi8LSJLnNsYZ/sIEflaRJY5//Zztl8nIm+JyPvAJyJypojME5GZ4lv74LXaWvvO9nTnfolTeG+FiCwSkc7O9t7O4yUi8kgjz3IWcrAoYYKIfCYi34qv3v8Ep80TQG/nrOOPTtt7nddZKSK/C+KP0bQCliBMa/QX4GlVPQW4HHjR2b4OOF1Vh+Krovp7v2NGA5NV9Wzn8VDg58BAoBcwJsDrxAOLVPVkYD7wE7/X/4vz+kesj+PUEjoH3+x2gP3Apao6DN8aJH9yEtR9QI6qDlHVe0XkfKAPMAIYAgwXkdOP9HrG1LJifaY1OhcY6Fc5s62IJALtgH+KSB98lS+j/I6Zo6r+NfwXq2oegIgsx1dbZ0Gd16nkYMHDpcB5zv3RHKzN/zrwVD1xtvF77qX4av2Dr7bO753/7L34ziw6Bzj+fOe2zHmcgC9hzK/n9Yw5hCUI0xp5gNGqWu6/UUT+Bnyuqpc6/fnz/HaX1nmOCr/7NQT+LFXpwYt89bVpSLmqDhGRdvgSzc+Av+JbJyIFGK6qVSKyCYgNcLwAj6vqP47ydY0BrIvJtE6f4FtnAQARqS2z3A7Y5ty/zsXXX4Svawtg4pEaq2oxvuVH7xGRKHxx7nKSw1lAD6fpPiDR79DZwA3OegOISFcR6RSk92BaAUsQJtzFiUie3+0X+P6zTXcu3GbiK9MO8AfgcRH5CohwMaafA78QkcVAF6D4SAeo6jJ8lT4n4luEJ11EMvCdTaxz2hQAXznDYv+oqp/g68JaKCKrgJkcmkCMaZANczWmiTmr35WrqorIRGCSqk440nHGNDW7BmFM0xsOPOuMPCqiiZZ1NeZo2RmEMcaYgOwahDHGmIAsQRhjjAnIEoQxxpiALEEYY4wJyBKEMcaYgP4ffKj54tc+Y/oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.437175</td>\n",
       "      <td>0.282718</td>\n",
       "      <td>0.864618</td>\n",
       "      <td>0.914374</td>\n",
       "      <td>02:58</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.387293</td>\n",
       "      <td>0.260588</td>\n",
       "      <td>0.876972</td>\n",
       "      <td>0.922479</td>\n",
       "      <td>02:56</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas1_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.433597</td>\n",
       "      <td>0.279311</td>\n",
       "      <td>0.875412</td>\n",
       "      <td>0.920975</td>\n",
       "      <td>02:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.366492</td>\n",
       "      <td>0.256188</td>\n",
       "      <td>0.887030</td>\n",
       "      <td>0.929599</td>\n",
       "      <td>02:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas1_sp15');\n",
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas2_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.352552</td>\n",
       "      <td>0.237124</td>\n",
       "      <td>0.947416</td>\n",
       "      <td>0.968906</td>\n",
       "      <td>03:27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.279644</td>\n",
       "      <td>0.209162</td>\n",
       "      <td>0.946463</td>\n",
       "      <td>0.968271</td>\n",
       "      <td>03:34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas2_sp15');\n",
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas3_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.283462</td>\n",
       "      <td>0.184935</td>\n",
       "      <td>0.924007</td>\n",
       "      <td>0.953850</td>\n",
       "      <td>04:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.217490</td>\n",
       "      <td>0.181429</td>\n",
       "      <td>0.934368</td>\n",
       "      <td>0.960415</td>\n",
       "      <td>04:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15');\n",
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas4_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.219500</td>\n",
       "      <td>0.176016</td>\n",
       "      <td>0.935062</td>\n",
       "      <td>0.960744</td>\n",
       "      <td>04:49</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas5_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas5_sp15')\n",
    "learn_c.save(f'{lang}clas_sp15')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_sp15');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_sp15')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Training (Loss = FlattenedLoss of LabelSmoothing CrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12.5 s, sys: 200 ms, total: 12.7 s\n",
      "Wall time: 12.7 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15', bs=bs, num_workers=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=0.5, \n",
    "                                  metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.loss_func = FlattenedLoss(LabelSmoothingCrossEntropy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of LabelSmoothingCrossEntropy()"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xUddb48c9JJyGQSoAEkhBCEwEhIGBDbNjAsrqwdlfRVXTV1X30cVdd/bl9H3etK66ra0VFXVFRsGEDhNA7hB5qKpAEUs/vj5ngECaFkJuZSc779ZoXc+9879xzSWZO7reKqmKMMcbUFeTrAIwxxvgnSxDGGGO8sgRhjDHGK0sQxhhjvLIEYYwxxqsQXwfQUhISEjQtLc3XYRhjTEBZtGhRvqomenutzSSItLQ0srOzfR2GMcYEFBHZWt9rVsVkjDHGK0sQxhhjvLIEYYwxxitLEMYYY7yyBGGMMcYrRxOEiIwTkXUikiMi93t5vaeIfCUiS0RkuYhc4N5/jogsEpEV7n/HOhmnMcaYoznWzVVEgoFngHOAXGChiMxQ1dUexX4DvK2qz4nIAGAmkAbkAxer6k4RGQjMApKditUYY8zRnBwHMQLIUdVNACIyDZgAeCYIBTq5n3cGdgKo6hKPMquACBEJV9VyB+M1fuz7nHzW7NpPWnwU6YlR9IiNJCzEakiNcZKTCSIZ2O6xnQucXKfMI8BsEbkDiALO9vI+lwNLvCUHEZkMTAbo2bNnC4Rs/E1ldQ1//nQtL3y7+Yj9wUHCjaek8eCFA3wUmTFtn5MJQrzsq7s60STgZVX9m4iMAl4VkYGqWgMgIicAfwLO9XYCVZ0KTAXIysqylY/amF37DjLljSUs2lrENSNTuWNsb3KLD7I5r5Qv1u7hhW83MzojgTP7dfF1qMa0SU4miFygh8d2Cu4qJA8/B8YBqOo8EYkAEoC9IpICvA9cq6obHYzT+KEFmwu59bVFlFdW8+Skkxg/uDsAXTpFMLRnLBcN7kbO3u+4/73lzL7rDDpHhvo4YmPaHicrcRcCmSKSLiJhwERgRp0y24CzAESkPxAB5IlIDPAx8ICqfu9gjMYP7Sur5PY3FtO5Qygz7jj1cHLwFB4SzN+uGEJ+SQWPfrTay7sYY46XYwlCVauAKbh6IK3B1VtplYg8KiLj3cV+BdwsIsuAN4Hr1bVI9hSgN/BbEVnqflg9QitbkbuPLfmlrX7ex2euprC0gqcmnURGYsd6y52Y0pnbxmTw7uJcPl+9pxUjNKZ9ENf3ceDLyspSm8215WzJL2XcP74hNDiIl28YwbDU2FY573cb8rn6xR/4xZgM/mdcv0bLV1TVMP7p7ygoreCzu08nJjKsFaI0pu0QkUWqmuXtNesnaI6iqtz/3nJCg4KIiwrjmhd/YN7GgkaP+WZ9Hnv3H2r2ecsqqrj/veX0Sojil2dlNumYsJAg/nrFYIpKK7hv+nJqatrGHzzG+ANLEOYoby7YzvxNhfzvhf1555ZRJMd04PqXFjBn3V6v5VWVxz9ew7X/XsDIP3zBtf9ewH+X7KCsouqosnsPHGLOur08N2cjj3+8mtmrdlNS7ir311nryS06yB8vH0REaHCT4x2Y3JkHL+zPZ6v38NfZ65p30caYo7SZBYNMy9i17yB/mLmGUb3imTi8ByLCtMkjufbfC7j5lWx+N34gE4f3ICjI1YtZVfnTp+v413ebmTSiJwkdw3hv8Q7uemspYcFBRIUHExwkBAcJFVU1FJVVHj5XWHAQL3y7mZAgYUiPGBZtK+LqkT0ZkR53zHFfPzqN9XtKeHbORjKTOnLpSSkt9n9iTHtlbRDmMFXlpv9k8/3GfGbddTqp8VGHX9t3sJJbX13EvE0FDErpzMMXn8DQnjH8bfZ6nv4qh6tH9uSxCQMREWpqlIVbCvly3V4OVVRTVaNU1ygiQmaXjgzo3on+XTsRERbE4q3FfLMhj2835FFVrbxz6yiiI5rXZbWyuoZrXvyBxduKefPmka3WbmJMIGuoDcIShDnsg6U7+OW0pfzmwv7cdFqvo15XVWYs28nvZ65hz/5yhvaMYfG2YiYO78HvLz3x8F2FLxWVVnDJs99TWl7FB1NOJTmmg69DMsavWSO1aVTO3gM8+P5KhvSI4YZT0r2WEREmDEnmy1+N4bYxGazcsZ8rs1L8JjkAxEaF8eJ1WZRX1nDrq4s4VFnt65CMCVh2BxHgSsuriAo/vqak4rIKJjzzPaXl1cyYcgrdm/hX98GKaiJCgxDxj+Tg6fPVe7jplWx+MiyFv/xkkF/GaIw/sDuINuqrtXsZ8uhsPli6o9Gyq3fu55ZXs7noqW+ZvWo3tX8YVFbXcPsbi9lVfIjnrxnW5OQA0CEs2G+/eM8ekMSdZ2UyfVEur83f6utwjAlI1ospQNXUKH/6dC2V1cpv3l/J0J6x9IiLPKrc+j0H+Pvn65m5YjfRESEkdgxn8quLOC0zgYcvHsBr87fxfU4Bf71icJtr1L3rrExW7tjH7z5czYDunRiW2rTeUdU1Ss7eEvokdWzxBKjqarAPCba/zYz/syqmAPXx8l3c/sZi7j23D89/vYk+XaN5a/LIw188NTXKX2ev47mvNxIVFsKNp6Tx81N7ERkezGvzt/LEZ+spKa+iRuHm09Lb7LTZ+w5WMuHp7yitqOb1m06mT1J0o8f8fuYapn6zicwuHblmVCqXnpTc7J5VhyqrufmVbJZsK6aiqoaK6hpE4ObTevHA+f389g7MtB/Wi6mNqa5Rzvv7NwDMuut0Plq+k19OW8ovz8rk7nP6UFpexV1vLeWz1Xu4MiuFB87vT2zUkVNQFJSU8/fPN1BRVcPvLzuRYD9pZHbCut0HuOKfcykpr+KSk5K566w+9Iw/+m4LXFVxFz/9HaMz4ikuq2TFjn1EhQVz6dBkrhuVRmYTEkwtVeW+6ct5d3Euk0b0pFNEKGEhQWzKK+Gj5bu499w+TBnbtBHjxjiloQRhVUx+Zs2u/aTEdmjwL9YPl+0kZ28JT//sJIKDXD2Lvl6Xx1NfbqBXYhTPzdnI+j0HePjiAVw/Os3rX6nxHcN57JKBTl6K3+jbNZo5953JP7/eyH/mbmHG0p1MHNGD/xnX74j/55oa5cH/riCmQyhPTTqJzh1CWZa7j1fmbeHt7Fxem7+N0RnxXDsqjbP7d2m0muithduZviiXO8f25p5z+x5xntDgIP46ez2xUWFcdXKqU5duzHGxOwg/8tHyndz55hJOTInhrckjvU43UVVdwzlPfEN4SBAz7zztcPfSA4cqufDJ79hWWEZ0RAhP/2woZ/RJbO1L8Ht79h/iqS838OaC7fRNiublG4fTJToCgDd+2Mb/vr+C/7tyMJcNPXIkdkFJOW9lb+f1+dvYUXyQzC4d+dd1WUcMJvS0csc+LntuLienx/HyDSOOukOrrK5h8ivZzFmfxzM/G8oFJ3Zz5oKNaYT1YgoAX63dy13TltIrsSPLthfzv++vwFvyfm/JDjbnl3LPOX2OGHsQHRHKMz8byrkDknj/ttGWHOqR1CmC/3fJibx4XRab80v5yXPz2JJfSn5JOX/8ZA0je8Vx6UnJRx0X3zGc28b05uv7xvDsVUPJKynn0mfnsmhr0VFl95VVcutri4iPCuMfE0/yWn0XGhzEs1cNY2jPWO6atpTnv954eE4qY/yF3UH4gXkbC7j+pQVkJnXkjZtH8tJ3W3ji8/VHjWjeve8QP/nnXOKiwvjg9lOsgfM4LdlWxI0vLyQ4SOjfrRPzNxXwyS9Pp3eX+tegqLUpr4QbXl7I7n2HeOKnQ7jgxG7sKD7Ipyt38072djbmlfDWLaMY2rPhnmH7yiqZ8uZivt2QT6eIEK4dlcb1p6SR0DG8pS7TmAZZI7UfW7q9mKtemE/3mA68dcso4qLCqKlRbn9jMbNW7eblG0bQv1snnpuzkdd+2Iqq8p8bRjC6d4KvQ28TNuaVcO2LC9hRfJA7xvbmVx5tBY0pKCln8quLWLS1iH5do1m7+wAA/bpGc8fYTC4c1PRqo2Xbi/nn1xv5dNVuwkOC+Ne1wzk1037GpnFv/LCN8qrqemdAaIwlCD9UXlXN819v4pmvckjqFME7t44iqVPE4ddLy6u4/Lm55BYdpLpGqaiu4fKhydwxNtPreAfTfHv2H+LDZTu5emTqMU0zDq5urL/7cBXr95Rwdv8kxg3sSnqC93aJptiYV8Ktry6isLSCT355Gl08fieM8Wbi1HlU1yjv3Dq6Wcf7LEGIyDjgH0Aw8C9V/WOd13sC/wFi3GXuV9WZ7tceAH4OVAN3quqshs4VSAni2w15PPTBKjbnl3LhoG48fNEAr18E2wvLuO7fCzgxpTN3nd3nuL54TODYsOcA45/+nsE9OvP6TSPbdBdkc/zG/f0besRF8sK1Xr/jG+WTbq4iEgw8A5wD5AILRWSGqnquMP8bXGtVPyciA4CZQJr7+UTgBKA78LmI9FHVgJ55rayiit+8v5L3luwgLT6SV24cwekNNCb3iIvky3vHtF6Axi9kJkXz6IQTuG/6cv7xxQbuOaePr0MyfqyorIJBKZ0deW8nx0GMAHJUdROAiEwDJgCeCUKBTu7nnYGd7ucTgGmqWg5sFpEc9/vNczBeR23KK+HW1xaRs7eEO8f25rYzex9zdYZpP67I6sG8TQU89eUGTk6P4xR3m9P+Q64Flzo1c2S3aVtUlaLSyqMGwrYUJxNEMrDdYzsXOLlOmUeA2SJyBxAFnO1x7Pw6xx7V91BEJgOTAXr27NkiQTvh05W7ufedZYSFBPHKjSdb46NpkscmDGTZ9mJ+8doiunSKYPe+Q5SUVxEeEsSrPz+5WSvvmbalrKKaiuoaYiMDL0F4qzit2+AxCXhZVf8mIqOAV0VkYBOPRVWnAlPB1QZxnPG2mF37DrJqx35W7tzH8tx9fLl2L4NTOvPs1cNsARvTZFHhIfzz6mH8fuYawkOCOS0zgW6dI3j9h23c+eYSPr7zVOKtO2y7VlRWAUBspDN3lE4miFygh8d2Cj9WIdX6OTAOQFXniUgEkNDEY1tEYWkFT3+Zw5XDU+jXtVPjBzTiDzPX8Pw3mwAQgfSEKG45vRf3nNuH8BCrUjLHJjMpmpduGHHEvtEZCVz27FzueXsZL10/3G8WazKtr9i9xntMAN5BLAQyRSQd2IGr0flndcpsA84CXhaR/kAEkAfMAN4Qkf/D1UidCSxwIshgEaYt3Ma+g5X87crBx/Veczfm8/w3m5gwpDvXjEylf7dOx72YjzF1DUzuzG8vHsBv/7uSf36zkdvG9PZ1SMZHCktddxBxDrVBODbVhqpWAVOAWcAaXL2VVonIoyIy3l3sV8DNIrIMeBO4Xl1WAW/jatD+FLjdqR5MnSNDuTKrBzOW7WDP/kPNfp/S8ip+PX05afGR/PGyQWSlxVlyMI65+uSeXDioG3+bvZ4Fmwt9HY7xEaermBydi0lVZ6pqH1XNUNXH3fseUtUZ7uerVfUUVR2sqkNUdbbHsY+7j+urqp84GeeNp6RTXaO8PHdLs9/jj5+sZUfxQf5yxWA6hFlVknGWiPDHy04kJbYDU95YzOb8Ul+HZHzA6Somm6wP6BkfybiBXXl9/lZKmzFh2tycfF6dv5UbRqczPM16lpjWER0RyvPXDKOqRvnp8/PYsOeAr0Myraz2DiKmQwDeQQSSm07rxf5DVbyTvb3xwh5Ky6v49bvLSU+I4r7zmj6PjzEtoV/XTkybPBIFfjp1Pqt27vN1SKYVFZdVEh0R4tgStpYg3Ib2jGVYaiwvfr+Z6pofe8zuP1TJx8t3UVVd4/W4Z+fksKP4IH/+ySCrWjI+0ScpmrdvGUVESBCTps7n05W7+Xz1Ht5auI1nvsrhtflb2Xug+e1rxn8VllY41kANtqLcEW4+LZ1bX3PNonrBid34Ys0eHnx/Jbv3H+KxSwZyzcgjV/46cKiSV+Zt5fyBXa1qyfhUekIUb90yip/9az63vrboqNd/+8FKRqTFcdGgbpwzoCtdO9skgG1BUVmFY+0PYAniCOcM6EpqfCTPzdnI7FW7+e/SnfRNiiY2KoxnvszhimEpR0yP8cYP2zhwqIpbz8jwYdTGuPSIi+TDKaeydHsxcVFhxHcMJz4qjG2FZXy8fBcfr9jFbz9YxW8/WEXvLh05tXcCozPiOaNvoo3RCVDFZZXEd7QE0SqCg4Sfn5rOQx+sYu3u/dx1dia3jelN9tZCfvbCD7zxwzZuPNU153p5VTUvfreZU3rHMyglxseRG+MSExnGmL5djtjXJymaPudEc/c5fVi/5wBfr8vju5x83lq4nZfnbuHs/l144dosW4AqABWVVTRpgavmsgRRx5VZPSgsrWDcwK6HR1aPzkhgVK94np2Tw8QRPYgMC+G/S3aw90D5cQ+uM6Y19UmKpk9SNDef3ouKqhqem7ORJz5fz+zVezjvhK6+Ds8co+KySmIcGgMB1kh9lIjQYO46u89R02786tw+5JdU8Mq8rVTXKM9/vYmByZ041VZ2MwEqLCSI287MoG9SNI9+uJqyClsTO5BUVNVQUl5FnINtEJYgmigrLY4z+iTy/NcbeX/JDjbll3LrGRl2W24CWmhwEI9dMpAdxQd5+sscX4djjkFx7RgIB3sxWYI4Bvec04eiskoeeG85qfGRnD+w6WsOG+OvRqTHcdnQZF74dhM5e0t8HY5poiL3KGqnptkASxDHZHCPGM7un0RltTL59F62FKRpMx44vz8RocE8PGMlbWWd+rbux3mY7A7Cb/z2ov5cPzqNy4em+DoUY1pMYnQ4953Xl+9zCvh4xS5fh2Oa4HAVk91B+I/U+CgeGX+CLRdq2pyrTk6lX9do/jprHZX1zBxg/EdhqauKycmR1JYgjDGAaxzQfef1ZUtBGdMX5fo6HNMIq2IyxrSqsf26MLRnDP/4fAOHKh1ZgsW0kOKyCiJCgxytzbAEYYw5TES477x+7N5/iNfmb/V1OKYBRWWVjt49gCUIY0wdozLiOS0zgWfnbKSkGeujmNZRVOrsRH3gcIIQkXEisk5EckTkfi+vPyEiS92P9SJS7PHan0VklYisEZEnxUakGdNq7j23L4WlFfz7u82+DsXUo6isgrgo53owgYMJQkSCgWeA84EBwCQRGeBZRlXvdi81OgR4CnjPfexo4BRgEDAQGA6c4VSsxpgjDe4Rw7gTuvLCN5soKq3wdTjGC9c8TIF7BzECyFHVTapaAUwDJjRQfhLwpvu5AhFAGBAOhAJ7HIzVGFPHr87tQ2lFFbe8togDhyp9HY6po6iswtFR1OBsgkgGPNfvzHXvO4qIpALpwJcAqjoP+ArY5X7MUtU1Xo6bLCLZIpKdl5fXwuEb075lJkXz94knsXhrEVf96we7k/Aj1TXKvoOB3Ujtrc2gvjH8E4HpqloNICK9gf5ACq6kMlZETj/qzVSnqmqWqmYlJia2UNjGmFrjB3fn+WuGsXb3Aa58fh579tvSpf5g/8FKapSArmLKBXp4bKcAO+spO5Efq5cALgXmq2qJqpYAnwAjHYnSGNOgs/on8fINw9lZfJAr/jmP7YVlvg6p3asdJBewjdTAQiBTRNJFJAxXEphRt5CI9AVigXkeu7cBZ4hIiIiE4mqgPqqKyRjTOkZnJPD6zSPZd7CSiVPnW5LwsdqZXAP2DkJVq4ApwCxcX+5vq+oqEXlURMZ7FJ0ETNMjp5CcDmwEVgDLgGWq+qFTsRpjGjekRwyv33QyJeVVTJw6n20FliR8pbgVptkAh5ccVdWZwMw6+x6qs/2Il+OqgVucjM0Yc+wGJnfm9ZtO5uoXf2Di1Hm8OXkkqfFRvg6r3WmNtSDARlIbY47RwOTOvHHTSA5WVvPT5+eTX1Lu65DandoeZQFbxWSMabsGdO/Ef24cwe79h3jXZn5tdUVlFQQHCZ0iHK0EsgRhjGmeQSkxnNQzhvcW77BV6FqZa6K+UJyegcgShDGm2S4bmsK6PQdYvWu/r0NpV4rLnJ+oDyxBGGOOw8WDuhEaLLy3eIevQ2lXWmOaDbAEYYw5DjGRYYzt14UPlu6gypYpbTVFpc5P1AeWIIwxx+myoSnkl1Tw7YZ8X4fSbhSVVRBnCcIY4+/O7NuFmMhQ3lti1UytQVVdU307PM0GWIIwxhynsJAgxg/uzuxVu9lv04I7rqyimorqGsdHUYMlCGNMC7hsaArlVTV8smKXr0Np8wpLa6fZsDsIY0wAGJzSmV4JUbxrvZkcV9xKE/WBJQhjTAsQES4bmsyCzYXc8/ZSNuWV+DqkNuvHqb6dTxDOjtM2xrQbN53Wi6KySl7/YSv/XbKD8YO7M2VsJr27dPR1aG1KUZlVMRljAkxEaDC/vWgA3/3PWG4+rRezVu3hgie/ZcHmQl+H1qZYFZMxJmAldAzngQv6882vzyQlpgOTX822KqcWVNtIHdPB7iCMMQEqMTqcl24YTpAIN7688PAXmzk+xWUVREeEEBLs/Ne3JQhjjGNS46N44dosdu47xM2vZHOostrXIQW8jXmlJMd0aJVzOZogRGSciKwTkRwRud/L60+IyFL3Y72IFHu81lNEZovIGhFZLSJpTsZqjHHGsNRYnrhyCIu2FvHr6ct9HU5AO1RZzYIthZzaO6FVzudYLyYRCQaeAc4BcoGFIjJDVVfXllHVuz3K3wGc5PEWrwCPq+pnItIRsJnAjAlQFw7qxsa8PvzfZ+u55KTujO2X5OuQAtKCzYVUVNVwWp/EVjmfk3cQI4AcVd2kqhXANGBCA+UnAW8CiMgAIERVPwNQ1RJVtRXSjQlgt56RQa/EKB77aA3lVVbV1BzfbsgjLDiIEWlxrXI+JxNEMrDdYzvXve8oIpIKpANfunf1AYpF5D0RWSIif3HfkdQ9brKIZItIdl5eXguHb4xpSWEhQTx00QA255fy0vdbfB1OQPp2Qz7D02PpEHbU16EjnEwQ3tbCq29dwonAdFWt/bMiBDgNuBcYDvQCrj/qzVSnqmqWqmYlJrbOLZcxpvnG9O3C2f2TeOqLDezZf8jX4QSUvfsPsXb3AU7LbL3vOicTRC7Qw2M7BdhZT9mJuKuXPI5d4q6eqgL+Cwx1JEpjTKv67UX9qaxW/vTJWl+HElC+y3Gtt9FaDdTgbIJYCGSKSLqIhOFKAjPqFhKRvkAsMK/OsbEiUpsqxwKr6x5rjAk8qfFR3Hx6Ou8t2cGirTbKuqm+3ZBPfFQYA7p1arVzOpYg3H/5TwFmAWuAt1V1lYg8KiLjPYpOAqapqnocW42reukLEVmBq7rqBadiNca0rtvP7E3XThE8PGMV1TX11TybWqrKtxvyOTUzgaAgb7X3znB0sj5VnQnMrLPvoTrbj9Rz7GfAIMeCM8b4TGRYCA9c0I9fTlvKO9nbmTiip69D8mtrdx8gv6S8VauXwEZSG2N8ZPzg7gxPi+Uvs9ax76CtRNeQbze4emm2ZgM1WIIwxviIiPDwxSdQWFbBk19s8HU4fu3bDfn0SepI184RrXpeSxDGGJ8ZmNyZn2b14D9zt5Cz12Z89eZQZTU/bC7k1N6t35XfEoQxxqfuPa8vHcKCeeyj1Xj0VTFuC7fUTq/Ruu0PYAnCGONjCR3D+eVZmXy9Po8v1+71dTh+55v1ruk1Tk5vnek1PFmCMMb43HWj08hIjOJ3H662KcHrmLMujxHpcUSGtf4K0ZYgjDE+FxocxGOXDGRbYRlPfWkN1rV2FB9kw94SxvT1zVRCTUoQIpIhIuHu52NE5E4RiXE2NGNMezI6I4HLhiYz9ZtNbNhzwNfh+IU561xVbn6dIIB3gWoR6Q28iGvm1Tcci8oY0y49eEF/osJDePD9ldTYCGvmrMsjOaYDGYkdfXL+piaIGvfUGZcCf3cv9NPNubCMMe1RfMdwHji/Hwu2FDJ9Ua6vw/Gp8qpqvs/JZ0zfRERab3oNT01NEJUiMgm4DvjIvS/UmZCMMe3ZFcN6MDwtlt9/soaCknJfh+Mz2VuKKKuoZkzfLj6LoakJ4gZgFK4lQDeLSDrwmnNhGWPaq6Ag4fFLT6TkUBX3vrOs3a4+N2fdXsKCgxidEe+zGJqUIFR1tareqapvikgsEK2qf3Q4NmNMO9UnKZpHxp/AV+vy+MVri9tl19fa7q1R4a3fvbVWU3sxzRGRTiISBywDXhKR/3M2NGNMe3b1yFQev3QgX67dyy2vLmpXScLX3VtrNbWKqbOq7gcuA15S1WHA2c6FZYwxcNXJqfzp8hP5ZkMeN7+S3W6ShK+7t9ZqaoIIEZFuwJX82EhtjDGO++nwnvz58kF8l5PP4x+v8XU4rcLX3VtrNTVBPIprZbiNqrpQRHoBNtzRGNMqrsjqwc9G9GTawm3kFpX5OhxHlVdVM9fH3VtrNbWR+h1VHaSqv3Bvb1LVyxs7TkTGicg6EckRkfu9vP6EiCx1P9aLSHGd1zuJyA4RebqpF2SMaZtuP7M3AM/O2ejjSJyVvaWIUh93b63V1EbqFBF5X0T2isgeEXlXRFIaOSYYeAY4HxgATBKRAZ5lVPVuVR2iqkOAp4D36rzNY8DXTb0YY0zb1T2mAz8d3oN3sre36buIuRvzCQ4Sn3ZvrdXUKqaXgBlAdyAZ+NC9ryEjgBz33UYFMA2Y0ED5ScCbtRsiMgxIAmY3MUZjTBt3+5m9EYRnvsrxdSiOWbiliIHdO/m0e2utpiaIRFV9SVWr3I+Xgcaa15OB7R7bue59RxGRVFzzO33p3g4C/gbc18T4jDHtQLfOHZg4ogfvZOeyvbDt3UVUVNWwbHsxw1Jbf+0Hb5qaIPJF5GoRCXY/rgYKGjnGW+tKfbNvTQSmq2ptH7bbgJmqur2e8q4TiEwWkWwRyc7Ly2skHGNMW3DbmN4ESdu8i1i5cx/lVTUMT4v1dShA0xPEjbi6uO4GdgE/wTX9RkNygR4e2ynAznrKTsSjegnXtB5TRGQL8FfgWhE5auS2qk5V1SxVzUpM9G1/YWNM6+jaOYJJI3owfVHbu4vI3lIIwLBAShCquk1Vx6tqoqp2UdVLcA2aa8hCIFNE0kUkDFcSmFG3kIj0BWKBeR7nu0pVe6pqGnAv8IqqHtULyhoLJG0AABb7SURBVBjTPt12pusu4sXvNvs6lBaVvaWItPhIukRH+DoU4PhWlLunoRfd04NPwTV+Yg3wtqquEpFHRWS8R9FJwDS11cqNMU2U1CmCC07syruLciktr/J1OC1CVcneWuQ37Q8Ax9NM3ugIDlWdCcyss++hOtuPNPIeLwMvH3N0xpg27ZpRqfx36U4+WLqTn53c09fhHLdN+aUUllb4TfsDHN8dhP3Fb4zxmaE9Y+nfrROvzNtCW6iAqG1/yErznzuIBhOEiBwQkf1eHgdwjYkwxhifEBGuGZnK2t0HWLytyNfhHLfsLUXERoaSkRjl61AOazBBqGq0qnby8ohWVd+P4jDGtGsThnQnOjyEV+dt9XUox622/cHX8y95Op4qJmOM8amo8BAuH5bCzBW7yQ/g5UnzDpSzOb/Ur9ofwBKEMSbAXT0ylYrqGt7ObnBcrV9btLW2/cEShDHGtJjeXToyOiOe1+dvo7omMBurs7cUERYSxMDkzr4O5QiWIIwxAe+akansKD7IZ6v3+DqUZlm4tYghKTGEhwT7OpQjWIIwxgS8cwYkkZ4Qxd8/X09NgN1FHKyoZtWOfX4zvYYnSxDGmIAXEhzEXWdnsnb3AT5ascvX4RyTZbnFVNWo3zVQgyUIY0wbcfGg7vTrGs0Tn62nqrrG1+E0We2Eg70To30cydEsQRhj2oSgIOGec/qwOb+Udxfn+jqcJissrQAgrmOYjyM5miUIY0ybcc6AJAb3iOHJL3Ior6pu/AA/UFhWQVhwEFFh/tVADZYgjDFtiIhw77l92FF8kDd/2ObrcJqkqLSCuKgwvxpBXcsShDGmTTm1dwIje8Xx9FcbKavw/6nAC0sriI3yv+olsARhjGljRIS7z+5Dfkk5Hy/3/x5NhaUVxEWF+joMryxBGGPanBHpcaTEduCjAEgQRWWVxEbaHYQxxrQKEeGiQd35Pif/cC8hf1VQUk68VTEZY0zruWhQN6pqlE9X7vZ1KPWqrK5h/6Gq9tkGISLjRGSdiOSIyP1eXn9CRJa6H+tFpNi9f4iIzBORVSKyXER+6mScxpi254TunUhPiOKj5Tt9HUq9issqAYhrbwlCRIKBZ4DzgQHAJBEZ4FlGVe9W1SGqOgR4CnjP/VIZcK2qngCMA/4uIjFOxWqMaXtEhIsHdWP+pgL2Hjjk63C8KipzD5JrbwkCGAHkqOomVa0ApgETGig/CXgTQFXXq+oG9/OdwF4g0cFYjTFt0EWDu1Oj8MkK/6xmKihxJ4h22EidDHiu4JHr3ncUEUkF0oEvvbw2AggDNnp5bbKIZItIdl5eXosEbYxpO/okRdMnqaPfVjPV3kG0xzYIb8MC65uHdyIwXVWPGBsvIt2AV4EbVPWo2bdUdaqqZqlqVmKi3WAYY4528aDuLNxSxM7ig74O5Si1PazaYy+mXKCHx3YKUF8an4i7eqmWiHQCPgZ+o6rzHYnQGNPmXTS4OwAz/XAa8NoEEdMOq5gWApkiki4iYbiSwIy6hUSkLxALzPPYFwa8D7yiqu84GKMxpo1LT4hiYHInPvTDQXOFpRVEh4cQFuKfIw4ci0pVq4ApwCxgDfC2qq4SkUdFZLxH0UnANFX1rH66EjgduN6jG+wQp2I1xrRtFw3qzrLtxbyTvb3xwq2oqMx/52ECCHHyzVV1JjCzzr6H6mw/4uW414DXnIzNGNN+XD0ylW835HHf9OVs2FvC/4zrR3CQ72dPLXTP5Oqv/PO+xhhjWlDH8BBevmEE14xMZeo3m7j5lWwOHKr0dViWIIwxxh+EBgfx2CUDeWzCCXy9Po8r/jmPiirfLk1aVFrhtxP1gSUIY0w7c82oNP58+SDW7j7A/E0FPo2lsKyCeD9carSWJQhjTLtz4aBuRIYF84kPJ/Irq6jiUGWN3UEYY4w/iQgN5sx+Xfhs9W6qa+obv+us2jEQ/rpYEFiCMMa0U+NO6Ep+SQWLthb55PxFpbUzuYb75PxNYQnCGNMundmvC2EhQXyy0jcD6ApKywG7gzDGGL/TMTyE0zMTmLVyN0eO020dhyfqszYIY4zxP+ed0JWd+w6xYse+Vj93Yal/LxYEliCMMe3YOQOSCA4Sn/RmKiqtIDhI6BRhVUzGGON3YiLDGNUrnk99UM1UUFpBbGQoQX4w5Ud9LEEYY9q18wZ2ZXN+KRv2lrTqef19FDVYgjDGtHPnDUhCpPWXJS0s8+95mMAShDGmnevSKYJhPWP5aPlOqqpbb24mf5+oDyxBGGMM14xKZcPeEh6esarV2iKKSv17LQhweD0IY4wJBBOGJLN29wGem7ORlNhIfjEmw9Hz1dQoRWUVfrsWdS1LEMYYA9x3bl92FB3kT5+upXtMBBOGJDt2rv2HKqlR/x4kBw5XMYnIOBFZJyI5InK/l9ef8FhSdL2IFHu8dp2IbHA/rnMyTmOMCQoS/nLFIE5Oj+O+d5Y7OhV4weGJ+tppghCRYOAZ4HxgADBJRAZ4llHVu1V1iKoOAZ4C3nMfGwc8DJwMjAAeFpFYp2I1xhiA8JBgpl6TRY+4Dtz7zjLH2iOK3AnC39sgnLyDGAHkqOomVa0ApgETGig/CXjT/fw84DNVLVTVIuAzYJyDsRpjDACdI0O5fnQauUUH2VpQ5sg5aqf69vc2CCcTRDKw3WM7173vKCKSCqQDXx7LsSIyWUSyRSQ7Ly+vRYI2xphRGQkAzN3oTDVTod1B4G38eH33axOB6apafSzHqupUVc1S1azExMRmhmmMMUfKSIwiqVM4czfmO/L+he6ZXOPacSN1LtDDYzsF2FlP2Yn8WL10rMcaY0yLEhFGZyQwb2OBI+0QRaUVdAgNpkNYcIu/d0tyMkEsBDJFJF1EwnAlgRl1C4lIXyAWmOexexZwrojEuhunz3XvM8aYVjEqI56C0grW7TnQ4u9dWFrp9z2YwMEEoapVwBRcX+xrgLdVdZWIPCoi4z2KTgKmqUeaVtVC4DFcSWYh8Kh7nzHGtIrRGfEAzM1p+XaIwtJyYv14Jblajg6UU9WZwMw6+x6qs/1IPcf+G/i3Y8EZY0wDUmIjSY2PZO7GAm48Nb1F37uwrNLvB8mBzcVkjDH1Gp0Rzw+bClp8Er+iUv+fZgMsQRhjTL1GZSRwoLyKVTv3t+j7FgbARH1gCcIYY+o1qperHeL7FuzuWl5VTUl5ld93cQVLEMYYU6/E6HD6JkUzrwUHzBWXVQIQ19EShDHGBLRRGfEs3FJIeVV144WboKAkMAbJgSUIY4xp0OiMeA5V1rB0W3HjhZtgz4FDgOvuxN9ZgjDGmAac3CueIIHvW6iaaZt7AsCecZEt8n5OsgRhjDEN6NwhlIHJnZmb0zIN1VsLyugQGmx3EMYY0xaMzkhg6fZiSsurjvu9thWW0jMuEhFvc5L6F0sQxhjTiFN7J1BVoyzYfPwz/mwtKKNnvP9XL4ElCGOMaVRWWixhIUF8f5zVTDU1yrbCMlIDoP0BLEEYY0yjIkKDGdYz9rgbqvceKKe8qoZUu4Mwxpi249TMBNbs2k9+SXmz32NrQSkAPeOjWiosR1mCMMaYJqid/vt4RlVvLXR1cbUqJmOMaUNOTO5MdHjIcS1Duq2gjOAgITm2QwtG5hxLEMYY0wQhwUGMzIjnu+NoqN5aWEb3mAhCgwPjqzcwojTGGD9wSkY82wsPst1dVXSsthWUkhoXGO0P4HCCEJFxIrJORHJE5P56ylwpIqtFZJWIvOGx/8/ufWtE5EkJhFElxpg27dTMBIBmd3fdWhg4YyDAwQQhIsHAM8D5wABgkogMqFMmE3gAOEVVTwDucu8fDZwCDAIGAsOBM5yK1RhjmiIjsSNdosObVc2072AlxWWVAdNADc7eQYwAclR1k6pWANOACXXK3Aw8o6pFAKq6171fgQggDAgHQoE9DsZqjDGNEhFO6Z3AvI0F1NToMR1bO0lfoIyBAGcTRDKw3WM7173PUx+gj4h8LyLzRWQcgKrOA74Cdrkfs1R1Td0TiMhkEckWkey8vDxHLsIYYzyd0juBgtIK1u05cEzHbS10j4GwNggAvLUZ1E25IUAmMAaYBPxLRGJEpDfQH0jBlVTGisjpR72Z6lRVzVLVrMTExBYN3hhjvDmlt3sZ0mOsZtpaO8233UEArjuGHh7bKcBOL2U+UNVKVd0MrMOVMC4F5qtqiaqWAJ8AIx2M1RhjmqRb5w5kJEbx+Zpjq/XeXlhGQscwOoaHOBRZy3MyQSwEMkUkXUTCgInAjDpl/gucCSAiCbiqnDYB24AzRCREREJxNVAfVcVkjDG+MGFIMvM3FZJb1PTurlsLygJikSBPjiUIVa0CpgCzcH25v62qq0TkUREZ7y42CygQkdW42hzuU9UCYDqwEVgBLAOWqeqHTsVqjDHH4tKTXM2p7y/e0eRjthWWkRogczDVcvReR1VnAjPr7HvI47kC97gfnmWqgVucjM0YY5qrR1wko3rF8+7iXKaM7d3o4j/lVdXs3HfQ7iCMMaY9uHxYClsKyli0tajRsrlFB1ENrC6uYAnCGGOa5fyBXYkMC2b6otxGywbiGAiwBGGMMc0SFR7C+QO78fHyXRysqG6w7OF1IAJoDARYgjDGmGa7fFgyB8qrmL16d4PlthaWERkWTELHsFaKrGVYgjDGmGYamR5PckyHRquZtrm7uAbanKOWIIwxppmCgoTLhybzXU4+u/YdrLfc1sLAGwMBliCMMea4XD4sBVV4ZMYqdhYfnSRqatQ9BiLwEkTgjPk2xhg/lBofxZ1nZfLcnBy+WpvHVSN7ctuY3lTV1PD5mr3MXrWbiqqagBskByCusWqBLysrS7Ozs30dhjGmncotKuPJLzbw7uIdBAlUVru+W9PiIzlnQBJTzsykc2Soj6M8mogsUtUsr69ZgjDGmJazKa+EV+ZtpWvnCM7un0RGYpRfN043lCCsiskYY1pQr8SOPDL+BF+H0SKskdoYY4xXliCMMcZ4ZQnCGGOMV5YgjDHGeGUJwhhjjFeWIIwxxnhlCcIYY4xXliCMMcZ41WZGUotIHrDVy0udgX2N7PPc9va89t8EIL+ZIXqLo6llGruG+q7HWxknr6Gh1xv6P6+73dhzX1xDS/weeT5v7jU4+XtUd7uhzwL45zU05Xr87fPc1G2nPgupqprotYSqtukHMLWxfZ7b3p57/JvdknE0tUxj11Df9dRzLY5dQ0OvN/R/3pSfga+voSV+j1riGpz8PWpi3J77/O4amnI9/vZ5bup2a38WVLVdVDF92IR9Hzby3Nt7tEQcTS3T2DXUdz0NlWmOxt6jodcb+j+vu92U583V3Gtoid+jppy/MU7+HtXdbkufBc/n/nYNTd1u7c9C26liag0ikq31TGoVKOwa/INdg+8Fevzg/DW0hzuIljTV1wG0ALsG/2DX4HuBHj84fA12B2GMMcYru4MwxhjjlSUIY4wxXrXbBCEi/xaRvSKyshnHDhORFSKSIyJPisdyUSJyh4isE5FVIvLnlo36qDha/BpE5BER2SEiS92PC1o+8iPicOTn4H79XhFREUlouYi9xuHEz+ExEVnu/hnMFpHuLR/54RiciP8vIrLWfQ3vi0hMy0d+RBxOXMMV7s9xjYg41hB8PLHX837XicgG9+M6j/0Nfl68am4f2kB/AKcDQ4GVzTh2ATAKEOAT4Hz3/jOBz4Fw93aXALyGR4B7A/nn4H6tBzAL1+DJhEC7BqCTR5k7gX8GWPznAiHu538C/hSAP4P+QF9gDpDlb7G740qrsy8O2OT+N9b9PLah62zo0W7vIFT1G6DQc5+IZIjIpyKySES+FZF+dY8TkW64Przz1PW//gpwifvlXwB/VNVy9zn2BuA1tCoHr+EJ4NeA470wnLgGVd3vUTQKB6/Dofhnq2qVu+h8IMWp+B28hjWqus7JuI8n9nqcB3ymqoWqWgR8Boxr7me+3SaIekwF7lDVYcC9wLNeyiQDuR7bue59AH2A00TkBxH5WkSGOxqtd8d7DQBT3FUD/xaRWOdCrddxXYOIjAd2qOoypwNtwHH/HETkcRHZDlwFPORgrN60xO9RrRtx/cXa2lryGlpbU2L3JhnY7rFdez3Nus6QJp60zRORjsBo4B2Pqrlwb0W97Kv96y4E123dSGA48LaI9HJnbMe10DU8Bzzm3n4M+BuuD3irON5rEJFI4EFcVRw+0UI/B1T1QeBBEXkAmAI83MKhetVS8bvf60GgCni9JWNsTEteQ2trKHYRuQH4pXtfb2CmiFQAm1X1Uuq/nmZdpyWIHwUBxao6xHOniAQDi9ybM3B9gXreLqcAO93Pc4H33AlhgYjU4JpMK8/JwD0c9zWo6h6P414APnIyYC+O9xoygHRgmfvDlQIsFpERqrrb4dhrtcTvkqc3gI9ppQRBC8XvbiC9CDirtf5I8tDSP4PW5DV2AFV9CXgJQETmANer6haPIrnAGI/tFFxtFbk05zqdangJhAeQhkfDEDAXuML9XIDB9Ry3ENddQm1jzwXu/bcCj7qf98F1qycBdg3dPMrcDUwLtJ9DnTJbcLiR2qGfQ6ZHmTuA6QEW/zhgNZDo9P+9079HONxI3dzYqb+RejOumoxY9/O4plyn17ha64fnbw/gTWAXUIkru/4c11+enwLL3L/cD9VzbBawEtgIPM2PI9LDgNfcry0GxgbgNbwKrACW4/oLq1ugXUOdMltwvheTEz+Hd937l+OaVC05wOLPwfUH0lL3w7FeWA5ew6Xu9yoH9gCz/Cl2vCQI9/4b3f//OcANx/J5qfuwqTaMMcZ4Zb2YjDHGeGUJwhhjjFeWIIwxxnhlCcIYY4xXliCMMcZ4ZQnCtGkiUtLK5/uXiAxoofeqFtdsritF5MPGZkQVkRgRua0lzm0M2Ipypo0TkRJV7diC7xeiP05C5yjP2EXkP8B6VX28gfJpwEeqOrA14jNtn91BmHZHRBJF5F0RWeh+nOLeP0JE5orIEve/fd37rxeRd0TkQ2C2iIwRkTkiMl1cax68Xju3vnt/lvt5iXvCvWUiMl9Ektz7M9zbC0Xk0Sbe5czjx8kIO4rIFyKyWFzz+09wl/kjkOG+6/iLu+x97vMsF5HfteB/o2kHLEGY9ugfwBOqOhy4HPiXe/9a4HRVPQnX7Km/9zhmFHCdqo51b58E3AUMAHoBp3g5TxQwX1UHA98AN3uc/x/u8zc6H457/qCzcI1sBzgEXKqqQ3GtQfI3d4K6H9ioqkNU9T4RORfIBEYAQ4BhInJ6Y+czppZN1mfao7OBAR4zZXYSkWigM/AfEcnENdNlqMcxn6mq55z9C1Q1F0BEluKaS+e7Ouep4MfJDhcB57ifj+LHufjfAP5aT5wdPN57Ea65/cE1l87v3V/2NbjuLJK8HH+u+7HEvd0RV8L4pp7zGXMESxCmPQoCRqnqQc+dIvIU8JWqXuquz5/j8XJpnfco93hejffPUqX+2MhXX5mGHFTVISLSGVeiuR14Etf6EInAMFWtFJEtQISX4wX4g6o+f4znNQawKibTPs3Gtb4CACJSO61yZ2CH+/n1Dp5/Pq6qLYCJjRVW1X24lh29V0RCccW5150czgRS3UUPANEeh84CbnSvL4CIJItIlxa6BtMOWIIwbV2kiOR6PO7B9WWb5W64XY1rmnaAPwN/EJHvgWAHY7oLuEdEFgDdgH2NHaCqS3DN7DkR1+I7WSKSjetuYq27TAHwvbtb7F9UdTauKqx5IrICmM6RCcSYBlk3V2NamXvVu4OqqiIyEZikqhMaO86Y1mZtEMa0vmHA0+6eR8W04pKuxhwLu4MwxhjjlbVBGGOM8coShDHGGK8sQRhjjPHKEoQxxhivLEEYY4zx6v8D0xNOhL+/L+gAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.callbacks import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "hidden": true,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.810097</td>\n",
       "      <td>0.690017</td>\n",
       "      <td>0.581065</td>\n",
       "      <td>0.708460</td>\n",
       "      <td>03:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.801332</td>\n",
       "      <td>0.692924</td>\n",
       "      <td>0.577293</td>\n",
       "      <td>0.705548</td>\n",
       "      <td>03:09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXxU9b3/8deHQNiRfRGERMAFlIJGpGoRlVUs2luuxdZbbW1prRaXthKtC2K9Ulv1an96rVpurbWidaWCoiAWF1CCLLITMEiIQNjCGrJ9fn/MZJhJZpIJJkYO7+fjkUfmnPM953znO+e855zvnDlj7o6IiBz9GtR3BUREpHYo0EVEAkKBLiISEAp0EZGAUKCLiAREw/pacWrz1t7v1F71tXoRkaPSokWLtrt7h3jT6i3Qm7btTFZWVn2tXkTkqGRmGxNNU5eLiEhA1FugO/pCk4hIbdIRuohIQNRbH7qISE0VFxeTm5tLYWFhfVelzjVp0oRu3brRqFGjpOdRoIvIUSM3N5eWLVuSlpaGmdV3deqMu7Njxw5yc3NJT09Pej51uYjIUaOwsJB27doFOswBzIx27drV+ExEgS4iR5Wgh3m5I3meCnQRkYBQoIuIJGn37t089thjNZ7v4osvZvfu3XVQo1gKdBGRJCUK9NLS0irnmzlzJq1bt66rakXoKhcRkSRlZmayfv16+vfvT6NGjWjRogVdunRhyZIlrFy5kssuu4xNmzZRWFjIDTfcwPjx4wFIS0sjKyuLffv2MWrUKM477zw+/PBDunbtymuvvUbTpk1rpX4KdBE5Kt39rxWszNtTq8vsc3wr7vp234TTp0yZwvLly1myZAnvvvsuo0ePZvny5ZFLC6dOnUrbtm05ePAgZ511Ft/97ndp165dzDLWrVvHc889x5NPPsnll1/OSy+9xJVXXlkr9Vegi4gcoYEDB8ZcJ/7II4/wyiuvALBp0ybWrVtXKdDT09Pp378/AGeeeSY5OTm1Vh8F+jHsg+ztDExvS6OU5D5KKSopY8Jzi5k0pi+dj2tSx7Wr2lPvbeCkTi0ZfFLcu4jWmbzdB2lgVmfPf9f+Ilo3a3TMXJr3ZVR1JB2Pu7O/qJQWjWsv9po3bx55/O677zJ79mzmz59Ps2bNGDJkSNzryBs3bhx5nJKSwsGDB2utPvX6oejlj88nLXPGl1rGnsJi3Gt2o68XFm4iLXMGRSVlX2rdtWnz7oOkZc5g+eaCGs9bUlpGWuaMGj2nR+as4wdPfUTv376R9HreWb2VN1dsYdB9c4BQqP7qhaWUltX+jdYKi0sjz2VvYTFb98TuGL+bsYofTv044fz3zVyV9LY189MvWLd1b1Jlz5nyTuT5JyN7214eenttzLjSMufWlz+t9Hrl7jrAgHveZsobq8nK2Zn0OgBW5u1h44797Nh3iLteW87BosQf0pWWOW98+kWl/abgYHGN1/tV2neohM27D7LnYPERzf/5zgNsyN9HwYGiI65Dy5Yt2bs3/rZSUFBAmzZtaNasGatXr2bBggVHvJ7SsjK2FBRSVsNsq7dAP1RSxsfhjadiCBUWl7I2iR1s36ES+k16i+889mGN1n3LS8sAOPu/Z1e54Sdr3da9fPzZl9sRbpy2GIBL/vQ+ZWXOz59ZFBNixeHQHnz/3Erz5u0+XO6k2ysHdElpGS8uyo15s3iwQsgko3GjlMjj0jLndzNW8dInufS8bWaNl1WVVxbncsodb0aey+mT3uLs/44fojv3x985/zxvQ6Se1fnFs58w7KF5CacXFpeypaCQXRXWNW9tPmmZMxK+GeTvPcTQB+fx8Jx1bNsbeo0+zN5Oz9tm8tzHnwOxr9fU93MidR/7+Hx+8eyiyPb59wUbScucQVmC53PxI+9x/h/e5czfzebp+RsZeO/shM/n7ws2cu2zn3Dnaysi49bn7+Mbd7/F2MfnU1hcygNvraGwuOp9483lW0jLnFFlG/9pzjrSMmfwk6cr//bB3sJipi/NS/qAbP22fbhDzo79FCQZ6qVlZRw4VAIczpmN4WBPVpl7pI7t2rXj3HPP5bTTTuM3v/lNTLnhI0ZQVFzM6f36cccddzBo0KCoZcCKvAJWfbEncp9Zd2f/oZJKz3/X/iJW5O1h295Ctu05RFrmDHK270+qrl+LyxaX5sZen/nL5xYz/KF5lXagwuJSPly/PTI8e+VWAJZs2s2+8ItWfqQa/YIdLCpl0vQVbN93KGZ5uw4Uc+qdbyas14GiEp5f+Hm1oTDsoXlc/uf5lcbvPlDE/kMlkZ2wvG7vrcuPKefuLMzZFRk+8baZvLliC2f/9xxufn4JubsOsKUgFAif7zxQaQNYUqH99hQWR5a7t7CYqR98xq//uZRL/vQ+2/YUVtpRf/zXhXyYvT2y3E07D5CWOYPpS/N4/N/r2X2giLTMGfzo/xYC0LNDc74oiD1N/N3rKzlUUsptr3zKo3OzWbRxJ2mZM1iRV8Cijcm/2b38SS43Pb80MlxSWvmMY+OOwxt3dUdrz8zPiTt+Q/4+VuQVVArIgoPFzFsben0u+dN7pGXOYOzjHzLovjkMuOftSLm0zBmRM4TyN4PvPPZB5A23rMw5KypUB94bekP6/lMfVarLzv1F/PyZRUz94LOY8TM/3RLZPm9/dXmo3tsPb9fTPv6c1Vvifyi491AJE55bTHFpGWu37qW07HAovbJ4c6htFhz+nYSLHvh35PGE5xbzp3eyeTL8phjP5t0H+fnfFwHw8Jx1cUM5f+8hHggfOMxetbXS9O/9eQETnlvM95+s3CbxLNp4eB/ZuGM/2/YWsnnXgZgy2/cdYuf+0H5eVFLKirw9ZOfvo+BgMSkNDndj7TtUQlFJWZUHdCWlZRQWl7J8cwGfbi6gtKyMZbm7ybz/MZYvX87ChQv52/Mvsyx3N9nb9rFueyF/eGoaz86cx3PTnmfOO+8wZMgQAN76aBlt2raj6wndef3d0PNdu3Uf37nq51z/69si63R3NkU9p5Lw9jnkj+9y38xVzF+/o8o2+lr0oe87VMKWgkIWf76Lj3N28nY4qAfc8zY5U0bz0qJcfvXPpWT0aEPWxl3Mvvl8enVswY3PL4ks47S7ZjH50sN9ahNfWsY/f34OL2Rt4pYXQ0fkf/0wh8/uu7jS+gdMfovFdw6PnKJn3zuKhikN6HPnrPCyPuXqc9K44aLekf7NpZt2c0qXljRuePioddaKLYzo2zky3H/y4QDImTI68vi//vIxOVNG4+6YGT97ZlHCtnl58WZeXryZW0aeHBl32WMf8sex/fjFs5/wr1+ex4TnFsfM88O/fMyr151L+q2hI+eMHm0i03YeKGJghaPdd1Zv453V2xh31gnMW5tPXvjNo3y5U95YHVN+ff5+zvt97JnCU+9/xmldj+MfH30eM370I+8D8PSPB3J+gv7uktIyGob78W9+YWnMtMse+yDy+Jn5OWzccYAurQ9f4lXxSG1PYTH9Jr0VGZ70r5V8nLOTb57Yjkn/WsmKu0fQpFEKF0YFWLlNOw/wrXAgjx98Iss3h8Ky/H9VPvl8F4s/P/zGum5b5SPAROF7w7TFvLdue9xpEPumNvTBecz99RCWbtpN5sufhtZ176i4801fmsf0pXkx43KmjGbJptgDgB0VDnTeCu9/D85ey9a9hYzo25nC4jKG9enEvkMlfJC9PWabfWTOOl5alMsHmRcCcO6Udxh7Zjd6dWwRs9x+k2axbNKIyPDKL0LtMX/DDkrLPHKmt+7eUZGuwOj9ZvLrK3lyTJfIcPlBzo79RXRq1STmjPZgcVnM84o+CChX/nr06xb/+vDy+pXbf+hw+BccLI5Z5oGikrjztm6WSvPUlJheiL3hbuLy95cvCg7SrkUq2/cdijyneP48b0PkzDMRS+Z0x8xGAg8DKcBT7j6lwvTuwNNA63CZTHev8jy8cZfe3uWq/6l23e/dckFkJyt3w0W9uWnYSaTfOoOqqv+XqzK4psKp3tXnpPHXD3MqlR0/+ESeiGqsxXcMizkiK/ebESfT5/hWkaPVnCmj4/bVzr55MEMfTHwaX1taNG4YOTuJtuZ3Izn59sRnH3WhWWoKBxIc8TRu2IBFdwyr9IFUedv94OzuXDmoB6Mefq9G65xwUW8GndiWUzq3YvbKrZHutGNJwwYWOZKrTsXt+uFx/blh2pIq5jjs8SvPjByVxzNx5Ckc17QRt73yacIyq+8ZSZNw1130fpOa0oCi8BtX2+apka60y/ofz/+MG8DbK7fy079l8eSYLnTqfmJS9U1Wzw4tInUqP4o/VFzKmiQ/VzlSzVIbVnojqGjr5xv46fQvYsZt/P0li9w9I175agPdzFKAtcAwIBdYCFzh7iujyjwBLHb3/zWzPsBMd0+rarnJBnoin913ceQItD4lCvSvUtNGKRwsLmXChb145J3spOc7O70tH33Jvv+aWvjboTRp1ICGDRqwfd+hmDfri07pyJzV277S+gTR3F8P4YI/vlvf1Uhowa0XsfjzXVz77CdJlY8+Yq+LQI+W1q45DVOM7DhnWPWhpoGeTB/6QCDb3Te4exEwDbi0QhkHWoUfHwfkUcdqeiRX0cSRp/Dbi0/lpE4tqi9chTPiHMUfqQZHeKXawXCf+M3DT66mZKznf/bNI1thlOhT4mScde9sTp/0Fqfe+Sa//mds90pth/nD4/rX6vKOxCu/OKdG5S/rfzwAI/t25j/O6HpE60xv37z6QnFcOaj7Ec1XnWvOi72f96D75iQd5kClK7HatWicoOSXl7Njf43D/PjjDncBntK5VRUloXlqcr3cp3c9jn7dWtOueWqN6pJMoHcFNkUN54bHRZsEXGlmucBM4JfJrDwlQYK9P/ECJo48pcp5V2+Jfzr07E/OTmbV7DpQxE8Hn8gPv5mWVPlEEl1lUe69Wy6IGf6f7yUOmZeuPbzz//2as2nfIpUZE86LKdOpVdUb8wP/+Y244y8+vXPMcHm9Ft8xjNX3jKxymd3aNGXV5FCZruH+6/duuYDFdwwD4I5L+sSUz7p9aOTx9OvPTbjcRGcHZ3Sv/p4XVwxMHD5P/TCDz+67mEv7d2Vk384Jy1WnX7fjyJkymgW3XkTOlNFMuKg3ALddfAq/u+w0Vtw9gp9+q+ofHxjQvU3Caa9eV7ltHry8P7+8sBcTR53CPZeeFjMtmTeH34wIvak/csWAyLi3bhrMmG8cH1Nu0IltK807/ls9q11+RTeE26Qqd1zS54jfnCpqlppCx5ZHHuhNU1OqL1RD0V8ZqO6grGfHFnRsWfV3GNLbN498D6FpakqNtuFkAj1eFSv201wB/NXduwEXA8+YWaVlm9l4M8sysyyAVk0acuPQwxvEuntHkTNlNN3aNGP84MqnVZO+3afSOAgdJU64sBcj+nbi3F7tefzKMyuVmfIfp8cExZCTQx/QXXRqx5hy110Qu1F/Oml43HXGU/6hULRubZoyom8nAP7xk7O5bEBXcqaMZvU9I1k1eSTfDu9o08YPYkD3NnRt3ZSrvtmD83q3J+v2YfQ9/rjIspbeOZyPbhtaaR3Rvntmt5jh2Tefzy+G9OTR759Bk0ahl+T+sf04oW0zANo0T6VJo6p3kvcnXkjT1BRypozmg8wLyZkymhPaNqNN+OjhnJ6HvwnXKMVo36IxyyYNZ9HtQxN+4BRtzq/Ojxl+5pqzefPGb1U5z/UX9oo7vn2LVIb26RTZIR65YgCv//I8/pjgja7cysmhD+uuGNid45qGfvLrtXDgln+J6LoLenL76FP50bnpXDmoB80bN4w5IuvZoTmzbx7MzwafyG9GnMyy8LYzMD02PHOmjCZnymj6nxDbNtdf0IsGDYxfDT+Z9PbNad64Ie3DR6Pn9GzHgO5tuHtMX757RjdO6dySpXce3jZ/d9lpLLp9KNddEGqXMd84PrKekzq1jAl4gD+MrdwebZrH/tRZvLOv/7v6rJjHv0zwOlTcFx74z2/QvAZh+sw1A+OOb9WkIY1SGtCrYwv6dGlFl+Ni74HSp0urSoF1cqeWkce9OrTgtK7Hkaw+XVrRO2r+Ns0OHzGfdvxxnN71uMj20qJxQxqmNODE8BlSWrvYM6Xe4Q+JE30prXlqQ/p1a03LJrGvw+P/dSbvT7wgsj1VJZk+9G8Ck9x9RHj4VgB3vy+qzApgpLtvCg9vAAa5e8Jz6MZdevvc9+ezPn8/d4Qvyaq4AUX3TS++Yxhtmqfi7jF950vvHM5xzSr/5l5hcSlfFBTSvkUqLRo3xMyYv34HVzwZutj/nz//JmelhXa0P8xaTYoZ3xvYnfXb9kUuR/t00nBaNmlEwcFivnH3W2TdPpT2LRrH7TOfcGEvbh5+Mu6Oe+jSw/LntLewmLVb93Jmj8pHRckoKikjpYFFzmiu/fsi3li+pVK58vaLrl90m7o7xaVOasPK7+OFxaWccseb3DLyZE7u1DLmw+RkulWWbtrNrgNFDDm5Y6VpD769lkfmrEs4b/TnEMsmDadVeIP+2TNZzFqxlRV3j2Dw/XPZET4buvqcNCaN6cvO/UXc+vIyHh43IPKhVlXKypwf/XUh/16bT/sWjZl5w3kMvHcOw/p04skfxu2STGqZ05fmcUm/LpErdSp6ePY6Hpp9+Lr/iu1ZcKCYMvfIG2S0wuJS9haW0KGKN9zVW/ZwcqeW1X67tLyNV00eSdPUFH7ydBazV23lB2d3p1XTRtwy4uTIvjX01I48ddVZvLN6Kz/+axYTLuzFgO5tuOCUyq9v9LIhdEFAr44teWZ+Duf0ak/PDqEgy9m+nyEV+vZP7dKKVeErQhoY3D2mL4NObEfvTi0j6y6XM2U0q1at4tRTT41ZxrLwZbsdWzWhc6tQWGZv28uBolLaNk+lW5tmkTLlBxjLcnfTLDWFVk0b0bBBA9o0a0RhcSkFB0si3xlIb988Eq7LcnfTplkqJ7QNLat5akN6dqy+u7Z58xasz8unbP8uJkyYwIsvvgjAvsJiNmzfT1q75owYdhE3/XYyl48aUmn+is933da9nNS5VcI+9HAAJf4jdGnjBiAdSAWWAn0rlHkDuDr8+FRCfehW1XJTO/dyd/eS0jIf+sC7/sG6fK9oyB/meo+Jr3uPia/HjC8fd9dryyvNU50vdh/0B2at9uKS0rjTP9m403tMfN0veuDdhMt4a8UWX7G5wLNydniPia/73dNXVCoTr961pbS0zGev3OLu7hvy9/ljc7N9+97CmPr1mPi6P/DWmiNex56DRd5j4ut+2l1vfun6uruvzCvwnzy9MNIu0X/u1bdX7q4DPnf11lqpS872fb5z3yF3d39n1Vbfc7CoVpabSElpma/ftrdO15GM3fuL/GBRSZVllny+y+es2lJndXhn9Vaft3abf7Jxp89dvdVLSsv81cW53ueON3xLwcFK5cu3iw+yQ/mwcuXKSmW27TnoSzft8tLSspjxRVH7+Mq8Al+3tfrXoLik1Jdu2uVLN+2KGV9WVuZlZWUJ5kqsefPm1ZY5//zzfeHChXGnxXu+QJYnyutEEzw2sC8mdKXLeuC34XGTgTHhx32AD8JhvwQYXt0yywO9KivzCrzHxNf96Q8/ixn/YfZ2n79+e7XzH4ndB0JB9uri3DpZ/rGsrKzMpy/Z7Lv3F/lFD7wbE+LZ2/b65l0H6rmG8nXzWf4+f23J5shwvICrTWVlZb500y5f88WeuNNvueUWf/TRRyPDd911l0+aNMkvvPBCHzBggJ922mn+6quvRqaXB/pnn33mffv2dXf3AwcO+Pe+9z0//fTT/fLLL/eBAwfWWqAndR16XWjcpbcf+iLxabgEm7szf8MOzk5vl/DDcZGKYrog3siELYmveT8inU+HUVMSTl68eDE33ngj//536Itpffr04c0336R169a0atWK7du3M2jQINatW4eZ0aJFC/bt20dOTg6XXHIJy5cv58EHH2T58uVMnTqVZcuWccYZZ7BgwQIyMir3osTrYjKzhF0uX4tvisqxx8w4p2f7+q6GSI0MGDCAbdu2kZeXR35+Pm3atKFLly7cdNNNzJs3jwYNGrB582a2bt1K587xr06ZN28eEyZMAKBfv37069ev1uqnQBeRo1MVR9J1aezYsbz44ots2bKFcePG8eyzz5Kfn8+iRYto1KgRaWlpcW+bG62ubo9cbzfnahznSgsRka+7cePGMW3aNF588UXGjh1LQUEBHTt2pFGjRsydO5eNGzdWOf/gwYN59tlnAVi+fDnLltXe7Srq7Qj9pKhrO0VEjhZ9+/Zl7969dO3alS5duvCDH/yAb3/722RkZNC/f39OOaXqL0Vee+21/OhHP6Jfv37079+fgQPjX3N/JOrtQ9GMjAzPyqp8j2QRkUTifUgYZDX9UFT9HiIiAaFAFxEJCAW6iBxV6qub+Kt2JM9TgS4iR40mTZqwY8eOwIe6u7Njxw6aNKn6zowV6Tp0ETlqdOvWjdzcXPLz86svfJRr0qQJ3bp1q75gFAW6iBw1GjVqRHp61fegP5apy0VEJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAIiqUA3s5FmtsbMss0sM870h8xsSfhvrZntrv2qiohIVaq9bNHMUoBHgWFALrDQzKa7+8ryMu5+U1T5XwIDKi1IRETqVDJH6AOBbHff4O5FwDTg0irKXwE8VxuVExGR5CXzxaKuwKao4Vzg7HgFzawHkA68k2D6eGA8QPfu3WtUURGReuUe+qOa/15WRRnijC9LbrnRy0ggmUCP91tJiZY6DnjR3UvjTXT3J4AnIHQ/9CTWLfLV2v05rHqd5Hdaqi9b5U7LEcwT/Z84dapBONSobFmS9U4m3BK129fsOZfX6SiRTKDnAidEDXcD8hKUHQdc92UrJVJvtq+FWbfW0sIMzMAaHH6c1H+OYJ4G4UOvJOeptPzq5m2QeFqDcM9tTeaJ+c8RttOXfc6WoN7VzVNVuyX7nMuXX4Oy5f/vviLhFpdMoC8EeptZOrCZUGh/v2IhMzsZaAPMT2KZIl9P6efDxByOONwiO6lIXfkSge7uJWZ2PTALSAGmuvsKM5sMZLn79Ki1TPOg39dSgi2lETRtU9+1EDkiSd1t0d1nAjMrjLuzwvCk2quWiIjUlL4pKiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCQV6GY20szWmFm2mWUmKHO5ma00sxVm9o/araaIiFSn2h+4MLMU4FFgGKHfF11oZtPdfWVUmd7ArcC57r7LzDrWVYVFRCS+ZI7QBwLZ7r7B3YuAacClFcr8FHjU3XcBuPu22q2miIhUJ5lA7wpsihrODY+LdhJwkpl9YGYLzGxkvAWZ2XgzyzKzrPz8/COrsYiIxJVMoMf7CfOKPwTdEOgNDCH0Y9FPmVnrSjO5P+HuGe6e0aFDh5rWVUREqpBMoOcCJ0QNdwPy4pR5zd2L3f0zYA2hgBcRka9IMoG+EOhtZulmlgqMA6ZXKPMqcAGAmbUn1AWzoTYrKiIiVas20N29BLgemAWsAl5w9xVmNtnMxoSLzQJ2mNlKYC7wG3ffUVeVFhGRysy9Ynf4VyMjI8OzsrLqZd0iIkcrM1vk7hnxpumboiIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBERSgW5mI81sjZllm1lmnOlXm1m+mS0J//2k9qsqIiJVaVhdATNLAR4FhhH67dCFZjbd3VdWKPq8u19fB3UUEZEkJHOEPhDIdvcN7l4ETAMurdtqiYhITSUT6F2BTVHDueFxFX3XzJaZ2YtmdkK8BZnZeDPLMrOs/Pz8I6iuiIgkkkygW5xxFX+I9F9Amrv3A2YDT8dbkLs/4e4Z7p7RoUOHmtVURESqlEyg5wLRR9zdgLzoAu6+w90PhQefBM6sneqJiEiykgn0hUBvM0s3s1RgHDA9uoCZdYkaHAOsqr0qiohIMqq9ysXdS8zsemAWkAJMdfcVZjYZyHL36cAEMxsDlAA7gavrsM4iIhKHuVfsDv9qZGRkeFZWVr2sW0TkaGVmi9w9I940fVNURCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiAREUoFuZiPNbI2ZZZtZZhXlxpqZm1ncm6+LiEjdqTbQzSwFeBQYBfQBrjCzPnHKtQQmAB/VdiVFRKR6yRyhDwSy3X2DuxcB04BL45S7B7gfKKzF+omISJKSCfSuwKao4dzwuAgzGwCc4O6vV7UgMxtvZllmlpWfn1/jyoqISGLJBLrFGRf5ZWkzawA8BPyqugW5+xPunuHuGR06dEi+liIiUq1kAj0XOCFquBuQFzXcEjgNeNfMcoBBwHR9MCoi8tVKJtAXAr3NLN3MUoFxwPTyie5e4O7t3T3N3dOABcAYd8+qkxqLiEhc1Qa6u5cA1wOzgFXAC+6+wswmm9mYuq6giIgkp2Eyhdx9JjCzwrg7E5Qd8uWrJSIiNaVvioqIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhBJBbqZjTSzNWaWbWaZcab/3Mw+NbMlZva+mfWp/aqKiEhVqg10M0sBHgVGAX2AK+IE9j/c/XR37w/cDzxY6zUVEZEqJXOEPhDIdvcN7l4ETAMujS7g7nuiBpsDXntVFBGRZCTzE3RdgU1Rw7nA2RULmdl1wM1AKnBhvAWZ2XhgPED37t1rWlcREalCMkfoFmdcpSNwd3/U3XsCE4Hb4y3I3Z9w9wx3z+jQoUPNaioiIlVKJtBzgROihrsBeVWUnwZc9mUqJSIiNZdMoC8EeptZupmlAuOA6dEFzKx31OBoYF3tVVFERJJRbR+6u5eY2fXALCAFmOruK8xsMpDl7nYDUaMAAAg8SURBVNOB681sKFAM7AKuqstKi4hIZcl8KIq7zwRmVhh3Z9TjG2q5XiIiUkP6pqiISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGRVKCb2UgzW2Nm2WaWGWf6zWa20syWmdkcM+tR+1UVEZGqVBvoZpYCPAqMAvoAV5hZnwrFFgMZ7t4PeBG4v7YrKiIiVUvmCH0gkO3uG9y9iNCPQF8aXcDd57r7gfDgAkI/JC0iIl+hZAK9K7Apajg3PC6Ra4A34k0ws/FmlmVmWfn5+cnXUkREqpVMoFuccR63oNmVQAbwh3jT3f0Jd89w94wOHTokX0sREalWMj8SnQucEDXcDcirWMjMhgK/Bc5390O1Uz0REUlWMkfoC4HeZpZuZqnAOGB6dAEzGwD8GRjj7ttqv5oiIlKdagPd3UuA64FZwCrgBXdfYWaTzWxMuNgfgBbAP81siZlNT7A4ERGpI8l0ueDuM4GZFcbdGfV4aC3XS0REakjfFBURCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCCSCnQzG2lma8ws28wy40wfbGafmFmJmY2t/WqKiEh1qg10M0sBHgVGAX2AK8ysT4VinwNXA/+o7QqKiEhykvnFooFAtrtvADCzacClwMryAu6eE55WVgd1FBGRJCTT5dIV2BQ1nBseJyIiXyPJBLrFGedHsjIzG29mWWaWlZ+ffySLEBGRBJIJ9FzghKjhbkDekazM3Z9w9wx3z+jQocORLEJERBJIJtAXAr3NLN3MUoFxwPS6rZaIiNRUtYHu7iXA9cAsYBXwgruvMLPJZjYGwMzOMrNc4D+BP5vZirqstIiIVJbMVS64+0xgZoVxd0Y9XkioK0ZEROqJvikqIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAIiqUA3s5FmtsbMss0sM870xmb2fHj6R2aWVtsVFRGRqlUb6GaWAjwKjAL6AFeYWZ8Kxa4Bdrl7L+Ah4Pe1XVEREalaMkfoA4Fsd9/g7kXANODSCmUuBZ4OP34RuMjMrPaqKSIi1UnmN0W7ApuihnOBsxOVcfcSMysA2gHbowuZ2XhgfHjwkJktP5JKH0PaU6ENJYbap3pqo6odje3TI9GEZAI93pG2H0EZ3P0J4AkAM8ty94wk1n/MUhtVTe1TPbVR1YLWPsl0ueQCJ0QNdwPyEpUxs4bAccDO2qigiIgkJ5lAXwj0NrN0M0sFxgHTK5SZDlwVfjwWeMfdKx2hi4hI3am2yyXcJ349MAtIAaa6+wozmwxkuft04C/AM2aWTejIfFwS637iS9T7WKE2qprap3pqo6oFqn1MB9IiIsGgb4qKiASEAl1EJCDqJdCru5VAkJlZjpl9amZLzCwrPK6tmb1tZuvC/9uEx5uZPRJup2VmdkbUcq4Kl19nZlclWt/RwMymmtm26O8l1GabmNmZ4TbPDs97VH3pLUH7TDKzzeHtaImZXRw17dbwc11jZiOixsfd78IXPHwUbrfnwxc/HDXM7AQzm2tmq8xshZndEB5/7G1D7v6V/hH6YHU9cCKQCiwF+nzV9aivPyAHaF9h3P1AZvhxJvD78OOLgTcIXec/CPgoPL4tsCH8v034cZv6fm5fok0GA2cAy+uiTYCPgW+G53kDGFXfz7kW2mcS8Os4ZfuE96nGQHp4X0upar8DXgDGhR8/Dlxb38+5hu3TBTgj/LglsDbcDsfcNlQfR+jJ3ErgWBN964Sngcuixv/NQxYArc2sCzACeNvdd7r7LuBtYORXXena4u7zqPy9hVppk/C0Vu4+30N75t+ilnVUSNA+iVwKTHP3Q+7+GZBNaJ+Lu9+FjzQvJHTLDoht66OCu3/h7p+EH+8FVhH69voxtw3VR6DHu5VA13qoR31x4C0zWxS+FQJAJ3f/AkIbJ9AxPD5RWx0LbVhbbdI1/Lji+CC4PtxlMLW8O4Gat087YLe7l1QYf1Sy0J1eBwAfcQxuQ/UR6EndJiDAznX3MwjdvfI6MxtcRdlEbXUst2FN2ySobfW/QE+gP/AF8EB4/DHbPmbWAngJuNHd91RVNM64QLRRfQR6MrcSCCx3zwv/3wa8QuhUeGv4tI7w/23h4ona6lhow9pqk9zw44rjj2ruvtXdS929DHiS0HYENW+f7YS6HBpWGH9UMbNGhML8WXd/OTz6mNuG6iPQk7mVQCCZWXMza1n+GBgOLCf21glXAa+FH08Hfhj+VH4QUBA+dZwFDDezNuFT7eHhcUFSK20SnrbXzAaF+4t/GLWso1Z5UIV9h9B2BKH2GWehH51JB3oT+kAv7n4X7hOeS+iWHRDb1keF8Ov6F2CVuz8YNenY24bq45NYQp8yryX0qftv6/uT4a/weZ9I6OqCpcCK8udOqB9zDrAu/L9teLwR+nGR9cCnQEbUsn5M6AOvbOBH9f3cvmS7PEeo26CY0NHQNbXZJkAGocBbD/w/wt+QPlr+ErTPM+Hnv4xQQHWJKv/b8HNdQ9TVGIn2u/B2+XG43f4JNK7v51zD9jmPUBfIMmBJ+O/iY3Eb0lf/RUQCQt8UFREJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQg/j83MGQgDd4CRAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.5810647010803223.\n"
     ]
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',monitor='accuracy',\n",
    "                                                   name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas1_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.810106</td>\n",
       "      <td>0.669316</td>\n",
       "      <td>0.607031</td>\n",
       "      <td>0.732028</td>\n",
       "      <td>03:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.801576</td>\n",
       "      <td>0.667449</td>\n",
       "      <td>0.610543</td>\n",
       "      <td>0.735263</td>\n",
       "      <td>03:15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5d3/8feXkBBWWWSTgIkKBVQEjAguuCCKWMG2aLHt81Or0tpa16cVtCqij0t9am0fqYqttlUUKWqlAoKillplCYqsIgFZwhrCvmT//v6YyTBJJpMJJkYOn9d1cTHnnHvO3HPnzOfc5z7nzJi7IyIiR74G9V0BERGpHQp0EZGAUKCLiASEAl1EJCAU6CIiAdGwvl44pWlL79XjpPp6eRGRI9LChQu3u3vbWMvqLdAbt+5AVlZWfb28iMgRyczWVbWs3oZcHF3/LiJSmzSGLiISEAp0EZGAqLcxdBGRmioqKiInJ4f8/Pz6rkqdS01NJS0tjeTk5ISfo0AXkSNGTk4OzZs3Jz09HTOr7+rUGXcnLy+PnJwcMjIyEn6ehlxE5IiRn59PmzZtAh3mAGZGmzZtanwkokCvZTk7D5A+ehov/OfL+q5KJf/4dCN//WhtfVdD5CsJepiXOZz3Wa+Bvie/KPL4l3//jOxte+uxNrXjnMfeB+CBfy6PuXz7vgK27yv4OqsUcduri7h/6rJ6ee1EvfFpDoOf+BdVfa3zurz9PPb2519zraQ+bduTT35RSX1XA3dn14HCKrfNb4J6C/SC4lJ6jZ3FjCWb2bY3n78vzOGiJ+ZQVFLK/oLi+qpWRGmpU1Ja9R9u5/5Ctuyu2eHQ1RPmkvnQu2Q+9O5XrV5g3f7qZ6zato8d+wtjLj/v8Q94+oPVfLZhV6Vl+wuK2X2wiMLi0rquZkLu/cdS5q7Jq9Fz/r0ql90HiqovWMue//BL0kdPY+uefHYdiN329aXfw7Ppfu/bpI+eRmmcz6S7U1qHYbtpdz5Lv9zMg795stqy+/KLWL1tX2R66NCh7NpVeZtN1BOzVnL6g+/w7vKtccvV+5DL/VOXMfT3H0amf/ineZx8/8zIXvAPs1fR9Z7pkeV1safesOMAo19bzAP/XMbPJi6kx71vc8Ld0znx7ukx98buTp8H36H/I7MpKqk6PCp+MD+u4Yc7UflFJUzO2sBH2du58pmPqu1B1GYP48NV25m8YAOnPTAr7g4QYOK8dUyct47/nbmSdXn745Z97/NtcZcPH/+fctPzv9zB6Q+9w2kPzKLbr2ckVvkaunPyZwkHdFFJKS/OXcfICXMTXv+z/1rNf/15PqeNmwWE/k41/Vvd9+ZSLvjfD2r0HIBxb4WOKM98eDa9x71To+fe/uoistbuqPFrJqLizmXT7nyKq/jMLdm4m6Ubd1NQXMLqbfvihv/hKC119u7ZzYsvPFdpWUnJoVxyd9Zs38/+wmIOFobmT58+nZYtWx7W627bk88f3ssmb38hN/wt/t319R7oF5/cvtwQxPwvQxvGS3PXUVRSyhPvfEFRiZNfVEL66Gl0v/dt3lm+lTlf5JI+elq5P/jug0XVhgqEhnqGPfVhZMM49zfvM2nBBl74z1qmL9nCwaidxvodBwBYtGEXP/rTPE4dO5OMMYd2MF3vmUHfB98hffS0Sr3Ksg9mUUkp6aOnlVv25qKN1dazrNf0yvz1LNqwi39+tok1uYf2+qWlzrq8/Vz17Mf8aspifvCneSxYu5NJCzbEXe+zc9YAsDc85PVR9nZ+8cqnlT4A7s7ug1X3Fh+evoIf/Xkev3ptMbsPFjFxXpV3JANwzxtLueeNpTz1fjbnPf4B6/MOsHN/IQcLS1iTu69ceP1yyuJKz1+fd6BS/QBWbd3LVc9+TH7RoQ/6h6u2x61LPLsPFEU6Drl7C0gfPY300dN47ZMcRk6Yy9MfrI7UJ1YHY/eBIrreU3mnsnzTHtJHT6Ok1Pl4dR7ZUT24u99YwiMzDg0l7TpQSMaY6dz00icJ1/vd5Vv528fr+HJ75Z3l4pxdPPjW8ko7iJ37C/n5xMqvkT56WqRDMv79bG6d9Cklpc4n63eWK7d9XwFvfLqREc98nHA9q+PubNx1ECDm+1++eQ/FJaXs3F/I4pxdbNl9sNxw7cote9lfWBxZR9k643W+EtEwyfj9I2PZsPZLevfuzRlnnEH/cwYy9IoRnHrqqQBcccUV9D39dL4zaABTJv6F1eHPa3p6Otu3b2ft2rX06NGDG2+8kZNPPpmBFwxi997Qtl9Vdu2rwYhFvV+2WFUH5N43l3Hvm4fGe3N2Hvowz1iymdc/DQXirOVbuSqzM4s27OKKCr2233yvF727tKRru2bk7i2goLiUopJSLvztvwA46Z4ZDOreLm79yhqz4rqjlQV53wcr92xGTviYVVv3VZo/ZWEOw3t3YsHaHVwZ9WF4/tpMLuzenpteWsiMpVsAGPP6knLPXfvoZZWeF23M60u4ul+XKuv73Jw1HNM4mTGvL+Hinu2ZFT6Me/g7p9A89dA1r7+d9QVPvZ/Nh3ddQFqrJuXW8cu/f8bfF+aUm3ffm8u4sHs70lo14UBhMbOWbeXsk45l14FCcqI+XGUGPh4633BKpxYs3biHV27sX2WdAT7fsqfc9N1vLOGR7/Yid2/lcxI/+vM8Tji2Kdednc5/DUiPzHd39heW0KzRoU1/X0Exuw4UktaqCa/MXx9p7/fuPC+yrUR77O3Pufjk9gz67b/ol96ayT8dQN6+An760kKuyuxcaWe0YccBDhSWMPQP/wbgxLunl1v+l+vO4OV568vN++4fPwLg7WVb2H2wiGMaJ7MkZzfj3lrGi9efSWpyUqV6Rffetu7Jp32LVH42cSGrt+0nb38h2/cVMHPZFt694zxSk5PYub+QPjG22TJPzv6CX1/Wk8dnrgTgzUWbALj+nAy+27cTI57+uFznB2Db3nyOaZxMo4ZJ4eeEPqeXnNwBoFy93Z2Hpq3g2rPS6dy6SWRedIepKlv25PPY25/zZW78I70mKUkcLCqttCNrnJJEgxgnHXse14L7Lz8ZgLx9BWzcdZBTjjsGLLRzv3XMWLJXruCTTz7lhdemcfM13+e1dz/i3L49Q2321DPsIZX8gwf5wbcv5KKhwyg9rkXkva3YvIdVq1bxyiuv8MT//ZERV17F+Bcm8uNrrmHb3nx6dmxBw6RQP7uoJFTvWNtgVay+BvgbdezqHa+pfiwqWpfWTSI95mgpDRvU2bhp784t6ZfRmgnhXm1tmnf3IM58eHal+Q9ecQr3/mNplc9L9P2ecGxT3vvv84HQxhGr1xht/t2DaNciNTI94umPyFq3k6d+0IeT2jVjwZc7uPfNZdw6qCu/n70q7rqGntqB6Uu2VFvHeMZc2p0urZtw6akdASod5UBo5/bmoo3cOmlRletZ++hlkcdl6+jUsjH/GX0hAKfcP5N9BcV0atm4XK8uUR/edUHkZHgsT1x1GndM/qzG6412Wa+OzF2dR1648/DeneeR1qoJ495axhdb9vHqT/qXC8LU5AasGDckMq9d80Zsi9rxrX30spjtWVGvtGNYnLM7oTq2SG3Invxi+qW3pv+Jbbjh3Ax6jZ0Vs2yfLi2599s9IzuusrYf2K0tc77IrVR+8diL6TV2Fs8N60j7LicA8Ny/11Qb6PHECvWex7XgzsHdKC71clnTvkUqW/fks3HDen5x7fdZ8OlnvD5tFs/87jH+PPmfkXJPP/Eo7739FgCbctbz9Euv0b//AC7qdwrT3vuQ9Vt38NMffIdlKz5nzfb9PP/HJykuKmbUrf8NhK5scffI/1vXr+HGqZvL1XHdY99e6O6Zsd5TQj10MxsC/B5IAv7k7o9WWN4F+CvQMlxmtLtXv5utoVhhDtTpSbBFG3axKMYJuHhuvuAknno/O+ay2y7qypPvhsIwVpgDccMcEn+/a7bvp6iklOSkBuTtq/5E179XbWfsP5fRtV0zPll/6D3f/PKn5cpVF+bAVw5zIDIEMfknAzgjvVVk/gvXncF1LywAQofXd1c4gonlD7NX8cQ7X0SmN+46yBXj/1Pub3s4YQ7EDXOA0QnUrzrTFpf/UFfsta3YXP4Ksfyi0nIBv63CUUyi21CiYQ6wJz90NDt/7Q7mr93BH+JsJ5+u3xUJczjU9rHCHKBFajKrHx7Ke3MPbYs3nntCwnWriTUxhqy27il/AcSGcBY1bnLoyHXBxx8y98MP+Nubs2jcuAnXX/ltCgoKOFBYTHGpR4aWk1NSIq+R1CCJgpJD6y7rYMfqaPfLaB0Zkq5KtWPoZpYEjAcuBXoCV5tZzwrFfg1Mdvc+wEjgj9Wt90h2z9AevHbTABaPvTjm8s6tG3P/5RWbCO4c3I3bLupW19Urp+s9M0gfPY3+j8TeeUS78++fsTe/uFyYV+ekds2+SvUSUlxayktRQxIXfOvQMNklT85hf2H8E+W7DxSVC/MyNd1Rv3DdGTUqX+bruOqmbDgnUbFOHH/+4JDaqk6dSGqQ2HXZvdIO7+RjIlq2aMGB/ZWHUAH27dlDi2Na0rhxE77M/oLFnx4aAquNkZBJ1QxJQmInRfsB2e6+xt0LgUnA8AplHGgRfnwMsKkG9QQOb2PqEDU88FX17XJoI7jk5PZxy9448AROP741LVJjf8dC3y6t6N6hRaX5vxjUFYDTj29Vadkt4WUVvR8eMvmmOuvENjUq/5OBJzD/7kHlhkEqmnjDmeWmP1iZW+1RC8C1Z6Uz/+5B/PbK08rNLzs5fbgm3nAmax+9jAu+1Y5Ztw9M+HlTbz477vJJo2J/QD+7L3ZHoa6lJiex9tHLuO7s9Hp5/WjP/KjvYT2vbOy+YRXh36V1k5jzE9W8ZSt6Z57JdwcN4HcP3VfuXMzZ5w+ipLiYEYPP5pknHmFA/+oDuCYaNLBqMy+RIZdOQPRlEznAmRXKjAVmmdkvgKbARbFWZGajgFEAKR3K/1pRanISqx8eyp6DRbz3+TaOa9mYOatyI1cUlHnh2jO47i+hw+0tUYdBZ53Yhr35xSzZeOgQccTpaRQWlzL1s/L7l7/+uB/Ht27C+eHLu+aOGcSxzVI4KTzG/OT3+7Bs024aNUxi8cZd3PNG1WHyxx/2pVWTFJo1asjlT4UuvzypXTO6tjce/s6pnNC2KSMnzC138nXSqP6VxrPvGNyN7/XtxHmPfxCZVxZ6ax+9jGmLN/Pzlyuf8c/69UVs21PASe2aRXpdLVIbUlLqMXuuGcc2rXQVRNavLzrsa+MfGHYyTRs15Nqz0nnmX6v5YGVuzKssylx/TkZknP7t285lyJOHepaDe7bnmgHplXZ47k7vzi2r7VGPHRY6mfW909O4ok8nbn75k8iJ5erMuPVcLv19qC4fj7mQ/3svm5fnree0zi05+6RjI+W6tW/Ou3ecx4wlmzk17RiuDQ/9xFrft9o3rzQ/OckoKgn11k5La8nSBy7hwv/9gN99v3fkddydjsekcsO5J/DgW+VvUPtu3068/knVV0iVbTNnPTKbTQncJ3HtWen8+OwMurQ5FHT3X34yL/xnbbly0285l4LiEnJ2HuSd5VsrfaZqQ9d2zVgVvvJnyCkdI/PHRh3ttmhcPrJ6dGzBtr0FHCws4UBhMd3ah44Yi2NcMdKmWSNaNknhmMbJFJc6q7fto7CGV75079CcR5/6U2S6V1pLtu7JZ+uefFIaNeKPL04B4NROx2BmLM4JbbMzPg6dKO9yXAden33oYoaHx97Nyi2Vb6gsy5St4QPTKT8dAMD4H/Yh856q65dIoMfa1VVsrauBv7j7b81sAPCimZ3i7uVay90nABMgdFK04kqTGhitmqbwvdPTABhwYhvuGtKdTbsOctaj7/G9vmlcEOOqlOaNGvLyjf0pLC4tdyj5yHdPJcmMWwZ15a3Fm+iX0ZrRry3h9ONblbsBocMxoYCZdftAVmzeQ+OUJDLTWwNwatoxXH1GF/61Kjcybhtt6KmHNryKvc4fnBm60uS9O88r1zNITmrAHYO7VRoGOL5NU9LbNGFt3gE+qNAzv6xXR9bmfYvHZ66kWaOGLH3gksiyY5s1AmDNw0OB0J58X0Exp9w/s1J9//HzszntgVCP9cO7LqBRwySap8beDJ79r9P5yYsLYy6DULubGXcN6Q6EguD+y+HsR98rNx79f1f3YU3ufvqf0LrcSdfuHVpwy4Uncd632lUK8bWPXka//3mXbXsLeO7flb9G4cZzM2LOL5PUwGjTLCXmsuG9j4tcsVGmW1T4tm6awsPfOZWbzjsx5jpOatcscrS1fNwlvLN8K7dOWsRbvziHUzodU2WdAAZ2bcvs8DX2jVNCvcn595Tv/5gZH48ZBMA1A45nT35x5AqqscNOjhvoZT4aM4ju984odynnv391Ac0aNSx3ZUvZTrCiSaP6l7uGvkfH5pgZfbq0onlqQ6Z+tombzj8x8rd/Z/lWzu16LOvyDnDJk3MizzsjvRWGMX/tDl676SxWbtnL8s27eWlu+at6zkhvxaujBnBC1BVAXz4ylHlf7uDMjNaReS1Sk+mR1pLtewto2SSZhkkN6NSycaX6p7dpytoK9zq0ax76nJgZyUlGtw7NWRruALZITS5353rz1OTIZb1lundoHrkCpeJ6WzdNYcXm0FVYqclJkdv2K64nvU0T9ocv0+3WvjnJUetLa9UkcjVf59ZNcHeSk4zfj+wdyaPTjz/UFrFUe5VLOKDHuvsl4ekxAO7+SFSZZcAQd98Qnl4D9Hf3Ku8OadSxq/9t6mwu6tGenQcKOald5d5MVcrOAj8yfQXPzlnDovsG07JJ6IOXvW0fDRsY6cc2rXYdGWOmc0Xv43hyZJ+EXvetxZvYl1/MyDiXBCZqb34RI57+mCsz0xjYrW25QKmKu/PHD1Zz/TkZMS9bq1i24uVfax4eSoMqDkWvevZj5n+5g0mj+tM0pSGvfZLD/Zf3jPR0nnz3C8a/v5rxP+gbOVJY/fDQKsc1563Jo1OrxpUud6ypildifHLvYFo3Tam08373jvMqjefvLyjm5Bg7tUX3Dab3uHcYeUbnyDX7ax+9jH0FxWzZnX9Y5wVKSz1m227bm8/W3QX06NicP8xexe2Du9Hv4dkkNzA+Cod2IgqLS0lOMtzhhLun07tzS+67vCetm6REjjSf/mHfyBVB0fWaOH89V56eFtlm3l66hac/yObxK0+Lu91t31dAm6Yplb5TxN2ZtmQzg3u2jwxx1NQNf13Auyu20a55o3I7tFfmr6dpo4YMO+24mM9bsWIFPXr0SOg1Vm7Zy7HNUmjdNIVSd5IaVA7jguISdh0ool3zRuQXlXCwqJTU5AY0SWlIqTsGkaP+srH5zbsPRi6VjR6vL+uNn9SuGU1SQp2k4pJSlm8+dLltrPH9suf1SmsZybZ479fMqrzKJZFAbwh8AQwCNgILgB+4+7KoMjOAV939L2bWA5gNdPI4K2/UsasXbK7+aom6tK+gmNSGDWLudYNgzOtLWLllD8e3acotg7qSEWcn5+4s2rCL3p1bxvxSoJJSZ9Oug5Hrhb8uFQM93th7det46xfnUFhSSt8urdhXUEyTqJ1iVTu6b6I1ufto3yKVpuHx2yfe+YK5a/KY/JMB9VyzxO0+UMTT/1rNHYO7kdIw8c9fTQK9thwsLKbUibR3WUgf17Jx5OgYQifvDxaWlLuXo8y6vP10atk4ZtZEB3pFtR7o4RUMBZ4kdEni8+7+P2Y2Dshy96nhq16eA5oRGo75lbvHPRPV+vgevmPdimpfW45uE+as5uHpocsXPxp9IcfFOLyuzvJNe9iy5yAXdo9/slu++eoj0OvawcJiGpjRKMZRd50Eel3IzMz0rKz430sgAoS/4Q5aNY09Ji5HjyAGejw1DfR6v/VfpDpl50dEJL5gDh6LiHwDNGsWOsG+adMmRowYEbPM+eefT22NVijQRUTq2HHHHceUKVPq/HU05CIikqC77rqL448/np/97GcAjB07FjNjzpw57Ny5k6KiIh566CGGDy9/M/3atWv59re/zdKlSzl48CDXXXcdy5cvp0ePHhw8eHjfIRSLAl1EjkwzRsOWr/7FZ+V0OBUufbTKxSNHjuS2226LBPrkyZN5++23uf3222nRogXbt2+nf//+DBs2rMrfBH366adp0qQJixcvZvHixfTte3hfcxCLAl1EJEF9+vRh27ZtbNq0idzcXFq1akXHjh25/fbbmTNnDg0aNGDjxo1s3bqVDh06xFzHnDlzuOWWWwDo1asXvXr1qrX6KdBF5MgUpyddl0aMGMGUKVPYsmULI0eOZOLEieTm5rJw4UKSk5NJT08nPz/+9+hU1Xv/qnRSVESkBkaOHMmkSZOYMmUKI0aMYPfu3bRr147k5GTef/991q2L/1OMAwcOZOLEiQAsXbqUxYsr/9zi4VIPXUSkBk4++WT27t1Lp06d6NixIz/84Q+5/PLLyczMpHfv3nTv3j3u82+66Sauu+46evXqRe/evenXr1+t1U13iorIEUN3isa/U1RDLiIiAaFAFxEJCAW6iBxR6muY+Ot2OO9TgS4iR4zU1FTy8vICH+ruTl5eHqmpNfvdZF3lIiJHjLS0NHJycsjNza3vqtS51NRU0tLSavQcBbqIHDGSk5PJyMio72p8YyU05GJmQ8xspZllm9noGMt/Z2aLwv++MLP4P88uIiK1rtoeupklAeOBwUAOsMDMprr78rIy7n57VPlfAIn96rKIiNSaRHro/YBsd1/j7oXAJGB4nPJXA6/URuVERCRxiQR6J2BD1HROeF4lZnY8kAG8V8XyUWaWZWZZR8NJDRGRr1MigR7ra8GqumZoJDDF3UtiLXT3Ce6e6e6Zbdu2TbSOIiKSgEQCPQfoHDWdBmyqouxINNwiIlIvEgn0BUBXM8swsxRCoT21YiEz+xbQCvi4dqsoIiKJqDbQ3b0YuBmYCawAJrv7MjMbZ2bDoopeDUzyoN/CJSLyDZXQjUXuPh2YXmHefRWmx9ZetUREpKZ0p6hItPzdkLc69NgMsPD/VPHYypet9nlVlK2N14s8ppbrHON5dfQTavLVKNBFouUsgJe+V9+1OMLU9k7PwsXqYKcX2Q/VxY4uRp3rpDNQNQW6SLQOp8HVrwIO7kSu0I31OHK6qKrHh/O8mr5eFWVr7bXjvb9vehtxmM9L9PWopXpW0y7uMZ4XmwJdJFqztvCtIfVdC5Gq3VB1L13fhy4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBIKdDMbYmYrzSzbzEZXUeYqM1tuZsvM7OXaraaIiFSn2m9bNLMkYDwwmNAPRi8ws6nuvjyqTFdgDHC2u+80s3Z1VWEREYktkR56PyDb3de4eyEwCRheocyNwHh33wng7ttqt5oiIlKdRAK9E7AhajonPC9aN6Cbmf3HzOaaWcwvlDazUWaWZWZZubm5h1djERGJKZFAj/Vt6hV/NqMh0BU4H7ga+JOZtaz0JPcJ7p7p7plt27ataV1FRCSORAI9B+gcNZ0GbIpR5k13L3L3L4GVhAJeRES+JokE+gKgq5llmFkKMBKYWqHMP4ALAMzsWEJDMGtqs6IiIhJftYHu7sXAzcBMYAUw2d2Xmdk4MxsWLjYTyDOz5cD7wC/dPa+uKi0iIpWZV/Mr0nUlMzPTs7Ky6uW1RUSOVGa20N0zYy3TnaIiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCASCnQzG2JmK80s28xGx1h+rZnlmtmi8L8bar+qIiIST8PqCphZEjAeGEzox6AXmNlUd19eoeir7n5zHdRRREQSkEgPvR+Q7e5r3L0QmAQMr9tqiYhITSUS6J2ADVHTOeF5FX3PzBab2RQz6xxrRWY2ysyyzCwrNzf3MKorIiJVSSTQLca8ir8s/U8g3d17Ae8Cf421Inef4O6Z7p7Ztm3bmtVURETiSiTQc4DoHncasCm6gLvnuXtBePI54PTaqZ6IiCQqkUBfAHQ1swwzSwFGAlOjC5hZx6jJYcCK2quiiIgkotqrXNy92MxuBmYCScDz7r7MzMYBWe4+FbjFzIYBxcAO4No6rLOIiMRg7hWHw78emZmZnpWVVS+vLSJypDKzhe6eGWuZ7hQVEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGRUKCb2RAzW2lm2WY2Ok65EWbmZhbzy9dFRKTuVBvoZpYEjAcuBXoCV5tZzxjlmgO3APNqu5IiIlK9RHro/YBsd1/j7oXAJGB4jHIPAr8B8muxfiIikqBEAr0TsCFqOic8L8LM+gCd3f2teCsys1FmlmVmWbm5uTWurIiIVC2RQLcY8yK/LG1mDYDfAXdWtyJ3n+Dume6e2bZt28RrKSIi1Uok0HOAzlHTacCmqOnmwCnAB2a2FugPTNWJURGRr1cigb4A6GpmGWaWAowEppYtdPfd7n6su6e7ezowFxjm7ll1UmMREYmp2kB392LgZmAmsAKY7O7LzGycmQ2r6wqKiEhiGiZSyN2nA9MrzLuvirLnf/VqiYhITelOURGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAkFupkNMbOVZpZtZqNjLP+pmS0xs0Vm9qGZ9az9qoqISDzVBrqZJQHjgUuBnsDVMQL7ZXc/1d17A78Bnqj1moqISFyJ9ND7AdnuvsbdC4FJwPDoAu6+J2qyKeC1V0UREUlEIr8p2gnYEDWdA5xZsZCZ/Ry4A0gBLoy1IjMbBYwC6NKlS03rKiIicSTSQ7cY8yr1wN19vLufCNwF/DrWitx9grtnuntm27Zta1ZTERGJK5FAzwE6R3SDAkIAAAiNSURBVE2nAZvilJ8EXPFVKiUiIjWXSKAvALqaWYaZpQAjganRBcysa9TkZcCq2quiiIgkotoxdHcvNrObgZlAEvC8uy8zs3FAlrtPBW42s4uAImAncE1dVlpERCpL5KQo7j4dmF5h3n1Rj2+t5XqJiEgN6U5REZGAUKCLiASEAl1EJCAU6CIiAaFAFxEJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCQW6mQ0xs5Vmlm1mo2Msv8PMlpvZYjObbWbH135VRUQknmoD3cySgPHApUBP4Goz61mh2KdAprv3AqYAv6ntioqISHyJ9ND7AdnuvsbdC4FJwPDoAu7+vrsfCE/OBdJqt5oiIlKdRAK9E7AhajonPK8q1wMzYi0ws1FmlmVmWbm5uYnXUkREqpVIoFuMeR6zoNmPgEzg8VjL3X2Cu2e6e2bbtm0Tr6WIiFSrYQJlcoDOUdNpwKaKhczsIuAe4Dx3L6id6omISKIS6aEvALqaWYaZpQAjganRBcysD/AsMMzdt9V+NUVEpDrVBrq7FwM3AzOBFcBkd19mZuPMbFi42ONAM+DvZrbIzKZWsToREakjiQy54O7TgekV5t0X9fiiWq6XiIjUkO4UFREJCAW6iEhAKNBFRAJCgS4iEhAKdBGRgFCgi4gEhAJdRCQgFOgiIgGhQBcRCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBkVCgm9kQM1tpZtlmNjrG8oFm9omZFZvZiNqvpoiIVKfaQDezJGA8cCnQE7jazHpWKLYeuBZ4ubYrKCIiiUnkJ+j6AdnuvgbAzCYBw4HlZQXcfW14WWkd1FFERBKQyJBLJ2BD1HROeF6NmdkoM8sys6zc3NzDWYWIiFQhkUC3GPP8cF7M3Se4e6a7Z7Zt2/ZwViEiIlVIJNBzgM5R02nAprqpjoiIHK5EAn0B0NXMMswsBRgJTK3baomISE1VG+juXgzcDMwEVgCT3X2ZmY0zs2EAZnaGmeUAVwLPmtmyuqy0iIhUlshVLrj7dGB6hXn3RT1eQGgoRkRE6onuFBURCQgFuohIQCjQRUQCQoEuIhIQCnQRkYBQoIuIBIQCXUQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGAUKCLiASEAl1EJCAU6CIiAZFQoJvZEDNbaWbZZjY6xvJGZvZqePk8M0uv7YqKiEh81Qa6mSUB44FLgZ7A1WbWs0Kx64Gd7n4S8DvgsdquqIiIxJdID70fkO3ua9y9EJgEDK9QZjjw1/DjKcAgM7Paq6aIiFQnkd8U7QRsiJrOAc6sqoy7F5vZbqANsD26kJmNAkaFJwvMbOnhVPoociwV2lDKUftUT20U35HYPsdXtSCRQI/V0/bDKIO7TwAmAJhZlrtnJvD6Ry21UXxqn+qpjeILWvskMuSSA3SOmk4DNlVVxswaAscAO2qjgiIikphEAn0B0NXMMswsBRgJTK1QZipwTfjxCOA9d6/UQxcRkbpT7ZBLeEz8ZmAmkAQ87+7LzGwckOXuU4E/Ay+aWTahnvnIBF57wleo99FCbRSf2qd6aqP4AtU+po60iEgw6E5REZGAUKCLiAREvQR6dV8lEGRmttbMlpjZIjPLCs9rbWbvmNmq8P+twvPNzP4QbqfFZtY3aj3XhMuvMrNrqnq9I4GZPW9m26LvS6jNNjGz08Ntnh1+7hF101sV7TPWzDaGt6NFZjY0atmY8HtdaWaXRM2P+bkLX/AwL9xur4YvfjhimFlnM3vfzFaY2TIzuzU8/+jbhtz9a/1H6MTqauAEIAX4DOj5ddejvv4Ba4FjK8z7DTA6/Hg08Fj48VBgBqHr/PsD88LzWwNrwv+3Cj9uVd/v7Su0yUCgL7C0LtoEmA8MCD9nBnBpfb/nWmifscB/xyjbM/yZagRkhD9rSfE+d8BkYGT48TPATfX9nmvYPh2BvuHHzYEvwu1w1G1D9dFDT+SrBI420V+d8Ffgiqj5f/OQuUBLM+sIXAK84+473H0n8A4w5OuudG1x9zlUvm+hVtokvKyFu3/soU/m36LWdUSoon2qMhyY5O4F7v4lkE3oMxfzcxfuaV5I6Cs7oHxbHxHcfbO7fxJ+vBdYQeju9aNuG6qPQI/1VQKd6qEe9cWBWWa2MPxVCADt3X0zhDZOoF14flVtdTS0YW21Safw44rzg+Dm8JDB82XDCdS8fdoAu9y9uML8I5KFvum1DzCPo3Abqo9AT+hrAgLsbHfvS+jbK39uZgPjlK2qrY7mNqxpmwS1rZ4GTgR6A5uB34bnH7XtY2bNgNeA29x9T7yiMeYFoo3qI9AT+SqBwHL3TeH/twFvEDoU3ho+rCP8/7Zw8ara6mhow9pqk5zw44rzj2juvtXdS9y9FHiO0HYENW+f7YSGHBpWmH9EMbNkQmE+0d1fD88+6rah+gj0RL5KIJDMrKmZNS97DFwMLKX8VydcA7wZfjwV+H/hs/L9gd3hQ8eZwMVm1ip8qH1xeF6Q1EqbhJftNbP+4fHi/xe1riNWWVCFfYfQdgSh9hlpoR+dyQC6EjqhF/NzFx4Tfp/QV3ZA+bY+IoT/rn8GVrj7E1GLjr5tqD7OxBI6y/wFobPu99T3meGv8X2fQOjqgs+AZWXvndA45mxgVfj/1uH5RujHRVYDS4DMqHX9mNAJr2zguvp+b1+xXV4hNGxQRKg3dH1ttgmQSSjwVgNPEb5D+kj5V0X7vBh+/4sJBVTHqPL3hN/rSqKuxqjqcxfeLueH2+3vQKP6fs81bJ9zCA2BLAYWhf8NPRq3Id36LyISELpTVEQkIBToIiIBoUAXEQkIBbqISEAo0EVEAkKBLiISEAp0EZGA+P9tnPhH2he+vgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.6070314049720764.\n",
      "Better model found at epoch 1 with accuracy value: 0.6105427145957947.\n"
     ]
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas1_sp15_labelsmoothing');\n",
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',\n",
    "                                                   monitor='accuracy',name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas2_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.317804</td>\n",
       "      <td>0.280220</td>\n",
       "      <td>0.951968</td>\n",
       "      <td>0.972167</td>\n",
       "      <td>03:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.314769</td>\n",
       "      <td>0.274971</td>\n",
       "      <td>0.953008</td>\n",
       "      <td>0.972907</td>\n",
       "      <td>03:45</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU1fnA8e9LyEZYE3YCJGCQTQgQdkQURBYBW9GCdcGflmqLa6vGBURqldqqtYq1aGltqyLFWqgEEBQERJYgW9gDBBPCEpZAgCRkOb8/ZjJMJjOTCUwy5Ob9PE8e7tx75t53Lsl7z5x77jlijEEppVT1VyvQASillPIPTehKKWURmtCVUsoiNKErpZRFaEJXSimLqB2oA4dENDDdOsUF6vBKKVUtbdq06YQxpom7bQFL6GGRzUlOTg7U4ZVSqloSkUOetmmTi1JKWYQmdKWUsghN6EopZREBa0NXSqmKKigoICMjg7y8vECHUunCwsKIjo4mODjY5/cELKHrCDJKqYrKyMigXr16xMTEICKBDqfSGGM4efIkGRkZxMbG+vy+wDW5aEZXSlVQXl4eUVFRlk7mACJCVFRUhb+JaBu6UqpasXoyL3E5n1MTulJKWUTAErq2uCilqpvs7GzefffdCr9v1KhRZGdnV0JEpWkNXSmlfOQpoRcVFXl9X1JSEg0bNqyssBy026JSSvkoMTGR/fv3Ex8fT3BwMHXr1qVFixZs2bKFnTt3ctttt5Genk5eXh6PPfYYkydPBiAmJobk5GTOnTvHyJEjGTRoEGvXrqVVq1YsWLCA8PBwv8SnCV0pVS299L8d7Mw869d9dm5ZnxfHdPG4febMmaSkpLBlyxZWrlzJ6NGjSUlJcXQtnDNnDpGRkeTm5tK7d29uv/12oqKiSu1j3759fPLJJ7z//vvceeedfPbZZ9x9991+iV8TulJKXaY+ffqU6if+pz/9ic8//xyA9PR09u3bVyahx8bGEh8fD0CvXr1IS0vzWzyBS+h6V1QpdQW81aSrSkREhGN55cqVLF++nO+++446deowZMgQt/3IQ0NDHctBQUHk5ub6LZ4A9nLRjK6Uql7q1atHTk6O221nzpyhUaNG1KlTh927d7Nu3boqjk6bXJRSymdRUVEMHDiQrl27Eh4eTrNmzRzbRowYwXvvvUe3bt249tpr6devX5XHJ8YEpqYc0aqDOX94b0COrZSqnnbt2kWnTp0CHUaVcfd5RWSTMSbBXXnth66UUhbhU0IXkREiskdEUkUk0c32NiKyQkQ2i8g2ERnl/1CVUkp5U25CF5EgYBYwEugMTBSRzi7FXgDmGWN6ABOAcp+N1VuiSinlX77U0PsAqcaYA8aYi8BcYJxLGQPUty83ADL9F6JSSilf+JLQWwHpTq8z7OucTQfuFpEMIAl4xN2ORGSyiCSLSLIp1jq6Ukr5ky8J3d2gvK7ZeCLwd2NMNDAK+KeIlNm3MWa2MSbBGJNQXPFYlVJKeeFLQs8AWju9jqZsk8oDwDwAY8x3QBjQuLwdZ+Xk+xalUkpVQ3Xr1gUgMzOT8ePHuy0zZMgQkpOT/XI8XxL6RiBORGJFJATbTc+FLmV+AIYCiEgnbAk9q7wdnzyvCV0pZX0tW7Zk/vz5lX6ccp8UNcYUisgUYCkQBMwxxuwQkRlAsjFmIfAr4H0ReQJbc8wk48MTS8FB2g1eKVV9PPPMM7Rt25Zf/OIXAEyfPh0RYdWqVZw+fZqCggJefvllxo0r3W8kLS2NW2+9lZSUFHJzc7n//vvZuXMnnTp18utYLj49+m+MScJ2s9N53TSn5Z3AwIoefFtGNu2b1K3o25RSChYnwtHt/t1n8+tg5EyPmydMmMDjjz/uSOjz5s1jyZIlPPHEE9SvX58TJ07Qr18/xo4d63FO0D//+c/UqVOHbdu2sW3bNnr27Om38ANaRX7i060UaW8XpVQ10aNHD44fP05mZiZbt26lUaNGtGjRgueee45u3boxbNgwDh8+zLFjxzzuY9WqVY7xz7t160a3bt38Fl/AB+f6dGM6d/VtE+gwlFLVjZeadGUaP3488+fP5+jRo0yYMIGPPvqIrKwsNm3aRHBwMDExMW6HzXXmqfZ+pQLeiL33mPuhKJVS6mo0YcIE5s6dy/z58xk/fjxnzpyhadOmBAcHs2LFCg4dOuT1/YMHD+ajjz4CICUlhW3btvkttoDX0ONbV/7EqUop5S9dunQhJyeHVq1a0aJFC376058yZswYEhISiI+Pp2PHjl7f//DDD3P//ffTrVs34uPj6dOnj99iC3hCz8kvJCZxEfcPjLkqZiBRSqnybN9+6WZs48aN+e6779yWO3fuHGCbJDolJQWA8PBw5s6dWylxBazJpWVD2yzXU/9r+5B/+zYtUKEopZQlBCyhR0WEBOrQSillSQG/Keos+8LFQIeglLrKBWqWtap2OZ/zqkro8TOWUVhkG7arqNiwT3vAKKWchIWFcfLkScsndWMMJ0+eJCwsrELvC+hN0Z5tGvL9D9ml1l3z/GLSZo6m07QlXCws5ukR1/KLIdcEKEKl1NUkOjqajIwMsrLKHSqq2gsLCyM6OrpC7wnYJNEJCQkmOTmZQyfPc+jkBe6ds8Fj2bSZo6swMqWUunp5myQ64N0W20ZF0CA82GuZ4mJDrVqV82SVUkpZxVXRht6wTghvT+zhcfsfv9qHMUZvmiqllBcBb3JxlldQRMepS7y+LzhI2PfbUZUZmlJKXbW8NblcFTX0EmHBQY7lOxOiWf7kDWXKFBRZ++62UkpdLp/a0EVkBPAWtgkuPjDGzHTZ/iZwo/1lHaCpMeayBmlxvgFa0oXR1f+2ZjKme8vL2b1SSllWuTV0EQkCZgEjgc7ARBHp7FzGGPOEMSbeGBMPvA38xx/B1fYwo9Ejn2y2fD9UpZSqKF+aXPoAqcaYA8aYi8BcYJyX8hOBT/wRHEDf2EjH8s9vaOdYnrUi1V+HUEopS/AlobcC0p1eZ9jXlSEibYFY4GsP2yeLSLKIJPv6YMCvb7kWgEeHxpE44tKwlH/4cq9P71dKqZrCl4TurgO4p/aOCcB8Y0yRu43GmNnGmARjTEKTJk18CrB3TCSfPdyfx4bGlZnlIyZxEWcuFPi0H6WUsjpfEnoG0NrpdTSQ6aHsBPzY3FKiV9tIguwPFn3xyKBS27rP+NLfh1NKqWrJl4S+EYgTkVgRCcGWtBe6FhKRa4FGgPuR3v2ka6sGlbl7pZSqtspN6MaYQmAKsBTYBcwzxuwQkRkiMtap6ERgrqmC7icHX9UHi5RSypVP/dCNMUlAksu6aS6vp/svLO8qa8ZspZSqzq6qJ0UrYuuLw4lrWjfQYSil1FWj2ib0BuHB7Dtum4D1bJ72dFFKqWqb0AGesvdRzy9wP0SAUkrVJNU6oTetFwrYRmlUSqmarlon9JLRGTWhK6VUNU/o4Y6Erk0uSilVrRP6qfO2GYzGvLMmwJEopVTgVeuEbjwOKaOUUjVPtU7o43u1Lr+QUkrVENU6oZcM2AVw4lx+ACNRSqnAq9YJ3dm3qScCHYJSSgWUZRL6lvTsQIeglFIBVe0TeskUdSmHzwQ4EqWUCqxqn9Cvs4+Pvl0TulKqhqv2Cb1kwgt9uEgpVdNV+4Qe0zgi0CEopdRVwaeELiIjRGSPiKSKSKKHMneKyE4R2SEiH/s3TM+iG4VX1aGUUuqqVu6MRSISBMwCbsY2YfRGEVlojNnpVCYOeBYYaIw5LSJNKytgV43rhlbVoZRS6qrmSw29D5BqjDlgjLkIzAXGuZT5GTDLGHMawBhz3L9hevf4sDgAiop1KAClVM3lS0JvBaQ7vc6wr3PWAeggIt+KyDoRGeFuRyIyWUSSRSQ5Kyvr8iJ2I/uCbcairRnaF10pVXP5ktDdzcjsWhWuDcQBQ4CJwAci0rDMm4yZbYxJMMYkNGnSpKKxerT+4CkAnp6/zW/7VEqp6saXhJ4BOI+CFQ1kuimzwBhTYIw5COzBluCrxEM3tAMg1T7HqFJK1US+JPSNQJyIxIpICDABWOhS5r/AjQAi0hhbE8wBfwbqzbBOzarqUEopddUqN6EbYwqBKcBSYBcwzxizQ0RmiMhYe7GlwEkR2QmsAJ4yxpysrKBdRYSW21lHKaUsz6dMaIxJApJc1k1zWjbAk/YfpZRSAVDtnxRVSillY5mEPqZ7y0CHoJRSAWWZhB7bOAIRKNaHi5RSNZRlEnrd0CCMgQsFRYEORSmlAsIyCb2kp8v5/MIAR6KUUoFhmYRe157Qz2lCV0rVUJZJ6CXjuUz9b0qAI1FKqcCwTEJPP3UBgLX7q+x5JqWUuqpYJqE/P7oTAHf0ig5wJEopFRiWSegiQmzjCPIKdW5RpVTNZJmEDlA/PJjsCxcDHYZSSgWEpRJ6w/BgzuYWBDoMpZQKCEsl9HphtcnJ026LSqmayVIJ/XhOPgdOnA90GEopFRCWSugb7FPRFRbpjVGlVM3jU0IXkREiskdEUkUk0c32SSKSJSJb7D8P+j/U8j0zoiMA5/N1PBelVM1T7gQXIhIEzAJuxjZ36EYRWWiM2elS9FNjzJRKiNFnkRHBAOTkF9CgTnAgQ1FKqSrnSw29D5BqjDlgjLkIzAXGVW5Yl6duqC2Jaw1dKVUT+ZLQWwHpTq8z7Otc3S4i20Rkvoi0drcjEZksIskikpyVlXUZ4XoXERoEwLl87bqolKp5fEno4mad6ywS/wNijDHdgOXAh+52ZIyZbYxJMMYkNGnSpGKR+qBkCN2Uw2f9vm+llLra+ZLQMwDnGnc0kOlcwBhz0hiTb3/5PtDLP+FVzKnztqdEX1y4IxCHV0qpgPIloW8E4kQkVkRCgAnAQucCItLC6eVYYJf/QvTd8M7NAnFYpZS6KpTby8UYUygiU4ClQBAwxxizQ0RmAMnGmIXAoyIyFigETgGTKjFmj0TctQ4ppVTNUG5CBzDGJAFJLuumOS0/Czzr39CUUkpVhKWeFAUID7b1dMnTyaKVUjWM5RL6Qze0B+DombwAR6KUUlXLcgn9mqZ1ATh6VhO6UqpmsVxCLyy2Dcylk0UrpWoayyX0/u2iAOjYon6AI1FKqapluYTetH4Y4cFBNKsXGuhQlFKqSlkuoQPkFhTx1e7jgQ5DKaWqlCUTOsBBnblIKVXDWDahg85cpJSqWSyd0M/l64TRSqmaw5IJvWQquqyc/HJKKqWUdVgyoS/abhvd9+Y3VwU4EqWUqjqWTOjDOzcPdAhKKVXlLJnQ+8RGBjoEpZSqcpZM6D3aNHQsz93wQwAjUUqpquNTQheRESKyR0RSRSTRS7nxImJEJMF/IVZcaO0gro9rDEDif7YHMhSllKoy5SZ0EQkCZgEjgc7ARBHp7KZcPeBRYL2/g7wcq/edCHQISilVpXypofcBUo0xB4wxF4G5wDg35X4DvAZcdePWarOLUqom8CWhtwLSnV5n2Nc5iEgPoLUx5gtvOxKRySKSLCLJWVlZFQ62Ij5+sK9j+f3VByr1WEopdTXwJaG7m3nZODaK1ALeBH5V3o6MMbONMQnGmIQmTZr4HuVlGHBNY8fy/iwd10UpZX2+JPQMoLXT62gg0+l1PaArsFJE0oB+wMJA3xgFmHLjNYEOQSmlqowvCX0jECcisSISAkwAFpZsNMacMcY0NsbEGGNigHXAWGNMcqVEXAH39m8b6BCUUqrKlJvQjTGFwBRgKbALmGeM2SEiM0RkbGUHeCUa1700ycX0hTuYtzGdmMRFrN2vPWCUUtZT25dCxpgkIMll3TQPZYdceVj+UavWpeb/v69Ncyy/v+oAA9rb2th3HTlL3dDatI6sU9XhKaWUX1nySdHyZOcWAHA4O5eRb63m+tdWcPKcbWTGecm2WvzP/pHMqr2le+JsSc/mpf/twBhTZp//+T6D/VnnKj94pZTywKcautUcP2tL3s/M3+ZY99MP1rNgykCetq9btvMYy3YeY+A1UQzt2IxdR87y700ZABQVG2aM6+p4b15BEU/O2wpA2szRVfUxPNqWkU2HZvUICw4KdChKqSpk+Rr6p5P7lVl3ODuXmMRFrEm91Ja++2gO176wpEzZb1NPMuOLnY5kDvCP7w6VKnMl464XFhWzcGum21r/5cjJK2DsO98y5ePv/bI/pVT1YfmE3rddVKXXml9dvMvnsifP5dP7t8spKrYl8B+9u5ZHP9nMh05t/M6Onc0jJnERW9Oz3W4vLCpmz9EcAL7adYzrpn8JwPJdOkm2UjWN5RN6ZVm+8xg5eba2+Nq1Lp3GmMRF/Mre/AK2bwOvLt7FfzcfZmt6Nr1eXk5WTj7tn0ti0bYjbD98BoCXvtjJlvRszrtMm9f3la8AGDfrW77bf5Jffvy9ozb/928PMnPxbm754yrWHzjJAx+67ymaV1BEgdP8qmknzjPqrdWcOn+RNftOEJO4iIzTF674nKzdf6LUcZRSVatGtqH7YvFj1zPyrdUetz/4D1vyPPjqKC5cLCq17bPvM3jlx10JrR3EwJlfe9zHL52aRYyB22Z9C1xqh5+z5mCp8hPfXwfAb8Z15eCJc0z/307Htp/MXufxOB2nLim137+s2s/OI2dJ2n6EF/6bAsDIP65m+0u3eNyHNyv3HGfS3zaWWndTx6bMmdT7svbnLxcLiwmprXUWVXPUmN/2ZvVtfdI/uDeBOiFB/PW+Sw+yPnXLtaXKHnx1FNc2q+fTfndknmX5rmNl1j/y8WZOnb942fGmnTjPjC92ut2291gOuRfLrwl/tesYcc9f6m36s38kM27Wt3yywTY0z2mn+HKuYEJt12QO8PVuW5PP43M3E5O4yNHEVFXe+2Y/HV5Y7LhIVoX9Wef47+bDbrdlZudy1v6NLpA2HTpNTOIizlwIfCzK/8RfN+MqKiEhwSQnV93DpDGJiwD45qkhtI2KAODQyfPUDqpFq4bhpW5Kitj6r3+w+gAvL/LePh4eHERuQZHXMhW1+ukbyS0oYngVzInarH4ox+y9ft6aEM8tXZoTElSLVfuy6NcuirDgIJK2H+EXH9m+TWx6YRhR9ge2Nh06zb/WHeJzD0ksbeZox3lv3ySCAe0bM2NcF8f59ZedmWdpVj/UERfAwJlfczg7F4C9L4+skpp63PNJFBQZDr46qsxnLDkP/r6fY4whv7DY5x5NJXFURiyqaojIJmOM26FVakyTy7pnhzLn24OOZA6UWnaXZB68vh3tm9Tl3ZWpTOzThifnbSUqIoSTTjVbfydzgKkLUmjfpK7f9+tO47qXEvpjc7eU2b7glwMdyRxsNe87Elpz8lw+t/95rdd9rztw0rG8P+s8+7POE9M4ggcGxXp8z4sLUmjVKJzJg9s71h0/m8eZ3ALinL41GWN4edEubotvxZh31gC2BPVt6gl6tW3kSOYAHV5YTNrM0RQVG4Jq+fdiUqKo2FBQZKsULNt5jOFdLs1rW5mVpjHvrCHl8Fm2vjics7kFNG8QRnBQjfnirVzUmBq6PznXckoM79yML3eWbXrxhyeGdeDN5XsrZd8V9fbEHozp3tLtOfDVgVdGMeyNb+jZthF/uKM7BUXFBAfVIjM7lwFO9xxKvk2VHOutCfEUG8OPekRzPr+QLi8uLbXfT37Wz3GfwdWLYzrzkv2ew4FXRlGrlpB94SLxM5YB8P3Um4mMCAFgwZbDPDZ3CzteuoWIUN/qPNMX7ij1NLJz7bfj1MXkFRSXWX8lzlwo4IM1B3j769Qy27wd43Jr6KnHz1E/rDZN64dVLFA/+3TjD2Tl5DPlpriAxhFI3mroein3k/fu7uVz2am3lpnwiZYNwtj64vAy638+uB2PDYsj3MNX6jHdW7J12nD+cEd3AGrXEiJCKu+Bokc+2UxOXgHDOjUrtf7Ga30bDvknCa25/rUVHDhxnvmbMliz7wRxzy8mJnFRqWQOcMPvV5L42aWHvx6bu4UnPrX1IHLXFOYpmQOOZA6QknmGlXuOs8WpK2jP3yxzLL+5zHbxnDB7HV/tOsaFi+XfX8jz8E1tw8FTjmQOtqEmoGK1dndl312Z6jaZg62psMS6AyeJSVzEkpQjLNhSumks+4Jv93gKiooZ9sY39LH3uAqUpTuO8sxn2/nDl75Xbmav2k/c80nkFRRd0T2tymKModjD/aWiYlPhp881oftJrVrC+ueGAvDKj64rtc15so1JA2J4YFAsb0/s4Vi34tdDWPvsUBqEB5fZ77OjOgHw5RODHetm32O7ePRrF8nbE3vQoE4w43tF88efxLPuuaEseXxwmf140zumUYXKH87OpW9spON18/phvH5nPMueGOy1OQXg0+T0Us0hd//V+4yFczeml1kXk7iIT65gFqqx73zLpL9tdHsz98LFQtJO2rpwbj98hgc+TKbztKWObc6Je1tGNnPWHCQmcZHbOAG+2JZZ6vXIt1YTk7iI2GdtN6vnbviBmMRFPPhhsuNGZcrhM7z8xU5uen0lczf8QOyzSRw9U3oisLN5ni8yLy/axco9xzlxLp8J9t5PD/3re9YfPFWq3D+dHpDLyskv03V1Z+ZZvtiWyZi315Q6P+64XnQKi4oprGAX1jeW7eXzzbYH+J6Zv40Vu0s/S/Hzf27yeV//25rJ/E0ZvJK0m4IiQ8epS+j5m2Us3n6kQjGVOHIml/xC782rxcWG7AsXOXjivM83wJ+av412zyWRe7GIi4Wlz9ecNQcZ+vo3pNi7Nt/+57X8+t9b3e3Goca0ofvTrLt68suPv2fNMzcy6HcrHOub1Q9zfI197nPb5NTv35vAgGsaM+iaxqxJPcEt9rbVMd1b0qFZPf665gBtnAYGe218N8fwA85KRo6MbhTO8C7N+X7qzdRxqYnf1sM2kVSuSzfKtJmj6f/qVxxxSgovjO7kqOVuTDtdoc8/4o+radUw3PF64SMDiYwIITIihBdGd+LEuXwWbMn0soer0+p9Wdzz1w1ut7k2VcxcvJv3vtnvdX/7juWUearY2aOfbGbhVtt5Wr7rGN1nfMnBV0dxq1MCLZnkfNXeLG7u3Iwev1nGh//Xh/ZNItzus4S7i9XH60tfBF9ftpcLBUX0atPI0Q036dHrWbUvi9HXtWDUn8p22+08bWmZppr0UxcY/PsV/G1SbwbHNWFzenap+ysfP9i31IQzxhimL9zBhx7Oza3dWvJpcjqfJqfz8m1duSMhuszv9NrUE8S3aUidkNIp7OCJ8xQVF/PIJ5vd7vvhj77n3Z/2ZNR1Ldxud+dcfiH9X7V9e9w89WYa2ZvmTp7LJzIiBBFxNNOVaN8kgq9+NQSwfXsr6ToMMH1MZyYNtFV85tufQO80zbb9f1MGcV10AwA+Wm87P1szsunaqgGbDp1m0yHvf6vahn6F7vnresZ0a8mdvVuXWl9YVIyIlLoJl1dQVG5vhOJiQ7vnLnU1dP7j+Xj9Dwy6pjFtosofGdI1AeXkFZD4n+1MH9OFqIgQatUSR5noRuGseeYmUo/nMOwNW8+aN+7s7hif5i/39KJHm4YsSTnKtAU7Sh1n64vDy3yzyL1YxI1/WMnRs7YLyHOjOvJK0u5yY/a3e/q15Z/rPCfUy/Xzwe34yyr30xq2iazDD6cu8M5dPZjy8aWk0r9dFN853ST25LXbu/H0Z2Uv6K6cL8gV9fkvBvCjd73f0Pbm7/f3Zsi1TQFb2/rmH07z1PxtjOzanE4t6vPGsrJNIgPaR/H3+/vw+rI9/OWbik0J2ToynPRTuW63Of99fLg2jRcX7nBbztmILs1ZsuMo/36oPwu3ZPLPdYeIighhw/PD3N40d71fFFK7Fosfu56hr3/DS2O7cN+AGB7+1yYWpxwtVe77qTdz+HSu46a9u7jd3Ysq2XbHe2sdla1XfnSdo5J46He3emxD14R+FZq5eDcFRcX8pHdrOvjYH95VyY24lb8eQkxj97W5M7kFTFuQwu/Hd3fbre+D1QcY3KGJIwZjjKOpoISnG2vOFyZvv7y+ePMn3R1t585ui2/J8Zx8nri5A3e8912Z7a/f0Z1fOX1F3fricLq/9GWpMn+/vzfbM87wupskdDkeHxbHH5fvK7PeuQunPzx1y7X8fumey3rvgVdGlao0XI537urBqfMXy1zga9cSCqvwmYNHh8YR17Suxxp5RcS3bsifJvSgdWQ4py8U8LvFu3l2VEfHjXNPxsW3pEF4sNdvY67WPTuUpO1H3D5rsvflkYCtd5Y7mtCV38zbmF6qBumtp8SK3ceJbRzhuKCknTjPkD+sdGxv0SCMI2fy6Ncuktdu786FgkKa1w+jYR3bV9qYxEV0j27AgimDHMlwbeJNjpunDw6K5QX7DeaS7XMmJdCwTgg/fncty54YzM32vvz/fqg/vWMi3fbycHeh6hsbWabNuTyPDo2jflhttzVnfyb0hLaNSLZ/9f74Z325633bfYi/3NPLbTvz7Ht6MdlpfdrM0by4IMVjk4cn9UJrl/sAWqM6wZzWh5YqlbeE7tNNUREZISJ7RCRVRBLdbH9IRLaLyBYRWSMiZbtxKEvYnO57e/uNHZuW+nYQ0ziCtJmjWfDLgax/bijfPTuUtJmjmTu5P22i6tCxeX1HMgfY8dItzH94AABbpw1n9dM30rJhOD/t2wawPSfgakD7xvRs04iDr44irlk9Nk+9mffu7kXvmMgyZUu4PoPw8YN9+fTn/TnwyiifP+ukATE8eXMH/vN92Yesrmnq/ZmCtybEl1m35+URHssnO7WjlkzUAnBLl+asf24owUGXPs/mqTeX6hNf4iWn4Z995cvTxK7JvF6Y99t0o7t5bst2dw5eHON7annv7p7Mf6i/z+UDaXAH33qJfeTUwcKdchO6iAQBs4CRQGdgopuE/bEx5jpjTDzwGvCGT9GpaueefjGO5aRHr7+sfXRv3ZBmPvRnjgit7XhIpkGdYMesUr/90XWkzRxN8waX9jH11s5c26weIfbyJUm6UUQII7peSmjLnxzM9DGd2e+SrL95agjv35tA2szRjht4tWoJq5++kXk/78/Lt7lPgJ893J+7+rbhsaG2ftE77d0SnS1/8gYAdv/GlqCeG9WRVU/dCMDDQ9ozLr5VmT/U0NpBvDS2C5ERIeyc4X2MnXLivBYAAAzsSURBVLSZox3fNprVD2Pfb0c51pXcwPvb/b1Z/uQNpb5ROSfHrS8O59mRHQH4sf3m+opfD6FfO88XwvJsmz6cd+7q6XH77Ht68cpt1zHETZfX9+7uRWjt0vebhlzbhEkDYnw6dt3Q2ozo2oKEmEhSf2trwvjikUG+B+8irpyLcnleGN3J47aJfdrwj//r4+gl56xZ/VCc6xsD2kd5PU65TS4i0h+Yboy5xf76WQBjzKseyk8E7jXGjPS2X21yqb6WpByhV9tImtQLLb+wxdw3ZwOFxcW8NLYrq/Zm8X8u3TSPn80r1V/7vbt7lbqglCqbk0eTuqGICAVFxcQ9b2szXf/c0DIXPE/NNZX9+H7uxSLu+9sGHrnpGhLaRjp6Y5Rn4ZSBdItuiDGGm99cxYguzXlnRSqv39Gd23tFlypb8kDZzB9fR+eW9cnMzmVEV1vN/e4P1jvmLSj5rCmHz3Do5AUGd2jsGC7a1QODYt0+77EtI5ux73zLqOuak7T9qJt3urf7NyN44MONGANr99tubvdq24hzeYXsOZZT7vtLhswYN+vbMkNh739lFEG1xG3T378e6EvXVvV5/r8p/GF8d8JDgrw+WORLQh8PjDDGPGh/fQ/Q1xgzxaXcL4EngRDgJmNMmTtDIjIZmAzQpk2bXocO+b8HglKBFpO4iC4t67Oogt9gMk5f4IdTF0o1o5QoLCrmrvfXsyHtUrv+WxPiGRff6orjraiUw2dYsfs4H36XxolzFwkOEtpE1mF/1nnA1tRRkpB9lVdQRGjtWmWav47n5DHm7TUsfmyw40leZyUXuhW/HkKLBmHcN2cD6w+eYuu04TSoU/a5jhJLUo7y0L9s9xVWP30jK/dmMdU+8qirSQNimD62S5ljOl9M3/tmP5MGxPD55sNknL5AeHAQ3aIbcn1c41KfaWfmWWatSOWlcV04ff4iDcKDSz19W7Lvu/q2KfM8S4krTeh3ALe4JPQ+xphHPJS/y17+Pm/71Rq6sqqsnHzqhdWulCkAe8z4ktMXCnh8WByPD+vg9/1XVH5hkaNp5O2v9vH6sr2lujVWtte/3EOLBuHcZb+vYozBmNITxLtTUFRMx6lL+PwXA+gW3RCAd77e53gKdcNzQwkLCaJ+WNmLwlP/3krbqDqVMvxAZnYuhUXGa9fkK03oFW1yqQWcNsY08LZfTehKWUt+YRFf7zrO8C7NK20QtMq2dv8J6oUGOx7uuRpd6WiLG4E4EYkFDgMTgLtcDhDn1MQyGijbEVep6iA/B7LTQWo5/YjLa0/bvJVzKmNRobWDGFmBJzCvRu6au6qTchO6MaZQRKYAS4EgYI4xZoeIzACSjTELgSkiMgwoAE4DXptblLpqpa+Hf91euccoL+GXLOPpAlHehcOX7f7Yh8u6MvH6eIHzaXslxHtZMfv6mQJzcfdpLBdjTBKQ5LJumtPyY36OS6nAaHYd3PEhmGL7j3Fa9vRjgPLKXe52U4F9+Hic4qJy4r2Sz2LKfz+BeZjxqnRZFyHPdHAupZzVawZdbgt0FNbmS9J3d+FwexHy00XOL/txKePxonmFF388D/mgCV0pVbVKmiR09O7L9I7HLXpGlVLKIjShK6WURWhCV0opi9CErpRSFqEJXSmlLEITulJKWYQmdKWUsghN6EopZRGa0JVSyiI0oSullEVoQldKKYvQhK6UUhahCV0ppSzCp4QuIiNEZI+IpIpIopvtT4rIThHZJiJfiUhb/4eqlFLKm3ITuogEAbOAkUBnYKKIdHYpthlIMMZ0A+YDr/k7UKWUUt75UkPvA6QaYw4YYy4Cc4FxzgWMMSuMMRfsL9cB0f4NUymlVHl8SeitgHSn1xn2dZ48ACx2t0FEJotIsogkZ2Vl+R6lUkqpcvmS0N3NZOp2UkARuRtIAH7vbrsxZrYxJsEYk9CkSRPfo1RKKVUuX6agywBaO72OBjJdC4nIMOB54AZjTL5/wlNKKeUrX2roG4E4EYkVkRBgArDQuYCI9AD+Aow1xhz3f5hKKaXKU25CN8YUAlOApcAuYJ4xZoeIzBCRsfZivwfqAv8WkS0istDD7pRSSlUSX5pcMMYkAUku66Y5LQ/zc1xKKaUqSJ8UVUopi9CErpRSFqEJXSmlLEITulJKWYQmdKWUsghN6EopZRGa0JVSyiI0oSullEVoQldKKYvQhK6UUhahCV0ppSxCE7pSSlmEJnSllLIITehKKWURmtCVUsoifEroIjJCRPaISKqIJLrZPlhEvheRQhEZ7/8wlVJKlafchC4iQcAsYCTQGZgoIp1div0ATAI+9neASimlfOPLjEV9gFRjzAEAEZkLjAN2lhQwxqTZtxVXQoxKKaV84EuTSysg3el1hn1dhYnIZBFJFpHkrKysy9mFUkopD3xJ6OJmnbmcgxljZhtjEowxCU2aNLmcXSillPLAl4SeAbR2eh0NZFZOOEoppS6XLwl9IxAnIrEiEgJMABZWblhKKaUqqtyEbowpBKYAS4FdwDxjzA4RmSEiYwFEpLeIZAB3AH8RkR2VGbRSSqmyfOnlgjEmCUhyWTfNaXkjtqYYpZRSAaJPiiqllEVoQldKKYvQhK6UUhahCV0ppSxCE7pSSlmEJnSllLIITehKKWURmtCVUsoiNKErpZRFaEJXSimL0ISulFIWoQldKaUsQhO6UkpZhCZ0pZSyCE3oSillEZrQlVLKInxK6CIyQkT2iEiqiCS62R4qIp/at68XkRh/B6qUUsq7chO6iAQBs4CRQGdgooh0din2AHDaGHMN8CbwO38HqpRSyjtfauh9gFRjzAFjzEVgLjDOpcw44EP78nxgqIiI/8JUSilVHl/mFG0FpDu9zgD6eipjjCkUkTNAFHDCuZCITAYm21/mi0jK5QRdgzTG5RyqUvT8lE/PkXfV8fy09bTBl4TurqZtLqMMxpjZwGwAEUk2xiT4cPwaS8+Rd3p+yqfnyDurnR9fmlwygNZOr6OBTE9lRKQ20AA45Y8AlVJK+caXhL4RiBORWBEJASYAC13KLATusy+PB742xpSpoSullKo85Ta52NvEpwBLgSBgjjFmh4jMAJKNMQuBvwL/FJFUbDXzCT4ce/YVxF1T6DnyTs9P+fQceWep8yNakVZKKWvQJ0WVUsoiNKErpZRFBCShlzeUgJWJSJqIbBeRLSKSbF8XKSLLRGSf/d9G9vUiIn+yn6dtItLTaT/32cvvE5H7PB2vOhCROSJy3Pm5BH+eExHpZT/nqfb3VquH3jycn+kictj+e7RFREY5bXvW/ln3iMgtTuvd/t3ZOzyst5+3T+2dH6oNEWktIitEZJeI7BCRx+zra97vkDGmSn+w3VjdD7QDQoCtQOeqjiNQP0Aa0Nhl3WtAon05EfidfXkUsBhbP/9+wHr7+kjggP3fRvblRoH+bFdwTgYDPYGUyjgnwAagv/09i4GRgf7Mfjg/04Ffuynb2f43FQrE2v/Wgrz93QHzgAn25feAhwP9mSt4floAPe3L9YC99vNQ436HAlFD92UogZrGeeiED4HbnNb/w9isAxqKSAvgFmCZMeaUMeY0sAwYUdVB+4sxZhVln1vwyzmxb6tvjPnO2P4y/+G0r2rBw/nxZBww1xiTb4w5CKRi+5tz+3dnr2nehG3IDih9rqsFY8wRY8z39uUcYBe2p9dr3O9QIBK6u6EEWgUgjkAxwJcissk+FAJAM2PMEbD9cgJN7es9nauacA79dU5a2Zdd11vBFHuTwZyS5gQqfn6igGxjTKHL+mpJbCO99gDWUwN/hwKR0H0aJsDCBhpjemIbvfKXIjLYS1lP56omn8OKnhOrnqs/A+2BeOAI8Lp9fY09PyJSF/gMeNwYc9ZbUTfrLHGOApHQfRlKwLKMMZn2f48Dn2P7KnzM/rUO+7/H7cU9nauacA79dU4y7Muu66s1Y8wxY0yRMaYYeB/b7xFU/PycwNbkUNtlfbUiIsHYkvlHxpj/2FfXuN+hQCR0X4YSsCQRiRCReiXLwHAghdJDJ9wHLLAvLwTutd+V7wecsX91XAoMF5FG9q/aw+3rrMQv58S+LUdE+tnbi+912le1VZKo7H6E7fcIbOdngtgmnYkF4rDd0HP7d2dvE16BbcgOKH2uqwX7/+tfgV3GmDecNtW836FA3InFdpd5L7a77s8H+s5wFX7udth6F2wFdpR8dmztmF8B++z/RtrXC7bJRfYD24EEp339H7YbXqnA/YH+bFd4Xj7B1mxQgK029IA/zwmQgC3h7Qfewf6EdHX58XB+/mn//NuwJagWTuWft3/WPTj1xvD0d2f/vdxgP2//BkID/ZkreH4GYWsC2QZssf+Mqom/Q/rov1JKWYQ+KaqUUhahCV0ppSxCE7pSSlmEJnSllLIITehKKWURmtCVUsoiNKErpZRF/D/c8mzX+YF95QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.9519680738449097.\n",
      "Better model found at epoch 1 with accuracy value: 0.9530084729194641.\n"
     ]
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas2_sp15_labelsmoothing');\n",
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',\n",
    "                                                   monitor='accuracy',name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas3_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.301391</td>\n",
       "      <td>0.272535</td>\n",
       "      <td>0.959598</td>\n",
       "      <td>0.976547</td>\n",
       "      <td>04:16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.291589</td>\n",
       "      <td>0.270674</td>\n",
       "      <td>0.961332</td>\n",
       "      <td>0.977425</td>\n",
       "      <td>04:29</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd5wU5f3A8c/3Kr2DIh1p0qQcCKKIgBRJxCgqdqM/jYUkakzEFhVjgjVGxUIiiTEiUWwYlSJFRerRuxxFOI5ejgOOq8/vj53dm92d3Z2FO+4Yvu/Xa1+3O/PM7LNzu9955mkjxhiUUkp5V0J5Z0AppVTZ0kCvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyuKTyzkCoxCo1TbtWLamckljeWVFKqdPGkiVL9hlj6jutq3CBPqlmAyZ9NYdOjWuWd1aUUuq0ISI/RVqnVTdKKeVxFTLQG3QQl1JKlZYKGehL0/4jeRQUFZd3NpRSqtxUuDr60mSMofufvuHn55/Da9d3Le/sKKXKSEFBAZmZmRw/fry8s1LmKlWqROPGjUlOTna9TYUM9KU1/U5eoa8k/8WKLA30SnlYZmYm1atXp3nz5ohIeWenzBhj2L9/P5mZmbRo0cL1dp6uusnNLyrvLCilToHjx49Tt25dTwd5ABGhbt26cV+5VOhA/9P+o3ywaNsJb3/93xeUYm6UUhWZ14O834l8zgoZ6O94dzEAP39tLo98sopoUyl/9+Netu0/5rhu/a6cMsmfUkqdTipkoN93JB+Aw8cLASgqjhzob5mwiL4vzD4l+VJKKSeHDh3ijTfeiHu7yy+/nEOHDpVBjoJVyEAPsP1ASSnd36ha1jL2HOFYfmHg9f9WZnHoWP4peW+l1OkrUqAvKoreTvjVV19Rq1atsspWQIUN9D/ZqmOOFzgfLHtQjiXWnbSMMQx8+Vvu+Fc6ADsO5TJq4jLum7jU9XuUpic+W02XMdMpjnI1UxEcLyjiw8XbYx5fpbxs9OjRbNq0iS5dutCjRw8uvfRSbrjhBjp16gTAlVdeSffu3enQoQPjx48PbNe8eXP27dvH1q1bOe+887jzzjvp0KEDgwYNIjc3t9TyVyG7VwLY2xuORyjR26t0jDHsO5LPW99uYvKSTOaN7h+UtqDIkJIUuRFjm3UFMX/zfgAKrUFW2w441/87WbTlANsOHGNE98aut3FyvKCI9xb4pq34YPE2bryg2UntryyN+d9aJi7cxjm1KnNR63onvb/s3AJSkxKolJwYtGzfkTzOrV/tpPevvO/pL9awNutwqe6z/Tk1ePLnHSKuHzt2LKtXr2b58uXMmTOHYcOGsXr16kAXyAkTJlCnTh1yc3Pp0aMHV199NXXr1g3ax8aNG/nggw/4+9//zrXXXsvHH3/MTTfdVCr5r7Al+hv/sTDw/OtVOx3TFNvi/6z1e+jx7De8M3cL2bkF/G3mxqC03Z+ZEfG9cvOLuOSFOUHLEqwzTXEctUbXvj2fhz5a4X6DCOyF48c+XX3S+ztZ+47k8eNu54btiQt9vaJK68rn/Kenc/mr3wctu+7t+Qx46dtS2b9Sp0LPnj2D+rm/+uqrnH/++fTq1Yvt27ezcePGsG1atGhBly5dAOjevTtbt24ttfxU2BK93Z++XMf/XdwybHmBLQq/MG1D0LrQHjc5eYUYY4K6Jh0vKOKRT1axaMuBoLTNR38ZuCKI1hBcViraXD8DXvqW7NwCto4dFjFNdm5Bqb3f5r1Hg177/5c7s3N5dWYGT1/RgZSkCltGUeUsWsn7VKlatWrg+Zw5c/jmm2+YP38+VapUoV+/fo794FNTUwPPExMTS7Xq5rT+tRQWlQTE0MBePTX8HPbmt5vI2JNDy0e+ZG9OHgNf/pZPl+1gx6HIB7QwniK95WRPDm6ru4uLDec9MZU352xiz+GyGfqdnVsQFsQ/W7aDb3/cW2rv8fnyHSz56WDMdH+YvJIPFm3ju1J8b6VKQ/Xq1cnJcb7qzc7Opnbt2lSpUoX169ezYMGpH99z2gT6qauDq2/25Byn119mxrUPY2Dgy99RbKDHs9+QeTBygPdPhLbvSD7NR3/JleN+iJh2b04ez365NvA6tJF4bdZh31XCpn1h227YlcOGkJNUrDi/budhXp6+gfyiYnILinhu6np6/rnkWBQXGz5fvqNUrkaGvPJd2LL7/7ucWycsClseqXonlt9OWs7Vb86LesIFAuuLtOFXVTB169alT58+dOzYkd///vdB64YMGUJhYSGdO3fmiSeeoFevXqc8f6dF1Q34uj7ajXhzftT0s9bvAeD3g9sGqnWa160abZMgx0KmT1i+3dfXNTe/iMc/W82jl7ejbrVUbn5nId9vDA7gP2TsY0jHhoHX/jrnG/6+MKj6Y/fh4wy2AunaMYPJOnScVg2qsW5n9Iaka96az5G8Qm6/yHmui0+W7eChj1awNyfPscorHjuzS64U9h3Jo1611IhpB/31u6jVO07sPap+88Ey2/vm8u68n/jD4LaBZf4qnV+9t4RP7r2Qbk1rx/VeSpWliRMnOi5PTU3l66+/dlznr4evV68eq1eXtMc99NBDpZq306ZE/+L0H/n5a3N5cdoGiotNzN4wuVYAGdT+rMCyKqnub0+4JkKr/T++38zHSzP505frOJpXGBbkAT5euoNPlmZG3Pfcjfs4cDSftbaA/n/vpjPw5W8pLjbkxxg3cCTPd8UQmu7deVsB3wkEYMHm/ezKLr0qnb05eWHLGtas5Hr7iQu3hZ2wX7U1mu/JKcnr7z5cwVvfbmJhSPuJ31VvzHP9vkqd6U6bEj3Aqh3ZrNqRHdf9ZJMTS85lr3wT3tIdSa3KzlOAvjTjRwA+XbYjYv/+GWt3M2Ptbq7q1jgsTfPRXwLQqVFN7rv03MDyeZt83TpvmbCIp65oH7RNaCOyX27Ivp+csoZbejcLXMF8s24P36ybGVbK9vd5j3fOjFpVYk+L+umyTBpUr0SfVvX4atVO6lZN4YKWdTHG8Oinq6iUnMD6Z4ayYVcOrRpU42heSTXX9gMlVTeb9vpOCC9O3xD2Hkqp+Jw2JXq70B420ew4lMv0B/oCsGK7+6HG/sAbzderd0VdP3Pdbto9MdVx3aod2Wx1mKNnbsY+CoqC66DfmLPJcR9H88JPNOkuGjV7/2UWLR75ipWZ8Q29drrSCK0Oe+C/KwJdY+99fynXjfc1PPnbC44XFNN89JcMfuU7rnlrHsszsx3f68BR34hkN420XjFv0764vqNKuXVaBvp4dG9Wm8SE+Gd7m/DDlrBl8Y7+vOPd9Kjrx3693nF56B2xXrCqq0I5jQy+5q3obRcAu6yqnSte/yHiVM6z1u9m9Y7gIBxadbN8+yGqVXK+KPx8+Y6g1xscGmqXbjsUMbDZr8ROtf+tzOLTZSVVb8XFhuxjJ959dOm2gzz6qW9yvuJiwyOfrHJsh7nh7wsZHqXR361IV5rxMMbwnwU/lWq3WVV+PB/oUxITAoOfTtbcjPD6+LKw3yrNdmtaMgfGvqO+IJtzvOSHtzGkvjuSaO0Fm/c57+P2f6Xzs9fmBi0LrTJ77uv1HDnuPA3FbyctDzxfmXmIYa/OdUwXSeIpnnJ2V/ZxXp+1EWMMoyYu44H/lgx8+9OX6zh/zPSgYx+Pq96Yx8SF28grLGbHoVw+WLSNoX/7nuajv2TpttK9YvkhYx/tnpjK4q3ObRturduZw+Ofreb8p6cHLT+SV6j3eTgNVfhA//iw805q+4QEoXKyuzr9etVSoq6/+Z3wLoVl4S2rqmbptpLSbs9nZ5J1KDeoZ8ojn6xytb8HP1zBsm0HKS42/F/IVYb9JLgr+zjbDxxjY4RukscLgq805m/eH5gyIpo3I1Q9RZOT524eo+xjBVz28reOQfjg0Xymr9mFMYZxszNYtOUA901cSl5heKAaNXEpL07/MahNYPIS3wnSf3W3MjObXdnHufxv3wemqIikuNgwY+3uoKvAI3mFJIRcXZZWo/Kew8fZk3M80IV3vlX1+NzU9fx3cfz3dHA6RgAdn5xGn+dmRd12zBdrWeDie3Giuo6ZTvs/OleJKmcVPtC76R54b79zuaprI+4f2Npx/dkueoa0qFeVmQ/24/e27nxOWjUo+/lWLmlbH4BbegfPcfP67AzW7TyxvurjZm/isc9W8c263UHL7QXnXn+ZycXPz2b8d5sd93H1m/MingSiOViGM4CeP2Y6G/ccodNTwSXPPTnH6frMDO56bwktHvmKF6Zt4Nq35/Plyp1MdWhb8fdkGje75KQUOp3Fjf9YSK+/zGTtzsM88ZmvK9y0NbuYs2FPULo1WdlcMW4ud/47nae/KBlfkVdYTKRaxJkh/5dYdhzK5fynpwd6MfX880x6PjuTxATfT7rQqup7c84mHv7YVyDIzS9yfQUR7SrY337ixBjDhB+2MHJ8yaCgzXuP0O+F2YEqJWMMf/lqXcz5aIwxvDx9Q+B+E89NXU/z0V9y8FhBWPfn0021ar44kpWVxYgRIxzT9OvXj/T06NW/uflFPPDf5TGvNl0FehEZIiIbRCRDREY7rL9bRFaJyHIRmSsi7W3rHrG22yAig928n9//fn2Rq3Tn1KrMy9d14bcDWvPEz9rH3sDBoA5nUbNKcsy+9qHdA8uCf1Rvs5C8TFy4LVC/Hq+jeYV8sGh72PKhf/s+bNlHSyJX9Vz21/ABVLEs2Hxy1QjxyNiTw4LN+5m9fk/ENPZqJfC1dUS6SU20UbjGGH713hJu++fioOXDXp3L6h2+IDbJVprOLyxGcA6gsdpzQn29aifZuQWBuYb8KiX7ftJOJfJHPlnJVW/Mc9Xl9kTatQDyi8Ib7Pu/9C1b9x/j9n/5jtORvELe/m4z170dvT1pb04er87K4MZ3FpBXWHRCV4YV3TnnnMPkyZNPePveY2fy6bIdYQWdUDEDvYgkAuOAoUB74Hp7ILdMNMZ0MsZ0AZ4HXra2bQ+MBDoAQ4A3rP25UsWqE/74nt5R0/kvjkWEOyIMIgotHYe6qqtvxsluzeKfG/r7P1wa9zbRTLQCstsqJzciVbH4axb+FkfX07Lw7u09494mtHF44+4cBr78HSPHL6BBDff9+2+bsDjiulscRgD7FTo0kIc2Lturu3LzixxH9WYedD9Dqp+/sTo/ZL7z1CTfd+btb4Ovyl755sfA2BA3Dayx+h38uDvHcSRzYVHkDedt2s+rMzcGuvXGGuHsX7v9QC5frAif2PDFOHrflbasQ7mszDwUqJp7+OGHg+ajf+qpp3j66acZMGAA3bp1o1OnTnz++edh+9myZQvnte/A4dwCjh07xsiRI+ncuTPXXXedq7luDrnsJOCmH31PIMMYsxlARCYBw4HANakxxn4NVpWS/9FwYJIxJg/YIiIZ1v5idw2BQI+Ojo1qRk1Xr2r0unWAP/6sPf+eH1yv2vas6oHeIGdbgaFhzcpushakSZ0qrHhyEOO/2xR06X+i/D0yyrLKw87ft/9k9GxRJ2xyuHhc0qY+NSolBe4q5sZ144O/RvarjXhOkotOsOHy7W9L/tcvTtvAQ4PbRu01k19UzCyHKpqLnovvDmnfrN3Nk1PW+PYZ0uW1qq3B3F7F8so3G2l3dnUg+lxMG3fncPu7i3nKYWIwe2FgkHWsQ8doOJ387F6e8SO/7NMcKOldlldYRNvHp3Lbhc156oqS97X3HnLqdfb67AweslW1Hs4tYOv+o3RY+WcSd68JSmswCEKxMRQUFZOSlBDx6srR2Z1g6NjAy31HfIWMwmJDcqIwcuRI7r//fu69914APvzwQ6ZOncoDDzxAjRo12LdvH7169aLfZUOpXqlkPMruw3kUFBWzdf9RPn9vPFWqVGHlypWsXLmSbt26uc9fDG6qbhoB9mv+TGtZEBG5T0Q24SvR/ybObe8SkXQRCVy/3nZhc+pbw+1TkxJZ/8wQNv/58kBAtru4Tf2YHyIppLvexa3r8fmoPiz/42VsHTuMmi4GAzn5zQBfu0DNysn8fnC7E9pHJCdS0rN7eEjk/FxcCnPH223ZdzR2ohg+ufdCru/Z1HFdx0Y1wpaFznJpl+Si6uHw8YKTOsm9OP3HwPPXZ2fETJ9fWMz7C+NvGA317FfrgvZpN3aqr8vuFeefw/cbg6udAiXpKMH4xekb2H4gl+lrwk9If/3mR4ctgtn3vTM717H3j7/U7x8vsvQn31XQv6yR3X5BBYcI/84jtkb7rft93wf7VZTBcCy/kKN5ReQXFXO8oIiCIuN64kCAYmM4cCzfcdyJ//27du3Knj17yMrKYsWKFdSuXZuGDRvy6KOP0rlzZwYOHEjmjh0sWb+F1Vkl3Zazc0tOxj/M/T4w/3znzp3p3LlzzLw1qeOuYOqmRO90iMMOkzFmHDBORG4AHgdujWPb8cB4gNSGrQ0QdGYHAjeimHr/xXQZ45tb/rP7+lC3agrVHGaqBLi+Z5Og1ymJCYE6xPfuuCBov3bPDO/AE5+vCVvu5Irzz4mZZvLdvRnhon97qFYNqkdcd1aNVHYfLqm66Nq0FstsvXSGdW7IL/s057mpzn31nxnekX4vzok7T5G0rFfVcYqEeLRqUJ1nhnfgg0XhwbBrk9qBem83vnOYmiLUjiiT2p2IWCeNF6dtcHXD+l3ZJRP2zX+kP3WrpvLDpn1c2rYB787bGnRS/Wx5Fq+M7Bp47b+Un7IiiykrsoL262+wc6pHB9/JepoV4O11/EfzCqka4Tf220nLeObKjtSwSqn2Cfoue/k7juQVMrJHEyYtLinv2bv0Nh/9JX8b2cVx37+fvDLwPNJp+7FPV/E32+cH2NT9cc6pVZm6VVPYtPdo0HiT1KRE8gqLaHNWdcfffrExGGMCjdoAq20B/sjxAqrZSuR5BUWszDxEi3pVGTFiBJMnT2bXrl2MHDmS999/n71797JkyRKSk5Np1KQpeXnBvxH758ovLI6rwHToWD7n1q8WNKI8Ejcl+kzAHjEbA1kR0gJMAq48wW1jqlWlpJqmdYNqNKlTJSzN1d189e1tzgoOlP/8ZQ8AfjvAuXeO3829m7N17DDeuql7YJn9ktg/0hbCe+GkPz6QZ3/Rkc/u6xNY1tLhzkjjbii5LHvwsjYADOlwdlCaC1rUiZjHBtWDr2yu71FSEn76ig48e2VHKiUnRuzl0axu+HELNfnu6G0jfokJQlOH/8OJiNQIWDdG19dQr86M3uawKjPb1ejnk2UvcbmtIhr9SUmA6/2XWQx4eQ6//OdiZq0vqbI5Ef7ZWvMcBlS9v/AnLrWd+D9bXvIzjTYA6/PlWbxnqxK9/u8lvW38pd3Qkd6hU2/YZ5F9beZGJi/JJGNPTtRt/OZH+B9mHcplddbhsEGF/t0Y45tTyj8vVHZuASszD7FhV07Eea4ANu87GnQ8/J/sp/2++vVJkyYxefJkRowYQXZ2Ng0aNCA5OZnZs2eTlek72dlPMPZD0+2CC/ny048AWL16NStXlnwPnAx7dS5zNribsttNoF8MtBaRFiKSgq9xdYo9gYjYI+cwwP8rmwKMFJFUEWkBtAZKrTN6pMYc/5wsoQ1DfVrVY+vYYTxgBdZYhnT0Bd4qKYm8c1uPwPLQE4hdvWqp3HhBM7o0KWnUTUwQujYNbuQdcF6DwPNf92/Fpj9fTpuzg/fb4Zzw6grwVROtso1avbpbY4Z2KjlJ3Hph88AJceGjAx334Waem7OsarJYc9x0alQzYokvXpHydUGLktuu3dTLuXonHv/8YQvP/G9t7IQnyU1pK1Toj9e/j20OU2acSIP9cYceOdHuZDZj7e6oM6oeLyiK2u3245ABewWF4SO//V6a8SMPfbQi7Oot0rd1T05eUPWNndNI9kAXT2D9rsPsPnycVTuy+cmq9vG3GxhjKCwqDpqLyc8+Hbf/dhXFxtD03Dbk5OTQqFEjGjZsyLUjr+e7eQvp2q0777//Pi1atQnkwan27Nqbb+fY0SN07tyZ559/np49o3dQiDWtt13MX6cxplBERgHTgERggjFmjYiMAdKNMVOAUSIyECgADuKrtsFK9yG+httC4D5jzEl3gP3gzl5MXLQtcLkY6s6LW5K+9QA/d1GtEsu80f2pnJwYNq/NNw/25cfd0btaPjSoDS9O/5GalZP56Fe9mbR4O49b/a9TbXdIEhESBe7o0yKoJCoijLuhGw1qpAZNbfC/X1/Exc/7GvAa1arMS9eeD8CE29KoWzV4GuH61cOnFW5Uy129XpM6VZj78KXUrpJChyenRUz3h8FtyS0oCtSxPj7sPP705bqwdP6Gu+vHL+B4YVFQVVMs9vj/4eLI3T/d+mTZjtiJKpglDscrt6Ao7iqz0IFvsa5+RscYmPfarAxem5XhuufU/1Y63xo0OE3whf87c8OnJPHr+OQ0vrrFubddJPZ5opxOCKt2OM/BFKrYtu2WfUeZt3hpoLG1Ws3avPf5dKqkJNKqQfWgOv4FGzI5lldIoyZN+WSm77ddqXJlnn9jAp0a1QwUeA4dy+fA0XzquOhwEo2rYpgx5ivgq5Blf7Q9/22UbZ8Fnj3RDDrpfW5dep9bN+L6s2tW4vNR7vrgx3KOFRTbhpS2WzWoHrUOHWBU/9aM6u+72ElKTOCmXs0Cgd7/j+xpq55xahAe1rlh2LImdaqQ/vhALn5udtBYg/7tzgpL6+Q//3eBq3QAjWs7V8lUSUnkWH4RM393SdhNu++4qAU9W9QhJSmBIa+E99P/4K5ejP16fcRA/8Woixg3O4NVO7IDpRb7+IXU5ISI9cyl4fkRnfnD5OiXzeXhixXOtZ49nv0mrv3Yqx6mrt7JyzNiN7K64XQzGiexRhWDb+ZVu7Ux7tFwLL+IOqlJEUv3oXZml077zOGQgUqhde7+vDlVf2XsdS4oHssv4tCxAo7kFQbaShIkuNo6XhVymuKW9d3fIORU6dok/v71Th68rE1gdOqixwZEvCqJpV61VNY9MyTu7To3rkmLerGP7/Mjglv8e7esy5KfDgYCrD/QpzhMPiYidG4cfLw+uDP4rjoPD2nL3pw8bundLKxqqFPjmrx1c3cWbz3ANW/N5+N7LqSqdS+BRrUqc3e/cwMjU91688Zu3PO+uxuYX5vWpEIG+hN1WfuzmLG2pBfNmqzDrNqxhuqpSbw6K3ZvodPB0fwi6p7i+ZEgvAfTrsN5tLJ+0/a2iXjuvrbJ4QSw7cAxiooNda2eiE7dTaOpkIHe6X6v5S10jpIT9ZsBrQNdMkMbVN2Y/0j/k3r/KS6vdM4K6cb6wV29eHnGj4HLfP8QeXvj6Q+j+0f8AoaWtEQkUOUUSY/mdYL6aa95ejBVUhJPaHK5oZ0a8tZN3bj7P5GD/Xt39Iw4xUVKUgL5hcV8dHdvOjeuyfJthwJTMMerRb2qjr0rGtWqHFe9q1ujh7YLCvTRqkEqutSkBPIKi8N6mRUWFVN8Avd3Lm3H8gvJOV5A9UrJQTfSKQ07DuVSt1oqeQVF7DkSXF23deww5LnI21bMuW7K4czsRtWURO68OL66wHilP+7cePrBnb147PLzTmhA1y+6+oYu3Nwr+uhgO6fZI/u38zUg+we8QPCcKI1qVXbsBQWQlHjy/9OqqUm+9oyQk+66MUO4vmcT/viz9kwZ1Sdo3ds3d+er31wMwJCODVkf5Sro4tb1A8e3zVnBAX/N04P5/g+X0qN5HVKTEmPe4ey2C5tHXBdpPqUZD/Z1XH6ySnO2ydFD23FtWmOe+vmJTTUSSVWXNxPKs6pDQqv9cgqF4znZcU8lXha27Dsa970e3MrYk8PqzZlk7I+vXabiFZ0hYpfA8rZmTPxVJfGKdE/WWO0S0fz1ui789Trnvsp2a8cMpv0ffY2u+4+Gf5G6NKnFD6P7c07NSqQ1q8MfP1/tupGoe7PSu7+r/ST0nzsuoHJKIn+5ynlwyeCQLqtOfaeBwKhRv3du7RFo8AbflAP2k5j9BPezzg3DGhjtt1hsWLNS0L13+7apz/9+fRGVUxIZ8NK3gG8ajSoppf9zHNrxbM5r6Nx760Rc070xdaulxj0JWzQzf3cJuflFYdNig6+HWaQpG67q1ohPlvoa1f+9IoehOXk0q5Uc34jX04zB8NOhAl5bGN/01hUy0MfTG0OVjss7nR0UaBZuOcDwLmGDmAM9doZ1bujYUBzqoUFteG/BTyfcFuGkm+2k0adV/Ce/OQ/1CxssFloCb1izUqAdwkmnxiXTcrx8bReWbz/E6zd0Y/qaXbwxZ1PQzWPmje5Pi0dK+jJUSU6kY6OaQaXPSFdCJ+tN21iQ0uAfYR460tyNRy9vx5+/Ch7A16lRTc6tXy2s37zfCyM6c9d7SwKv7VU2L11zfiDQL83MIdJtF/59e0/embuFb6NMUHeyHh7SLuLgRDfeuqk7d/9nSeyEJ6hiVt2c4Z6/unPMidxKwzcP9uXNG7vxyb0X8sp1waML46nmiWZU/9YR+/KfqOTEBLaOHearl3SoYnr7Zl9wG2gbq2DXvF5VXrFd4dxxUQt+0S34pJaUmMDqpyJPtmofS5GUIMx9uD9dmtRiaEffya9f2wZ8fl8ffjugdVge/e098d6zN9S1aY1PavtYnGaPTbaq4EKrSH5mO+mfH6Hjwl19zw1bdo31GZJsI1HtAxVD72D26/6tAN8snSISNtdOqLVjBtO3Tf0TmjQv1GXtI/dqu6df+Gdzq33DGq67PJ+oClmiP9Nd26NJ7ESlIFoX0dIa6VoeBnc4m3E3dOPSdpHnQBre5RyO5RcxrHNDaka4EXxCQuxA4k/n16lxzaBt/EFPJPaMkOBr3Lb35Pj94LYR75E8ZnhHfsjYH7EB114NGM+0Hva8hPIHZHtgvvuSc/nNgFaB6iun20O+eWP4BF33D2zNSGtEt328x5COZ9O9WW2W/HQwMBsn+EZz92lVj06NavKYyxsS2a9Sb+3djHfnx+7aGclfr+vCmh3ZYY3wPZtHHsEeqkH1VPaEjHu4s28LqqSW3ky1TrRErxyVRuNpeRrWuWHUOm8R4YYLmkYM8qVt9u/6uUq34Zkh/Pv2nmQ8O5Sp91/MPZdELilWSk7kh9H9WfToAAA+vfdC7urbkt/0b8Wap93r4ocAABQKSURBVAcz9+GS6bNv7t08aqnz+as7h5Uq8wqLWRfSLuUv0dsLAqOHtgs71jMe6Ms7t6YFXvtnoP2dbVT6/QPbkGINHPSPrPaP9J1wWw8+urt30Mjfb39/KalJiXzx64vo1TJ2lV3oBIi39YmvI8ULIV2MUxITuMDhfa+/wH3B7JyQYzxmeAd+0bUxtU+ij/xjl8c+6WmJXgVpd3Z11u/KITlBywClqbmLsQvgqzLqa83G2u7syI2o9t5fDWpUClxFdG0audH74SHtwm7esejRARw8VkDbs6tzbY8m7DuSx5Z9R3ns01W0Pas6lVMS2Tp2GP9Z8BPrdh4OVDc1rVuFUZe2ChrzMvuhflz64hxqVk6m9VnVaW2r3mpQw1diX2H1Rkl2KEi8dn1XOlknhJqVk+nRvE5g0NHPXLQH2d3SuxljhncMWtasThVu6d0saLpye916aJtM67OqU80ahDWie+PASSnUv37Yyi+6Nmbr2GGOE9vVrpLMwWMF1KuWGvjc/kZm//GsWTk58F6h+rWtH3FOGzdXnKAlehXi37f35J+/7FFq4wa8rF/b2NNj231674X845a02AlDTL3/YuY+fCkt61dl/iP9+cctaaUyJbZveo1KQaO+61VLpUfzOkx/4JKgm8Hf1KsZz/6iU9D2Dw1uy1XdStoJmtSuTIt6VcMG20HJDVH8o7edxpD8/Pxzwk6IKUm+9pjXb3A3N/uTVrfPAoeR0wkJwpjhHYOuEuxXdAPPC66DTxRh5u8u4cHL2vD81cGf6Rxbr6qDUW7+8cWoiwIzYX5674X80rqq8HffPWYF9sQEYfXTg/mN1QYx6tJWgX0khRS67rWuzN67w327g5boVZAGNSrFdXemM8pRa6CW+H54b1/TiqN5LSHviG+ZiPXX/ig5YUYrbUfjL9nPsqp/TmQshd+6MUM4749T+fieC0u1yyv4rkZmP9QvaFloibNdQ99JpUfz0nvv2y5szr/mbeXje3rT7uwaLN12iPsHRp648MNf9ebnr/u6cvqPwZs3duO8hjWYt2l/4KYiOXkFdKpRMzDA0W/RYwOokpLEC1PX8+78n5h0V8mob/sI7BsuaBrUOys5MYHLOzUMKvl/vXoXv7JVzz04qC2/HtCaomITuMfBQtvd4T66uzc9mtfhD1HuNeFEKsIAA7vUhq1N51Fvsvix0u2podRJe7ENHDmR/uMSHPhtJ4JiBERIcDpB2F9b6aKmEQl5r0j7InYax32d/PsZSWD97iOcW786KclJcb6fc94LDRw4VkCDGlWipAve187DeZxVozIJCQnkF0NKUlIg3TVvL6QY4ckrOtK5cW2H7Z3yVfL/+WzFTi5sVS+Qn1+8OZ/tB/OY/kBf6lSrDCIMefUHMg8dp1WD6nw26uKQ/SRgRGjx6NeAMKxTQ75ctZPxN3dnUMi4kKBvmcgSY4zjJWOFC/TtOnUxS5ekl8ngEaVOyrL3oeAYmGLrYWzPrQfGYZ2JP02Z7KsCvp+KylgnkGKEhIREJOSEYD/xyOitEQN9hYum1VKTNMiriqnrjeWdA+8JOwGcipOifV34vm74xwISMLx1YxeqpSTGeL8o+8Lps5Us23M4lwbVkh3SEXgu1t/EGPvyvd9LEQ+zRlSlVPkRAUnEd6uLiuHNxy9ky/6jVCulGWsjcR7OdzI00CullCs1qyTTpUrZBvlTTbtXKqWUx2mgV0opj9NAr5RSHqeBXimlPE4DvVJKeZwGeqWU8jgN9Eop5XGuAr2IDBGRDSKSISKjHdY/KCJrRWSliMwUkWa2dUUistx6TCnNzCullIot5oApEUkExgGXAZnAYhGZYoxZa0u2DEgzxhwTkXuA54HrrHW5xpjYd6ZWSilVJtyU6HsCGcaYzcaYfGASMNyewBgz2xhzzHq5ACjbm1kqpZRyzU2gbwRst73OtJZFcgfwte11JRFJF5EFInKl0wYicpeVJn3v3rK7U7tSSp2J3Mx143SrIcf5RUXkJiANuMS2uKkxJktEWgKzRGSVMSbofmbGmPHAeIC0tDSdu1QppUqRmxJ9JmC/+21jICs0kYgMBB4DrjDGBG5zbozJsv5uBuYAXU8iv0oppeLkJtAvBlqLSAsRSQFGAkG9Z0SkK/A2viC/x7a8toikWs/rAX0AeyOuUkqpMhaz6sYYUygio4Bp+CaNnmCMWSMiY4B0Y8wU4AWgGvCRdVfzbcaYK4DzgLdFpBjfSWVsSG8dpZRSZazC3UowLS3NpKenl3c2lFLqtBLtnrE6MlYppTxOA71SSnmcBnqllPI4DfRKKeVxGuiVUsrjNNArpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyOA30SinlcRrolVLK4zTQK6WUx2mgV0opj9NAr5RSHqeBXimlPE4DvVJKeZwGeqWU8jgN9Eop5XEa6JVSyuM00CullMe5CvQiMkRENohIhoiMdlj/oIisFZGVIjJTRJrZ1t0qIhutx62lmXmllFKxxQz0IpIIjAOGAu2B60WkfUiyZUCaMaYzMBl43tq2DvAkcAHQE3hSRGqXXvaVUkrF4qZE3xPIMMZsNsbkA5OA4fYExpjZxphj1ssFQGPr+WBghjHmgDHmIDADGFI6WVdKKeWGm0DfCNhue51pLYvkDuDreLYVkbtEJF1E0vfu3esiS0oppdxyE+jFYZlxTChyE5AGvBDPtsaY8caYNGNMWv369V1kSSmllFtuAn0m0MT2ujGQFZpIRAYCjwFXGGPy4tlWKaVU2XET6BcDrUWkhYikACOBKfYEItIVeBtfkN9jWzUNGCQita1G2EHWMqWUUqdIUqwExphCERmFL0AnAhOMMWtEZAyQboyZgq+qphrwkYgAbDPGXGGMOSAiz+A7WQCMMcYcKJNPopRSypEY41jdXm7S0tJMenp6eWdDKaVOKyKyxBiT5rROR8YqpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyOA30SinlcRrolVLK4zTQK6WUx2mgV0opj9NAr5RSHqeBXimlPE4DvVJKeZwGeqWU8jgN9Eop5XEa6JVSyuM00CullMdpoFdKKY/TQK+UUh6ngV4ppTxOA71SSnmcBnqllPI4V4FeRIaIyAYRyRCR0Q7r+4rIUhEpFJERIeuKRGS59ZhSWhlXSinlTlKsBCKSCIwDLgMygcUiMsUYs9aWbBtwG/CQwy5yjTFdSiGvSimlTkDMQA/0BDKMMZsBRGQSMBwIBHpjzFZrXXEZ5FEppdRJcFN10wjYbnudaS1zq5KIpIvIAhG5Mq7cKaWUOmluSvTisMzE8R5NjTFZItISmCUiq4wxm4LeQOQu4C6Apk2bxrFrpZRSsbgp0WcCTWyvGwNZbt/AGJNl/d0MzAG6OqQZb4xJM8ak1a9f3+2ulVJKueAm0C8GWotICxFJAUYCrnrPiEhtEUm1ntcD+mCr21dKKVX2YgZ6Y0whMAqYBqwDPjTGrBGRMSJyBYCI9BCRTOAa4G0RWWNtfh6QLiIrgNnA2JDeOkoppcqYGBNPdXvZS0tLM+np6eWdDaWUOq2IyBJjTJrTOh0Zq5RSHqeBXimlPE4DvVJKeZwGeqWU8jgN9Eop5XEa6JVSyuM00CullMdpoFdKKY/TQK+UUh6ngV4ppTxOA71SSnmcBnqllPI4DfRKKeVxGuiVUsrjNNArpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyOA30SinlcRrolVLK4zTQK6WUx7kK9CIyREQ2iEiGiIx2WN9XRJaKSKGIjAhZd6uIbLQet5ZWxpVSSrkTM9CLSCIwDhgKtAeuF5H2Icm2AbcBE0O2rQM8CVwA9ASeFJHaJ59tpZRSbrkp0fcEMowxm40x+cAkYLg9gTFmqzFmJVAcsu1gYIYx5oAx5iAwAxhSCvlWSinlkptA3wjYbnudaS1zw9W2InKXiKSLSPrevXtd7loppZQbbgK9OCwzLvfvaltjzHhjTJoxJq1+/foud62UUsoNN4E+E2hie90YyHK5/5PZVimlVClwE+gXA61FpIWIpAAjgSku9z8NGCQita1G2EHWMqWUUqdIzEBvjCkERuEL0OuAD40xa0RkjIhcASAiPUQkE7gGeFtE1ljbHgCewXeyWAyMsZYppZQ6RcQYt9Xtp0ZaWppJT08v72wopdRpRUSWGGPSnNbpyFillPI4DfRKKeVxGuiVUsrjNNArpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyOA30SinlcRrolVLK4zTQK6WUx2mgV0opj9NAr5RSHqeBXimlPE4DvVJKeZwGeqWU8jgN9Eop5XEa6JVSyuM00CullMdpoFdKKY/TQK+UUh7nKtCLyBAR2SAiGSIy2mF9qoj811q/UESaW8ubi0iuiCy3Hm+VbvaVUkrFkhQrgYgkAuOAy4BMYLGITDHGrLUluwM4aIxpJSIjgeeA66x1m4wxXUo530oppVxyU6LvCWQYYzYbY/KBScDwkDTDgXet55OBASIipZdNpZRSJ8pNoG8EbLe9zrSWOaYxxhQC2UBda10LEVkmIt+KyMVObyAid4lIuoik7927N64PoJRSKjo3gd6pZG5cptkJNDXGdAUeBCaKSI2whMaMN8akGWPS6tev7yJLSiml3HIT6DOBJrbXjYGsSGlEJAmoCRwwxuQZY/YDGGOWAJuANiebaaWUUu65CfSLgdYi0kJEUoCRwJSQNFOAW63nI4BZxhgjIvWtxlxEpCXQGthcOllXSinlRsxeN8aYQhEZBUwDEoEJxpg1IjIGSDfGTAHeAd4TkQzgAL6TAUBfYIyIFAJFwN3GmANl8UGUUko5E2NCq9vLV1pamklPTy/vbCil1GlFRJYYY9Kc1unIWKWU8jgN9Eop5XEa6JVSyuM00CullMdpoFdKKY/TQK+UUh6ngV4ppTxOA71SSnmcBnqllPI4DfRKKeVxGuiVUsrjNNArpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyOA30SinlcRrolVLK4zTQK6WUx2mgV0opj9NAr5RSHqeBXimlPM5VoBeRISKyQUQyRGS0w/pUEfmvtX6hiDS3rXvEWr5BRAaXXtaVUkq5ETPQi0giMA4YCrQHrheR9iHJ7gAOGmNaAX8FnrO2bQ+MBDoAQ4A3rP0ppZQ6RdyU6HsCGcaYzcaYfGASMDwkzXDgXev5ZGCAiIi1fJIxJs8YswXIsPanlFLqFElykaYRsN32OhO4IFIaY0yhiGQDda3lC0K2bRT6BiJyF3CX9TJPRFa7yv2Zqx6wr7wzUcHpMYpOj09sp9sxahZphZtALw7LjMs0brbFGDMeGA8gIunGmDQX+Tpj6TGKTY9RdHp8YvPSMXJTdZMJNLG9bgxkRUojIklATeCAy22VUkqVITeBfjHQWkRaiEgKvsbVKSFppgC3Ws9HALOMMcZaPtLqldMCaA0sKp2sK6WUciNm1Y1V5z4KmAYkAhOMMWtEZAyQboyZArwDvCciGfhK8iOtbdeIyIfAWqAQuM8YUxTjLcef+Mc5Y+gxik2PUXR6fGLzzDESX8FbKaWUV+nIWKWU8jgN9Eop5XEVKtDHmmrBy0Rkq4isEpHlIpJuLasjIjNEZKP1t7a1XETkVes4rRSRbrb93Gql3ygit0Z6v9OBiEwQkT32cRWleUxEpLt1zDOsbZ26A1doEY7RUyKyw/ouLReRy23rHKckifTbszphLLSO3X+tDhmnDRFpIiKzRWSdiKwRkd9ay8+s75ExpkI88DX0bgJaAinACqB9eefrFH7+rUC9kGXPA6Ot56OB56znlwNf4xun0AtYaC2vA2y2/ta2ntcu7892EsekL9ANWF0WxwRfD7De1jZfA0PL+zOX0jF6CnjIIW1763eVCrSwfm+J0X57wIfASOv5W8A95f2Z4zw+DYFu1vPqwI/WcTijvkcVqUTvZqqFM419aol3gStty/9tfBYAtUSkITAYmGGMOWCMOQjMwDfH0GnJGPMdvl5cdqVyTKx1NYwx843v1/pv275OGxGOUSSRpiRx/O1ZJdP++KY1geDjfVowxuw0xiy1nucA6/CNzj+jvkcVKdA7TbUQNl2ChxlguogssaaEADjLGLMTfF9YoIG1PNKxOhOOYWkdk0bW89DlXjHKqnqY4K+WIP5jVBc4ZIwpDFl+WhLfrLpdgYWcYd+jihToXU2X4GF9jDHd8M0Sep+I9I2S9qSmnPCoeI+Jl4/Vm8C5QBdgJ/CStfyMPUYiUg34GLjfGHM4WlKHZaf9MapIgf6Mni7BGJNl/d0DfIrvcnq3dWmI9XePlTzSsToTjmFpHZNM63no8tOeMWa3MabIGFMM/J2SGWPjPUb78FVdJIUsP62ISDK+IP++MeYTa/EZ9T2qSIHezVQLniQiVUWkuv85MAhYTfDUErcCn1vPpwC3WD0EegHZ1uXnNGCQiNS2LtcHWcu8pFSOibUuR0R6WXXRt9j2dVrzBzDLL/B9lyDylCSOvz2rznk2vmlNIPh4nxas/+07wDpjzMu2VWfW96i8W4PtD3wt3j/i6wHwWHnn5xR+7pb4ejqsANb4Pzu+OtKZwEbrbx1rueC7GcwmYBWQZtvX7fga2TKAX5b3ZzvJ4/IBvqqHAnwlpztK85gAafiC4CbgdayR4qfTI8Ixes86BivxBa6GtvSPWZ93A7beIZF+e9Z3c5F17D4CUsv7M8d5fC7CV5WyElhuPS4/075HOgWCUkp5XEWqulFKKVUGNNArpZTHaaBXSimP00CvlFIep4FeKaU8TgO9Ukp5nAZ6pZTyuP8HAhNmvDhSm+oAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.9595977067947388.\n",
      "Better model found at epoch 1 with accuracy value: 0.9613317251205444.\n"
     ]
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_labelsmoothing');\n",
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',monitor='accuracy',\n",
    "                                                   name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas4_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.290772</td>\n",
       "      <td>0.268086</td>\n",
       "      <td>0.962979</td>\n",
       "      <td>0.978403</td>\n",
       "      <td>06:32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXwU9f348debcISbcMkpCYIgKIJEREEERS4VbKUWq636tUWr1qu2gmhVPIrUWmuLB7XW/rwQsSqVSxEQFUGCIjcSMEC47zOBJLx/f+zsZnZ3NjuBDQnb9/PxyCO7M5+Z/czO7ns+13xWVBVjjDHJq1J5Z8AYY0zZskBvjDFJzgK9McYkOQv0xhiT5CzQG2NMkrNAb4wxSc5XoBeRASKyWkSyRWRECemGioiKSKZr2Uhnu9Ui0j8RmTbGGONf5XgJRCQFGAdcDuQCC0VksqquiEhXG7gLWOBa1gEYBnQEmgEzReRMVS1K3CEYY4wpiZ8SfTcgW1XXqepRYAIwxCPd48BYIN+1bAgwQVWPqOoPQLazP2OMMSdJ3BI90BzY6HqeC1zgTiAiXYCWqvqRiNwfse38iG2bl/RiKTXqanqrVqTVrOoja8YYYwAWLVq0U1Ubea3zE+jFY1lo3gQRqQT8BbiptNu69jEcGA5QtUkbbv/ru/y2XzsfWTPGGAMgIutjrfPTdJMLtHQ9bwFsdj2vDZwNzBGRHKA7MNnpkI23LQCqOl5VM1U1M/DcR66MMcb44ifQLwTaikiGiFQl0Lk6ObhSVfepakNVTVfVdAJNNYNVNctJN0xEqolIBtAW+DrhR2GMMSamuE03qlooIncCM4AU4FVVXS4io4EsVZ1cwrbLRWQisAIoBO6wETfGGHNy+WmjR1WnAlMjlv0hRtreEc+fBJ48zvwZY0xcBQUF5Obmkp+fHz/xKS41NZUWLVpQpUoV39v4CvTGGFOR5ebmUrt2bdLT0xHxGgOSHFSVXbt2kZubS0ZGhu/tKuQUCBo9MMcYY2LKz8+nQYMGSR3kAUSEBg0alLrmUiEDvTHGlFayB/mg4zlOC/TGGJPkKmSgt3H0xphTyd69e3nhhRdKvd2gQYPYu3dvGeQoXIUM9MYYcyqJFeiLikoeTT516lTq1atXVtkKsVE3xhhzgkaMGMHatWvp3LkzVapUoVatWjRt2pTFixezYsUKrr76ajZu3Eh+fj533303w4cPByA9PZ2srCwOHjzIwIED6dmzJ/PmzaN58+Z8+OGHVK9ePSH5s0BvjEkqj/13OSs270/oPjs0q8MjV3WMuX7MmDEsW7aMxYsXM2fOHK644gqWLVsWGgL56quvUr9+ffLy8jj//PO55ppraNCgQdg+1qxZw9tvv80//vEPrr32Wt577z1uuOGGhOTfAr0xxiRYt27dwsa5P//887z//vsAbNy4kTVr1kQF+oyMDDp37gxA165dycnJSVh+KmSgt75YY8zxKqnkfbLUrFkz9HjOnDnMnDmTr776iho1atC7d2/PcfDVqlULPU5JSSEvLy9h+bHOWGOMOUG1a9fmwIEDnuv27dtHWloaNWrUYNWqVcyfP98zXVmqkCV6Y4w5lTRo0IAePXpw9tlnU716dU477bTQugEDBvDSSy/RqVMn2rVrR/fu3U96/izQG2NMArz11luey6tVq8a0adM81wXb4Rs2bMiyZctCy++//37P9MfLmm6MMSbJVchAb3fGGmNM4lTIQG+MMSZxLNAbY0ySs0BvjDFJzlegF5EBIrJaRLJFZITH+ttEZKmILBaRL0Skg7M8XUTynOWLReSlRB+AMcaYksUN9CKSAowDBgIdgOuCgdzlLVU9R1U7A2OBZ13r1qpqZ+fvNj+Zsl+YMsYks1q1agGwefNmhg4d6pmmd+/eZGVlJeT1/JTouwHZqrpOVY8CE4Ah7gSq6p5BqCY2i4ExxsTVrFkzJk2aVOav4+eGqebARtfzXOCCyEQicgdwH1AVuNS1KkNEvgX2Aw+p6ufHn11jjKl4HnjgAVq1asXtt98OwKOPPoqIMHfuXPbs2UNBQQFPPPEEQ4aElZHJycnhyiuvZNmyZeTl5XHzzTezYsUKzjrrrITOdeMn0Hv9QGFUiV1VxwHjRORnwEPAjcAW4HRV3SUiXYEPRKRjRA0AERkODAeo2qRNKQ/BGGNcpo2ArUsTu88m58DAMTFXDxs2jHvuuScU6CdOnMj06dO59957qVOnDjt37qR79+4MHjw45m++vvjii9SoUYMlS5awZMkSzjvvvIRl30+gzwVaup63ADaXkH4C8CKAqh4BjjiPF4nIWuBMIKzhSVXHA+MBqjVta80+xphTSpcuXdi+fTubN29mx44dpKWl0bRpU+69917mzp1LpUqV2LRpE9u2baNJkyae+5g7dy533XUXAJ06daJTp04Jy5+fQL8QaCsiGcAmYBjwM3cCEWmrqmucp1cAa5zljYDdqlokIq2BtsC6uK9ood4Yc7xKKHmXpaFDhzJp0iS2bt3KsGHDePPNN9mxYweLFi2iSpUqpKene05P7BartH+i4nbGqmohcCcwA1gJTFTV5SIyWkQGO8nuFJHlIrKYQDv9jc7yXsASEfkOmATcpqq7E34UxhhTzoYNG8aECROYNGkSQ4cOZd++fTRu3JgqVaowe/Zs1q9fX+L2vXr14s033wRg2bJlLFmyJGF58zV7papOBaZGLPuD6/HdMbZ7D3jvRDJojDGngo4dO3LgwAGaN29O06ZNuf7667nqqqvIzMykc+fOtG/fvsTtf/3rX3PzzTfTqVMnOnfuTLdu3RKWN5um2BhjEmTp0uJO4IYNG/LVV195pjt48CAQ+HHw4PTE1atXZ8KECWWSL5sCwRhjklyFDPTWF2uMMYlTIQO9McaUlv6P/JDF8RynBXpjzCkvNTWVXbt2JX2wV1V27dpFampqqbarkJ2xBUXHyjsLxphTSIsWLcjNzWXHjh3lnZUyl5qaSosWLUq1TYUM9F1OTyvvLBhjTiFVqlQhIyOjvLNRYVXIppuUMro7zBhj/hdVyEB/LMnb2Ywx5mSyQG+MMUmuQgZ6i/PGGJM4FTLQW4neGGMSp0IG+i37Sp7K0xhjEkFV/yeGc1fIQP+nGavLOwvGmP8B/56XQ9tR09hx4Eh5Z6VMVchAb4wxJ8P7324CYOOew+Wck7Jlgd4Yk7SW5O5l1db9MddXTgmEwKJjyd0vWCHvjDXGmEQY/PcvAcgZc4Xn+pRKgZszk72d3kr0xsSw/UA+bR6cyrcb9pR3Vsxx8DPBWZWUQKA/WlixAr2qsjAncb+66ivQi8gAEVktItkiMsJj/W0islREFovIFyLSwbVupLPdahHpn7CcG1PG5q/bTeEx5ZXPfyjvrABw8Ehh0s/OmEg7Dx6Nm6ZypUAIzC8oKuvslMo7Czfyk5e+YvqyLQnZX9xALyIpwDhgINABuM4dyB1vqeo5qtoZGAs862zbARgGdAQGAC84+zOmwtuw6xAAyzbvK+ecQM7OQ5z9yAwmLNxY3lk5Zfi5H+ez7wOzXW6tYEO6g0PMV2yO3b9QGn5K9N2AbFVdp6pHgQnAEHcCVXXnpibFPxI1BJigqkdU9Qcg29nf/4SFObt59pPvy2z/B/ILuO31Rew5FL/kcjwKi47xwKQlZG8/WCb7DzpSWFThSlRQXCJcv6v8R2Tk7skD4L/fbS71th8v38rfZ61JdJZKtPfwUTbuLt/37UhBcXNMvJrQo/9dUdbZKZV9eQUA7DlcEDdtzs5DPDV1ZYlp/AT65oC7GJHrLAsjIneIyFoCJfq7SrNtsvrJS1/x/KdrOFZGPfr3vvMd05dvpcvjn5TJ/lduOcA7WRu5e8K3ZbL/oHYPTaf9w9PL9DUiqSoDnpvLh4s3xUxzfnr9k5ijklWrEviqLttUutrFsWPK8NcX8czHZVfg8NLnmTlcPHb2SX3NSPmFxYWHvIiCxPYD+bz02doye+2ComP0GDOL6cu2Htf2r83LAeD1+etLTLcvr4Dez8xh/Nx1JabzE+i95gyOilyqOk5VzwAeAB4qzbYiMlxEskQky0d+KqQPF29i0948z3V784qvyqrKtv35CSntzFy57YT3UZLgl2P55v3sO1zAqPeXxjzGRJixPP6XYtKiXNJHTOHPH5/YTXWFx5RVWw9w94TFMdO4axmFxzkq44U52aSPmELe0cC+Vm3dT/qIKaSPmOLZAVh0TD1rN7udWtv+/MJSvX7rB6eGHsc6hg27DtNr7Gy27EvcuQ2WRMuyRluS7O0HOXy0+H3cHVHrveGVBYyZtiohr+V1vj5duY1Ne/O47Y1FCXmNWA7kxy/xg79Anwu0dD1vAZRUf5wAXF2abVV1vKpmqmpmvMzsOXSUhz9YxuGjhdzx1jd8t3Fv3AOINGz8V/zu3e9KvV0sRceUuycs5poX5oWWrd56IPT40JHiL+dj/13BBU99mtDSTqcWdRO2L7e73i4uyZ87+mPeXLCBe0sIjMfDPazt1tcXxW0rvd85b3+blc3Og8d/N+NiH58bd4nwQCkDbNDY6YEL0mYniE5ZUty5dvho9D7vmvCtZ+3m1teLA8ZzM7+nx5hZ/Nsp9bkVFh3j8Y9WcOkzc6LGhl/27Gec9fB0Vm4Jb/d9Y8F6Nuw+HLp5KJGe/7R0TUb5BUUnXANes+0AfZ/9jEcnLw8tiyxZf78tMc2Rs1Zto73He/qfbxL/Xnrx2zfvJ9AvBNqKSIaIVCXQuTrZnUBE2rqeXgEEz+5kYJiIVBORDKAt8LW/rHn708ereX3+esbPXceUJVtKfcU8WniM+et28+6i3BPJRphgsNq6vzhI9X9ubuixO6i/5vpybkhQ2++S3H1sLoOStvt4ghbn7o2qjew9fJRc152F05dtpc8zc2K2ux8+WhjqBDt8JDxN9z9+6jt/mU/M9J020guzs+OmyXe18XZ5/BM27j5c6oJFw1pVgUBN7tOV2zjiKsX3fXYuizfu5bPvd4Q+Q8ELQUnB7rmZa9i0N49HXIEs6JI/zeGfX/zAup2H+Oz77WHr1u86TF5BEQP/+nnY8mqVA2Fgf97xXczi8bqgeck7WkT7h6fz509OrLZ246uBEOO+mD8xpeQ27NQqlTh2THnw/aXsPey/zyt4AYn8XJys3046eMTfexs30KtqIXAnMANYCUxU1eUiMlpEBjvJ7hSR5SKyGLgPuNHZdjkwEVgBTAfuUNUT6nULdqqsca7I7gnQVJXxc9eW2DlZeKz4i+ZnfPShI4Xc9fa3YYEsbH9Fx3hzwYa4+1m/6xBzVod/8U5knGxk59JFY2aFHhcUHQtbr6r8e15O3GreuNnZ/OiFL0tMc7TwGBePnc0POw+FlvV9di49n54dWn/bG4v4YeehmG2gf/nke2589WsWrd/DIR9BQFXZn1/AfRNPvDaxNHcfA56by+zV8X9bNPJCdfHY2QwZV/L7E/Tthj1c+swcqjp3Xk5evJlb/p0V1pa68+ARrh73JTe++jVtR03j0j/PCa2LfF8a1a4WN4/b9+eHNa19uyH2Ren9b4sLOn+bFbjovfTZWvb7bAoojQ5/mOErXfCY/XyfSrI5Tq3Q6+ao/IJjtH5wKm8t2EDn0bH7vPILinh08nK+WLOTBet2hebIOeQ0E93x1jf0ffYzBpzd5ASOABrUrBp6XFJH8sMfLPO1P1/j6FV1qqqeqapnqOqTzrI/qOpk5/HdqtpRVTurah8nwAe3fdLZrp2qTvOVK8IP7tgx5c0F6zlSWBR6Y6csLa4CP+u0167Ysp+npq7inncWo6q8/lV0cAuOXgD40QvzGPmfJZ6vv2VfHukjpjDyP0uZ/N3mUCCL9PbCjTz+UXSPfbO64b/SvvvQUf4c0SH22xNoPjoS0b6bUklQVdJHTKHtqGm8MKc4yE7M2sgjk5fzUIwPxb68AtJHTOFPM1bz7Ya9vtqjN7hK9cEmlCOFRZz5UPEpfm7mmrAL5LjZgfbq7c45/Gb9Hpbkxu9cnPzdZjo9+rGv6vANrywINe9A4Iv5+vz1oRLyPe98yypXsxrELj37LYm65RcUsevgEX70wjzW7TwUCjobfPTJrNtRfPGMbCqKNelW+4enh/oAnonotwgGcC/3vvMdm/bm8U1EYcddSNp7+CgZI6dEFVDKSjAA+715Kb+gdKO1cpzCyaotB+Kk9LbvcAHtH57Oa/NyuOGfC/jp+PmhAkOwRj1lyRaytx88od/U2HXwCLvCzkPsi2/Wen8381XYO2OPuoLNjOVbGfX+Mv44dRW92zWOSvv8rGzemL+ed7MCpZScXYf46fj5PPzhcgY89znrdx1i3+ECNu4+HBX43/7ae1zy0Be/AgJBJsjrynooouoULFHVTq0StnzHgSP0bNswavulPgJdsPPOzT10DODOPm0Y52qOcM8A+sB7S4HYwwQjayttRk0LK7F78fqCvTk/uiQ28LniZoJgnj5cHHhPn5y6MtT0dk/f4ta/fXnh5yjYzBOPqvJF9k4mOc1y+QVFPPbf5Tz8wTL+uyTwmmt3RB+Xu8My6EB+AeNmBy6WZzWtE7Zu6778mNX7W19fRFePJqWjpezM/c83pWta/Mfn65iYVbpteoyZRd3q4Z/T2974hj7PzGHmim1MzNqIKtz0r4W+9ldYdIwX56wNNZ24RfanLNu0L+r7FGwqO3y0iPQRU8IKDUE3/+trbnltIekjptD+4el0evRjIDDefNPePCaWcJ9BMHhW8hH1cjw+/+eO/jhqWffWgZFZkU03902MXYg7dqzkqZHdnchAzFF1Rwr9X+QqXKAPVlnc7aPBAtdr83JYu8O7E+WhD5aF2r/X7zrM1z8EmkU27c3jkj/NoefTs7h47GyqV4me3idydMWmvXmeo0siRzwcKSwKVc2Dhr4Y6JBt3ahm2PKdB49SuVJ0w93PX13geTzxPPh+IHg/+aOzqVa5EvkFRWFD6Fo3rBm1TbCD+LI/zwkbCTIve1dU2qfjjEi4861vOO/xTzj/yeKgNtqjZtOuSW0fRwPb9ufTsFageWLf4QJUlec/XcPw/5dVYkneXRp3X5Tf/noD7R+eHrqQb95buhtiznm0+Es9atBZYeu6//FTz+p9QdGxmBel2av8XayCnvn4+1DJ9qu1gfPT96zTYqY/3tEtkbWZlVv288POQ/zy/2Xx1NTiz0D6iCm8OKfk4YjvfZPL09NXhd6DPu0ahda5+1Ne/mwtV/7tC3o+PTus4DV2evhn7mjhMeat3RkI+qOmcaSwiNmrd/DpquIaxtGiY3QZ/TGDnv+cHmNm8fcS+l6CwTUykHrp/cycuGkgcPc0BErWkSN73MfhNmTcl7QdNQ1V5ZsNe8I6cl//Ksf3QI1t+/wPRqhwgT74sfsye2fojatXo7jU8a8vc45rvweckrdXU427I7PNqGn0cLV3u72zsLjE+uHiTbR7aHpUiT7YZ1BQdIyOzerw7m0XArB8875QVfquS9uE0pdULYNAu2uQu0kl2HSVWjmFOtWrRJWCtx84wtSlW8Judspz2heDpdpFTrXvSY+bLabHGerYp11jdh86Gnce720H/AXY0UPO5m6nVD/qg6X87B8LePaT7/l4RfQQ0vNOrxd6vMQ1rtw9VHLkf5aGbbN4o78q7oeLN0XVnlrWr17iNkXHlDve+ob+f5kbM03kOG4/gsNnr/vHfGeJsuTRfqXez629Wsdcd3kJeY709PRVJXb+RX6Wa1QLL1QFCxZ/dAoRm/bmcc6jH7P70FEWrd/NNI8x5z/7R6AgdLToWFRzZZD7piKvTtAnrj4bKC7o+O3ALO3on1gDCdzfza378lnqfGYnLNzIj1+Yx8C/fs6c1ds5eKSQhz+M7mAHmL16O+kjpvDOwg2hJsXDBcXH8fWDl5WYtwoX6INtk7e/GSgxpo+YEpphLhG+82gqef/bTew6eCTuyJWnpq7iw8Wb2L4/PxRUJnvcqXjoSCEzV24nZ+chzk+vT70aVcI6mO7r1y4s/a2vZ/H09FWkj5gSdcuz+8M94j9Lo8aa5xcWcehIIbMj2lEPHink9jcDHUNur83LobbzBdxxMJ8v1uws8Zhj8QrAXjbuDvR1xButUiWlEgXOsX6+ZidfrYuuZQS1rF8j9Pjlz9aGvpAllXhnLI/O70s3nBe1LHJc/Su/yKRVg+jaERQ35T3/6RqmLNnCujjNXUGxZlIEwmp9t7/5DdeNnx96PnPlduqkVuHS9tHNl7FUq1yJEQPb88ovSh65fHXnZr7298cS7sBs07hW2PMjBUUMPrd4v2f9ITBsNDgSKajfX+ZyjdNUWpKZPj5zXs2T3TICzSvBUUrBIbwPXRGoqX30m55c2r5xWAEM4DeuGwX9BP1YfQvb9ufz6hc/sO9wQdjF4GPXd/mmfy1kRgk3VwVHiT3w3tJQ5/aDrsJM4zqpntsFVbhAn+JxSY5V6r2zTxvP5aX1t1nZdH1iJvf4GCN+94TFPO+6nXyNU2J++eddQ8s6PhI4EcGe+Ea1vEdMBM1Yvi1ULXaPhhjx3hJ+9krxF33SolxufX1R2PjoXQePcvhoEdv2+6/G1asZqCHd+8533PDPkpuOvnigT+jx8BJKhm7fPHw5P/xxUGjYHuBrtMrQzBZx0zSrm8qgc5qGnk9btpVZTlU+3g1kH0SME29erwb39j0TiD1Nbd8OgYtHzzbR/Su5e/L48Qtf8k4p5p9ZOKpv1LLa1WLPFu51wXv1pvPJGXMFz157ruc2XzzQh89+15vHrz6bpY/2R0To2+E0OrWoy8NXdqB6lejppmql+puxvKQRMZHDbutWr8rQruHndPaq7VGTjfm9H6Kkdu9I80dexsz7evH57/uENWPuyysIja/vdWYjcsZcwdnN6/LqTedzX792YZ/ZKUu20MtpRjmRezaemLKC0R+tiGrjjxz5FXl3612XFfdbLcwJr5Gu3nqAb5xRVVd0ako8FS7QV6kcHei9xsp/8UAfftvvTLIeiv7i+FHL48v1tc/hjm94dDq2alDDI2XAGo+5Yvp39C59BtuUJy3KZcLCjWzcHV3LmO1qo7yhe6uo9ZElpkiplWPPK9exWXjH42l1UunRpgEALdKql1gaDeTndOrXrIqIxKxqf/Sbnp7L60R0YEf6Zc8MXrnxfPp3bMLkO3uElt/7zuKwZrVY7nkncCFvXq86rRvWpE3jWqQ5F719eQUlltqe+tE5vH5L+DRNF4+dzTcb9nrebxD03E87hz0Pfu5m3ncJ/Tqcxn2Xn8mXIy8Nra9XI/a5czdZAfz4vBasHD0gKl2LtBq0alCTn3dvRVVX4Jp8Z09u6ZnBysejt/n9gPZhHeIlec91D4qq8pOX5nHtS19FzRfzyOAOVK8a/ll7zeMmr7LQpG4qbRrXpmX9GqEfFwE497GPQ5/L5vWim+Tmj7yM67qdHnq+YfdhDh8tDOtMv+pcf7Wf0D7X+Ysrq7eFjwa65MzowkXQy3OL+0vOOz0t7r4rXKBvEqcKApD95EBapNVARGhYqxoLHrys1G++n3a6FmmBD0Lfs+JXlVNEGD2kY9x0v+sfaLZ54fquPO60HUbmS1XDhghG+uX/K54pon7N6MDQuWW9qGXukorXhSeodkTJrkpKJX55caAkf3bz+HfgPnH1OaHHv+59RtT6S9s35uzmdengGsnSOMYY8UgPXdmBDs6FqHHt4s/JgSOFoZFFkX51cUbUsk1785h1f2+qV00JjTrZe7iAHRGlthevL27aOb1BDS5u24i7L/MXDCHQNn51l/CpnYKBr03jWoz/RSZ3XdY27AJ3pLCIH/44yHN/Xl/oyEDqPs8lcR8bBC6yv7m0LVd3bkbTiKHBf7uuS9g4/t+++x3pI6YwfdlW7n93CQtz9kQVksZe04k6qVWiRvX4HUHlh59YEeT1Gavq8V6l1azKY4PDv8cd/jAjbHDI367r4lkzi9Te50CEWDq3jB3A81wdysGCWEkqXKCvFKc9vmOzOmFXaAiUOn/ulGy7tip+c16/pRtv/vICz/1kPdSXfh28S9VXnNOUX/bM4PbegaahRrWreVZ3I/P9iwvTw5a99+tAR6y7hHC7E/xSKgk/796KF66PbieObG+PJdjRG2yDhEBw8aqtLH+sP7+4MLr0HxQM8PkFx7g84n3p064xK0b3DwWaIR7tub+6OINFEbWrrh6B6c8/CTQ39GkfGJEx+/7efDmiuEQbWWoOurlHethzdwd9LEO7tuDBQWdFjYAK30/gQrn38NESL65B915+JqufiC4RewneLzDjnl50Ob0eY358Tsy0qVWKf9JOPJovHxzUngcGtvfc9rZLii+oXz/or4Y70NX8lf3kQCDwmXxuWBc++k1Pbu99Bt8+fDlLHu3HVec2Y8HI6M6+Ef9Zwnsew0DXPDmQa88PzHzStnEtRg06i7d+Ff099LpoXpBRP24fxNnN6/DdI/1Cn38/nr22c9Qyr1Fw4H0BmOoMfgjWxGPdwAbQ2xlt9LfruniuP6OEz6NbSX2T7o7r9k3qxEwXVOECfbxu17ObeZcqu7ZK45aeGTzvenMvbtuIHm0asnL0gKjOqIa1qjH+F5meHXiPXNWBh67swKBzmtC5ZT1uvCidBaNK7tX26lvo2ioQgPu5mmkiv8SDzmlKrzMbhS37v9f8ze0WLNEEh5IC3N6nDTkRHVK3XtKayimVOK2EElCwE/zcFnW569LoL2CNqsUXD3dzxDDnCz3qig40iOiLSPOobQRLoPdd3o7Pf9+HjIY1qeK6cJ95mncp6P96hJfMU2NceIPtlXVSK/PMT85FRMhsFX7Bue/yM0OP6zuBfuhLX/F5RMd0rJEykUNq3dy1v+Dsl+2a1Ob923swzHXBjzTptouA4qF/X0d83jq3TAt7n9yCF/DHh3Skro8LYKTIglODWtX4/YD2pNWsGqptVKok1IyoPXj1nQ07v2VYPkWEX/VqzUVnRDdD3Os6D0HXd28VVQvKGXNFWC3jjVsuoG71KlEX+wsyYs82Gqydu3ldUGMJDl+9/oLiwlKwALFwVF/eGd6dLx7ow4rR/fmX04fSulEtz329/avuYTdUrnsqvAY3b8SlfPeHwOiqv/8s/EwxTNIAABWMSURBVGIx875LfOfZrcIFeveH7jcRveAf/aYnj8VoHkmpJDx8ZQea16tORsOaYe2N1aum0LfDaSx7LPADV+7xveOuj77qBjum6tWoygd39KB9kzrUSa3CazefH0rTvF71sA7Y4DbB0u6FrYurU63qx26/h/DAUxrBYOdu865bvUrYHB/T77mYEQMCJcF4TSTT7r6YUVd0iKpuR3J/QcZc0ylmu32aR9AJNi2kVJKw0TNB7o7r0531yx/r75n2eY8S05VOSdV9z8PYoeGdlu5+Da9OyGDzWqySUkkBYubK7Tw4KPB+/6iL/xm5a0bUwtxNUwA1qsauUTarV50Vo/t79teUZOzQTrwVo8br5YsHLo2b5gmP5sjSOLtZHQaf24zvHulH7dTKoVL/G658Bi8kkTcllvQD335qgG5rnFpOJHcBY8pvLubrUZfRqHY1LmjdgBZpNahRtXLo8xGrRN64TmqowNPrzEZUqiRhMalmtcqhC/aVnZqF1dgjRzaNvaaTr+OpcIFegDd/eQHT7r6Y3/Zrx/yRl3Fx24YsebQfZzevG7Mk5zb7/t7c0zc6eNaqVpn3fn1RWICo5tEx6S69urlHXmzam8flrtpA8EavYKB3V+3SSuhgg9hteWuf8m6rDQo2t0S2nbuHyrVvUif0wbvmvBZh7XnN6qaGjaQ5q2kdqlauRJ3q8UdgdG2VVmL1FbyPO14pqlIl4Y1bLuDz3/dh8p09mPu7PlFBMGjwuc246aL0sGV1Ylyk3P0W7gtZs3rRtZyrOjVj5egBof6A0vi/HhkM73UGOWOuiGo/L0nLtOr0Peu0sIu2u7OwpEAfWF+5VCVUgGszW3KRx2iiWLxqaJEiawduwQsowCqnQ/id4d0B6HJ6PbIe6hsqBdetXoWlj/YPlfrPaFSLsdd0okHNqmExYOKtF3JLz0Btr3rVFP46rDPfPHx51GvH6+iPFKv2FGxiC75e5AXZr+C9LMF7Y9yflch+lldvChQwH70q8of9IMNnM1CFC/QAPdo0DN123qRuKq/fckGpT1QsXVulRZUE/HJ/iK/NbBHWnxD8kvVq24hbL2nNg667KWMFn6DUKimsHD0g1FYKgbZ2rxKBu5rn/kB89rvezHfaUf/y0+j2SAgE0eCXomurNOaNvIzf928XlS74/pTU8fjery+K2yHlJzB46dm2IS3r16BejaqcXsJoJiB0PEGqgSAx7mfhfR9/u64LPz6vOd8/MTDsffW60CsaN0gP6Bg9adVfh3Xmnsv9d9a6VU6pxCs3ZoZdtKfefXHocWkuGifDHX2iO9rjb9OGVY8PYPlj/UPBOhioMlulhe6MjuXa81uy6OHLw85ft4z6XH9BoEns3Bb1GNK5uecAhUqVJNSs1rJ+dRbEucEoljwfd9W63XpJoCB1WYx+h3nOXc/3Xd6OhrWqMnJg+6jCbK1qlckZcwU3Oc2X7mZov53v/gbPJrm1Tw1CgH7PzfVd3f7p+YEP18JRfcPufKucUomRA8Nvmfdzw1fwi/zcTzuzMGc3I50LxdqnBpGz6xDpDWqiqlSqJLx4/Xms2nogrATnvqlHRHjzlxd4zs4Z/DIddJo2KqdUYvKdPWiRVhxQUypJ3GGUfi0c1ZcdB44w6PnP43ZoH4+W9WuQM+YKnpmxmr/PzqZDs0BHnVc6rw45L/FqKgADzm4SdffwkM6J/fE0d80jUQWdE/Wvm87nb7PWcH+/dqG5gIIuiehr8hIZxBrXTuWLB/qU2H8UT+tGtfjk3l5keEz74fbbfu2YuXI7N12U4ev1Vj0+gJv/tTDsXoZzPUa0lWTkwLMYOfAs8o4W8eKc7FCwblCzatjEZW0a1yLroeiaiJe+HU6jX4fT+HjFNs7xMRIOQCrar8pnZmZqVlbF/aGp4O3xX4641HMcbrztEhVAj9fG3Ye5eOxsmterHjba5X/Z32etCc0TNPv+3nEDRtCKzft5bd4PTMzK5a5L20Td8ZwI2/bns27HIS48I/4QupPt+lfm82X2Lj797SWcEaPjsaL5Yech0hvUKFUz1zmPzKBbRn3+8YvMuKMC/dqyL48L/ziLd2+7MGE/WSkii2L9eJOV6Etp7VOD2Lw3r1RBHgJt96Xdpiw0qZtKeoMaYU1L/+tu6N6Kyd9tpkebhr6DPECHZnXo0aYhE7NyOaNx2QS60+qknlBptyz95aedmb5s6ykT5IFSnd+gpc4gjkRqWjf+zYeJZCV6Y06AqrJ44146t6xX6s5QYxLJSvTGlBERoYuPW9CNKU8VctSNMcaYxLFAb4wxSc5XoBeRASKyWkSyRWSEx/r7RGSFiCwRkU9FpJVrXZGILHb+Jicy88YYY+KL20YvIinAOOByIBdYKCKTVdU9J+m3QKaqHhaRXwNjgZ866/JU1d8AZmOMMQnnp0TfDchW1XWqehSYAAxxJ1DV2aoanElrPhD/FySMMcacFH4CfXPA/RM6uc6yWG4B3D/fnioiWSIyX0Su9tpARIY7abJ27EjcfNXGGGP8Da/0GhzsOfheRG4AMgH3XJqnq+pmEWkNzBKRpaoadu+0qo4HxkNgHL2vnBtjjPHFT4k+F2jpet4CiPpFbBHpC4wCBqtq6Kd6VHWz838dMAfwno3fGGNMmfAT6BcCbUUkQ0SqAsOAsNEzItIFeJlAkN/uWp4mItWcxw2BHkD4D0saY4wpU3GbblS1UETuBGYAKcCrqrpcREYDWao6GfgTUAt417kNfIOqDgbOAl4WkWMELipjIkbrGGOMKWM2140xxiSBkua6sTtjjTEmyVmgN8aYJGeB3hhjkpwFemOMSXIW6I0xJslZoDfGmCRngd4YY5KcBXpjjElyFuiNMSbJWaA3xpgkZ4HeGGOSnAV6Y4xJchbojTEmyVmgN8aYJGeB3hhjkpwFemOMSXIW6I0xJslZoDfGmCTnK9CLyAARWS0i2SIywmP9fSKyQkSWiMinItLKte5GEVnj/N2YyMwbY4yJL26gF5EUYBwwEOgAXCciHSKSfQtkqmonYBIw1tm2PvAIcAHQDXhERNISl31jjDHx+CnRdwOyVXWdqh4FJgBD3AlUdbaqHnaezgdaOI/7A5+o6m5V3QN8AgxITNaNMcb44SfQNwc2up7nOstiuQWYVpptRWS4iGSJSNaOHTt8ZMkYY4xffgK9eCxTz4QiNwCZwJ9Ks62qjlfVTFXNbNSokY8sGWOM8ctPoM8FWrqetwA2RyYSkb7AKGCwqh4pzbbGGGPKjp9AvxBoKyIZIlIVGAZMdicQkS7AywSC/HbXqhlAPxFJczph+znLjDHGnCSV4yVQ1UIRuZNAgE4BXlXV5SIyGshS1ckEmmpqAe+KCMAGVR2sqrtF5HECFwuA0aq6u0yOxBhjjCdR9WxuLzeZmZmalZVV3tkwxphTiogsUtVMr3V2Z6wxxiQ5C/TGGJPkLNAbY0ySs0BvjDFJzgK9McYkOQv0xhiT5CzQG2NMkrNAb4wxSc4CvTHGJDkL9MYYk+Qs0BtjTJKzQG+MMUnOAr0xxiQ5C/TGGJPkLNAbY0ySs0BvjDFJzgK9McYkOQv0xhiT5HwFehEZICKrRSRbREZ4rO8lIt+ISKGIDI1YVyQii52/yZHbGmOMKVtxfxxcRFKAccDlQC6wUEQmq+oKV7INwE3A/R67yFPVzgnIqzHGmOMQN9AD3YBsVV0HICITgCFAKNCrao6z7lgZ5NEYY8wJ8NN00xzY6Hqe6yzzK1VEskRkvohc7ZVARIY7abJ27NhRil0bY4yJx0+gF49lWorXOF1VM4GfAc+JyBlRO1Mdr6qZqprZqFGjUuzaGGNMPH4CfS7Q0vW8BbDZ7wuo6mbn/zpgDtClFPkzxhhzgvwE+oVAWxHJEJGqwDDA1+gZEUkTkWrO44ZAD1xt+8YYY8pe3ECvqoXAncAMYCUwUVWXi8hoERkMICLni0gu8BPgZRFZ7mx+FpAlIt8Bs4ExEaN1jDHGlDFRLU1ze9nLzMzUrKys8s6GMcacUkRkkdMfGsXujDXGmCRngd4YY5KcBXpjjElyFuiNMSbJWaA3xpgkZ4HeGGOSnAV6Y4xJchbojTEmyVmgN8aYJGeB3hhjkpwFemOMSXIW6I0xJslZoDfGmCRngd4YY5KcBXpjjElyFuiNMSbJWaA3xpgkZ4HeGGOSnK9ALyIDRGS1iGSLyAiP9b1E5BsRKRSRoRHrbhSRNc7fjYnKuDHGGH/iBnoRSQHGAQOBDsB1ItIhItkG4CbgrYht6wOPABcA3YBHRCTtxLNtjDHGLz8l+m5AtqquU9WjwARgiDuBquao6hLgWMS2/YFPVHW3qu4BPgEGJCDfxhhjfPIT6JsDG13Pc51lfpzItsYYYxLAT6AXj2Xqc/++thWR4SKSJSJZO3bs8LlrY4wxfvgJ9LlAS9fzFsBmn/v3ta2qjlfVTFXNbNSokc9dG2OM8cNPoF8ItBWRDBGpCgwDJvvc/wygn4ikOZ2w/ZxlxhhjTpK4gV5VC4E7CQTolcBEVV0uIqNFZDCAiJwvIrnAT4CXRWS5s+1u4HECF4uFwGhnmTHGmJNEVP02t58cmZmZmpWVVd7ZMMaYU4qILFLVTK91dmesMcYkOQv0xhiT5CzQG2NMkrNAb4wxSc4CvTHGJDkL9MYYk+Qs0BtjTJKzQG+MMUnOAr0xxiQ5C/TGGJPkLNAbY0ySs0BvjDFJzgK9McYkOQv0xhiT5CzQG2NMkrNAb4wxSc4CvTHGJDkL9MYYk+R8BXoRGSAiq0UkW0RGeKyvJiLvOOsXiEi6szxdRPJEZLHz91Jis2+MMSaeyvESiEgKMA64HMgFForIZFVd4Up2C7BHVduIyDDgaeCnzrq1qto5wfk2xhjjk58SfTcgW1XXqepRYAIwJCLNEODfzuNJwGUiIonLpjHGmOPlJ9A3Bza6nuc6yzzTqGohsA9o4KzLEJFvReQzEbn4BPNrjDGmlOI23QBeJXP1mWYLcLqq7hKRrsAHItJRVfeHbSwyHBgOcPrpp/vIkjHGGL/8lOhzgZau5y2AzbHSiEhloC6wW1WPqOouAFVdBKwFzox8AVUdr6qZqprZqFGj0h+FMcaYmPwE+oVAWxHJEJGqwDBgckSaycCNzuOhwCxVVRFp5HTmIiKtgbbAusRk3RhjjB9xm25UtVBE7gRmACnAq6q6XERGA1mqOhn4J/C6iGQDuwlcDAB6AaNFpBAoAm5T1d1lcSDGGGO8iWpkc3v5yszM1KysrPLOhjHGnFJEZJGqZnqtsztjjTEmyVmgN8aYJGeB3hhjkpwFemOMSXIW6I0xJslZoDfGmCRngd4YY5KcBXpjjElyFuiNMSbJWaA3xpgkZ4HeGGOSnAV6Y4xJchbojTEmyVmgN8aYJGeB3hhjkpwFemOMSXIW6I0xJslZoDfGmCTnK9CLyAARWS0i2SIywmN9NRF5x1m/QETSXetGOstXi0j/xGXdGGOMH3EDvYikAOOAgUAH4DoR6RCR7BZgj6q2Af4CPO1s24HAD4V3BAYALzj7M8YYc5L4KdF3A7JVdZ2qHgUmAEMi0gwB/u08ngRcJiLiLJ+gqkdU9Qcg29mfMcaYk8RPoG8ObHQ9z3WWeaZR1UJgH9DA57bGGGPKUGUfacRjmfpM42dbRGQ4MNx5ekRElvnIVzJoCOws70ycBHacycWOs2JqFWuFn0CfC7R0PW8BbI6RJldEKgN1gd0+t0VVxwPjAUQkS1UzfeTrlPe/cqx2nMnFjvPU46fpZiHQVkQyRKQqgc7VyRFpJgM3Oo+HArNUVZ3lw5xRORlAW+DrxGTdGGOMH3FL9KpaKCJ3AjOAFOBVVV0uIqOBLFWdDPwTeF1EsgmU5Ic52y4XkYnACqAQuENVi8roWIwxxnjw03SDqk4FpkYs+4PrcT7wkxjbPgk8WYo8jS9F2lPd/8qx2nEmFzvOU4wEWliMMcYkK5sCwRhjklyFCvTxplqo6ESkpYjMFpGVIrJcRO52ltcXkU9EZI3zP81ZLiLyvHO8S0TkPNe+bnTSrxGRG2O9ZnkRkRQR+VZEPnKeZzjTX6xxpsOo6iw/pafHEJF6IjJJRFY55/XCJD2f9zqf2WUi8raIpCbDORWRV0Vku3vIdiLPn4h0FZGlzjbPi4jXkPLyp6oV4o9AR+9aoDVQFfgO6FDe+SrlMTQFznMe1wa+JzBtxFhghLN8BPC083gQMI3A/QbdgQXO8vrAOud/mvM4rbyPL+JY7wPeAj5ynk8EhjmPXwJ+7Ty+HXjJeTwMeMd53ME5x9WADOfcp5T3cXkc57+BXzqPqwL1ku18EriJ8Qeguutc3pQM5xToBZwHLHMtS9j5IzCK8EJnm2nAwPI+n57vQ3lnwPXmXwjMcD0fCYws73yd4DF9CFwOrAaaOsuaAqudxy8D17nSr3bWXwe87Foelq68/wjcD/EpcCnwkfMh3wlUjjyXBEZrXeg8ruykk8jz605XUf6AOk4AlIjlyXY+g3ew13fO0UdA/2Q5p0B6RKBPyPlz1q1yLQ9LV5H+KlLTTVJNl+BUZ7sAC4DTVHULgPO/sZMs1jFX9PfiOeD3wDHneQNgrwamv4Dw/J7K02O0BnYA/3KaqV4RkZok2flU1U3AM8AGYAuBc7SI5DynkLjz19x5HLm8wqlIgd7XdAmnAhGpBbwH3KOq+0tK6rHM99QR5UFErgS2q+oi92KPpBpnXYU9RpfKBKr9L6pqF+AQgap+LKfksTpt1EMINLc0A2oSmK02UjKc05KU9rhOmeOtSIHe13QJFZ2IVCEQ5N9U1f84i7eJSFNnfVNgu7M81jFX5PeiBzBYRHIIzGR6KYESfj0JTH8B4fkNHYscx/QY5SwXyFXVBc7zSQQCfzKdT4C+wA+qukNVC4D/ABeRnOcUEnf+cp3HkcsrnIoU6P1MtVChOT3u/wRWquqzrlXuKSJuJNB2H1z+C6e3vzuwz6lKzgD6iUiaU9rq5ywrd6o6UlVbqGo6gXM0S1WvB2YTmP4Coo/xlJweQ1W3AhtFpJ2z6DICd3knzfl0bAC6i0gN5zMcPM6kO6eOhJw/Z90BEenuvG+/cO2rYinvToKITpNBBEaqrAVGlXd+jiP/PQlU3ZYAi52/QQTaLz8F1jj/6zvphcCPuqwFlgKZrn39H4H5+7OBm8v72GIcb2+KR920JvClzgbeBao5y1Od59nO+tau7Uc5x76aijpaAToDWc45/YDAqIukO5/AY8AqYBnwOoGRM6f8OQXeJtDvUECgBH5LIs8fkOm8Z2uBvxPRcV9R/uzOWGOMSXIVqenGGGNMGbBAb4wxSc4CvTHGJDkL9MYYk+Qs0BtjTJKzQG+MMUnOAr0xxiQ5C/TGGJPk/j/XHR+PDmIIlQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.9629790186882019.\n"
     ]
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15_labelsmoothing');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',monitor='accuracy',\n",
    "                                                   name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas5_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas5_sp15_labelsmoothing');\n",
    "learn_c.save(f'{lang}clas_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.281384</td>\n",
       "      <td>0.265413</td>\n",
       "      <td>0.962849</td>\n",
       "      <td>0.978329</td>\n",
       "      <td>06:18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.275690</td>\n",
       "      <td>0.285689</td>\n",
       "      <td>0.964106</td>\n",
       "      <td>0.979017</td>\n",
       "      <td>06:17</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.273247</td>\n",
       "      <td>0.272905</td>\n",
       "      <td>0.964280</td>\n",
       "      <td>0.979153</td>\n",
       "      <td>06:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3xV5f3A8c83GxJm2DNhyBJkhCEoggMZClZpRapVa7Wun1VbK9aFiIraWrXFAa2tWhQVR2lFERTEAUoYskdARpgBkjBCyHp+f9xzb+44996TcEOS2+/79cor957znHOfc8f3POdZR4wxKKWUil4x1Z0BpZRSVUsDvVJKRTkN9EopFeU00CulVJTTQK+UUlEurroz4C+2bgPTIT2NBnXiqzsrSilVa6xYseKQMaap3boaF+jjGjRj0qsfcdN56dWdFaWUqjVEZGewdTWy6kb79iulVOTUyECvlFIqcmpkoNcCvVJKRU6Nq6MHMGikV0o5V1xcTHZ2NoWFhdWdlSqXlJREmzZtiI933mHFUaAXkZHAC0As8DdjzDS/9bcCdwClwHHgFmPMBhFJAzYCm62ky4wxt4Z7PS3RK6UqIjs7m3r16pGWloaIVHd2qowxhsOHD5OdnU16uvMOK2EDvYjEAtOBS4BsYLmIzDXGbPBK9pYx5hUr/VjgOWCktW6bMaa34xyBlueVUhVSWFgY9UEeQERITU0lJyenQts5qaMfAGQZY7YbY4qA2cA47wTGmKNeT5M5zVitJXqlVEVFe5B3q8xxOgn0rYHdXs+zrWX+L36HiGwDngHu8lqVLiKrRORLETnf7gVE5BYRyRSRTNA6eqWUiiQngd7u9BEQiY0x040xHYH7gYesxfuAdsaYPsC9wFsiUt9m2xnGmAxjTIbrudPsK6VU9cvLy+Oll16q8HajR48mLy+vCnLky0mgzwbaej1vA+wNkX42cAWAMeaUMeaw9XgFsA04q3JZVUqpmilYoC8tLQ253bx582jYsGFVZcvDSaBfDnQWkXQRSQAmAHO9E4hIZ6+nY4Ct1vKmVmMuItIB6Axsj0TGlVKqppg0aRLbtm2jd+/e9O/fn+HDhzNx4kR69uwJwBVXXEG/fv3o0aMHM2bM8GyXlpbGoUOH2LFjB926dePmm2+mR48ejBgxgpMnT0Ysf2F73RhjSkTkTmA+ru6Vrxlj1ovIFCDTGDMXuFNELgaKgVzgemvzocAUESnB1fXyVmPMEQevWbmjUUr9z3vsP+vZsPdo+IQV0L1VfR69vEfQ9dOmTWPdunWsXr2axYsXM2bMGNatW+fpAvnaa6/RuHFjTp48Sf/+/bnqqqtITU312cfWrVt5++23mTlzJj/72c94//33ufbaayOSf0f96I0x84B5fsse8Xr8myDbvQ+8X9FMaZxXStVmAwYM8Onn/uKLL/Lhhx8CsHv3brZu3RoQ6NPT0+nd29UTvV+/fuzYsSNi+amhI2OVUqpyQpW8z5Tk5GTP48WLF7Nw4UKWLl1K3bp1GTZsmO0I3sTERM/j2NjYiFbd6Fw3Sil1murVq8exY8ds1+Xn59OoUSPq1q3Lpk2bWLZs2RnOXY0t0WukV0rVHqmpqQwZMoSzzz6bOnXq0Lx5c8+6kSNH8sorr9CrVy+6dOnCoEGDznj+amag1zivlKpl3nrrLdvliYmJfPLJJ7br3PXwTZo0Yd26dZ7lv/vd7yKat5pZdVPdGVBKqShSIwO9UkqpyKmZgV7rbpRSKmJqZKAv0zivlFIRUyMDvfa6UUqpyKmZgV7jvFJKRUyNDPRKKRXNUlJSANi7dy/jx4+3TTNs2DAyMzMj8no1MtBrgV4p9b+gVatWzJkzp8pfRwdMKaXUabr//vtp3749t99+OwCTJ09GRFiyZAm5ubkUFxczdepUxo3zuQsrO3bs4LLLLmPdunWcPHmSG2+8kQ0bNtCtW7czO01xddDGWKVUpX0yCfavjew+W/SEUdOCrp4wYQJ33323J9C/++67fPrpp9xzzz3Ur1+fQ4cOMWjQIMaOHRv0nq8vv/wydevWZc2aNaxZs4a+fftGLPs1MtBrnFdK1SZ9+vTh4MGD7N27l5ycHBo1akTLli255557WLJkCTExMezZs4cDBw7QokUL230sWbKEu+5y3W67V69e9OrVK2L5q5GBXuO8UqrSQpS8q9L48eOZM2cO+/fvZ8KECcyaNYucnBxWrFhBfHw8aWlpttMTewtW2j9dNbMxVivplVK1zIQJE5g9ezZz5sxh/Pjx5Ofn06xZM+Lj41m0aBE7d+4Muf3QoUOZNWsWAOvWrWPNmjURy1vNLNFrnFdK1TI9evTg2LFjtG7dmpYtW/Lzn/+cyy+/nIyMDHr37k3Xrl1Dbn/bbbdx44030qtXL3r37s2AAQMiljepaaXnxJadTcvrn2fHtDER22dhcSkJsTHExFTNZZFSqnpt3LiRbt26VXc2zhi74xWRFcaYDLv0NbLqBuBUSWlE9lNSWkbXhz9lyn83RGR//8vWZufzydp9YdMZY8g6aH+3HaXUmeco0IvISBHZLCJZIjLJZv2tIrJWRFaLyNci0t1r3QPWdptF5FKnGTtZFBjol247zLPzNzndBQCnSsoAmL18l+36eWv3cej4KUf7euw/63n03+vCJ4xSl//1a26btZIjJ4pCppu9fDcXP7eEpdsOn6GcKaVCCRvoRSQWmA6MAroD13gHcstbxpiexpjewDPAc9a23YEJQA9gJPCStb+wiqwA7e2amcuYvmibk80D9hNr05p9tLCY22et5Irp3zja1z++2cHrS0M3qPwvWLY9dACf+dV2AF7/dscZyI1SLjWtGrqqVOY4nZToBwBZxpjtxpgiYDbgM7zLGHPU62ky5T0kxwGzjTGnjDE/AlnW/sI6WlgcdN3Owyec7IKDxwrJzg0+usxY55JQaYJZsTOXt7+3v0qw8+/VeyqU3qnSMsO9765m0/6j4RP7yT1RRNbB4xXebuXO3JDr8wtcn93GSuSpJtu8/xiFxZGpUqwqOw6d4M2lO6o7G2dcUlIShw8fjrpgX1pmKC0rL/QaYzh8+DBJSUkV2o+TXjetgd1ez7OBgf6JROQO4F4gAbjQa1vvW55nW8vCemPpTqaMO9t23QXPLub7By+iWb3QBzvgic+dvBQA0xdlcWmPFnRqluIo/VUvfwvANQPaOUr/m9mrK5TeqW05x/lg5R4yd+Sy5PfDK7Tt6Be/Yl9+oafh2xjjqB9vk3qJIdcftqp2dh4uqFB+QnFX27118yCS4h1dFEbU8VMlXPr8EoZ3aco/boxcb4hI++mrS8k5doqr+7cjIa7GNsFFXJs2bcjOziYnJ6e6sxJR7kJom0Z1PMuSkpJo06ZNhfbjJNDb/fIDTpvGmOnAdBGZCDwEXO90WxG5BbgFIKFFJ8C+jt7bmBe/ZvmDF4fJus9rBCwr9Tr7Pzt/M8/O31zh3j7fbjvE4I5NKrRNJJVad2nZdaSAtEkfc+OQNB69vIejbffluwZvfLklh4Hpjen68KfcdWEn7h3RxTZ9ckIsJ4pKmfbJJm69oGNkDsCha2a6ygtLtuQwoof9yMKqVFLqKlUt2hz5QJJ18BidmtWLyL5yjrnamwpLSv+nAn18fDzp6enVnY0K25d/koTYGFJT7AtPoyZ9DHDavRCdfBOygbZez9sAe0Oknw1cUZFtjTEzjDEZ3l2DxvRqGTJT7i+0v798vpVxf/065LbWa1ao2iLY5fDEmd/ZLj98/FTEeg6FUup3O65/fLOjwvvIzi0gz6pueSdzd9B0MZUYtbcmOy+il9M7HFbb+csrKApa7bJkS07IqkLwveuZ3fGUlpmwjdR2bp+1goufW8LMJdsrvG0o/6zE9+B0GGM4eCz0qE8V6NynvqDf1IV8vfVQhbY7fqqED1dlc82MZTw1b2PY9E4C/XKgs4iki0gCrsbVud4JRKSz19MxwFbr8Vxggogkikg60Bn43sFrkhDrmzWnweJPC7bwQ3Y+ZWHuRzh7+W5+9upSn2VxXv3s31m+i91HyqseHv73es/jgqISn+3WZucH7L/f1IV0eehTT0nQLa/AFQxmLtlO2qSPHR1XSWlZldYNP/jhOs8xHTh6irRJH7MvP7DdIlScP1pYzDvLd9HpD/N8lo/96zf8fk7kRvi1T02u1Ha9pyxgwoxlAcuzDh7jF699z3nTvgi5vfcJ9WObLqYd/zCPvo8vIM0qgTk1b+1+IHivsMqy60m2P7/Q8/2LtHczdzPgic9ZuSt0+02kzVu7jwNHz+wJpqS0jOOnSsIn9DPru52sCNK+9eBHFZuE7dq/fcc97/zA0u2HedVBISFsoDfGlAB3AvOBjcC7xpj1IjJFRMZaye4UkfUishpXPf311rbrgXeBDcCnwB3GGEcRq8T6Ye3JO0lZmeHlLyvW26YkRKBfuOEAD3wQ+MaWlBmWbjtMz0fnc//7a20DAwTe0zY717cu+sst5Zf3f1qwhUWbDnqebz14nP35hTxhnYUnz13PQx+tJW3Sx0G/BBNmLKPrw596nucVFJE26WM+WbvP9jhX786z3U8oby7z7U10x6yVnseP/Wc9aZM+5mhh8C/3/XPWcP/7a23z896K7ArnJ5h6SeW1jb96PZO0SR9z2GH32NW785i5ZDvr9uRz4lQJpWWGb7JcPYj8j23BhgNs3n/MZ1u3O99adTqHYOvEqfA/i7RJHzs+keQWBF6hDHrqc3pPWVDhvDnxrdWV9sqXvmVXBNtm/BUWl/LkvI2cLCqluLSM22etZPwr31bZ69m5a/Yqzn50foW3e/DDdZ62PX927VlnPfRJ0H1V9DfuqBLPGDPPGHOWMaajMeYJa9kjxpi51uPfGGN6GGN6G2OGWwHeve0T1nZdjDHBc+6n1BiycwsYMu0Lnl+4hWc+3RyQprg0sAum279X7/F5ftKrRPyrN4LfteWamcs4Zp2t9+TZ98Yp8yuFF1pVNCdOlbBqVy7Xv1Z+0fLy4m3c+M/lnucCTPqgvIT7+tKd/GuZqzR31cvf2lb3ZHqdAIpLyzw/1ttmrbTthuq0u6g3/yqflbvygq7zl19QzCfr9lf4Nb2t3p3Hv1fvYZ5faTk7t4CMqfbBaeHGAwD8eeGWkPv2vrp7Yt5GLvvL1/R4dD4PfriWYIOlb34jk0ufX+J5vmRL6Lp57/pw/+o0J/bblEqzcwtsrxad+M8PwWtX3aX9ktKyiJXwv//xiOfx0GcXceubKyq8j7IyQ/oDrpNZSWkZuw4XBFzx/uObHcxYsp2ZX23nhPU73X3kZNBCUlVwX4VVVtqkj4MGfG/ev+1w36lwBYAa21pTWmo4dNz1JXzxiyzbNKHeLO/SGFTuxxfMbf/y/RLf884PvL8im3veWc1PXgr9AcbFxrD1QPC2Af/+/t6X4MaYgCBele0AxwqLuffd1bbrVuw8wn6rMfecKZ+d9mtdMf0bfjN7NbfPWump7tq8/xi/fnOF53sAUGZzbg9X+3XK5mQIruo7pz14/K94/Hn/KL/7sXycQWFxKd9mHeKz9eXBYeGGAz5XfcGc9/QiLrdpb6pIm8fSbYdJm/Qxs7269uZabQndH5lP7ykLAqoiK8P/RPXp+ooHw7k/7PV8lku25jD02UV85Fdgc7/PxaVlPp/rVS9/G7QKp6zMMOqFrzzf10g5nbanUCemsjIT0Gbj3f7xSCUGbdbYQF9SVhZQT+9vTXY+Ow+fYOWuXPILij1neIC/ff1jQPrKfDDr9waWqNyX+95++94PfLbhQNj9FRaXBg08APknfS+5Bz1Z3kU0r6CY9Xt9+6bblehdacOX1Iae1TTk+ucXbuWDlXts11318lIGPeW8+2qo+mv/z+WRues5YXVn9D/eD1cF5mfWd7712yeLSj29tt5Zvotuj3wasI1bR6s7bROr10NJaVmFvyf+VRUTZ37nGUvQ9eFPmfi377jlzRWM++vXLNp8kF+9kelz1RfO+r35Pp+z/3sSiru30iSvqkp3b7Mi64T6XqZ91dof529miNV28eOhE7wfogquW4v6Acse/HAtuw4XhG3odvP+zi7Z4mqcvOedH2zTCoEn+IFP2n8f//jZZjbuO2r7fT1aWBz0NxROZRrfveXbVK+Bq+r3Cb8GVu9R5m9UYtBmjZy9EqC41PDcgtCX5ODqU+/Uv77bRbvGdSuUjzEvhu/BUxGFxaUhp1w4VVLGQx+tpXOzelw/OM2nztuudB2sZPDAB2vJzj3J2j35bH1iFPGxMSzZksPJ4lIutbonxobpRPN3m5NlZbl/FKVlhli/+pIn/b7Ub323i7e+s2+cfH9lNk/8xH58BcDuIwWc/8wiAP55Y3/ufz90I5e7WkfElce+jy/grgs7edbnFRRRNyH0z2Tos4sClj05byNPj/e9ccQP2fnc+I/lAWnDGfPi11w7qHz8Re5pVrfsyyukayOho+yhhRzh+LJMKK4Lx/dDXBLUaQhJDdn55Q46kkzJ7vr86uU1HCpL5qo+4yEm8Cpow77Ak8+s73Yx67tdNKobzxe/HcbiLQdp3bAuA9Ib2+brT5+V/97/6TWquri0jFW78hiQ3thz97l/fruDUT1D98xz8y88uf2wO49x07/h3A6pvH3LIM/yoc8sYlCHxjwz/pyAbS77y1eex/2mLrTt9niyqJT3V2bz84HtQo5L2X7I/sp+8eaDAcsKiyt3MnKrsYE+6+BxTx1spDz8UfXPU1NcGrq0+OLnW5m93NXFccsB3+onuz7cLy22b6Q+UVTK2j2uq5FRL3zFwnsv4BdWKXLoWU1ZsiWHZmEGPjnxbVbFuoX9/evt3Hx+B58fwMyvKnZC6Tl5vu3VXlmZ8QR5gBscBFV3ySnn2Cn+YJV6vasKe09ZwLAugVc+R04U0Tg5Ieh+38ncHRDoK8L/qmKVV5vJdX//PiDAFBaX8tBH6wBDQ47TUo7w4NN/4ppYVzBvyRFaiOsvfU4+lBznc/fHfxRYBCQ1hNIiKHZdofzFfXh/f5rP3Y+n/JqCmGSOSwrNmrVwnRTqNGJq3FHySSbPpJBPMvnW/zyTQn5BMhOmf87mI6WAsGPaGAqKSrjypW/54PbBnhPpsSA9Wfo/sZC8gmJeubafZ3T50cISRr3wlW36/3t7FSt35vLNJNe4zXi/78qn6/Zzq1f161K/KT12HSlg15ECT6Bfk53H1gPHuapfG9btCX819cz8Tfzjmx1M/XgDmx4fFTTdmiDtL+/bXEV3aeFsIGcwNTbQv/D5Vtvl4/u1YU4Ee3GcaZ+Fqbt0B3kIrJKoiMwd5Y1j/uMF3A2LB4OMRaiIiX+zH0cQzJPzNnHoeBF/GO2aYtVpjxlvxaWG4tLAtolgjeeheAfQYPXKi21OsA9+uJaXr+1Xqdd0+3CV7/fYGMOSrYdITU6gawvfAVTr9x4lhjJSyaelHKFo3VwSTuyHo3vg6F6O7v2RO3J2MTXxCElilWBPAvFQZoSDNGS/acx204pvCs9mv2nMPtOY/aYx+2nMAdOIjY9cQUyMcNakj2jACRrIcRpwgoae/77LRiXWIfFUPuRnMyr2AA04QZwEKXkWQFFiLPmkkPVIMnmkcJ9JZsETKYwb1APqNOKG2D1eJ4pknxMFxLF480EOHA39fSkoKvE0RKdN+pgd08Z4qqjcbv1XxRqKx/7V1S7mf+UJrs/syy059G3fiPpJ8UD5FURhcRnZuQUs2nSQHq0bBGy7168L86pduXy77bBt181wBcRwamygD+aRy7vX6kAfya6GoRT4jSwO1UPpTPt84wH+MLobOw6dYNgfF0dkn7sOF/DG0h0R2ZcTX2w6yHuZu7kvxBiBFTuPBF0H8OqX5Q1ucZQw4Zl3KcnbQwvJZUKXGB6K22yVwnNpIUdoTi7xYn2uc1z/ikwspSktORWTyjqTzoKyDN8gbhqTQwNKHPzUl20/zOBOTSginhwakmMaulYEiTFxfftwXqcm/HnBFl7fsxMwpHDSc1KoLydoyHEaeP1v4Hl+guaSSxd2U7ZqNTFFx5gcHzxvx00SeWtS+EVCMnneJwGfq4dk7nt6Ez0k0bMcY3yqAc9/xn68xIINB7jgrKaeqiEoP1G4Hbapk3/h8608v3Ar9RLjOHaqhHduGeRTnXre064rzLoJgdVd3p8/ELIjx5vLdjKoQ2rQ9eHU2BuPBLPtydF09BuUo6rf2HNaMTdElz5/j43twdOfbgo4IUWTP/70HH733g8kccpVfWIF65ZyhBZy2PVcXM+bkE+M+P4WC0yiV6m7kRXAU60A3oj9JpXD1MNEqE/FL4ekc+OQNJ/qr3BEInNHuFhKqe++arCuGOoHXE24ThQ+yzhOooToNSQxHCmra10hpJBvXFcTrv/JnuV5xnddPsmcInjVXHVY+fAlNKobT/oD9vFv59OXBb3xSK0q0f/zxv4BDXnh/PjU6KBvjKq4ey85y7aRvKL9mB+duz58ohqoc7MUtnqqwgz1OOmp+y6vCz9MRuOTpHyWw6rEAzSSwEa3fFPXE7Q3lrVjP75BfJ9pzFGSsZ8uqmq89s2PvPZNxdpLIlVOLCWWXOqTa6zeO473a0iiyK+KyX31cILr+zRg8eotXlcVx2nPARrEuNb7n1y9FZp4r5NCCkdN6KsJ9/KjJFNWBR0acwuK+LqCbWJutSrQd7W6cD02tofjQBFuNsZRZ7fgrxP7/k9fJcy9cwivff0jH612lcinT+zLHW+ttE0b7ERbUlZGu8Z12XWkoFIlw5rH0JhjVqnbVRof3PQUiQX7SYrf76lOSZHAvtk5pj778xqz06Sy36R7SuX7KA/kJ6nYNLMqGKGQRApJ5ICxevN4xe6/ZQKcF2TLMupx0qvd4YTPCcN/eRvJoXvMDhpynGQJ3VZw1NT1XDXYnxRSyDPJHHWvt04mBSQS7OR+0Z++pEGdEPVbIdSqQO8OMtcPTqtQiXDz1JF0eci3L/WwLk3ZvP8YL1/bz/F+GtaN90z+BXBJ9+as3p1nO8Fau8Z1ubp/W56dHziit6bp1aYhz0/o4wn0Y3q15I637NP6D0RzG9KxCc9d3buqsshbvxpY4YbfYGIooyl5PkG8hdf/FrjqxRPFt1ueyY/hEI3ZLQ3ZbNqwpKxXeRA3jdlPKgdNQ4qo3I+xNmqSkuAzoK02McRwlGSOmmTXPOwVuDqJpyRIg3X51YR320QrDlM/xtU24WlnsVFsYsmz8pRHCkn1Utl8NM5TnZRflExeTHmPpg/uHc2Kg4YJb4aOM7Uq0DcN0R2wa4t6bAoShBLjfBtCNj0+slJzmn9533DOeax8FOhfrunDb2avYv76wG6ggzum0iQldB3fxd2aV7oL6e3DOgbtWlmVmte3/wweHRs4NfIPj4zghc+3Oq4OyHzoYjKmLrRdN7iTs6mgEyimmeT6dCf0DuTN5QjNyAvoHXLKxHl6oKwyndhf1jigUXPx5J+xfOsRbp9lf7XTrWV9nhvbI2CyvGg2cUC7oCPXo1kxcRyiAYeM1ZumAlVNdTlV3kgtx/nN4KZ8uHS91f5wwucE0qz0CBmSS8OYE9QXmzmEpk+mH7A1KXQlX60K9KEEGxRhp7I3rmhQJ95Tgnnqyp4kxcfSor7vJfh//+88Nu47yrjerUmIiwk6YGfVw5eQkhRH5wdd0/9snDIy5AhOb1PG9fC56coNg9N8BphU1rjerfj36sAG1eUPXsx7K3bzzKebg96Yxe6SskHdeCYObOc40DcJMie32+B2ddi/e5tXqfsIPVKOk1Cw3xPEm0pgP+cTJpF9JpV9pjFZZWezn8Zce8kgGrZI49ucJD7MKmPOppOEqw+vk5TI6J4tiY0R2yk1jDFBBwNFqtHSiWfG92LhhgOc3bqBo0GHpyOSg+r+NwgFJFFAEntNEzAwKOks3iltZpt65tgMbrbm5oqhjPpeVwoZzWN4+KKWnDx6mOnzlgN/D/qqtSLQ//PG/gGDU67OaOszd/rV/dvy/EL7vvfeerUJ7M8KUD8pjqOFJUy+vDuT/7MhYP1//89Vz5f50CVsPXDME/D8b+6QnBjHTzPaBmzv7eWf96WRdTzXDWrPvvxC6th0vwJo2SDJc4MQt1+cm8a/vOZeqV/Jejt/L0zowwsT+gBwZd/WnukPmtZL5JbzO9CwTgJX9W3jOXkN79KUF6/pQ72k4K9v160skGH1fQPgwHqGxay2eqIcpgXlpXGm3cZbhfngdy7ILUxhv7hK3mvKOlh14eWl8QOmMceog3cQ976iG9IFhpwHc6zpGZxcZQWbNynUjKkv/7wvt/7L/koglLduHuhzz4NP7z6fkc/bDxRyS2+SzIxfuDpfOAn0913apdJVjCci3Gtq6xOjPIWf/xXB5hr64rcX0KFpCj88OoJzHvuMMmLIox55ph7dz27BzZf3gAZJxJSU8tf/tKHWB/phXQLPdk+P7+UT6AempwJbuWZAW+655KyA2wiue+xS5mTuZkKQW/l98bth/PWLLCYObM+/f9jrM5AG4GyvAQ+dm5cPZrljeCcOHy9i4cYDHC0s8ZnT3l+/9o24blB7n6Hbj18RfDg/uHq53DdnDTECL/28L9sPuW68Md9rcM/twzryYpABZpX11JU9fea5iYuNYeJA3/fOyS31WtVPZGKPJNrH55G5Zl15EPcZrZlL3b+42jn+aZ3Py4xQUqcJGwvqscO0oEvPUeynMdO+OWb1UHEN8unWrnnAZ+Vt7eQR9JzsO+laqCu6V67tSyebQNOyQfjG01AdwhrUSaBVgyT2Wift//7feVz2F9f0Gv3aNwraa2lwxyaM6dWSj9e4ZvXs2qI+O6aN4cI/LWZ7jv1NWHraDM4J5Y7hnWpMW5L/KNZIGtmjRcCgOLvvB8AfRnflyXmbPM+Hd2kaMDL9k9+cH3R0bkUMSG9sO6d8h6auwqTd1bJ322K4OcGgBk5q1rN1A5Y9cJHn+drJIxxtd27HVNY9dilPXdnL9l6yKYlx3DAkPeiPvElKIpPH9iAhLob3fn0u915ylqPXbVg3geeu7k1Kouuc6d/Jp3XDOvRt15BXru3HrF8N5Io+wW+Z676r1tf3l9/7tUerBrx/27lsenwUI89uye3DXKGzb7AAABidSURBVPOwXH5OK08a72PyHxr/8V3lPQ5uHJIW8JorHrK/HWNiXCzPXNWLP18dON8HwIVdbS41186B+Q/CezfC3y+F53vC1GY8ue1Kfr3pl8xMeI6p8f/gltiPGRiziQQpYYNJ463SC2HEEzD+H9wgUxlc+CJnnXqdhElZjCuayq+L74Uxf6Rk8N18VHYey8q6s9O0oJBEMto3CshGvNckPv7tM97z2XvLfOhiVj18CXGxMfx+ZBe6tyyfpGv+3UNZ6vWdfPiy7rb7mHGdbRdmnrqyJ+d2TKW1130/05uU30Dl+TCN2LcPc9228Y8/Lf8s3vil70nWuzeU9/dh9SOX8N0fyvN+Orek+95rP6E0SUmgT7uGlX6dzCDfSacG2lSfdWqWwivX+Xa86NAkOejV6C1DfW+VecOQwNsU2sWSx2zaqsLpZ/Mdrggn93mukSX6Fl6lp1DVAu6z7nu3ngvgCbanKy42hrsu6szoni25+LkvHW3jfrP962Hd8204MX1iX6ZP9F3WvVXgrIAAI7o35/cBeQhM16NVAzo0SWb7oRM+JYPHrzibhnXig96rEuBn/e2roLZMHWXfzXL9h5C1EOq3gvqtod25rsf1WlnLWtH/Lxs5RH0MMUzo35YVO3Ndc8kMdgXPg59/xd6T5fXsfxjdldRkVx7tSnsN6wY2eP80o61nNGR8rJD1xCiKSsvo/sh8rg5SrebdPnD7sE7cPqyTZ7bNLn7TEdx0XjofrdrjmUtofL82/Or8dNKa2N/9arR1Bff2zYO4ZuYyXr0ugzpeQaJtmIn2erRqwJrJIzxD7AFaNajjk2bBPUM5Vljic98FcL0/pWWG8zs3CXmf369+P5yLn/syYGbVrCdGkX+yGBEhMcQ9aO++uDNX9mlDu9TyY3n8vxv4+9c/8vBl3Xn8v4HVof42ThkJlH8W7u+tv/su7cLb3+/y3DgbYEinVM+ssk9d2ZML/+T7u5131/mAbztUvTBVnt5jRvqnBQZju/fD3WFkTM+Wtncic78XLeoneaZ2tru3b7hbqfrLemIU8U8HX18jAz24enccD3FHI4Cbz+/AqLNbBv2hXNk3eOnZiWANj3ZaNkhiT97JCg/oCmV8v+B3em9QJ54rerfiGqsqyrukdtuwjry8eBs3DE4DXAF72iebuKhrc55fuJWXft7XE3wqI+hNp6/6m2v2wxAljBzKv/yFxaUsuPcCn/WPX9GDq15eygXWFMreJSv3ifzuizvzm4s6IyL89t3AaWwHd0z1BHoRIS5WiIuNYdPjIx1d5jrx9i2DPHcZ8i5pA3w76UIGe92aMCne9ZpxsTG8d+vgsPued9f57M07ybFT5R0M6vsVeGJihFeu7eeZt6V+nXjPpb6/2BjhzZsG2q6badXlt21cl9uGdfRp5+qf1og4rxtXh5pG4+6LA6+AHxrTjQdGdaXUmIBAnxAX4zM98K+HdvBpp1o7eQTxsTHc9fYqPttwgHsuPstzg5n2qXV9gvzaySNIjItlzopsdh4+QYemKSy5b7jPrKLu72zP1g08gf5Kv6vrr+8f7pmyAGB0zxaeQF83IY4tU0d57vrUsWmybaDv3rI+b940gIHpqbaB/qbz0rnpvHSGeo0xSYiNYe3kEZ57Mr+budvzu3Z79bp+/DrEzVziwnyva2ygX/L74WF7KYhI0CC/Y9qYiN6UOpxXruvHok0HadWwTvjEDoS7xBYRnrcaTv3dP7Irv7+0i+cq49dDO3DjkDQS42JP+27yIcWHP/YfHhnB28t3Me2TTbYNef3aN+aNXw6w7b2SnBjHpsdHkhgX4zm2CQPa8v5K1/xBj1zWnQUbDnBJ9+a2r13R3lYTB7ZjZZC6c/dJ54rerQLW+X8H/KuP3AZ3TGXk2a4po3u3bei5PVy71LpBr+S8ubf1zo8Tc+8cQv7JYnq1bkiDuuUnkFsvKA/0j17enSv7+hY04mKEO4d3YuTZLTztC6G4T7KxXr/DJimJDD2rCc/9rDfPLdjiaVt6wJrkzs19Je8+rjaN6rDtydEs2HCAS3s0Z3LKek//fXda7zYk7ysLb1f1bcPUjzcy9Yqz+bmV/i/X9KFt47q0aeS7TVO/KuCEuBhPY/h1g9p7vk8JcTG8+csBPP3pJlo1rOO5snPPf+PmXZ3lXRaKi42hnlegvtGmmsg9tXhl1dhAH+zHURFO6q7C+fcdQwJmf7TTJCUxbG+bM8n72F2X3af/fkZCg7rxdLEas4Pd8CHUDVH8g3X/tMYsf/BizyXzL89z/Uj++NNzwk4qFs6TP+kZ8ibzayeP8KmC8fb1/cN5Y+lOOgUpZQO8dXP5HOgdm6Z4An1Fgva5HVJZuv1whU5ivdrY15977+P6c9OI8bs6FRF+d2kXIHDwYCgiwt0Xd+aCs5rSp115FUhDB73Fzj+rCR+s2kPn5inExojn5PbAqG789r0fePEa+8JOMI2SEwIKO97tXd4a1Inn47vO87khfdcW9fl20oW0bJDk6X1VVFLGwA6pfHD7EJ/t1z52KVB+mz/vKz/3Ozt9Yl/HeX/yJz35w4cVu4m4W40N9DXFOW0bck7byjcsqUDuS95I3QbRbiDd+H5tQlZ9OeUf7LyFaj9q06iuZypmJ7q3qs/7K0NUiwXx+i8HVMntJEMdN8C/bhroqFTvZle107Bu+ED/kz5tOL9z04AxFlf2bU3HZin0jvBv011l5NajVWAPJvcVW1ysMLxLU64d1D7kPl+Y0JtFmw7S0euk/4tz05jy3w0M6eR8RsoLrPsiPHq5fWeAUDTQqzMuwRPoa87UydXthsFpdGiazLAwt3f0lxAXU+GTQyirHr7Ec6vBUM5u3cDTv/tX5wVWNTjR3Bps6G5LCsZuIJ2IhA3yqckJHD5RxNmtw1eDuYU6edtx0sV4XO/WjOvt2x7wy/PSue7c9hXqTtq6oav6qjLNgI5eRURGishmEckSkUk26+8VkQ0iskZEPheR9l7rSkVktfU3t+JZVNGmW8v6NK+fyH0julR3VmqM2BhheJdmEaluPB2NkhPCjlB2a1Annh3TxvBQkO6m4QzumMqfrz6H+0d2rdT24dx1UWeAKtv/6arMmIHYGKnUdyTsfPQiEgtsAS4BsoHlwDXGmA1eaYYD3xljCkTkNmCYMeZqa91xY4zj7isZGRkmMzOzwgeilFLejDGs23OUnkFGw0cbEQk6H72TU8oAIMsYs90YUwTMBsZ5JzDGLDLGuGfcWQacfuWoUkqdBhH5nwny4TgJ9K2B3V7Ps61lwdwEeI8hTxKRTBFZJiJX2G0gIrdYaTJzcgLvz6mUUqrynDTG2lUI2db3iMi1QAbgPQqmnTFmr4h0AL4QkbXGGJ/5dY0xM4AZ4Kq6cZRzpZRSjjgp0WcD3h3E2wABc9mKyMXAg8BYY4znThzGmL3W/+3AYqBiHV+VUkqdFieBfjnQWUTSRSQBmAD49J4RkT7Aq7iC/EGv5Y1EJNF63AQYAoSf9EIppVTEhK26McaUiMidwHwgFnjNGLNeRKYAmcaYucCzQArwntX1Z5cxZizQDXhVRMpwnVSmeffWUUopVfXCdq8807R7pVJKVdzpdq9USilVi2mgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKOQr0IjJSRDaLSJaITLJZf6+IbBCRNSLyuYi091p3vYhstf6uj2TmlVJKhRc20ItILDAdGAV0B64Rke5+yVYBGcaYXsAc4Blr28bAo8BAYADwqIg0ilz2lVJKheOkRD8AyDLGbDfGFAGzgXHeCYwxi4wxBdbTZUAb6/GlwAJjzBFjTC6wABgZmawrpZRywkmgbw3s9nqebS0L5ibgk4psKyK3iEimiGTm5OQ4yJJSSimnnAR6sVlmbBOKXAtkAM9WZFtjzAxjTIYxJqNp06YOsqSUUsopJ4E+G2jr9bwNsNc/kYhcDDwIjDXGnKrItkoppaqOk0C/HOgsIukikgBMAOZ6JxCRPsCruIL8Qa9V84ERItLIaoQdYS1TSil1hsSFS2CMKRGRO3EF6FjgNWPMehGZAmQaY+biqqpJAd4TEYBdxpixxpgjIvI4rpMFwBRjzJEqORKllFK2xBjb6vZqk5GRYTIzM6s7G0opVauIyApjTIbdOh0Zq5RSUU4DvVJKRTkN9EopFeU00CulVJTTQK+UUlFOA71SSkU5DfRKKRXlNNArpVSU00CvlFJRTgO9UkpFOQ30SikV5TTQK6VUlNNAr5RSUU4DvVJKRTkN9EopFeU00CulVJTTQK+UUlFOA71SSkU5DfRKKRXlNNArpVSU00CvlFJRzlGgF5GRIrJZRLJEZJLN+qEislJESkRkvN+6UhFZbf3NjVTGlVJKORMXLoGIxALTgUuAbGC5iMw1xmzwSrYLuAH4nc0uThpjekcgr0oppSohbKAHBgBZxpjtACIyGxgHeAK9MWaHta6sCvKolFLqNDipumkN7PZ6nm0tcypJRDJFZJmIXGGXQERusdJk5uTkVGDXSimlwnES6MVmmanAa7QzxmQAE4HnRaRjwM6MmWGMyTDGZDRt2rQCu1ZKKRWOk0CfDbT1et4G2Ov0BYwxe63/24HFQJ8K5E8ppdRpchLolwOdRSRdRBKACYCj3jMi0khEEq3HTYAheNXtK6WUqnphA70xpgS4E5gPbATeNcasF5EpIjIWQET6i0g28FPgVRFZb23eDcgUkR+ARcA0v946SimlqpgYU5Hq9qqXkZFhMjMzqzsbSilVq4jICqs9NICOjFVKqSingV4ppaKcBnqllIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKOco0IvISBHZLCJZIjLJZv1QEVkpIiUiMt5v3fUistX6uz5SGVdKKeVM2EAvIrHAdGAU0B24RkS6+yXbBdwAvOW3bWPgUWAgMAB4VEQanX62lVJKOeWkRD8AyDLGbDfGFAGzgXHeCYwxO4wxa4Ayv20vBRYYY44YY3KBBcDICORbKaWUQ04CfWtgt9fzbGuZE462FZFbRCRTRDJzcnIc7loppZQTTgK92CwzDvfvaFtjzAxjTIYxJqNp06YOd62UUsoJJ4E+G2jr9bwNsNfh/k9nW6WUUhHgJNAvBzqLSLqIJAATgLkO9z8fGCEijaxG2BHWMqWUUmdI2EBvjCkB7sQVoDcC7xpj1ovIFBEZCyAi/UUkG/gp8KqIrLe2PQI8jutksRyYYi1TSil1hogxTqvbz4yMjAyTmZlZ3dlQSqlaRURWGGMy7NbpyFillIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinKOAr2IjBSRzSKSJSKTbNYnisg71vrvRCTNWp4mIidFZLX190pks6+UUiqcuHAJRCQWmA5cAmQDy0VkrjFmg1eym4BcY0wnEZkAPA1cba3bZozpHeF8K6WUcshJiX4AkGWM2W6MKQJmA+P80owDXrcezwEuEhGJXDaVUkpVlpNA3xrY7fU821pmm8YYUwLkA6nWunQRWSUiX4rI+XYvICK3iEimiGTm5ORU6ACUUkqF5iTQ25XMjcM0+4B2xpg+wL3AWyJSPyChMTOMMRnGmIymTZs6yJJSSimnnAT6bKCt1/M2wN5gaUQkDmgAHDHGnDLGHAYwxqwAtgFnnW6mlVJKOeck0C8HOotIuogkABOAuX5p5gLXW4/HA18YY4yINLUacxGRDkBnYHtksq6UUsqJsL1ujDElInInMB+IBV4zxqwXkSlApjFmLvB34E0RyQKO4DoZAAwFpohICVAK3GqMOVIVB6KUUsqeGONf3V69MjIyTGZmZnVnQymlahURWWGMybBbpyNjlVIqymmgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKaaBXSqkop4FeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinIa6JVSKsppoFdKqSingV4ppaKcBnqllIpyGuiVUirKOQr0IjJSRDaLSJaITLJZnygi71jrvxORNK91D1jLN4vIpZHLulJKKSfCBnoRiQWmA6OA7sA1ItLdL9lNQK4xphPwZ+Bpa9vuwASgBzASeMnan1JKqTPESYl+AJBljNlujCkCZgPj/NKMA163Hs8BLhIRsZbPNsacMsb8CGRZ+1NKKXWGxDlI0xrY7fU8GxgYLI0xpkRE8oFUa/kyv21b+7+AiNwC3GI9PSUi6xzlvuZqAhyq7kychtqef6j9x6D5r3617RjaB1vhJNCLzTLjMI2TbTHGzABmAIhIpjEmw0G+aqzafgy1Pf9Q+49B81/9ouEY3JxU3WQDbb2etwH2BksjInFAA+CIw22VUkpVISeBfjnQWUTSRSQBV+PqXL80c4HrrcfjgS+MMcZaPsHqlZMOdAa+j0zWlVJKORG26saqc78TmA/EAq8ZY9aLyBQg0xgzF/g78KaIZOEqyU+wtl0vIu8CG4AS4A5jTGmYl5xR+cOpMWr7MdT2/EPtPwbNf/WLhmMAQFwFb6WUUtFKR8YqpVSU00CvlFJRrkYF+nBTLVQnEdkhImtFZLWIZFrLGovIAhHZav1vZC0XEXnROo41ItLXaz/XW+m3isj1wV4vQnl+TUQOeo9LiGSeRaSf9Z5kWdvadaeNdP4ni8ge63NYLSKjvdbZTrcR7HtldTD4zjqud6zOBpHMf1sRWSQiG0VkvYj8xlpeKz6DEPmvTZ9Bkoh8LyI/WMfwWKjXlUpM5xLs2GoUY0yN+MPV0LsN6AAkAD8A3as7X1752wE08Vv2DDDJejwJeNp6PBr4BNc4gkHAd9byxsB2638j63GjKszzUKAvsK4q8oyrB9W51jafAKPOQP4nA7+zSdvd+s4kAunWdyk21PcKeBeYYD1+BbgtwvlvCfS1HtcDtlj5rBWfQYj816bPQIAU63E88J313tq+LnA78Ir1eALwTmWPrSb91aQSvZOpFmoa76kfXgeu8Fr+hnFZBjQUkZbApcACY8wRY0wusADXHEBVwhizBFcvqIjn2VpX3xiz1Lh+CW947asq8x9MsOk2bL9XVsn3QlxTdoDvexGp/O8zxqy0Hh8DNuIaGV4rPoMQ+Q+mJn4Gxhhz3Hoab/2ZEK9b0elcakXcqkmB3m6qhVBfqjPNAJ+JyApxTdkA0NwYsw9cPwqgmbU82LHUhGOMVJ5bW4/9l58Jd1pVG6+5qz2oeP5TgTxjTInf8iphVQH0wVWirHWfgV/+oRZ9BiISKyKrgYO4TpLbQryuz3QugPd0LjX1Nx1WTQr0jqZLqEZDjDF9cc3ieYeIDA2R9rSmhKgmFc1zdR3Ly0BHoDewD/iTtbzG5l9EUoD3gbuNMUdDJQ2Sp2o9Bpv816rPwBhTaozpjWtk/gCgW4jXrZHHcLpqUqCv0dMlGGP2Wv8PAh/i+sIcsC6fsf4ftJIHO5aacIyRynO29dh/eZUyxhywfrhlwEzKZ0OtaP4P4aoaifNbHlEiEo8rSM4yxnxgLa41n4Fd/mvbZ+BmjMkDFuOqow/2uhWdzqUm/KbDq+5GAvcfrlG623E1dLgbNXpUd76svCUD9bwef4urbv1ZfBvVnrEej8G3Ue17a3lj4EdcDWqNrMeNqzjvafg2ZkYsz7imxxhEeUPg6DOQ/5Zej+/BVW8KrnseeDeWbcfVUBb0ewW8h2+D3O0Rzrvgqjd/3m95rfgMQuS/Nn0GTYGG1uM6wFfAZcFeF7gD38bYdyt7bDXpr9oz4PehjMbVsr8NeLC68+OVrw7WB/gDsN6dN1x1d58DW63/7h+f4LpZyzZgLZDhta9f4mrIyQJurOJ8v43r0roYV8njpkjmGcgA1lnb/BVrpHUV5/9NK39rcM2l5B10HrTyshmv3ifBvlfW5/q9dVzvAYkRzv95uC7j1wCrrb/RteUzCJH/2vQZ9AJWWXldBzwS6nWBJOt5lrW+Q2WPrSb96RQISikV5WpSHb1SSqkqoIFeKaWinAZ6pZSKchrolVIqymmgV0qpKKeBXimlopwGeqWUinL/D8izKB4hbS8VAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better model found at epoch 0 with accuracy value: 0.962848961353302.\n",
      "Better model found at epoch 1 with accuracy value: 0.9641061425209045.\n",
      "Better model found at epoch 2 with accuracy value: 0.9642795324325562.\n"
     ]
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas5_sp15_labelsmoothing');\n",
    "learn_c.fit_one_cycle(3, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7), \n",
    "                      callbacks=[ShowGraph(learn_c),\n",
    "                                 SaveModelCallback(learn_c.to_fp32(),every='improvement',mode='max',monitor='accuracy',\n",
    "                                                   name='bestmodel_clas_sp15_labelsmoothing')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas6_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas6_sp15_labelsmoothing');\n",
    "learn_c.save(f'{lang}clas_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_sp15_labelsmoothing');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_sp15_labelsmoothing')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13.3 s, sys: 512 ms, total: 13.8 s\n",
      "Wall time: 13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15', bs=bs, num_workers=1)\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, pretrained=False, drop_mult=0.5, \n",
    "                                  metrics=[accuracy,f1])\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15');\n",
    "\n",
    "learn_c.load(f'{lang}clas_sp15');\n",
    "\n",
    "# put weight on cpu\n",
    "loss_weights = torch.FloatTensor(trn_weights).cpu()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEGCAYAAAAJw7AFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAY4ElEQVR4nO3debxVZb3H8c8PRQUBBcccSQLJgRBy4l4NNU3NITPnMm2wLO1mapp5HetqZoNl5nAdyxzKLEWvaWYqKKYoKiYqTpVizqLgCL/7x1oHN0c4nHPgnIPP+bxfr/Nir2c9e+3fhrP57vXsZz07MhNJkkrRo6sLkCRpYTLYJElFMdgkSUUx2CRJRTHYJElFMdgkSUUx2KSFKCJ6RcTVEfFKRPx2AY6zT0RcvzBr6yoRsVlEPNTVdaj7CK9jU3cUEXsD3wKGAq8CE4HvZ+bYBTzu54CDgVGZ+c4CF7qIi4gEBmfmlK6uRWriGZu6nYj4FvBT4H+AlYA1gDOAnRfC4dcEHu4OodYaEbF4V9eg7sdgU7cSEcsAJwBfz8zfZ+b0zHw7M6/OzMPrPktGxE8j4un656cRsWS9b3RE/CsiDo2IZyNiakTsX+87HjgG2CMiXouIL0bEcRHx64bHHxgR2fQffkTsFxGPRcSrEfF4ROzT0D624X6jIuLOeojzzogY1bDvrxFxYkSMq49zfUQsP4/n31T/txvq/1REbB8RD0fEixFxVEP/jSLi9oh4ue57ekQsUe+7pe52b/1892g4/hER8QxwflNbfZ9B9WOMqLdXiYjnI2L0Av3DSg0MNnU3mwJLAVe20Oe7wCbAcOAjwEbA0Q37VwaWAVYFvgj8IiL6Z+axVGeBl2Vmn8w8t6VCImJp4GfAdpnZFxhFNSTavN8A4Jq673LAj4FrImK5hm57A/sDKwJLAIe18NArU/0drEoVxOcAnwVGApsBx0TEWnXfmcAhwPJUf3dbAV8DyMzN6z4fqZ/vZQ3HH0B19npA4wNn5qPAEcDFEdEbOB+4IDP/2kK9UpsYbOpulgOen89Q4T7ACZn5bGY+BxwPfK5h/9v1/rcz81rgNWDtdtYzC1gvInpl5tTMfGAufT4JPJKZv8rMdzLzEmAysGNDn/Mz8+HMfB24nCqU5+Vtqs8T3wYupQqt0zLz1frxHwCGAWTmhMwcXz/uE8BZwMda8ZyOzcw363rmkJnnAI8AdwAfoHojIS00Bpu6mxeA5efz2c8qwJMN20/WbbOP0SwYZwB92lpIZk4H9gC+CkyNiGsiYmgr6mmqadWG7WfaUM8LmTmzvt0UPP9u2P960/0jYkhEjImIZyJiGtUZ6VyHORs8l5lvzKfPOcB6wM8z88359JXaxGBTd3M78AbwqRb6PE01jNZkjbqtPaYDvRu2V27cmZl/ysytqc5cJlP9hz+/eppqeqqdNbXFL6nqGpyZ/YCjgJjPfVqcah0Rfagm75wLHFcPtUoLjcGmbiUzX6H6XOkX9aSJ3hHRMyK2i4hT6m6XAEdHxAr1JIxjgF/P65jzMRHYPCLWqCeufKdpR0SsFBE71Z+1vUk1pDlzLse4FhgSEXtHxOIRsQewDjCmnTW1RV9gGvBafTZ5YLP9/wbWes+9WnYaMCEzv0T12eGZC1yl1MBgU7eTmT+muobtaOA54J/AQcAf6i7fA+4C7gPuB+6u29rzWDcAl9XHmsCcYdQDOJTqjOxFqs+uvjaXY7wA7FD3fQH4NrBDZj7fnpra6DCqiSmvUp1NXtZs/3HAhfWsyd3nd7CI2BnYlmr4Fap/hxFNs0GlhcELtCVJRfGMTbNFxOoRcVNEPBgRD0TEfzXbf1h9Ddby9XZExM8iYkpE3Nd0bVJD/34R8VREnN6Zz0NaGCLivPpav0nN2g+OiIfq18gpddvWETEhIu6v/9yya6oWgKsCqNE7wKGZeXdE9AUmRMQNmfn3iFgd2Br4R0P/7YDB9c/GVBMNNm7YfyJwc+eULi10FwCnAxc1NUTEFlQr1AzLzDcjYsV61/PAjpn5dESsB/yJOWetqhN5xqbZ6uuo7q5vvwo8yLsvzp9QfbbTOHa9M3BRVsYDy0bEBwAiYiTVclVFLOSr7iczb6H67LPRgcDJTZcoZOaz9Z/3ZGbTzNkHgKWiXq1Gna/Dgi2qpYMejIhz6lP266Na+XxQRFxXn67f2nTdTt0+Pqrlgk6IiNc6qjbNX0QMBDYA7oiInYCnMvPeZt1WpZp40eRfwKoR0QP4EXB4J5QqdaYhwGYRcUdE3BwRG86lz67APV6f13U6+oxtMPCLzFwXeJnqH/xs4ODMHEk14+qMuu9pVKsfbEj7rxnSQlBfZ3QF8E2q4cnvUk15f0/XubQl1cy+azPzn3PZL72fLQ70p1py7XDg8oiY/TqIiHWBHwBf6ZryBB04K7J+x39DZg6ut48AelL9J9n43UxLZuaHI+IFYKXMfCci+gFPZ+Z7Vk+IiAOo15/r3XvpkYMGt3clI81NZvLEY1Po068fK6ywEm+8/jqPPfYIPaJ6D/T222/Rs2dPBg0eyrPPTGXpPn1Ytn91fe1Dkx9grUFDmPr0v5gx/TUgmDVrJpnJcsutwMqr+JHDwtRzsfldJ60F9eabb/Loo1NYZ511AZgy5RFWWmll+vbtC8CkSfez9tpD6dmzJ2+99RaPPPIwa645kD592rwQjdrh7rsnPJ+ZKzRv7+jJI42n4jOpPnN5OTNbWseuRZl5NtVZH8OGj8yr/jxuwSrUbJnJoQd9iQ03GcUx3z91rn3+c8TaXHXDOAYstzx/uf7/uOjcMzn/0j8wccLfOO6oQ/nj9XN+ndnvLvkV902cwAk/+GlnPIVuZeVll+rqEor35BNP8OlP7cC4O+4C4JyzzmTq1Kc55rgTeOThh9nuE1txx4SJvPLKK2yz5cc4/6KL2eXTu3Zx1d1Hr57RfKk5oPMnj0wDHo+I3WD2dPGP1PvGUw1VAuzZyXUJuOuO27jy8t9w29ib2X70xmw/emNuuuG6efbfYuttWX3NDzJ6o3X5zre+zomnnNaJ1Uoda9/P7sXozTbl4YceYtDA1bjgvHP5/P5f4PHHHmPk8PXYd589+d/zLiQiOPOM03n00Smc/P0T2XjkcDYeOZxnn322q59Ct9XRQ5FjMnO9evswqoVVL6SaFv4BqqHJSzPzhIgYTLVsUVAts3NAZrY4duUZm7ozz9jU3fXqGRMy86PN2ztsKLL+iov1GrYbx7a2nctdngI2ycyMiD2pljSSJKlNFqULtEcCp9czjF4GvtDF9UiS3ocWmWDLzFupvq1YkqR2c+URSVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRFp/Xjoi4Gsh57c/MnTqkIkmSFsA8gw04tdOqkCRpIZlnsGXmzZ1ZiCRJC0NLZ2wARMRg4CRgHWCppvbMXKsD65IkqV1aM3nkfOCXwDvAFsBFwK86sihJktqrNcHWKzNvBCIzn8zM44AtO7YsSZLaZ75DkcAbEdEDeCQiDgKeAlbs2LIkSWqf1pyxfRPoDXwDGAl8Dvh8RxYlSVJ7zfeMLTPvrG++BuzfseVIkrRgWjMr8ibmcqF2Zvo5myRpkdOaz9gOa7i9FLAr1QxJSZIWOa0ZipzQrGlcRHjxtiRpkdSaocgBDZs9qCaQrNxhFbVBjx7QZ6nWnHRK5em/4UFdXYK0SGpNKkyg+owtqIYgHwe+2JFFSZLUXq0Jtg9n5huNDRGxZAfVI0nSAmnNdWy3zaXt9oVdiCRJC0NL38e2MrAq0CsiNqAaigToR3XBtiRJi5yWhiI/AewHrAb8iHeDbRpwVMeWJUlS+7T0fWwXAhdGxK6ZeUUn1iRJUru15jO2kRGxbNNGRPSPiO91YE2SJLVba4Jtu8x8uWkjM18Ctu+4kiRJar/WBNtijdP7I6IX4HR/SdIiqTXXsf0auDEizq+39wcu7LiSJElqv9asFXlKRNwHfJxqZuR1wJodXZgkSe3RmqFIgGeAWVQr+28FPNhhFUmStABaukB7CLAnsBfwAnAZEJm5RSfVJklSm7U0FDkZuBXYMTOnAETEIZ1SlSRJ7dTSUOSuVEOQN0XEORGxFe+uPiJJ0iJpnsGWmVdm5h7AUOCvwCHAShHxy4jYppPqkySpTeY7eSQzp2fmxZm5A9W6kROBIzu8MkmS2qG1syIByMwXM/OszNyyowqSJGlBtCnYJEla1BlskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhskqSiGGySpKIYbJKkohhsmsM3DvwSH/7gKmy20fDZbSedeCwf22QDRo8ayW47b8czU58G4P/GXDW7/eObb8z428YCMPaWvzJ61MjZP6st34drr/5jlzwfaX7OPHYfnrzxJO767VGz29Yfsip/vfBQ7rz8KH7306/Qd+mlABiwzNJcd/Y3eG7cj/jJEbvNcZyeiy/G6UfvxX1/OIaJvz+aT21VvYY+u+PG/OMvJzH+0iMZf+mR7LfLpp335LqpyMyurqHdho8YmX++5Y6uLqMot429laX7LM1BB3yBW/82EYBXp02jb79+AJz9y5/z8OQHOfW0M3jttddYeumliQgemHQfX9p3b26/e9Icx3vpxRfZaPhQ7p38BL179+7051Oy1Tf7ZleXUIT/GDGI6TPe5H9P3JeP7vY/AIz99eEc+ZMrGTthCvvuvAkDV12OE864ht5LLcHwoauxzodWYd1BH+CQH/x29nGO/ur2LNajB8efMYaIYMAyvXnh5el8dseNGbnOGnP01cLxxsRfTMjMjzZv94xNcxj1n5vRv/+AOdqaQg1gxvQZRAQAffr0mX17xvTps283uvoPV7DV1p8w1LTIGnf3o7z4yow52gavuSJjJ0wB4C/jJ88++5rxxlvcNvEx3njz7fcc5/M7b8oPz7segMzkhZend3DlmpcODbaIGBgRkyPiwoi4LyJ+FxG9I2KriLgnIu6PiPMiYsm6/8kR8fe676kdWZva5vvH/zcfGfpBrrj8Eo747nGz26+56g9sOmI99t5tZ0474+z33O/KKy7n05/ZsxMrlRbc3x+dyg6j1wfg01uPYLWV+rfYf5k+vQA49us7cNtvjuDiU77AigP6zt6/81bD+dtl3+E3P/wiq620bMcVLqBzztjWBs7OzGHANOBbwAXAHpm5PrA4cGBEDAB2Adat+36vE2pTK3332BO5d/Lj7Lr7Xpx79hmz2z+506e4/e5JXPibKzj5e8fNcZ9nnpnKgw9MYouPb9O5xUoL6CvHXcxXdt+ccRd/mz69l+Stt2e22H/xxXuw2sr9uX3iY4za+wfccd8TnHTILgBce8skhn7yWDba4yT+csdDnHPC5zrjKXRrHfoZW0QMBG7JzDXq7S2B/wYWy8zN67atgK8DuwMTgLuAa4AxmfnWXI55AHBAvbk28FCHPYHuawlgMPDAfPYtDzzfsG994EHgnXp7RaAX8GSHVSotBEOGDFlizJgxg4cMGfKe3/n1119/yYsvvnitYcOGPdjUdvDBBy83bNiw5b785S8/DBARTJ8+fYM+ffrcM2vWLAYNGtTzuuuuGzJ48OA5jrfYYovx0ksvDe/Xr9/Ejn9W3cKamblC88bFO+GBW5WcmflORGwEbAXsCRwEbDmXfmcD7x3z0kJTvyEZ0/ShbEQMzsxH6tsHAx/LzM9ExP3AhpmZETECuBrYIOt3SxExHvhOZt7UFc9DaoOBNPzOU70pe5ZqVOsC4CeZeV5D//0uuuiiU5pNXLh05syZZwN/AfYDPpmZuwEfAKbWfXYBjsjMTTrqiahzztgeB0Zl5u0RcQ7wBPAVYMvMnBIRFwD3AOcCvTPz2XpYckpmDpjrgdVhIuISYDTV2di/gWOB7anOjmdRnX19NTOfioh/Aa8AbwOvA4dn5tj6OAOBccDqmTmrc5+F1CZz+53vQzWSBPB74Du8+yb9CaDfjBkzlundu/dUYBvg78CawK+AZYHngP2BfwAnATtRjWS8CBwITO7g59StdUawXQvcAowCHgE+B2wKnEp1xngn1T/0AOCPwFJAAKdm5oUdVpwWWETcNbeptlJ34O//oqszhiJnZeZXm7XdCGzQrG0qsFEn1KOFxyFhdWf+/i+iOuOMbUxmrtdhDyJJUoP39cojkiQ158ojkqSiGGySpKIYbJKkohhsapOIeDUipjX7+WdEXBkRa3V1fVJHiohTIqJfRPSMiBsj4vmI+GxX16U5GWxqqx8DhwOrAqsBhwHnAJcC57VwP6kE22TmNGAH4F/AEKrXgxYhBpvaatvMPCszX83MafUSZ9tn5mVAy0ugS+9/Pes/twcuycwXu7IYzZ3BpraaFRG7R0SP+mf3hn1eO6LSXR0Rk4GPAjdGxArAG11ck5rxOja1Sf052mlUy6IlMB44BHgKGNm0VqRUqojoD0zLzJkR0Rvol5nPdHVdepfBJkmtFBE9qda23bxuuhk4MzPf+5Xa6jIORapNImJIPRtsUr09LCKO7uq6pE7yS2AkcEb9M6Ju0yLEMza1SUTcTDUL7KzM3KBum+R6oOoOIuLezPzI/NrUtTxjU1v1zsy/NWt7Z649pfLMjIhBTRv1Z84zu7AezUVnfG2NyvJ8/cJu+pbsz/DutwNLpTscuCkiHqu3B1J9oagWIQ5Fqk3qd6hnU31x7EtU35C+T2Y+2aWFSZ0gIpYCDgW2qptuAH6SmU75X4QYbGqTiFgS+AzVO9UBwDQgM/OErqxL6gwRcTnV7/zFddNeQP/M3K3rqlJzDkWqrf4IvAzcDTzdxbVInW3tZhNFboqIe7usGs2Vwaa2Wi0zt+3qIqQuck9EbJKZ4wEiYmNgXBfXpGYcilSbRMTZwM8z8/6urkXqbBHxILA28I+6aQ3gQWAW1ZD8sK6qTe8y2NQmEfF34ENUk0beBAJf0OomImLNlvY7iWrRYLCpTeb1wvYFLWlRYbBJkoriyiOSpKIYbJKkohhsUgeLiJkRMTEiJkXEb+vv8GrvsUZHxJj69k4RcWQLfZeNiK+14zGOi4jD2luj1NUMNqnjvZ6Zw+tvQHgL+Grjzqi0+bWYmVdl5sktdFkWaHOwSe93BpvUuW4FPhQRAyPiwYg4g2oVl9UjYpuIuD0i7q7P7PoARMS2ETE5IsYCn246UETsFxGn17dXiogrI+Le+mcUcDIwqD5b/GHd7/CIuDMi7ouI4xuO9d2IeCgi/kx1nZb0vmWwSZ0kIhYHtgOaLm5fG7io/l676cDRwMczcwRwF/CtetHdc4Adgc2Aledx+J8BN9fLPY0AHgCOBB6tzxYPj4htgMHARsBwYGREbB4RI4E9gQ2ognPDhfzUpU7lklpSx+sVERPr27cC5wKrAE82Lc0EbAKsA4yLCIAlgNuBocDjmfkIQET8GjhgLo+xJbAvQGbOBF6JiP7N+mxT/9xTb/ehCrq+wJWZOaN+jKsW6NlKXcxgkzre65k5vLGhDq/pjU3ADZm5V7N+w6m/+24hCOCkzDyr2WN8cyE+htTlHIqUFg3jgf+IiA8BRETviBgCTAY+2PCtzXvN4/43AgfW910sIvoBr1KdjTX5E/CFhs/uVo2IFYFbgF0ioldE9KUa9pTetww2aRGQmc8B+wGXRMR9VEE3tP4CywOAa+rJI/Nauuy/gC0i4n5gArBuZr5ANbQ5KSJ+mJnXA78Bbq/7/Q7om5l3A5cBE4ErqIZLpfctl9SSJBXFMzZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUf4fAebaq6agzW4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2404   162]\n",
      " [ 1337 19165]]\n",
      "accuracy global: 0.9350182070400554\n",
      "accuracy on negative reviews: 93.68667186282151\n",
      "accuracy on positive reviews: 93.47868500634084\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁voila ▁une ▁intégrale ▁bienvenue ▁que ▁xxup ▁decca ▁sort ▁judicieusement ▁de ▁ses ▁carton s , ▁ce ▁ne ▁sont ▁que ▁des ▁enregistrements ▁de ▁qualité ▁de ▁l ' immense ▁pianiste ▁( dont ▁de ▁nombreux ▁introuvable s ). ▁xxmaj ▁ses ▁concertos ▁de ▁xxmaj ▁beethoven ▁avec ▁xxmaj ▁ kn apper ts bus ch ▁sont ▁fabuleux ▁tout ▁comme ▁les ▁deux ▁concertos ▁de ▁xxmaj ▁brahms , ▁pour ▁moi ▁la ▁meilleure ▁version ▁jamais ▁enregistrée . ▁xxmaj</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁h tt p : ▁/ ▁/ ▁w ww . amazon . co . uk ▁/ ▁sony - ber n stein - edition ▁/ ▁for um ▁/ ▁ fx 1 n p 1 l t py n 3 x 6 c ▁/ ▁ t x 2 ry vi 6 c j hy mes ▁/ ▁1 ? _ en co ding = ut f 8 &amp; asin = b 00 ll</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁même ▁si ▁je ▁vais ▁me ▁faire ▁des ▁ennemis , ▁je ▁persiste ▁et ▁signe , ▁xxmaj ▁pierce ▁xxmaj ▁brosnan ▁est ▁crédible ▁dans ▁le ▁rôle , s e an ▁xxmaj ▁connery ▁l ' acteur ▁fondateur ▁du ▁rôle ▁titre , ▁donc ▁ intouchable , ▁xxmaj ▁george ▁xxmaj ▁la z en by ▁et ▁xxmaj ▁thi mo thy ▁xxmaj ▁d al ton , ▁crédibles ▁malgré ▁un ▁passage ▁très ▁court ▁( ▁1 ▁film ▁pour</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁wa ou h , ▁au ▁boulot . ▁xxmaj ▁je ▁re ç ois ▁ce ▁bloc ▁( di vin , ▁salut ▁xxmaj ▁vol odi a !) ▁- ▁et , ▁je ▁le ▁souligne , ▁à ▁20 ▁euros ▁de ▁moins ▁que ▁proposé ▁ici ▁en ▁passant ▁par ▁un ▁vendeur ▁d ' amazon ▁xxup ▁uk ▁- ▁alors ▁même ▁que ▁j ' ent a mais ▁( tra î nais ▁depuis ▁un ▁moment , ▁je ▁suis</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁xxbos ▁xxmaj ▁tout ▁y ▁est , ▁l ' histoire , ▁la ▁réalisation ▁du ▁très ▁grand ▁xxup ▁ridley ▁xxup ▁scott , ▁xxmaj ▁les ▁acteurs ▁et ▁surtout ▁xxup ▁russell ▁xxup ▁crowe , ▁xxmaj ▁les ▁xxmaj ▁décors , ▁enfin ▁tout ▁et ▁surtout ▁en ▁version ▁longue . ▁xxmaj ▁ gladiator ▁( ou ▁xxmaj ▁gladiateur ▁au ▁xxmaj ▁québec ▁et ▁au ▁nouveau - br un s wick ) ▁est ▁un ▁film ▁ américan o - br</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neg tensor([0.9970, 0.0030])\n"
     ]
    }
   ],
   "source": [
    "# Trying out some random sentences I made up\n",
    "\n",
    "review = 'Ce produit est bizarre.'\n",
    "pred = learn_c.predict(review)\n",
    "print(pred[0], pred[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fine-tuning \"backward Classifier\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Databunch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.42 s, sys: 376 ms, total: 2.79 s\n",
      "Wall time: 2.79 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_lm = load_data(path, f'{lang}_databunch_lm_aws_sp15_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 7min 26s, sys: 7.3 s, total: 7min 33s\n",
      "Wall time: 4min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = (TextList.from_df(df_trn_val, path, vocab=data_lm.vocab, cols='review_body', \n",
    "                              processor=[OpenFileProcessor(), SPProcessor(max_vocab_sz=15000)])\n",
    "    .split_by_rand_pct(0.1, seed=42)\n",
    "    .label_from_df(cols='label')\n",
    "    .databunch(bs=bs, num_workers=1, backwards=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5.9 s, sys: 1.1 s, total: 7 s\n",
      "Wall time: 7.49 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas.save(f'{lang}_textlist_class_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Get weights to penalize loss function of the majority class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13.2 s, sys: 432 ms, total: 13.6 s\n",
      "Wall time: 13 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_bwd', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(207616, 23068, 230684)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_trn = len(data_clas.train_ds.x)\n",
    "num_val = len(data_clas.valid_ds.x)\n",
    "num_trn, num_val, num_trn+num_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 23071, 184545]), array([ 2566, 20502]))"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_LabelCounts = np.unique(data_clas.train_ds.y.items, return_counts=True)[1]\n",
    "val_LabelCounts = np.unique(data_clas.valid_ds.y.items, return_counts=True)[1]\n",
    "trn_LabelCounts, val_LabelCounts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0.888876579839704, 0.11112342016029597],\n",
       " [0.8887636552800416, 0.11123634471995836])"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trn_weights = [1 - count/num_trn for count in trn_LabelCounts]\n",
    "val_weights = [1 - count/num_val for count in val_LabelCounts]\n",
    "trn_weights, val_weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training (Loss = FlattenedLoss of weighted CrossEntropyLoss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12.6 s, sys: 424 ms, total: 13 s\n",
      "Wall time: 12.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_bwd', bs=bs, num_workers=1, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_bwd');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "#### Change loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FlattenedLoss of CrossEntropyLoss()"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "loss_weights = torch.FloatTensor(trn_weights).cuda()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "functools.partial(<function cross_entropy at 0x7f6389801200>, weight=tensor([0.8889, 0.1111], device='cuda:0'))"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn_c.loss_func"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYkAAAEGCAYAAACQO2mwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de3xcdZ3/8dcnk0tzbZomvd8vgFVbLgFhQSyoiOiCLF7o6uJ1+a274ILirrv+FhTXxct6WUXF4gK6KiyCulzFGywoIKRc2lJaKKW06S1p01yaZJK5fPaPmbRDmsmlzcmcSd7Px2MenTnnOzOfbyeZTz7f7znfY+6OiIjIQApyHYCIiISXkoSIiGSlJCEiIlkpSYiISFZKEiIiklVhrgMYqdraWl+wYEGuwxARyStr1qzZ6+51I31e3iWJBQsW0NDQkOswRETyipm9ciTP03CTiIhkpSQhIiJZKUmIiEhWShIiIpKVkoSIiGSlJCEiIlkpSYiISFZKEiIiIXTP2p1s2t2R6zCUJEREwujK/36G86//Az9r2J7TOAJLEmZ2k5k1mdn6LPtXmlmbmT2Tvl0dVCwiIvkklkgSSziRAuPTd6zlM3euJRpL5CSWIJfluAW4HvjRIG0ecfd3BhiDiEje6YknAfjEm5fSEY3xnQdfYt2ONr73/pOYN7VsTGMJrJJw94eBlqBeX0RkvOpJVw1lxRE+/bbj+MEl9Wxv6eKWR7eOeSy5XuDvNDN7FtgJXOXuzw3UyMwuBS4FmDdv3hiGJyIy9qLpSmJSYQSAtyybzr2feCN1lSVjHksuJ66fAua7+wrg28AvszV099XuXu/u9XV1I17pVkQkr/RVEiVFh76i59aUMakoMuax5CxJuHu7ux9I378PKDKz2lzFIyISFtFYqpIoKRz7pNBfzpKEmc0wM0vfPyUdy75cxSMiEhY98cMriVwJbE7CzG4FVgK1ZtYIXAMUAbj7DcC7gY+bWRzoBi52dw8qHhGRfHGokhjHScLdVw2x/3pSh8iKiEiGvkoiF3MQ/eU+TYmIyKv0nScRhkoi9xGIiMir9J1drUpCREQOo0pCRESy6lElISIi2aiSEBGRrPrmJCb0yXQiIjKwnniSAoOiiOU6FCUJEZGwicYSlBRGSC9KkVNKEiIiIdMTTzIpBEtygJKEiEjo9MSSoZiPACUJEZHQicYTqiRERGRgqiRERCQrVRIiIpKVKgkREckqGk+E4oJDEGCSMLObzKzJzNYP0e5kM0uY2buDikVEJJ9MlEriFuDcwRqYWQT4MvBAgHGIiOSVCVFJuPvDQMsQzS4H7gSagopDRCTf9MSSTJoAlcSgzGw2cCFwQ65iEBEJo554cvxXEsPwTeAf3T0xVEMzu9TMGsysobm5eQxCExHJnZ5YIjSVRGEO37seuC29gFUtcJ6Zxd39l/0buvtqYDVAfX29j2mUIiJjLEyVRM6ShLsv7LtvZrcA9wyUIEREJpJE0ulNhGdOIrAkYWa3AiuBWjNrBK4BigDcXfMQIiID6O27Kt14ryTcfdUI2n4oqDhERPLJoavShSNJhCMKEREBDl3felJROIablCREREKkJ65KQkREsojGVEmIiEgWqiRERCQrVRIiIpKVKgkREcmqr5KYCEuFi4jICPVVErp8qYiIHEaVhIiIZKVKQkREsupRJSEiItlE+45uUiUhIiL9HaokwvH1HI4oREQESFUSJYUFpC/IlnNKEiIiIdITS4amigAlCRGRUOmJJygJyZIcEGCSMLObzKzJzNZn2X+Bma01s2fMrMHMzggqFhGRfNETS4bm8FcItpK4BTh3kP2/A1a4+/HAR4AfBBiLiEhe6IknQ3P4KwSYJNz9YaBlkP0H3N3TD8sBz9ZWRGSiiMYSE6aSGJKZXWhmG4F7SVUT2dpdmh6Samhubh67AEVExtiEqSSGw91/4e7HAe8CvjBIu9XuXu/u9XV1dWMXoIjIGFMlMYD00NRiM6vNdSwiIrmkSiLNzJZY+mwRMzsRKAb25SoeEZEwCFslURjUC5vZrcBKoNbMGoFrgCIAd78BuAi4xMxiQDfwvoyJbBGRCSlslURgScLdVw2x/8vAl4N6fxGRfBSNJXTGtYiIDKwnnmTSRDjjWkRERq4nrkpCREQG4O5EY8mJsXaTiIiMTG8iXNeSACUJEZHQiKYvOKQ5CREROUxP36VLVUmIiEh/Ybt0KShJiIiERl8loeEmERE5TFSVhIiIZKNKQkREstKchIiIZBVVJSEiItkcrCRCtFR4eCIREZngDlYSIVoqXElCRCQkVEmIiEhW0VjfGdcToJIws5vMrMnM1mfZ/34zW5u+PWpmK4KKRUQkH/TE+9ZuCs/f70FGcgtw7iD7Xwbe5O7LgS8AqwOMRUQk9PqSRJgqiSAvX/qwmS0YZP+jGQ8fB+YEFYuISD6IxhIURYxIgeU6lIPCUtN8FLg/204zu9TMGsysobm5eQzDEhEZOz3xZKiqCAhBkjCzs0gliX/M1sbdV7t7vbvX19XVjV1wIiJjKBpLhGo+AgIcbhoOM1sO/AB4u7vvy2UsIiK5pkoig5nNA34O/JW7v5CrOEREwiIaS4TqHAkIsJIws1uBlUCtmTUC1wBFAO5+A3A1MBX4rpkBxN29Pqh4RETCLoyVRJBHN60aYv/HgI8F9f4iIvkmGkuEagVYCMHEtYiIpPTEk6GbuA5XNCIiE1gYh5uUJEREQqInhIfAhisaEZEJTJWEiIhkFcaT6cIVjYjIBKZKQkREssrbQ2DNbLGZlaTvrzSzT5hZdbChiYhMLKlDYPOzkrgTSJjZEuA/gYXATwOLSkRkgoknkiSSnp+VBJB09zhwIfBNd78SmBlcWCIiE0v04FXp8rOSiJnZKuCDwD3pbUXBhCQiMvH09F3fOk+PbvowcBrwRXd/2cwWAj8OLiwRkYnlYCURsqObhrXAn7tvAD4BYGZTgEp3/1KQgYmITCR5XUmY2UNmVmVmNcCzwM1m9vVgQxMRmTiisVQlka/nSUx293bgL4Cb3f0k4C3BhSUiMrH0xPO4kgAKzWwm8F4OTVyLiMgoOVRJ5GeSuBZ4AHjJ3Z80s0XAi4M9wcxuMrMmM1ufZf9xZvaYmfWY2VUjC1tEZHzpqyTy8hBYd/+Zuy9394+nH29x94uGeNotwLmD7G8hNRn+78OJQURkPOuJ53ElYWZzzOwX6cpgj5ndaWZzBnuOuz9MKhFk29/k7k8CsZGFLCIy/kRjeVxJADcDdwGzgNnA3eltY8LMLjWzBjNraG5uHqu3FREZM3ldSQB17n6zu8fTt1uAugDjehV3X+3u9e5eX1c3Zm8rIjJmevK8kthrZh8ws0j69gFgX5CBiYhMJPleSXyE1OGvu4FdwLtJLdUhIiKjIKxzEsNdlmMbcH7mNjO7AvhmtueY2a3ASqDWzBqBa0gvCujuN5jZDKABqAKS6ddblj5pT0RkQumJJykwKCywXIfyKsNKEll8kkGShLuvGuzJ7r4bGPQIKRGRiSJ1VboIZuFKEkcz+BWunoiI5LHUVenCNR8BR5ckfNSiEBGZ4HpiydAt7gdDDDeZWQcDJwMDSgOJSERkAorGE6GsJAZNEu5eOVaBiIhMZGGtJMKXtkREJqCwVhLhi0hEZAJSJSEiIllF44nQXXAIlCREREJBlYSIiGTVo0pCRESyicaSTFIlISIiA+mJJ1VJiIjIwHpiCVUSIiIyMB3dJCIiA4rGEsQSTuWko1mYOxhKEiIiOdbWHQNgcmlRjiM5XGBJwsxuMrMmM1ufZb+Z2bfMbLOZrTWzE4OKRUQkzFq7UkmiurQ4x5EcLshK4hbg3EH2vx1Ymr5dCnwvwFhEREJrQlYS7v4w0DJIkwuAH3nK40C1mc0MKh4RkbCakEliGGYD2zMeN6a3iYhMKK1dvQBUlylJZBro8qcDXu3OzC41swYza2hubg44LBGRsdVXSVSpkniVRmBuxuM5wM6BGrr7anevd/f6urq6MQlORGSstHXHMIPKEh0Cm+ku4JL0UU6nAm3uviuH8YiI5ERbd4zJpUUUFAw0wJJbgaUtM7sVWAnUmlkjcA1QBODuNwD3AecBm4Eu4MNBxSIiEmZ9SSKMAksS7r5qiP0O/F1Q7y8iki9au2JUhzRJ6IxrEZEca+uOhXLSGpQkRERyrj3Ew01KEiIiOdbaHQvlORKgJCEiklPuHuqJayUJEZEc6uxNkEi6koSIiBzu4JIcIVwBFpQkRERyKsxLcoCShIhIToV5BVhQkhARyam2vgsO6egmERHpT5WEiIhkpSQhIiJZtXbHKIoYZcWRXIcyICUJEZEc6juRzix8y4SDkoSISE6FeXE/UJIQEcmpthAvEw5KEiIiORXmdZsg4CRhZuea2SYz22xmnxlg/3wz+52ZrTWzh8xsTpDxiIiEzYRNEmYWAb4DvB1YBqwys2X9mv078CN3Xw5cC1wXVDwiImHU2tVLdVk4122CYCuJU4DN7r7F3XuB24AL+rVZBvwuff/BAfaLiIxbyaTT0ROfsBPXs4HtGY8b09syPQtclL5/IVBpZlP7v5CZXWpmDWbW0NzcHEiwIiJjrSMaxz28J9JBsElioIN+vd/jq4A3mdnTwJuAHUD8sCe5r3b3enevr6urG/1IRURyoLW7b5nw8CaJwgBfuxGYm/F4DrAzs4G77wT+AsDMKoCL3L0twJhEREIj7EtyQLCVxJPAUjNbaGbFwMXAXZkNzKzWzPpi+CfgpgDjEREJlYNJIqQrwEKAScLd48BlwAPA88Dt7v6cmV1rZuenm60ENpnZC8B04ItBxSMiEjatfcuEh7iSCHK4CXe/D7iv37arM+7fAdwRZAwiImE10YebRERkEGG/dCkoSYiI5Exbd4xJRQVMKgrnMuGgJCEikjNtXeFekgOUJEREcibs6zaBkoSISM60dvdSXRredZtASUJEJGfausO9bhMoSYiI5ExbV6+Gm0REZGBt3TGqQ3y2NShJiIjkRCyRpLM3oUpCREQOlw9nW4OShIhITvQlCQ03iYjIYfJhSQ5QkhARyYm2Lg03iYhIFgeHm5QkRESkP01ci4hIVn0XHJrQcxJmdq6ZbTKzzWb2mQH2zzOzB83saTNba2bnBRmPiEhYtHXHqCgppCgS7r/VA4vOzCLAd4C3A8uAVWa2rF+z/0/qsqYnkLoG9neDikdEJEzyYQVYCLaSOAXY7O5b3L0XuA24oF8bB6rS9ycDOwOMR0QkNNq6e0M/1ATBJonZwPaMx43pbZk+B3zAzBpJXQv78oFeyMwuNbMGM2tobm4OIlYRkTHV1h0L/ZFNEGySsAG2eb/Hq4Bb3H0OcB7wX2Z2WEzuvtrd6929vq6uLoBQRUTGloabUpXD3IzHczh8OOmjwO0A7v4YMAmoDTAmEZFQaM2DS5dCsEniSWCpmS00s2JSE9N39WuzDXgzgJm9hlSS0HiSiIxb7s5vNuyhpbM39Os2ARQG9cLuHjezy4AHgAhwk7s/Z2bXAg3ufhfwKeBGM7uS1FDUh9y9/5DUuJVIOpGCgUblRGQ82tJ8gM/fvYH/faGZpdMquPiUebkOaUiBJQkAd7+P1IR05rarM+5vAE4PMoYwWtfYxvUPvshvNuxhcV0FpyysOXibObk01+GJyChzd77x2xf53kObmVQY4V/euYxLTpsf+nMkIOAkIYf0xpM8tW0/333oJR5+oZmqSYW8/w3z2b6/i/95Zic/+dM2AE5fMpX3v2E+b102PS9+gERkaHesaeRbv3uR81fM4l/euYy6ypJchzRsShIB2bq3k9WPbOHl5k62tXSxq62bpMPU8mL+4dxj+atT51M5KTUemUg6z+9q58GNTdz25Hb+9idPUVdZwruOn0VFSRG9iQSxhFMUMS45bQHTqybluHciMlzbW7r4/N0bOGVhDd943/F5N8Rs+TYFUF9f7w0NDaP+urFEkkdf2kdxpIDaimKmVpRQXVpEwQg/UHfn9obtfP7uDQC8ZmYV82rKmFtTxuK6cs5ZNoPS4kjW5yeSzkObmvjJn7bx4KYm3CFSYBRFjFjCqSgp5NoLXsv5K2ZhNnBs7s5T21r57fN72NvRQ2t3jLauGF2xOPNryjl2RiXHTK/kmOkVzK0pU8UiEpBE0rl49WNs3NXB/Ve8kTlTynIWi5mtcff6kT5PlQSpL9XP/mIdtzc0vmp7eXGEv3/LUj5y+kIKh/FFur+zl3/6+Tp+9dxuTls0la+/b8WI5xgiBcabXzOdN79mOr3xJJECO/iXx5bmA3zqZ8/y97c9w/3rdvOvF76O2oqSg33Y3R7lF0/v4I41jWxp7qQoYtRWlDC5tIjqsiKmlpewfmcb963fRd/fBpECY3Z1KfOnphJZbUUJNWVFTCkvpraihGOmV+ZVaSwSJt9/+CWe3Lqfr793RU4TxNFQJQH88NGtXHPXc3zsjIWcfdw09nb2su9AD4+8uJffb2zidbOr+NJfLOd1sydnfY2ntu3n4z9eQ0tnL1edcyx//cZFI65ChiORdG58ZAtf//ULFEWMspJCunsTdPXGSaY/ypMXTOE9J83lvOUzqSg5/O+Art44L+45wKY9HWzb18UrLV28sq+T7S1dtHbH6P8jMa2yhNfOqmL5nGo++GcLqCkvHvV+jZS78/yuDqLxBBFLJdKSwgLmTS2jpPDwSq1xfxfrd7SxsLaCpdMqAvlsRDKt39HGhd/9I+csm8H1f3lC1sp/rBxpJTHhk8RjL+3jA//5J1YeU8eNl9S/6svD3bl//W6uues5Wjp7+egZC7n87CUH5xL63L9uF1f89zNMr5rEd99/4qDJZLS8sKeDm//4MgClRYWUFUeYXFrEW5dNZ0Ft+RG/biLptHXHaOnspak9yoZd7WzY2c5zO9t5samDmvJivnjh63nba2eMVldG7Klt+7nuvud5cuv+w/YVRYzjZlSxfM5klkyrYMPOdh7bso/G/d0H21RNKuSk+VNYPqea9miMxv3dbG/poqWzlwtPnM1lZx3+GYuMRDSW4M+//QfaumM8cMWZTAnBH1ZKEkegcX8X51//R6rLivjl351OVZYvhrbuGF+6fyO3PrGNKWVFXHb2Uj5w6jyKIwXc+MgWrrt/IyfMrebGS+qZWjF+h2ae39XOp25/lg272rnwhNl87s9fy+QxPBloS/MBvvrAJu5fv5vaihIuO2sxC2rLSSSdRNLpjiXYsKuddY1trGtso6MnzuTSIk5dVMOpi6ayYm41W5o7WfNKC09u3c/mpgOUFUeYO6WMOVNKKSgwfrNhD1PLi/nUOcfyvpPn5t0ko4TDdfc/z/f/dwu3fPhkVh47LdfhAEoSI9YRjfG+7z/O9pYufnnZ6SyuqxjyOesa2/jKAxt55MW9zK4uZcXcydy3bjfveP1MvvbeFUwqyj4hPV70xpN858HNfOfBzVSXFXPqohoW1VWwuK6cWdWl7GztZnPTAV7cc4Adrd3MmVLKsTMqOW5GJUumVVJWHKHAjIICMIwDPXHaumO0d8fo7IlTU17MrOpSplWVUFRQwPqdbTy0qZkHNzXxzPZWSosi/L8zF/OxNy6kfIChtD7JpNPU0cO0ypKsQ0vRWIKSwoJXDQOsa2zjC/ds4ImtLSydVsGxMyopihRQWGAURgqIFJAe3iqgwKCzN8GBnjgHojE6e1OvV1FSSFlxIaXFBbR3x9nX2cO+A710ROOcdVwdHz1jEQuPotobyoad7fz0iVe4b91uZlRNOngOzskLao5qfqlxfxelRRFqyotzPnQSZk9t28+7v/co762fy5cuWp7rcA5SkhiBhzY18c8/X8fu9ij/+cGTOeu4kWX6P7y4ly//aiPrdrTxN29azD+87dgJN8a9fkcb3/ztC2za00Hj/u5XzWNECoz5NWXMnlLK9pbUnMeR/JiVFUfo6k1gBivmVHPWsdP4yzfMC3wi3d351frdfP/hLbR3x4glk8QTTizhJD1VtSSTTsKdsuJCKicVUlFSSGlxhN54ks6eOF3peaKq0iKmlqeOlouY8fuNTcSSSd583HQ+esZCTl4wZVgHRWRqj8bYtLuDl5oOEEs6ETMKC4zuWIJfPrODp7e1UlJYwFuXTaels5entu0nGksCMKNqEsfNrOS4GVUsnVZBVyzBnrYou9uj7DvQw5JpFZy6aConL6yhalIRTe1R/ueZnfz86R08v6sdSB3QMbemjHk1ZSxJJ9JjpleyqK58wPmgoMQSSXa2drO9pfvgYeYzJk9i+exqjp1RSXHh2B+1F40leMe3HqGrN8EDV56ZdXQiF5QkhqG1q5dr79nAz5/awZJpFXzl3cs5cd6UI3qtZNLZ0drN3Jr8PGJhNEVjCV7Z18XO1m5mT0kdKZX5ZdE3Uf5S8wF640kSnvqSdaCipJDJpUVMLi2itDhCS2cvu1qj7GqLsr+rlxVzJ3Pm0rpxM4zX1BHlx4+9wn89/gr7u2IURwpYVFd+8JDkY6anvsD7hr9640nW7Wjjya0tNGxt4fldHexo7c76+ovryvnLN8znohNnU12WGgfvjSdZv7ONNVv38/yudp5PJ5jeRCpxRAqMuooSqsuK2NLcSW8iSYHBwtpyXt7bSdJhxZzJ/PmKWRSYsa2l62Dy37q3k3j6iInCAuP0JbVcdNIczlk2/Ygr6237uvjZmu3cu3YXZSURlk6rZOn0ChbVlrO7LcpzO9tZv7OdzU0dxBIDf38VFxbwmhmVlBUXEk8mDyb4ZTOrOOu4aZyxpPZgJRqNJVL/L7s6WDKtgvr5U474j77r7nue7z+8hR9+5BTedEy4VqxWkhjCo5v38onbnqG1q5ePr1zMZWcvGdO/ekQydfcm+PWG3WzY2c4Lezp4IT0816esOMK8mjK27us8WAUsqivn9bMnHxy+WzqtkpKiApJJSKR/j2dNnjSsoaBYIsn2li4qSgpTVU76SzEaS/DUtv08vqWFZ7e38vrZk3nXCbNZMm3g4djeeJKX93ayaU8H63e0cc+zO9nZFqWypJB3LJ/JxafMY8WcyUPGlEg6967bxW1PbOPRl/ZhBqcvrsUMXtxzgN3t0YNtp5YX89rZk1k2s4pFdeUHz0OaXlnCztYozza2sm5HG+t3tNEbT1IYMYoiBbjDs9tb6eiJUxwp4OSFU2jvjrNxd/urks30qhLe/rqZvGP5TJZOq6CipHBY1V5Yh5n6KEkM4YU9HXzmzrX867tez7JZVUM/QWSMHeiJ88KeDjbtTt1e3tvJorpy3rCwhvoFNQfPiQmzZNJ5fMs+7niqkV+t301Xb4LXza7iA2+Yz/nHz6KsuPCw9veu28U3fvsCW5o7mVtTyntPmstFJ81hVvWhc4zaozG27u1kWuUkpleVHPGcSG88ScPWFn6/sYk/bN5LTXkxy+dUc/zcyRw3o4q1O9q4d+1OHtzUTG88efB5pUURqkoLWXXKPC4/e+lhBzRs3dvJh25+gt54kgeuPDOUR8cpSQyDu2vCTWSMdERj/PLpHfz48W1s2tNBZUkhr5s9mXk1ZcybWkZVaRE/efwVNu7u4JjpFXzyrcdwzrIZoZjfO9AT5+EXmtndFuVAT5yOaIyXmjv5/cYmzlhSyzcvPv5g0r5n7U4+c+c6IgXGjZfUc8rCmhxHPzAlCREJJXen4ZX93NHQyItNHWxr6WbvgR4gNe9xxVuW8s7ls0J/uHHfkjtX/89zVJcV8bX3HM+vntvFjx/fxgnzqvn2qhNCfVa1koSI5I2u3jh72nuYO6V0xEd35dqGne387U/WsHVfFwCXnrmIT7/t2NCvgaa1m0Qkb5QVF7KwNj+/fpbNquLuy8/g27/fzGmLpo74EPp8E+inZGbnAv9B6sp0P3D3L/Xb/w3grPTDMmCau1cHGZOIyNGqnFTEP5/3mlyHMSYCSxJmFgG+A7wVaASeNLO70lejA8Ddr8xofzlwQlDxiIjIyAU5iHYKsNndt7h7L3AbcMEg7VcBtwYYj4iIjFCQSWI2sD3jcWN622HMbD6wEPh9lv2XmlmDmTU0NzePeqAiIjKwIJPEQMezZTuU6mLgDndPDLTT3Ve7e72719fVhetUdxGR8SzIJNEIzM14PAfYmaXtxWioSUQkdIJMEk8CS81soZkVk0oEd/VvZGbHAlOAxwKMRUREjkBgScLd48BlwAPA88Dt7v6cmV1rZudnNF0F3Ob5dlafiMgEEOh5Eu5+H3Bfv21X93v8uSBjEBGRI5d3y3KYWTPwygC7JgNtR/i4737fv7XA3iMMsf/7jLRNWPoxVJxD7R/NfkCwn8lI+jHQtoFiz7yvfgw/zqHaqB9H3o/57j7yI3/cfVzcgNVH+rjvfsa/DaMVx0jbhKUfw+nLWPUj6M9kJP0Ybuzqx5H3Y7A26sfo92OoW7hXpBqZu4/i8d1Z2oxGHCNtE5Z+DOd1JmI/Bto2UOyZ99WPoWMZbhv1Y/T7Mai8G24aC2bW4EewWmLYjJd+wPjpi/oRLurH0MZTJTGaVuc6gFEyXvoB46cv6ke4qB9DUCUhIiJZqZIQEZGslCRERCSrcZ8kzOwmM2sys/VH8NyTzGydmW02s2+ZmWXsu9zMNpnZc2b2ldGNesBYRr0fZvY5M9thZs+kb+eNfuSHxRLI55Hef5WZuZnVjl7EWWMJ4vP4gpmtTX8WvzazWaMf+WGxBNGPr5rZxnRffmFmgV9ILKB+vCf9+500s0Ant48m/iyv90EzezF9+2DG9kF/hwYU1LG1YbkBZwInAuuP4LlPAKeRWtH2fuDt6e1nAb8FStKPp+VpPz4HXJXvn0d631xSS8C8AtTmYz+Aqow2nwBuyNN+nAMUpu9/GfhynvbjNcCxwENAfRjjT8e2oN+2GmBL+t8p6ftTBuvrYLdxX0m4+8NAS+Y2M1tsZr8yszVm9oiZHdf/eWY2k9Qv7WOe+t/9EfCu9O6PA19y9570ezQF24vA+jHmAuzHN4B/IPty9KMqiH64e3tG03LGoC8B9ePXnlq7DeBxUitAByqgfjzv7puCjv1o4s/ibcBv3L3F3fcDvwHOPdLvgnGfJLJYDVzu7icBVwHfHaDNbFLLnffJvGjSMcAbzexPZva/ZnZyoNFmdx2XNxcAAAV5SURBVLT9ALgsPSxwk5lNCS7UQR1VPyy1YOQOd3826ECHcNSfh5l90cy2A+8HriY3RuPnqs9HSP3Fmguj2Y9cGE78A8l2wbcj6mugC/yFkZlVAH8G/CxjOK5koKYDbOv7y66QVBl3KnAycLuZLUpn5zExSv34HvCF9OMvAF8j9Us9Zo62H2ZWBnyW1BBHzozS54G7fxb4rJn9E6lVlK8Z5VAHNVr9SL/WZ4E48JPRjHE4RrMfuTBY/Gb2YeDv09uWAPeZWS/wsrtfSPY+HVFfJ1ySIFU9tbr78ZkbzSwCrEk/vIvUF2hmmZx50aRG4OfppPCEmSVJLbA1ltdWPep+uPuejOfdCNwTZMBZHG0/FpO69O2z6V+mOcBTZnaKu+8OOPZMo/FzlemnwL2McZJglPqRnix9J/DmsfzjKcNofx5jbcD4Adz9ZuBmADN7CPiQu2/NaNIIrMx4PIfU3EUjR9LXICdjwnIDFpAxIQQ8Crwnfd+AFVme9ySpaqFvkue89Pa/Aa5N3z+GVGlnediPmRltriR1XY+8+zz6tdnKGExcB/R5LM1oczmpS/rmYz/OBTYAdWMRf9A/V4zBxPWRxk/2ieuXSY12TEnfrxlOXweMayw/xFzcSF0WdRcQI5VJP0rqL89fAc+mf5ivzvLcemA98BJwPYfOUC8Gfpze9xRwdp7247+AdcBaUn9VzczHfvRrs5WxObopiM/jzvT2taQWb5udp/3YTOoPp2fSt7E4SiuIflyYfq0eYA/wQNjiZ4Akkd7+kfTnsBn48Eh+h/rftCyHiIhkNVGPbhIRkWFQkhARkayUJEREJCslCRERyUpJQkREslKSkHHBzA6M8fv9wMyWjdJrJSy18ut6M7t7qFVTzazazP52NN5bZCg6BFbGBTM74O4Vo/h6hX5okbpAZcZuZj8EXnD3Lw7SfgFwj7u/bizik4lNlYSMW2ZWZ2Z3mtmT6dvp6e2nmNmjZvZ0+t9j09s/ZGY/M7O7gV+b2Uoze8jM7rDU9RF+0rf+fnp7ffr+gfTCfM+a2eNmNj29fXH68ZNmdu0wq53HOLRwYYWZ/c7MnrLUNQAuSLf5ErA4XX18Nd320+n3WWtmnx/F/0aZ4JQkZDz7D+Ab7n4ycBHwg/T2jcCZ7n4CqZVW/y3jOacBH3T3s9OPTwCuAJYBi4DTB3ifcuBxd18BPAz8dcb7/0f6/YdcIye9rtCbSZ39DhAFLnT3E0ldw+Rr6ST1GeAldz/e3T9tZucAS4FTgOOBk8zszKHeT2Q4JuICfzJxvAVYlrGKZpWZVQKTgR+a2VJSq2AWZTznN+6eua7/E+7eCGBmz5BaX+cP/d6nl0OLI64B3pq+fxqH1uv/KfDvWeIszXjtNaTW/4fU+jr/lv7CT5KqMKYP8Pxz0ren048rSCWNh7O8n8iwKUnIeFYAnObu3ZkbzezbwIPufmF6fP+hjN2d/V6jJ+N+goF/Z2J+aHIvW5vBdLv78WY2mVSy+TvgW6SuKVEHnOTuMTPbCkwa4PkGXOfu3x/h+4oMScNNMp79mtQ1GQAws75llycDO9L3PxTg+z9OapgL4OKhGrt7G6nLll5lZkWk4mxKJ4izgPnpph1AZcZTHwA+kr4GAWY228ymjVIfZIJTkpDxoszMGjNunyT1hVufnszdQGqJd4CvANeZ2R+BSIAxXQF80syeAGYCbUM9wd2fJrXq58WkLtZTb2YNpKqKjek2+4A/pg+Z/aq7/5rUcNZjZrYOuINXJxGRI6ZDYEUCkr5qXre7u5ldDKxy9wuGep5ImGhOQiQ4JwHXp49IamWMLw0rMhpUSYiISFaakxARkayUJEREJCslCRERyUpJQkREslKSEBGRrP4PPOmZ2fnnxn0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 2e-2\n",
    "lr *= bs/48\n",
    "\n",
    "wd = 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.501302</td>\n",
       "      <td>0.387383</td>\n",
       "      <td>0.823348</td>\n",
       "      <td>0.885338</td>\n",
       "      <td>02:57</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.523619</td>\n",
       "      <td>0.373636</td>\n",
       "      <td>0.802974</td>\n",
       "      <td>0.871117</td>\n",
       "      <td>03:04</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas1_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.468178</td>\n",
       "      <td>0.407151</td>\n",
       "      <td>0.740550</td>\n",
       "      <td>0.822256</td>\n",
       "      <td>02:52</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.457562</td>\n",
       "      <td>0.391507</td>\n",
       "      <td>0.795084</td>\n",
       "      <td>0.865942</td>\n",
       "      <td>03:05</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas1_sp15_bwd');\n",
    "learn_c.fit_one_cycle(2, lr, wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas2_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.391392</td>\n",
       "      <td>0.283015</td>\n",
       "      <td>0.895570</td>\n",
       "      <td>0.936219</td>\n",
       "      <td>03:39</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.363020</td>\n",
       "      <td>0.253934</td>\n",
       "      <td>0.898908</td>\n",
       "      <td>0.938087</td>\n",
       "      <td>03:27</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas2_sp15_bwd');\n",
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(2, slice(lr/(2.6**4),lr), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas3_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.332751</td>\n",
       "      <td>0.218908</td>\n",
       "      <td>0.912043</td>\n",
       "      <td>0.946311</td>\n",
       "      <td>04:13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.285485</td>\n",
       "      <td>0.208085</td>\n",
       "      <td>0.915641</td>\n",
       "      <td>0.948510</td>\n",
       "      <td>04:21</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas3_sp15_bwd');\n",
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(2, slice(lr/2/(2.6**4),lr/2), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas4_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.254880</td>\n",
       "      <td>0.205224</td>\n",
       "      <td>0.922057</td>\n",
       "      <td>0.952664</td>\n",
       "      <td>04:55</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15_bwd');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>f1</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.273147</td>\n",
       "      <td>0.203318</td>\n",
       "      <td>0.931810</td>\n",
       "      <td>0.958756</td>\n",
       "      <td>05:32</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.load(f'{lang}clas4_sp15_bwd');\n",
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(1, slice(lr/10/(2.6**4),lr/10), wd=wd, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas5_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas5_sp15_bwd')\n",
    "learn_c.save(f'{lang}clas_sp15_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.load(f'{lang}clas_sp15_bwd');\n",
    "learn_c.to_fp32().export(f'{lang}_classifier_sp15_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Confusion matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 16 s, sys: 1.11 s, total: 17.1 s\n",
      "Wall time: 15.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15_bwd', bs=bs, num_workers=1, backwards=True)\n",
    "\n",
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True\n",
    "\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=0.5, \n",
    "                                  metrics=[accuracy,f1])\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_sp15_bwd');\n",
    "\n",
    "learn_c.load(f'{lang}clas_sp15_bwd');\n",
    "\n",
    "# put weight on cpu\n",
    "loss_weights = torch.FloatTensor(trn_weights).cpu()\n",
    "learn_c.loss_func = partial(F.cross_entropy, weight=loss_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAbYAAAEGCAYAAAAJw7AFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAY2UlEQVR4nO3deZgcVbnH8e8b1iwEErJBWIWENREIAnKviKAYEFGvyiIqIIqggLKJCgICImquqCDrRRbBGDcUAiqIIGsUEhZREMImhkUWIYFAgPDeP6omdMIkmZmkp8OZ7+d55qGr6nT1W2F6fn1OV52KzESSpFL0anUBkiQtTgabJKkoBpskqSgGmySpKAabJKkoBpskqSgGm7QYRUTviLgsIp6LiJ8vwn72jIgrF2dtrRIR74iIf7S6DvUc4XVs6oki4mPAocD6wAzgduAbmXnDIu73E8BBwNaZ+eoiF7qEi4gERmTm1FbXIrWxx6YeJyIOBb4HnAQMBdYATgc+sBh2vyZwb08ItY6IiKVbXYN6HoNNPUpErAgcD3w+M3+VmS9k5iuZeVlmHlG3WS4ivhcRj9Y/34uI5ept20bEvyLisIj4d0Q8FhH71Nu+DhwD7BYRz0fEvhFxXERc1PD6a0VEtv3Bj4i9I+KBiJgREQ9GxJ4N629oeN7WEXFLPcR5S0Rs3bDt2og4ISJurPdzZUQMms/xt9X/pYb6PxgRO0XEvRHxTER8taH9FhFxc0Q8W7c9LSKWrbddVze7oz7e3Rr2f2REPA6c17aufs469WtsVi+vGhFPRcS2i/Q/VmpgsKmneTuwPHDJAtocBWwFbAK8FdgCOLph+zBgRWA4sC/ww4gYkJnHUvUCJ2Rmv8w8d0GFRERf4AfAjpm5ArA11ZDovO0GApfXbVcGvgtcHhErNzT7GLAPMARYFjh8AS89jOrfYDhVEJ8DfBwYA7wDOCYi3lK3nQ0cAgyi+rfbHvgcQGZuU7d5a328Exr2P5Cq97pf4wtn5v3AkcDFEdEHOA84PzOvXUC9UqcYbOppVgaeWshQ4Z7A8Zn578x8Evg68ImG7a/U21/JzCuA54H1uljPa8DGEdE7Mx/LzL+10+Z9wH2Z+ePMfDUzxwP3AO9vaHNeZt6bmS8CP6MK5fl5her7xFeAn1KF1vczc0b9+n8DRgNk5uTMnFS/7kPAWcA7O3BMx2bmrLqeuWTmOcB9wJ+BVag+SEiLjcGmnuZpYNBCvvtZFXi4Yfnhet2cfcwTjDOBfp0tJDNfAHYD9gcei4jLI2L9DtTTVtPwhuXHO1HP05k5u37cFjxPNGx/se35ETEyIiZGxOMRMZ2qR9ruMGeDJzPzpYW0OQfYGDg1M2ctpK3UKQabepqbgZeADy6gzaNUw2ht1qjXdcULQJ+G5WGNGzPz95n5Hqqeyz1Uf/AXVk9bTdO6WFNnnEFV14jM7A98FYiFPGeBp1pHRD+qk3fOBY6rh1qlxcZgU4+Smc9Rfa/0w/qkiT4RsUxE7BgR366bjQeOjojB9UkYxwAXzW+fC3E7sE1ErFGfuPKVtg0RMTQidqm/a5tFNaQ5u519XAGMjIiPRcTSEbEbsCEwsYs1dcYKwHTg+bo3ecA8258A3vKGZy3Y94HJmflpqu8Oz1zkKqUGBpt6nMz8LtU1bEcDTwKPAAcCv66bnAjcCtwJ/BWYUq/rymtdBUyo9zWZucOoF3AYVY/sGarvrj7Xzj6eBnau2z4NfAnYOTOf6kpNnXQ41YkpM6h6kxPm2X4ccEF91uSuC9tZRHwAGEs1/ArV/4fN2s4GlRYHL9CWJBXFHpvmiIjVI+KaiLg7Iv4WEV+o158QEXdGxO31NVKr1uvXr69xmhURh8+zr0PqfdwVEeMjYvlWHJPUVfN7PzRsP7y+JnFQvbxiVNOp3VG336c1lcsem+aIiFWAVTJzSkSsQDV09kHgX5k5vW5zMLBhZu4fEUOoTmr4IPCfzBxXtxkO3FC3ezEifgZckZnnd/9RSV0zv/dDZv49IlYH/o9qSrYxmflUfWH7ipl5ZEQMBv4BDMvMl1t2ED2UPTbNUV9HNaV+PAO4GxjeFmq1vtRnvdXXed1CdV3UvJYGeten1feh62cVSi0xv/dDvfkUqu86G3sGCawQEUF1ucQzgFOrtUDTgi2qqYPujohz6m75lVHNfL5ORPwuIiZHxPVt1+3U6ydFNV3Q8RHxfLNq08JFxFrAplQX0RIR34iIR6guXj5mQc/NzGnAOOCfwGPAc5lZxEz16pka3w8RsQswLTPvmKfZacAGVB/i/gp8ITNf6846VWl2j20E8MPM3Ah4FvgwcDZwUGaOoTrj6vS67fepZj94G366b6n6OqNfAl9s661l5lGZuTpwMdUZhAt6/gCqCYXXprq4uG9EfLy5VUvN0fh+oOqBHUX7H+7eS3V5x6pUM7+cFhH9u6tOva5p37HVn3CuyswR9fKRwDJUvxSN92ZaLjM3iIingaGZ+Wr9y/BoZr5h9oSI2I96/rk+ffqOWWdEV2cyUnsyk4cemEq//v0ZPHjoG7a//PIsHnrwfkaut+GcdU88/ii9ei3F4CFV++ee/Q8zZkxntdWra4r/88zTzJz5AsNXW6N7DqKHWGaphV0nrUWVmUydOpX+/fszdOhQXnzxRe6771569ar6BC+//DLLLLMM66+/AQ8//BDDhg2jX78VALj33nsZPnw4ffv2beUhFG3KlMlPZebgedc3O9gmZubG9fLhVJ9k9sjMVdpp36FgazR6kzE58Y83Lfbae6rM5NDP7ctKAwZy7Enj5qx/8P6prL3OugCcd/bp/Pmm6znz/PFztp/yrRPo07cfnz3wEABuu/UvHHHwZ7nsDzeyfO/eHPb5TzNqkzHss98bLtHSIhjSf7lWl1C0zOTT++zFgIEDGffd77XbZr111+LGSbcyaNAgDv78AQwZOpSjjzmOJ554grdvsRl/mXwHgwYtbAYydVXvZWJyZm4+7/ruvlfSdODBiPhoZv68/pJ1dD1WPYlqqHICsHs31yXg1j/fxK9+9hPW33BjdnznFgAccfTxTLjofB6YWn1KHb76Gpw07lQA/v3E47x/+//i+RnT6dWrFz868zT+cNNtbLr5Fuy0y4d437u2Yqmll2ajUW/lY3vt28pDkzrtphtv5CcX/5iNNx7FlmOqOaW/fuJJjN1xp3bbf/mor7Hfvnuz+SajSJJvnPQtQ61FurvH1g+4gGr+uVWohiZ/mpnHR8QIqmmLgmqanf0yc3g7u57DHpt6Mnts6um6vcdW3+Ji44blcQ2bx7bzlGnAVpmZEbE71ZRGkiR1ypJ02/YxVGcRBdUZlJ9qcT2SpDehJSbYMvN6qrsVS5LUZc48IkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqisEmSSqKwSZJKorBJkkqytLz2xARlwE5v+2ZuUtTKpIkaRHMN9iAcd1WhSRJi8l8gy0z/9SdhUiStDgsqMcGQESMAL4JbAgs37Y+M9/SxLokSeqSjpw8ch5wBvAq8C7gQuDHzSxKkqSu6kiw9c7Mq4HIzIcz8zhgu+aWJUlS1yx0KBJ4KSJ6AfdFxIHANGBIc8uSJKlrOtJj+yLQBzgYGAN8AtirmUVJktRVC+2xZeYt9cPngX2aW44kSYumI2dFXkM7F2pnpt+zSZKWOB35ju3whsfLAx+mOkNSkqQlTkeGIifPs+rGiPDibUnSEqkjQ5EDGxZ7UZ1AMqxpFXVCr17Qd9mlWl2G1BID3nZgq0uQlkgdGYqcTPUdW1ANQT4I7NvMoiRJ6qqOBNsGmflS44qIWK5J9UiStEg6ch3bTe2su3lxFyJJ0uKwoPuxDQOGA70jYlOqoUiA/lQXbEuStMRZ0FDke4G9gdWA/+X1YJsOfLW5ZUmS1DULuh/bBcAFEfHhzPxlN9YkSVKXdeQ7tjERsVLbQkQMiIgTm1iTJEld1pFg2zEzn21byMz/ADs1ryRJkrquI8G2VOPp/RHRG/B0f0nSEqkj17FdBFwdEefVy/sAFzSvJEmSuq4jc0V+OyLuBN5NdWbk74A1m12YJEld0ZGhSIDHgdeoZvbfHri7aRVJkrQIFnSB9khgd2AP4GlgAhCZ+a5uqk2SpE5b0FDkPcD1wPszcypARBzSLVVJktRFCxqK/DDVEOQ1EXFORGzP67OPSJK0RJpvsGXmJZm5G7A+cC1wCDA0Is6IiB26qT5JkjploSePZOYLmXlxZu5MNW/k7cCXm16ZJEld0NGzIgHIzGcy86zM3K5ZBUmStCg6FWySJC3pDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRDDZJUlEMNklSUQw2SVJRlm51AVqyHHzAp7nyd1cwaPAQbvjL7QB884Rj+e3ll9KrVy8GDR7CqWeeyyqrrMrPJ/yEU0/5DgB9+/bjO987jY1HvXXOvmbPns27t9mSYasMZ/wvftOS45EW5sxj92THbTbmyWdmsPlHTwJg1MjhnHrU7vTtvRwPP/o0+xx1ATNeeImBK/blJ9/ZlzEbrclFl07ikG/9HIB+fZbjDz86ZM4+hw9ZiZ9ecQtHjPslqw8bwDnHf4IVV+jNUr168bVTf8Pvb/h7S461p7DHprnsvudeTLhk4lzrDvzCYVw36TauvWkyO4zdiXEnnwjAmmuuxaW//SPXTbqNw448ikMPPmCu5511+g8Ysd4G3Va71BU/vmwSH/j8D+dad8YxH+PoH/yGt+16EpdecweH7LU9AC/NeoXjT5/IV065ZK72z8+cxVa7nzzn55+PPcOv/1h9MDzy02P55VVTePse3+KTXzmP739lt+45sB7MYNNctv7vdzBgwMC51q3Qv/+cxzNfmElEALDFVluz0oABAGz+ti15dNq0Oe0enfYvrvr9b/n4Xp/qhqqlrrtxyv0889zMudaNWHMIN0yeCsAfJ93DB7ffBICZL73MTbc/wEuzXpnv/tZZYzBDBq7AjVPuByAz6d93eQBW7Nebx558rhmHoQZNHYqMiLWA3wF/BjYF7gU+CbwdGFe//i3AAZk5KyJOBnYBXgWuzMzDm1mfOu4bX/8aE8ZfRP/+K/Lry696w/aLLjyP7d/z3jnLRx15GMee8E2ef/757ixTWiz+fv9j7LztKCZe+1f+5z2bsdrQAR1+7q5jx/CLK6fMWf7GWVdw2ekHcsDu76RP7+V43/6nNqNkNeiOHtt6wNmZORqYDhwKnA/slpmjqMLtgIgYCHwI2Khue2I31KYOOurYE7jzngf5yK578H9nnz7Xtuuvu5aLLzyPY4//JgC//+3lDBo8mE02HdOKUqVF9tnjLuazu27DjRd/iX59luPlV2Z3+Lkffe8Yfva7W+cs7zp2cy66bBLrjv0aHzroDM498ZNzRj3UHJGZzdt51WO7LjPXqJe3A74GLJWZ29Trtgc+D+wKTAZuBS4HJmbmy+3scz9gv3pxPeAfTTuAnmtZYATwt4VsGwS8AKwD3AfMqtsMB1YGkurDUy/gWeDBplYtddHIkSOXnThx4oiRI0e+4Xd+1KhRy1188cVvGT169N1t6w466KCVR48evfJnPvOZexvbbrXVVr3Hjx+/ztprr31X27r77rtvo7Fjx957//33vwLwyCOPjNpyyy3vfvTRR19t5jH1EGtm5uB5V3bHWZEdSs7MfDUitgC2B3YHDgS2a6fd2cDZi7VCzaX+QDIxMzevl0dk5n3144OAd2bmRyLiTqAPsF1m3jSffW0LHJ6ZO3dH7VIXrUXD7zwwBPg31Yey84FTMvNHDe33vvDCC7/d0L7NycCszDy2Yd1vp06dOqHezwbA1dOmTduUDv5tVOd1R4/tQWDrzLw5Is4BHgI+S/XHcGpEnA/cBpwL9MnMf9fDklMzc2C7O1bTRMR4YFuq3tgTwLHATlS949eAh4H9M3NaRDxJ9eHo4frpr877RjfY9CbQ3u98P6qRJIBfAV/h9SB6COg/c+bMFfv06fMYsAPQdv7+A1Tvl3sa9r8hcE69zwS+BFzZnEMRdE+wXQFcB2xNNVz1Cdo5eQQYCPwGWB4IYFxmXtC04rTIIuLWdj6xSj2Cv/9Lru4YinwtM/efZ93VVGdJNnoM2KIb6tHi45CwejJ//5dQ3dFjm5iZGzftRSRJatDUYJMkqbs584gkqSgGmySpKAabJKkoBps6JSJmRMT0eX4eiYhLIuItra5PaqaI+HZE9I+IZSLi6oh4KiI+3uq6NDeDTZ31XeAIqmmzVgMOp7r49KfAjxbwPKkEO2TmdGBn4F/ASKr3g5YgBps6a2xmnpWZMzJzej3F2U6ZOQHo+BTo0pvTMvV/dwLGZ+YzrSxG7TPY1FmvRcSuEdGr/tm1YZvXjqh0l0XEPcDmwNURMRh4qcU1aR5ex6ZOqb9H+z7VtGgJTAIOAaYBYzLzhhaWJzVdRAwApmfm7IjoA/TPzMdbXZdeZ7BJUgdFxDJUc9tuU6/6E3BmZs7/ltrqdg5FqlMiYmR9Nthd9fLoiDi61XVJ3eQMYAxwev2zWb1OSxB7bOqUiPgT1VlgZ2XmpvW6u5wPVD1BRNyRmW9d2Dq1lj02dVafzPzLPOu8E7B6itkRsU7bQv2d8+wW1qN2dMdta1SWp+o3dgJExEeobjkk9QRHANdExAP18lrAPq0rR+1xKFKdUn9CPZvqxrH/obpD+p6Z+fACnygVICKWBw4Dtq9XXQWckpme8r8EMdjUKRGxHPARqk+qA4HpQGbm8a2sS+oOEfEzqt/5i+tVewADMvOjratK83IoUp31G+BZYArwaItrkbrbevOcKHJNRNzRsmrULoNNnbVaZo5tdRFSi9wWEVtl5iSAiNgSuLHFNWkeDkWqUyLibODUzPxrq2uRultE3A2sB/yzXrUGcDfwGtWQ/OhW1abXGWzqlIj4O7Au1Ukjs4DAN7R6iIhYc0HbPYlqyWCwqVPm98b2DS1pSWGwSZKK4swjkqSiGGySpKIYbFKTRcTsiLg9Iu6KiJ/X9/Dq6r62jYiJ9eNdIuLLC2i7UkR8rguvcVxEHN7VGqVWM9ik5nsxMzep74DwMrB/48aodPq9mJmXZubJC2iyEtDpYJPe7Aw2qXtdD6wbEWtFxN0RcTrVLC6rR8QOEXFzREype3b9ACJibETcExE3AP/TtqOI2DsiTqsfD42ISyLijvpna+BkYJ26t/idut0REXFLRNwZEV9v2NdREfGPiPgD1XVa0puWwSZ1k4hYGtgRaLu4fT3gwvq+di8ARwPvzszNgFuBQ+tJd88B3g+8Axg2n93/APhTPd3TZsDfgC8D99e9xSMiYgdgBLAFsAkwJiK2iYgxwO7AplTB+bbFfOhSt3JKLan5ekfE7fXj64FzgVWBh9umZgK2AjYEbowIgGWBm4H1gQcz8z6AiLgI2K+d19gO+CRAZs4GnouIAfO02aH+ua1e7kcVdCsAl2TmzPo1Ll2ko5VazGCTmu/FzNykcUUdXi80rgKuysw95mm3CfW97xaDAL6ZmWfN8xpfXIyvIbWcQ5HSkmES8F8RsS5ARPSJiJHAPcDaDXdt3mM+z78aOKB+7lIR0R+YQdUba/N74FMN390Nj4ghwHXAhyKid0SsQDXsKb1pGWzSEiAznwT2BsZHxJ1UQbd+fQPL/YDL65NH5jd12ReAd0XEX4HJwEaZ+TTV0OZdEfGdzLwS+Alwc93uF8AKmTkFmADcDvySarhUetNySi1JUlHssUmSimKwSZKKYrBJkopisEmSimKwSZKKYrBJkopisEmSimKwSZKK8v99cwzuv1soTwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "preds,y,losses = learn_c.get_preds(with_loss=True)\n",
    "predictions = np.argmax(preds, axis = 1)\n",
    "\n",
    "interp = ClassificationInterpretation(learn_c, preds, y, losses)\n",
    "interp.plot_confusion_matrix()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 2318   248]\n",
      " [ 1324 19178]]\n",
      "accuracy global: 0.9318536500780302\n",
      "accuracy on negative reviews: 90.33515198752923\n",
      "accuracy on positive reviews: 93.54209345429713\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "cm = confusion_matrix(np.array(y), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>on z ur ▁c ▁xxmaj ord ▁cliff ▁xxmaj 39 : ▁7 po ▁trop ▁non ▁ma allegro ▁ ▁xxmaj . 4 . ▁50 :14 ▁14 ) za ez cat li ▁de ▁con ce viva ▁ allegro ▁( zo cher s ▁ ▁xxmaj ▁3. . 49 ▁ 17 : ▁9 to tenu s ▁so ante ▁and ▁xxmaj ▁2. . 48 ▁ 12 : ▁13 to ra ▁mode to mol ▁ ▁xxmaj ▁1.</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁m : \" ▁figaro ▁xxmaj ▁of ▁marriage ▁xxmaj the ▁\" overture ▁ ▁xxmaj ) ▁1968 , ▁york ▁xxmaj ▁new ▁xxmaj , ▁5 arch m ▁( ] 77 disc ▁[ ▁philharmonic ▁xxmaj ▁york ▁xxmaj ▁new ▁xxmaj ] ing play ▁[ ri ra ▁fer ▁xxmaj ▁= ▁w : \" ▁madonna ▁xxmaj ▁of wel je ▁\" interlude ▁ ▁- ) ▁1968 , ▁2 ary u br ▁fe ▁xxmaj known un ▁( ) ▁hall</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>▁\" ▁\\ ▁name ▁xxmaj ▁my ▁xxmaj ▁know ▁xxmaj you ▁\" ▁\\ ▁pour ▁visuel a ▁médi ▁autre ▁un ▁ou ▁télévision ▁la , ▁cinéma ▁le ▁pour ▁écrite ▁chanson ▁meilleure ▁: ▁2008 ▁awards ▁xxmaj ▁grammy ▁xxmaj d air ▁b ▁xxmaj ▁stuart ▁xxmaj ▁pour ▁dramatique film un ' ▁d ▁montage ▁meilleur ▁: ▁2007 ▁awards ▁xxmaj eddie ▁ ▁xxmaj our ▁park suite pour - ▁course ▁la ▁pour ▁scène ▁meilleure ▁: ▁2007 ▁awards ▁xxmaj empire ▁</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>) . ▁inconnu asin ▁ ▁xxup ▁numéro , ▁introuvable , ▁là ▁de ▁partir ▁à ▁ensuite z ▁recherche ▁le ▁vous ▁si , ▁mais asin ▁ ▁xxup ▁numéro ▁un ▁article ▁cet ▁pour indique ▁ ▁nous ▁on , ▁surcroît ▁par ▁xxmaj ! bin ja r k s ▁ ▁xxmaj ▁carrément , ▁aussi écrit ' s ▁ ▁cela , ▁à ▁échappé ▁même ▁quand ▁a ▁on ▁mais ... é ▁francis ▁non , bin ria ▁sc</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>. ▁morricone ▁xxmaj ennio ▁ ▁xxmaj ▁par ▁musique ▁en ▁mis , hara ▁sa ▁xxmaj ▁the ▁of ▁secret ▁xxmaj ▁the ▁xxmaj , ▁1988 ▁de ▁téléfilm un ' ▁d ▁thème ▁un ▁reprend ▁scène ▁même ▁cette ▁de ▁fin ▁la ▁à vient ▁inter ▁qui ▁lent ▁morceau ▁le , ▁enfin ▁xxmaj . ▁scott ▁xxmaj ▁ridley ▁xxmaj ▁de ▁préféré ▁films ▁des un ' ▁l , lou ou z ▁ ▁xxmaj ▁film ▁au ▁emprunté ▁est ouverture '</td>\n",
       "      <td>pos</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.show_results()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "neg tensor([0.9985, 0.0015])\n"
     ]
    }
   ],
   "source": [
    "# Trying out some random sentences I made up\n",
    "\n",
    "review = 'Ce produit est bizarre.'\n",
    "pred = learn_c.predict(review)\n",
    "print(pred[0], pred[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 18"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = awd_lstm_clas_config.copy()\n",
    "config['qrnn'] = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas = load_data(path, f'{lang}_textlist_class_sp15', bs=bs, num_workers=1)\n",
    "learn_c = text_classifier_learner(data_clas, AWD_LSTM, config=config, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c.load(f'{lang}clas_sp15', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9351), tensor(0.9624))"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds,targs = learn_c.get_preds(ordered=True)\n",
    "accuracy(preds,targs),f1(preds,targs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_bwd = load_data(path, f'{lang}_textlist_class_sp15_bwd', bs=bs, num_workers=1, backwards=True)\n",
    "learn_c_bwd = text_classifier_learner(data_clas_bwd, AWD_LSTM, config=config, drop_mult=0.5, metrics=[accuracy,f1]).to_fp16()\n",
    "learn_c_bwd.load(f'{lang}clas_sp15_bwd', purge=False);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9318), tensor(0.9606))"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds_b,targs_b = learn_c_bwd.get_preds(ordered=True)\n",
    "accuracy(preds_b,targs_b),f1(preds_b,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "preds_avg = (preds+preds_b)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(0.9370), tensor(0.9636))"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "accuracy(preds_avg,targs_b),f1(preds_avg,targs_b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "predictions = np.argmax(preds_avg, axis = 1)\n",
    "cm = confusion_matrix(np.array(targs_b), np.array(predictions))\n",
    "print(cm)\n",
    "\n",
    "## acc\n",
    "print(f'accuracy global: {(cm[0,0]+cm[1,1])/(cm[0,0]+cm[0,1]+cm[1,0]+cm[1,1])}')\n",
    "\n",
    "# acc neg, acc pos\n",
    "print(f'accuracy on negative reviews: {cm[0,0]/(cm[0,0]+cm[0,1])*100}') \n",
    "print(f'accuracy on positive reviews: {cm[1,1]/(cm[1,0]+cm[1,1])*100}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}