MovieReviewClassification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sentiment classification of the Standford Movie Review Database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will be using the handlabelled Standford [movie review database](https://www.cs.cornell.edu/people/pabo/movie-review-data/) (0) to build a sentiment classifier. Our work will highlight how to use [Jupyter Notebooks](https://jupyter.org/) with Python, [Scikit-learn](http://scikit-learn.org/), and [Pandas](http://pandas.pydata.org/) (including [Numpy](http://www.numpy.org/)) to build and crossvalidate a sentiment classifier. We will also throw in a bit of EDA with the help of [Matplotlib](http://matplotlib.org/) and [Seaborn](http://seaborn.pydata.org).\n",
    "\n",
    "Make sure you have downloaded the movie review data base into the same directory as this notebook under 'review_polarity' (leave the remaining directory file structure intact), and have all the necessary python modules (Numpy, Pandas, Seaborn, Scikit-learn, etc.) loaded.\n",
    "\n",
    "(0) Bo Pang and Lillian Lee, ``A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts'',  Proceedings of the ACL, 2004"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import os\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Provide file paths to positive and negative review files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pos_dir = './review_polarity/txt_sentoken/pos'\n",
    "neg_dir = './review_polarity/txt_sentoken/neg'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The original dataset broke up each review into sentences contained in one file for each review, likely for better feature construction. We want to explore simple sentiment classification, so we have to undo this process.\n",
    "\n",
    "For each directory, load file, take each line in file and put into one long string, combine it with the associated polarity tag. Do so for all files in the directory. Return a lsit of tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def reviews(polarity_dir, polarity):\n",
    "    file_list = os.listdir(polarity_dir)\n",
    "    print len(file_list)\n",
    "    review_bank = []\n",
    "    for fname in file_list:\n",
    "        f = open(os.path.join(polarity_dir, fname))\n",
    "        file_string = ''\n",
    "        for line in f:\n",
    "            line = line.strip('\\n')\n",
    "            file_string = file_string + ' ' + line\n",
    "        review_bank.append(file_string)\n",
    "    label_list = [polarity for f in file_list]\n",
    "    return zip(review_bank, label_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000\n",
      "1000\n"
     ]
    }
   ],
   "source": [
    "pos_reviews = reviews(pos_dir, 1)\n",
    "neg_reviews = reviews(neg_dir, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Dealing with Data: DataFrames\n",
    "Dataframes are convenient format for dealing with small to medium sized data (data which can fit into memory ~8GB). Pandas is a library which implements dataframes for python (largely inspired by the same concept in R). Dataframes are essentially spreadsheets, or objects with data and columns. The columns can hold different types of data, where each row is indexed. From this arrangement, one can do a lot of data munging. From selecting data, dealing with missing values (a.k.a., imputation; very important in the real world of noisey, messy data), simple statistics, time series, to more database-esque type functions: merging, and grouping.\n",
    "\n",
    "This piece of software is crucial for exploritory data analysis, and quick mockups of classifiers. It is widely used in the industry (check most data science job listings). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# place the parsed data into a dataframe\n",
    "review_array = np.concatenate((pos_reviews, neg_reviews), axis=0)\n",
    "review_df = pd.DataFrame(review_array, columns=['review', 'sentiment'])\n",
    "# the pandas lib. interprets the sentiment as a string. So we correct\n",
    "# this.\n",
    "review_df['sentiment'] = pd.to_numeric(review_df.sentiment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1995</th>\n",
       "      <td>if anything , \" stigmata \" should be taken as...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1996</th>\n",
       "      <td>john boorman's \" zardoz \" is a goofy cinemati...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1997</th>\n",
       "      <td>the kids in the hall are an acquired taste . ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>there was a time when john carpenter was a gr...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>two party guys bob their heads to haddaway's ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 review  sentiment\n",
       "1995   if anything , \" stigmata \" should be taken as...          0\n",
       "1996   john boorman's \" zardoz \" is a goofy cinemati...          0\n",
       "1997   the kids in the hall are an acquired taste . ...          0\n",
       "1998   there was a time when john carpenter was a gr...          0\n",
       "1999   two party guys bob their heads to haddaway's ...          0"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "review_df.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Employ a little Exporitory Data Analysis (EDA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                              review  sentiment  review_len\n",
      "0   films adapted from comic books have had plent...          1        4227\n",
      "1   every now and then a movie comes along from a...          1        4096\n",
      "2   you've got mail works alot better than it des...          1        2421\n",
      "3    \" jaws \" is a rare film that grabs your atte...          1        6092\n",
      "4   moviemaking is a lot like being the general m...          1        3898\n"
     ]
    }
   ],
   "source": [
    "review_df[\"review_len\"] = review_df[\"review\"].apply(lambda x: len(x))\n",
    "print review_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      ":0: FutureWarning: IPython widgets are experimental and may change in the future.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x11d0e5310>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgUAAAFkCAYAAACw3EhvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xl01fd95//n9/u9+6INEBiD2EwA2wlhaxYHdewMsTOd\n5ucO2IFQ/OuxG7vppDMdxkw0OXGM2/KD5MQ9p5PEcc8kNSmltmOnbhsnaSZAa1KcGNUE28BgbAxi\nR7t09+37/f3xlS7ISDIYSffq6vU4hxNL3+V+riLd+7rvz2Y4juMgIiIiE55Z6gaIiIhIeVAoEBER\nEUChQERERPooFIiIiAigUCAiIiJ9FApEREQEAM9wB23bZvPmzRw7dgyv18uWLVtoaGgoHt+zZw9P\nPPEEHo+H1atXc8899wx5TUtLC01NTZimyfz583n00UcxDIOdO3fywgsvYBgG999/P5/+9KdxHIfG\nxkZmz54NwJIlS9i4ceOo/iBEREQmPGcYP/vZz5ympibHcRzn4MGDzhe+8IXisWw266xatcrp7e11\nstmss3r1aqe9vX3Iax566CFn//79juM4zle/+lXn5z//udPR0eH8x//4H518Pu/E43HnN3/zNx3H\ncZyTJ086Dz300HBNExERkRE2bPfBgQMHWLlyJQCLFy/m0KFDxWPHjx+noaGBaDSK1+tl2bJlNDc3\nD3nNkSNHWLFiBQCNjY28/PLL1NXV8fd///dYlkVrayt+vx+Aw4cP09rayn333ceDDz7IiRMnRj4N\niYiIyADDdh/E43EikUjxa8uysG0b0zSJx+NEo9HisXA4TCwWG/SaQqGAc9nCiaFQiFgsVjy+c+dO\n/tf/+l/cd999ANTX1/PQQw9x55138uqrr7Jp0yaef/75IduZTqc5dOgQU6ZMwbKsa/wRiIiIjC+F\nQoG2tjZuvfVWAoHAiN132FAQiURIJBLFr/sDAUA0Gh1wLJFIUFVVNeg1lmUVr7v83H7r16/n3nvv\n5fOf/zyvvPIKixcvLr65L1u2jNbW1mGfxKFDh1i/fv3VPF8REZGKsXPnTpYvXz5i9xs2FCxdupR/\n/ud/5tOf/jQHDx5kwYIFxWNz586lpaWFnp4egsEgzc3NPPDAAxiGMeg1ixYtYv/+/fzGb/wGe/fu\n5WMf+xgnTpzg8ccf51vf+hYejwefz4dpmnz729+murqa3//93+fo0aNMnz592CcxZcoUwP3hTJs2\n7Xp/JiIiImXtwoULrF+/vvj+N1KGDQWrVq1i3759rF27FoCtW7fy4osvkkwmuffee2lqauKBBx7A\ntm3WrFlDfX39oNcANDU18cgjj5DL5Zg3bx533XUXhmGwcOFCPvvZz2IYBo2NjaxYsYKFCxeyadMm\nXnrpJTweT/EeQ+mvKkybNo0ZM2Zc9w9FRERkPBjpLnPDccb/Lolnzpzhk5/8JLt371YoEBGRijda\n73tavEhEREQAhQIRERHpo1AgIiIigEKBiIiI9Bl29oGIiMhYsW2beDw+oveMRCID1smR4SkUiIhI\nWYjH4/zjv54mEAyNyP3SqSSf+cTMAYvlyfAUCkREpGwEgiFC4eh7nzhCXnnlFf74j/+Ym266CcMw\nyGQy/PZv/za/+7u/e9X3+KM/+iO++c1v8uabbxKLxVi+fDkbN27ka1/7Gl6vdxRbP/IUCkREZMIy\nDIOPf/zjPP744wBks1nuuusu7r777gH7+Aznm9/8JgD/5//8H6ZMmcLy5cv58z//81Fr82hSR4uI\niExYjuMM2LAvHo9jWRZHjx5l3bp1bNiwgQceeIDz58+TyWT4gz/4AzZs2MCaNWvYt28fALfddhsX\nL17khRdeYPv27bz++uvccccdJBIJPvWpT5FOpwH43ve+x/bt2zl//jyf//zn2bBhA5///Oe5cOFC\nSZ77YFQpEBGRCe1Xv/oVGzZswDRNPB4PX/nKV9i6dStbtmxh4cKF7N69m61bt/Jf/st/obu7m+9+\n97t0dnZy4sQJwK02TJ06lf/0n/4TU6ZM4UMf+hAAXq+XT33qU/zTP/0Td999Nz/+8Y/5q7/6KzZv\n3syGDRtobGzkl7/8Jd/4xjf4xje+UcofQZFCgYiITGgf/ehHryj3f+UrX2HhwoUALF++nMcff5yb\nbrqJtWvX8t//+38nn8+zYcOGAdcMtmvAPffcw+bNm5k7dy5z5syhpqaGY8eO8Zd/+Zf87//9vwHK\natyBQoGIiJSNdCo5wvea9L6ura+v580332TBggU0NzczZ84cjh07RiKR4C//8i9pbW1l3bp1/Lt/\n9++K15imSaFQGHCfWbNm4TgO3/ve9/jc5z4HwLx587j//vtZsmQJx44d47XXXnu/T3HEKRSIiEhZ\niEQifOYTM0fwjpPec7CgYRgYhnHF9//sz/6MP/3TP8VxHDweD1u2bKG+vp5vfetb/PSnP8W2bf7r\nf/2vA6659dZb+frXv868efMG3HPNmjV885vf5CMf+QgA/+N//A82b95MNpslnU7zla98ZQSe68jQ\nLokiIiLjjHZJFBERkVGlUCAiIiKAQoGIiIj0USgQERERQLMPRESkTGiXxNJTKBARkbIQj8c5tu8k\noRHaJTGZSvKB22Zrl8RroPgkIiJlIxQMEQlHR+Tf1YSLV155heXLlw/Yf+Ab3/gGL7zwwnU/l2w2\ny3PPPQfACy+8wJ49e677nqNNoUBERCY0n8/H//yf/7P49WCLGb0fra2tPP/88wD8zu/8DnfccceI\n3Hc0qftAREQmLMMw+OhHP4rjOOzcuZP169cXj+3YsYMf//jHAPzWb/0WGzZsoKWlhaamJrxeLzfe\neCNnzpxhx44d/M3f/A0///nPSaVS1NbW8q1vfYsnn3ySt99+m29/+9s4jsPkyZM5efIkCxcu5O67\n76atrY2HHnqIv/u7v+Pxxx/n1VdfxbZtfu/3fo+77rqrJD8PVQpERGTC6l/U99FHH2X79u2cOnUK\ngFQqxU9/+lOefvppdu7cya5duzhx4gRf//rX+cIXvsBf//Vfs3TpUgzDwHEcuru72b59Oz/4wQ/I\n5/O88cYbfOELX+Cmm27iP//n/1x8vHvuuafYNfEP//APrF69mpdeeomzZ8/yt3/7t3z/+9/nySef\nJBaLjf0PA1UKREREqKmp4ctf/jJf+tKXWLp0KclkknPnznHfffcBEIvFaGlp4Z133mHp0qUALF26\nlH/8x3/EMAy8Xi8bN24kFApx8eJFCoXCFbsmGobBvHnzKBQKnDt3jp/+9Kd8//vf5+mnn+bw4cPF\nXRcLhQJnz54t7tI4lhQKRESkbCRHcJfEa73X7bffzs9//nNeeOEF/uAP/oCbbrqJ7373uwA89dRT\nLFiwgPnz53PgwAEaGxuLuxu++eab7N69mx/84AekUilWr16N4ziYpolt24BbkegPCWvWrOHrX/86\n8+fPJxKJMG/ePD7ykY/wJ3/yJ+TzeZ588klmzhzJjaGunkKBiIiUhUgkwgdumz3i9xzOu3dJ/PKX\nv8yvfvUrqqqq+NjHPsa6devIZDJ8+MMfZurUqWzatIkvf/nL/NVf/RXRaBSfz8esWbMIBoOsX7+e\n2tpabr75ZlpbW/nwhz9MLpfjG9/4BoFAoPg4d955J1u2bOE73/kOAHfccQf79+9n/fr1JJNJVq1a\nRTgcHtGfw9XSLokiIiJX6Uc/+hGLFy+moaGB5557joMHD7Jly5Yxb8dove+pUiDSZ7jV1LQqmogA\n3HDDDfy3//bfCAaDWJZVkkAwmhQKRPoMtZqaVkUTkX7Lly/nhz/8YambMWoUCkQu07+amojIRKR6\nqIiIiAAKBSIiItJH3Qcy4Qw1oDAWi2GP/8k4IiLvm0KBTDhDDShs62gjEtFgQhGZuBQKZEIabEBh\nIjn4dEQRkYli2FBg2zabN2/m2LFjeL1etmzZQkNDQ/H4nj17eOKJJ/B4PKxevZp77rlnyGv6d5Yy\nTZP58+fz6KOPYhgGO3fu5IUXXsAwDO6//34+/elPk06n2bRpE52dnYTDYbZt20ZdXd2o/zBEREQm\nsmEHGu7atYtcLsczzzzDww8/zLZt24rHcrkc27Zt46mnnmLHjh08++yzdHR0DHnN1q1b2bhxIzt3\n7sRxHHbv3k1nZyfPPPMMzz77LNu3b+drX/saAE8//TQLFixg586d3H333cWlIEVERGT0DBsKDhw4\nwMqVKwFYvHgxhw4dKh47fvw4DQ0NRKNRvF4vy5Yto7m5echrjhw5wooVKwBobGzk5Zdfpq6ujr//\n+7/HsixaW1vx+/3Fx21sbARg5cqV/PKXvxzhpy0yUKIzTdehDMnz2VI3RUSkZIYNBfF4fMBmEpZl\nFXd8isfjRKOX+mTD4TCxWGzQa969hWQoFCruFW1ZFjt37mTt2rV85jOfueJx++8rMlo6TvTyzr7z\n5GIOvccznDvUccWWpyIiE8GwoSASiZBIJIpf27ZdXP89Go0OOJZIJKiqqhr0GsuyBqwb339uv/Xr\n1/Ov//qvNDc388orrxCJRIpTxt59rshISvVkOPdGB5bXpOomL56QScc7vXSc6C1100RExtywoWDp\n0qXs3bsXgIMHD7JgwYLisblz59LS0kJPTw/ZbJbm5maWLFky5DWLFi1i//79AOzdu5fly5dz4sQJ\nvvjFLwLg8Xjw+XyYpjngHv3nioyGzpNuFWrGh6cQmGJR98EghmXQfrwHx1a1QEQmlmFnH6xatYp9\n+/axdu1awB0s+OKLL5JMJrn33ntpamrigQcewLZt1qxZQ319/aDXADQ1NfHII4+Qy+WYN28ed911\nF4ZhsHDhQj772c9iGAaNjY2sWLGCD37wg3zpS1/ic5/7HD6fj8cff3yUfwwyERXyNt1n4niDFtGp\nQVLtPZhek9qZETpPxug5n6DmxuH3YhcRqSSGUwGdp6O1r7RUpt7eXs4caCXT5nDu9Q6mLqyh/gO1\nXGw7j2l5qfJVc2zPGYLVPuY1TieRjDNjab26sUSkbIzW+572PpAJq/NkDAyobRi4iJE/4qXqhhCp\nnizJzkyJWiciMvYUCmRCyqcLpHuzRKcE8Qau7EWrm+UGhZ7ziSuOiYhUKoUCmZDSXTkAwlOCgx4P\nTw5iekx6zyc0PVFEJgyFApmQUt1uKIhMCgx63DQNqqYGyaUKZGOFsWyaiEjJKBTIhJTuymF6DALV\nviHPqbohDECiTeMKRGRiUCiQCScTz5FP2YQnBTAMY8jzovVBDNMg2aalj0VkYlAokAmn53QSgPAQ\nXQf9TI9JtD5ILlHQLAQRmRAUCmTC6TnlzigITxp8kOHlolNDAHS9Ex/VNomIlAOFAplwes4mMSyD\n4DDjCfpF693g0HlCoUBEKp9CgUwo+WyBVEcWf9TCMIceT9DPG/Tgi1j0nE5SyGoWgohUNoUCmVBi\nF93xBL7IsNt+DBCc5MMpOLRr50QRqXAKBTKhxC70hYLotYUCgLa3ukelTSIi5eLqXxlFxhnbtonH\nB44FaD/lvrF7wtZV3ydQ7cHymbQe68ZxnGGnMYqIjGcKBVKx4vE4x/adJBQMFb/XeSIGODi+qx8f\nYJgGNbPDdByLkWhPExliaWQRkfFO3QdS0ULBEJFwlEg4SjgUIRcvYAXNqxpkeLm6uRFAXQgiUtkU\nCmTCyCbz2AUHT/jay/+1c9xQ0KpQICIVTKFAJox0j7tcsSd07b/2/oiXqmkhOk/2amqiiFQshQKZ\nMNK9faHgfVQKAKbMr8HOa2qiiFQuhQKZMFJ9ocD7PioFAFM+UANoXIGIVC6FApkwMrEsls/E9L2/\nSkHtzAgev1WcmigiUmkUCmRCsAsO2WQef8T7vu9hWiZT5teQ6soUV0YUEakkCgUyIWSTOXC4rlAA\nMO3mOgAuHOkciWaJiJQVhQKZEDLxHHD9oaD+AzWYHoMLR7pGolkiImVFoUAmhJEKBR6/xeR5NcQu\nJkl0pEaiaSIiZUOhQCaES6HAd933mnZzLQAXDqsLQUQqi0KBTAiZeA4M8IWuf7uPqQtrMUw4r3EF\nIlJhFAqk4jmOQyaewx/2XvOeB4PxhbxMnldNz9kE8TZ1IYhI5VAokIpXyNrYOfu6xxNc7sbFUwA4\n+1r7iN1TRKTUFAqk4o3UIMPLTVtUi+UzOftaO46thYxEpDIoFEjFy8Td5Y19IxgKLJ/FDbfUkerO\n0HUqNmL3FREppesfdSVS5q63UmDbNrHYlW/8NfNDnPk1nHmtnbrZVdfVRhGRcqBQIBUvk8gD7z8U\npNJJupt7qKupHfD9RDKBL+Lh/KEObvkPs7G8KryJyPimUCAVL5vIYXrM63rTDgZDRMJRAGzHIZPJ\nYFoeam7y0HowRsvBc0xecKlaEIlEME2FBBEZXxQKpKI5zqWNkAzj+qcjAmQyGV57u5NsNkMuBPXA\na784T7zVXfo4nUpxz7//ADU1NSPyeCIiY0UfZaSiFbIOTsEZkUWLLuf1+rANG6cnje038HYVCLQW\nCHTYWBeSJBKJEX08EZGxoFAgFS2fKgDgC4/czIPLBfwhzLoQBhBKewgFIwT8oVF5LBGR0Tbsxyfb\nttm8eTPHjh3D6/WyZcsWGhoaisf37NnDE088gcfjYfXq1dxzzz1DXtPS0kJTUxOmaTJ//nweffRR\nDMNg+/bt/OQnPwGgsbGRL37xiziOQ2NjI7NnzwZgyZIlbNy4cfR+ClKxcv2hYIQrBQPU+OF8Anoy\nMDk4eo8jIjLKhn2l3LVrF7lcjmeeeYbXXnuNbdu28cQTTwCQy+XYtm0bP/zhDwkEAqxbt4477riD\nV199ddBrtm7dysaNG1mxYgWPPvoou3fvZsGCBfzoRz/i+eefxzAM1q1bx6pVqwgEAtxyyy08+eST\nY/JDkMrVXynwj1KlAACvBSEPJHKQs0fvcURERtmw3QcHDhxg5cqVACxevJhDhw4Vjx0/fpyGhgai\n0Sher5dly5bR3Nw85DVHjhxhxYoVgFsRePnll7nhhhv47ne/WxwAls/nCQQCHD58mNbWVu677z4e\nfPBBTpw4MfLPXCaEfMp9k34/lYJUzubVsyn+8S3Y+Uaet9qzQ59c7Xf/tzfzfpopIlIWhn2ljMfj\nRCKR4teWZWHbNqZpEo/HiUajxWPhcJhYLDboNYVCAce5tBRsKBQiFovh8Xiora3FcRy+/vWvc/PN\nNzNr1iza2tp46KGHuPPOO3n11VfZtGkTzz///Eg+b5kgcqkCGOANXlso6E4V2PYv7XSl+j/5O/zf\nvR18ZGaQ314wyPbL1Zd1IdRdf7tFREph2FfKSCQyYBR1fyAAiEajA44lEgmqqqoGvcayrAFztvvP\nBXd615e//GUikQibN28G4NZbb8WyLACWLVtGa2vrdT5NmajyqQK+oOeadkdM5Qp8c18HXSmbTzT4\nmRuJYxomu077eOV0igu9Gf799HcV2XwWBD0Qz2FUj8zURxGRsTZs98HSpUvZu3cvAAcPHmTBggXF\nY3PnzqWlpYWenh6y2SzNzc0sWbJkyGsWLVrE/v37Adi7dy/Lly/HcRz+8A//kIULF/LYY48VuxG+\n/e1v8/3vfx+Ao0ePMn369BF+2jIRFHI2hey1TUe0HYfv7e/iTG+BhTUGiyIFkt1x0r1J7pphMLfK\noKXH5t8u5q+8uK8LwaPdlEVknBr21XLVqlXs27ePtWvXArB161ZefPFFkskk9957L01NTTzwwAPY\nts2aNWuor68f9BqApqYmHnnkEXK5HPPmzePOO+9k165dNDc3k8vlikHi4Ycf5sEHH2TTpk289NJL\neDye4j1ErkW6u28jpGsYZPjaRYc3LuaZHja4fU4YyzDw+v14DA+BQJBPznFoPRzjtU6DOp/JjMtn\nH1b54EICT1q7JorI+GQ4l3f2j1Nnzpzhk5/8JLt372bGjBmlbo6UiRMHznLkhdNMW1TLlPmXVhe8\n2HYe0/IypW7ygPPPt57nW69CVxo+e5OHuqg7vbCj6yIew0N1zSQATrb18OPTJgHL5u6ZeSbVud/H\nceD/dmLbNvPX1Qxa4dLyxyIyEkbrfU/LHEvFutZKwaE2aE/Bx2b6iPiGzspTAvDBWni9y+StmMmk\n/oGFhgERL2Z3htOvtGPPGvjnlUwl+cBts4vjaUREyo0+skjFShVDwXtnX9txeOm0+wfxyXmB9zz/\n1jrwGA6Huy3y9mUBIuLOTLAyPiLh6IB/oaBWOhSR8qZQIBUr3Z0DwBe6slLgOA6pdLr475WTMdqS\n8MEpNhEzx3t1qgUsgw9U5UkVDA615y4d6NueOd877nvlRGQCUveBVKx0TxbTawy6ZXI2k+G18zm8\nXveT/c9OuLMJbrSSHHonh8/33tWChdU5jvV6OHAhy62TvXhMA3wWecuGmIljO9c0FVJEpNRUKZCK\n5NgO6Z4cnqA15Dlerw+fP0DG8HE+6VAfKDAp5MPrvboxCH4LFlbbJPMORzsvVQuyPhtsSHZrdUMR\nGV8UCqQipWNZnIKDN/jev+Jv9r2hz44MsvbAe/hAVQEDOHJZF0LW666CmOxUKBCR8UWhQCpS/xvy\ncJUCcMcWvNmZxzRgZqhwzY8T9sCsaovWpE1r0r0+1xcKUqoUiMg4o1AgFSnZmQbA+x6hoD1l05W2\nmVPtwTf8qUO6ZbI7LqG/WlAwHQyPug9EZPxRKJCKlOgLBQWrMGCWQSqdJpXJkslmcZxLXQcL6t7/\nmNtZVRYRr8GbnTlyBQcMsEIGuWSefObaqw8iIqWi2QdSkXpb4wC0dKa5mIsPONbZlSCXzlA3Kcix\nLgO/BbOqPHT3vL/HMg2DRZO8NF/I8lZXnjrAEzbI9zqkujNEp2p9AhEZH1QpkIqU7sniGOAJ+fH5\nAwP+ef1+PF4vbWlI5hzm1niwrnPq4M2T3RkL/bMQPGH3fupCEJHxRKFAKlKqO4ftN9ylh4dwum+H\n79nV118wi/pMpkcszsULJAsGVl8o0GBDERlPFAqk4uTSefKpAoXA8L/epxO4sw6iI9OLNr/Wvc+Z\nlA/Ta+ANWqS6MlTAnmMiMkEoFEjFSXa5n85t/9BVgmTBpDMDN0YsfNbIrDo4r8aDgRsKAII1fvJZ\nm1zq2tc/EBEpBYUCqTj90xHtYSoFFzLuGICR6DroF/KazIhadOY8dKXdUACQ6smO2GOIiIwmhQKp\nOP2VgsIwlYLzGffT/EiGArjUhXCkwyBQ5T5GJpYb7hIRkbKhUCAV570qBXkbWrNeanxQ7R/ZP4G5\nNV4MHA53GASibjUiHVOlQETGB4UCqTjvVSm4mDaxMZgZHvnHDngMpvpzXEwadBfAtAzSvQoFIjI+\nKBRIxUl2ZvCGLBhiAOGFlLue8YxRCAUANwbc7oLXLmTwV/nIxHPYtmYgiEj5UyiQimIX3FUEA9W+\nIc+5mLIwcZgSGJ023BDIAQ6vnc+4XQgOZOMaVyAi5U+hQCpKujeDYzsEagYPBcmcTU/OZLIvh+c6\nVzEcSsBymBGBt9uzGCGNKxCR8UOhQCpK/3iCQI130ONnYu4GRfW+0f3kvqDOwQHOZN1uA40rEJHx\nQKFAKkqy0w0FwSEqBWMWCmrdMPB6j/t4mpYoIuOBQoFUlGSXOx0xUD14peB0LI/XdKjxjO6WxpOC\nMC1q8Vp7FstnqlIgIuOCQoFUlP5KwWBjCnoyNrGsw9RAYbh9kkbMh28IkCsY5P0essk8dkEzEESk\nvCkUSEVJdqUxLQNf5MqVCs/E3D0IpgbtMWnLh25wpzd02G4CySVGtzohInK9FAqkoiQ7MwRr/RiD\nlAL6xxNMC4zNm/OcOi9hn8GJlBtCckmFAhEpbwoFUjFyqTy5VJ5Q7ZULEDiOw9lYgZDHIOodmzK+\naRjcXO/nfN8YQ4UCESl3CgVSMfqnI4bq/Fcc68k4JPMO0yPWmIwn6HfrND9dTl/3gUKBiJQ5hQKp\nGP0zD0J1V1YKzvX1598Qsca0TTfX+4ljUEChQETKn0KBjHu2bdPb20vX+R4ADL9NLBYjnUyA43YV\nnI+7gwynj3EoqApYNNT46HYMcskCjqMZCCJSvhQKZNyLx+Mc23eS9rd7Aeg5H+PCoW5yLZ3ksu76\nAOfiBXwmTAqO/q+8g0MmmyWVTpNKp1k42aLbMXAKDl2tPdj22Mx+EBG5VlfO2xIZh0LBEIlsAoDq\nSdVk81n8/iAAiZxNT8ZhVpWFOQYDCgq5HMdO5enOxAEIFGxacR/3pX3n+Z2pNVRVVY16O0RErpUq\nBVIxsokcHp+J5Rn4a30+PvbjCSyPF58/gM8f4MaaIHHDbVPAHnylRRGRcqBQIBXBsR2yqTze8JVv\nuuf6QsFYjyfoZxoGgbBblMvH1HUgIuVLoUAqQj5jgwO+0JU9YufiBSwDpoZKEwoA6vr2YrDjCgUi\nUr6GHVNg2zabN2/m2LFjeL1etmzZQkNDQ/H4nj17eOKJJ/B4PKxevZp77rlnyGtaWlpoamrCNE3m\nz5/Po48+imEYbN++nZ/85CcANDY28sUvfpF0Os2mTZvo7OwkHA6zbds26urqRvcnIeNaPuVWA/zv\nqhRkbWhP2UyPWFjmGC5Q8C4zar2kz4Eno9kHIlK+hq0U7Nq1i1wuxzPPPMPDDz/Mtm3bisdyuRzb\ntm3jqaeeYseOHTz77LN0dHQMec3WrVvZuHEjO3fuxHEcdu/ezenTp/nRj37Es88+yw9+8AP27dvH\nm2++ydNPP82CBQvYuXMnd999N9/5zndG96cg416+bynhd1cK2tNuELghXLoqAUDQYxI3DMK2Q0J7\nIIhImRo2FBw4cICVK1cCsHjxYg4dOlQ8dvz4cRoaGohGo3i9XpYtW0Zzc/OQ1xw5coQVK1YAbkXg\n5Zdf5oYbbuC73/1ucZ36fD6P3+/nwIEDNDY2ArBy5Up++ctfjvDTlkqT66sU+MIDQ0Fbxv3dmlbi\nUABgew0sA9442lvqpoiIDGrYUBCPx4lEIsWvLcsqzrGOx+NEo9HisXA4TCwWG/SaQmHgoi2hUIhY\nLIbH46G2thbHcfja177GzTffzOzZswfco/++IsPp7z7whQZ2H7Sn3V/xqeHSD5/xBdyA8tbxZIlb\nIiIyuGFsQv7zAAAgAElEQVRfKSORCIlEovi1bduYpntJNBodcCyRSFBVVTXoNZZlFa+7/FyATCbD\nww8/TDKZZPPmzcXHjcfjV5wrMpRc2sYwwRO4VBFwHGjPGFT7DULe0oeCYN/CSa1nU1rZUETK0rCv\nlEuXLmXv3r0AHDx4kAULFhSPzZ07l5aWFnp6eshmszQ3N7NkyZIhr1m0aBH79+8HYO/evSxfvhzH\ncfjDP/xDFi5cyGOPPVbsRrj8Hv3nigwnnyzgC3kHbJkcy5tkbaMsug4AHJ/bNm+2wMkL6RK3RkTk\nSsPOPli1ahX79u1j7dq1gDtY8MUXXySZTHLvvffS1NTEAw88gG3brFmzhvr6+kGvAWhqauKRRx4h\nl8sxb9487rzzTnbt2kVzczO5XK4YAh5++GHWrVvHl770JT73uc/h8/l4/PHHR/NnIONcPl3Azjt4\n3zXIsCPnfl0uocDuq1ZU4XDg7RhzbgiWuEUiIgMNGwoMw+Cxxx4b8L05c+YU//v222/n9ttvf89r\nAGbPns2OHTsGfG/VqlW8/vrrgz72X/zFXwzfcpE+6R53f4N3zzzozJZXKMAC24Rqx+Hg2zFWr6wv\ndYtERAYofUeryHVKdbmhwB8ZOMiwI+vBYzhjsgnSVTEM7IBJjeHwxjsxsnktZCQi5aVMXi1F3r/+\nUOC7bOGiVM6hN28yye+MySZIV8v2G3gAM+/wf1sS73m+iMhYUiiQca9YKbgsFJzqzgMGUwLlNcq/\nEHD/5Kpx+PXbmmorIuVFoUDGvVRnFoyBYwpOducBmOwvrxK97XerFrWWw4G3FApEpLwoFMi4l+rO\n4gmYGJftbXCqx13MaJK/3CoFbhvnVXt4+1yKWDJf4haJiFyiUCDjWi6VJ58q4H3XDoine/IETZtB\nNk0sKbuv+2B60MRx4ODxeIlbJCJyiUKBjGuJDncRoMtDQVeqQG/GodZXfp/CbZ8BhrtWAaBxBSJS\nVsrsc5TItSmGguClUNDSlQOg1lsAymSNgj42Dr6oRSGWIeT38+qxHnp7qwecE4lEBiwLLiIyVhQK\nZFxLdKSAgZWCk8VQkKfcQkE2nSLmGPiTDtNqDd5py/P8L1qp6mt/OpXkM5+Yqf0+RKQk9HFExrX+\nSoEnNEilwFcoSZveU99OjvPrAgC0xz2EwlFC4SiBYKiULRORCU6hQMa1REcawzLw9A3gcxyHlu4s\nk0ImfrO8Zh70u3ywIcCp1kwpmyMiUqRQIOOW4zgkOtIEai7tjtieLJDIOjRUl1e3AbjbiKdTCVKG\nu9iSN5UiEjA43ZomXyjTqoaITCgKBTJuZZN58ukCwVpf8Xv9XQczq8tvuEwmm8K8kMZMuLMijPYU\n0wMO2bzD2Yu9JW6diIhCgYxjiTZ3kGGozl/8Xv8gw3KsFAD4fQF8YXfcgLdgMqfWHVdwtqP8pk+K\nyMSjUCDjVrzdDQXBy0JBS1cOA5hRhpWCItMAjwHZAjOibnhRKBCRcqBQIONWvM2deRCqc7sPbMfh\nVHeOaVEPAU/57Iw4KJ8FWZugZTDJD63dBfKF8hwYKSITh0KBjFvvrhRcjOVJ5x1m1XqHu6w8+Pq6\nN7I208JgO3ChU7MQRKS0FApk3Eq0pfCFvcXVDFu63fEE4ysUFLihb2mCM+0KBSJSWgoFMi4VcjbJ\n7gyRKYHi9/oHGc6qGQehwH8pFEztCwVn2xQKRKS0FApkXEp2psGByORg8XstXTlMA2aOh1DQXynI\nFPBbBpOiJhe6shpXICIlpVAg41K8bzpieIobCgq2w+nuHDdWefBZZT7IEAZ0HwDcUOfBtqG1R4sY\niUjpKBTIuNQ/yDAy2e0+uBi3ydnjZDwBuFMSTS6Fglp3CuWFTk1NFJHSUSiQcal/OmKkr1JwqtcG\nYNZlqxuWNcPom5ZYAMdhal8oOKdQICIlpFAg41KiPYXpMQhWu9MRT/eV3WePh/EE/XwW2EAB/F6D\nKTVe2noKZHJ2qVsmIhOUQoGMO47tEG9PEZ4UxDDd8QNnem08Jkwv55UM361vXIHZFwJunOzHduCt\nc5qFICKloVAg406qJ0MhaxOtd7sOsnmbczGbGdVePOY4GGTYr29aopFzZxxMn+RWPd48nS5Zk0Rk\nYlMokHEn1to3yLDeneB/qjWL7YyjQYb9ipWC/lDgjoc4ekahQERKQ6FAxp34xSQA0alupeDExSww\nThYtuty7KgVBv0V12OTY2TQFrVcgIiWgUCDjTn+lINpXKThxwe2DbxhvocDr/vkZlw0snFpjkc46\nnLiQKlWrRGQCUyiQccG2bXp7e+nt7aX7fBzTY5C3MvT29nL8XArLgBuqxtEgQyhOS+zvPgCYVuM+\nh8MnE6VqlYhMYOPsVVQmqng8zrF9JwkGgiTb03jDFmcPtrkrGbblmBo2xtcgw34+EyNegLxbLehf\nr+DQyTj/z21TStkyEZmAVCmQcSMUDOEjiGNDqDpIJBwlZgcpOAY3RMdhIIBL4wpS7joL0aBBbcTi\ncEsCx9G4AhEZWwoFMq6kY+6gwkDUHT9wum+75OmRcRoKfANDgWEYLJgRoCuW53xntpQtE5EJSKFA\nxpV0r/tG6a9yp++dqpBQYKYubYS0cKa7n4PGFYjIWFMokHElE3NDQCB6KRQYwNTxGgre1X0AsHCG\nGwqOtMRL0iQRmbgUCmRcSceymB4Db9DCdhxO9+SZHGJ8bJc8GN+VoaCh3offa3L0dLJUrRKRCWrY\n2Qe2bbN582aOHTuG1+tly5YtNDQ0FI/v2bOHJ554Ao/Hw+rVq7nnnnuGvKalpYWmpiZM02T+/Pk8\n+uijGIb7Qt7Z2cm6dev40Y9+hM/nw3EcGhsbmT17NgBLlixh48aNo/dTkHHBsR0y8RzBGj+GYdAa\ny5PJO0yvK3XLroNpYHvASF7aHdEyDebPCHL4ZIJkpkCor5ogIjLahq0U7Nq1i1wuxzPPPMPDDz/M\ntm3bisdyuRzbtm3jqaeeYseOHTz77LN0dHQMec3WrVvZuHEjO3fuxHEcdu/eDcAvfvEL7r//fjo6\nOor3PnXqFLfccgs7duxgx44dCgQCQC5ZAOfSIMNTPW5XwrRwKVt1/RyviZGxcS5bxXDhzDCOA8fO\nqFogImNn2FBw4MABVq5cCcDixYs5dOhQ8djx48dpaGggGo3i9XpZtmwZzc3NQ15z5MgRVqxYAUBj\nYyMvv/wyAJZlsX37dqqqqor3Pnz4MK2trdx33308+OCDnDhxYgSfsoxX2YRbYg9EfdiOwzvt7qp/\nk/w5Muk0qcv+pdNpxsuEPsdrYACkcsXvLWxwV2s8ekqhQETGzrDdB/F4nEgkUvzasixs28Y0TeLx\nONFotHgsHA4Ti8UGvaZQKAyYcx0KhYjFYgB8/OMfv+Jx6+vreeihh7jzzjt59dVX2bRpE88///z7\nf5ZSEXJxt8Tuj/rIZDIcvZAGDOLdCd5KWFyMXRqYl0zEsfP5Ie5UXmxv33iIRA76/nQWznTLH2+e\n1gwEERk7w4aCSCRCInHpRak/EABEo9EBxxKJBFVVVYNeY1lW8brLzx3KrbfeimW5/ajLli2jtbX1\nGp+WVKJipaDKS87J0ZExqPIZhIM+PIYHnz9QPDeXzZSqmdfM6Q8FyRxE3N/7SVVeplR7OXo6ieM4\nxfE3IiKjadjug6VLl7J3714ADh48yIIFC4rH5s6dS0tLCz09PWSzWZqbm1myZMmQ1yxatIj9+/cD\nsHfvXpYvXz7k437729/m+9//PgBHjx5l+vTp1/EUpVLkEgUsr4nHb9GddsgUYEpo/A/Cc/o2RiKR\nG/D9hQ1huuN5LnZpESMRGRvDVgpWrVrFvn37WLt2LeAOFnzxxRdJJpPce++9NDU18cADD2DbNmvW\nrKG+vn7QawCampp45JFHyOVyzJs3j7vuumvAY13+SejBBx9k06ZNvPTSS3g8nuI9ZOKy8za5ZIFQ\nnTvz4HSP2zUwJTT+Z9Ve3n1g275i19rsepNfAL9+s4PbbrnUJReJRAZU3kRERsqwocAwDB577LEB\n35szZ07xv2+//XZuv/3297wGYPbs2ezYsWPIx+qfjQBu18STTz45fMtlQkl29C1v3LeS4dletyth\nStACe8jLxgfLwPEYGMkcmXSSf3qlm+raLF0xN/jsPthNIu12h6RTST7ziZnDdr+JiLxf2iVRxoVk\nexq4tJLhmf5QEDJJVcDCf07QwkjkwfESCIYJhaPMDDiYRoL2GITC0fe+iYjIdVINUsaFRIf7Sdnf\nt0bBmd48YQ+EvJXxK2wHLbAdzOylWToey2BytZe2niyFwniZYCki41llvKJKxUu2u6EgEPXRmy7Q\nk3aYFKicEflO0C3amemBb/7T6nzYNrT1aLChiIw+hQIZFxLtGSyfgcdvFXdGnByspFDgzqKwMgMH\nSEyr8wNwQdsoi8gYUCiQspfPFMj05PCG3U/T/TMPJldUpaBvC+V3Vwpq3TEUFzQtUUTGgEKBlL14\nm7ucsS/svnEWKwUVFArsvvUWrPTASkFV2CLgM7moSoGIjAGFAil7sVZ3/X9vxK0UnOrOEfYahL2l\nbNUI85ngMTFTA0OBYRhMq/XRmyyQTBeGuFhEZGQoFEjZi128VClI5WzaEwVurLIqa+lfw4CIFyvt\ngHPlYENQF4KIjD6FAil78b5KgS9icbZvPMGNVeN/eeMrhL0YDpjvGmw4tT8UqAtBREaZQoGUvVhr\nCl/Ug+kxOd3jjieoyFAQcd/8392FMLVvsKHGFYjIaFMokLKWS+VJ92YJT3an5p3u7g8FFbgYZ8Qd\nJGGlBo4d8HtN6qIeLnZnsR0tYiQio0ehQMparNUdTxCa1BcKevJ4TKgPV+CvbnjwSgG44wpyeYfu\n+Hjf6EFEylkFvrJKJemfeRCeEqBgO5zrzTG9yotlVtAgw3590ynM9JVv/P1dCK09moEgIqNHoUDK\nWvyySkFrwiZvw8zqCuw6AAyPScFnXNF9AJdWNmzrG2gpIjIaFAqkrMUuupWC0CQ/52LuJ+gZNZW0\nQMFAdtDAzDo4+YHVgroqD16PQWu3KgUiMnoUCqSsxVpTBGv9WD6zGApmVlduKCgE+v4k4wNnGpiG\nwdQaH90Jm+Qg3QsiIiNBoUDKViqWIZvIEazzEovFinseTPIXSKfT717jpyIUgn1/koncFcf61yt4\n+3x6LJskIhOIQoGUrbaWTgAu5m12/7qDMz02US+cOJfg0DvdFPKV179erBTErlyToH9lw7fOZcay\nSSIygSgUSNlKtrtvft66KI4VJmMbTAl78PkDeL2V2YVQCPXNqhgsFPTNQHj7rEKBiIwOhQIpW4m+\nUEDUR0fMHWA3JVjZv7KO18D2GIOGglDAIho0eOtcGqcS+05EpOQq+xVWxrVkexoHIOKlsy8UTA5W\n4PLGlzMM7KAJiRxO4coBhVOqPcRSNue15LGIjAKFAilLjuOQbM/ghCwMyyxWCiaHKv9XthDqCz7x\nKwcbTql2jx09lRjLJonIBFH5r7AyLmViOfIZGyfsLlTUGbPxmRDxVuBKhu9SDAWDdCHU94eC08mx\nbJKITBAKBVKW+pc3tsMesjmb3qTNpAAYRuWHAjs09AyESVUWHkuVAhEZHQoFUpb6N0Jywh46et0y\nel2glC0aO4Xg0JUCyzSYM9XPO+dTZHJaxEhERpZCgZSl/j0P7LCHtp6+UOAvZYvGjuMzwWdCbPCp\nhzdN91Ow4e2z6kIQkZGlUCBlKd6WBAOckEV7z8SqFAAQ9UMyf8UeCADzb3R/EG9qXIGIjDCFAik7\njuP07XngA9OgrSeLaUDNBKkUABB1Fyp69x4IAPOnuz8IDTYUkZGmUCBlJxPPkU8XCE3yY9sOHT05\naiIm1gQYZFjUHwp6rwwFU6o91EY8GmwoIiNOoUDKTrzNHU8QmuSnN2lTsGFStMIXLXq36r6ySM+V\n4woMw2BhQ4i2nhztPVrESERGjkKBlJ3+QYahSf7iokV1Ey0UVPnAYNBQALBgZhjQuAIRGVkKBVJW\nbNum82wPAE4wz8VOd5vgiC9LRe6VjPucU8k4yUSMdDJBKhknlU5ghzw4PWnsQuGKaxbODAEaVyAi\nI8tT6gaIXC4ej9P+jhsKes4m6O4sACbBtjZyoSiEStu+0ZDJpsifyBKosvHFcxiGjZPoxTYdPDak\nO3oJ1tcOuGb+jBCmAUdPa1yBiIwcVQqk7BRSDr6Qh2g0SnfOJOoziAaDpW7WqAr4Q4SCEYKBMMFA\nmFAwgifqTj00Y1fugRDyW8yaGuCtM0nyhcqsoIjI2FMokLKSS+axcw7+qJdYxiaVh0kVvl3ykIJe\nAIxYftDDi2aFyeQcLWIkIiNm2Fdb27b56le/ytq1a9mwYQOnTp0acHzPnj2sWbOGtWvX8txzzw17\nTUtLC+vWrWP9+vVs3rx5wH7wnZ2d3HnnnWSz7kjqdDrNH/3RH7F+/XoefPBBOjs7R/RJS/lKdroD\n6/xRH+d6J8h2yUMJeHAYvFIA8ME5EQDeOBEfw0aJSCUbNhTs2rWLXC7HM888w8MPP8y2bduKx3K5\nHNu2beOpp55ix44dPPvss3R0dAx5zdatW9m4cSM7d+7EcRx2794NwC9+8Qvuv/9+Ojo6ivd++umn\nWbBgATt37uTuu+/mO9/5zmg8dylDyXY3GAYiXs72b5c8USsFloHjNTBj+QEhut+lUKBxBSIyMoZ9\ntT1w4AArV64EYPHixRw6dKh47Pjx4zQ0NBCNRvF6vSxbtozm5uYhrzly5AgrVqwAoLGxkZdffhkA\ny7LYvn07VVVVAx63sbERgJUrV/LLX/5ypJ6vlLlkx5WVgkkTtVIA2AETo+BA8spqwaQqL9Mn+Th8\nMk7B1rgCEbl+w4aCeDxOJBIpfm1ZFrZtF49Fo9HisXA4TCwWG/SaQqEw4JNOKBQiFosB8PGPf5ya\nmpohH7f/vjIxXAoFbqXAY0C1fwKtZPgutr/vT7Rr8PUKPjgnQjJjc+J8agxbJSKVathQEIlESCQu\nlSZt28Y03Uui0eiAY4lEgqqqqkGvsSyreN3l5w73uPF4/KrOlcqS7Mhg+U1sw+BivEBdwMCcSMsb\nv4vd33XSOfibvsYViMhIGjYULF26lL179wJw8OBBFixYUDw2d+5cWlpa6OnpIZvN0tzczJIlS4a8\nZtGiRezfvx+AvXv3snz58qt63Pc6VypHLp0nG8/jC1tciOWxHagLTNxAAGD7DRwT6EoPevxWhQIR\nGUHDLl60atUq9u3bx9q1awF3sOCLL75IMpnk3nvvpampiQceeADbtlmzZg319fWDXgPQ1NTEI488\nQi6XY968edx1110DHsu47NPgunXr+NKXvsTnPvc5fD4fjz/++Ig+aSlP/XseeMMWZ/q2S540kbZL\nHoxhYFd5sbqzOLkrt1GeWuujvsbLoRMJbNvBNCd2iBKR6zNsKDAMg8cee2zA9+bMmVP879tvv53b\nb7/9Pa8BmD17Njt27BjysfpnIwAEAgH+4i/+YviWS8Xp3/PAG/Zwpsedmz9pglcKAOxqL1Z3DrrT\n2AH7ijE2C2f42XsozpF3Omio9xGJRAZ014mIXC0tcyxlo79S4AtbnLnoVgrqJvAgw352tQ9IQmea\nTF2Gf3qlm+raS7sjOo47S+P5f23npvoCn/nETI3DEZH3RaFASsK27eJg0n5d591PwGbQ4ExPjkkh\nE5+lUGBXuSsb0pmCOpNAMEwofGnmz7wZeX5xOMX5bodbZ1Xg5hAiMmYUCqQk4vE4x/adJBS89CYW\nO5fE8Dh0ZfLEszZzar0lbGEZ8ZkQ9kJ3Gpwr94CIhjzURT2cbcuQL/hL0EARqRTqeJSSCQVDRMJR\nIuEoIX+YfNrGE7K4EHfXtJgenbiLFl1hUhDyDlbiysGGALOmBijYcKFr8H0SRESuhkKBlIVM3B1D\n4AkanO8LBTdWKRQUTXYrBN6ewqCHZ01zp2mcblcoEJH3T6FAykK6b9MfK2RwIdFXKVAouKQ/FPQO\nHgqm1/nxWgZnFQpE5DooFEhZyMTd0fSeoMGFuEPAY1A3UTdCuoxt26SScVL5FHbYg9Vrk4rHSCZi\nxSXHASzLYMYUPz1Jm4tdg++qKCLyXvSqK2Uh01cpcAIm7Um4sdozoZc37pfJpsid6MA53UvB62A6\nEDqbI/32BdKpgbsj9nchHHwnWYqmikgFUCiQspCOZbG8Jh0ZcIAZ1Zp50C/gDxEKRvDWhAEIFXz4\n/VdOPZxV74aCXx9XKBCR90ehQErOLjhkE3n8US8Xkm51QKFgEGEvDmCmBh9XUBX2UBc1ef1EilhK\nYwtE5NopFEjJZRNu10Eg6uNCX0V8ZrWW0LiCZWL7DcyMAwVn0FPmTvNSsOHlwz1j3DgRqQQKBVJy\n6Zg7yNAf9XIhAQYwvUqhYDCFkIkBWMnBqwVzp/oAeOm17jFslYhUCoUCKRnbcUil08S73D0P8Dlc\niDvUBhzsfJZ0Oo0z+AfiCasQdLtXhlrEKBoyuWm6n9eOx+iOaxaCiFwbhQIpmUw2y2tvd9LRmgbg\nSFuGdMEgbOQ50hLn0DvdFPLqG7+c7TOwLQMrUWCoxPTxRRFsB/71DXUhiMi1USiQkvJ6fZg5B0yD\nbtzSd13AwOcP4PVqsOEVDMPtQrDB7Bm8EvCxRWEMA156vWuMGyci451CgZSW40C2AAGLjrRbEq/1\nqc9gOHbIXenR7MgMerwu6uGDcyIcOpngQufg54iIDEahQEora7sLE/gt2lNuKKjzKxQMpxA0cQyw\n2rNDnvOpZXUAvPir9rFqlohUAIUCKSkj2zdgLuChPVXAazqEtOXB8EwDO2RiJvI48cGDwcoP1VAT\n8fCz5k7S2cFnKoiIvJtCgZSUkXFDQd5r0Z1xqPXZaHXj95YP9yWnC4lBj/s8Jv/hNyYRTxfY82uN\nLRCRq6NQICXVXynoxE0CNb7Bp9rJQIWIhWMA5+NDnvNbH5mMZcI/vNyOo7mdInIVFAqkpIyMDaZB\na85901IouEqWgV3jg+4MTmrwWQh1VV5WfrCWU61pDh4fOjyIiPRTKJCScWzHrRQELDpS/aFAn2iv\nVmGK3/2P84N3IQDcfdsUAP5m1wVVC0TkPSkUSMkU0rbbaeD30J5y/7vaq0rB1SqGggtDVwEWzAzx\nsZurONKS4JWjvWPUMhEZrxQKpGTySTcAOH6L9lSBmoCJR7+RV89vQW0AOtI4maFXfvx/PzUd04Dt\nPztPwVa1QESGpl1npGRyfev3Jz0mObvA5KASwdWybZtsMo5nkgdvF2RbOijcGCKVjGPbtQPOnTU1\nwCeX1vHzVzv5ycvn+M0PRYe8byQSwTT1/4PIRKVQICXTXylot92ZBwoFVy+TTZE/kSUfrMYLWKcT\n5O08ud4uEh+uo6amZsD5v/vvp/EvB7t46udtxFIZfJ4r532mU0k+84mZVFVVjdGzEJFyo1dhKZl8\n0sYx4WLWLWlPDmrVomsR8IcIVlVB0IOVtAn5QgT8oUHPra/x8ZmPVpPKwmsnC4TC0Sv+BYKDXysi\nE4dCgZSEXXAopG0cn0l738yDSaoUvD9V7kZS9A697DHA73y8luqwyevvJDg/xL4JIjKx6VVYSiLV\nmQHHHWTYlioQ9BiEvVrK8H2p7puF0DP8G73XY/CJm4MA7Pl1lwYdisgVFAqkJJJ9n1RzHoNY1mFK\nyMTQ+sbvT8ADAQtiWXiPN/pptR5unR2mM5bnV0d6xqiBIjJeaKChlESy3Q0FXX3LG9drF6TrUxOA\nCwm8SYd4PE5v75VrEsRiMRzb4bZbqznTluHAW3FmTgnQMDVQggaLSDlSKJCSSPRVCi4U3FAwReMJ\nrk+1Hy4k8MRtdv3bRaZO911xSldHG6FwFeGoyZ0r6njupVZ+/mon6z45lZBfoUxE1H0gJZJsz2BY\ncL6vG3yKKgXXx29ByIM3axC0gkPMLggWT6+v9fGxW6pJZmx2vdqlJZBFBFAokBIo5G1SXVk8IZO2\nNAQsg6hP4wmuW7UfA4NA99W9wS+5KUJDvZ+Wi2le04ZJIoJCgZRAoj0NDhAwieXQIMORUuPHwSHY\ndXX7RxiGwapldQT9JvsO9dDeWxjlBopIuVMokDEXb026/2u5v37qOhghXouc38GXBCc2/JoF/UIB\ni1XL6rAd+JfXkyQz2pBKZCIbNhTYts1Xv/pV1q5dy4YNGzh16tSA43v27GHNmjWsXbuW5557bthr\nWlpaWLduHevXr2fz5s3FPswf/OAHrF69ms9+9rP8y7/8CwCO47By5Uo2bNjAhg0b+PM///ORft5S\nQrHWFABtdv/MA2XTkZIO9r2pn7n6HRFnTQ2wdH6EnqTNd37cqvEFIhPYsLMPdu3aRS6X45lnnuG1\n115j27ZtPPHEEwDkcjm2bdvGD3/4QwKBAOvWreOOO+7g1VdfHfSarVu3snHjRlasWMGjjz7K7t27\nWbx4MTt27ODv/u7vyGQyrFu3jttuu41z585xyy238OSTT47JD0HGVqyvUnA62zfzQJWCEZMNONgm\nmGdiOAsnXXW3zMduruZce4r9byb54S/aWNNYP8otFZFyNOxHtAMHDrBy5UoAFi9ezKFDh4rHjh8/\nTkNDA9FoFK/Xy7Jly2hubh7ymiNHjrBixQoAGhsbefnll3njjTdYunQpXq+XSCTCrFmzOHr0KIcP\nH6a1tZX77ruPBx98kBMnTozKk5fSiLem8PhNTsYd/BZUaZDhyDEgXWNAugBtqau+zDQNbv9QiNqI\nxVM/O8fr72jgochENGwoiMfjRCKR4teWZWHbdvFYNHppC9ZwOEwsFhv0mkKhMKAkefm5775HPB6n\nvr6ehx56iL/+67/moYceYtOmTdf/TKUs5LMFEp1pApP8dKQNJgUMDTIcYclJfX/Wp65txcKQ3+SP\nf2cqBrDt6ZN09OZGvG0iUt6GDQWRSIREIlH82rbt4l7r0Wh0wLFEIkFVVdWg11iWNWCP9ng8Pui5\n/W+yHZEAACAASURBVPe49dZbueOOOwBYtmwZra2t1/k0pVzELibBgWzI7bmqDyoQjLRcCHeTpAsJ\nnFT+mq5dOCPA7/+HG+mK5/n//vYk+YLGF4hMJMOGgqVLl7J3714ADh48yIIFC4rH5s6dS0tLCz09\nPWSzWZqbm1myZMmQ1yxatIj9+/cDsHfvXpYvX86HPvQh/u3f/o1s9v9v777DoyrTxo9/z5zpk0mF\nUENJQEAxamgqiMArq7wCggprQ71EQd7l2tX9iXWxvBawrbsruqu7oou6or5iQRALKqzCUsQg0gUS\nCOl1Mr2c5/fHkMHQhYRQ7s91DQln5px5zpOTM3eedoepr69n27ZtdO/enRdeeIF//vOfAGzatIn2\n7ds3/ZmLFuEpiY8nqFTxS0+CgmagadAlJT7t8xe2FgBccWErBuemsqHQx8sLdjd9+YQQJ6xDDjQc\nPnw43377Lddccw0AM2bM4OOPP8bv9zN+/HjuvfdeJk6ciGEYXH311WRmZh5wH4B7772X6dOnE4lE\nyMnJ4bLLLkPTNG688Uauu+46DMPg97//PVarlUmTJjFt2jSWLFmC2WxOHEOc/Dyl8Zahbb54N1Qb\npwQFzaKDGzZUQaEH1T0dzXTk9axpGndcmUVhWZD5yyvplGln5PmtmrGwQogTxSGDAk3TeOSRRxpt\n69q1a+L7oUOHMnTo0MPuA9ClSxdef/31/baPGzeOcePGNdrmdrtl5sEpylPiQ9M1fqiIkmYHh1mC\nguagmU2oLDfsqINiL3R0H36nn3HYdB6+sSu/e3ELf51fRIdWNs7r9suOIYQ4+cgEcXHcGDGFp9SP\nLc1OfUjRKVkCgmaVnQoa8NPR5TZom25j+g1dMWkaj7+5g4LSI5/NIIQ4OUlQII4bX1UQI6oIOy0A\nZElQ0Kw0pyXejVAfhlLf4Xc4gN5dkrjzqix8QYM/vLqd8tojWylRCHFykqBAHDeekvgHU/melQwl\nKDgOuqXFv26tgaNcqXDYeelMHNGeKk+E+/+xld2lNXg8nv0eDdOVhRAnr0OOKRCiKTUMMtxaH8Ni\n1mibpLE71MKFOsUYyiAY8OH31cc3mMCSacNcHsJcZcXItDZ+vRF/PUDA76W+fu/zSUlJianEV13U\nmtJKHwtW1XHP7F2M6OvCZtn7N0Uw4Gf0oCySk5Ob+QyFEM1JggJx3DRMR1xfHSGngwPdJFn5mlo4\nFEQriaGMvbkPIg7QUTgLw3gzHI1eHwz4CP5Uis3mxBQKUhgNY7PXEgj46dqvfaPFxa7ob2FrsYUt\nuyN89n2QKwa2xm6VxkYhTiUSFIjjQhmK2t1ezMlWgtUa3TvYAH9LF+uUZLM6cDr2riqKAyJpMSw1\nUWzFIdgnrYHN5sTpSEJFYxSUxXA4IRBQ7MivxuHcO4agpqqCftluzGYnGwr9fLisgjEXtsYmgYEQ\npwz5bRbHhbcySDQYw++IDzLs3sHewiU6vUTSzMTMYN8dRHkPPljQYrFitdmx2Gw4nEk4Xe7Ew+5w\noGkaw85Lo1cnJ+U1ET5cVkEoImMJhDhVSFAgjovaongf9849n0dnZklQcFzpGnXtQTOANWUo4+iX\nL9Y0jWF5afTMclJWE+GjbysJRWQ5ZCFOBRIUiOOitiiedW9tTZTsdg7cki75uAumQri1BepCsLn6\nmI5l0jT+q08aPbKclNaEWbDKS3X9L8uzIIQ48UhQII6L2l1eNF2jLArndks6/A6iWfi7OsFpji9o\nVFx/TMcyaRrD+6SRm+2ixmvw4OvF7K6U6SRCnMwkKBDNLhqK4SnzE3JZMdA4J1uWy20xZg36tYt/\n/b4cre7YFiPSNI3Buank5dioqIvy//62lZ92ywBSIU5WEhSIZldX7AUFxTEN3QS9u7paukinNS3Z\nBnltwVDY1taiBY9toKCmaZyXY+fWS1vh8Ue5++8/kb/t2FohhBAtQ4IC0exqdsXHE2zyxOiR5cJp\nk/EELU1r44JzMyGqsO8OgT9yzMccnpfMfdd2IRJV/GH2Nj5dVdUEJRVCHE8SFIhmV7Mz/ldjiTJx\nTraMJzhRaFnJRHolgwHsqMMUOvaphRedncrjt+TgtOn8ad4uXv54N7FjmOkghDi+ZPEi0ayMmEHV\nDg8RuxmfV5NBhieYWDsHqjqArSyCvTRKqK0OR/EjMgyD+vp48NelFTx2U3ueereU97+toKDUy9RR\nrXHaTIllkw/k58sqCyFahgQFolnVFvmIhQ2KbGasZo2eWTKe4EQTSzaD1Q676rGVhMAW/sV3hlDQ\nz6IVtaSk7R24OCzXwVc/KL7fFuCOvxUwsLuiQ7v0A+4vuROEODFIUCCaVeW2OgA2+BV5Pd1YLfKX\n4AkpzU4oFMBWEYUddehtLByoM8EwjL3Jln4m6PdhdybjdO2dWeIErhjk5tsf68jf5uXLjfArl5mu\n7Rz77S+EODHIHVo0q8ptdShgtzJx4VmpLV0ccQgxl4lQOyvoGrayCOYCH2qfdMuhoJ/gT6WoXZ7G\njyIPoeD+UxFNJo2LclM5v7tGzICP/1PF8vV1GEeZxlkI0bykpUA0m0gwSm1RPbVmnZihcVZHDY8n\nnr2vvr5+zweO1rKFPE00pEjWTGZsdltie8Dvxf6zD2jDrkOOC2N7LZbtXgiBys1EM+/9+6EhgdLP\n+W0eDjV/IbuNiXS3zjebY6zeUk9pTZjL+qXjkJkoQpxQJCgQzaa6sB5lwE9RyEnTqd1YRe2e5yqq\nKrCYrYDkQDgeQuEAptogVrcD5dubVjlYV4FuT4639Tewmwll2bBWxdB3e6E2hOrd6pjjt/QkE9cM\nacXn31WzozTI3K/K+VXfdDq0sh1+ZyHEcSHdB6LZlG+JhwBFykTfLBdJLnfi4XRIv/LxZrPacdhd\nOB1JiYfN5jzga5VZI5yXBjmp4IvAihLcGwPY6gw4hqZ/m9XE5edncMGZyfgCMeb9uyLenSDTFoU4\nIUhLgWgWRkxRur6KsEmjJGri3PbSInDSMWloZ7ZCdXTD+koslQHS60CZgpBkQJIVkizwC7sANE2j\nb49kOrSy8dnqalZvqaegVCc3OxmZfCBEy5KWAtEsqnbUEfZF2Rw10TlVJ8UufccnKy3ZhnZBB+rO\ntuNrpaFMGnjCUOyFLTWwsZqkWh1bnbHfwMRDaZdh49phbejZyUmlJ8a9r+5m4cpKaTUQogVJS4Fo\nFsXr4kvcbjF0utlCbCkob/R8VU0VRjiMPbVTSxRPHIWYS8ffUUdLNePUneANgzcCvjD2gAn7DgNK\nC1Hd0qBTMprp8IMQrBYTw/uk0y5VsXJLiOffL2LxmhqmXtFRpi4K0QKkpUA0uVjUoHRDNX5No0KH\nLikWQjj3eTjwR+XyOylpWrzLIMMBnZOhVwY1rSL4MjQIx2BdBXy9E1XuO+JDZre18vTEjgzqncKG\nQh9TZ23mL+/vouIYszgKIX4ZuSuLJlextZZoMMbmmIns1mDRZdrhKU3TiFnAk6XDf3WGLinxBEsr\nSlDfl6FCsSM6TEaymQeu78qjN2fTPsPGJyurmPjMRl78qIjiylAzn4QQAqT7QDSDHctLAdhk6FzQ\n1oCQjCc41Rkqvg5CIBqAbDtaax3rJg+monpUmRdLlplYmyO73fTtkcx53dx8mV/NG1+UMn95JR//\np5J+PZIZ0S+Dvj2SMUugKUSzkKBANKmaXfVU7/Cw0zCRlZNMiqMW5I+8U144FEQriaGM+BoICgi2\nMWO2gaUqgnt7hEi1D9UnFc1pOezxdF1jeJ8Mhp6bzjfranjv32Ws3ORh5SYPKS6dwb2TGJLrpmMr\nKyDJlIRoKhIUiCa17d/FAHyndG4ekE7BjtrD7CFOFTarY7+VDnFCIMmDVhnCXhuFr3aiuqZAtzQ0\n6+FbkMy6Rl62maJig96dkthSHGZbSYT5K+qYv6KOzBSdrplw+6gs2raWZbSFOFYSFIgmU1/mp2xj\nDWVKw9nOxYCeyRTsaOlSiZYW06G8rY/0sJvkMjBtq0XtqCXaxo7eJR0tw9Eo9bJhGPh8ewcper1e\njFiMjGSdoW1aMeRcje2lATYU+thZFqK8DvKfL+Sis+sY3ieDs7u60DTpXhDiaEhQIJqEMhQ/fLgd\ngFWGmd+OzsJ0BFPSxKmvYYnlmNtBsKMTc10Mc20Uc0kQSorBYsLi1li6VWFtm4pfDxIprMa+Z7VF\nv9+L1WonaPZBt7Y4XW66d3DSvYOTen+UddtqKKqM8cWaGr5YU0O7dCvD+6ZzSV46rVOsLXz2Qpxc\nJCgQTWL7shJqd3nZapjIOjOFDmkxPB4PgYAPm0pp6eKJX8gwDMJ+b6NtQb+PUChEylH8PBNLLLuS\nwAW0UwQq69BiCludgbU6hhVgdzU2DQybA91tg2Qb5iQdzWSGA/Q2uJ1mcrtYmXypjd21Fr5aW89/\nNvmY81kpr39eyjldHQzJdXPROZk47ebEuXm93v0PhoxNEEKCAnHMand72fzFLgLAcs3CHa0MitaU\nYxiK6E4PkVRH44Q74oQXCgeI7ghjTzYS26zeCMF6D5H0Jvh5ahp+c4ioEcadlUqwrh5LRMeuLOAL\nYw7qEAxARQCHDtEURTT1wLerUNDPZytrSUlLp3t7nc6ZbraXRtiyO0z+9gD52wP8dUEF/Xum0K9H\nMl1bK2rX78bpaHwS/oCfMwZ2IVnWWhanMQkKxDHxlPpY8dpGjJjiy5iFsXmpdMiI32wNw8Bmk5wH\nJyv7PimSVTRGKBRs+vdwulExA03TsSQlUVVThllppJjdUBdCqw1hrY5iqYkSC2moM2xoSY27BewO\nF06XG4jHK3kpkNcDqj0R1m2robQmxr/X1fLvdfGBr24rtHMHae3ScVnBYdbQlUbp2koyUgO47Cac\nNhOpSTrpqcnSeiBOGxIUiKNWvqWG/Pe2EQ3GWByz0LWLnQs7S5OAaAImDZLj3Qf+5Fos9QqzJ4Z5\ndwB270RlOqFrCrQ+9PWWnmyhd8cY3VuFMSwpFFdH2V0RpLpOsaUqypaqaOMdNnsAT6NNyU4THVrZ\n6drWQdd2Dnp1ctKlrQNdxsyIU5AEBeIXC9aH2fr1bnauLMMAvoqZST7DzYjOcpMUzcCkEU01E003\no9tt2HaHoNwffyRZsKdDNMOEUuqgsw4cThdpGal0aK04u00d7KpH16wEwgb1wSABzYRP1winODCw\nEI4qgmGDOm8Iw4DNu/xs3OlPHM9u1eje3s4ZHWyc2z2NXp2TcEnSL3EKOGRQYBgGDz/8MFu2bMFi\nsfD444/TqdPeBDZffvklL774Imazmauuuopx48YddJ/CwkLuvfdeTCYT3bt356GHHkLTNN555x3e\nfvttzGYzU6ZMYciQIQSDQaZNm0Z1dTUul4uZM2eSnp7e7JUhDk4pRW2Rl6LvKyj6vgIjqqhFY1HU\nQt9+rbh+SDLlP1S2dDHFKcxQimBSFOO8FDSPE3ORH70siNML7IygfvQSc5lRNh10DQyFjo7bF8AU\nCaCiHoga7O3QCmEBGkYQKED5opjctnj/QqqVunQ/QVMUR2oKFTUBqr2Kynqo8CjWFQRYVxDgvW9r\n0YCs1lZ6dLRxRgc7Z3S0k9MxFV2XQEGcXA4ZFHzxxRdEIhHmzp3L2rVrmTlzJi+++CIAkUiEmTNn\n8t5772G327n22msZNmwY33333QH3mTFjBr///e/p168fDz30EIsXL+acc87h9ddfZ968eYRCIa69\n9louvPBC3nrrLXr06MHUqVNZuHAhf/3rX3nggQeOS4WIOGUo/LUhand5qS7wUL61luCe5DQ+YGXM\nzFZN5/Izdfqmelj95RY6pndu2UKLU9rPBz8qIOzWwGHHu7sMl+HAgQ29NgJEGu1nBpRZA4cFbDpR\nXaFCESx2O2gQDgQxRUGLKkxRY28rBJCy56GoIklTdNVNoAEaGA4IR2NgMhFVGuHqEEZ1PUVrNbYA\nMYuJpBQb1mSd5HQbrdraycy00irZgt2qoWnaL57tEDMUdb4o1fURqurClFZ6qfHG8AYMghGDYDie\ndlrTFDaLRrLTTJLDhNuu43bqpLp00tw6KU6d5GT3UY+VONQMDpBZHCezQwYFa9as4aKLLgLgnHPO\n4ccff0w8t23bNjp16oTbHR/c06dPH1atWkV+fv4B99mwYQP9+vUDYPDgwXz77beYTCby8vKwWCxY\nLBY6d+7M5s2bWbNmDbfddhsAF110USIQEUdHKYW/OoQRNTAMhYqpxNegP0JFRZCwP0rYFyXsCRGp\nCxH1hCG2N699GNhhmNisdEo1EwOzXdzQPYl0Z/wvIb/Pf5B3F6Lp7Dv4EaDaVUZMj+Fq3woMBTED\nYopA2A8d3QQifky6lbSMVgBEfPWoXR4se2YfROojaJoOOmhZyTgse9JC14cJltegh8BkaGiBKCYV\nb4GAeDY5Exo6xNNEmxSGAfF/iJejOgrVQAH4gS0KVqPhRcNn0tAdFqxOM5pVR5lNmMwmTLqGZoJA\nROEPG9RHoS6iqPNHqfNGG97+mGhAapJOqxQr6ckW0t0WXHYdq1nDZjVhs5iwmDWMGERiBtGYIhJV\ne7pVYnh8IbYXezGUHn8uxp6vimhUYbfqWCwmzLqGWdew6BoOm47TZsJh03HYTDisJpwN3+/znM1i\nwqRp6KZ4Uk6TSUv8PyXJjNshPd/N5ZA16/V6SUra+wuo6zqGYWAymfB6vYmAAMDlclFfX3/AfWKx\nGEqpA75232N4vV68Xi8ul6vRaw8lFotnYSstLT2Scz7tbPt3MTtXlR3x68MKPGh4lEalMlGhTARt\nOj07u+idHOQySxSntZaySmg4ak1tJXXWWlwOV+I4hlKUV+zAYnFQ72k8zctTX0MsEsYbDGAxWxpt\n1006tXUVBAJ+NE3Dbnc02g4c8rmG573+OpweV6PtDe8RCYfweGuw2x37vbfJZE4ce98y/fy9w5Hg\nfu8LUF65G7PJQjjSOFA63Pvue84t+d6/tE4P9r5N8d5He86hcIBYzEYoFMRk0qmujK+vEAr40StD\n2KyORsfXTBAL2rD9bKqix6jGZNOxWCzo4b37NKiuLadLaxetMloD8aUUlKEI+AK07pKOzxNl009+\nTGHQQgbmiIEtZpDMnm6Lg/+xTcNvkqHgU92GYTfT0a2T7NRJcenY9RhltSHcLmv8Q1wnMfjR46nB\nUDoWexKhiCIcUYSiimBIEQgr/MEoMZ/G1qr4h/2x0k3xJanNpngrha5ZiEY0QjFF1FBEIgahiNEk\nAY3VYuIvvzkDu/X0bolo+Lxr+PxrKocMCpKSkhotN9oQEAC43e5Gz/l8PpKTkw+4j67rjZqSvF7v\nAV/r8/lwu92Ntjcc91AqKuI3ieuvv/6wJyyO3saWLoAQQgCXf9zSJThxVFRU0Llz03XdHjIoyMvL\n46uvvmLEiBHk5+fTo0ePxHPZ2dkUFhZSV1eHw+Fg1apVTJw4EU3TDrhPr169WLlyJf3792fp0qVc\ncMEF5Obm8txzzxEOhwmFQmzbto0zzjiDvLw8li5dSm5uLkuXLqVv376HPInevXvz5ptv0rp1axnY\nI4QQ4pQXi8WoqKigd+/eTXpcTf28XX8fSikefvhhNm/eDMCMGTNYv349fr+f8ePH89VXX/HCCy9g\nGAZXX30111133QH36dq1KwUFBUyfPp1IJEJOTg6PPfYYmqbx7rvv8vbbb2MYBlOmTGH48OEEg0Hu\nueceKioqsFqtPPvss2RkZDTpiQshhBCisUMGBUIIIYQ4fZzeIzWEEEIIkSBBgRBCCCEACQqEEEII\nsYcEBUIIIYQATpKESEopBg8eTJcuXYD4VMk777yT/Px8nnjiCXRdZ+DAgUydOhWAWbNmsWTJEnRd\n5/777yc3N5fq6mruuusuQqEQmZmZzJgxA7td0voeicPlwBBHZuzYsYmFvbKyspg8ebLkA2kCa9eu\n5ZlnnuH1119vkhwrB7uvnO5+Xs8bNmzg9ttvT8yPv+666xgxYoTU8zGIRCLcf//9FBcXEw6HmTJl\nCjk5Ocf/elYngYKCAjV58uT9tl9xxRVq586dSimlbrvtNrVhwwb1448/qhtvvFEppVRxcbG66qqr\nlFJKPfroo+r9999XSin10ksvqVdfffX4FP4U8Omnn6p7771XKaVUfn6+mjJlSguX6OQTDAbVmDFj\nGm2bPHmyWrlypVJKqQcffFB9/vnnqry8XI0cOVKFw2FVX1+vRo4cqUKhkJo9e7Z6/vnnlVJKLViw\nQD322GPH/RxORC+//LIaOXKk+vWvf62Uapo6HT169H73ldPdvvX8zjvvqNmzZzd6jdTzsXnvvffU\nE088oZRSqra2Vl188cXq9ttvP+7X80nRfbB+/XrKy8u58cYbmTRpEjt27MDr9RIOh8nKygJg0KBB\nLFu2jDVr1jBw4EAA2rVrRywWo7q6ulEeh8GDB7N8+fIWO5+TzaFyYIgjs2nTJgKBABMnTuSmm24i\nPz9/v3wgy5YtY926dYl8IElJSY3ygQwePBiI5wOR6zeuc+fOzJo1K7GM+rHWqdfrJRKJ7HdfOd3t\nW88//vgjX3/9NTfccAMPPPAAPp+PH374Qer5GFx22WX89re/BeKts2azuUWu5xMuKHj33XcZNWpU\no0dmZiaTJ09mzpw5TJ48mWnTpuHz+RrlWDiSfAoN248kn4LY62A5MMSRczgcTJw4kVdeeYVHHnmE\nu+66q9HzTZ0P5HTxq1/9qtEqpuoYc6wc7L5yutu3ns855xzuuece3njjDbKyspg1a1ZimfoGUs+/\njNPpTNTZ7373O+64445G99njdT2fcGMKxo0bx7hx4xptCwaDiQuyT58+lJeX43K5GuVNaMinYLFY\nDppPwev1kp6efkT5FMReh8qBIY5Mly5dEv2vXbp0ITU1lY0b92aTaOp8IKerY82xcrD7imhs+PDh\niQ+m4cOH8+ijj9KvXz+p52NUUlLC1KlTuf766xk5ciRPP/104rnjdT2fFHf2F154gX/+859AvBm2\nffv2JCUlYbFY2LVrF0opvv32W/r27UteXh7ffPMNSimKi4tRSpGWlkZeXh5LliwBOKJ8CmKvhlwU\nwH45MMSRmTdvHjNnzgSgrKwMn8/HwIEDWblyJbD3mszNzWX16tWEw2Hq6+v3ywfy89eK/TXkWIGj\nq9OD3VdEY7feeis//PADAMuWLaN3795Sz8eosrKSW265hWnTpnHllVcCLXM9nxTLHNfX1ye6DMxm\nMw8++CBdu3Zl7dq1PPHEE8RiMQYNGsQdd9wBxGcfLF26FMMwuP/++8nLy6Oqqop77rkHn89Heno6\nzz77rMw+OELqIPksxJGLRqPcd999FBcXAzBt2jRSU1MlH0gTKCoq4q677mLu3LlNkmPlYPeV093P\n63nTpk088sgjmM1mMjMz+d///V9cLpfU8zF47LHHWLRoUaN76wMPPMDjjz9+XK/nkyIoEEIIIUTz\nOym6D4QQQgjR/CQoEEIIIQQgQYEQQggh9pCgQAghhBCABAVCCCGE2EOCAiGEEEIAEhQI0WSKioro\n3bs3Y8aMYcyYMYwePZphw4bx/PPPH9Xx/vKXv/Dll182aRlfffXVRPmuvPJKFi5cmHiuZ8+eTfpe\nh7Jr1y4eeOCBo95/3rx53HfffU1Yov3dd999lJSUADBs2LDEGhNCnMpOuGWOhTiZZWZm8sEHHyT+\nX15ezqWXXsrll19Odnb2LzpWQ3KUpvLHP/6RTZs28cYbb5CUlERZWRk33HADaWlpXHDBBU36XodT\nXFzMzp07j3p/TdOasDQHtmLFikZrz8uSLuJ0IEGBEM2ovLwcIJGo5OWXX2bRokWJ1cWmTZvGjBkz\naNOmDbfccgsQDwZGjRrF4sWLGTBgAGPHjuWDDz5gzpw5GIbBWWedxUMPPcSTTz5Jt27duPbaa3nn\nnXd47bXXWLhwIZFIhOHDh7N48eJEzhCfz8ecOXNYuHBhIkFKmzZt+OMf/4jT6UyU96GHHiI/Px+A\n559/nk6dOvHJJ5/w2muvEQwGCQaDPP744/Tt25cJEyaQmprK1q1b+dOf/sTq1av56KOPCAQCaJrG\nc889R05ODsuWLePJJ5/EMAw6dOjAM888w2OPPUZRURGPPvoo06dPP2C9FBUVceutt5Keno7dbmf2\n7NmHre9YLMZTTz3FqlWriMVijB07lptvvpkVK1bw0ksv4XA4EsvCPvvss1gsFubMmcObb76J2+0m\nOzubTp06YbVaKS8vZ/LkybzxxhtAfLn1jRs3EggEeOqpp8jNzW2iq0SIE4d0HwjRhMrLyxkzZgwj\nRozg/PPP589//jOzZs2iTZs2LF26lPXr1/N///d/vP/++5SWlvLRRx8xZswYFixYAMQTlnz//fdc\nfPHFaJqGpmls3bqVd999l7lz5/LBBx+Qnp7OK6+8wpAhQxIplJcvX47H46GqqorvvvuO8847r1FW\nu+3bt+NyuWjfvn2j8p599tnk5OQk/j9w4EA+/PBDLrzwQubOnYtSirfffpuXXnqJDz/8kNtuu41/\n/OMfidf36NGDRYsW0bFjRxYvXswbb7zB/PnzueSSS3jrrbcIh8NMmzaNJ598kvnz59OjRw8++OAD\npk+fTu/evZk+ffpB6wWgoKCAZ5555ogCAoB33nkHTdOYN28e7777LosXL2b16tUAfP/99zz44IN8\n8sknlJSU8M0337Bp0yb+9a9/MW/ePP71r39RWFgIwKRJk8jMzOTll18mNTUVgO7du/P+++8zYcIE\nXnnllV90XQhxspCWAiGaUEP3gVKKmTNnsnnzZgYMGADEP7h/+OGHRLKTUChEx44dGT16NOFwmJ07\nd7JmzRqGDh2K1WoF4k3WK1asoLCwkPHjxwMQiUQ466yzmDhxIg8++CCGYbBjxw7++7//m1WrVrFu\n3TqGDBnSqFwmk+mImr8vueQSIP4BuGrVKjRNY9asWXz55Zfs2LGDVatW7ZdCF+KZNJ999lnmzJ9B\nVQAABAJJREFUz59PQUEB33zzDb169WLLli20adMmMV7hzjvvBOJN8w0OVi99+vQhIyNjv0DmUJYv\nX86mTZv4z3/+A0AgEGDr1q3k5ORwxhln0KZNGwBycnKoq6ujsLCQoUOHJlpyLr/8cjwezyHrJicn\nh08//fSIyyTEyUSCAiGagaZp3H333YwZM4bZs2czadIkDMPgpptu4uabbwagrq4Oszn+Kzhq1CgW\nLFhAfn4+t912W6NjGYbBZZddxh/+8Acg3hUQi8WwWq306NGDjz76iOzsbPr378+yZctYs2bNfsfI\nyckhEAhQUlJCu3btEtsXLFhAVVUVN954I8B+KbH9fj9XXXUVY8eOpX///vTs2TPRnA4kkoqVlJQw\nYcIEJkyYwMUXX0zr1q3ZuHFj4vwaeL3eRqlcG87vQPVSU1ODzWY74jpvONbdd9+d+ACvrq7G5XKR\nn5+fCLQg/vNRSmEymY543EBDMNSwrxCnIuk+EKKZ6LrO3Xffzd/+9jcqKys5//zz+fDDD/H7/USj\nUaZOncrnn38OwOjRo1m4cCGFhYX7pTbt378/X3zxBdXV1YmMlXPmzAFgyJAhvPDCCwwYMID+/fuz\nePFinE5nosm7gd1u54YbbuDhhx/G6/UC8dkSzz33HN26dTvoORQUFKDrOpMnT2bAgAEsWbKk0Ydo\ng3Xr1tG5c2duuukmcnNzWbJkCbFYjOzsbKqrq9m2bRsAf//733nrrbcwm83EYjGAQ9bLwRzsQ/n8\n88/n7bffJhqN4vV6ue666xIpfg/kggsuYMmSJXi9XsLhMJ999lliEKPZbCYajR6yHEKcaqSlQIgm\ntO+o+Isuuohzzz2XP//5zzz66KNs2rSJ8ePHE4vFGDx4MGPGjAGgbdu2pKenc+655+53vJ49e/Kb\n3/yGm266CcMwOPPMM5k0aRIAF198MY888gj9+/cnOTmZjIyM/boOGtx5553MmjWLX//615jNZnRd\n56677uLCCy/cr+wN3/fs2ZNevXoxYsQI0tPTufTSSxNN8z83aNAg5s6dy8iRI0lLS2PQoEEsXboU\nq9XK008/zd13300kEqFz58489dRThEIh6uvrueeee3jyyScPWC9FRUUHnWWgaRrz589n0aJFiW1T\npkzhlltuoaCggLFjxxKNRrn66qvp168fK1eu3O9YmqbRvXt3JkyYwDXXXIPT6SQtLS3R+jFkyBAm\nTZrUaAxFw37HY/aDEC1BUicLIU5bBQUFfP3114mui//5n/9h/PjxBw2shDjVSUuBEOK01b59e9at\nW8eoUaOAeMuOBATidCYtBUIIIYQAZKChEEIIIfaQoEAIIYQQgAQFQgghhNhDggIhhBBCABIUCCGE\nEGKP/w+T/L19J3fs0gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x11d022e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plot the distribution of review lengths\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "sns.set(style=\"white\", palette=\"muted\", color_codes=True)\n",
    "sns.distplot(review_df[review_df[\"sentiment\"] == 1]['review_len'], color='b', label='Positive')\n",
    "sns.distplot(review_df[review_df[\"sentiment\"] == 0]['review_len'], color='m', label='Negative')\n",
    "plt.legend()\n",
    "plt.xlabel('Review Character Length')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can attempt some for yourself: try to find the most common words in the corpus, and see the overlap with so-called 'stop words'. What should you do if the correlate heavily? What are the most common words from both camps? How much over lap is there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Split the data into training and testing data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Splitting data is an important step in any classification process, where one has a hold out set (test set) which the classifier never sees), and one trains the classifier on the training set. A usual split is anywhere between 60/40, and 80/20. Here we choose 75/25.\n",
    "\n",
    "Be sure to 'stratify' your split, namely, have the same proportion of each class in both the training and test sets, as it is in the whole data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(review_df.review.values, review_df.sentiment.values,\n",
    "                                                    stratify=review_df.sentiment.values, test_size=0.25,\n",
    "                                                    random_state=42) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1500,)\n",
      "(500,)\n"
     ]
    }
   ],
   "source": [
    "print X_train.shape\n",
    "print X_test.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Feature Creation in NLP Problems"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we create the usual common features for NLP ML tasks: the bag of words. We construct a dictionary of all words in the training set of reviews. From here, we break down the semantic structure of the review and create a vector whose elemental values correspond to the count of a given word in a review. Therefore, each word is a feature, and the feature space is extremely sparse. Note: we can also save some semantic structure by treating word pairs (bigrams) as features, or generalize this to ngrams.\n",
    "\n",
    "The transformation of the written word to some numerical representation is important step to using most classifiers in Scikit Learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1500, 434163)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "count_vect = CountVectorizer(ngram_range=(1, 2))\n",
    "X_train_counts = count_vect.fit_transform(X_train)\n",
    "X_train_counts.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we further extend the transformation of the feature space by the inclusion of term-frequency inverse-document transformations. Instead of the raw word-count vector above, we can normalize the counts over the vector, weighting the words with respect to the phrase itself.\n",
    "\n",
    "Inverse document scaling, scales each word weight in the particular review to how common it is in the corpus (the review data set). More uncommon words are weighted more highly (hence 'inverse' in the transformation name), whereas more common words are weighted less, such as articles in a sentence (these common words are often also noted as 'stop words', see below)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1500, 434163)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "tf_transformer = TfidfTransformer(use_idf=False).fit(X_train_counts)\n",
    "X_train_tf = tf_transformer.transform(X_train_counts)\n",
    "X_train_tf.shape\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1500, 434163)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_transformer = TfidfTransformer()\n",
    "X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
    "X_train_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Training your Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "clf = MultinomialNB().fit(X_train_tfidf, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_test_counts = count_vect.transform(X_test)\n",
    "X_test_tfidf = tfidf_transformer.transform(X_test_counts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "predicted = clf.predict(X_test_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC Accuracy:  0.832\n"
     ]
    }
   ],
   "source": [
    "# Checking the accuracy of our classifier\n",
    "from sklearn.metrics import roc_auc_score\n",
    "print 'ROC Accuracy: ', roc_auc_score(y_test, predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROC Accuracy:  0.804\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "clf = SGDClassifier(loss='hinge', penalty='l2', alpha=1e-3, n_iter=5, random_state=42).fit(X_train_tfidf, y_train)\n",
    "predicted = clf.predict(X_test_tfidf)\n",
    "print 'ROC Accuracy: ', roc_auc_score(y_test, predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Grid Searching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An important part of machine learning is known as hypertuning. This is process of fitting the hyperparameters of your model to the training data.\n",
    "\n",
    "As always one should also always be cautious of overfitting your data when hypertuning your model. We can circumvent this problem through crossvalidation.\n",
    "\n",
    "Below, we use a grid search method, to iterate over various combinations of parameters for our different models to find the optimum, cross-validated set of hyperparameters for our data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on training set:\n",
      "{'penalty': 'elasticnet', 'alpha': 0.0001}\n",
      "0.921617777778\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "tuning_parameters = {'alpha': [1e-2, 1e-3, 1e-4, 1e-7], 'penalty': ['none', 'l2', 'l1','elasticnet']}\n",
    "clf = GridSearchCV(SGDClassifier(random_state=42, n_iter=5), tuning_parameters, cv=5, verbose=0, n_jobs=-1,\n",
    "                   scoring='roc_auc')\n",
    "clf.fit(X_train_tfidf, y_train)\n",
    "\n",
    "\n",
    "print(\"Best parameters set found on training set:\")\n",
    "print(clf.best_params_)\n",
    "print clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.842\n"
     ]
    }
   ],
   "source": [
    "predicted_SGD = clf.predict(X_test_tfidf)\n",
    "print roc_auc_score(y_test, predicted_SGD)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best parameters set found on training set:\n",
      "{'alpha': 0.7, 'fit_prior': True}\n",
      "0.83\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "tuning_parameters = {'alpha': [1e-2, 1e-3, 1e-4, 1e-1,0.25, 0.5, 0.7, 0.9, 1], 'fit_prior':[True, False]}\n",
    "clf = GridSearchCV(MultinomialNB(), tuning_parameters, cv=5, n_jobs=-1, verbose=0, scoring='roc_auc')\n",
    "clf.fit(X_train_tfidf, y_train)\n",
    "\n",
    "\n",
    "print(\"Best parameters set found on training set:\")\n",
    "print(clf.best_params_)\n",
    "predicted_mb = clf.predict(X_test_tfidf)\n",
    "print roc_auc_score(y_test, predicted_mb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Building Pipelines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pipelines are useful tools when transforming data, and then fitting it. There is often a fixed series of steps from data to classifier training, and pipelines help this.\n",
    "\n",
    "Interestingly, one can apply a number of transforms to the data and then apply a classifier. What's interesting, is that this can also be used in conjuction with GridSearcCV (which we applied earlier), to further tune our classifier, and the transformation stage.\n",
    "\n",
    "Below, we build a pipeline for the SGD classifier, but we include different preprocessing steps; for example, the selection between using only unigrams, or uni-, and bigrams; or whether to weight the ngrams with respect to inverse document frequency, or to use stop words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer()),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', SGDClassifier()),\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'vect__max_df': (0.5, 0.75, 1.0),\n",
    "    'vect__stop_words': ('english', None),\n",
    "    #'vect__max_features': (None, 5000, 10000, 50000),\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
    "    'tfidf__use_idf': (True, False),\n",
    "    'tfidf__norm': ('l1', 'l2'),\n",
    "    'clf__alpha': (1e-4, 0.00001, 0.000001),\n",
    "    'clf__penalty': ('l2', 'elasticnet'),\n",
    "    'clf__n_iter': (5, 10, 50)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 864 candidates, totalling 2592 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:  1.6min\n",
      "[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  7.6min\n",
      "[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 17.9min\n",
      "[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 32.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 51.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed: 69.4min\n",
      "[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed: 95.3min\n",
      "[Parallel(n_jobs=-1)]: Done 2592 out of 2592 | elapsed: 102.0min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=None, error_score='raise',\n",
       "       estimator=Pipeline(steps=[('vect', CountVectorizer(analyzer=u'word', binary=False, decode_error=u'strict',\n",
       "        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        st...   penalty='l2', power_t=0.5, random_state=None, shuffle=True,\n",
       "       verbose=0, warm_start=False))]),\n",
       "       fit_params={}, iid=True, n_jobs=-1,\n",
       "       param_grid={'vect__ngram_range': ((1, 1), (1, 2)), 'tfidf__use_idf': (True, False), 'tfidf__norm': ('l1', 'l2'), 'vect__stop_words': ('english', None), 'vect__max_df': (0.5, 0.75, 1.0), 'clf__penalty': ('l2', 'elasticnet'), 'clf__n_iter': (5, 10, 50), 'clf__alpha': (0.0001, 1e-05, 1e-06)},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score=True,\n",
       "       scoring='roc_auc', verbose=1)"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search = GridSearchCV(pipeline, parameters, n_jobs=-1, verbose=1, scoring='roc_auc')\n",
    "grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Pipeline Findings\n",
    "Below we see what the best combination of preprocessing transformations, and then gridsearch, hyperparameters achieve the best trained classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best score: 0.849\n",
      "Best parameters set:\n",
      "\tclf__alpha: 0.0001\n",
      "\tclf__n_iter: 10\n",
      "\tclf__penalty: 'elasticnet'\n",
      "\ttfidf__norm: 'l2'\n",
      "\ttfidf__use_idf: True\n",
      "\tvect__max_df: 0.75\n",
      "\tvect__ngram_range: (1, 2)\n",
      "\tvect__stop_words: None\n"
     ]
    }
   ],
   "source": [
    "print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(alpha=0.0001, average=False, class_weight=None, epsilon=0.1,\n",
       "       eta0=0.0, fit_intercept=True, l1_ratio=0.15,\n",
       "       learning_rate='optimal', loss='hinge', n_iter=10, n_jobs=1,\n",
       "       penalty='elasticnet', power_t=0.5, random_state=None, shuffle=True,\n",
       "       verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters['clf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CountVectorizer(analyzer=u'word', binary=False, decode_error=u'strict',\n",
       "        dtype=<type 'numpy.int64'>, encoding=u'utf-8', input=u'content',\n",
       "        lowercase=True, max_df=0.75, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 2), preprocessor=None, stop_words=None,\n",
       "        strip_accents=None, token_pattern=u'(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "        tokenizer=None, vocabulary=None)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_parameters['vect']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we apply the best preprocessing transformations to our test data, and then predict the test results from our best trained classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "best_vect = best_parameters['vect']\n",
    "X_test_counts = best_vect.transform(X_test)\n",
    "X_test_tfidf = best_parameters['tfidf'].transform(X_test_counts)\n",
    "\n",
    "best_predicted = best_parameters['clf'].predict(X_test_tfidf)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.86\n"
     ]
    }
   ],
   "source": [
    "print roc_auc_score(y_test, best_predicted)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see a marked increase in accuracy of our classifier, via searching the preprocessing space. The usefulness of Pipelines in Scikit-learn show their worth in this instance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Futher improvements\n",
    "There are a number of things that we can do to improve our classifier, that were mentioned in a previous lecture:\n",
    "- Inclusion of external semantic libraries\n",
    "- Incorporating 'part of speech' (POS) tagging.\n",
    "- Using negation methods\n",
    "- Using skip n-grams\n",
    "- Using char n-grams (useful for spelling mistakes)\n",
    "- Create an ensemble model to play to the features of two or more classifiers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##Further reading and references\n",
    "- Great tutorial to get into Pandas: [Ten minutes to Pandas](http://pandas.pydata.org/pandas-docs/stable/10min.html)\n",
    "- [Jupyter Notebook Guide](https://www.datacamp.com/community/tutorials/tutorial-jupyter-notebook#gs.PvzdTDk)\n",
    "- [Introduction to Seaborn](http://seaborn.pydata.org/tutorial.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}