ex01-the-scikit-learn-ecosystem.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This exercise takes inspiration from the tutorial found here: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html, but here served in a notebook format.\n",
    "\n",
    "Try to follow this notebook by itself but if you should get in trouble you can find help by refering to the original tutorial\n",
    "\n",
    "We will go through how to do the following with scikit-learn:\n",
    "1. Download the data\n",
    "1. Load the data\n",
    "1. Extract Features\n",
    "1. Build a classifier\n",
    "1. Put it all together into a scikit-learn pipeline\n",
    "1. Evaluation\n",
    "1. Optimization of parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from sklearn import datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.0 Download the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the tutorial above we will also be using the 20 newsgroups dataset with the official description (http://people.csail.mit.edu/jrennie/20Newsgroups/):\n",
    "\n",
    "> The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of our knowledge, it was originally collected by Ken Lang, probably for his paper “Newsweeder: Learning to filter netnews,” though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering.\n",
    "\n",
    "Run the script `fetch_data.py` from the root folder or first load code into the workspace by running the IPython magic-command `%load fetch_data.py` in the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the structure of the files in your local filesystem. This is important for the next step. See here why: http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_files.html#sklearn.datasets.load_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.1 Loading the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will load the data using the native function of scikit-learn suitable for loading text data which you have (hopefully) just read about"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First you need to set `file_path` variable below to point at your data directory with the 20 news data. Use this variable and the `categories` variable to only choose a subset of the document topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path_train='/path/to/your_data_directory/20news-bydate-train'\n",
    "\n",
    "# Only select a subset of the categories\n",
    "categories = ['alt.atheism', 'soc.religion.christian', 'comp.graphics', 'sci.med'] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "twenty_train= ... # instert code here to load the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look at the data. Run the following cells and inspect their output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'sklearn.utils.Bunch'>\n",
      "dict_keys(['data', 'filenames', 'target_names', 'target', 'DESCR'])\n"
     ]
    }
   ],
   "source": [
    "print(type(twenty_train))\n",
    "print(twenty_train.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data is of type <class 'list'>\n",
      "filenames is of type <class 'numpy.ndarray'>\n",
      "target_names is of type <class 'list'>\n",
      "target is of type <class 'numpy.ndarray'>\n",
      "DESCR is of type <class 'NoneType'>\n"
     ]
    }
   ],
   "source": [
    "for key, value in twenty_train.items():\n",
    "    print('{} is of type {}'.format(key,type(value)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'From: clipper@mccarthy.csd.uwo.ca (Khun Yee Fung)\\nSubject: Re: looking for circle algorithm faster than Bresenhams\\nOrganization: Department of Computer Science, The University of Western\\n\\tOntario, London, Ontario, Canada\\nIn-Reply-To: graeme@labtam.labtam.oz.au\\'s message of Wed, 14 Apr 1993 04:49:46 GMT\\n\\t<1993Apr13.025240.8884@nwnexus.WA.COM>\\n\\t<1993Apr14.044946.12144@labtam.labtam.oz.au>\\nNntp-Posting-Host: mccarthy.csd.uwo.ca\\nLines: 41\\n\\n>>>>> On Wed, 14 Apr 1993 04:49:46 GMT, graeme@labtam.labtam.oz.au (Graeme Gill) said:\\n\\nGraeme> \\tYes, that\\'s known as \"Bresenhams Run Length Slice Algorithm for\\nGraeme> Incremental lines\". See Fundamental Algorithms for Computer Graphics,\\nGraeme> Springer-Verlag, Berlin Heidelberg 1985.\\n\\n> I have tried to extrapolate this to circles but I can\\'t figure out\\n> how to determine the length of the slices. Any ideas?\\n\\nGraeme> \\tHmm. I don\\'t think I can help you with this, but you might\\nGraeme> take a look at the following:\\n\\nGraeme> \\t\"Double-Step Incremental Generation of Lines and Circles\",\\nGraeme> X. Wu and J. G. Rokne, Computer Graphics and Image processing,\\nGraeme> Vol 37, No. 4, Mar. 1987, pp. 331-334\\n\\nGraeme> \\t\"Double-Step Generation of Ellipses\", X. Wu and J. G. Rokne,\\nGraeme> IEEE Computer Graphics & Applications, May 1989, pp. 56-69\\n\\nAnother paper you might want to consider is:\\n\\n@article{fungdraw,\\n  title=\"A Run-Length Slice Line Drawing Algorithm without Division Operations\",\\n  author=\"Khun Yee Fung and Tina M. Nicholl and A. K. Dewdney\",\\n  journal=\"Computer Graphics Forum\",\\n  year=1992,\\n  volume=11,\\n  number=3,\\n  pages=\"C-267--C-277\"\\n}\\n\\nKhun Yee\\n--\\nKhun Yee Fung    clipper@csd.uwo.ca\\nDepartment of Computer Science\\nMiddlesex College\\nUniversity of Western Ontario\\nLondon, Ontario\\nCanada N6A 5B7\\nTel: (519) 661-6889\\nFax: (519) 661-3515\\n'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism', 'comp.graphics', 'sci.med', 'soc.religion.christian']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_train.target_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2257,)\n",
      "[1 0 2 2 0]\n"
     ]
    }
   ],
   "source": [
    "print(twenty_train.target.shape)\n",
    "print(twenty_train.target[0:5,])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Small extra exercise for working with data in python:\n",
    "- count the number of samples in total and per category"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.2 Extracting features from text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it is time to make use of this data in a scikit-learn friendly way.\n",
    "\n",
    "We need to transform the text into the format required by scikit-learn and machine learning in general. Remember we want $n_{samples} \\times n_{features}$ in a numpy array $X$ and another array with the targets $y$ for each sample.\n",
    "\n",
    "A well known representation is bag of words (BoW). To transform the text strings into a BoW representation find a suitable method here http://scikit-learn.org/stable/modules/classes.html#module-sklearn.feature_extraction.text \n",
    "\n",
    "Run the below cell if you get errors due to encoding/decoding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Function for fixing potential issue with encoding\n",
    "def fix_unicode(data):\n",
    "     return [text.decode(\"utf-8\", \"ignore\") for text in data]\n",
    "\n",
    "# Uncomment line below if you get decoding errors\n",
    "twenty_train.data=fix_unicode(twenty_train.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35787)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Instert your code here\n",
    "\n",
    "vect_method = ...\n",
    "\n",
    "\n",
    "X_train_counts = vect_method.fit_transform(twenty_train.data)\n",
    "X_train_counts.shape # check dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BoW have some issues as explained in the original tutorial:\n",
    "\n",
    "> Occurrence count is a good start but there is an issue: longer documents will have higher average count values than shorter documents, even though they might talk about the same topics.\n",
    "> To avoid these potential discrepancies it suffices to divide the number of occurrences of each word in a document by the total number of words in the document: these new features are called tf for Term Frequencies.\n",
    "> Another refinement on top of tf is to downscale weights for words that occur in many documents in the corpus and are therefore less informative than those that occur only in a smaller portion of the corpus.\n",
    "> This downscaling is called tf–idf for “Term Frequency times Inverse Document Frequency”.\n",
    "\n",
    "Therefore we will quickly transform our counts into frequencies with the code below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2257, 35787)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "tfidf_transformer = TfidfTransformer()\n",
    "X_train_tfidf = tfidf_transformer.fit_transform(X_train_counts)\n",
    "X_train_tfidf.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.3 Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the flowchart to determine a suitable classifier for the 20 news data (it contains link directly to the API of the different models)\n",
    "\n",
    "http://scikit-learn.org/stable/tutorial/machine_learning_map/index.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you have determined which classifier to use, train the model and test it on the the following sentences by using the function `clf.predict(X_test)`. You will need to transform them similar to the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_new = ['God is love', 'OpenGL on the GPU is fast']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# clf = ... #your trained classifier\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should get\n",
    "\n",
    "```python\n",
    "'God is love' => soc.religion.christian\n",
    "'OpenGL on the GPU is fast' => comp.graphics\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.4 Putting It All in a Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-learn makes it easy to work with many data processing steps from the raw data to a classifier. See http://scikit-learn.org/stable/modules/pipeline.html#pipeline\n",
    "\n",
    "Combine the steps from exercise 1.21.3 into a pipeline. Try to test it on the same two sentences as above to verify you get similar results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "text_clf = Pipeline(...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "     steps=[('vect', CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "        dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "        lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "        ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "        strip...inear_tf=False, use_idf=True)), ('clf', MultinomialNB(alpha=1.0, class_prior=None, fit_prior=True))])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text_clf.fit(twenty_train.data, twenty_train.target) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.5 Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will test our trained classfier on the test data. Start by loading the test data and use your pipeline from exercise 1.4 to predict its targets. Afterwards compute the accuracy using the *metric* module of scikit-learn.\n",
    "\n",
    "If you did not choose the classifier `SGDClassifier`, try to compare your solution with a pipeline using that instead of your classifier.\n",
    "\n",
    "Use the `metrics.classification_report` function to a detailed report of results, or `metrics.confusion_matrix` for obvious reasons\n",
    "\n",
    "**Extra exercise:** implement a function which takes the test data and a metric (e.g. accuracy) and a pipeline as input returning the result of the metric. Bonus if you make it take a list of metrics for comparing them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_path_test='/path/to/your_data_directory/20news-bydate-test'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load test\n",
    "twenty_test= ...\n",
    "\n",
    "# Uncomment below to avoid decoding errors\n",
    "# twenty_test.data = fix_unicode(twenty_test.data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1.6 Parameter tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Read the user guide on parameter tuning using grid search here http://scikit-learn.org/stable/modules/grid_search.html#grid-search\n",
    "\n",
    "Use it to tune the parameters of your pipeline. A suggestion would the `ngram_range` for the vectorization, `use_idf` for the term frequency transformation, and `alpha` for the classifier.\n",
    "\n",
    "Setting `n_jobs=-1` when calling the grid search function detects the number of cpus in your system and uses them for parallelization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "parameters = {'vect__ngram_range': [(1, 1), (1, 2)],\n",
    "               'tfidf__use_idf': (True, False),\n",
    "               'clf__alpha': (1e-2, 1e-3),}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "gs_clf = GridSearchCV(text_clf, parameters, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extras\n",
    "\n",
    "Try the following the things:\n",
    "- load your own data/other data (or some available from  e.g. https://archive.ics.uci.edu/ml/index.php)\n",
    "- if you choose to work with non-text data (image, csv etc.), use suitable methods as described in http://scikit-learn.org/stable/datasets/index.html#loading-from-external-datasets\n",
    "- for more advanced integration between pandas and sklearn, see https://github.com/scikit-learn-contrib/sklearn-pandas\n",
    "- build some pipelines using different vectorized features and models for comparison"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:scikit_intro]",
   "language": "python",
   "name": "conda-env-scikit_intro-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}