Skip to content

Commit 0e0fd04

Browse files
justmarkhamjustmarkham
committed
add tutorial materials
1 parent 61c51b2 commit 0e0fd04

File tree

6 files changed

+1884
-1
lines changed

6 files changed

+1884
-1
lines changed

‎.gitignore

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22
.DS_Store
33
*.pyc
44
extras/
5+
*.tpl

‎README.md

Lines changed: 46 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ Presented by [Kevin Markham](http://www.dataschool.io/about/) at PyCon 2016 (Por
44

55
### Welcome!
66

7-
This repository will contain the data files and the notebooks/scripts that you will need for the tutorial. I will email enrolled students as soon as those files have been added to the repository.
7+
This repository contains the data files and the notebooks/scripts that you will need for the tutorial.
88

99
A detailed description of the tutorial is below, including a list of **required software** and **knowledge prerequisites**. If you need a refresher on any of the prerequisite material, I have listed my recommended resources.
1010

@@ -54,3 +54,48 @@ In this tutorial, we'll answer all of those questions, and more! We'll start by
5454
### About the Instructor
5555

5656
Kevin Markham is the founder of [Data School](http://www.dataschool.io/) and the former lead instructor for [General Assembly's Data Science course](https://github.com/justmarkham/DAT8) in Washington, DC. He is passionate about teaching data science to people who are new to the field, regardless of their educational and professional backgrounds, and he enjoys teaching both online and in the classroom. Kevin's professional focus is supervised machine learning, which led him to create the popular [scikit-learn video series](https://github.com/justmarkham/scikit-learn-videos) for Kaggle. He has a degree in Computer Engineering from Vanderbilt University.
57+
58+
### Tutorial Introduction
59+
60+
* Required files for today:
61+
* Clone or download this repository: [http://bit.ly/pycon2016](http://bit.ly/pycon2016)
62+
* IPython/Jupyter notebooks ([tutorial.ipynb](tutorial.ipynb), [exercise.ipynb](exercise.ipynb)) or Python scripts ([tutorial.py](tutorial.py), [exercise.py](exercise.py))
63+
* Datasets in the `data` subdirectory ([sms.tsv](data/sms.tsv), [yelp.csv](data/yelp.csv))
64+
* Required software for today:
65+
* [scikit-learn](http://scikit-learn.org/stable/install.html) and [pandas](http://pandas.pydata.org/pandas-docs/stable/install.html) (and their dependencies)
66+
* [Anaconda distribution of Python](https://www.continuum.io/downloads) is an easy way to install both of these
67+
* Both Python 2 and 3 are welcome
68+
* Flash drives are available with Anaconda installers and tutorial files
69+
* About me:
70+
* Founder of Data School: [blog](http://www.dataschool.io/), [YouTube](https://youtube.com/user/dataschool)
71+
* Twitter: [@justmarkham](https://twitter.com/justmarkham)
72+
* Email: [[email protected]](mailto:[email protected])
73+
* How the tutorial will work
74+
* What we'll be learning today
75+
* What I expect you already know
76+
* Agenda
77+
78+
### Related Resources
79+
80+
**Text classification:**
81+
* Read Paul Graham's classic post, [A Plan for Spam](http://www.paulgraham.com/spam.html), for an overview of a basic text classification system using a Bayesian approach. (He also wrote a [follow-up post](http://www.paulgraham.com/better.html) about how he improved his spam filter.)
82+
* Coursera's Natural Language Processing (NLP) course has [video lectures](https://class.coursera.org/nlp/lecture) on text classification, tokenization, Naive Bayes, and many other fundamental NLP topics. (Here are the [slides](http://web.stanford.edu/~jurafsky/NLPCourseraSlides.html) used in all of the videos.)
83+
* [Automatically Categorizing Yelp Businesses](http://engineeringblog.yelp.com/2015/09/automatically-categorizing-yelp-businesses.html) discusses how Yelp uses NLP and scikit-learn to solve the problem of uncategorized businesses.
84+
* [How to Read the Mind of a Supreme Court Justice](http://fivethirtyeight.com/features/how-to-read-the-mind-of-a-supreme-court-justice/) discusses CourtCast, a machine learning model that predicts the outcome of Supreme Court cases using text-based features only. (The CourtCast creator wrote a post explaining [how it works](https://sciencecowboy.wordpress.com/2015/03/05/predicting-the-supreme-court-from-oral-arguments/), and the [Python code](https://github.com/nasrallah/CourtCast) is available on GitHub.)
85+
* [Identifying Humorous Cartoon Captions](http://www.cs.huji.ac.il/~dshahaf/pHumor.pdf) is a readable paper about identifying funny captions submitted to the New Yorker Caption Contest.
86+
* In this [PyData video](https://www.youtube.com/watch?v=y3ZTKFZ-1QQ) (50 minutes), Facebook explains how they use scikit-learn for sentiment classification by training a Naive Bayes model on emoji-labeled data.
87+
88+
**Naive Bayes and logistic regression:**
89+
* Read this brief Quora post on [airport security](http://www.quora.com/In-laymans-terms-how-does-Naive-Bayes-work/answer/Konstantin-Tt) for an intuitive explanation of how Naive Bayes classification works.
90+
* For a longer introduction to Naive Bayes, read Sebastian Raschka's article on [Naive Bayes and Text Classification](http://sebastianraschka.com/Articles/2014_naive_bayes_1.html). As well, Wikipedia has two excellent articles ([Naive Bayes classifier](http://en.wikipedia.org/wiki/Naive_Bayes_classifier) and [Naive Bayes spam filtering](http://en.wikipedia.org/wiki/Naive_Bayes_spam_filtering)), and Cross Validated has a good [Q&A](http://stats.stackexchange.com/questions/21822/understanding-naive-bayes).
91+
* My [guide to an in-depth understanding of logistic regression](http://www.dataschool.io/guide-to-logistic-regression/) includes a lesson notebook and a curated list of resources for going deeper into this topic.
92+
* [Comparison of Machine Learning Models](https://github.com/justmarkham/DAT8/blob/master/other/model_comparison.md) lists the advantages and disadvantages of Naive Bayes, logistic regression, and other classification and regression models.
93+
94+
**scikit-learn:**
95+
* The scikit-learn user guide includes an excellent section on [text feature extraction](http://scikit-learn.org/stable/modules/feature_extraction.html#text-feature-extraction) that includes many details not covered in today's tutorial.
96+
* The user guide also describes the [performance trade-offs](http://scikit-learn.org/stable/modules/computational_performance.html#influence-of-the-input-data-representation) involved when choosing between sparse and dense input data representations.
97+
* To learn more about evaluating classification models, watch video #9 from my [scikit-learn video series](https://github.com/justmarkham/scikit-learn-videos) (or just read the associated [notebook](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb)).
98+
99+
**pandas:**
100+
* Here are my [top 8 resources for learning data analysis with pandas](http://www.dataschool.io/best-python-pandas-resources/).
101+
* As well, I have a new [pandas Q&A video series](http://www.dataschool.io/easier-data-analysis-with-pandas/) targeted at beginners that includes two new videos every week.

‎exercise.ipynb

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Tutorial Exercise: Yelp reviews"
8+
]
9+
},
10+
{
11+
"cell_type": "markdown",
12+
"metadata": {},
13+
"source": [
14+
"## Introduction\n",
15+
"\n",
16+
"This exercise uses a small subset of the data from Kaggle's [Yelp Business Rating Prediction](https://www.kaggle.com/c/yelp-recsys-2013) competition.\n",
17+
"\n",
18+
"**Description of the data:**\n",
19+
"\n",
20+
"- **`yelp.csv`** contains the dataset. It is stored in the repository (in the **`data`** directory), so there is no need to download anything from the Kaggle website.\n",
21+
"- Each observation (row) in this dataset is a review of a particular business by a particular user.\n",
22+
"- The **stars** column is the number of stars (1 through 5) assigned by the reviewer to the business. (Higher stars is better.) In other words, it is the rating of the business by the person who wrote the review.\n",
23+
"- The **text** column is the text of the review.\n",
24+
"\n",
25+
"**Goal:** Predict the star rating of a review using **only** the review text.\n",
26+
"\n",
27+
"**Tip:** After each task, I recommend that you check the shape and the contents of your objects, to confirm that they match your expectations."
28+
]
29+
},
30+
{
31+
"cell_type": "markdown",
32+
"metadata": {},
33+
"source": [
34+
"## Task 1\n",
35+
"\n",
36+
"Read **`yelp.csv`** into a pandas DataFrame and examine it."
37+
]
38+
},
39+
{
40+
"cell_type": "markdown",
41+
"metadata": {},
42+
"source": [
43+
"## Task 2\n",
44+
"\n",
45+
"Create a new DataFrame that only contains the **5-star** and **1-star** reviews.\n",
46+
"\n",
47+
"- **Hint:** [How do I apply multiple filter criteria to a pandas DataFrame?](http://nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb#9.-How-do-I-apply-multiple-filter-criteria-to-a-pandas-DataFrame%3F-%28video%29) explains how to do this."
48+
]
49+
},
50+
{
51+
"cell_type": "markdown",
52+
"metadata": {},
53+
"source": [
54+
"## Task 3\n",
55+
"\n",
56+
"Define X and y from the new DataFrame, and then split X and y into training and testing sets, using the **review text** as the only feature and the **star rating** as the response.\n",
57+
"\n",
58+
"- **Hint:** Keep in mind that X should be a pandas Series (not a DataFrame), since we will pass it to CountVectorizer in the task that follows."
59+
]
60+
},
61+
{
62+
"cell_type": "markdown",
63+
"metadata": {},
64+
"source": [
65+
"## Task 4\n",
66+
"\n",
67+
"Use CountVectorizer to create **document-term matrices** from X_train and X_test."
68+
]
69+
},
70+
{
71+
"cell_type": "markdown",
72+
"metadata": {},
73+
"source": [
74+
"## Task 5\n",
75+
"\n",
76+
"Use multinomial Naive Bayes to **predict the star rating** for the reviews in the testing set, and then **calculate the accuracy** and **print the confusion matrix**.\n",
77+
"\n",
78+
"- **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains how to interpret both classification accuracy and the confusion matrix."
79+
]
80+
},
81+
{
82+
"cell_type": "markdown",
83+
"metadata": {},
84+
"source": [
85+
"## Task 6 (Challenge)\n",
86+
"\n",
87+
"Calculate the **null accuracy**, which is the classification accuracy that could be achieved by always predicting the most frequent class.\n",
88+
"\n",
89+
"- **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains null accuracy and demonstrates two ways to calculate it, though only one of those ways will work in this case. Alternatively, you can come up with your own method to calculate null accuracy!"
90+
]
91+
},
92+
{
93+
"cell_type": "markdown",
94+
"metadata": {},
95+
"source": [
96+
"## Task 7 (Challenge)\n",
97+
"\n",
98+
"Browse through the review text of some of the **false positives** and **false negatives**. Based on your knowledge of how Naive Bayes works, do you have any ideas about why the model is incorrectly classifying these reviews?\n",
99+
"\n",
100+
"- **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains the definitions of \"false positives\" and \"false negatives\".\n",
101+
"- **Hint:** Think about what a false positive means in this context, and what a false negative means in this context. What has scikit-learn defined as the \"positive class\"?"
102+
]
103+
},
104+
{
105+
"cell_type": "markdown",
106+
"metadata": {},
107+
"source": [
108+
"## Task 8 (Challenge)\n",
109+
"\n",
110+
"Calculate which 10 tokens are the most predictive of **5-star reviews**, and which 10 tokens are the most predictive of **1-star reviews**.\n",
111+
"\n",
112+
"- **Hint:** Naive Bayes automatically counts the number of times each token appears in each class, as well as the number of observations in each class. You can access these counts via the `feature_count_` and `class_count_` attributes of the Naive Bayes model object."
113+
]
114+
},
115+
{
116+
"cell_type": "markdown",
117+
"metadata": {},
118+
"source": [
119+
"## Task 9 (Challenge)\n",
120+
"\n",
121+
"Up to this point, we have framed this as a **binary classification problem** by only considering the 5-star and 1-star reviews. Now, let's repeat the model building process using all reviews, which makes this a **5-class classification problem**.\n",
122+
"\n",
123+
"Here are the steps:\n",
124+
"\n",
125+
"- Define X and y using the original DataFrame. (y should contain 5 different classes.)\n",
126+
"- Split X and y into training and testing sets.\n",
127+
"- Create document-term matrices using CountVectorizer.\n",
128+
"- Calculate the testing accuracy of a Multinomial Naive Bayes model.\n",
129+
"- Compare the testing accuracy with the null accuracy, and comment on the results.\n",
130+
"- Print the confusion matrix, and comment on the results. (This [Stack Overflow answer](http://stackoverflow.com/a/30748053/1636598) explains how to read a multi-class confusion matrix.)\n",
131+
"- Print the [classification report](http://scikit-learn.org/stable/modules/model_evaluation.html#classification-report), and comment on the results. If you are unfamiliar with the terminology it uses, research the terms, and then try to figure out how to calculate these metrics manually from the confusion matrix!"
132+
]
133+
}
134+
],
135+
"metadata": {
136+
"kernelspec": {
137+
"display_name": "Python 2",
138+
"language": "python",
139+
"name": "python2"
140+
},
141+
"language_info": {
142+
"codemirror_mode": {
143+
"name": "ipython",
144+
"version": 2
145+
},
146+
"file_extension": ".py",
147+
"mimetype": "text/x-python",
148+
"name": "python",
149+
"nbconvert_exporter": "python",
150+
"pygments_lexer": "ipython2",
151+
"version": "2.7.11"
152+
}
153+
},
154+
"nbformat": 4,
155+
"nbformat_minor": 0
156+
}

‎exercise.py

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# # Tutorial Exercise: Yelp reviews
2+
3+
# ## Introduction
4+
#
5+
# This exercise uses a small subset of the data from Kaggle's [Yelp Business Rating Prediction](https://www.kaggle.com/c/yelp-recsys-2013) competition.
6+
#
7+
# **Description of the data:**
8+
#
9+
# - **`yelp.csv`** contains the dataset. It is stored in the repository (in the **`data`** directory), so there is no need to download anything from the Kaggle website.
10+
# - Each observation (row) in this dataset is a review of a particular business by a particular user.
11+
# - The **stars** column is the number of stars (1 through 5) assigned by the reviewer to the business. (Higher stars is better.) In other words, it is the rating of the business by the person who wrote the review.
12+
# - The **text** column is the text of the review.
13+
#
14+
# **Goal:** Predict the star rating of a review using **only** the review text.
15+
#
16+
# **Tip:** After each task, I recommend that you check the shape and the contents of your objects, to confirm that they match your expectations.
17+
18+
# ## Task 1
19+
#
20+
# Read **`yelp.csv`** into a pandas DataFrame and examine it.
21+
22+
# ## Task 2
23+
#
24+
# Create a new DataFrame that only contains the **5-star** and **1-star** reviews.
25+
#
26+
# - **Hint:** [How do I apply multiple filter criteria to a pandas DataFrame?](http://nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb#9.-How-do-I-apply-multiple-filter-criteria-to-a-pandas-DataFrame%3F-%28video%29) explains how to do this.
27+
28+
# ## Task 3
29+
#
30+
# Define X and y from the new DataFrame, and then split X and y into training and testing sets, using the **review text** as the only feature and the **star rating** as the response.
31+
#
32+
# - **Hint:** Keep in mind that X should be a pandas Series (not a DataFrame), since we will pass it to CountVectorizer in the task that follows.
33+
34+
# ## Task 4
35+
#
36+
# Use CountVectorizer to create **document-term matrices** from X_train and X_test.
37+
38+
# ## Task 5
39+
#
40+
# Use multinomial Naive Bayes to **predict the star rating** for the reviews in the testing set, and then **calculate the accuracy** and **print the confusion matrix**.
41+
#
42+
# - **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains how to interpret both classification accuracy and the confusion matrix.
43+
44+
# ## Task 6 (Challenge)
45+
#
46+
# Calculate the **null accuracy**, which is the classification accuracy that could be achieved by always predicting the most frequent class.
47+
#
48+
# - **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains null accuracy and demonstrates two ways to calculate it, though only one of those ways will work in this case. Alternatively, you can come up with your own method to calculate null accuracy!
49+
50+
# ## Task 7 (Challenge)
51+
#
52+
# Browse through the review text of some of the **false positives** and **false negatives**. Based on your knowledge of how Naive Bayes works, do you have any ideas about why the model is incorrectly classifying these reviews?
53+
#
54+
# - **Hint:** [Evaluating a classification model](https://github.com/justmarkham/scikit-learn-videos/blob/master/09_classification_metrics.ipynb) explains the definitions of "false positives" and "false negatives".
55+
# - **Hint:** Think about what a false positive means in this context, and what a false negative means in this context. What has scikit-learn defined as the "positive class"?
56+
57+
# ## Task 8 (Challenge)
58+
#
59+
# Calculate which 10 tokens are the most predictive of **5-star reviews**, and which 10 tokens are the most predictive of **1-star reviews**.
60+
#
61+
# - **Hint:** Naive Bayes automatically counts the number of times each token appears in each class, as well as the number of observations in each class. You can access these counts via the `feature_count_` and `class_count_` attributes of the Naive Bayes model object.
62+
63+
# ## Task 9 (Challenge)
64+
#
65+
# Up to this point, we have framed this as a **binary classification problem** by only considering the 5-star and 1-star reviews. Now, let's repeat the model building process using all reviews, which makes this a **5-class classification problem**.
66+
#
67+
# Here are the steps:
68+
#
69+
# - Define X and y using the original DataFrame. (y should contain 5 different classes.)
70+
# - Split X and y into training and testing sets.
71+
# - Create document-term matrices using CountVectorizer.
72+
# - Calculate the testing accuracy of a Multinomial Naive Bayes model.
73+
# - Compare the testing accuracy with the null accuracy, and comment on the results.
74+
# - Print the confusion matrix, and comment on the results. (This [Stack Overflow answer](http://stackoverflow.com/a/30748053/1636598) explains how to read a multi-class confusion matrix.)
75+
# - Print the [classification report](http://scikit-learn.org/stable/modules/model_evaluation.html#classification-report), and comment on the results. If you are unfamiliar with the terminology it uses, research the terms, and then try to figure out how to calculate these metrics manually from the confusion matrix!

0 commit comments

Comments
 (0)