tweet_location.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Off-Platform Project: Classifying Tweets\n",
    "\n",
    "In this off-platform project, you will use a Naive Bayes Classifier to find patterns in real tweets. We've given you three files: `new_york.json`, `london.json`, and `paris.json`. These three files contain tweets that we gathered from those locations.\n",
    "\n",
    "The goal is to create a classification algorithm that can classify any tweet (or sentence) and predict whether that sentence came from New York, London, or Paris."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Investigate the Data\n",
    "\n",
    "To begin, let's take a look at the data. We've imported `new_york.json` and printed the following information:\n",
    "* The number of tweets.\n",
    "* The columns, or features, of a tweet.\n",
    "* The text of the 12th tweet in the New York dataset.\n",
    "\n",
    "Run the cell below to see the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4723\n",
      "Index(['created_at', 'id', 'id_str', 'text', 'display_text_range', 'source',\n",
      "       'truncated', 'in_reply_to_status_id', 'in_reply_to_status_id_str',\n",
      "       'in_reply_to_user_id', 'in_reply_to_user_id_str',\n",
      "       'in_reply_to_screen_name', 'user', 'geo', 'coordinates', 'place',\n",
      "       'contributors', 'is_quote_status', 'quote_count', 'reply_count',\n",
      "       'retweet_count', 'favorite_count', 'entities', 'favorited', 'retweeted',\n",
      "       'filter_level', 'lang', 'timestamp_ms', 'extended_tweet',\n",
      "       'possibly_sensitive', 'quoted_status_id', 'quoted_status_id_str',\n",
      "       'quoted_status', 'quoted_status_permalink', 'extended_entities',\n",
      "       'withheld_in_countries'],\n",
      "      dtype='object')\n",
      "Be best #ThursdayThoughts\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>id_str</th>\n",
       "      <th>text</th>\n",
       "      <th>display_text_range</th>\n",
       "      <th>source</th>\n",
       "      <th>truncated</th>\n",
       "      <th>in_reply_to_status_id</th>\n",
       "      <th>in_reply_to_status_id_str</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>...</th>\n",
       "      <th>lang</th>\n",
       "      <th>timestamp_ms</th>\n",
       "      <th>extended_tweet</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>quoted_status_id</th>\n",
       "      <th>quoted_status_id_str</th>\n",
       "      <th>quoted_status</th>\n",
       "      <th>quoted_status_permalink</th>\n",
       "      <th>extended_entities</th>\n",
       "      <th>withheld_in_countries</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-26 13:32:33+00:00</td>\n",
       "      <td>1022474755625164800</td>\n",
       "      <td>1022474755625164800</td>\n",
       "      <td>@DelgadoforNY19 Calendar marked.</td>\n",
       "      <td>[16, 32]</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>False</td>\n",
       "      <td>1.022208e+18</td>\n",
       "      <td>1.022208e+18</td>\n",
       "      <td>8.290618e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:32:33.060</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-26 13:32:34+00:00</td>\n",
       "      <td>1022474762491183104</td>\n",
       "      <td>1022474762491183104</td>\n",
       "      <td>petition to ban more than one spritz of cologne</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:32:34.697</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-26 13:32:35+00:00</td>\n",
       "      <td>1022474765750226945</td>\n",
       "      <td>1022474765750226944</td>\n",
       "      <td>People really be making up beef with you in th...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:32:35.474</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-26 13:32:36+00:00</td>\n",
       "      <td>1022474768736546816</td>\n",
       "      <td>1022474768736546816</td>\n",
       "      <td>30 years old.. wow what a journey... I moved t...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://instagram.com\" rel=\"nofollow\"&gt;...</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:32:36.186</td>\n",
       "      <td>{'full_text': '30 years old.. wow what a journ...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-26 13:32:36+00:00</td>\n",
       "      <td>1022474769260838913</td>\n",
       "      <td>1022474769260838912</td>\n",
       "      <td>At first glance it looked like asparagus with ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:32:36.311</td>\n",
       "      <td>{'full_text': 'At first glance it looked like ...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 36 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 created_at                   id               id_str  \\\n",
       "0 2018-07-26 13:32:33+00:00  1022474755625164800  1022474755625164800   \n",
       "1 2018-07-26 13:32:34+00:00  1022474762491183104  1022474762491183104   \n",
       "2 2018-07-26 13:32:35+00:00  1022474765750226945  1022474765750226944   \n",
       "3 2018-07-26 13:32:36+00:00  1022474768736546816  1022474768736546816   \n",
       "4 2018-07-26 13:32:36+00:00  1022474769260838913  1022474769260838912   \n",
       "\n",
       "                                                text display_text_range  \\\n",
       "0                   @DelgadoforNY19 Calendar marked.           [16, 32]   \n",
       "1    petition to ban more than one spritz of cologne                NaN   \n",
       "2  People really be making up beef with you in th...                NaN   \n",
       "3  30 years old.. wow what a journey... I moved t...                NaN   \n",
       "4  At first glance it looked like asparagus with ...                NaN   \n",
       "\n",
       "                                              source  truncated  \\\n",
       "0  <a href=\"http://twitter.com/download/android\" ...      False   \n",
       "1  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "2  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "3  <a href=\"http://instagram.com\" rel=\"nofollow\">...       True   \n",
       "4  <a href=\"http://twitter.com/download/iphone\" r...       True   \n",
       "\n",
       "   in_reply_to_status_id  in_reply_to_status_id_str  in_reply_to_user_id  ...  \\\n",
       "0           1.022208e+18               1.022208e+18         8.290618e+17  ...   \n",
       "1                    NaN                        NaN                  NaN  ...   \n",
       "2                    NaN                        NaN                  NaN  ...   \n",
       "3                    NaN                        NaN                  NaN  ...   \n",
       "4                    NaN                        NaN                  NaN  ...   \n",
       "\n",
       "   lang            timestamp_ms  \\\n",
       "0    en 2018-07-26 13:32:33.060   \n",
       "1    en 2018-07-26 13:32:34.697   \n",
       "2    en 2018-07-26 13:32:35.474   \n",
       "3    en 2018-07-26 13:32:36.186   \n",
       "4    en 2018-07-26 13:32:36.311   \n",
       "\n",
       "                                      extended_tweet possibly_sensitive  \\\n",
       "0                                                NaN                NaN   \n",
       "1                                                NaN                NaN   \n",
       "2                                                NaN                NaN   \n",
       "3  {'full_text': '30 years old.. wow what a journ...                0.0   \n",
       "4  {'full_text': 'At first glance it looked like ...                NaN   \n",
       "\n",
       "  quoted_status_id quoted_status_id_str  quoted_status  \\\n",
       "0              NaN                  NaN            NaN   \n",
       "1              NaN                  NaN            NaN   \n",
       "2              NaN                  NaN            NaN   \n",
       "3              NaN                  NaN            NaN   \n",
       "4              NaN                  NaN            NaN   \n",
       "\n",
       "   quoted_status_permalink  extended_entities  withheld_in_countries  \n",
       "0                      NaN                NaN                    NaN  \n",
       "1                      NaN                NaN                    NaN  \n",
       "2                      NaN                NaN                    NaN  \n",
       "3                      NaN                NaN                    NaN  \n",
       "4                      NaN                NaN                    NaN  \n",
       "\n",
       "[5 rows x 36 columns]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "new_york_tweets = pd.read_json(\"new_york.json\", lines=True)\n",
    "print(len(new_york_tweets))\n",
    "print(new_york_tweets.columns)\n",
    "print(new_york_tweets.loc[12][\"text\"])\n",
    "new_york_tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code block below, load the London and Paris tweets into DataFrames named `london_tweets` and `paris_tweets`.\n",
    "\n",
    "How many London tweets are there? How many Paris ones are there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5341\n",
      "I saw this on the BBC and thought you should see it:\n",
      "\n",
      "The precious metal sparking a new gold rush - https://t.co/ScW4MOSobZ\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>id_str</th>\n",
       "      <th>text</th>\n",
       "      <th>display_text_range</th>\n",
       "      <th>source</th>\n",
       "      <th>truncated</th>\n",
       "      <th>in_reply_to_status_id</th>\n",
       "      <th>in_reply_to_status_id_str</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>...</th>\n",
       "      <th>retweeted</th>\n",
       "      <th>filter_level</th>\n",
       "      <th>lang</th>\n",
       "      <th>timestamp_ms</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>quoted_status_id</th>\n",
       "      <th>quoted_status_id_str</th>\n",
       "      <th>quoted_status</th>\n",
       "      <th>quoted_status_permalink</th>\n",
       "      <th>extended_entities</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-26 13:39:30+00:00</td>\n",
       "      <td>1022476504855400449</td>\n",
       "      <td>1022476504855400448</td>\n",
       "      <td>@bbclaurak i agree Laura but the Party you see...</td>\n",
       "      <td>[11, 140]</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>True</td>\n",
       "      <td>1.022447e+18</td>\n",
       "      <td>1.022447e+18</td>\n",
       "      <td>6.118357e+07</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>low</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:39:30.109</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-26 13:39:30+00:00</td>\n",
       "      <td>1022476506075942912</td>\n",
       "      <td>1022476506075942912</td>\n",
       "      <td>@masturbacaolove Why?</td>\n",
       "      <td>[17, 21]</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>1.021997e+18</td>\n",
       "      <td>1.021997e+18</td>\n",
       "      <td>9.003777e+17</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>low</td>\n",
       "      <td>und</td>\n",
       "      <td>2018-07-26 13:39:30.400</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-26 13:39:31+00:00</td>\n",
       "      <td>1022476510089949190</td>\n",
       "      <td>1022476510089949184</td>\n",
       "      <td>@JackRobinson80 @pgroresearch Yeah not great b...</td>\n",
       "      <td>[30, 65]</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>1.022444e+18</td>\n",
       "      <td>1.022444e+18</td>\n",
       "      <td>7.355633e+08</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>low</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:39:31.357</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-26 13:39:33+00:00</td>\n",
       "      <td>1022476519845883905</td>\n",
       "      <td>1022476519845883904</td>\n",
       "      <td>Penalty shit out Arsenal</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>low</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:39:33.683</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-26 13:39:36+00:00</td>\n",
       "      <td>1022476532684648448</td>\n",
       "      <td>1022476532684648448</td>\n",
       "      <td>Obviously need some pen practice 🙈</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>False</td>\n",
       "      <td>low</td>\n",
       "      <td>en</td>\n",
       "      <td>2018-07-26 13:39:36.744</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 created_at                   id               id_str  \\\n",
       "0 2018-07-26 13:39:30+00:00  1022476504855400449  1022476504855400448   \n",
       "1 2018-07-26 13:39:30+00:00  1022476506075942912  1022476506075942912   \n",
       "2 2018-07-26 13:39:31+00:00  1022476510089949190  1022476510089949184   \n",
       "3 2018-07-26 13:39:33+00:00  1022476519845883905  1022476519845883904   \n",
       "4 2018-07-26 13:39:36+00:00  1022476532684648448  1022476532684648448   \n",
       "\n",
       "                                                text display_text_range  \\\n",
       "0  @bbclaurak i agree Laura but the Party you see...          [11, 140]   \n",
       "1                              @masturbacaolove Why?           [17, 21]   \n",
       "2  @JackRobinson80 @pgroresearch Yeah not great b...           [30, 65]   \n",
       "3                           Penalty shit out Arsenal                NaN   \n",
       "4                 Obviously need some pen practice 🙈                NaN   \n",
       "\n",
       "                                              source  truncated  \\\n",
       "0  <a href=\"http://twitter.com/download/iphone\" r...       True   \n",
       "1  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "2  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "3  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "4  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "\n",
       "   in_reply_to_status_id  in_reply_to_status_id_str  in_reply_to_user_id  ...  \\\n",
       "0           1.022447e+18               1.022447e+18         6.118357e+07  ...   \n",
       "1           1.021997e+18               1.021997e+18         9.003777e+17  ...   \n",
       "2           1.022444e+18               1.022444e+18         7.355633e+08  ...   \n",
       "3                    NaN                        NaN                  NaN  ...   \n",
       "4                    NaN                        NaN                  NaN  ...   \n",
       "\n",
       "   retweeted filter_level lang            timestamp_ms possibly_sensitive  \\\n",
       "0      False          low   en 2018-07-26 13:39:30.109                NaN   \n",
       "1      False          low  und 2018-07-26 13:39:30.400                NaN   \n",
       "2      False          low   en 2018-07-26 13:39:31.357                NaN   \n",
       "3      False          low   en 2018-07-26 13:39:33.683                NaN   \n",
       "4      False          low   en 2018-07-26 13:39:36.744                NaN   \n",
       "\n",
       "  quoted_status_id  quoted_status_id_str  quoted_status  \\\n",
       "0              NaN                   NaN            NaN   \n",
       "1              NaN                   NaN            NaN   \n",
       "2              NaN                   NaN            NaN   \n",
       "3              NaN                   NaN            NaN   \n",
       "4              NaN                   NaN            NaN   \n",
       "\n",
       "  quoted_status_permalink  extended_entities  \n",
       "0                     NaN                NaN  \n",
       "1                     NaN                NaN  \n",
       "2                     NaN                NaN  \n",
       "3                     NaN                NaN  \n",
       "4                     NaN                NaN  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "london_tweets = pd.read_json(\"london.json\", lines=True)\n",
    "print(len(london_tweets))\n",
    "print(london_tweets.loc[12][\"text\"])\n",
    "london_tweets.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2510\n",
      "Hauts-de-Seine : l’incendie d’Issy-les-Moulineaux prive aussi 16 000 foyers de courant https://t.co/Hlb02Fpliy\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>created_at</th>\n",
       "      <th>id</th>\n",
       "      <th>id_str</th>\n",
       "      <th>text</th>\n",
       "      <th>source</th>\n",
       "      <th>truncated</th>\n",
       "      <th>in_reply_to_status_id</th>\n",
       "      <th>in_reply_to_status_id_str</th>\n",
       "      <th>in_reply_to_user_id</th>\n",
       "      <th>in_reply_to_user_id_str</th>\n",
       "      <th>...</th>\n",
       "      <th>lang</th>\n",
       "      <th>timestamp_ms</th>\n",
       "      <th>display_text_range</th>\n",
       "      <th>extended_entities</th>\n",
       "      <th>possibly_sensitive</th>\n",
       "      <th>quoted_status_id</th>\n",
       "      <th>quoted_status_id_str</th>\n",
       "      <th>quoted_status</th>\n",
       "      <th>quoted_status_permalink</th>\n",
       "      <th>extended_tweet</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2018-07-27 17:40:45+00:00</td>\n",
       "      <td>1022899608396156928</td>\n",
       "      <td>1022899608396156928</td>\n",
       "      <td>Bulletin météo parisien : des grêlons énormes ...</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>fr</td>\n",
       "      <td>2018-07-27 17:40:45.854</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2018-07-27 17:40:47+00:00</td>\n",
       "      <td>1022899613550956544</td>\n",
       "      <td>1022899613550956544</td>\n",
       "      <td>Prêt pour le match #USORCL https://t.co/V5jw0S...</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>fr</td>\n",
       "      <td>2018-07-27 17:40:47.083</td>\n",
       "      <td>[0, 26]</td>\n",
       "      <td>{'media': [{'id': 1022899599336525825, 'id_str...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2018-07-27 17:40:50+00:00</td>\n",
       "      <td>1022899626041651200</td>\n",
       "      <td>1022899626041651200</td>\n",
       "      <td>MAIS QOIDBDNND'SLS'SLSLLSLS''D DBDODNDNODJDBKD...</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>in</td>\n",
       "      <td>2018-07-27 17:40:50.061</td>\n",
       "      <td>[0, 111]</td>\n",
       "      <td>{'media': [{'id': 1022899571884744706, 'id_str...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2018-07-27 17:40:57+00:00</td>\n",
       "      <td>1022899655347249152</td>\n",
       "      <td>1022899655347249152</td>\n",
       "      <td>@ToursFC Où peut on le championnat de National...</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/android\" ...</td>\n",
       "      <td>False</td>\n",
       "      <td>1.022888e+18</td>\n",
       "      <td>1.022888e+18</td>\n",
       "      <td>978599220.0</td>\n",
       "      <td>978599220.0</td>\n",
       "      <td>...</td>\n",
       "      <td>fr</td>\n",
       "      <td>2018-07-27 17:40:57.048</td>\n",
       "      <td>[9, 50]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018-07-27 17:40:57+00:00</td>\n",
       "      <td>1022899656685223936</td>\n",
       "      <td>1022899656685223936</td>\n",
       "      <td>Les tismey ils sont bas qu’a tromper leur go e...</td>\n",
       "      <td>&lt;a href=\"http://twitter.com/download/iphone\" r...</td>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>fr</td>\n",
       "      <td>2018-07-27 17:40:57.367</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 35 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                 created_at                   id               id_str  \\\n",
       "0 2018-07-27 17:40:45+00:00  1022899608396156928  1022899608396156928   \n",
       "1 2018-07-27 17:40:47+00:00  1022899613550956544  1022899613550956544   \n",
       "2 2018-07-27 17:40:50+00:00  1022899626041651200  1022899626041651200   \n",
       "3 2018-07-27 17:40:57+00:00  1022899655347249152  1022899655347249152   \n",
       "4 2018-07-27 17:40:57+00:00  1022899656685223936  1022899656685223936   \n",
       "\n",
       "                                                text  \\\n",
       "0  Bulletin météo parisien : des grêlons énormes ...   \n",
       "1  Prêt pour le match #USORCL https://t.co/V5jw0S...   \n",
       "2  MAIS QOIDBDNND'SLS'SLSLLSLS''D DBDODNDNODJDBKD...   \n",
       "3  @ToursFC Où peut on le championnat de National...   \n",
       "4  Les tismey ils sont bas qu’a tromper leur go e...   \n",
       "\n",
       "                                              source  truncated  \\\n",
       "0  <a href=\"http://twitter.com/download/android\" ...      False   \n",
       "1  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "2  <a href=\"http://twitter.com/download/android\" ...      False   \n",
       "3  <a href=\"http://twitter.com/download/android\" ...      False   \n",
       "4  <a href=\"http://twitter.com/download/iphone\" r...      False   \n",
       "\n",
       "   in_reply_to_status_id  in_reply_to_status_id_str  in_reply_to_user_id  \\\n",
       "0                    NaN                        NaN                  NaN   \n",
       "1                    NaN                        NaN                  NaN   \n",
       "2                    NaN                        NaN                  NaN   \n",
       "3           1.022888e+18               1.022888e+18          978599220.0   \n",
       "4                    NaN                        NaN                  NaN   \n",
       "\n",
       "   in_reply_to_user_id_str  ... lang            timestamp_ms  \\\n",
       "0                      NaN  ...   fr 2018-07-27 17:40:45.854   \n",
       "1                      NaN  ...   fr 2018-07-27 17:40:47.083   \n",
       "2                      NaN  ...   in 2018-07-27 17:40:50.061   \n",
       "3              978599220.0  ...   fr 2018-07-27 17:40:57.048   \n",
       "4                      NaN  ...   fr 2018-07-27 17:40:57.367   \n",
       "\n",
       "  display_text_range                                  extended_entities  \\\n",
       "0                NaN                                                NaN   \n",
       "1            [0, 26]  {'media': [{'id': 1022899599336525825, 'id_str...   \n",
       "2           [0, 111]  {'media': [{'id': 1022899571884744706, 'id_str...   \n",
       "3            [9, 50]                                                NaN   \n",
       "4                NaN                                                NaN   \n",
       "\n",
       "  possibly_sensitive  quoted_status_id  quoted_status_id_str  quoted_status  \\\n",
       "0                NaN               NaN                   NaN            NaN   \n",
       "1                0.0               NaN                   NaN            NaN   \n",
       "2                0.0               NaN                   NaN            NaN   \n",
       "3                NaN               NaN                   NaN            NaN   \n",
       "4                NaN               NaN                   NaN            NaN   \n",
       "\n",
       "   quoted_status_permalink  extended_tweet  \n",
       "0                      NaN             NaN  \n",
       "1                      NaN             NaN  \n",
       "2                      NaN             NaN  \n",
       "3                      NaN             NaN  \n",
       "4                      NaN             NaN  \n",
       "\n",
       "[5 rows x 35 columns]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "paris_tweets = pd.read_json(\"paris.json\", lines=True)\n",
    "print(len(paris_tweets))\n",
    "print(paris_tweets.loc[12][\"text\"])\n",
    "paris_tweets.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifying using language: Naive Bayes Classifier\n",
    "\n",
    "We're going to create a Naive Bayes Classifier! Let's begin by looking at the way language is used differently in these three locations. Let's grab the text of all of the tweets and make it one big list. In the code block below, we've created a list of all the New York tweets. Do the same for `london_tweets` and `paris_tweets`.\n",
    "\n",
    "Then combine all three into a variable named `all_tweets` by using the `+` operator. For example, `all_tweets = new_york_text + london_text + ...`\n",
    "\n",
    "Let's also make the labels associated with those tweets. `0` represents a New York tweet, `1`  represents a London tweet, and `2` represents a Paris tweet. Finish the definition of `labels`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_york_text = new_york_tweets[\"text\"].tolist()\n",
    "london_text = london_tweets['text'].tolist()\n",
    "paris_text = paris_tweets['text'].tolist()\n",
    "\n",
    "all_tweets = new_york_text + london_text + paris_text\n",
    "labels = [0] * len(new_york_text) + [1] * len(london_text) + [2] * len(paris_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making a Training and Test Set\n",
    "\n",
    "We can now break our data into a training set and a test set. We'll use scikit-learn's `train_test_split` function to do this split. This function takes two required parameters: It takes the data, followed by the labels. Set the optional parameter `test_size` to be `0.2`. Finally, set the optional parameter `random_state` to `1`. This will make it so your data is split in the same way as the data in our solution code. \n",
    "\n",
    "Remember, this function returns 4 items in this order:\n",
    "1. The training data\n",
    "2. The testing data\n",
    "3. The training labels\n",
    "4. The testing labels\n",
    "\n",
    "Store the results in variables named `train_data`, `test_data`, `train_labels`, and `test_labels`.\n",
    "\n",
    "Print the length of `train_data` and the length of `test_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10059\n",
      "2515\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "train_data, test_data, train_labels, test_labels = train_test_split(all_tweets, labels, test_size=0.2, random_state=1)\n",
    "print(len(train_data))\n",
    "print(len(test_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making the Count Vectors\n",
    "\n",
    "To use a Naive Bayes Classifier, we need to transform our lists of words into count vectors. Recall that this changes the sentence `\"I love New York, New York\"` into a list that contains:\n",
    "\n",
    "* Two `1`s because the words `\"I\"` and `\"love\"` each appear once.\n",
    "* Two `2`s because the words `\"New\"` and `\"York\"` each appear twice.\n",
    "* Many `0`s because every other word in the training set didn't appear at all.\n",
    "\n",
    "To start, create a `CountVectorizer` named `counter`.\n",
    "\n",
    "Next, call the `.fit()` method using `train_data` as a parameter. This teaches the counter our vocabulary.\n",
    "\n",
    "Finally, let's transform `train_data` and `test_data` into Count Vectors. Call `counter`'s `.transform()` method using `train_data` as a parameter and store the result in `train_counts`. Do the same for `test_data` and store the result in `test_counts`.\n",
    "\n",
    "Print `train_data[3]` and `train_counts[3]` to see what a tweet looks like as a Count Vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saying bye is hard. Especially when youre saying bye to comfort.\n",
      "  (0, 5022)\t2\n",
      "  (0, 6371)\t1\n",
      "  (0, 9552)\t1\n",
      "  (0, 12314)\t1\n",
      "  (0, 13903)\t1\n",
      "  (0, 23994)\t2\n",
      "  (0, 27146)\t1\n",
      "  (0, 29397)\t1\n",
      "  (0, 30274)\t1\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "counter = CountVectorizer()\n",
    "counter.fit(train_data)\n",
    "train_counts = counter.transform(train_data)\n",
    "test_counts = counter.transform(test_data)\n",
    "\n",
    "print(train_data[3])\n",
    "print(train_counts[3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train and Test the Naive Bayes Classifier\n",
    "\n",
    "We now have the inputs to our classifier. Let's use the CountVectors to train and test the Naive Bayes Classifier!\n",
    "\n",
    "First, make a `MultinomialNB` named `classifier`.\n",
    "\n",
    "Next, call `classifier`'s `.fit()` method. This method takes two parameters &mdash; the training data and the training labels. `train_counts` contains the training data and `train_labels` containts the labels for that data.\n",
    "\n",
    "Calling `.fit()` calculates all of the probabilities used in Bayes Theorem. The model is now ready to quickly predict the location of a new tweet. \n",
    "\n",
    "Finally, let's test our model. `classifier`'s `.predict()` method using `test_counts` as a parameter. Store the results in a variable named `predictions`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "classifier = MultinomialNB()\n",
    "classifier.fit(train_counts, train_labels)\n",
    "predictions = classifier.predict(test_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluating Your Model\n",
    "\n",
    "Now that the classifier has made its predictions, let's see how well it did. Let's look at two different ways to do this. First, call scikit-learn's `accuracy_score` function. This function should take two parameters &mdash;  the `test_labels` and your `predictions`. Print the results. This prints the percentage of tweets in the test set that the classifier correctly classified.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6779324055666004\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "print(accuracy_score(test_labels, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The other way you can evaluate your model is by looking at the **confusion matrix**. A confusion matrix is a table that describes how your classifier made its predictions. For example, if there were two labels, A and B, a confusion matrix might look like this:\n",
    "\n",
    "```\n",
    "9 1\n",
    "3 5\n",
    "```\n",
    "\n",
    "In this example, the first row shows how the classifier classified the true A's. It guessed that 9 of them were A's and 1 of them was a B. The second row shows how the classifier did on the true B's. It guessed that 3 of them were A's and 5 of them were B's.\n",
    "\n",
    "For our project using tweets, there were three classes &mdash; `0` for New York, `1` for London, and `2` for Paris. You can see the confustion matrix by printing the result of the `confusion_matrix` function using `test_labels` and `predictions` as parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[541 404  28]\n",
      " [203 824  34]\n",
      " [ 38 103 340]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "print(confusion_matrix(test_labels, predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Test Your Own Tweet\n",
    "\n",
    "Nice work! The confusion matrix should line up with your intuition. The classifier predicts tweets that were actually from New York as either New York tweets or London tweets, but almost never Paris tweets. Similarly, the classifier rarely misclassifies the tweets that were actually from Paris. Tweets coming from two English speaking countries are harder to distinguish than tweets in different languages.\n",
    "\n",
    "Now it's your chance to write a tweet and see how the classifier works! Create a string and store it in a variable named `tweet`. \n",
    "\n",
    "Call `counter`'s `.transform()` method using `[tweet]` as a parameter. Save the result as `tweet_counts`. Notice that your variable has to be in an array &mdash; `.transform()` can't take just a string, it must be a list. \n",
    "\n",
    "Finally, pass `tweet_counts` as parameter to `classifier`'s `.predict()` method. Print the result. This should give you the prediction for the tweet. Remember a `0` represents New York, a `1` represents London, and a `2` represents Paris. Can you write different tweets that the classifier predicts as being from New York, London, and Paris?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2]\n"
     ]
    }
   ],
   "source": [
    "tweet = 'oui'\n",
    "\n",
    "tweet_counts = counter.transform([tweet])\n",
    "print(classifier.predict(tweet_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}