Assignment+3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "_You are currently looking at **version 1.0** of this notebook. To download notebooks and datafiles, as well as get help on Jupyter notebooks in the Coursera platform, visit the [Jupyter Notebook FAQ](https://www.coursera.org/learn/python-text-mining/resources/d9pwm) course resource._\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3\n",
    "\n",
    "In this assignment you will explore text message data and create models to predict if a message is spam or not. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Go until jurong point, crazy.. Available only ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Ok lar... Joking wif u oni...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Free entry in 2 a wkly comp to win FA Cup fina...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>U dun say so early hor... U c already then say...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Nah I don't think he goes to usf, he lives aro...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>FreeMsg Hey there darling it's been 3 week's n...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Even my brother is not like to speak with me. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>As per your request 'Melle Melle (Oru Minnamin...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>WINNER!! As a valued network customer you have...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Had your mobile 11 months or more? U R entitle...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text  target\n",
       "0  Go until jurong point, crazy.. Available only ...       0\n",
       "1                      Ok lar... Joking wif u oni...       0\n",
       "2  Free entry in 2 a wkly comp to win FA Cup fina...       1\n",
       "3  U dun say so early hor... U c already then say...       0\n",
       "4  Nah I don't think he goes to usf, he lives aro...       0\n",
       "5  FreeMsg Hey there darling it's been 3 week's n...       1\n",
       "6  Even my brother is not like to speak with me. ...       0\n",
       "7  As per your request 'Melle Melle (Oru Minnamin...       0\n",
       "8  WINNER!! As a valued network customer you have...       1\n",
       "9  Had your mobile 11 months or more? U R entitle...       1"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "spam_data = pd.read_csv('spam.csv')\n",
    "\n",
    "spam_data['target'] = np.where(spam_data['target']=='spam',1,0)\n",
    "spam_data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(spam_data['text'], \n",
    "                                                    spam_data['target'], \n",
    "                                                    random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 1\n",
    "What percentage of the documents in `spam_data` are spam?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_one():\n",
    "    \n",
    "    return len(spam_data[spam_data['target'] == 1]) / len(spam_data) * 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "13.406317300789663"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_one()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 2\n",
    "\n",
    "Fit the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "What is the longest token in the vocabulary?\n",
    "\n",
    "*This function should return a string.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "def answer_two():\n",
    "    import operator\n",
    "\n",
    "    vectorizer = CountVectorizer()\n",
    "    vectorizer.fit(X_train)\n",
    "    \n",
    "    return sorted([(token, len(token)) for token in vectorizer.vocabulary_.keys()], key=operator.itemgetter(1), reverse=True)[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'com1win150ppmx3age16subscription'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_two()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 3\n",
    "\n",
    "Fit and transform the training data `X_train` using a Count Vectorizer with default parameters.\n",
    "\n",
    "Next, fit a fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1`. Find the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "def answer_three():\n",
    "    vectorizer = CountVectorizer()\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "    X_test_transformed = vectorizer.transform(X_test)\n",
    "\n",
    "    clf = MultinomialNB(alpha=0.1)\n",
    "    clf.fit(X_train_transformed, y_train)\n",
    "\n",
    "    y_predicted = clf.predict(X_test_transformed)\n",
    "    \n",
    "    return roc_auc_score(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.97208121827411165"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_three()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 4\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer with default parameters.\n",
    "\n",
    "What 20 features have the smallest tf-idf and what 20 have the largest tf-idf?\n",
    "\n",
    "Put these features in a two series where each series is sorted by tf-idf value and then alphabetically by feature name. The index of the series should be the feature name, and the data should be the tf-idf.\n",
    "\n",
    "The series of 20 features with smallest tf-idfs should be sorted smallest tfidf first, the list of 20 features with largest tf-idfs should be sorted largest first. \n",
    "\n",
    "*This function should return a tuple of two series\n",
    "`(smallest tf-idfs series, largest tf-idfs series)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "def answer_four():\n",
    "    import operator\n",
    "\n",
    "    vectorizer = TfidfVectorizer()\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "\n",
    "    feature_names = vectorizer.get_feature_names()\n",
    "    idfs = vectorizer.idf_\n",
    "    names_idfs = list(zip(feature_names, idfs))\n",
    "\n",
    "    smallest = sorted(names_idfs, key=operator.itemgetter(1))[:20]\n",
    "    smallest = pd.Series([features[1] for features in smallest], index=[features[0] for features in smallest])\n",
    "\n",
    "    largest = sorted(names_idfs, key=operator.itemgetter(1), reverse=True)[:20]\n",
    "    # largest = sorted(names_idfs, key=operator.itemgetter(1,0), reverse=True)[:20]\n",
    "    largest = sorted(largest, key=operator.itemgetter(0))\n",
    "    largest = pd.Series([features[1] for features in largest], index=[features[0] for features in largest])\n",
    "    \n",
    "    return (smallest, largest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(to      2.198406\n",
       " you     2.265645\n",
       " the     2.707383\n",
       " in      2.890761\n",
       " and     2.976764\n",
       " is      3.003012\n",
       " me      3.111530\n",
       " for     3.206840\n",
       " it      3.222174\n",
       " my      3.231044\n",
       " call    3.297812\n",
       " your    3.300196\n",
       " of      3.319473\n",
       " have    3.354130\n",
       " that    3.408477\n",
       " on      3.463136\n",
       " now     3.465949\n",
       " can     3.545053\n",
       " are     3.560414\n",
       " so      3.566625\n",
       " dtype: float64, 000pes         8.644919\n",
       " 0089           8.644919\n",
       " 0121           8.644919\n",
       " 01223585236    8.644919\n",
       " 0125698789     8.644919\n",
       " 02072069400    8.644919\n",
       " 02073162414    8.644919\n",
       " 02085076972    8.644919\n",
       " 021            8.644919\n",
       " 0430           8.644919\n",
       " 07008009200    8.644919\n",
       " 07099833605    8.644919\n",
       " 07123456789    8.644919\n",
       " 0721072        8.644919\n",
       " 07753741225    8.644919\n",
       " 077xxx         8.644919\n",
       " 078            8.644919\n",
       " 07808247860    8.644919\n",
       " 07808726822    8.644919\n",
       " 078498         8.644919\n",
       " dtype: float64)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_four()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 5\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **3**.\n",
    "\n",
    "Then fit a multinomial Naive Bayes classifier model with smoothing `alpha=0.1` and compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_five():\n",
    "    vectorizer = TfidfVectorizer(min_df=3)\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "    X_test_transformed = vectorizer.transform(X_test)\n",
    "\n",
    "    clf = MultinomialNB(alpha=0.1)\n",
    "    clf.fit(X_train_transformed, y_train)\n",
    "\n",
    "    # y_predicted_prob = clf.predict_proba(X_test_transformed)[:, 1]\n",
    "    y_predicted = clf.predict(X_test_transformed)\n",
    "\n",
    "    # return roc_auc_score(y_test, y_predicted_prob) #Your answer here\n",
    "    return roc_auc_score(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.94162436548223349"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_five()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 6\n",
    "\n",
    "What is the average length of documents (number of characters) for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average length not spam, average length spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_six():\n",
    "    spam_data['length'] = spam_data['text'].apply(lambda x:len(x))\n",
    "    \n",
    "    return (np.mean(spam_data['length'][spam_data['target'] == 0]), np.mean(spam_data['length'][spam_data['target'] == 1]))#Your answer here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71.02362694300518, 138.8661311914324)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_six()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "The following function has been provided to help you combine new features into the training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_feature(X, feature_to_add):\n",
    "    \"\"\"\n",
    "    Returns sparse feature matrix with added feature.\n",
    "    feature_to_add can also be a list of features.\n",
    "    \"\"\"\n",
    "    from scipy.sparse import csr_matrix, hstack\n",
    "    return hstack([X, csr_matrix(feature_to_add).T], 'csr')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 7\n",
    "\n",
    "Fit and transform the training data X_train using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5**.\n",
    "\n",
    "Using this document-term matrix and an additional feature, **the length of document (number of characters)**, fit a Support Vector Classification model with regularization `C=10000`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC\n",
    "\n",
    "def answer_seven():\n",
    "    vectorizer = TfidfVectorizer(min_df=5)\n",
    "\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "    X_train_transformed_with_length = add_feature(X_train_transformed, X_train.str.len())\n",
    "\n",
    "    X_test_transformed = vectorizer.transform(X_test)\n",
    "    X_test_transformed_with_length = add_feature(X_test_transformed, X_test.str.len())\n",
    "\n",
    "    clf = SVC(C=10000)\n",
    "\n",
    "    clf.fit(X_train_transformed_with_length, y_train)\n",
    "\n",
    "    y_predicted = clf.predict(X_test_transformed_with_length)\n",
    "    \n",
    "    return roc_auc_score(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.95813668234215565"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_seven()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 8\n",
    "\n",
    "What is the average number of digits per document for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average # digits not spam, average # digits spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_eight():\n",
    "    spam_data['length'] = spam_data['text'].apply(lambda x: len(''.join([a for a in x if a.isdigit()])))\n",
    "    \n",
    "    return (np.mean(spam_data['length'][spam_data['target'] == 0]), np.mean(spam_data['length'][spam_data['target'] == 1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.2992746113989637, 15.759036144578314)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_eight()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 9\n",
    "\n",
    "Fit and transform the training data `X_train` using a Tfidf Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **word n-grams from n=1 to n=3** (unigrams, bigrams, and trigrams).\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* **number of digits per document**\n",
    "\n",
    "fit a Logistic Regression model with regularization `C=100`. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "*This function should return the AUC score as a float.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "\n",
    "def answer_nine():\n",
    "    vectorizer = TfidfVectorizer(min_df=5, ngram_range=[1,3])\n",
    "\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "    X_train_transformed_with_length = add_feature(X_train_transformed, [X_train.str.len(),\n",
    "                                                                        X_train.apply(lambda x: len(''.join([a for a in x if a.isdigit()])))])\n",
    "\n",
    "    X_test_transformed = vectorizer.transform(X_test)\n",
    "    X_test_transformed_with_length = add_feature(X_test_transformed, [X_test.str.len(),\n",
    "                                                                      X_test.apply(lambda x: len(''.join([a for a in x if a.isdigit()])))])\n",
    "\n",
    "    clf = LogisticRegression(C=100)\n",
    "\n",
    "    clf.fit(X_train_transformed_with_length, y_train)\n",
    "\n",
    "    y_predicted = clf.predict(X_test_transformed_with_length)\n",
    "\n",
    "    return roc_auc_score(y_test, y_predicted)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.96787090640544626"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_nine()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 10\n",
    "\n",
    "What is the average number of non-word characters (anything other than a letter, digit or underscore) per document for not spam and spam documents?\n",
    "\n",
    "*This function should return a tuple (average # non-word characters not spam, average # non-word characters spam).*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_ten():\n",
    "    spam_data['length'] = spam_data['text'].str.findall(r'(\\W)').str.len()\n",
    "    \n",
    "    return (np.mean(spam_data['length'][spam_data['target'] == 0]), np.mean(spam_data['length'][spam_data['target'] == 1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(17.29181347150259, 29.041499330655956)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_ten()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question 11\n",
    "\n",
    "Fit and transform the training data X_train using a Count Vectorizer ignoring terms that have a document frequency strictly lower than **5** and using **character n-grams from n=2 to n=5.**\n",
    "\n",
    "To tell Count Vectorizer to use character n-grams pass in `analyzer='char_wb'` which creates character n-grams only from text inside word boundaries. This should make the model more robust to spelling mistakes.\n",
    "\n",
    "Using this document-term matrix and the following additional features:\n",
    "* the length of document (number of characters)\n",
    "* number of digits per document\n",
    "* **number of non-word characters (anything other than a letter, digit or underscore.)**\n",
    "\n",
    "fit a Logistic Regression model with regularization C=100. Then compute the area under the curve (AUC) score using the transformed test data.\n",
    "\n",
    "Also **find the 10 smallest and 10 largest coefficients from the model** and return them along with the AUC score in a tuple.\n",
    "\n",
    "The list of 10 smallest coefficients should be sorted smallest first, the list of 10 largest coefficients should be sorted largest first.\n",
    "\n",
    "The three features that were added to the document term matrix should have the following names should they appear in the list of coefficients:\n",
    "['length_of_doc', 'digit_count', 'non_word_char_count']\n",
    "\n",
    "*This function should return a tuple `(AUC score as a float, smallest coefs list, largest coefs list)`.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def answer_eleven():\n",
    "    vectorizer = CountVectorizer(min_df=5, analyzer='char_wb', ngram_range=[2,5])\n",
    "\n",
    "    X_train_transformed = vectorizer.fit_transform(X_train)\n",
    "    X_train_transformed_with_length = add_feature(X_train_transformed, [X_train.str.len(),\n",
    "                                                                        X_train.apply(lambda x: len(''.join([a for a in x if a.isdigit()]))),\n",
    "                                                                        X_train.str.findall(r'(\\W)').str.len()])\n",
    "\n",
    "    X_test_transformed = vectorizer.transform(X_test)\n",
    "    X_test_transformed_with_length = add_feature(X_test_transformed, [X_test.str.len(),\n",
    "                                                                      X_test.apply(lambda x: len(''.join([a for a in x if a.isdigit()]))),\n",
    "                                                                      X_test.str.findall(r'(\\W)').str.len()])\n",
    "\n",
    "    clf = LogisticRegression(C=100)\n",
    "\n",
    "    clf.fit(X_train_transformed_with_length, y_train)\n",
    "\n",
    "    y_predicted = clf.predict(X_test_transformed_with_length)\n",
    "\n",
    "    auc = roc_auc_score(y_test, y_predicted)\n",
    "\n",
    "    feature_names = np.array(vectorizer.get_feature_names() + ['length_of_doc', 'digit_count', 'non_word_char_count'])\n",
    "    sorted_coef_index = clf.coef_[0].argsort()\n",
    "    smallest = feature_names[sorted_coef_index[:10]]\n",
    "    largest = feature_names[sorted_coef_index[:-11:-1]]\n",
    "\n",
    "    return (auc, list(smallest), list(largest))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.97885931107074342,\n",
       " ['. ', '..', '? ', ' i', ' y', ' go', ':)', ' h', 'go', ' m'],\n",
       " ['digit_count', 'ne', 'ia', 'co', 'xt', ' ch', 'mob', ' x', 'ww', 'ar'])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answer_eleven()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "python-text-mining",
   "graded_item_id": "Pn19K",
   "launcher_item_id": "y1juS",
   "part_id": "ctlgo"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}