NLP_in_Pysparks_MLlib.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# NLP in Pyspark's MLlib\n",
    "\n",
    "Natural Language Processing (NLP) is a very trendy topic in the data science area today that is really handy for tasks like **chat bots**, movie or **product review analysis** and especially **tweet classification**. In this notebook, we will cover the **classification aspect of NLP** and go over the features that Spark has for cleaning and preparing your data for analysis. We will also touch on how to implement **ML Pipelines** to a few of our data processing steps to help make our code run a bit faster. \n",
    "\n",
    "As we learned in the NLP concept review lectures, the text you process must first be cleaned, tokenized and vectorized. Essentially, we need to covert our text into a vector of numbers. But how do we do that? Spark has a variety of built in functions to accomplish all of these tasks very easily. We will cover all of it here!\n",
    "\n",
    "### Agenda\n",
    "\n",
    "    1. Review Data (quality check)\n",
    "    2. Clean up the data (remove puncuation, special characters, etc.)\n",
    "    3. Tokenize text data\n",
    "    4. Remove Stopwords\n",
    "    5. Zero index our label column\n",
    "    5. Create an ML Pipeline (to streamline steps 3-5)\n",
    "    6. Vectorize Text column\n",
    "         - Count Vectors\n",
    "         - TF-IDF\n",
    "         - Word2Vec\n",
    "    7. Train and Evaluate Model (classification)\n",
    "    8. View Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You are working with 1 core(s)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "            <div>\n",
       "                <p><b>SparkSession - in-memory</b></p>\n",
       "                \n",
       "        <div>\n",
       "            <p><b>SparkContext</b></p>\n",
       "\n",
       "            <p><a href=\"http://orcuns-mbp-2:4043\">Spark UI</a></p>\n",
       "\n",
       "            <dl>\n",
       "              <dt>Version</dt>\n",
       "                <dd><code>v2.4.4</code></dd>\n",
       "              <dt>Master</dt>\n",
       "                <dd><code>local[*]</code></dd>\n",
       "              <dt>AppName</dt>\n",
       "                <dd><code>NLP</code></dd>\n",
       "            </dl>\n",
       "        </div>\n",
       "        \n",
       "            </div>\n",
       "        "
      ],
      "text/plain": [
       "<pyspark.sql.session.SparkSession at 0x111dbc6d0>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# First let's create our PySpark instance\n",
    "# import findspark\n",
    "# findspark.init()\n",
    "\n",
    "import pyspark # only run after findspark.init()\n",
    "from pyspark.sql import SparkSession\n",
    "# May take awhile locally\n",
    "spark = SparkSession.builder.appName(\"NLP\").getOrCreate()\n",
    "\n",
    "cores = spark._jsc.sc().getExecutorMemoryStatus().keySet().size()\n",
    "print(\"You are working with\", cores, \"core(s)\")\n",
    "spark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Import Dependencies**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import * #CountVectorizer,StringIndexer, RegexTokenizer,StopWordsRemover\n",
    "from pyspark.sql.functions import * #col, udf,regexp_replace,isnull\n",
    "from pyspark.sql.types import * #StringType,IntegerType\n",
    "from pyspark.ml.classification import *\n",
    "from pyspark.ml.evaluation import *\n",
    "from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "\n",
    "# For pipeline development\n",
    "from pyspark.ml import Pipeline "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in Dataset\n",
    "\n",
    "#### Kickstarter Dataset\n",
    "\n",
    "##### What is Kickstarter?\n",
    "\"Kickstarter is an American public-benefit corporation based in Brooklyn, New York, that maintains a global crowdfunding platform, focused on creativity and merchandising. The company's stated mission is to \"help bring creative projects to life\". Kickstarter, has reportedly received more than $1.9 billion in pledges from 9.4 million backers to fund 257,000 creative projects, such as films, music, stage shows, comics, journalism, video games, technology and food-related projects.\n",
    "\n",
    "People who back Kickstarter projects are offered tangible rewards or experiences in exchange for their pledges. This model traces its roots to subscription model of arts patronage, where artists would go directly to their audiences to fund their work\" ~ Wikipedia\n",
    "\n",
    "So, what if you can predict if a project will be or not to be able to get the money from their backers?\n",
    "\n",
    "#### Content\n",
    "\n",
    "The datastet contains the blurbs or short description of 215,513 projects runned along 2017, all written in english and all labeled with \"successful\" or \"failed\", if they get the money or not, respectively. From those texts you can train linguistics models for description, and even embeddings relative to the case.\n",
    "\n",
    "**Source:** https://www.kaggle.com/oscarvilla/kickstarter-nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path =\"Datasets/\"\n",
    "\n",
    "# CSV\n",
    "df = spark.read.csv(path+'kickstarter.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_c0</th>\n",
       "      <th>blurb</th>\n",
       "      <th>state</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>Using their own character, users go on educati...</td>\n",
       "      <td>failed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>MicroFly is a quadcopter packed with WiFi, 6 s...</td>\n",
       "      <td>successful</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>A small indie press, run as a collective for a...</td>\n",
       "      <td>failed</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>Zylor is a new baby cosplayer! Back this kicks...</td>\n",
       "      <td>failed</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  _c0                                              blurb       state\n",
       "0   1  Using their own character, users go on educati...      failed\n",
       "1   2  MicroFly is a quadcopter packed with WiFi, 6 s...  successful\n",
       "2   3  A small indie press, run as a collective for a...      failed\n",
       "3   4  Zylor is a new baby cosplayer! Back this kicks...      failed"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.limit(4).toPandas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+-----------------------------------------------------------------------------------------------------------------------------------+----------+\n",
      "|_c0|blurb                                                                                                                              |state     |\n",
      "+---+-----------------------------------------------------------------------------------------------------------------------------------+----------+\n",
      "|1  |Using their own character, users go on educational quests around a virtual world leveling up subject-oriented skills (ie Physics). |failed    |\n",
      "|2  |MicroFly is a quadcopter packed with WiFi, 6 sensors, and 3 processors for ultimate stability -- and fits in the palm of your hand.|successful|\n",
      "|3  |A small indie press, run as a collective for authors who want to self-publish, and a sexy, smart , hilarious novel!                |failed    |\n",
      "|4  |Zylor is a new baby cosplayer! Back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world!    |failed    |\n",
      "+---+-----------------------------------------------------------------------------------------------------------------------------------+----------+\n",
      "only showing top 4 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's read a few full blurbs\n",
    "df.show(4,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the output above that the blurb text contains a good bit of punctuation and special characters. We'll need to clean that up. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- _c0: string (nullable = true)\n",
      " |-- blurb: string (nullable = true)\n",
      " |-- state: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**See how many rows are in the df?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "223627"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How many null values do we have?\n",
    "\n",
    "Let's use our handy dandy function!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------+-----------------+------------------+\n",
      "|Column_Name|Null_Values_Count|Null_Value_Percent|\n",
      "+-----------+-----------------+------------------+\n",
      "|      blurb|             1488|0.6653937136392296|\n",
      "|      state|            13157| 5.883457722010312|\n",
      "+-----------+-----------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import *\n",
    "\n",
    "def null_value_calc(df):\n",
    "    null_columns_counts = []\n",
    "    numRows = df.count()\n",
    "    for k in df.columns:\n",
    "        nullRows = df.where(col(k).isNull()).count()\n",
    "        if(nullRows > 0):\n",
    "            temp = k,nullRows,(nullRows/numRows)*100\n",
    "            null_columns_counts.append(temp)\n",
    "    return(null_columns_counts)\n",
    "\n",
    "null_columns_calc_list = null_value_calc(df)\n",
    "spark.createDataFrame(null_columns_calc_list, ['Column_Name', 'Null_Values_Count','Null_Value_Percent']).show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not too bad! Less than 1% for the blurb column and about 5% of the state column. Unfortunatly though, we will need each row of data to contain value in both of these columns to conduct our analysis, so let's how many rows that actually effects. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Rows that contain at least one null value: 13157\n",
      "Percentage of Rows that contain at least one null value: 0.058834577220103115\n"
     ]
    }
   ],
   "source": [
    "# Of course you will want to know how many rows that affected before you actually execute it..\n",
    "og_len = df.count()\n",
    "drop_len = df.na.drop().count()\n",
    "print(\"Total Rows that contain at least one null value:\",og_len-drop_len)\n",
    "print(\"Percentage of Rows that contain at least one null value:\", (og_len-drop_len)/og_len)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So dropping all rows that have at least one null value would impact just under 6% of our dataframe. I can live with that, so I'll go ahead and drop them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the null values\n",
    "# It's only about 6% so that's okay\n",
    "df = df.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "210470"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# New df row count\n",
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to make this notebook run faster, you can slice the df like this..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sliced row count: 400\n"
     ]
    }
   ],
   "source": [
    "# Slice the dataframe down to make this notebook run faster\n",
    "df = df.limit(400)\n",
    "print('Sliced row count:',df.count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quality Assurance Check (QA)\n",
    "\n",
    "Let's make sure our dependent variable column is clean before we go any further. This an important step in our analysis. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------------------------------------------------------------------------+-----+\n",
      "|state                                                                      |count|\n",
      "+---------------------------------------------------------------------------+-----+\n",
      "|failed                                                                     |240  |\n",
      "|successful                                                                 |156  |\n",
      "| best experienced with Oculus Rift.\"                                       |1    |\n",
      "| mixing pixel art graphics with deep storytelling and action combat system\"|1    |\n",
      "| smart                                                                     |1    |\n",
      "| Adventure with RPG elements.\"                                             |1    |\n",
      "+---------------------------------------------------------------------------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Quick data quality check on the state column....\n",
    "# This is going to be our category column so it's important\n",
    "df.groupBy(\"state\").count().orderBy(col(\"count\").desc()).show(truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the query above that we have some invalid data in the label (state) column. Let's delete those."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------+-----+\n",
      "|state     |count|\n",
      "+----------+-----+\n",
      "|failed    |240  |\n",
      "|successful|156  |\n",
      "+----------+-----+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df = df.filter(\"state IN('successful','failed')\")\n",
    "# Make sure it worked\n",
    "df.groupBy(\"state\").count().orderBy(col(\"count\").desc()).show(truncate=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|blurb                                                                                                                              |\n",
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Using their own character, users go on educational quests around a virtual world leveling up subject-oriented skills (ie Physics). |\n",
      "|MicroFly is a quadcopter packed with WiFi, 6 sensors, and 3 processors for ultimate stability -- and fits in the palm of your hand.|\n",
      "|A small indie press, run as a collective for authors who want to self-publish, and a sexy, smart , hilarious novel!                |\n",
      "|Zylor is a new baby cosplayer! Back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world!    |\n",
      "|Hatoful Boyfriend meet Skeletons! A comedy Dating Sim that puts you into a high school full of Skeletons. Rattle some Bones!       |\n",
      "|FastMan is a Infinite running platformer. Go in FastMan's shoes and run through the platform dodging obstacles.                    |\n",
      "|FADE. A dark and somber RPG about survival and hope.(Legend of Zelda/Fable Inspired)                                               |\n",
      "|The next generation of space combat with online progression, leveling, an arsenal of ships, weapons and much more!                 |\n",
      "|Whip around planets and smash your way to victory in this video game of galactic proportions!                                      |\n",
      "|Sneak in, find treasures, avoid cats and collect the loot before time runs out!                                                    |\n",
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's check the quality of the blurbs\n",
    "df.select(\"blurb\").show(10,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see some punctuation proper casing and some slashes which might making parsing problematic. Let's clean this up a bit!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the blurb column\n",
    "\n",
    "Keep in mind that you can/should do all of this in one call...\n",
    "But we will show each individually for the purpose of learning."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|blurb                                                                                                                              |\n",
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Using their own character, users go on educational quests around a virtual world leveling up subject-oriented skills  ie Physics . |\n",
      "|MicroFly is a quadcopter packed with WiFi, 6 sensors, and 3 processors for ultimate stability -- and fits in the palm of your hand.|\n",
      "|A small indie press, run as a collective for authors who want to self-publish, and a sexy, smart , hilarious novel!                |\n",
      "|Zylor is a new baby cosplayer! Back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world!    |\n",
      "|Hatoful Boyfriend meet Skeletons! A comedy Dating Sim that puts you into a high school full of Skeletons. Rattle some Bones!       |\n",
      "|FastMan is a Infinite running platformer. Go in FastMan's shoes and run through the platform dodging obstacles.                    |\n",
      "|FADE. A dark and somber RPG about survival and hope. Legend of Zelda Fable Inspired                                                |\n",
      "+-----------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 7 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Replace Slashes and parenthesis with spaces\n",
    "# You can test your script on line 7 of the df \"(Legend of Zelda/Fable Inspired)\"\n",
    "df = df.withColumn(\"blurb\",translate(col(\"blurb\"), \"/\", \" \")) \\\n",
    "        .withColumn(\"blurb\",translate(col(\"blurb\"), \"(\", \" \")) \\\n",
    "        .withColumn(\"blurb\",translate(col(\"blurb\"), \")\", \" \"))\n",
    "df.select(\"blurb\").show(7,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "|blurb                                                                                                                          |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Using their own character users go on educational quests around a virtual world leveling up subjectoriented skills  ie Physics |\n",
      "|MicroFly is a quadcopter packed with WiFi  sensors and  processors for ultimate stability  and fits in the palm of your hand   |\n",
      "|A small indie press run as a collective for authors who want to selfpublish and a sexy smart  hilarious novel                  |\n",
      "|Zylor is a new baby cosplayer Back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world  |\n",
      "|Hatoful Boyfriend meet Skeletons A comedy Dating Sim that puts you into a high school full of Skeletons Rattle some Bones      |\n",
      "|FastMan is a Infinite running platformer Go in FastMans shoes and run through the platform dodging obstacles                   |\n",
      "|FADE A dark and somber RPG about survival and hope Legend of Zelda Fable Inspired                                              |\n",
      "|The next generation of space combat with online progression leveling an arsenal of ships weapons and much more                 |\n",
      "|Whip around planets and smash your way to victory in this video game of galactic proportions                                   |\n",
      "|Sneak in find treasures avoid cats and collect the loot before time runs out                                                   |\n",
      "+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Removing anything that is not a letter\n",
    "df = df.withColumn(\"blurb\",regexp_replace(col('blurb'), '[^A-Za-z ]+', ''))\n",
    "df.select(\"blurb\").show(10,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "|blurb                                                                                                                         |\n",
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie Physics |\n",
      "|MicroFly is a quadcopter packed with WiFi sensors and processors for ultimate stability and fits in the palm of your hand     |\n",
      "|A small indie press run as a collective for authors who want to selfpublish and a sexy smart hilarious novel                  |\n",
      "|Zylor is a new baby cosplayer Back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world |\n",
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 4 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Remove multiple spaces\n",
    "df = df.withColumn(\"blurb\",regexp_replace(col('blurb'), ' +', ' '))\n",
    "df.select(\"blurb\").show(4,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "|blurb                                                                                                                         |\n",
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "|using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie physics |\n",
      "|microfly is a quadcopter packed with wifi sensors and processors for ultimate stability and fits in the palm of your hand     |\n",
      "|a small indie press run as a collective for authors who want to selfpublish and a sexy smart hilarious novel                  |\n",
      "|zylor is a new baby cosplayer back this kickstarter to help fund new cosplay photoshoots to share his cuteness with the world |\n",
      "+------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 4 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Lower case everything\n",
    "df = df.withColumn(\"blurb\",lower(col('blurb')))\n",
    "df.select(\"blurb\").show(4,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a pause here and go look at your Spark UI. You'll notice that only the \"show strings\" calls (as opposed to each of the data manipulation calls) are creating jobs. This is because of Sparks lazy computation. \n",
    "\n",
    "<br>\n",
    "So when you want to speed up your notebook those are some calls you can take out. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prep Data for NLP \n",
    "\n",
    "Alright so here is where our analysis turns from basic text cleaning to actually turning our text into number (the backbone of NLP). These next several steps in our analysis are very unique to NLP. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Split text into words (Tokenizing)\n",
    "\n",
    "Yo'll see a new column is added to our dataframe that we call \"words\". This column contains an array of strings as opposed to just a string (current data type of the blurb column)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------------------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|_c0|blurb                                                                                                                         |state     |words                                                                                                                                            |\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|1  |using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie physics |failed    |[using, their, own, character, users, go, on, educational, quests, around, a, virtual, world, leveling, up, subjectoriented, skills, ie, physics]|\n",
      "|2  |microfly is a quadcopter packed with wifi sensors and processors for ultimate stability and fits in the palm of your hand     |successful|[microfly, is, a, quadcopter, packed, with, wifi, sensors, and, processors, for, ultimate, stability, and, fits, in, the, palm, of, your, hand]  |\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 2 rows\n",
      "\n",
      "root\n",
      " |-- _c0: string (nullable = true)\n",
      " |-- blurb: string (nullable = true)\n",
      " |-- state: string (nullable = true)\n",
      " |-- words: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "regex_tokenizer = RegexTokenizer(inputCol=\"blurb\", outputCol=\"words\", pattern=\"\\\\W\")\n",
    "raw_words = regex_tokenizer.transform(df)\n",
    "raw_words.show(2,False)\n",
    "raw_words.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Stopwords\n",
    "\n",
    "**Recall from the content review lecture**\n",
    "Recall that \"stopwords\" are any word that we feel would \"distract\" our model from performing it's best. This list can be customized, but for now, we will just use the default list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', 'your']"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# from pyspark.ml.feature import StopWordsRemover\n",
    "\n",
    "# Define a list of stop words or use default list\n",
    "remover = StopWordsRemover(inputCol=\"words\", outputCol=\"filtered\")\n",
    "stopwords = remover.getStopWords() \n",
    "\n",
    "# Display default list\n",
    "stopwords[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+\n",
      "|_c0|blurb                                                                                                                         |state |words                                                                                                                                            |filtered                                                                                                                  |\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+\n",
      "|1  |using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie physics |failed|[using, their, own, character, users, go, on, educational, quests, around, a, virtual, world, leveling, up, subjectoriented, skills, ie, physics]|[using, character, users, go, educational, quests, around, virtual, world, leveling, subjectoriented, skills, ie, physics]|\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "words_df = remover.transform(raw_words)\n",
    "words_df.show(1,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now we need to encode state column to a column of indices\n",
    "\n",
    "Remember that MLlib requres our dependent variable to not only be a numeric data type, but also zero indexed. We can Sparks handy built in StringIndexer function to accomplish this, just like we did in the classification lectures. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+--------------------+----------+--------------------+--------------------+-----+\n",
      "|_c0|               blurb|     state|               words|            filtered|label|\n",
      "+---+--------------------+----------+--------------------+--------------------+-----+\n",
      "|  1|using their own c...|    failed|[using, their, ow...|[using, character...|  0.0|\n",
      "|  2|microfly is a qua...|successful|[microfly, is, a,...|[microfly, quadco...|  1.0|\n",
      "|  3|a small indie pre...|    failed|[a, small, indie,...|[small, indie, pr...|  0.0|\n",
      "|  4|zylor is a new ba...|    failed|[zylor, is, a, ne...|[zylor, new, baby...|  0.0|\n",
      "|  5|hatoful boyfriend...|    failed|[hatoful, boyfrie...|[hatoful, boyfrie...|  0.0|\n",
      "+---+--------------------+----------+--------------------+--------------------+-----+\n",
      "only showing top 5 rows\n",
      "\n",
      "root\n",
      " |-- _c0: string (nullable = true)\n",
      " |-- blurb: string (nullable = true)\n",
      " |-- state: string (nullable = true)\n",
      " |-- words: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- filtered: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- label: double (nullable = false)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "indexer = StringIndexer(inputCol=\"state\", outputCol=\"label\")\n",
    "feature_data = indexer.fit(words_df).transform(words_df)\n",
    "feature_data.show(5)\n",
    "feature_data.printSchema()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creat an ML Pipeline\n",
    "\n",
    "We could also create an ML Pipeline to accomplish the previous three steps in a more streamlined fashion. Pipelines allow users to combine any transformer call(s) and ONE estimator call in their ML workflow. Si a Pipeline can be a continuous set of transformer calls until you reach a point where you need to call \".fit()\" which is an estimator call. \n",
    "<br>\n",
    "\n",
    "Notice in the script below that we reduced our .transform calls from 3 to 1. So the benefit here is not necessarily speed but a bit less and more organized code (always nice) and little more streamlined. This feature can be esspecially useful when you get to the point where you want to move your model into production. You can save this pipeline to be called on whenever you need to prep new text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from pyspark.sql.functions import *\n",
    "# from pyspark.ml.feature import StopWordsRemover"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "|_c0|blurb                                                                                                                         |state |words                                                                                                                                            |filtered                                                                                                                  |label|\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "|1  |using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie physics |failed|[using, their, own, character, users, go, on, educational, quests, around, a, virtual, world, leveling, up, subjectoriented, skills, ie, physics]|[using, character, users, go, educational, quests, around, virtual, world, leveling, subjectoriented, skills, ie, physics]|0.0  |\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "######################## BEFORE #############################\n",
    "# Tokenize\n",
    "regex_tokenizer = RegexTokenizer(inputCol=\"blurb\", outputCol=\"words\", pattern=\"\\\\W\")\n",
    "raw_words = regex_tokenizer.transform(df)\n",
    "\n",
    "# Remove Stop words\n",
    "remover = StopWordsRemover(inputCol=\"words\", outputCol=\"filtered\")\n",
    "words_df = remover.transform(raw_words)\n",
    "\n",
    "# Zero Index Label Column\n",
    "indexer = StringIndexer(inputCol=\"state\", outputCol=\"label\")\n",
    "feature_data = indexer.fit(words_df).transform(words_df)\n",
    "\n",
    "feature_data.show(1,False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "|_c0|blurb                                                                                                                         |state |words                                                                                                                                            |filtered                                                                                                                  |label|\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "|1  |using their own character users go on educational quests around a virtual world leveling up subjectoriented skills ie physics |failed|[using, their, own, character, users, go, on, educational, quests, around, a, virtual, world, leveling, up, subjectoriented, skills, ie, physics]|[using, character, users, go, educational, quests, around, virtual, world, leveling, subjectoriented, skills, ie, physics]|0.0  |\n",
      "+---+------------------------------------------------------------------------------------------------------------------------------+------+-------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------+-----+\n",
      "only showing top 1 row\n",
      "\n"
     ]
    }
   ],
   "source": [
    "################# AFTER ##################\n",
    "\n",
    "# Tokenize\n",
    "regex_tokenizer = RegexTokenizer(inputCol=\"blurb\", outputCol=\"words\", pattern=\"\\\\W\")\n",
    "# raw_words = regex_tokenizer.transform(df)\n",
    "\n",
    "# Remove Stop words\n",
    "remover = StopWordsRemover(inputCol=regex_tokenizer.getOutputCol(), outputCol=\"filtered\")\n",
    "# words_df = remover.transform(raw_words)\n",
    "\n",
    "# Zero Index Label Column\n",
    "indexer = StringIndexer(inputCol=\"state\", outputCol=\"label\")\n",
    "# feature_data = indexer.fit(words_df).transform(words_df)\n",
    "\n",
    "# Create the Pipeline\n",
    "pipeline = Pipeline(stages=[regex_tokenizer,remover,indexer])\n",
    "data_prep_pl = pipeline.fit(df)\n",
    "# print(type(data_prep_pl))\n",
    "# print(\" \")\n",
    "# Now call on the Pipeline to get our final df\n",
    "feature_data = data_prep_pl.transform(df)\n",
    "feature_data.show(1,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now take a look at the Spark UI again. You'll see the last 2 \"countbyvalue\" job ids for each one of these. If you take a look at how long it took each of those job ids to run, you will see that the second job id actually took just a bit less time to run. Since we do not have much data here it only saved us .2 seconds but that may translate to a couple of miniutes on a much larger df. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting text into vectors\n",
    "\n",
    "We will test out the following three vectorizors:\n",
    "\n",
    "1. Count Vectors\n",
    "2. TF-IDF\n",
    "3. Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count Vector (count vectorizer and hashingTF are basically the same thing)\n",
    "# cv = CountVectorizer(inputCol=\"filtered\", outputCol=\"features\")\n",
    "# model = cv.fit(feature_data)\n",
    "# countVectorizer_features = model.transform(feature_data)\n",
    "\n",
    "# Hashing TF\n",
    "hashingTF = HashingTF(inputCol=\"filtered\", outputCol=\"rawfeatures\", numFeatures=20)\n",
    "HTFfeaturizedData = hashingTF.transform(feature_data)\n",
    "\n",
    "# TF-IDF\n",
    "idf = IDF(inputCol=\"rawfeatures\", outputCol=\"features\")\n",
    "idfModel = idf.fit(HTFfeaturizedData)\n",
    "TFIDFfeaturizedData = idfModel.transform(HTFfeaturizedData)\n",
    "TFIDFfeaturizedData.name = 'TFIDFfeaturizedData'\n",
    "\n",
    "#rename the HTF features to features to be consistent\n",
    "HTFfeaturizedData = HTFfeaturizedData.withColumnRenamed(\"rawfeatures\",\"features\")\n",
    "HTFfeaturizedData.name = 'HTFfeaturizedData' #We will use later for printing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Word2Vec\n",
    "word2Vec = Word2Vec(vectorSize=3, minCount=0, inputCol=\"filtered\", outputCol=\"features\")\n",
    "model = word2Vec.fit(feature_data)\n",
    "\n",
    "W2VfeaturizedData = model.transform(feature_data)\n",
    "# W2VfeaturizedData.show(1,False)\n",
    "\n",
    "# W2Vec Dataframes typically has negative values so we will correct for that here so that we can use the Naive Bayes classifier\n",
    "scaler = MinMaxScaler(inputCol=\"features\", outputCol=\"scaledFeatures\")\n",
    "\n",
    "# Compute summary statistics and generate MinMaxScalerModel\n",
    "scalerModel = scaler.fit(W2VfeaturizedData)\n",
    "\n",
    "# rescale each feature to range [min, max].\n",
    "scaled_data = scalerModel.transform(W2VfeaturizedData)\n",
    "W2VfeaturizedData = scaled_data.select('state','blurb','label','scaledFeatures')\n",
    "W2VfeaturizedData = W2VfeaturizedData.withColumnRenamed('scaledFeatures','features')\n",
    "\n",
    "W2VfeaturizedData.name = 'W2VfeaturizedData' # We will need this to print later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Evaluate your model\n",
    "\n",
    "From here on out, is straight up classification. So we can go and use our trusty function! I'll just go ahead and copy and paste it in here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ClassTrainEval(classifier,features,classes,train,test):\n",
    "\n",
    "    def FindMtype(classifier):\n",
    "        # Intstantiate Model\n",
    "        M = classifier\n",
    "        # Learn what it is\n",
    "        Mtype = type(M).__name__\n",
    "        \n",
    "        return Mtype\n",
    "    \n",
    "    Mtype = FindMtype(classifier)\n",
    "    \n",
    "\n",
    "    def IntanceFitModel(Mtype,classifier,classes,features,train):\n",
    "        \n",
    "        if Mtype == \"OneVsRest\":\n",
    "            # instantiate the base classifier.\n",
    "            lr = LogisticRegression()\n",
    "            # instantiate the One Vs Rest Classifier.\n",
    "            OVRclassifier = OneVsRest(classifier=lr)\n",
    "#             fitModel = OVRclassifier.fit(train)\n",
    "            # Add parameters of your choice here:\n",
    "            paramGrid = ParamGridBuilder() \\\n",
    "                .addGrid(lr.regParam, [0.1, 0.01]) \\\n",
    "                .build()\n",
    "            #Cross Validator requires the following parameters:\n",
    "            crossval = CrossValidator(estimator=OVRclassifier,\n",
    "                                      estimatorParamMaps=paramGrid,\n",
    "                                      evaluator=MulticlassClassificationEvaluator(),\n",
    "                                      numFolds=2) # 3 is best practice\n",
    "            # Run cross-validation, and choose the best set of parameters.\n",
    "            fitModel = crossval.fit(train)\n",
    "            return fitModel\n",
    "        if Mtype == \"MultilayerPerceptronClassifier\":\n",
    "            # specify layers for the neural network:\n",
    "            # input layer of size features, two intermediate of features+1 and same size as features\n",
    "            # and output of size number of classes\n",
    "            # Note: crossvalidator cannot be used here\n",
    "            features_count = len(features[0][0])\n",
    "            layers = [features_count, features_count+1, features_count, classes]\n",
    "            MPC_classifier = MultilayerPerceptronClassifier(maxIter=100, layers=layers, blockSize=128, seed=1234)\n",
    "            fitModel = MPC_classifier.fit(train)\n",
    "            return fitModel\n",
    "        if Mtype in(\"LinearSVC\",\"GBTClassifier\") and classes != 2: # These classifiers currently only accept binary classification\n",
    "            print(Mtype,\" could not be used because PySpark currently only accepts binary classification data for this algorithm\")\n",
    "            return\n",
    "        if Mtype in(\"LogisticRegression\",\"NaiveBayes\",\"RandomForestClassifier\",\"GBTClassifier\",\"LinearSVC\",\"DecisionTreeClassifier\"):\n",
    "  \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"LogisticRegression\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "#                              .addGrid(classifier.regParam, [0.1, 0.01]) \\\n",
    "                             .addGrid(classifier.maxIter, [10, 15,20])\n",
    "                             .build())\n",
    "                \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"NaiveBayes\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "                             .addGrid(classifier.smoothing, [0.0, 0.2, 0.4, 0.6]) \\\n",
    "                             .build())\n",
    "                \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"RandomForestClassifier\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "                               .addGrid(classifier.maxDepth, [2, 5, 10])\n",
    "#                                .addGrid(classifier.maxBins, [5, 10, 20])\n",
    "#                                .addGrid(classifier.numTrees, [5, 20, 50])\n",
    "                             .build())\n",
    "                \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"GBTClassifier\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "#                              .addGrid(classifier.maxDepth, [2, 5, 10, 20, 30]) \\\n",
    "#                              .addGrid(classifier.maxBins, [10, 20, 40, 80, 100]) \\\n",
    "                             .addGrid(classifier.maxIter, [10, 15,50,100])\n",
    "                             .build())\n",
    "                \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"LinearSVC\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "                             .addGrid(classifier.maxIter, [10, 15]) \\\n",
    "                             .addGrid(classifier.regParam, [0.1, 0.01]) \\\n",
    "                             .build())\n",
    "            \n",
    "            # Add parameters of your choice here:\n",
    "            if Mtype in(\"DecisionTreeClassifier\"):\n",
    "                paramGrid = (ParamGridBuilder() \\\n",
    "#                              .addGrid(classifier.maxDepth, [2, 5, 10, 20, 30]) \\\n",
    "                             .addGrid(classifier.maxBins, [10, 20, 40, 80, 100]) \\\n",
    "                             .build())\n",
    "            \n",
    "            #Cross Validator requires all of the following parameters:\n",
    "            crossval = CrossValidator(estimator=classifier,\n",
    "                                      estimatorParamMaps=paramGrid,\n",
    "                                      evaluator=MulticlassClassificationEvaluator(),\n",
    "                                      numFolds=2) # 3 + is best practice\n",
    "            # Fit Model: Run cross-validation, and choose the best set of parameters.\n",
    "            fitModel = crossval.fit(train)\n",
    "            return fitModel\n",
    "    \n",
    "    fitModel = IntanceFitModel(Mtype,classifier,classes,features,train)\n",
    "    \n",
    "    # Print feature selection metrics\n",
    "    if fitModel is not None:\n",
    "        \n",
    "        if Mtype in(\"OneVsRest\"):\n",
    "            # Get Best Model\n",
    "            BestModel = fitModel.bestModel\n",
    "            print(\" \")\n",
    "            print('\\033[1m' + Mtype + '\\033[0m')\n",
    "            # Extract list of binary models\n",
    "            models = BestModel.models\n",
    "            for model in models:\n",
    "                print('\\033[1m' + 'Intercept: '+ '\\033[0m',model.intercept,'\\033[1m' + '\\nCoefficients:'+ '\\033[0m',model.coefficients)\n",
    "\n",
    "        if Mtype == \"MultilayerPerceptronClassifier\":\n",
    "            print(\"\")\n",
    "            print('\\033[1m' + Mtype,\" Weights\"+ '\\033[0m')\n",
    "            print('\\033[1m' + \"Model Weights: \"+ '\\033[0m',fitModel.weights.size)\n",
    "            print(\"\")\n",
    "\n",
    "        if Mtype in(\"DecisionTreeClassifier\", \"GBTClassifier\",\"RandomForestClassifier\"):\n",
    "            # FEATURE IMPORTANCES\n",
    "            # Estimate of the importance of each feature.\n",
    "            # Each feature’s importance is the average of its importance across all trees \n",
    "            # in the ensemble The importance vector is normalized to sum to 1. \n",
    "            # Get Best Model\n",
    "            BestModel = fitModel.bestModel\n",
    "            print(\" \")\n",
    "            print('\\033[1m' + Mtype,\" Feature Importances\"+ '\\033[0m')\n",
    "            print(\"(Scores add up to 1)\")\n",
    "            print(\"Lowest score is the least important\")\n",
    "            print(\" \")\n",
    "            print(BestModel.featureImportances)\n",
    "            \n",
    "            if Mtype in(\"DecisionTreeClassifier\"):\n",
    "                global DT_featureimportances\n",
    "                DT_featureimportances = BestModel.featureImportances.toArray()\n",
    "                global DT_BestModel\n",
    "                DT_BestModel = BestModel\n",
    "            if Mtype in(\"GBTClassifier\"):\n",
    "                global GBT_featureimportances\n",
    "                GBT_featureimportances = BestModel.featureImportances.toArray()\n",
    "                global GBT_BestModel\n",
    "                GBT_BestModel = BestModel\n",
    "            if Mtype in(\"RandomForestClassifier\"):\n",
    "                global RF_featureimportances\n",
    "                RF_featureimportances = BestModel.featureImportances.toArray()\n",
    "                global RF_BestModel\n",
    "                RF_BestModel = BestModel\n",
    "\n",
    "        if Mtype in(\"LogisticRegression\"):\n",
    "            # Get Best Model\n",
    "            BestModel = fitModel.bestModel\n",
    "            print(\" \")\n",
    "            print('\\033[1m' + Mtype,\" Coefficient Matrix\"+ '\\033[0m')\n",
    "            print(\"You should compares these relative to eachother\")\n",
    "            print(\"Coefficients: \\n\" + str(BestModel.coefficientMatrix))\n",
    "            print(\"Intercept: \" + str(BestModel.interceptVector))\n",
    "            global LR_coefficients\n",
    "            LR_coefficients = BestModel.coefficientMatrix.toArray()\n",
    "            global LR_BestModel\n",
    "            LR_BestModel = BestModel\n",
    "\n",
    "        if Mtype in(\"LinearSVC\"):\n",
    "            # Get Best Model\n",
    "            BestModel = fitModel.bestModel\n",
    "            print(\" \")\n",
    "            print('\\033[1m' + Mtype,\" Coefficients\"+ '\\033[0m')\n",
    "            print(\"You should compares these relative to eachother\")\n",
    "            print(\"Coefficients: \\n\" + str(BestModel.coefficients))\n",
    "            global LSVC_coefficients\n",
    "            LSVC_coefficients = BestModel.coefficients.toArray()\n",
    "            global LSVC_BestModel\n",
    "            LSVC_BestModel = BestModel\n",
    "        \n",
    "   \n",
    "    # Set the column names to match the external results dataframe that we will join with later:\n",
    "    columns = ['Classifier', 'Result']\n",
    "    \n",
    "    if Mtype in(\"LinearSVC\",\"GBTClassifier\") and classes != 2:\n",
    "        Mtype = [Mtype] # make this a list\n",
    "        score = [\"N/A\"]\n",
    "        result = spark.createDataFrame(zip(Mtype,score), schema=columns)\n",
    "    else:\n",
    "        predictions = fitModel.transform(test)\n",
    "        MC_evaluator = MulticlassClassificationEvaluator(metricName=\"accuracy\") # redictionCol=\"prediction\",\n",
    "        accuracy = (MC_evaluator.evaluate(predictions))*100\n",
    "        Mtype = [Mtype] # make this a string\n",
    "        score = [str(accuracy)] #make this a string and convert to a list\n",
    "        result = spark.createDataFrame(zip(Mtype,score), schema=columns)\n",
    "        result = result.withColumn('Result',result.Result.substr(0, 5))\n",
    "        \n",
    "    return result\n",
    "    #Also returns the fit model important scores or p values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Declare the algorithims you want to test plus declare a list of all the different feature vectors we want to test out that we created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from pyspark.ml.classification import *\n",
    "# from pyspark.ml.evaluation import *\n",
    "# from pyspark.sql import functions\n",
    "# from pyspark.ml.tuning import CrossValidator, ParamGridBuilder\n",
    "\n",
    "# Comment out Naive Bayes if your data still contains negative values\n",
    "classifiers = [\n",
    "                LogisticRegression()\n",
    "                ,OneVsRest()\n",
    "               ,LinearSVC()\n",
    "               ,NaiveBayes()\n",
    "               ,RandomForestClassifier()\n",
    "               ,GBTClassifier()\n",
    "               ,DecisionTreeClassifier()\n",
    "               ,MultilayerPerceptronClassifier()\n",
    "              ] \n",
    "\n",
    "featureDF_list = [HTFfeaturizedData,TFIDFfeaturizedData,W2VfeaturizedData]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Loop through all feature types (hashingTF, TFIDF and Word2Vec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mHTFfeaturizedData  Results:\u001b[0m\n",
      " \n",
      "\u001b[1mLogisticRegression  Coefficient Matrix\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "DenseMatrix([[ 0.27734232, -0.12474694,  0.12819003,  0.46220788,  0.14395622,\n",
      "              -0.48257185,  0.19209362,  0.00889538, -0.083659  ,  0.01189857,\n",
      "              -0.37069719, -0.2463395 ,  0.20939558, -0.41442704, -0.02378179,\n",
      "              -0.22147773,  0.23617951,  0.31851057,  0.17207436,  0.03175833]])\n",
      "Intercept: [-0.522777788485201]\n",
      " \n",
      "\u001b[1mOneVsRest\u001b[0m\n",
      "\u001b[1mIntercept: \u001b[0m 0.5083832956482472 \u001b[1m\n",
      "Coefficients:\u001b[0m [-0.2583095507081701,0.11960380490476616,-0.12092628815687555,-0.4350427270787762,-0.1409907694260478,0.4450971379852503,-0.17792276324625714,-0.0009456983278041717,0.07507377877956131,-0.006872147792491524,0.34583501741911976,0.22975267873887523,-0.19367269425174413,0.3856943461336167,0.023845973440220766,0.20851301836141603,-0.21863640206444657,-0.29474942995355397,-0.154265696430576,-0.027986818410616347]\n",
      "\u001b[1mIntercept: \u001b[0m -0.5083832956482434 \u001b[1m\n",
      "Coefficients:\u001b[0m [0.2583095507081694,-0.11960380490476664,0.12092628815687538,0.435042727078776,0.14099076942604785,-0.4450971379852506,0.17792276324625672,0.0009456983278039056,-0.07507377877956156,0.006872147792491328,-0.3458350174191198,-0.22975267873887542,0.19367269425174366,-0.385694346133617,-0.02384597344022121,-0.2085130183614162,0.21863640206444618,0.2947494299535535,0.15426569643057544,0.027986818410615868]\n",
      " \n",
      "\u001b[1mLinearSVC  Coefficients\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "[0.1406649666240988,-0.23357207573860608,0.039226480217578505,0.5794476640599222,0.0,-0.2841800323875348,0.08347816156336452,-0.09927028460353089,-0.06384662503440326,-0.06182633125411611,-0.29188329332997176,-0.20584111738709532,0.13611803799322889,-0.22703450516493384,-0.10756646160622393,-0.14112501295928673,0.11875996399287413,0.09513354768014565,0.0,-0.08465345840925326]\n",
      " \n",
      "\u001b[1mRandomForestClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],[0.07258275192357437,0.05051743329308663,0.046589759113365964,0.06641522634853722,0.05533607484060241,0.053010968389498425,0.06617832163272681,0.03211164464490945,0.03812570308000519,0.03938511290447896,0.05407591661609379,0.042287397356641186,0.049321038862137345,0.04921926787399034,0.04753000631088697,0.06339120570205818,0.03491671120686464,0.05131064217401286,0.04319700959495807,0.04449780813157126])\n",
      " \n",
      "\u001b[1mGBTClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],[0.049005692964650384,0.0459558460058415,0.03737664686028332,0.044259541608297745,0.0651070992317022,0.044167214961073537,0.05368131996062493,0.028409263446276284,0.04549597589139785,0.03804820287867538,0.04919822137216936,0.049883069652167544,0.07573438023179623,0.04693169691919881,0.06410424059525437,0.05557935884443996,0.056142819241533824,0.05423179916240742,0.024910868976858788,0.07177674119535057])\n",
      " \n",
      "\u001b[1mDecisionTreeClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,8,11,15,16],[0.06512995880041486,0.10131326924508981,0.05789329671147988,0.1552500273991028,0.08683994506721977,0.0846785050160255,0.12349013693698746,0.07490501928362253,0.07236662088934982,0.10687993239042443,0.07125328826028313])\n",
      "\n",
      "\u001b[1mMultilayerPerceptronClassifier  Weights\u001b[0m\n",
      "\u001b[1mModel Weights: \u001b[0m 923\n",
      "\n",
      "+------------------------------+------+\n",
      "|Classifier                    |Result|\n",
      "+------------------------------+------+\n",
      "|LogisticRegression            |54.23 |\n",
      "|OneVsRest                     |55.08 |\n",
      "|LinearSVC                     |56.77 |\n",
      "|NaiveBayes                    |59.32 |\n",
      "|RandomForestClassifier        |56.77 |\n",
      "|GBTClassifier                 |54.23 |\n",
      "|DecisionTreeClassifier        |55.08 |\n",
      "|MultilayerPerceptronClassifier|54.23 |\n",
      "+------------------------------+------+\n",
      "\n",
      "None\n",
      "\u001b[1mTFIDFfeaturizedData  Results:\u001b[0m\n",
      " \n",
      "\u001b[1mLogisticRegression  Coefficient Matrix\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "DenseMatrix([[ 0.36065297, -0.11724828,  0.14301497,  0.57620933,  0.20103424,\n",
      "              -0.65955494,  0.21579041,  0.01181596, -0.10141948,  0.01422522,\n",
      "              -0.46865591, -0.2513694 ,  0.22417301, -0.67573492, -0.02923563,\n",
      "              -0.40574315,  0.26349324,  0.43532423,  0.17558788,  0.0385005 ]])\n",
      "Intercept: [-0.5227777884852061]\n",
      " \n",
      "\u001b[1mOneVsRest\u001b[0m\n",
      "\u001b[1mIntercept: \u001b[0m 0.5083832956482435 \u001b[1m\n",
      "Coefficients:\u001b[0m [-0.33590295839356715,0.11241430794867295,-0.13491118940985825,-0.5423440083093565,-0.19689299975583777,0.6083363893341058,-0.19987142830873733,-0.001256194974277538,0.09101164951764582,-0.008215930577196077,0.43722378154370595,0.23444390724448483,-0.20734052496256974,0.6288854476985172,0.029314532469713085,0.38199204030972483,-0.2439212970180739,-0.4028487012697272,-0.15741558628824842,-0.03392831091905026]\n",
      "\u001b[1mIntercept: \u001b[0m -0.5083832956482465 \u001b[1m\n",
      "Coefficients:\u001b[0m [0.33590295839356743,-0.11241430794867274,0.1349111894098586,0.5423440083093567,0.19689299975583818,-0.6083363893341058,0.19987142830873744,0.0012561949742778949,-0.09101164951764527,0.00821593057719613,-0.43722378154370556,-0.23444390724448455,0.20734052496257002,-0.6288854476985166,-0.029314532469712832,-0.38199204030972433,0.2439212970180744,0.4028487012697274,0.15741558628824884,0.03392831091905046]\n",
      " \n",
      "\u001b[1mLinearSVC  Coefficients\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "[0.18291920798835964,-0.21953183906814014,0.04376295000182664,0.722365756673936,0.0,-0.38840298009108115,0.09377608058595804,-0.13186322630354944,-0.07740101477478328,-0.07391587910573733,-0.36901502407577147,-0.21004410523826353,0.14572413299371018,-0.37018612757729613,-0.13223450656408012,-0.2585384455258051,0.1324943338686253,0.13002374978695505,0.006013676731819215,-0.10262505780909321]\n",
      " \n",
      "\u001b[1mRandomForestClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],[0.07258275192357437,0.05051743329308663,0.046589759113365964,0.06641522634853722,0.05533607484060241,0.053010968389498425,0.06617832163272681,0.03211164464490945,0.03812570308000519,0.03938511290447896,0.05407591661609379,0.042287397356641186,0.049321038862137345,0.04921926787399034,0.04753000631088697,0.06339120570205818,0.03491671120686464,0.05131064217401286,0.04319700959495807,0.04449780813157126])\n",
      " \n",
      "\u001b[1mGBTClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19],[0.049005692964650384,0.0459558460058415,0.03737664686028332,0.044259541608297745,0.0651070992317022,0.044167214961073537,0.05368131996062493,0.028409263446276284,0.04549597589139785,0.03804820287867538,0.04919822137216936,0.049883069652167544,0.07573438023179623,0.04693169691919881,0.06410424059525437,0.05557935884443996,0.056142819241533824,0.05423179916240742,0.024910868976858788,0.07177674119535057])\n",
      " \n",
      "\u001b[1mDecisionTreeClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(20,[0,1,2,3,4,5,6,8,11,15,16],[0.06512995880041486,0.10131326924508981,0.05789329671147988,0.1552500273991028,0.08683994506721977,0.0846785050160255,0.12349013693698746,0.07490501928362253,0.07236662088934982,0.10687993239042443,0.07125328826028313])\n",
      "\n",
      "\u001b[1mMultilayerPerceptronClassifier  Weights\u001b[0m\n",
      "\u001b[1mModel Weights: \u001b[0m 923\n",
      "\n",
      "+------------------------------+------+\n",
      "|Classifier                    |Result|\n",
      "+------------------------------+------+\n",
      "|LogisticRegression            |54.23 |\n",
      "|OneVsRest                     |55.08 |\n",
      "|LinearSVC                     |56.77 |\n",
      "|NaiveBayes                    |56.77 |\n",
      "|RandomForestClassifier        |56.77 |\n",
      "|GBTClassifier                 |54.23 |\n",
      "|DecisionTreeClassifier        |55.08 |\n",
      "|MultilayerPerceptronClassifier|54.23 |\n",
      "+------------------------------+------+\n",
      "\n",
      "None\n",
      "\u001b[1mW2VfeaturizedData  Results:\u001b[0m\n",
      " \n",
      "\u001b[1mLogisticRegression  Coefficient Matrix\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "DenseMatrix([[ 1.32480216,  1.0212857 , -3.47408144]])\n",
      "\n",
      "Intercept: [-0.12823293374201641]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      "\u001b[1mOneVsRest\u001b[0m\n",
      "\u001b[1mIntercept: \u001b[0m -0.8342440524601364 \u001b[1m\n",
      "Coefficients:\u001b[0m [-0.6641696721415832,-0.3178630879424359,3.8901315369066913]\n",
      "\u001b[1mIntercept: \u001b[0m 0.8342440524601217 \u001b[1m\n",
      "Coefficients:\u001b[0m [0.6641696721416108,0.31786308794243057,-3.890131536906682]\n",
      " \n",
      "\u001b[1mLinearSVC  Coefficients\u001b[0m\n",
      "You should compares these relative to eachother\n",
      "Coefficients: \n",
      "[-0.15243135081805612,-0.2385729229597902,-1.4085024464403189]\n",
      " \n",
      "\u001b[1mRandomForestClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(3,[0,1,2],[0.07974517933719513,0.12015444784960333,0.8001003728132016])\n",
      " \n",
      "\u001b[1mGBTClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(3,[0,1,2],[0.3384781655641985,0.4424744420443583,0.21904739239144314])\n",
      " \n",
      "\u001b[1mDecisionTreeClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(3,[0,1,2],[0.2808481209180027,0.16992749361121173,0.5492243854707857])\n",
      "\n",
      "\u001b[1mMultilayerPerceptronClassifier  Weights\u001b[0m\n",
      "\u001b[1mModel Weights: \u001b[0m 39\n",
      "\n",
      "+------------------------------+------+\n",
      "|Classifier                    |Result|\n",
      "+------------------------------+------+\n",
      "|LogisticRegression            |61.01 |\n",
      "|OneVsRest                     |59.32 |\n",
      "|LinearSVC                     |59.32 |\n",
      "|NaiveBayes                    |59.32 |\n",
      "|RandomForestClassifier        |61.86 |\n",
      "|GBTClassifier                 |64.40 |\n",
      "|DecisionTreeClassifier        |63.55 |\n",
      "|MultilayerPerceptronClassifier|59.32 |\n",
      "+------------------------------+------+\n",
      "\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "for featureDF in featureDF_list:\n",
    "    print('\\033[1m' + featureDF.name,\" Results:\"+ '\\033[0m')\n",
    "    train, test = featureDF.randomSplit([0.7, 0.3],seed = 11)\n",
    "    features = featureDF.select(['features']).collect()\n",
    "    # Learn how many classes there are in order to specify evaluation type based on binary or multi and turn the df into an object\n",
    "    class_count = featureDF.select(countDistinct(\"label\")).collect()\n",
    "    classes = class_count[0][0]\n",
    "\n",
    "    #set up your results table\n",
    "    columns = ['Classifier', 'Result']\n",
    "    vals = [(\"Place Holder\",\"N/A\")]\n",
    "    results = spark.createDataFrame(vals, columns)\n",
    "\n",
    "    for classifier in classifiers:\n",
    "        new_result = ClassTrainEval(classifier,features,classes,train,test)\n",
    "        results = results.union(new_result)\n",
    "    results = results.where(\"Classifier!='Place Holder'\")\n",
    "    print(results.show(truncate=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like the Decision Tree classifier with the W2VfeaturizedData was our best performing feature list/classifier combo. Let's go with that and create our final model and play around with the test dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " \n",
      "\u001b[1mDecisionTreeClassifier  Feature Importances\u001b[0m\n",
      "(Scores add up to 1)\n",
      "Lowest score is the least important\n",
      " \n",
      "(3,[0,1,2],[0.2808481209180027,0.16992749361121173,0.5492243854707857])\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "DataFrame[Classifier: string, Result: string]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifier = DecisionTreeClassifier()\n",
    "featureDF = W2VfeaturizedData\n",
    "\n",
    "train, test = featureDF.randomSplit([0.7, 0.3],seed = 11)\n",
    "features = featureDF.select(['features']).collect()\n",
    "\n",
    "# Learn how many classes there are in order to specify evaluation type based on binary or multi and turn the df into an object\n",
    "class_count = featureDF.select(countDistinct(\"label\")).collect()\n",
    "classes = class_count[0][0]\n",
    "\n",
    "#running this afain with generate all the objects need to play around with test data\n",
    "ClassTrainEval(classifier,features,classes,train,test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see some results!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predicted Failures:\n",
      "+------+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "|state |blurb                                                                                                                          |\n",
      "+------+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "|failed|a book about abuse and how to remove ones self from an abusive situation                                                       |\n",
      "|failed|a collection of essays about life as an alien artistponderer in the iconic city of angels                                      |\n",
      "|failed|a complete game and a great learning opportunity for aspiring web game developers learn how to build it and have fun playing it|\n",
      "+------+-------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n",
      " \n",
      "Predicted Success:\n",
      "+------+----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|state |blurb                                                                                                                             |\n",
      "+------+----------------------------------------------------------------------------------------------------------------------------------+\n",
      "|failed| yr old spirit of of daniel reveals antichrist horses apocalypse iraq war ot prophecy revelation into current events              |\n",
      "|failed|a point n click game mixing up hard boiled detective stories lovecraftian horrors and dark humor in an intriguing mysterious story|\n",
      "|failed|a style experience that will display style in several forms fashion art and music this is style chronicles                        |\n",
      "+------+----------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions = DT_BestModel.transform(test)\n",
    "print(\"Predicted Failures:\")\n",
    "predictions.select(\"state\",\"blurb\").filter(\"prediction=0\").orderBy(predictions[\"prediction\"].desc()).show(3,False)\n",
    "print(\" \")\n",
    "print(\"Predicted Success:\")\n",
    "predictions.select(\"state\",\"blurb\").filter(\"prediction=1\").orderBy(predictions[\"prediction\"].desc()).show(3,False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What could be next?\n",
    "\n",
    "Once we have our model and all the vectorizer the sky is really the limit! We could do any of the following for starters:\n",
    "\n",
    "1. Allow a user to input their own \"blurb\" and we could return a prediction of whether or not it would pass\n",
    "2. If we had a time variable here, we could show the most popular words over time\n",
    "3. Provide this algorithim to Kickstarter for prescreening so they can prioritize entries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}