3_Topic_modeling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Topic Modeling Lab\n",
    "## Contents\n",
    "1. [Setup](#Section-1%3A-Setup)\n",
    "    1. [Import](#1.1-Import-Packages)\n",
    "    1. [Download](#1.2-Download-and-Prepare-Data)\n",
    "    1. [Read](#1.3-Read-Data)\n",
    "    1. [Pick Your Data](#1.4-Pick-Your-Data)\n",
    "    1. [Clean Text](#1.5-Clean-Text)\n",
    "1. [Converting Text to Numbers](#Section-2%3A-Converting-Text-to-Numbers)\n",
    "1. [LDA Topic Model](#Section-3%3A-LDA-Topic-Model)\n",
    "1. [Interpret Topics](#Section-4%3A-Interpret-Topics)\n",
    "1. [Topic Quality](#Section-5%3A-Topic-Quality)\n",
    "1. [Topic Popularity](#Section-6%3A-Topic-Popularity)\n",
    "1. [NMF Topic Model](#Section-7%3A-NMF-Topic-Model)\n",
    "1. [What We Learned](#Section-7%3A-What-We-Learned)\n",
    "\n",
    "## Section 0: Background\n",
    "- In this lab, we'll learn about topic modeling. Topic modeling uses statistics to understand what text is about, that is, to find the topics in text.\n",
    "- We'll use the online dating profile text that OKCupid made public as our example, but of course topic modeling can be used on any text.\n",
    "\n",
    "@Author: [Jeff Lockhart](http://www-personal.umich.edu/~jwlock/) & [Ed Platt](https://elplatt.com/), with some code adapted from [Aneesha Bakharia](https://medium.com/@aneesha/topic-modeling-with-scikit-learn-e80d33668730)'s example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1: Setup\n",
    "### 1.1 Import Packages\n",
    "- Packages contain code others have written to make our work easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install lxml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "tqdm.pandas()\n",
    "from scipy.stats import pearsonr  \n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.decomposition import NMF, LatentDirichletAllocation\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "from bs4 import BeautifulSoup\n",
    "import warnings\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Download and Prepare Data\n",
    "This code checks whether you have the data. If you don't, it will download and prepare it for you. To see how it works, look at lab `1 Data munging` which explains it in detail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i \"download_and_clean_data.py\"\n",
    "print('Ready to go!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Read Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profiles = pd.read_csv('data/clean_profiles.tsv', sep='\\t')\n",
    "profiles.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Pick Your Data\n",
    "- Pick which section of the profiles you want to analyze.\n",
    "#### Options:\n",
    "- `text` - All of the text from a profile (**Recommended**)\n",
    "- `essay0` - My self summary (**Recommended**)\n",
    "- `essay1` - What I’m doing with my life\n",
    "- `essay2` - I’m really good at\n",
    "- `essay3` - The first thing people usually notice about me\n",
    "- `essay4` - Favorite books, movies, show, music, and food\n",
    "- `essay5` - The six things I could never do without\n",
    "- `essay6` - I spend a lot of time thinking about\n",
    "- `essay7` - On a typical Friday night I am\n",
    "- `essay8` - The most private thing I am willing to admit\n",
    "- `essay9` - You should message me if...\n",
    "\n",
    "#### Replace `essay0` in the cell below with the essay you want to look at.\n",
    "- `text` and `essay0` are both recommended, but it's your choice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_section_to_use = 'essay0'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5 Clean Text \n",
    "- For this lab, it is not so important that you understand this code. \n",
    "- For now, just run it and move on. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some of the essays have just a link in the text. BeautifulSoup sees that and gets \n",
    "# the wrong idea. This line hides those warnings.\n",
    "warnings.filterwarnings(\"ignore\", category=UserWarning, module='bs4')\n",
    "def clean(text):\n",
    "    if pd.isnull(text):\n",
    "        t = np.nan\n",
    "    else:\n",
    "        t = BeautifulSoup(text, 'lxml').get_text()\n",
    "        t = t.lower()\n",
    "\n",
    "        bad_words = ['http', 'www', '\\nnan']\n",
    "\n",
    "        for b in bad_words:\n",
    "            t = t.replace(b, '')\n",
    "    if t == '':\n",
    "        t = np.nan\n",
    "    \n",
    "    return t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Cleaning up profile text for', profile_section_to_use, '...')\n",
    "profiles['clean'] = profiles[profile_section_to_use].progress_apply(clean)\n",
    "\n",
    "print('We started with', profiles.shape[0], 'profiles.')\n",
    "print(\"Dropping profiles that didn't write anything for the essay we chose...\")\n",
    "profiles.dropna(axis=0, subset=['clean'], inplace=True)\n",
    "\n",
    "text_cols = ['text', 'essay0', 'essay1', 'essay2', 'essay3', 'essay4', \n",
    "             'essay5', 'essay6', 'essay7', 'essay8', 'essay9']\n",
    "profiles.drop(columns=text_cols, inplace=True)\n",
    "\n",
    "#what we will use as our documents, here the cleaned up text of each profile\n",
    "documents = profiles['clean'].values\n",
    "\n",
    "print('We have', documents.shape[0], 'profiles left.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 2: Converting Text to Numbers\n",
    "- Our first model takes \"count vectors\" as input, that is, a count of how many times each word shows up in each document. \n",
    "    - Here we tell it to only use the 1,000 most popular words, ignoring stop words like \"a\" and \"of\".\n",
    "    - We use the abbreviation `tf` for these because they represent \"text frequency,\" i.e., how often each word shows up in text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    " \n",
    "#### Short Answer No.1\n",
    "- (1) Why do we want to ignore stop words? (1-2 sentences)\n",
    "- (2) Give 3 more examples of stop words \n",
    "    \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_vectorizer = CountVectorizer(max_features=1000, stop_words='english')\n",
    "\n",
    "print(\"Vectorizing text by word counts...\")\n",
    "tf_text = tf_vectorizer.fit_transform(documents)\n",
    "\n",
    "tmp = tf_text.get_shape()\n",
    "print(\"Our transformed text has\", tmp[0], \"rows and\", tmp[1], \"columns.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See what words are being counted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf_words = tf_vectorizer.get_feature_names_out()\n",
    "\n",
    "print(\"The first few words (alphabetically) are:\\n\\n\", tf_words[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See an example of how a profile's text is encoded\n",
    "- `n` is the profile number you want to look at. Change the value of `n` and re-run the code to see different profiles.\n",
    "- Note that only some of the words are counted. This is because we set `max_features=1000` in the vectorizor function, so it is only counting the 1,000 most common words and ignoring the rest. \n",
    "    - You can change that number to be bigger or smaller and see what happens.\n",
    "    - We found in Lab 1 that 1,000 is a good choice for this data because words less popular than that show up in less than 1% of all profiles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 6\n",
    "\n",
    "def show_vector(x, words):\n",
    "    rows,cols = x.nonzero()\n",
    "    for row,col in zip(rows,cols):\n",
    "        print(words[col], '\\t', x[row,col].round(2))\n",
    "\n",
    "print('Profile text:\\n', documents[n])\n",
    "print('\\nTF (count) vector:')\n",
    "show_vector(tf_text[n], tf_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.2 \n",
    "\n",
    "- (1) Given the essay questions decribed above, list 5 words that you predict to be the most popular in essay4. \n",
    "    \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_section_to_use = 'essay4'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In this cell, let's find the top 10 most popular words and list them alphabetically. \n",
    "# Look the sample codes above and try fill in the \"???\".\n",
    "\n",
    "profiles = pd.read_csv('data/clean_profiles.tsv', sep='\\t')\n",
    "\n",
    "print('Cleaning up profile text for', profile_section_to_use, '...')\n",
    "profiles['clean'] = profiles[profile_section_to_use].progress_apply(clean)\n",
    "\n",
    "print('We started with', profiles.shape[0], 'profiles.')\n",
    "print(\"Dropping profiles that didn't write anything for the essay we chose...\")\n",
    "profiles.dropna(axis=0, subset=['clean'], inplace=True)\n",
    "\n",
    "text_cols = ['text', 'essay0', 'essay1', 'essay2', 'essay3', 'essay4', \n",
    "             'essay5', 'essay6', 'essay7', 'essay8', 'essay9']\n",
    "profiles.drop(columns=text_cols, inplace=True)\n",
    "\n",
    "#what we will use as our documents, here the cleaned up text of each profile\n",
    "documents = profiles['clean'].values\n",
    "\n",
    "\n",
    "tf_vectorizer_your_turn = CountVectorizer(???)\n",
    "\n",
    "print(\"Vectorizing text by word counts...\")\n",
    "tf_text_your_turn = tf_vectorizer_your_turn.fit_transform(documents)\n",
    "\n",
    "tmp_your_turn = tf_text_your_turn.get_shape()\n",
    "print(\"Our transformed text has\", tmp_your_turn[0], \"rows and\", tmp_your_turn[1], \"columns.\")\n",
    "\n",
    "tf_words_your_output = tf_vectorizer_your_turn.get_feature_names_out()\n",
    "\n",
    "print(\"The first few words (alphabetically) are:\\n\\n\", tf_words_your_output[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "- (2) Did any of your predicted words made to top 10? (No right or wrong answer!) \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now let's focus on essay0 again.\n",
    "\n",
    "profile_section_to_use = 'essay0'\n",
    "\n",
    "profiles = pd.read_csv('data/clean_profiles.tsv', sep='\\t')\n",
    "\n",
    "print('Cleaning up profile text for', profile_section_to_use, '...')\n",
    "profiles['clean'] = profiles[profile_section_to_use].progress_apply(clean)\n",
    "\n",
    "print('We started with', profiles.shape[0], 'profiles.')\n",
    "print(\"Dropping profiles that didn't write anything for the essay we chose...\")\n",
    "profiles.dropna(axis=0, subset=['clean'], inplace=True)\n",
    "\n",
    "text_cols = ['text', 'essay0', 'essay1', 'essay2', 'essay3', 'essay4', \n",
    "             'essay5', 'essay6', 'essay7', 'essay8', 'essay9']\n",
    "profiles.drop(columns=text_cols, inplace=True)\n",
    "\n",
    "#what we will use as our documents, here the cleaned up text of each profile\n",
    "documents = profiles['clean'].values\n",
    "\n",
    "\n",
    "tf_vectorizer_your_turn = CountVectorizer(max_features=10, stop_words='english')\n",
    "\n",
    "print(\"Vectorizing text by word counts...\")\n",
    "tf_text_your_turn = tf_vectorizer_your_turn.fit_transform(documents)\n",
    "\n",
    "tmp_your_turn = tf_text_your_turn.get_shape()\n",
    "print(\"Our transformed text has\", tmp_your_turn[0], \"rows and\", tmp_your_turn[1], \"columns.\")\n",
    "\n",
    "tf_words_your_output = tf_vectorizer_your_turn.get_feature_names_out()\n",
    "\n",
    "print(\"The first few words (alphabetically) are:\\n\\n\", tf_words_your_output[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 3: LDA Topic Model\n",
    "\n",
    "- LDA stands for Latent Dirichlet Allocation. The statistical math behind it is complicated, but its goals are simple:\n",
    "    - find groups of words that often show up together and call those groups topics. \n",
    "    - find topics that can be used to tell documents apart, i.e. topics that are in some documents but not others.\n",
    "- LDA is the most popular method for topic modeling.\n",
    "- [Learn more](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation) about LDA\n",
    "\n",
    "### Step 1: Decide how many topics we want to find\n",
    "- We must tell LDA how many topics we want it to look for (we will do this with the `ntopics` variable below.)\n",
    "\n",
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.3\n",
    "- How would the number of topics you choose affect the result in your opinion? (Think about extreme cases like you only pick 1 topic, vs if you pick 10,000 topics.) (1-2 sentences)\n",
    "- Choose 1 essay and describe the types of topics you would expect to find.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#how many topics we want our model to find\n",
    "ntopics = 15\n",
    "\n",
    "#how many top words we want to display for each topic\n",
    "nshow = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Run the LDA algorithm\n",
    "- LDA can be a little slow. We'll use a faster method later on.\n",
    "- Set `n_jobs=` to the number of processors you want to use to compute LDA. If you set it to `-1`, it will use all available processors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LatentDirichletAllocation(n_components=ntopics, max_iter=10, \n",
    "                                  learning_method='online', n_jobs=-1, random_state=42)\n",
    "\n",
    "print('Performing LDA on vectors. This may take a while...')\n",
    "lda = model.fit(tf_text)\n",
    "lda_topics = lda.components_\n",
    "\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 4: Interpret Topics\n",
    "#### Some helper functions \n",
    "Don't worry about how these work right now. Just run them and scroll down. We'll use them to make our analysis easier later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def describe_topic(topic, feature_names, n_words=10):\n",
    "    words = []\n",
    "    # sort the words in the topic by importance\n",
    "    topic = topic.argsort() \n",
    "    # select the n_words most important words\n",
    "    topic = topic[:-n_words - 1:-1]\n",
    "    # for each important word, get it's name (i.e. the word) from our list of names\n",
    "    for i in topic:\n",
    "        words.append(feature_names[i])\n",
    "    # print the topic number and its most important words, separated by spaces\n",
    "    return \" \".join(words)\n",
    "\n",
    "def display_topics(components, feature_names, n_words=10):\n",
    "    # loop through each topic (component) in the model; show its top words\n",
    "    for topic_idx, topic in enumerate(components):\n",
    "        print(\"Topic {}:\".format(topic_idx), \n",
    "              describe_topic(topic, feature_names, n_words))\n",
    "    return\n",
    "\n",
    "def find_intersection(idxa, idxb, n):\n",
    "    a = set()\n",
    "    b = set()\n",
    "    both = set()\n",
    "    i = 0\n",
    "    while len(both) < n:\n",
    "        a.add(idxa[i])\n",
    "        b.add(idxb[i])\n",
    "        both = a.intersection(b)\n",
    "        i += 1\n",
    "    return list(both)\n",
    "\n",
    "def compare_topic_words(topics, a, b, words, how='overlap', n_words=nshow):\n",
    "    b_sort = False\n",
    "    if how == 'difference':\n",
    "        b_sort = True\n",
    "    \n",
    "    dfa = pd.DataFrame(topics, columns=words).T\n",
    "    idxa = dfa.sort_values(by=a, ascending=False).index.values\n",
    "    idxb = dfa.sort_values(by=b, ascending=b_sort).index.values\n",
    "    both = find_intersection(idxa, idxb, n=n_words)\n",
    "    \n",
    "    out = how + ' between ' + str(a) + ' and ' + str(b) + ':'\n",
    "    for w in both:\n",
    "        out += ' ' + w\n",
    "    print(out)\n",
    "    return\n",
    "\n",
    "def blue_matrix(cells, xl, yl, t, x_labels=None):\n",
    "    n = cells.shape[0]\n",
    "    plt.figure()\n",
    "    fig, ax = plt.subplots(figsize=(8,8))\n",
    "    plt.imshow(cells, cmap='Blues')\n",
    "    ax.xaxis.tick_top()\n",
    "    ax.xaxis.set_label_position('top') \n",
    "    if x_labels is not None:\n",
    "        plt.xticks(range(n), x_labels)\n",
    "    else:\n",
    "        plt.xticks(range(n))\n",
    "    plt.yticks(range(n))\n",
    "    plt.ylabel(yl)\n",
    "    plt.xlabel(xl)\n",
    "    plt.title(t)\n",
    "    # show a colorbar legend\n",
    "    plt.colorbar()\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Show our topics with the top words in each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_topics(lda_topics, tf_words, n_words=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.4\n",
    "- Pick three topics.\n",
    "- Look at the words that make up each one. Say which one it is, and briefly answer these questions about it (3-5 sentences per topic):\n",
    "    - What does the topic seem to be about?\n",
    "    - Do any of the other topics seem similar to this one?\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Examine the words that make two topics similar or different\n",
    "- We can also compare two topics to each other by looking at words that are common in both, or words that are common in one but not the other.\n",
    "- Try changing `topic_a` and `topic_b` to different topic numbers.\n",
    "- Notice the `how` option will let you see either the `overlap` or `difference` between two topics\n",
    "    - Notice also that the difference between topics a and b is not the same as between b and a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_a = 0\n",
    "topic_b = 2\n",
    "\n",
    "compare_topic_words(lda_topics, topic_a, topic_b, tf_words, how='overlap', n_words=nshow)\n",
    "compare_topic_words(lda_topics, topic_a, topic_b, tf_words, how='difference', n_words=nshow)\n",
    "compare_topic_words(lda_topics, topic_b, topic_a, tf_words, how='difference', n_words=nshow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Interpret these topics\n",
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.5\n",
    "- This part is for you to do: code can't do it for you.\n",
    "- Try to come up with a short, catchy name for each topic and write it down.\n",
    "    - For example, if the words were \"san francisco city moved living born years raised lived live\", you might call it \"places lived\" because the topic seems to be about where people currently live (San Francisco) and where they were born / raised / moved from.  \n",
    "    \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...\n",
    "\n",
    "- Topic 0:\n",
    "- Topic 1: \n",
    "- Topic 2:\n",
    "- Topic 3: \n",
    "- Topic 4:\n",
    "- Topic 5: \n",
    "- Topic 6:\n",
    "- Topic 7: \n",
    "- Topic 8: \n",
    "- Topic 9:\n",
    "- Topic 10: \n",
    "- Topic 11:\n",
    "- Topic 12: \n",
    "- Topic 13:\n",
    "- Topic 14: \n",
    "\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Check whether your interpretations match with the text\n",
    "\n",
    "#### Helper functions\n",
    "- Run this code and scroll down. \n",
    "- You don't need to understand these details right now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_profiles_from_topics(data, transformed, topic_a, topic_b=None, pick_from=10):\n",
    "    #get our data ready\n",
    "    df = pd.DataFrame(transformed)\n",
    "    df = df.sort_values(by=topic_a, ascending=False)\n",
    "    n = df.shape[0]\n",
    "    \n",
    "    if topic_b is None:\n",
    "        #if we only want things high in one topic, take randomly from the top\n",
    "        keep = df.head(pick_from).sample(1)\n",
    "        pid = keep.index.values[0]\n",
    "    else:\n",
    "        #if we want things high in two topics, find them and pick one of the top\n",
    "        idxb = df.sort_values(by=topic_b, ascending=False).index.values\n",
    "        both = find_intersection(df.index.values, idxb, pick_from)\n",
    "        keep = df.loc[both, :].sample(1)\n",
    "        pid = keep.index.values[0]    \n",
    "        \n",
    "    #output text to show our results\n",
    "    match_text = 'Profile number ' + str(pid)\n",
    "    match_text += ' has more of topic ' + str(topic_a)\n",
    "    match_text += ' than {:.2f}%'.format(100 - (np.where(df.index==pid)[0][0] / n)*100)\n",
    "    match_text += ' of other profiles'\n",
    "    if topic_b is not None:\n",
    "        match_text += ' and more of topic ' + str(topic_b)\n",
    "        match_text += ' than {:.2f}%'.format(100 - (np.where(idxb==pid)[0][0] / n)*100)\n",
    "        match_text += ' of other profiles.'\n",
    "        \n",
    "    #print results\n",
    "    text = data[pid]\n",
    "    print(match_text)\n",
    "    print('Here is the text:\\n\\n', text)\n",
    "    return\n",
    "\n",
    "def visualize_profile(profile_topics, profile_id):\n",
    "    # plot a stem diagram for a single profile\n",
    "    plt.figure(figsize=(8,4))\n",
    "    plt.xticks(range(profile_topics.shape[1]))\n",
    "    plt.xlabel('Topic number')\n",
    "    plt.ylabel('How much of profile is about each topic')\n",
    "    plt.title('Profile #'+str(profile_id))\n",
    "    plt.stem(profile_topics[profile_id,:])\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate what portion of each profile is about each topic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_topics = lda.transform(tf_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Look at the text of a profile that has a lot of a particular topic\n",
    "- This function randomly picks one of the top few profiles for a topic, so each time you run it you will see a different example.\n",
    "    - If you want it to pick from more or less topics, change the value of `pick_from`\n",
    "    - If you want to see a different topic, change the value of `topic_a`\n",
    "    - **Hint:** you can press `ctrl`+`enter` over and over to keep re-running the code in this cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_profiles_from_topics(documents, profile_topics, topic_a=7, pick_from=1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.6\n",
    "    \n",
    "- For each of the 3 topics you chose in **Step 1**, use the function get_profiles_from_topics defined above to find a document that strongly aligns with the topic. \n",
    "- Then discuss if the document truly aligns with the topic? (3-5 sentences) \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Look at the text of a profile that matches two topics well at the same time\n",
    "- Note that some topics might not happen together very often. If this is the case, the examples we find of both together might not be very good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "get_profiles_from_topics(documents, profile_topics, topic_a=1, topic_b=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See how much of a profile is about each topic\n",
    "- Try looking at some of the profiles you just found:\n",
    "    - Make the `pid` equal to the profile number from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pid = 40000 \n",
    "visualize_profile(profile_topics, profile_id=pid)\n",
    "\n",
    "\n",
    "### NEW\n",
    "documents[pid]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.7\n",
    "\n",
    "- Does the topic classification make sense to you? If so, which of the words in the example printed above are the most representitave of each of the topics? (3-5 sentences)\n",
    "    \n",
    "- Do your topic names match what you are seeing in the text? \n",
    "- Did any of your interpretations change after reading some profiles? (You would do so by changing pid and create several different charts. Then, based on the result, conclude if you want to make any changes to your topic names) \n",
    "    - If you need to update your topic names, do so here.\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 5: Topic Quality\n",
    "Let's see how good the topics we found are.\n",
    "\n",
    "### Step 1: See if the topics are each about different things.\n",
    "We want each topic to be about something different than the other topics. We can check this by comparing the words in each topic to the words in all the others. How to interpret:\n",
    "- Each square shows how similar two topics are. Darker means more similar, and lighter means more different.\n",
    "- The square in the very top left shows how similar topic 0 is to topic 0 (i.e. how similar it is to itself). \n",
    "- The square next to it in the top row shows how similar topic 0 is to topic 1, and so on. \n",
    "- For any two topics, you can see how similar they are by finding their numbers on the edges and seeing where they intersect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_topics(components):\n",
    "    sim = cosine_similarity(components)\n",
    "    blue_matrix(sim, xl='Topic number', yl='Topic number',\n",
    "                t = 'Word Similarity between Topics')\n",
    "    return\n",
    "\n",
    "plot_topics(lda_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: See if different topics show up in different profiles\n",
    "The point is to tell profiles apart based on what topics they're about, so we need to check whether the topics appear in different profiles.\n",
    "- This shows us something that looks similar to the topic similarity we saw before, but this time:\n",
    "    - We **don't** compare topics based on which words they use\n",
    "    - We **do** compare topics based on how often they appear in the same profile as one another"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def topic_cooccurance(topics):\n",
    "    co = pd.DataFrame(topics).corr()\n",
    "    blue_matrix(co, xl='Topic number', yl='Topic number',\n",
    "               t = 'Topic Co-occurance in Profiles')\n",
    "\n",
    "topic_cooccurance(profile_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Note that the topics are mostly uncorrelated. \n",
    "- The cells in the figure above are mostly very light blue\n",
    "- This doesn't mean that, for instance, topic 1 and 2 never show up in the same profile.\n",
    "- It does mean, however, that seeing any particular topic doesn't mean we're especially likely to also see any other topic.\n",
    "\n",
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.8\n",
    "- Why is the diagonal line so dark? (1-2 sentences)\n",
    "\n",
    "- Would you say that this topic classification result is good? Why? (explain in 1-3 sentences)\n",
    "\n",
    "- Do you think that the two figures above are sufficient to conclude that the topic quality is perfect? Why? (3-5 sentences)\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 6: Topic Popularity\n",
    "\n",
    "#### Helper functions to visualize and compare topics\n",
    "- Run this code and scroll down. The details of how it works aren't our focus right now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def common_topics_bars(topics):\n",
    "    popularity = pd.DataFrame(topics).mean()\n",
    "    popularity = popularity.rename_axis('Topic')\n",
    "    popularity = popularity.sort_values(ascending=False)\n",
    "    popularity.plot.bar(title='Topic popularity')\n",
    "    return\n",
    "\n",
    "def rank_groups(data, trait, topic):\n",
    "    groups = data[trait].value_counts().index.values\n",
    "    result = {}\n",
    "    \n",
    "    for g in groups:\n",
    "        result[g] = data[data[trait] == g][topic].mean()\n",
    "    \n",
    "    r = pd.DataFrame.from_dict(result, orient='index')\n",
    "    r.columns = [topic]\n",
    "    r = r.sort_values(by=topic, ascending=False)\n",
    "    \n",
    "    return r.round(3)\n",
    "\n",
    "def top_topics(data, trait, value, n_top_topics=3, distinctive=False):\n",
    "    topics = [col for col in data if col.startswith('topic_')]\n",
    "    vals = {}\n",
    "    means = {}\n",
    "    if distinctive:\n",
    "        for t in topics:\n",
    "            means[t] = data[t].mean()\n",
    "    else:\n",
    "        for t in topics:\n",
    "            means[t] = 1\n",
    "    \n",
    "    data = data[data[trait] == value]\n",
    "    \n",
    "    for t in topics:\n",
    "        vals[t] = data[t].mean() / means[t]\n",
    "    vals = pd.DataFrame.from_dict(vals, orient='index')    \n",
    "    vals = vals.sort_values(by=0, ascending=False).head(n_top_topics)\n",
    "\n",
    "    return list(vals.index.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Overall most common topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "common_topics_bars(profile_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Who is a topic most popular with?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Merge our information about topics with our information about people"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_info = pd.DataFrame(profile_topics).add_prefix('topic_')\n",
    "together = profiles.merge(topic_info, left_index=True, right_index=True)\n",
    "together.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: See the groups that have the most text about a given topic\n",
    "- The numbers here show how much of a profile, on average, is about a specific topic. For example, if you don't have a pet, you probably wouldn't be writing about your pet (topic X). \n",
    "\n",
    "#### Play around with the code in the next few cells:\n",
    "- Pick 3 of the traits we have data for. Here are the options (information we know about users from their profiles):\n",
    "    - `age_group` categories: ['10', '20', '30', '40', '50']\n",
    "    - `body` categories: ['average', 'fit', 'thin', 'overweight', 'unknown']\n",
    "    - `alcohol_use` categories: ['yes', 'no']\n",
    "    - `drug_use` categories: ['yes', 'no']\n",
    "    - `edu` (highest degree completed) categories: ['`<HS`', 'HS', 'BA', 'Grad_Pro', 'unknown'] \n",
    "    - `race_ethnicity` categories: ['Asian', 'Black', 'Latinx', 'White', 'multiple', 'other']\n",
    "    - `height_group` (whether someone is over or under six feet tall) categories: ['under_6', 'over_6']\n",
    "    - `industry` (what field they work in) categories: ['STEM', 'business', 'education', 'creative', 'med_law', 'other'] \n",
    "    - `kids` (whether they have children) categories: ['yes', 'no']\n",
    "    - `orientation` categories: ['straight', 'gay', 'bisexual']\n",
    "    - `pets_likes` (what pets they like) categories: ['both', 'dogs', 'cats', 'neither']\n",
    "    - `pets_has` (what pets they have) categories: ['both', 'dogs', 'cats', 'neither']\n",
    "    - `pets_any` (whether they have pets or not) categories: ['yes', 'no']\n",
    "    - `religion` categories: ['christianity', 'catholicism', 'judaism', 'buddhism', 'none', 'other'] \n",
    "    - `sex` categories: ['m', 'f']\n",
    "    - `smoker` categories: ['yes', 'no']\n",
    "    - `languages` categories: ['multiple', 'English_only'] \n",
    "- Change the topics and values in the code in the next few cells to explore how the trait you chose relates to the topics.\n",
    "    \n",
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.9\n",
    "\n",
    "- Before you run the code, predict what will the result be (1-2 sentences)\n",
    "\n",
    "- Write down which traits you chose to look at, and what you learned about different groups of people from the topics they used. What topics do they have in common? What topics make them different? Does this make sense given the groups? Why or why not? Remember: we interpreted the topics above, so explain your findings in terms of content, not just topic numbers. (1-2 paragraphs)\n",
    "\n",
    "- Does the actual result match your prediction? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rank_groups(together, trait='edu', topic='topic_5') ## we use this as an example, and ask students to find 3 more"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: See the topics that are most common for a given group\n",
    "- This example shows most common topics for different education groups.\n",
    "- You can change the arguments to compare different groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#show most popular topics for High School graduates\n",
    "top_topics(data=together, trait='edu', value='HS', n_top_topics=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#show most popular topics for High School graduates\n",
    "top_topics(data=together, trait='edu', value='BA', n_top_topics=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4: See the topics that distinguish a group from other groups\n",
    "- This example shows most distinctive topics for different education levels.\n",
    "- You can change the arguments to compare different groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_topics(data=together, trait='edu', value='HS', n_top_topics=3, distinctive=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_topics(data=together, trait='edu', value='Grad_Pro', n_top_topics=3, distinctive=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 7: NMF Topic Model\n",
    "- NMF is an alternative to LDA\n",
    "- NMF stands for Non-Negative Matrix Factorization. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Expand for more on how NMF works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- `Factoring` is something you may have done in math class before, for example:\n",
    "    - $ 10 $ can be factored as $ 2 \\times 5 $\n",
    "    - $ x^2+3x+2 $ can be factored as $ (x+2)(x+1)$\n",
    "- When we convert the text into numbers for the computer, it gets stored as something called a `matrix`.\n",
    "    - The matrix is `non-negative` because we can't have negative numbers of words: all the word counts are zero or more.\n",
    "- It is not important right now how exactly we find factors for these matrices, but if you're curious, you can learn more about it in a Linear Algebra class.\n",
    "- It turns out that finding factors for text is a really good way of finding topics. This makes sense intuitively: factors are simple things we can combine to get the more complicated output, and topics are simple things people combine to write profiles.\n",
    "- [Learn more](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization#Text_mining) about NMF."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Convert text to numbers the computer understands\n",
    "- NMF takes \"tf-idf vectors\" as input. Tf-idf stands for \"text frequency - inverse document frequency.\" \n",
    "    - Text frequency is the same as the count vectors we used for LDA above: how often does each word appear in the text?\n",
    "    - Inverse document frequency means we divide (\"inverse\") by the number of documents the word is in. (If everyone uses the word, it isn't very helpful for figuring out what makes people different. So this measurement looks for words that are used a lot in some documents, and not at all in others.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_vectorizer = TfidfVectorizer(max_features=1000, stop_words='english')\n",
    "\n",
    "print(\"Vectorizing text by TF-IDF...\")\n",
    "tfidf_text = tfidf_vectorizer.fit_transform(documents)\n",
    "\n",
    "tmp = tfidf_text.get_shape()\n",
    "print(\"Our transformed text has\", tmp[0], \"rows and\", tmp[1], \"columns.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The features are mostly the same as count vectors, because they are just the common words in the text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_words = tfidf_vectorizer.get_feature_names_out()\n",
    "print(\"The first few words (alphabetically) are:\\n\", tfidf_words[:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The values are different: the counts have been divided by the documents they show up in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 4\n",
    "\n",
    "print('Profile text:\\n', documents[n])\n",
    "print('\\nTF-IDF vector:')\n",
    "show_vector(tfidf_text[n], words=tfidf_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Build a topic model using NMF\n",
    "\n",
    "- NMF is faster than LDA and often works a little better for small documents like we have here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "sklearn_major_ver = int(sklearn.__version__.split('.')[0])\n",
    "\n",
    "if (sklearn_major_ver < 1):\n",
    "    model = NMF(n_components=ntopics, alpha=.1, l1_ratio=.5, init='nndsvd', random_state= 7)\n",
    "else:\n",
    "    # after sklearn v1.0, parameter alpha was removed, alpha_W and alpha_H with different approach.  \n",
    "    model = NMF(n_components=ntopics, alpha_W=.0001, alpha_H='same', l1_ratio=.5, init='nndsvd', random_state= 7)\n",
    "\n",
    "print('Performing NMF on vectors...')\n",
    "nmf = model.fit(tfidf_text)\n",
    "nmf_topics = nmf.components_\n",
    "\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Show our topics with the top words in each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_topics(nmf_topics, tfidf_words, nshow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Compare topics to each other\n",
    "We can compare topics visually by plotting the similarity of each topic's chosen words to each other topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_topics(nmf_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Examine the words that make two topics similar or different\n",
    "We can also compare two topics to each other by looking at words that are common in both, or words that are common in one but not the other. Try changing topic_a and topic_b to different topic numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_a = 1\n",
    "topic_b = 8\n",
    "\n",
    "compare_topic_words(nmf_topics, topic_a, topic_b, tfidf_words, n_words=nshow, how='overlap')\n",
    "compare_topic_words(nmf_topics, topic_a, topic_b, tfidf_words, n_words=nshow, how='difference')\n",
    "compare_topic_words(nmf_topics, topic_b, topic_a, tfidf_words, n_words=nshow, how='difference')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Interpret these topics\n",
    "\n",
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.10\n",
    "    \n",
    "- This part is for you to do: code can't do it for you.\n",
    "- Look at the list of important words for each topic, and think about these questions.\n",
    "    - What do the words have in common?\n",
    "    - What could someone write that would use most of those words?\n",
    "    - What does this topic seem to be about?\n",
    "- Try to come up with a short, catchy name for each topic.\n",
    "    - For example, if the words were \"san francisco city moved living born years raised lived live\", you might call it \"places lived\" because the topic seems to be about where people currently live (San Francisco) and where they were born / raised / moved from. \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6: Compare the topics from LDA and NMF\n",
    "\n",
    "#### Helper function to make a graph for us"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_confusion(x, y, x_label='', y_label='', t_label = '', sort=True):\n",
    "    n = x.shape[0]\n",
    "    corrs = cosine_similarity(x, y)\n",
    "    topic_similarity = pd.DataFrame(corrs)\n",
    "    new_order=None\n",
    "    \n",
    "    if sort:\n",
    "        matches = []\n",
    "        pairs = {}\n",
    "\n",
    "        for i in range(n):\n",
    "            for j in range(n):\n",
    "                tmp = {}\n",
    "                tmp['i'] = i\n",
    "                tmp['j'] = j\n",
    "                tmp['match'] = corrs[i][j]\n",
    "                matches.append(tmp)\n",
    "\n",
    "        matches = pd.DataFrame(matches).sort_values(by='match', ascending=False)\n",
    "\n",
    "        for row in matches.iterrows():\n",
    "            i = row[1]['i']\n",
    "            j = row[1]['j']\n",
    "            if i not in pairs.keys():\n",
    "                if j not in pairs.values():\n",
    "                    pairs[i] = row[1]['j']\n",
    "\n",
    "        new_order = list(range(n))\n",
    "        for k in pairs.keys():\n",
    "            new_order[int(k)] = int(pairs[k])\n",
    "\n",
    "        topic_similarity = topic_similarity[new_order]\n",
    "\n",
    "    blue_matrix(topic_similarity, xl=x_label, yl=y_label,\n",
    "                t = t_label,\n",
    "                x_labels=new_order)\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### See how similar the words in each topic from LDA are to the words in each topic from NMF\n",
    "- The NMF topics are along the X axis and the LDA are along the Y axis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion(x = nmf_topics, x_label='NMF topic number',\n",
    "               y = lda_topics, y_label='LDA topic number',\n",
    "               t_label = 'Word Similarity between LDA and NMF Topics',\n",
    "               sort=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### We can also sort the topics so that the most similar ones are aligned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_confusion(x = nmf_topics, x_label='NMF topic number',\n",
    "               y = lda_topics, y_label='LDA topic number',\n",
    "               t_label = 'Word Similarity between LDA and NMF Topics',\n",
    "               sort=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-info\">\n",
    "    \n",
    "#### Short Answer No.11\n",
    "\n",
    "Look at the LDA and NMF topic words and the confusion matrix, and consider the following questions:\n",
    "- Do any of the topics seem to be the same in both models?\n",
    "- Are some topics in one model but not the other?\n",
    "- Do the topics you get from one of the models make more sense than the ones you get from the other?\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 8: What We Learned\n",
    "- Two statistical methods for topic modeling\n",
    "    - LDA\n",
    "    - NMF\n",
    "- How to think about and interpret the topics our models find\n",
    "- How to compare and relate different topics\n",
    "- Different ways to see the distribution of topics in profiles\n",
    "- Which topics are most popular with social categories of people\n",
    "- Which social categories of people discuss a topic most"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">\n",
    "\n",
    "#### Reflection Question 1:\n",
    "- How is what we learned in this lab, using topic modeling, different from what we learned in the last lab, using just word frequencies? How is it similar? Write a paragraph explaining. \n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert-warning\">\n",
    "\n",
    "#### Reflection Question 2:\n",
    "- For both the LDA and NMF models, we specified 15 topics. Now, try running the LDA with other numbers of topics.\n",
    "    - If the topics seemed repetitive, you might want to try looking for fewer topics.\n",
    "    - If the topics seem confusing or vague, you might want to try looking for more topics (so that they can be more specific).\n",
    "- Run the code below and answer the following:\n",
    "    - What different numbers of topics did you try?\n",
    "    - How did your interpretations change in response to the number of topics? Were they it easier or more difficult to interpret?\n",
    "    - How did topic quality change in response to the number of topics? Were topics more similar or less similar?\n",
    "    \n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🤔 **Write your response here:**\n",
    "...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Decide how many topics we want to find\n",
    "- We must tell LDA how many topics we want it to look for (we did this above with the `ntopics` variable).\n",
    "    - We suggest picking a few values between 2-50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#how many topics we want our model to find\n",
    "ntopics = 2\n",
    "\n",
    "#how many top words we want to display for each topic\n",
    "nshow = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Run the LDA algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = LatentDirichletAllocation(n_components=ntopics, max_iter=10, \n",
    "                                  learning_method='online', n_jobs=-1)\n",
    "\n",
    "print('Performing LDA on vectors. This may take a while...')\n",
    "lda = model.fit(tf_text)\n",
    "lda_topics = lda.components_\n",
    "\n",
    "print('Done!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 3: Show our topics with the top words in each"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display_topics(lda_topics, tf_words, n_words=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 4: Examine the words that make two topics similar or different\n",
    "- We can also compare two topics to each other by looking at words that are common in both, or words that are common in one but not the other.\n",
    "- Try changing `topic_a` and `topic_b` to different topic numbers.\n",
    "- Notice the `how` option will let you see either the `overlap` or `difference` between two topics\n",
    "    - Notice also that the difference between topics a and b is not the same as between b and a"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "topic_a = 0\n",
    "topic_b = 1\n",
    "\n",
    "compare_topic_words(lda_topics, topic_a, topic_b, tf_words, how='overlap', n_words=nshow)\n",
    "compare_topic_words(lda_topics, topic_a, topic_b, tf_words, how='difference', n_words=nshow)\n",
    "compare_topic_words(lda_topics, topic_b, topic_a, tf_words, how='difference', n_words=nshow)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 5: Interpret these topics\n",
    "- This part is for you to do: code can't do it for you.\n",
    "- Try to come up with a short, catchy name for each topic (no writing required)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 6: Topic Quality\n",
    "Let's see how good the topics we found are.\n",
    "\n",
    "#### Step 6.1: See if the topics are each about different things.\n",
    "We want each topic to be about something different than the other topics. We can check this by comparing the words in each topic to the words in all the others. How to interpret:\n",
    "- Each square shows how similar two topics are. Darker means more similar, and lighter means more different.\n",
    "    - Notice that some topics may be negatively correlated.\n",
    "- The square in the very top left shows how similar topic 0 is to topic 0 (i.e. how similar it is to itself). \n",
    "- The square next to it in the top row shows how similar topic 0 is to topic 1, and so on. \n",
    "- For any two topics, you can see how similar they are by finding their numbers on the edges and seeing where they intersect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_topics(components):\n",
    "    sim = cosine_similarity(components)\n",
    "    blue_matrix(sim, xl='Topic number', yl='Topic number',\n",
    "                t = 'Word Similarity between Topics')\n",
    "    return\n",
    "\n",
    "plot_topics(lda_topics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 6.2: See if different topics show up in different profiles\n",
    "The point is to tell profiles apart based on what topics they're about, so we need to check whether the topics appear in different profiles.\n",
    "- This shows us something that looks similar to the topic similarity we saw before, but this time:\n",
    "    - We **don't** compare topics based on which words they use\n",
    "    - We **do** compare topics based on how often they appear in the same profile as one another"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "profile_topics = lda.transform(tf_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def topic_cooccurance(topics):\n",
    "    co = pd.DataFrame(topics).corr()\n",
    "    blue_matrix(co, xl='Topic number', yl='Topic number',\n",
    "               t = 'Topic Co-occurance in Profiles')\n",
    "\n",
    "topic_cooccurance(profile_topics)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}