examples/speaker-convo-attributes/speaker-convo-diversity-demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook provides examples of how to use convokit to perform analyses of behaviors of particular speakers within conversations. In other words, we will be dealing with attributes at the (speakers, conversation) level.\n",
    "Attributes at this granularity include linguistic diversity, described in the following paper : http://www.cs.cornell.edu/~cristian/Finding_your_voice__linguistic_development.html\n",
    "They can be used to perform longitudinal analyses of speaker behaviors across multiple conversations.\n",
    "\n",
    "Since we cannot publicly release the dataset of counseling conversations used in that paper, we will use the ChangeMyView subreddit as a test case---as such, this notebook is mostly to demonstrate how the functionality works, rather than to suggest any substantive scientific claims about longitudinal behavior change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import convokit\n",
    "from convokit import Corpus\n",
    "from convokit import download\n",
    "from convokit.text_processing import TextParser"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "imports and loading corpora:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OPTION 1: DOWNLOAD CORPUS\n",
    "# UNCOMMENT THESE LINES TO DOWNLOAD CORPUS\n",
    "# DATA_DIR = '<YOUR DIRECTORY>'\n",
    "# ROOT_DIR = convokit.download('subreddit-changemyview', data_dir=DATA_DIR)\n",
    "\n",
    "# OPTION 2: READ PREVIOUSLY-DOWNLOADED CORPUS FROM DISK\n",
    "# UNCOMMENT THIS LINE AND REPLACE WITH THE DIRECTORY WHERE THE CORPUS IS LOCATED\n",
    "# ROOT_DIR = '<YOUR DIRECTORY>'\n",
    "\n",
    "corpus = Corpus(ROOT_DIR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Speakers: 217100\n",
      "Number of Utterances: 5017556\n",
      "Number of Conversations: 117492\n"
     ]
    }
   ],
   "source": [
    "corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To start, we will set up a data structure mapping each speaker to their conversations, and each utterance they contributed in the conversation.\n",
    "\n",
    "To do this we call the `organize_speaker_convo_history` function, which annotates each `Speaker` in a corpus with a dict of conversations --> the speaker's utterances in that conversation, and the timestamp of their first utterance (i.e., when they \"entered\" the conversation).\n",
    "\n",
    "Note that we can specify what counts as participating in a conversation. Here, we omit posts and focus only on comments (such that a speaker doesn't count as participating if they only submitted the root post)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "SPEAKER_BLACKLIST = ['[deleted]', 'DeltaBot','AutoModerator']\n",
    "def utterance_is_valid(utterance):\n",
    "    return (utterance.id != utterance.conversation_id) and (utterance.speaker.id not in SPEAKER_BLACKLIST)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus.organize_speaker_convo_history(utterance_filter=utterance_is_valid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "example of what this function call gives us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1039"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.get_speaker('ThatBelligerentSloth').meta['n_convos']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1424463398"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.get_speaker('ThatBelligerentSloth').meta['start_time']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each speaker, we maintain a dictionary in their `meta` information of conversation ID to a record of the speaker's behavior in that conversation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'utterance_ids': ['cos7k4p', 'cos8ffz'],\n",
       " 'start_time': 1424491188,\n",
       " 'n_utterances': 2,\n",
       " 'idx': 2}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "corpus.get_speaker('ThatBelligerentSloth').meta['conversations']['2wm22t']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to speed up this demo, we will only take the top 100 most active speakers. \n",
    "\n",
    "To help with this, the `get_attribute_table` function call gives us a Pandas dataframe where indices correspond to speaker names, and which contains the number of comments each speaker participated in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "speaker_activities = corpus.get_attribute_table('speaker',['n_convos'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n_convos</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>cdb03b</td>\n",
       "      <td>7159.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Ansuz07</td>\n",
       "      <td>6501.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>garnteller</td>\n",
       "      <td>6290.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>hacksoncode</td>\n",
       "      <td>5947.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Nepene</td>\n",
       "      <td>5408.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>GnosticGnome</td>\n",
       "      <td>5211.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>huadpe</td>\n",
       "      <td>4847.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Grunt08</td>\n",
       "      <td>4623.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>caw81</td>\n",
       "      <td>4204.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Glory2Hypnotoad</td>\n",
       "      <td>3984.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 n_convos\n",
       "id                       \n",
       "cdb03b             7159.0\n",
       "Ansuz07            6501.0\n",
       "garnteller         6290.0\n",
       "hacksoncode        5947.0\n",
       "Nepene             5408.0\n",
       "GnosticGnome       5211.0\n",
       "huadpe             4847.0\n",
       "Grunt08            4623.0\n",
       "caw81              4204.0\n",
       "Glory2Hypnotoad    3984.0"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speaker_activities.sort_values('n_convos', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_speakers = speaker_activities.sort_values('n_convos', ascending=False).head(100).index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset_utts = []\n",
    "for speaker in top_speakers:\n",
    "    subset_utts += list(corpus.get_speaker(speaker).iter_utterances())\n",
    "subset_corpus = Corpus(utterances=subset_utts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Speakers: 100\n",
      "Number of Utterances: 539413\n",
      "Number of Conversations: 66051\n"
     ]
    }
   ],
   "source": [
    "subset_corpus.print_summary_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, to finish setting things up, we will tokenize the utterances using the `TextParser` transformer (this is somewhat slow; setting the mode to 'tokenize' means we avoid having to perform expensive dependency-parse computations, which we do not need for the present analysis)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit.text_processing import TextProcessor, TextParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000/539413 utterances processed\n",
      "2000/539413 utterances processed\n",
      "3000/539413 utterances processed\n",
      "4000/539413 utterances processed\n",
      "5000/539413 utterances processed\n",
      "6000/539413 utterances processed\n",
      "7000/539413 utterances processed\n",
      "8000/539413 utterances processed\n",
      "9000/539413 utterances processed\n",
      "10000/539413 utterances processed\n",
      "11000/539413 utterances processed\n",
      "12000/539413 utterances processed\n",
      "13000/539413 utterances processed\n",
      "14000/539413 utterances processed\n",
      "15000/539413 utterances processed\n",
      "16000/539413 utterances processed\n",
      "17000/539413 utterances processed\n",
      "18000/539413 utterances processed\n",
      "19000/539413 utterances processed\n",
      "20000/539413 utterances processed\n",
      "21000/539413 utterances processed\n",
      "22000/539413 utterances processed\n",
      "23000/539413 utterances processed\n",
      "24000/539413 utterances processed\n",
      "25000/539413 utterances processed\n",
      "26000/539413 utterances processed\n",
      "27000/539413 utterances processed\n",
      "28000/539413 utterances processed\n",
      "29000/539413 utterances processed\n",
      "30000/539413 utterances processed\n",
      "31000/539413 utterances processed\n",
      "32000/539413 utterances processed\n",
      "33000/539413 utterances processed\n",
      "34000/539413 utterances processed\n",
      "35000/539413 utterances processed\n",
      "36000/539413 utterances processed\n",
      "37000/539413 utterances processed\n",
      "38000/539413 utterances processed\n",
      "39000/539413 utterances processed\n",
      "40000/539413 utterances processed\n",
      "41000/539413 utterances processed\n",
      "42000/539413 utterances processed\n",
      "43000/539413 utterances processed\n",
      "44000/539413 utterances processed\n",
      "45000/539413 utterances processed\n",
      "46000/539413 utterances processed\n",
      "47000/539413 utterances processed\n",
      "48000/539413 utterances processed\n",
      "49000/539413 utterances processed\n",
      "50000/539413 utterances processed\n",
      "51000/539413 utterances processed\n",
      "52000/539413 utterances processed\n",
      "53000/539413 utterances processed\n",
      "54000/539413 utterances processed\n",
      "55000/539413 utterances processed\n",
      "56000/539413 utterances processed\n",
      "57000/539413 utterances processed\n",
      "58000/539413 utterances processed\n",
      "59000/539413 utterances processed\n",
      "60000/539413 utterances processed\n",
      "61000/539413 utterances processed\n",
      "62000/539413 utterances processed\n",
      "63000/539413 utterances processed\n",
      "64000/539413 utterances processed\n",
      "65000/539413 utterances processed\n",
      "66000/539413 utterances processed\n",
      "67000/539413 utterances processed\n",
      "68000/539413 utterances processed\n",
      "69000/539413 utterances processed\n",
      "70000/539413 utterances processed\n",
      "71000/539413 utterances processed\n",
      "72000/539413 utterances processed\n",
      "73000/539413 utterances processed\n",
      "74000/539413 utterances processed\n",
      "75000/539413 utterances processed\n",
      "76000/539413 utterances processed\n",
      "77000/539413 utterances processed\n",
      "78000/539413 utterances processed\n",
      "79000/539413 utterances processed\n",
      "80000/539413 utterances processed\n",
      "81000/539413 utterances processed\n",
      "82000/539413 utterances processed\n",
      "83000/539413 utterances processed\n",
      "84000/539413 utterances processed\n",
      "85000/539413 utterances processed\n",
      "86000/539413 utterances processed\n",
      "87000/539413 utterances processed\n",
      "88000/539413 utterances processed\n",
      "89000/539413 utterances processed\n",
      "90000/539413 utterances processed\n",
      "91000/539413 utterances processed\n",
      "92000/539413 utterances processed\n",
      "93000/539413 utterances processed\n",
      "94000/539413 utterances processed\n",
      "95000/539413 utterances processed\n",
      "96000/539413 utterances processed\n",
      "97000/539413 utterances processed\n",
      "98000/539413 utterances processed\n",
      "99000/539413 utterances processed\n",
      "100000/539413 utterances processed\n",
      "101000/539413 utterances processed\n",
      "102000/539413 utterances processed\n",
      "103000/539413 utterances processed\n",
      "104000/539413 utterances processed\n",
      "105000/539413 utterances processed\n",
      "106000/539413 utterances processed\n",
      "107000/539413 utterances processed\n",
      "108000/539413 utterances processed\n",
      "109000/539413 utterances processed\n",
      "110000/539413 utterances processed\n",
      "111000/539413 utterances processed\n",
      "112000/539413 utterances processed\n",
      "113000/539413 utterances processed\n",
      "114000/539413 utterances processed\n",
      "115000/539413 utterances processed\n",
      "116000/539413 utterances processed\n",
      "117000/539413 utterances processed\n",
      "118000/539413 utterances processed\n",
      "119000/539413 utterances processed\n",
      "120000/539413 utterances processed\n",
      "121000/539413 utterances processed\n",
      "122000/539413 utterances processed\n",
      "123000/539413 utterances processed\n",
      "124000/539413 utterances processed\n",
      "125000/539413 utterances processed\n",
      "126000/539413 utterances processed\n",
      "127000/539413 utterances processed\n",
      "128000/539413 utterances processed\n",
      "129000/539413 utterances processed\n",
      "130000/539413 utterances processed\n",
      "131000/539413 utterances processed\n",
      "132000/539413 utterances processed\n",
      "133000/539413 utterances processed\n",
      "134000/539413 utterances processed\n",
      "135000/539413 utterances processed\n",
      "136000/539413 utterances processed\n",
      "137000/539413 utterances processed\n",
      "138000/539413 utterances processed\n",
      "139000/539413 utterances processed\n",
      "140000/539413 utterances processed\n",
      "141000/539413 utterances processed\n",
      "142000/539413 utterances processed\n",
      "143000/539413 utterances processed\n",
      "144000/539413 utterances processed\n",
      "145000/539413 utterances processed\n",
      "146000/539413 utterances processed\n",
      "147000/539413 utterances processed\n",
      "148000/539413 utterances processed\n",
      "149000/539413 utterances processed\n",
      "150000/539413 utterances processed\n",
      "151000/539413 utterances processed\n",
      "152000/539413 utterances processed\n",
      "153000/539413 utterances processed\n",
      "154000/539413 utterances processed\n",
      "155000/539413 utterances processed\n",
      "156000/539413 utterances processed\n",
      "157000/539413 utterances processed\n",
      "158000/539413 utterances processed\n",
      "159000/539413 utterances processed\n",
      "160000/539413 utterances processed\n",
      "161000/539413 utterances processed\n",
      "162000/539413 utterances processed\n",
      "163000/539413 utterances processed\n",
      "164000/539413 utterances processed\n",
      "165000/539413 utterances processed\n",
      "166000/539413 utterances processed\n",
      "167000/539413 utterances processed\n",
      "168000/539413 utterances processed\n",
      "169000/539413 utterances processed\n",
      "170000/539413 utterances processed\n",
      "171000/539413 utterances processed\n",
      "172000/539413 utterances processed\n",
      "173000/539413 utterances processed\n",
      "174000/539413 utterances processed\n",
      "175000/539413 utterances processed\n",
      "176000/539413 utterances processed\n",
      "177000/539413 utterances processed\n",
      "178000/539413 utterances processed\n",
      "179000/539413 utterances processed\n",
      "180000/539413 utterances processed\n",
      "181000/539413 utterances processed\n",
      "182000/539413 utterances processed\n",
      "183000/539413 utterances processed\n",
      "184000/539413 utterances processed\n",
      "185000/539413 utterances processed\n",
      "186000/539413 utterances processed\n",
      "187000/539413 utterances processed\n",
      "188000/539413 utterances processed\n",
      "189000/539413 utterances processed\n",
      "190000/539413 utterances processed\n",
      "191000/539413 utterances processed\n",
      "192000/539413 utterances processed\n",
      "193000/539413 utterances processed\n",
      "194000/539413 utterances processed\n",
      "195000/539413 utterances processed\n",
      "196000/539413 utterances processed\n",
      "197000/539413 utterances processed\n",
      "198000/539413 utterances processed\n",
      "199000/539413 utterances processed\n",
      "200000/539413 utterances processed\n",
      "201000/539413 utterances processed\n",
      "202000/539413 utterances processed\n",
      "203000/539413 utterances processed\n",
      "204000/539413 utterances processed\n",
      "205000/539413 utterances processed\n",
      "206000/539413 utterances processed\n",
      "207000/539413 utterances processed\n",
      "208000/539413 utterances processed\n",
      "209000/539413 utterances processed\n",
      "210000/539413 utterances processed\n",
      "211000/539413 utterances processed\n",
      "212000/539413 utterances processed\n",
      "213000/539413 utterances processed\n",
      "214000/539413 utterances processed\n",
      "215000/539413 utterances processed\n",
      "216000/539413 utterances processed\n",
      "217000/539413 utterances processed\n",
      "218000/539413 utterances processed\n",
      "219000/539413 utterances processed\n",
      "220000/539413 utterances processed\n",
      "221000/539413 utterances processed\n",
      "222000/539413 utterances processed\n",
      "223000/539413 utterances processed\n",
      "224000/539413 utterances processed\n",
      "225000/539413 utterances processed\n",
      "226000/539413 utterances processed\n",
      "227000/539413 utterances processed\n",
      "228000/539413 utterances processed\n",
      "229000/539413 utterances processed\n",
      "230000/539413 utterances processed\n",
      "231000/539413 utterances processed\n",
      "232000/539413 utterances processed\n",
      "233000/539413 utterances processed\n",
      "234000/539413 utterances processed\n",
      "235000/539413 utterances processed\n",
      "236000/539413 utterances processed\n",
      "237000/539413 utterances processed\n",
      "238000/539413 utterances processed\n",
      "239000/539413 utterances processed\n",
      "240000/539413 utterances processed\n",
      "241000/539413 utterances processed\n",
      "242000/539413 utterances processed\n",
      "243000/539413 utterances processed\n",
      "244000/539413 utterances processed\n",
      "245000/539413 utterances processed\n",
      "246000/539413 utterances processed\n",
      "247000/539413 utterances processed\n",
      "248000/539413 utterances processed\n",
      "249000/539413 utterances processed\n",
      "250000/539413 utterances processed\n",
      "251000/539413 utterances processed\n",
      "252000/539413 utterances processed\n",
      "253000/539413 utterances processed\n",
      "254000/539413 utterances processed\n",
      "255000/539413 utterances processed\n",
      "256000/539413 utterances processed\n",
      "257000/539413 utterances processed\n",
      "258000/539413 utterances processed\n",
      "259000/539413 utterances processed\n",
      "260000/539413 utterances processed\n",
      "261000/539413 utterances processed\n",
      "262000/539413 utterances processed\n",
      "263000/539413 utterances processed\n",
      "264000/539413 utterances processed\n",
      "265000/539413 utterances processed\n",
      "266000/539413 utterances processed\n",
      "267000/539413 utterances processed\n",
      "268000/539413 utterances processed\n",
      "269000/539413 utterances processed\n",
      "270000/539413 utterances processed\n",
      "271000/539413 utterances processed\n",
      "272000/539413 utterances processed\n",
      "273000/539413 utterances processed\n",
      "274000/539413 utterances processed\n",
      "275000/539413 utterances processed\n",
      "276000/539413 utterances processed\n",
      "277000/539413 utterances processed\n",
      "278000/539413 utterances processed\n",
      "279000/539413 utterances processed\n",
      "280000/539413 utterances processed\n",
      "281000/539413 utterances processed\n",
      "282000/539413 utterances processed\n",
      "283000/539413 utterances processed\n",
      "284000/539413 utterances processed\n",
      "285000/539413 utterances processed\n",
      "286000/539413 utterances processed\n",
      "287000/539413 utterances processed\n",
      "288000/539413 utterances processed\n",
      "289000/539413 utterances processed\n",
      "290000/539413 utterances processed\n",
      "291000/539413 utterances processed\n",
      "292000/539413 utterances processed\n",
      "293000/539413 utterances processed\n",
      "294000/539413 utterances processed\n",
      "295000/539413 utterances processed\n",
      "296000/539413 utterances processed\n",
      "297000/539413 utterances processed\n",
      "298000/539413 utterances processed\n",
      "299000/539413 utterances processed\n",
      "300000/539413 utterances processed\n",
      "301000/539413 utterances processed\n",
      "302000/539413 utterances processed\n",
      "303000/539413 utterances processed\n",
      "304000/539413 utterances processed\n",
      "305000/539413 utterances processed\n",
      "306000/539413 utterances processed\n",
      "307000/539413 utterances processed\n",
      "308000/539413 utterances processed\n",
      "309000/539413 utterances processed\n",
      "310000/539413 utterances processed\n",
      "311000/539413 utterances processed\n",
      "312000/539413 utterances processed\n",
      "313000/539413 utterances processed\n",
      "314000/539413 utterances processed\n",
      "315000/539413 utterances processed\n",
      "316000/539413 utterances processed\n",
      "317000/539413 utterances processed\n",
      "318000/539413 utterances processed\n",
      "319000/539413 utterances processed\n",
      "320000/539413 utterances processed\n",
      "321000/539413 utterances processed\n",
      "322000/539413 utterances processed\n",
      "323000/539413 utterances processed\n",
      "324000/539413 utterances processed\n",
      "325000/539413 utterances processed\n",
      "326000/539413 utterances processed\n",
      "327000/539413 utterances processed\n",
      "328000/539413 utterances processed\n",
      "329000/539413 utterances processed\n",
      "330000/539413 utterances processed\n",
      "331000/539413 utterances processed\n",
      "332000/539413 utterances processed\n",
      "333000/539413 utterances processed\n",
      "334000/539413 utterances processed\n",
      "335000/539413 utterances processed\n",
      "336000/539413 utterances processed\n",
      "337000/539413 utterances processed\n",
      "338000/539413 utterances processed\n",
      "339000/539413 utterances processed\n",
      "340000/539413 utterances processed\n",
      "341000/539413 utterances processed\n",
      "342000/539413 utterances processed\n",
      "343000/539413 utterances processed\n",
      "344000/539413 utterances processed\n",
      "345000/539413 utterances processed\n",
      "346000/539413 utterances processed\n",
      "347000/539413 utterances processed\n",
      "348000/539413 utterances processed\n",
      "349000/539413 utterances processed\n",
      "350000/539413 utterances processed\n",
      "351000/539413 utterances processed\n",
      "352000/539413 utterances processed\n",
      "353000/539413 utterances processed\n",
      "354000/539413 utterances processed\n",
      "355000/539413 utterances processed\n",
      "356000/539413 utterances processed\n",
      "357000/539413 utterances processed\n",
      "358000/539413 utterances processed\n",
      "359000/539413 utterances processed\n",
      "360000/539413 utterances processed\n",
      "361000/539413 utterances processed\n",
      "362000/539413 utterances processed\n",
      "363000/539413 utterances processed\n",
      "364000/539413 utterances processed\n",
      "365000/539413 utterances processed\n",
      "366000/539413 utterances processed\n",
      "367000/539413 utterances processed\n",
      "368000/539413 utterances processed\n",
      "369000/539413 utterances processed\n",
      "370000/539413 utterances processed\n",
      "371000/539413 utterances processed\n",
      "372000/539413 utterances processed\n",
      "373000/539413 utterances processed\n",
      "374000/539413 utterances processed\n",
      "375000/539413 utterances processed\n",
      "376000/539413 utterances processed\n",
      "377000/539413 utterances processed\n",
      "378000/539413 utterances processed\n",
      "379000/539413 utterances processed\n",
      "380000/539413 utterances processed\n",
      "381000/539413 utterances processed\n",
      "382000/539413 utterances processed\n",
      "383000/539413 utterances processed\n",
      "384000/539413 utterances processed\n",
      "385000/539413 utterances processed\n",
      "386000/539413 utterances processed\n",
      "387000/539413 utterances processed\n",
      "388000/539413 utterances processed\n",
      "389000/539413 utterances processed\n",
      "390000/539413 utterances processed\n",
      "391000/539413 utterances processed\n",
      "392000/539413 utterances processed\n",
      "393000/539413 utterances processed\n",
      "394000/539413 utterances processed\n",
      "395000/539413 utterances processed\n",
      "396000/539413 utterances processed\n",
      "397000/539413 utterances processed\n",
      "398000/539413 utterances processed\n",
      "399000/539413 utterances processed\n",
      "400000/539413 utterances processed\n",
      "401000/539413 utterances processed\n",
      "402000/539413 utterances processed\n",
      "403000/539413 utterances processed\n",
      "404000/539413 utterances processed\n",
      "405000/539413 utterances processed\n",
      "406000/539413 utterances processed\n",
      "407000/539413 utterances processed\n",
      "408000/539413 utterances processed\n",
      "409000/539413 utterances processed\n",
      "410000/539413 utterances processed\n",
      "411000/539413 utterances processed\n",
      "412000/539413 utterances processed\n",
      "413000/539413 utterances processed\n",
      "414000/539413 utterances processed\n",
      "415000/539413 utterances processed\n",
      "416000/539413 utterances processed\n",
      "417000/539413 utterances processed\n",
      "418000/539413 utterances processed\n",
      "419000/539413 utterances processed\n",
      "420000/539413 utterances processed\n",
      "421000/539413 utterances processed\n",
      "422000/539413 utterances processed\n",
      "423000/539413 utterances processed\n",
      "424000/539413 utterances processed\n",
      "425000/539413 utterances processed\n",
      "426000/539413 utterances processed\n",
      "427000/539413 utterances processed\n",
      "428000/539413 utterances processed\n",
      "429000/539413 utterances processed\n",
      "430000/539413 utterances processed\n",
      "431000/539413 utterances processed\n",
      "432000/539413 utterances processed\n",
      "433000/539413 utterances processed\n",
      "434000/539413 utterances processed\n",
      "435000/539413 utterances processed\n",
      "436000/539413 utterances processed\n",
      "437000/539413 utterances processed\n",
      "438000/539413 utterances processed\n",
      "439000/539413 utterances processed\n",
      "440000/539413 utterances processed\n",
      "441000/539413 utterances processed\n",
      "442000/539413 utterances processed\n",
      "443000/539413 utterances processed\n",
      "444000/539413 utterances processed\n",
      "445000/539413 utterances processed\n",
      "446000/539413 utterances processed\n",
      "447000/539413 utterances processed\n",
      "448000/539413 utterances processed\n",
      "449000/539413 utterances processed\n",
      "450000/539413 utterances processed\n",
      "451000/539413 utterances processed\n",
      "452000/539413 utterances processed\n",
      "453000/539413 utterances processed\n",
      "454000/539413 utterances processed\n",
      "455000/539413 utterances processed\n",
      "456000/539413 utterances processed\n",
      "457000/539413 utterances processed\n",
      "458000/539413 utterances processed\n",
      "459000/539413 utterances processed\n",
      "460000/539413 utterances processed\n",
      "461000/539413 utterances processed\n",
      "462000/539413 utterances processed\n",
      "463000/539413 utterances processed\n",
      "464000/539413 utterances processed\n",
      "465000/539413 utterances processed\n",
      "466000/539413 utterances processed\n",
      "467000/539413 utterances processed\n",
      "468000/539413 utterances processed\n",
      "469000/539413 utterances processed\n",
      "470000/539413 utterances processed\n",
      "471000/539413 utterances processed\n",
      "472000/539413 utterances processed\n",
      "473000/539413 utterances processed\n",
      "474000/539413 utterances processed\n",
      "475000/539413 utterances processed\n",
      "476000/539413 utterances processed\n",
      "477000/539413 utterances processed\n",
      "478000/539413 utterances processed\n",
      "479000/539413 utterances processed\n",
      "480000/539413 utterances processed\n",
      "481000/539413 utterances processed\n",
      "482000/539413 utterances processed\n",
      "483000/539413 utterances processed\n",
      "484000/539413 utterances processed\n",
      "485000/539413 utterances processed\n",
      "486000/539413 utterances processed\n",
      "487000/539413 utterances processed\n",
      "488000/539413 utterances processed\n",
      "489000/539413 utterances processed\n",
      "490000/539413 utterances processed\n",
      "491000/539413 utterances processed\n",
      "492000/539413 utterances processed\n",
      "493000/539413 utterances processed\n",
      "494000/539413 utterances processed\n",
      "495000/539413 utterances processed\n",
      "496000/539413 utterances processed\n",
      "497000/539413 utterances processed\n",
      "498000/539413 utterances processed\n",
      "499000/539413 utterances processed\n",
      "500000/539413 utterances processed\n",
      "501000/539413 utterances processed\n",
      "502000/539413 utterances processed\n",
      "503000/539413 utterances processed\n",
      "504000/539413 utterances processed\n",
      "505000/539413 utterances processed\n",
      "506000/539413 utterances processed\n",
      "507000/539413 utterances processed\n",
      "508000/539413 utterances processed\n",
      "509000/539413 utterances processed\n",
      "510000/539413 utterances processed\n",
      "511000/539413 utterances processed\n",
      "512000/539413 utterances processed\n",
      "513000/539413 utterances processed\n",
      "514000/539413 utterances processed\n",
      "515000/539413 utterances processed\n",
      "516000/539413 utterances processed\n",
      "517000/539413 utterances processed\n",
      "518000/539413 utterances processed\n",
      "519000/539413 utterances processed\n",
      "520000/539413 utterances processed\n",
      "521000/539413 utterances processed\n",
      "522000/539413 utterances processed\n",
      "523000/539413 utterances processed\n",
      "524000/539413 utterances processed\n",
      "525000/539413 utterances processed\n",
      "526000/539413 utterances processed\n",
      "527000/539413 utterances processed\n",
      "528000/539413 utterances processed\n",
      "529000/539413 utterances processed\n",
      "530000/539413 utterances processed\n",
      "531000/539413 utterances processed\n",
      "532000/539413 utterances processed\n",
      "533000/539413 utterances processed\n",
      "534000/539413 utterances processed\n",
      "535000/539413 utterances processed\n",
      "536000/539413 utterances processed\n",
      "537000/539413 utterances processed\n",
      "538000/539413 utterances processed\n",
      "539000/539413 utterances processed\n",
      "539413/539413 utterances processed\n"
     ]
    }
   ],
   "source": [
    "tokenizer = TextParser(mode='tokenize', output_field='tokens', verbosity=1000)\n",
    "subset_corpus = tokenizer.transform(subset_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's what the tokenized output looks like for one utterance (for a more in-depth explanation, check out the `TextParser` documentation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'toks': [{'tok': 'Strictly'},\n",
       "   {'tok': 'speaking'},\n",
       "   {'tok': 'yes'},\n",
       "   {'tok': ','},\n",
       "   {'tok': 'they'},\n",
       "   {'tok': 'are'},\n",
       "   {'tok': 'probably'},\n",
       "   {'tok': 'entitled'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'their'},\n",
       "   {'tok': 'view'},\n",
       "   {'tok': 'if'},\n",
       "   {'tok': 'they'},\n",
       "   {'tok': 'live'},\n",
       "   {'tok': 'in'},\n",
       "   {'tok': 'a'},\n",
       "   {'tok': 'developed'},\n",
       "   {'tok': 'country'},\n",
       "   {'tok': '.'}]},\n",
       " {'toks': [{'tok': 'Typically'},\n",
       "   {'tok': 'these'},\n",
       "   {'tok': 'countries'},\n",
       "   {'tok': 'agree'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'by'},\n",
       "   {'tok': 'and'},\n",
       "   {'tok': 'large'},\n",
       "   {'tok': 'protect'},\n",
       "   {'tok': 'speech'},\n",
       "   {'tok': 'as'},\n",
       "   {'tok': 'free'},\n",
       "   {'tok': '.'}]},\n",
       " {'toks': [{'tok': 'You'},\n",
       "   {'tok': 'are'},\n",
       "   {'tok': 'literally'},\n",
       "   {'tok': 'entitled'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'say'},\n",
       "   {'tok': 'whatever'},\n",
       "   {'tok': 'you'},\n",
       "   {'tok': 'want'},\n",
       "   {'tok': '.'}]},\n",
       " {'toks': [{'tok': 'However'},\n",
       "   {'tok': 'this'},\n",
       "   {'tok': 'does'},\n",
       "   {'tok': 'not'},\n",
       "   {'tok': 'mean'},\n",
       "   {'tok': 'I'},\n",
       "   {'tok': 'am'},\n",
       "   {'tok': 'not'},\n",
       "   {'tok': 'similarly'},\n",
       "   {'tok': 'entitled'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'act'},\n",
       "   {'tok': 'in'},\n",
       "   {'tok': 'whatever'},\n",
       "   {'tok': 'way'},\n",
       "   {'tok': 'I'},\n",
       "   {'tok': 'wish'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'under'},\n",
       "   {'tok': 'existing'},\n",
       "   {'tok': ','},\n",
       "   {'tok': 'presumably'},\n",
       "   {'tok': 'constitutional'},\n",
       "   {'tok': 'law'},\n",
       "   {'tok': 'in'},\n",
       "   {'tok': 'response'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'your'},\n",
       "   {'tok': 'opinion'},\n",
       "   {'tok': '.'}]},\n",
       " {'toks': [{'tok': 'So'},\n",
       "   {'tok': 'yes'},\n",
       "   {'tok': ','},\n",
       "   {'tok': 'you'},\n",
       "   {'tok': 'are'},\n",
       "   {'tok': 'almost'},\n",
       "   {'tok': 'always'},\n",
       "   {'tok': 'entitled'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'your'},\n",
       "   {'tok': 'opinion'},\n",
       "   {'tok': ','},\n",
       "   {'tok': 'but'},\n",
       "   {'tok': 'another'},\n",
       "   {'tok': 'is'},\n",
       "   {'tok': 'also'},\n",
       "   {'tok': 'entitled'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': 'react'},\n",
       "   {'tok': 'however'},\n",
       "   {'tok': 'they'},\n",
       "   {'tok': 'legally'},\n",
       "   {'tok': 'want'},\n",
       "   {'tok': 'to'},\n",
       "   {'tok': '.'}]}]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_corpus.get_utterance('cos7k4p').retrieve_meta('tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this analysis is to examine how a speaker's conversational behavior looks like within a single conversation, and then how it evolves over the conversations they take. To demonstrate what this looks like we'll start with a simple attribute, wordcount. \n",
    "First, we count the words in each utterance using the `TextProcessor` transformer. Note this computes _per utterance_ statistics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/539413 utterances processed\n",
      "50000/539413 utterances processed\n",
      "75000/539413 utterances processed\n",
      "100000/539413 utterances processed\n",
      "125000/539413 utterances processed\n",
      "150000/539413 utterances processed\n",
      "175000/539413 utterances processed\n",
      "200000/539413 utterances processed\n",
      "225000/539413 utterances processed\n",
      "250000/539413 utterances processed\n",
      "275000/539413 utterances processed\n",
      "300000/539413 utterances processed\n",
      "325000/539413 utterances processed\n",
      "350000/539413 utterances processed\n",
      "375000/539413 utterances processed\n",
      "400000/539413 utterances processed\n",
      "425000/539413 utterances processed\n",
      "450000/539413 utterances processed\n",
      "475000/539413 utterances processed\n",
      "500000/539413 utterances processed\n",
      "525000/539413 utterances processed\n",
      "539413/539413 utterances processed\n"
     ]
    }
   ],
   "source": [
    "wordcounter = TextProcessor(input_field='tokens', output_field='wordcount', \n",
    "                           proc_fn=lambda sents: sum(len(sent['toks']) for sent in sents), verbosity=25000)\n",
    "subset_corpus = wordcounter.transform(subset_corpus) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "97"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_corpus.get_utterance('cos7k4p').retrieve_meta('wordcount')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_corpus.get_utterance('cos8ffz').retrieve_meta('wordcount')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we aggregate per-utterance statistics over all the utterances a particular speaker contributed in a conversation. That is, we will turn wordcount into a speaker,convo-level attribute.\n",
    "\n",
    "We call the `SpeakerConvoAttrs` transformer to do this. Here, `agg_fn=np.mean` means that the speaker,convo-level attribute is an _average_ over utterance lengths, but you could replace this with your own aggregation function (e.g., `max`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc_wordcount = convokit.speaker_convo_helpers.speaker_convo_attrs.SpeakerConvoAttrs('wordcount', agg_fn=np.mean)\n",
    "subset_corpus = sc_wordcount.transform(subset_corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This transformer annotates each conversation in each Speaker object with a (mean) wordcount:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'utterance_ids': ['cos7k4p', 'cos8ffz'],\n",
       " 'start_time': 1424491188,\n",
       " 'n_utterances': 2,\n",
       " 'idx': 2,\n",
       " 'wordcount': 64.5,\n",
       " 'lifestage': 0,\n",
       " 'tokens': ['strictly',\n",
       "  'speaking',\n",
       "  'yes',\n",
       "  ',',\n",
       "  'they',\n",
       "  'are',\n",
       "  'probably',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'their',\n",
       "  'view',\n",
       "  'if',\n",
       "  'they',\n",
       "  'live',\n",
       "  'in',\n",
       "  'a',\n",
       "  'developed',\n",
       "  'country',\n",
       "  '.',\n",
       "  'typically',\n",
       "  'these',\n",
       "  'countries',\n",
       "  'agree',\n",
       "  'to',\n",
       "  'by',\n",
       "  'and',\n",
       "  'large',\n",
       "  'protect',\n",
       "  'speech',\n",
       "  'as',\n",
       "  'free',\n",
       "  '.',\n",
       "  'you',\n",
       "  'are',\n",
       "  'literally',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'say',\n",
       "  'whatever',\n",
       "  'you',\n",
       "  'want',\n",
       "  '.',\n",
       "  'however',\n",
       "  'this',\n",
       "  'does',\n",
       "  'not',\n",
       "  'mean',\n",
       "  'i',\n",
       "  'am',\n",
       "  'not',\n",
       "  'similarly',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'act',\n",
       "  'in',\n",
       "  'whatever',\n",
       "  'way',\n",
       "  'i',\n",
       "  'wish',\n",
       "  'to',\n",
       "  'under',\n",
       "  'existing',\n",
       "  ',',\n",
       "  'presumably',\n",
       "  'constitutional',\n",
       "  'law',\n",
       "  'in',\n",
       "  'response',\n",
       "  'to',\n",
       "  'your',\n",
       "  'opinion',\n",
       "  '.',\n",
       "  'so',\n",
       "  'yes',\n",
       "  ',',\n",
       "  'you',\n",
       "  'are',\n",
       "  'almost',\n",
       "  'always',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'your',\n",
       "  'opinion',\n",
       "  ',',\n",
       "  'but',\n",
       "  'another',\n",
       "  'is',\n",
       "  'also',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'react',\n",
       "  'however',\n",
       "  'they',\n",
       "  'legally',\n",
       "  'want',\n",
       "  'to',\n",
       "  '.',\n",
       "  'nor',\n",
       "  'am',\n",
       "  'i',\n",
       "  ',',\n",
       "  'just',\n",
       "  'that',\n",
       "  'they',\n",
       "  'are',\n",
       "  '*',\n",
       "  '*',\n",
       "  'literally',\n",
       "  '*',\n",
       "  '*',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'their',\n",
       "  'opinions',\n",
       "  '.',\n",
       "  'you',\n",
       "  'are',\n",
       "  'similarly',\n",
       "  'entitled',\n",
       "  'to',\n",
       "  'react',\n",
       "  'in',\n",
       "  'whatever',\n",
       "  'legal',\n",
       "  'way',\n",
       "  'you',\n",
       "  'wish',\n",
       "  'to',\n",
       "  '.']}"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subset_corpus.get_speaker('ThatBelligerentSloth').meta['conversations']['2wm22t']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now use this aggregate statistic to analyze how speakers change behavior over time. The particular question here is whether or not speakers systematically increase or decrease in wordcount, and in the number of utterances contributed to each conversation.\n",
    "\n",
    "To facilitate further analyses, we'll load all the speaker,convo information pertaining to the attributes we want into a dataframe. We'll use the `get_full_attribute_table` function to do this (the particular call tells the function to load a table with wordcount and # of utterances at the speaker,conversation level, and # of conversations i.e., how active the speaker was, at the speaker level)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "speaker_convo_len_df = subset_corpus.get_full_attribute_table(speaker_convo_attrs=['wordcount','n_utterances'],\n",
    "                                             speaker_attrs=['n_convos'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>speaker</th>\n",
       "      <th>convo_id</th>\n",
       "      <th>convo_idx</th>\n",
       "      <th>wordcount</th>\n",
       "      <th>n_utterances</th>\n",
       "      <th>n_convos__speaker</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>cdb03b__18x6j5</td>\n",
       "      <td>cdb03b</td>\n",
       "      <td>18x6j5</td>\n",
       "      <td>0</td>\n",
       "      <td>24.5</td>\n",
       "      <td>2</td>\n",
       "      <td>7159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>cdb03b__1adg1v</td>\n",
       "      <td>cdb03b</td>\n",
       "      <td>1adg1v</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "      <td>7159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>cdb03b__1cciah</td>\n",
       "      <td>cdb03b</td>\n",
       "      <td>1cciah</td>\n",
       "      <td>2</td>\n",
       "      <td>25.0</td>\n",
       "      <td>2</td>\n",
       "      <td>7159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>cdb03b__1ccvs4</td>\n",
       "      <td>cdb03b</td>\n",
       "      <td>1ccvs4</td>\n",
       "      <td>3</td>\n",
       "      <td>35.0</td>\n",
       "      <td>1</td>\n",
       "      <td>7159</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>cdb03b__1e2r7u</td>\n",
       "      <td>cdb03b</td>\n",
       "      <td>1e2r7u</td>\n",
       "      <td>4</td>\n",
       "      <td>74.0</td>\n",
       "      <td>1</td>\n",
       "      <td>7159</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               speaker convo_id  convo_idx  wordcount  n_utterances  \\\n",
       "id                                                                    \n",
       "cdb03b__18x6j5  cdb03b   18x6j5          0       24.5             2   \n",
       "cdb03b__1adg1v  cdb03b   1adg1v          1        4.0             1   \n",
       "cdb03b__1cciah  cdb03b   1cciah          2       25.0             2   \n",
       "cdb03b__1ccvs4  cdb03b   1ccvs4          3       35.0             1   \n",
       "cdb03b__1e2r7u  cdb03b   1e2r7u          4       74.0             1   \n",
       "\n",
       "                n_convos__speaker  \n",
       "id                                 \n",
       "cdb03b__18x6j5               7159  \n",
       "cdb03b__1adg1v               7159  \n",
       "cdb03b__1cciah               7159  \n",
       "cdb03b__1ccvs4               7159  \n",
       "cdb03b__1e2r7u               7159  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speaker_convo_len_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We perform our longitudinal analyses at the level of life-stages: i.e., contiguous blocks of conversations. Here, we compare between the first two life-stages of 10 conversations: how the speaker behaves in their first 10, versus their 10th to 20th conversations. \n",
    "We say that speakers systematically increase (or decrease) in an attribute if for a significant majority of speakers the value of this attribute at one life-stage increases to the next. \n",
    "\n",
    "To this end, we need to aggregate attributes over a life-stage, e.g., mean wordcount. To perform this aggregation we'll use the `get_lifestage_attributes` function, specifying lifestages of 10 conversations each.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_lifestage_attributes(attr_df, attr, lifestage_size, agg_fn=np.mean):\n",
    "    aggs = attr_df.groupby(['speaker', attr_df.convo_idx // lifestage_size])\\\n",
    "        [attr].agg(agg_fn)\n",
    "    aggs = aggs.reset_index().pivot(index='speaker', columns='convo_idx',\n",
    "                                   values=attr)\n",
    "    return aggs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We focus on the first 20 conversations (i.e., 2 life-stages). We also ignore all speakers with less than 20 conversations---so we are not biased by survivorship."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "subset = speaker_convo_len_df[(speaker_convo_len_df.n_convos__speaker >= 20)\n",
    "                          & (speaker_convo_len_df.convo_idx < 20)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "stage_wc_df = get_lifestage_attributes(subset, 'wordcount', 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>convo_idx</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>speaker</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>ACrusaderA</td>\n",
       "      <td>99.640812</td>\n",
       "      <td>131.716667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>A_Mirror</td>\n",
       "      <td>71.400000</td>\n",
       "      <td>94.583333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>A_Soporific</td>\n",
       "      <td>287.816667</td>\n",
       "      <td>229.133333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>AlphaGoGoDancer</td>\n",
       "      <td>297.825000</td>\n",
       "      <td>418.933333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>Amablue</td>\n",
       "      <td>161.700000</td>\n",
       "      <td>298.008333</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "convo_idx                 0           1\n",
       "speaker                                \n",
       "ACrusaderA        99.640812  131.716667\n",
       "A_Mirror          71.400000   94.583333\n",
       "A_Soporific      287.816667  229.133333\n",
       "AlphaGoGoDancer  297.825000  418.933333\n",
       "Amablue          161.700000  298.008333"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stage_wc_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "convo_idx\n",
       "0    157.374162\n",
       "1    147.380913\n",
       "dtype: float64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stage_wc_df.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_lifestage_comparisons(stage_df):\n",
    "    for i in range(stage_df.columns.max()):\n",
    "        \n",
    "        mask = stage_df[i+1].notnull() & stage_df[i].notnull()\n",
    "        c1 = stage_df[i+1][mask]\n",
    "        c0 = stage_df[i][mask]\n",
    "        \n",
    "        print('stages %d vs %d (%d speakers)' % (i + 1, i, sum(mask)))\n",
    "        n_more = sum(c1 > c0)\n",
    "        n = sum(c1 != c0)\n",
    "        print('\\tprop more: %.3f, binom_p=%.2f' % (n_more/n, stats.binom_test(n_more,n)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stages 1 vs 0 (100 speakers)\n",
      "\tprop more: 0.410, binom_p=0.09\n"
     ]
    }
   ],
   "source": [
    "print_lifestage_comparisons(stage_wc_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "stage_convo_len_df = get_lifestage_attributes(subset, 'n_utterances', 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "convo_idx\n",
       "0    2.786\n",
       "1    2.733\n",
       "dtype: float64"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stage_convo_len_df.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just looking at the means, it looks like there's a slight decrease in wordcount across the population from the first to the second lifestage. To check significance, we can compute that % of speakers who experience this decrease, and see if it's significant per a binomial test against a null proportion of 50% of speakers (ie., people randomly increase or decrease)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that this is (almost) significant ... maybe more data would help!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "stages 1 vs 0 (100 speakers)\n",
      "\tprop more: 0.458, binom_p=0.48\n"
     ]
    }
   ],
   "source": [
    "print_lifestage_comparisons(stage_convo_len_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll compute some attributes related to linguistic diversity described in the following paper : http://www.cs.cornell.edu/~cristian/Finding_your_voice__linguistic_development.html \n",
    "\n",
    "In short, for each life-stage, we compare the words used by one speaker in one conversation to the words they use in their other conversations, or the words that others use. As such, this is a speaker,convo-level attribute. Given our small sample here (and the fact that CMV and crisis counseling conversations are very different), we're not going for any scientific claims, but use the following function calls to demostrate how the pipeline would work.\n",
    "\n",
    "These attributes are all computed through the `SpeakerConvoDiversityWrapper` transformer, which computes three attributes:\n",
    "\n",
    "* `div__self`: within-diversity in the paper, comparing language use across a speaker's own conversations\n",
    "* `div__other`: between-diversity in the paper, comparing language use across different speakers\n",
    "* `div__adj`: relative diversity: between - within. (intuitively, is the diversity coming from speakers being different from others, beyond being diverse in their own right?)\n",
    "\n",
    "Under the surface, `SpeakerConvoDiversityWrapper` calls a more general `SpeakerConvoDiversity` transformer, which allows for computation of how divergent a conversation is from any arbitrary reference set of conversations, beyond life-stages (see the documentation for details)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "from convokit import SpeakerConvoDiversityWrapper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "scd = convokit.SpeakerConvoDiversityWrapper(lifestage_size=10, max_exp=20,\n",
    "                sample_size=300, min_n_utterances=1, n_iters=50, cohort_delta=60*60*24*30*2, verbosity=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(this takes a while to run, especially with more speakers involved)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "getting lifestages\n",
      "getting within diversity\n",
      "joining tokens across conversation utterances\n",
      "100 / 708\n",
      "200 / 708\n",
      "300 / 708\n",
      "400 / 708\n",
      "500 / 708\n",
      "600 / 708\n",
      "700 / 708\n",
      "getting across diversity\n",
      "joining tokens across conversation utterances\n",
      "100 / 708\n",
      "200 / 708\n",
      "300 / 708\n",
      "400 / 708\n",
      "500 / 708\n",
      "600 / 708\n",
      "700 / 708\n",
      "getting relative diversity\n",
      "100 / 696\n",
      "200 / 696\n",
      "300 / 696\n",
      "400 / 696\n",
      "500 / 696\n",
      "600 / 696\n"
     ]
    }
   ],
   "source": [
    "subset_corpus = scd.transform(subset_corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "div_df = subset_corpus.get_full_attribute_table(['div__self','div__other','div__adj', 'tokens', 'n_utterances'], ['n_convos'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "note that one present limitation of this methodology is that it requires a speaker's activity in a conversation---and in their other conversations---to be substantive enough. if a speaker doesn't meet the minimum wordcount per conversation, then the function returns `np.nan` for that particular speaker,conversation. Filtering out these null values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "div_df = div_df[div_df.div__self.notnull() | div_df.div__other.notnull()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(696, 9)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "div_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "as with the wordcount example, we can make cross-lifestage comparisons. here we unfortunately see no significant population-wide change in either direction. This might be worth exploring with more speakers, though note that interpreting this result for CMV versus for counseling conversations where speakers are randomly assigned might be different. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "div__self\n",
      "stages 1 vs 0 (40 speakers)\n",
      "\tprop more: 0.500, binom_p=1.00\n",
      "\n",
      "\n",
      "===\n",
      "div__other\n",
      "stages 1 vs 0 (83 speakers)\n",
      "\tprop more: 0.530, binom_p=0.66\n",
      "\n",
      "\n",
      "===\n",
      "div__adj\n",
      "stages 1 vs 0 (36 speakers)\n",
      "\tprop more: 0.417, binom_p=0.41\n",
      "\n",
      "\n",
      "===\n"
     ]
    }
   ],
   "source": [
    "for attr in ['div__self','div__other','div__adj']:\n",
    "    print(attr)\n",
    "    stage_df = get_lifestage_attributes(div_df, attr, 10)\n",
    "    print_lifestage_comparisons(stage_df)\n",
    "    print('\\n\\n===')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}