topic-modeling-101/topic_modeling_part1.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:887cbfce4d5ff4f963164c74b13c9cd0f731910a4cb99e031146ed0cb8461d6c"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "\n",
      "#Definitions:\n",
      "* 1.) document - a body of text (eg. tweet)\n",
      "* 2.) text corpus - the set of documents that contains the text for the analysis (eg. many tweets)\n",
      "* 3.) dictionary - a mapping between tokens and their integer ids. In other words, the key:value pairs are token:unique_id for each unique token in the text corpus (eg. {'mathematics':1,'engineering':1,'physics':3})\n",
      "* 4.) vector corpus - the set of documents transformed such that each token is a tuple (token_id , doc_freq).\n",
      "\n",
      "\n",
      "#Stop words:\n",
      "**Topics are limited to tokens contained within the text corpus**. We can remove specific tokens from consideration using a set of stopwords, which can be edited per project requirement. \n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nltk.corpus import stopwords\n",
      "import pprint as pp\n",
      "\n",
      "stopset = set(stopwords.words('english'))\n",
      "print type(stopset)\n",
      "stopset.update([\"ruby tuesday\"]) # add token\n",
      "stopset.remove(\"own\")            # remove token\n",
      "\n",
      "# single lang\n",
      "print \"--English stopset\"\n",
      "print stopset\n",
      "\n",
      "# multi lang\n",
      "print\n",
      "print \"--Multi language stopset\"\n",
      "langs=['danish', 'dutch', 'english', 'french', 'german', 'italian','norwegian', 'portuguese', 'russian', 'spanish', 'swedish']\n",
      "stop_list = []\n",
      "for lang in langs:\n",
      "    stop_list.extend(stopwords.words(lang))\n",
      "\n",
      "stop_words_set=set(stop_list) #  -- could save to disk --\n",
      "print stop_words_set\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Text Corpus:\n",
      "The text corpus used for the demo is contained in a file with lines of text separated by carriage returns. Each line of text is it's own document. We will use the entire text corpus as our training set to build the dictionary and then remove stopwords; however, the dictionary need not be built from the entire text corpus if a smaller set of documents is sufficient. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "with open('text_corpus.txt', 'r') as f:\n",
      "    documents=[]\n",
      "    for line in f.readlines():\n",
      "        documents.append(line.strip())\n",
      "pp.pprint(documents)    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Dictionary:\n",
      "Next, we'll create a dictionary from the tokens in the entire text corpus. We're splitting the documents on white space for this demo; however, we'll use regex in later. We'll then remove stopwords and tokens that only appear once in the entire text corpus."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from gensim import corpora, models, similarities\n",
      "import logging\n",
      "logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO, filename=\"./log/topic-log\")\n",
      "logr = logging.getLogger(\"topic_model\")\n",
      "logr.info(\"#\"*15 + \" started \" + \"#\"*15)\n",
      "\n",
      "print \"Dictionary (full text corpus):\"\n",
      "dictionary = corpora.Dictionary(line.lower().split() for line in open('text_corpus.txt'))\n",
      "print dictionary\n",
      "print (dictionary.token2id)\n",
      "\n",
      "print\n",
      "\n",
      "print \"Dictionary (removed stopwords and once-ids):\"\n",
      "stop_ids = [dictionary.token2id[stopword] for stopword in stop_words_set if stopword in dictionary.token2id]\n",
      "\n",
      "once_ids = [tokenid for tokenid, corpus_freq in dictionary.dfs.iteritems() if corpus_freq == 1]\n",
      "#remove stop_ids,\"+\",once_ids\n",
      "dictionary.filter_tokens(bad_ids=stop_ids + once_ids,good_ids=None)\n",
      "## consider: dictionary.filter_extremes(no_below=2) \n",
      "dictionary.compactify()\n",
      "print dictionary\n",
      "print (dictionary.token2id)\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "We can also **add documents dynamically**, which is a huge advantage for real time data! Notice how the dictionary starts with 12 unique tokens (above) and ends with 25 tokens (below). Also note that we must add a list of lists such that docs=[[doc1],[doc2]...] where doc1 and doc2 are tokenized strings. "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import copy\n",
      "print \"Add documents to dictionary dynamically:\"\n",
      "print \"doc to add = \\\"Pooh bear says, 'People say nothing is impossible, but I do nothing every day.'\\\"\"\n",
      "print\n",
      "print \"doc tokenized =\",[item for item in \"Pooh bear says 'People say nothing is impossible, but I do nothing every day.'\".lower().split() if item not in stop_ids]\n",
      "print\n",
      "\n",
      "docs=[[item for item in \"Pooh bear says, 'People say nothing is impossible, but I do nothing every day.'\".lower().split() if item not in stop_ids]]\n",
      "d=copy.deepcopy(dictionary)\n",
      "d.add_documents(docs)\n",
      "\n",
      "d.compactify()\n",
      "\n",
      "print \"#NOTE: since we were only splitting on space, the punctuation is included.\"\n",
      "print\n",
      "print d\n",
      "print d.token2id\n",
      "\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Vectorize:\n",
      "Essentially a word frequency for each document is created in this step. Each document in the text corpus will be transformed into list of tuples \n",
      "[[(token_id , doc_freq),(token_id , doc_freq),(token_id , doc_freq)] , [(token_id , doc_freq),(token_id , doc_freq)]...]. We must iterate through the text corpus to create this set.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "\n",
      "vector_corpus=[]\n",
      "with open('text_corpus.txt', 'r') as f:\n",
      "    for line in f.readlines():\n",
      "        vector_corpus.append(dictionary.doc2bow(line.lower().split()))\n",
      "\n",
      "print \"Vector corpus:\"\n",
      "pp.pprint(vector_corpus)\n",
      "counter=0\n",
      "print dictionary\n",
      "\n",
      "# save to disk\n",
      "corpora.MmCorpus.serialize('vector_corpus.mm', vector_corpus)\n",
      "serialized_corpus = corpora.MmCorpus('vector_corpus.mm')\n",
      "pp.pprint(list(serialized_corpus))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "# [TfIdf](http://radimrehurek.com/gensim/models/tfidfmodel.html):\n",
      "The TfIdf, term frequency inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. The TfIdf value increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to control for the fact that some words are generally more common than others. \n",
      "\n",
      "To train a model using TfIdf, you first need to go through the corpus once and copute doc frequencies, which we already did above.  \n",
      "\n",
      "Typically, the TfIdf weight is composed by two terms: the first computes the normalized term frequency, which is the number of times a word appears in a document, divided by the total number of words in that document; the second term is the inverse document frequency (idf), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.\n",
      "\n",
      "$$TfIdf = {token\\ frequency\\ in\\ doc} * \\ln(\\frac{total\\ docs\\ in\\ corpus}{total\\ docs\\ w/\\ token})$$\n",
      "\n",
      "Tf: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization.\n",
      "\n",
      "\n",
      "Idf: Inverse Document Frequency, which measures how important a term is. While computing Tf, all terms are considered equally important. However it is known that certain terms, such as \"is\", \"of\", and \"that\", may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following: \n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from gensim import corpora, models, similarities\n",
      "tfidf = models.TfidfModel(vector_corpus, normalize=False) # trains the model \n",
      "print tfidf\n",
      "corpus_tfidf=tfidf[vector_corpus]\n",
      "print (dictionary.token2id)\n",
      "for doc in corpus_tfidf:\n",
      "    print doc\n",
      "#   tfidf = (<normalized> term frequency)                                     * (inverse document frequency) \n",
      "\n",
      "#   tfidf = (# of instances of word in single doc / # of words in single doc) * ln(# of total documents / # of docs in which word appears) = tfidf\n",
      "\n",
      "#   the tfidf matrix can be used to convert any vector ( uniq id, count per doc ) to ( uniq id, tfidf score )\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Latent Dirichlet Allocation vs Latent Semantic Indexing ---improve explanation---\n",
      "\n",
      "Inerteresting perspective: [link](http://stats.stackexchange.com/questions/32310/topic-models-and-word-co-occurrence-methods)\n",
      "\n",
      "\"I will just describe four milestones/popular models and their advantages/disadvantages and thus highlight (some of) the main differences (or at least what I think are the main/most important differences).\n",
      "\n",
      "The \"easiest\" approach, which would be to cluster the documents by matching them against a predefined query of terms (as in PMI). These lexical matching methods however might be inaccurate due to polysemy (multiple meanings) and synonymy (multiple words that have similar meanings) of single terms.\n",
      "\n",
      "As a remedy, latent semantic indexing (LSI) tries to overcome this by mapping terms and documents into a latent semantic space via a singular value decomposition. The LSI results are more robust indicators of meaning than individual terms would be. However, one drawback of LSI is that it lacks in terms of solid probabilistic foundation.\n",
      "\n",
      "This was partly solved by the invention of probabilistic LSI (pLSI). In pLSI models each word in a document is drawn from a mixture model specified via multinomial random variables (which also allows higher-order co-occurences as @sviatoslav hong mentioned). This was an important step forward in probabilistic text modeling, but was incomplete in the sense that it offers no probabilistic structure at the level of documents.\n",
      "\n",
      "Latent Dirichlet Allocation (LDA) alleviates this and was the first fully probabilistic model for text clustering. Blei et al. (2003) show that pLSI is a maximum a-posteriori estimated LDA model under a uniform Dirichlet prior.\n",
      "\n",
      "Note that the models mentioned above (LSI, pLSI, LDA) have in common that they are based on the \u201cbag-of-words\u201d assumption - i.e. that within a document, words are exchangeable, i.e. the order of words in a document can be neglected. This assumption of exchangeability offers a further justification for LDA over the other approaches: Assuming that not only words within documents are exchangeable, but also documents, i.e., the order of documents within a corpus can be neglected, De Finetti's theorem states that any set of exchangeable random variables has a representation as a mixture distribution. Thus if exchangeability for documents and words within documents is assumed, a mixture model for both is needed. Exactly this is what LDA generally achieves but PMI or LSI do not (and even pLSI not as beautiful as LDA).\"\n",
      "\n",
      "\\----\n",
      "\n",
      "The LSI process transforms documents from TfIdf-weighted space into a latent space of a lower dimensionality.\n",
      "\n",
      "LDA can be explained using plate notation. The boxes are \u201cplates\u201d representing replicates. The outer plate represents documents, while the inner plate represents the repeated choice of topics and words within a document. \n",
      "* M denotes the number of documents\n",
      "* N the number of words in a document\n",
      "* \u03b1 is the parameter of the Dirichlet prior on the per-document topic distributions\n",
      "* \u03b2 is the parameter of the Dirichlet prior on the per-topic word distribution\n",
      "* $\\theta_i$ is the topic distribution for document i\n",
      "* $\\phi_k$is the word distribution for topic k\n",
      "* z_{ij} is the topic for the jth word in document i\n",
      "* w_{ij} is the specific word.\n",
      "<img src=\"files/image.png\">\n",
      "\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from itertools import *\n",
      "number_of_clusters=3\n",
      "lsi = models.LsiModel(corpus_tfidf, id2word=dictionary, num_topics=number_of_clusters) # initialize an LSI transformation\n",
      "lda = models.ldamodel.LdaModel(corpus_tfidf, id2word=dictionary, num_topics=number_of_clusters,\\\n",
      "                               update_every=1, chunksize=10000, passes=1)\n",
      "corpus_lsi = lsi[corpus_tfidf] # create a double wrapper over the original corpus: bow->tfidf->fold-in-lsi\n",
      "corpus_lda = lda[corpus_tfidf] \n",
      "# for item in corpus_lsi:\n",
      "#    print (item)\n",
      "print \"-\"*10+\"LDA\"+\"-\"*10\n",
      "t=0\n",
      "for t, item in enumerate(lda.print_topics(number_of_clusters)):\n",
      "    print \"topic#{0}: {1}\".format(t,item)\n",
      "print\n",
      "for item in corpus_lda:\n",
      "    print item\n",
      "    #print lsi.show_topics()\n",
      "#print lsi.print_topic(0,topn=1)\n",
      "# save to disk\n",
      "#print lsi.projection.s\n",
      "#lsi.save('corpus_lsi.lsi')\n",
      "#lsi=models.LsiModel.load\n",
      "print \n",
      "print \n",
      "#models.lsimodel.clip_spectrum(0.1,4,discard=0.001)\n",
      "\n",
      "# Find the threshold, let's set the threshold to be 1/#clusters,\n",
      "# To prove that the threshold is sane, we average the sum of all probabilities:\n",
      "scores = list(chain(*[[score for topic,score in topic] \\\n",
      "                      for topic in [doc for doc in corpus_lda]]))\n",
      "threshold = sum(scores)/len(scores)\n",
      "print \"threshold:\",threshold\n",
      "print\n",
      "cluster1 = [j for i,j in zip(corpus_lda,documents) if i[0][1] > threshold]\n",
      "cluster2 = [j for i,j in zip(corpus_lda,documents) if i[1][1] > threshold]\n",
      "cluster3 = [j for i,j in zip(corpus_lda,documents) if i[2][1] > threshold]\n",
      "\n",
      "print \"topic#0: {0}\".format(cluster1)\n",
      "print \"topic#1: {0}\".format(cluster2)\n",
      "print \"topic#2: {0}\".format(cluster3)\n",
      "\n",
      "print \n",
      "print\n",
      "print \"-\"*10+\"LSI\"+\"-\"*10\n",
      "t=0\n",
      "for t, item in enumerate(lsi.print_topics(number_of_clusters)):\n",
      "    print \"topic#{0}: {1}\".format(t,item)\n",
      "print\n",
      "\n",
      "for item in corpus_lsi:\n",
      "    print item\n",
      "    #print lsi.show_topics()\n",
      "#print lsi.print_topic(0,topn=1)\n",
      "# save to disk\n",
      "#print lsi.projection.s\n",
      "#lsi.save('corpus_lsi.lsi')\n",
      "#lsi=models.LsiModel.load\n",
      "print \n",
      "print \n",
      "#models.lsimodel.clip_spectrum(0.1,4,discard=0.001)\n",
      "\n",
      "# Find the threshold, let's set the threshold to be 1/#clusters,\n",
      "# To prove that the threshold is sane, we average the sum of all probabilities:\n",
      "scores = list(chain(*[[score for topic,score in topic] \\\n",
      "                      for topic in [doc for doc in corpus_lsi]]))\n",
      "threshold = sum(scores)/len(scores)\n",
      "print \"threshold:\",threshold\n",
      "print\n",
      "\n",
      "cluster1 = [j for i,j in zip(corpus_lsi,documents) if i[0][1] > threshold]\n",
      "cluster2 = [j for i,j in zip(corpus_lsi,documents) if i[1][1] > threshold]\n",
      "cluster3 = [j for i,j in zip(corpus_lsi,documents) if i[2][1] > threshold]\n",
      "\n",
      "print \"topic#1: {0}\".format(cluster1)\n",
      "print \"topic#2: {0}\".format(cluster2)\n",
      "print \"topic#3: {0}\".format(cluster3)\n",
      "\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\n",
      "\n",
      "#play space\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#LDA\n",
      "<img src=\"files/table.png\">"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Inerteresting perspective: [link](http://stackoverflow.com/questions/10628262/inserting-image-into-ipython-notebook-markdown)\n",
      "\n",
      "Recently, a huge body of literature discussing how to extract information from written text has grown. Hence I will just describe four milestones/popular models and their advantages/disadvantages and thus highlight (some of) the main differences (or at least what I think are the main/most important differences).\n",
      "\n",
      "You mention the \"easiest\" approach, which would be to cluster the documents by matching them against a predefined query of terms (as in PMI). These lexical matching methods however might be inaccurate due to polysemy (multiple meanings) and synonymy (multiple words that have similar meanings) of single terms.\n",
      "\n",
      "As a remedy, latent semantic indexing (LSI) tries to overcome this by mapping terms and documents into a latent semantic space via a singular value decomposition. The LSI results are more robust indicators of meaning than individual terms would be. However, one drawback of LSI is that it lacks in terms of solid probabilistic foundation.\n",
      "\n",
      "This was partly solved by the invention of probabilistic LSI (pLSI). In pLSI models each word in a document is drawn from a mixture model specified via multinomial random variables (which also allows higher-order co-occurences as @sviatoslav hong mentioned). This was an important step forward in probabilistic text modeling, but was incomplete in the sense that it offers no probabilistic structure at the level of documents.\n",
      "\n",
      "Latent Dirichlet Allocation (LDA) alleviates this and was the first fully probabilistic model for text clustering. Blei et al. (2003) show that pLSI is a maximum a-posteriori estimated LDA model under a uniform Dirichlet prior.\n",
      "\n",
      "Note that the models mentioned above (LSI, pLSI, LDA) have in common that they are based on the \u201cbag-of-words\u201d assumption - i.e. that within a document, words are exchangeable, i.e. the order of words in a document can be neglected. This assumption of exchangeability offers a further justification for LDA over the other approaches: Assuming that not only words within documents are exchangeable, but also documents, i.e., the order of documents within a corpus can be neglected, De Finetti's theorem states that any set of exchangeable random variables has a representation as a mixture distribution. Thus if exchangeability for documents and words within documents is assumed, a mixture model for both is needed. Exactly this is what LDA generally achieves but PMI or LSI do not (and even pLSI not as beautiful as LDA)."
     ]
    }
   ],
   "metadata": {}
  }
 ]
}