Socrata API Open Data Portal Analysis - Final Version.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#standard imports\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "#API tools\n",
    "import requests\n",
    "import json\n",
    "from pandas.io.json import json_normalize\n",
    "from urllib.request import Request, urlopen\n",
    "import json\n",
    "\n",
    "#visuals\n",
    "import matplotlib as plt\n",
    "import seaborn as sns\n",
    "\n",
    "#Natural Language Processing\n",
    "import nltk\n",
    "import lda #Latent Dirichlet Allocation (create topics)\n",
    "import gensim\n",
    "from gensim import corpora, models #for constructing document term matrix\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "from nltk import stem\n",
    "from nltk.corpus import stopwords\n",
    "\n",
    "#magic\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.set_option('display.float_format', lambda x: '%.3f' % x) #otherwise we have scientific notation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Our Question: Among state and city government open datasets hosted on the Socrata portal, what dataset topics are the most popular as of August 2017?\n",
    "\n",
    "## However, this (unfortunately) isn't as easy as choosing a consistent category like \"crime and public safety\" and adding up views and downloads. We have two obstacles:\n",
    "- We need to control for the fact that some datasets just get more views because they are from popular portals. Our goal is determine what *types* of datasets are most popular so all cities and states can use this information. We don't want our conclusion to be that everyone should release data on \"New York Lottery\" (this is not a joke; our initial analysis found that to be a very popular tag).\n",
    "- Datasets themselves vary widely, and not all metadata is standardized or consistent. Some datasets are in broad categories, like annual budget, and some are very niche and specific, like dog licenses. Compounding this challenge is that cities choose all sorts of different tags and categories for similar data; \n",
    "\n",
    "### Therefore, we need a way to extract true topics that cut across multiple cities' and states' datasets.\n",
    "- We want \"budget\", \"fiscal year\" and \"expenditures\" to all be treated as a single topic.\n",
    "- **Fortunately, a machine learning model exists to do just this.** *Latent Dirichlet Allocation* is an algorithm that finds \"hidden\" topics in a group of documents. It posits that each \"document\" (in this case, our combined metadata) is a mixture of topics, and that each word in a document is derived from a topic. LDA will produce \"topics\" composed of words and their expected distribution across that topic. While expected distribution is not very intuitive to interpret, the word combinations in topics is extremely easy to interpret.\n",
    "    - **In order to run LDA on this dataset, we processed our metadata by combining \"description\", \"domain_tags\" and \"categories\" attributes into a new attribute called \"mash\".** Each row of \"mash\" will be a document in our model.\n",
    "        - We excluded dataset name and domain; these are often very specific, parochial tags that will inhibit our efforts to form \"topics\" that can apply across cities and datasets. As we will see later, proper and parochial names skew our model and we want to avoid them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in a saved dataframe created from Socrata API call on 8.14.17\n",
    "- Unpacked JSON dicts returned by API call.\n",
    "- Contains data on nearly 30,000 dataset's metadata, including description, topic tags, categories, views, and downloads.\n",
    "- **This metadata is not in a standard format**. City and state governments write their own descriptions and choose their own tags. One government may tag their annual budget as \"budget\", while another may choose \"fiscal year\", and yet another just \"government\".\n",
    "- **This metadata is often missing one or more attributes**. Some governments don't include a description, or don't include tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "big_mash = pd.read_csv('big_mash_archive_8_14.14')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>Homelessness PIT Transitional Age Youth</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['location_on_the_night_of_the_count_total_per...</td>\n",
       "      <td>['Location on the night of the count, Total Pe...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['point in time', 'homelessness', 'ss', 'dchs']</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>46.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>319.000</td>\n",
       "      <td>8.322</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['point', 'in', 'time', 'homelessness', 'ss', ...</td>\n",
       "      <td>['point', 'in', 'time', 'homelessness', 'ss', ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>Fair Housing Complaints</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['violations', 'percent_found_to_be_compliant'...</td>\n",
       "      <td>['Number of complaints', 'Percent of sites fou...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['housing']</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>31.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>278.000</td>\n",
       "      <td>8.124</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['housing']</td>\n",
       "      <td>['housing']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>Parking Complaints Bar Chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['percent_of_valid_parking_meter_problem_servi...</td>\n",
       "      <td>['Percent of valid parking meter problem servi...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['transportation']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>274.000</td>\n",
       "      <td>8.103</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['transportation']</td>\n",
       "      <td>['transportation']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>NVMHI Admissions</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['lipos_admissions_per_100k', 'nvmhi_admission...</td>\n",
       "      <td>['LIPOS Admissions per 100K', 'NVMHI Admission...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['delete', 'dchs']</td>\n",
       "      <td>official</td>\n",
       "      <td>5.000</td>\n",
       "      <td>38.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>263.000</td>\n",
       "      <td>8.044</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['delete', 'dchs']</td>\n",
       "      <td>['delete', 'dchs']</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>Property Owners Trainined</td>\n",
       "      <td>['Office', 'of', 'Housing', 'Data']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['number_of_property_owners_trained', 'percent...</td>\n",
       "      <td>['Number of property owners trained', 'Percent...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['housing']</td>\n",
       "      <td>official</td>\n",
       "      <td>11.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>246.000</td>\n",
       "      <td>7.948</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['housing']</td>\n",
       "      <td>['housing', 'Office', 'of', 'Housing', 'Data']</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                     name  \\\n",
       "0           0  Homelessness PIT Transitional Age Youth   \n",
       "1           1                  Fair Housing Complaints   \n",
       "2           2             Parking Complaints Bar Chart   \n",
       "3           3                         NVMHI Admissions   \n",
       "4           4                Property Owners Trainined   \n",
       "\n",
       "                           description attribution  \\\n",
       "0                                   []         NaN   \n",
       "1                                   []         NaN   \n",
       "2                                   []         NaN   \n",
       "3                                   []         NaN   \n",
       "4  ['Office', 'of', 'Housing', 'Data']         NaN   \n",
       "\n",
       "                                  columns_field_name  \\\n",
       "0  ['location_on_the_night_of_the_count_total_per...   \n",
       "1  ['violations', 'percent_found_to_be_compliant'...   \n",
       "2  ['percent_of_valid_parking_meter_problem_servi...   \n",
       "3  ['lipos_admissions_per_100k', 'nvmhi_admission...   \n",
       "4  ['number_of_property_owners_trained', 'percent...   \n",
       "\n",
       "                                        columns_name   type  \\\n",
       "0  ['Location on the night of the count, Total Pe...  chart   \n",
       "1  ['Number of complaints', 'Percent of sites fou...  chart   \n",
       "2  ['Percent of valid parking meter problem servi...  chart   \n",
       "3  ['LIPOS Admissions per 100K', 'NVMHI Admission...  chart   \n",
       "4  ['Number of property owners trained', 'Percent...  chart   \n",
       "\n",
       "           categories domain_category  \\\n",
       "0                  []             NaN   \n",
       "1                  []             NaN   \n",
       "2  ['transportation']             NaN   \n",
       "3                  []             NaN   \n",
       "4                  []             NaN   \n",
       "\n",
       "                                       domain_tags provenance  download_count  \\\n",
       "0  ['point in time', 'homelessness', 'ss', 'dchs']   official           4.000   \n",
       "1                                      ['housing']   official          21.000   \n",
       "2                                               []   official          17.000   \n",
       "3                               ['delete', 'dchs']   official           5.000   \n",
       "4                                      ['housing']   official          11.000   \n",
       "\n",
       "   page_views_last_month  page_views_last_week  page_views_total  \\\n",
       "0                 46.000                 2.000           319.000   \n",
       "1                 31.000                 0.000           278.000   \n",
       "2                 17.000                 2.000           274.000   \n",
       "3                 38.000                 1.000           263.000   \n",
       "4                 30.000                 0.000           246.000   \n",
       "\n",
       "   page_views_total_log                      domain  \\\n",
       "0                 8.322  dashboard.alexandriava.gov   \n",
       "1                 8.124  dashboard.alexandriava.gov   \n",
       "2                 8.103  dashboard.alexandriava.gov   \n",
       "3                 8.044  dashboard.alexandriava.gov   \n",
       "4                 7.948  dashboard.alexandriava.gov   \n",
       "\n",
       "                                                mash  \\\n",
       "0  ['point', 'in', 'time', 'homelessness', 'ss', ...   \n",
       "1                                        ['housing']   \n",
       "2                                 ['transportation']   \n",
       "3                                 ['delete', 'dchs']   \n",
       "4                                        ['housing']   \n",
       "\n",
       "                                            big_mash  \n",
       "0  ['point', 'in', 'time', 'homelessness', 'ss', ...  \n",
       "1                                        ['housing']  \n",
       "2                                 ['transportation']  \n",
       "3                                 ['delete', 'dchs']  \n",
       "4     ['housing', 'Office', 'of', 'Housing', 'Data']  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21807"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(big_mash.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building an LDA Model:\n",
    "\n",
    "- Running an LDA model is very computationally intensive as explained below; this is why we did it in a separate notebook. Fortunately, we can save and import models.\n",
    "- When building an LDA model, we pick the number of topics that we want the algorithm to find. This is a very important parameter.\n",
    "- LDA Model fitting is an iterative process. The algorithm starts out by assigning every word to a temporary topic. Then, for *every* word, it updates the topics by calculating:\n",
    "    - How prevalent is that word across topics? Topics with a high prevalance of the word in question get a higher weight for that word's assignment.\n",
    "     - How prevalent are topics within a document? If one topic within a document is more prevalent, it gets a higher weight.\n",
    "     - Based on these two criteria, LDA then updates a word's topic and document assignment.\n",
    "     - **An LDA's topic formations get better the more passes you can run. However, running 60 passes of this dataset's model (we do have a large number of documents, although their word count is relatively low) takes about 35 minutes on a MacBook Air with 8GB of RAM.** \n",
    "\n",
    "### If we understand nothing else, we need to understand this: LDA is a *probabilistic* model and its results can fluctuate based on small changes in the data and how the model itself randomly assigns topics to begin with!\n",
    "- This is a weakness in our analysis; it is not very replicable. It took several different LDA formations to settle on an ideal number of topics and to get consistently coherent topics.\n",
    "- Our source data changes based on our API call. Our LDA model changes based upon both the source data and the model's probabilistic results itself. **Thus, for stability and replicability, we've saved both a dataframe and an LDA model. These are snapshots in time, not current, definitive structures.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in saved LDA Model\n",
    "- Constructed from 52 topics; 60 passes on the tokens from the \"big_mash\" column of our dataframe:\n",
    "- The source code for this is in the ipynb \"Original LDA Code - Socrata API Dataset - Gensim LDA Groupings\". **Note that if you run this code yourself (encouraged), you WILL get different results, and perhaps not useful ones**. Finding the right LDA settings for your dataset is time consuming and basically takes trial and error. Again, this is a weakness of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "lda_52_sixty = gensim.models.ldamodel.LdaModel.load('lda_52_sixty_good_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0,\n",
       "  [('prevention', 0.039228265770020823),\n",
       "   ('hospital', 0.038899194905431915),\n",
       "   ('ny', 0.034882850346069173),\n",
       "   ('healthy', 0.032344049932466652),\n",
       "   ('statewide', 0.029269391777562728),\n",
       "   ('hospitals', 0.022294582283033028),\n",
       "   ('inpatient', 0.021854135158083089),\n",
       "   ('healthcare', 0.018663035226976647),\n",
       "   ('api', 0.018299534915519441),\n",
       "   ('quality-safety-costs', 0.016526700960621438)]),\n",
       " (1,\n",
       "  [('chart', 0.069963659585634108),\n",
       "   ('children', 0.059274004768666677),\n",
       "   ('income', 0.03908399441304488),\n",
       "   ('historic', 0.025093228857131812),\n",
       "   ('home', 0.021852521187430106),\n",
       "   ('assistance', 0.018091177706023875),\n",
       "   ('homes', 0.017940795661987072),\n",
       "   ('families', 0.017918354840225928),\n",
       "   ('low', 0.017857714487521079),\n",
       "   ('pay', 0.017610064270261648)]),\n",
       " (2,\n",
       "  [('population', 0.13151520321707977),\n",
       "   ('county', 0.091960434049722567),\n",
       "   ('demographics', 0.084275005086619365),\n",
       "   ('age', 0.0434468663431666),\n",
       "   ('total', 0.028221558698175012),\n",
       "   ('king', 0.019468810925607257),\n",
       "   ('washington', 0.01927613699149228),\n",
       "   ('montgomery', 0.018261080011821789),\n",
       "   ('number', 0.017900571442482898),\n",
       "   ('years', 0.016960472490539676)]),\n",
       " (3,\n",
       "  [('transportation', 0.1645420154231319),\n",
       "   ('traffic', 0.038249415092180432),\n",
       "   ('street', 0.029708854670228657),\n",
       "   ('parking', 0.026048447964619643),\n",
       "   ('infrastructure', 0.024499661836046696),\n",
       "   ('city', 0.024248488006745492),\n",
       "   ('safe', 0.023908406994518551),\n",
       "   ('vehicle', 0.023135110066475582),\n",
       "   ('streets', 0.021985229238776469),\n",
       "   ('bike', 0.01551025776156169)]),\n",
       " (4,\n",
       "  [('information', 0.045629510720610396),\n",
       "   ('tab', 0.02519267217634192),\n",
       "   ('projects', 0.024289165547602756),\n",
       "   ('concerning', 0.018274567627167695),\n",
       "   ('may', 0.017740368430513121),\n",
       "   ('contains', 0.017113371818170686),\n",
       "   ('project', 0.015854281382776003),\n",
       "   ('provided', 0.013839518701256619),\n",
       "   ('additional', 0.013692382248742174),\n",
       "   ('improvement', 0.012452002727572102)]),\n",
       " (5,\n",
       "  [('locations', 0.069775812645793758),\n",
       "   ('bacteria', 0.02648594877681025),\n",
       "   ('hours', 0.025954898196716671),\n",
       "   ('culture', 0.024828275368617661),\n",
       "   ('county', 0.023656957609001595),\n",
       "   ('levels', 0.02197024161214425),\n",
       "   ('contact', 0.02095100106449152),\n",
       "   ('directory', 0.019849859557347929),\n",
       "   ('e', 0.019300970726415546),\n",
       "   ('contains', 0.018603307582784098)]),\n",
       " (6,\n",
       "  [('property', 0.13136570965022701),\n",
       "   ('consumption', 0.051884459340934608),\n",
       "   ('real', 0.046286074447914313),\n",
       "   ('value', 0.03077786423134931),\n",
       "   ('properties', 0.025283903585934197),\n",
       "   ('significant', 0.021787551505509636),\n",
       "   ('easy', 0.021674540711116137),\n",
       "   ('usage', 0.019934752091226977),\n",
       "   ('assessment', 0.019827225768331354),\n",
       "   ('company', 0.015464303111405545)]),\n",
       " (7,\n",
       "  [('rates', 0.097364682402046362),\n",
       "   ('calculated', 0.069242760650261914),\n",
       "   ('rate', 0.052423706953771118),\n",
       "   ('year', 0.046226566488417858),\n",
       "   ('counts', 0.041696995181128169),\n",
       "   ('average', 0.026100750207484962),\n",
       "   ('seattle', 0.018955865652478202),\n",
       "   ('percent', 0.017770826386179429),\n",
       "   ('mortality', 0.016407853651273428),\n",
       "   ('number', 0.015107387053477006)]),\n",
       " (8,\n",
       "  [('health', 0.094596970812741571),\n",
       "   ('disease', 0.043755686476799373),\n",
       "   ('000', 0.039024957736277545),\n",
       "   ('per', 0.030917812190623686),\n",
       "   ('rate', 0.029034554086777568),\n",
       "   ('rates', 0.026226335228672973),\n",
       "   ('chronic', 0.025069794781680706),\n",
       "   ('risk', 0.022384318262765962),\n",
       "   ('agency', 0.02162102669270825),\n",
       "   ('due', 0.0195463066793998)]),\n",
       " (9,\n",
       "  [('recreation', 0.13128622968124695),\n",
       "   ('parks', 0.064528596736303123),\n",
       "   ('jersey', 0.060593984662716933),\n",
       "   ('park', 0.040013809141157139),\n",
       "   ('center', 0.029877716132019151),\n",
       "   ('neighborhoods', 0.0258798720295484),\n",
       "   ('centers', 0.02214072729834908),\n",
       "   ('facilities', 0.019767883044293812),\n",
       "   ('environment', 0.017226759948461606),\n",
       "   ('access', 0.01570620198722774)]),\n",
       " (10,\n",
       "  [('measures', 0.092380842032599053),\n",
       "   ('quality', 0.077820821260262546),\n",
       "   ('performance', 0.069476579222436344),\n",
       "   ('government', 0.049591497417367757),\n",
       "   ('life', 0.025916014373607278),\n",
       "   ('developed', 0.024796650118479639),\n",
       "   ('fair', 0.016476546976723017),\n",
       "   ('measurement', 0.014967900754065268),\n",
       "   ('definitions', 0.013866800035767382),\n",
       "   ('index', 0.013374835113969707)]),\n",
       " (11,\n",
       "  [('energy', 0.12371084945388265),\n",
       "   ('environment', 0.090690020883510858),\n",
       "   ('electricity', 0.027522591132607205),\n",
       "   ('air', 0.02448100005593962),\n",
       "   ('sustainable', 0.024139471140549176),\n",
       "   ('action', 0.023631696929809619),\n",
       "   ('climate', 0.023112523604460807),\n",
       "   ('city', 0.019150364298185181),\n",
       "   ('clean', 0.01909891884790434),\n",
       "   ('facilities', 0.01847151888251522)]),\n",
       " (12,\n",
       "  [('map', 0.07003231159311829),\n",
       "   ('\"about\"', 0.065737327359522146),\n",
       "   ('shows', 0.044643143524590485),\n",
       "   ('details', 0.037417466989238299),\n",
       "   ('tracking', 0.033865619389321588),\n",
       "   ('click', 0.028565691919877855),\n",
       "   ('view', 0.027447393223963716),\n",
       "   ('additional', 0.026872823680713778),\n",
       "   ('weekly', 0.021964112590207557),\n",
       "   ('information', 0.017908829658562275)]),\n",
       " (13,\n",
       "  [('gas', 0.059972589629464695),\n",
       "   ('natural', 0.038247506812764911),\n",
       "   ('food', 0.035123659997646883),\n",
       "   ('greenhouse', 0.02737167049995624),\n",
       "   ('environment', 0.026272618464459947),\n",
       "   ('emissions', 0.023135082043847805),\n",
       "   ('costs', 0.021398374769158977),\n",
       "   ('resources', 0.020865670871248309),\n",
       "   ('program', 0.020842524225516069),\n",
       "   ('fuel', 0.016081016497170121)]),\n",
       " (14,\n",
       "  [('health', 0.047486595563458037),\n",
       "   ('code', 0.041436774555512541),\n",
       "   ('statistics', 0.038542408744321223),\n",
       "   ('county', 0.035207719557086382),\n",
       "   ('codes', 0.029854335897479736),\n",
       "   ('zip', 0.029139372691221986),\n",
       "   ('public', 0.024872570253527144),\n",
       "   ('diego', 0.024784995832963057),\n",
       "   ('harbison', 0.023033852102136411),\n",
       "   ('crest', 0.021395606263276593)]),\n",
       " (15,\n",
       "  [('budget', 0.14154768271395413),\n",
       "   ('finance', 0.061629847818770116),\n",
       "   ('fund', 0.052866707925857535),\n",
       "   ('year', 0.027361528382526451),\n",
       "   ('operating', 0.026714548673113327),\n",
       "   ('city', 0.026151540354093086),\n",
       "   ('fiscal', 0.023587756171379842),\n",
       "   ('funds', 0.020153544191078145),\n",
       "   ('calendar', 0.018291264587569866),\n",
       "   ('communities', 0.016813255820410825)]),\n",
       " (16,\n",
       "  [('economy', 0.074831970475957757),\n",
       "   ('employment', 0.059024507694450813),\n",
       "   ('labor', 0.037518750233590671),\n",
       "   ('unemployment', 0.030420042880313702),\n",
       "   ('insurance', 0.022811919476438829),\n",
       "   ('credit', 0.02240291501623564),\n",
       "   ('workforce', 0.021053326053081162),\n",
       "   ('economic', 0.017523311551405966),\n",
       "   ('training', 0.016769979428771047),\n",
       "   ('federal', 0.015805104806649688)]),\n",
       " (17,\n",
       "  [('services', 0.27536646810144061),\n",
       "   ('social', 0.095962651687079945),\n",
       "   ('human', 0.053809687721545367),\n",
       "   ('dfps', 0.052589282054521021),\n",
       "   ('information', 0.03466089006854034),\n",
       "   ('programs', 0.033066788534579276),\n",
       "   ('us', 0.030089170965030963),\n",
       "   ('definition', 0.022733622690420504),\n",
       "   ('visit', 0.02151984124994798),\n",
       "   ('state', 0.020155768734965274)]),\n",
       " (18,\n",
       "  [('column', 0.063427308877555669),\n",
       "   ('update', 0.046675236579513182),\n",
       "   ('homeless', 0.041663950666934658),\n",
       "   ('annually', 0.030072376551628751),\n",
       "   ('frequency', 0.02926894249626533),\n",
       "   ('position', 0.021127522302048646),\n",
       "   ('city', 0.01984153972699073),\n",
       "   ('commercial', 0.01846514098874218),\n",
       "   ('daily', 0.016869053694575468),\n",
       "   ('salaries', 0.01558924337154113)]),\n",
       " (19,\n",
       "  [('death', 0.048126995262336554),\n",
       "   ('maryland', 0.04203981078656735),\n",
       "   ('database', 0.028903753013522989),\n",
       "   ('deaths', 0.027176667718008366),\n",
       "   ('office', 0.027062908105413903),\n",
       "   ('department', 0.024200455450014258),\n",
       "   ('new', 0.020639763071301036),\n",
       "   ('cdc', 0.020246707002184408),\n",
       "   ('metadata', 0.018528581835236948),\n",
       "   ('citizens', 0.017503462164537666)]),\n",
       " (20,\n",
       "  [('expected', 0.032218200719388315),\n",
       "   ('dept', 0.03192755749313185),\n",
       "   ('observed', 0.028518686601885064),\n",
       "   ('every', 0.023146698375314433),\n",
       "   ('300', 0.021255493076380451),\n",
       "   ('pacific', 0.021116246104674861),\n",
       "   ('physical', 0.020257410998302562),\n",
       "   ('performed', 0.018484869357599031),\n",
       "   ('contribution', 0.017489873926872384),\n",
       "   ('std', 0.01619190856680653)]),\n",
       " (21,\n",
       "  [('public', 0.071331431077657451),\n",
       "   ('safety', 0.069027661647336119),\n",
       "   ('fire', 0.047750379014825345),\n",
       "   ('month', 0.020581987820630462),\n",
       "   ('calls', 0.017375649507546796),\n",
       "   ('police', 0.017078124466635133),\n",
       "   ('emergency', 0.016522104916252903),\n",
       "   ('response', 0.015789103751618325),\n",
       "   ('discharge', 0.015068830152377888),\n",
       "   ('incident', 0.014034152768794762)]),\n",
       " (22,\n",
       "  [('arts', 0.030623882738573149),\n",
       "   ('grants', 0.027305070352776701),\n",
       "   ('live', 0.026616435603867498),\n",
       "   ('county', 0.02352575927246791),\n",
       "   ('includes', 0.019948024676078455),\n",
       "   ('library', 0.01873744681009169),\n",
       "   ('clearance', 0.01769201979004003),\n",
       "   ('grant', 0.01744185803724117),\n",
       "   ('inventory', 0.017416447546996693),\n",
       "   ('cps', 0.017216111917570089)]),\n",
       " (23,\n",
       "  [('police', 0.049599652100346163),\n",
       "   ('user', 0.032265367100513642),\n",
       "   ('information', 0.029626460569977066),\n",
       "   ('page', 0.024435878993697473),\n",
       "   ('department', 0.023728883932669205),\n",
       "   ('use', 0.023314363268950755),\n",
       "   ('orleans', 0.01831277092169864),\n",
       "   ('injury', 0.017497753573167622),\n",
       "   ('may', 0.016658073483472296),\n",
       "   ('records', 0.01319973742765217)]),\n",
       " (24,\n",
       "  [('official', 0.061094290910533561),\n",
       "   ('account', 0.04976604753946838),\n",
       "   ('accounts', 0.048301458649605035),\n",
       "   ('(openmichigan@michigan', 0.040820714695857956),\n",
       "   ('special', 0.029163998589317921),\n",
       "   ('nursing', 0.02877038841603027),\n",
       "   ('software', 0.026806546446223656),\n",
       "   ('gis', 0.023974220833955701),\n",
       "   ('use', 0.022528606483685758),\n",
       "   ('consumer', 0.022447395060642104)]),\n",
       " (25,\n",
       "  [('tax', 0.11694941744290623),\n",
       "   ('revenue', 0.058281674929582826),\n",
       "   ('sales', 0.038634383980769106),\n",
       "   ('taxes', 0.034713010510365863),\n",
       "   ('local', 0.033205416149379233),\n",
       "   ('colorado', 0.030687046796886448),\n",
       "   ('government', 0.029706917267673681),\n",
       "   ('state', 0.021996118000518896),\n",
       "   ('department', 0.019154178963866255),\n",
       "   ('sale', 0.015211224135670038)]),\n",
       " (26,\n",
       "  [('public', 0.16838176841104285),\n",
       "   ('safety', 0.1399822838525531),\n",
       "   ('crime', 0.063180867294822307),\n",
       "   ('police', 0.035971649940013783),\n",
       "   ('department', 0.01934244932463609),\n",
       "   ('illinois', 0.015211807554571162),\n",
       "   ('race', 0.013365201285222672),\n",
       "   ('reported', 0.011776909210955196),\n",
       "   ('crimes', 0.011550469075927439),\n",
       "   ('criminal', 0.010510578525453793)]),\n",
       " (27,\n",
       "  [('water', 0.096124708891673349),\n",
       "   ('environment', 0.048101960808142975),\n",
       "   ('protection', 0.020628382724667733),\n",
       "   ('tx', 0.020203690273746876),\n",
       "   ('filed', 0.019569864371137932),\n",
       "   ('waste', 0.019271624659872911),\n",
       "   ('site', 0.019200525696858466),\n",
       "   ('quality', 0.018557602041230845),\n",
       "   ('environmental', 0.017395095439341653),\n",
       "   ('monitoring', 0.015888456167874085)]),\n",
       " (28,\n",
       "  [('transportation', 0.049342201509275964),\n",
       "   ('plans', 0.0463408207807137),\n",
       "   ('iowa', 0.03724002146171048),\n",
       "   ('area', 0.031583902009011684),\n",
       "   ('transit', 0.029746652763280607),\n",
       "   ('operations', 0.029650998680989683),\n",
       "   ('region', 0.028630893370822634),\n",
       "   ('bus', 0.023940404924489834),\n",
       "   ('priority', 0.019765257375106392),\n",
       "   ('people', 0.0157596265394866)]),\n",
       " (29,\n",
       "  [('planning', 0.060932885606218332),\n",
       "   ('county', 0.059246046260552521),\n",
       "   ('district', 0.059082234604936598),\n",
       "   ('districts', 0.046091488648481185),\n",
       "   ('boundaries', 0.037211490590221732),\n",
       "   ('city', 0.029524721059023727),\n",
       "   ('areas', 0.027095006552687029),\n",
       "   ('zoning', 0.026634642336752109),\n",
       "   ('gis', 0.024439676328900622),\n",
       "   ('council', 0.023082692493783637)]),\n",
       " (30,\n",
       "  [('mateo', 0.04184248749834401),\n",
       "   ('city', 0.023985079812541651),\n",
       "   ('gis', 0.019805378260364801),\n",
       "   ('land', 0.019219281849626336),\n",
       "   ('county', 0.018870460262658366),\n",
       "   ('parcel', 0.017318355184099247),\n",
       "   ('map', 0.017293542672449272),\n",
       "   ('point', 0.017217088467835123),\n",
       "   ('survey', 0.016483910162221081),\n",
       "   ('feature', 0.016048488156929334)]),\n",
       " (31,\n",
       "  [('politics', 0.11069826014751941),\n",
       "   ('government', 0.098315050573643273),\n",
       "   ('election', 0.038259777750699074),\n",
       "   ('campaign', 0.030948562034191236),\n",
       "   ('elections', 0.029682305103395073),\n",
       "   ('commission', 0.029314992871497855),\n",
       "   ('results', 0.024928316671568231),\n",
       "   ('city', 0.02132785699829063),\n",
       "   ('460', 0.020080548927070931),\n",
       "   ('finance', 0.019735216016406935)]),\n",
       " (32,\n",
       "  [('open', 0.11136160624479899),\n",
       "   ('created', 0.066633337086342601),\n",
       "   ('items', 0.046975156803616497),\n",
       "   ('clear', 0.024971656082104495),\n",
       "   ('volume', 0.017569028429596321),\n",
       "   ('please', 0.01620818951157825),\n",
       "   ('presented', 0.015509271243001166),\n",
       "   ('difficult', 0.014152486549053123),\n",
       "   ('potential', 0.013902418105983163),\n",
       "   ('note', 0.012958764775942001)]),\n",
       " (33,\n",
       "  [('finance', 0.071043372892685447),\n",
       "   ('summary', 0.060625406434642808),\n",
       "   ('dashboard', 0.057306369737017436),\n",
       "   ('ethics', 0.046020336169751791),\n",
       "   ('form', 0.040459920451206687),\n",
       "   ('management', 0.032747889878733547),\n",
       "   ('board', 0.023358112210499077),\n",
       "   ('pension', 0.017133753473461615),\n",
       "   ('includes', 0.016256755260059545),\n",
       "   ('last', 0.016102898579562843)]),\n",
       " (34,\n",
       "  [('education', 0.18150475758690063),\n",
       "   ('school', 0.081857692569206983),\n",
       "   ('schools', 0.0317766682175891),\n",
       "   ('students', 0.029571629399532486),\n",
       "   ('public', 0.017173423581827232),\n",
       "   ('governance', 0.016783317325556272),\n",
       "   ('state', 0.016578579370884614),\n",
       "   ('year', 0.016159604793580536),\n",
       "   ('student', 0.016139734254594727),\n",
       "   ('district', 0.01600362788318669)]),\n",
       " (35,\n",
       "  [('infrastructure', 0.082445242432479721),\n",
       "   ('city', 0.065893376157151143),\n",
       "   ('public', 0.032845769972150357),\n",
       "   ('maps', 0.030519532936435358),\n",
       "   ('works', 0.027935011789728573),\n",
       "   ('dallas', 0.022443140628194077),\n",
       "   ('los', 0.019882704408148669),\n",
       "   ('angeles', 0.019192175434757104),\n",
       "   ('types', 0.015113120712693303),\n",
       "   ('way', 0.013887150827763371)]),\n",
       " (36,\n",
       "  [('census', 0.078804980837745758),\n",
       "   ('indicators', 0.077373830079416436),\n",
       "   ('survey', 0.048035159786913896),\n",
       "   ('includes', 0.044835173852544735),\n",
       "   ('indicator', 0.036924523941196177),\n",
       "   ('community', 0.033221116599808108),\n",
       "   ('demographics', 0.028209305968815383),\n",
       "   ('bureau', 0.023872041750842676),\n",
       "   ('u', 0.021139289477577056),\n",
       "   ('level', 0.019204313061601801)]),\n",
       " (37,\n",
       "  [('financial', 0.095961100691599369),\n",
       "   ('permits', 0.066863273960211606),\n",
       "   ('building', 0.063195390054562622),\n",
       "   ('permit', 0.054456521696982667),\n",
       "   ('expenditures', 0.042924738948058642),\n",
       "   ('guide', 0.025058053425860767),\n",
       "   ('issued', 0.023243432964197546),\n",
       "   ('construction', 0.022884617487689943),\n",
       "   ('information', 0.02279686241791478),\n",
       "   ('filter', 0.021954851645997791)]),\n",
       " (38,\n",
       "  [('plan', 0.043864302305066827),\n",
       "   ('strategic', 0.034170749236072259),\n",
       "   ('master', 0.02434141736730544),\n",
       "   ('community', 0.020477914960891377),\n",
       "   ('fewer', 0.019330699927391008),\n",
       "   ('limitations', 0.018044419984850159),\n",
       "   ('registry', 0.017693548198715405),\n",
       "   ('cause', 0.016906069114867663),\n",
       "   ('years', 0.015944418985748578),\n",
       "   ('information', 0.014420142106983924)]),\n",
       " (39,\n",
       "  [('health', 0.30531189584891172),\n",
       "   ('san', 0.061101772036273934),\n",
       "   ('county', 0.058639270651817503),\n",
       "   ('care', 0.028005505975059769),\n",
       "   ('counties', 0.019925395256915498),\n",
       "   ('system', 0.016377029581779941),\n",
       "   ('facilities', 0.014007603368699854),\n",
       "   ('network', 0.013561457306569633),\n",
       "   ('home', 0.011206164684491395),\n",
       "   ('francisco', 0.010310472003196668)]),\n",
       " (40,\n",
       "  [('finance', 0.16474209816597238),\n",
       "   ('year', 0.062366577351005315),\n",
       "   ('state', 0.053382253954750457),\n",
       "   ('monthly', 0.050696248199469038),\n",
       "   ('fiscal', 0.047622222135699312),\n",
       "   ('payments', 0.035719919718136567),\n",
       "   ('government', 0.02645402325871795),\n",
       "   ('june', 0.022577028747073),\n",
       "   ('july', 0.021675911556054386),\n",
       "   ('report', 0.021590010484289327)]),\n",
       " (41,\n",
       "  [('service', 0.087262714441883243),\n",
       "   ('requests', 0.035031694542933929),\n",
       "   ('inspection', 0.032678935192867728),\n",
       "   ('inspections', 0.031350351826353977),\n",
       "   ('311', 0.026311943104393398),\n",
       "   ('public', 0.025968462717016926),\n",
       "   ('violations', 0.023768792319548215),\n",
       "   ('request', 0.022587674832020798),\n",
       "   ('complaints', 0.019370415310915674),\n",
       "   ('days', 0.018002600321326952)]),\n",
       " (42,\n",
       "  [('information', 0.024602766215276441),\n",
       "   ('list', 0.021580669434979348),\n",
       "   ('code', 0.017461004706817183),\n",
       "   ('required', 0.016538859124753839),\n",
       "   ('political', 0.013997537552533643),\n",
       "   ('utah', 0.013858241707600623),\n",
       "   ('public', 0.013319630833329574),\n",
       "   ('city', 0.013266613262239497),\n",
       "   ('years', 0.012297917571542876),\n",
       "   ('see', 0.012277655731227213)]),\n",
       " (43,\n",
       "  [('state', 0.1756382967463313),\n",
       "   ('new', 0.13340351799805059),\n",
       "   ('michigan', 0.10266572561037926),\n",
       "   ('york', 0.077727077387110127),\n",
       "   ('information', 0.029283874552224455),\n",
       "   ('check', 0.02032209041652535),\n",
       "   ('measurements', 0.019708043153636445),\n",
       "   ('jobs', 0.017040272127524295),\n",
       "   ('licensing', 0.016976894067415114),\n",
       "   ('ny', 0.016777641736857838)]),\n",
       " (44,\n",
       "  [('book', 0.040135367577023996),\n",
       "   ('lead', 0.040111314911710098),\n",
       "   ('stations', 0.035230307653653078),\n",
       "   ('databook', 0.03363611816437783),\n",
       "   ('station', 0.032779991270326547),\n",
       "   ('closed', 0.023280298248918037),\n",
       "   ('ecology', 0.022906441104600394),\n",
       "   ('solar', 0.019267555473289202),\n",
       "   ('power', 0.01904494570812373),\n",
       "   ('residential', 0.016817032555105343)]),\n",
       " (45,\n",
       "  [('report', 0.11604973949315449),\n",
       "   ('performance', 0.072106784996400927),\n",
       "   ('city', 0.064553993348579072),\n",
       "   ('goal', 0.049701354025019934),\n",
       "   ('resultsnola', 0.045459898953553526),\n",
       "   ('administration', 0.044925526194230785),\n",
       "   (\"city's\", 0.03417954310339625),\n",
       "   ('management', 0.033852728584638718),\n",
       "   ('key', 0.033837307690521595),\n",
       "   ('topeka', 0.022145009545635606)]),\n",
       " (46,\n",
       "  [('address', 0.042514297658594498),\n",
       "   ('name', 0.028895265435991142),\n",
       "   ('number', 0.026095926621437948),\n",
       "   ('facility', 0.021195054756114773),\n",
       "   ('patient', 0.018728329073792953),\n",
       "   ('location', 0.017356509556266626),\n",
       "   ('contains', 0.014702734565212696),\n",
       "   ('type', 0.014078242116513321),\n",
       "   ('g', 0.013900974382997767),\n",
       "   ('transfer', 0.013899331867379633)]),\n",
       " (47,\n",
       "  [('care', 0.11036088229652928),\n",
       "   ('finance', 0.049033830573026793),\n",
       "   ('provider', 0.044146456939484949),\n",
       "   ('child', 0.041722110536953544),\n",
       "   ('cost', 0.034993745722234584),\n",
       "   ('plan', 0.030778145306684032),\n",
       "   ('day', 0.028558344716958277),\n",
       "   ('bay', 0.025286386344854205),\n",
       "   ('government', 0.024128574624377789),\n",
       "   ('fee', 0.018205044303512732)]),\n",
       " (48,\n",
       "  [('business', 0.080327130459714152),\n",
       "   ('license', 0.032302648156979613),\n",
       "   ('job', 0.022002624988460773),\n",
       "   ('contracts', 0.021934165455248577),\n",
       "   ('businesses', 0.019770850331225706),\n",
       "   ('vendor', 0.017832975411663454),\n",
       "   ('owned', 0.017528163358468088),\n",
       "   ('contract', 0.017516996738737415),\n",
       "   ('licenses', 0.016969889175799842),\n",
       "   ('certified', 0.016688461456116979)]),\n",
       " (49,\n",
       "  [('development', 0.17602775675090593),\n",
       "   ('housing', 0.15221265764931263),\n",
       "   ('economic', 0.058854109443175158),\n",
       "   ('community', 0.04859194305354804),\n",
       "   ('infrastructure', 0.019040125293198554),\n",
       "   ('managed', 0.018304696035360443),\n",
       "   ('medicaid', 0.017876406351462711),\n",
       "   ('department', 0.014023829088608938),\n",
       "   ('buildings', 0.013775405992175799),\n",
       "   ('economy', 0.012347394313369113)]),\n",
       " (50,\n",
       "  [('program', 0.03251146357097217),\n",
       "   ('employees', 0.028975172275721717),\n",
       "   ('adult', 0.024408505499127525),\n",
       "   ('employee', 0.02429680205275107),\n",
       "   ('annual', 0.020498612612776752),\n",
       "   ('names', 0.018217941353291824),\n",
       "   ('full', 0.017909118396460151),\n",
       "   ('abuse', 0.016209683412924534),\n",
       "   ('nutrition', 0.015767712459609552),\n",
       "   ('beginning', 0.014213770821721556)]),\n",
       " (51,\n",
       "  [('cook', 0.05830855785510166),\n",
       "   ('high', 0.057680528554684904),\n",
       "   ('cases', 0.039417055039367849),\n",
       "   ('youth', 0.03165394178442571),\n",
       "   ('case', 0.027938083160155187),\n",
       "   ('court', 0.02661638705111698),\n",
       "   ('enrollment', 0.018080429121202373),\n",
       "   ('college', 0.01773522389517114),\n",
       "   ('wa', 0.016770584173977882),\n",
       "   ('gis/maps', 0.016618223819135965)])]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_52_sixty.show_topics(num_topics=52, formatted=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis of LDA Model Topic Formation:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a very decent topic analysis. By my count, **we've created 48 genuinely useful \"clusters\" of latent topics.** Since topic analysis is probabilistic, and our data contains \"noise\" (that is, words either far too specific or far too common to add real meaning) to begin with, **we have a few topics that won't help us.**\n",
    "- Topic 32 is just too vague. It probably has something to do with some open government performance metrics, but there are no words to really a distinguish it.  This is not a failure of the algorithm; I don't doubt these words really are occuring together alot in the data. But \"open, created, items\" doesn't tell us much.\n",
    "- Topic 24 clearly has to do with some \"special\" software/gis for open data (probably having to do with nursing) -- it looks like through the openmichigan portal. Again, these words are quite likely appearing together. But the practical application of a topic like this when it comes to determinging what is popular in the real world is limited.\n",
    "- Topic 42 has a similar problem to topics 32 and 42 topics, but is a little more clear for content type. It looks like political information for several years. But we can't really tell more specifics.\n",
    "- Topic 43 is what I'd call a \"parochial\" topic; it's clearly about common information like jobs and licensing from new york and michigan. Again, not useful as its own topic - but my spin is that this helps isolate \"new york\" and \"ny\" to keep them from inflating other topic views.\n",
    "\n",
    "**Other topics are useful, but may mix content. It could be a vagary of the English language, or it might actually reveal new insights:**\n",
    "- Topic 51 is a topic that tells us something, but it appears to be a mix of youth court cases and youth college enrollment. This could be that \"enrollment\" is used in English in the context of college and court-mandated programs. Then again, this could be \"at-risk youth outcome statistics - either court or college (or both).\n",
    "- Topic 37 appears to be about the construction, permits, and financials of building projects; since these are are pretty closely related in real life, however, I'd argue this is a good topic formation.\n",
    "\n",
    "**Many topics contain a proper name, but clearly identify something useful:**\n",
    "- Topic 9 -- \"recreation\", \"parks\", \"jersey\", \"park\", \"centers\", \"neighborhood\", \"centers\", etc. is clearly about parks and recreation facilities. It just contains jersey. Because of how we'll award our views/downloads counts to each topic, \"jersey\" should only skew this slightly. Again, this is just \"error\" (but not really error, according to the model - jersey probably had a lot of parks and recreation datasets) we have to tolerate if we can't exclude proper names as stop words.\n",
    "\n",
    "**Topic 13 has a seemingly weird outlier:**\n",
    "- Topic 13 is clearly about gas & fuel emissions, but also contains \"food\". I'm guessing \"food\" is linked to \"gas\" by the word \"natural\" across many datasets (natural food, natural gas). This is just noise we will have to tolerate.\n",
    "\n",
    "**Many topics are absolutely beautiful. A few examples:**\n",
    "- Topic 11 -- energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities -- identifies a city's environmental initiatives with words I wouldn't have even thought of to group together.\n",
    "- Topic 31 -- politics, government, election, campaign, elections, commissions, results, etc -- leaves no doubt about its content.\n",
    "- Topic 5 -- locations, bacteria, hours, culture, county, levels, contact, directory, e [as in e coli], contains -- is amazing. Words that could be all sorts of different topics that become so clear in context together. This is about local bacteria levels! (Presumably in lakes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load Saved Corpus:\n",
    "- We are going to use our LDA model to transform this corpus so we can return the *topic composition* of each *document*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus = corpora.MmCorpus('8.14corpus.mm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_lda = lda_52_sixty[corpus] #just a wrapper -- will convert on the fly when you call it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Let's check that this lines up:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(31, 0.50961538461538491)]\n",
      "[(31, 0.50961538461538491)]\n",
      "[(31, 0.50961538461538214)]\n"
     ]
    }
   ],
   "source": [
    "for doc in corpus_lda[20000:20003]:\n",
    "    print(doc) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20000    ['politics']\n",
       "20001    ['politics']\n",
       "20002    ['politics']\n",
       "Name: big_mash, dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[20000:20003]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('politics', 0.11069826014751941),\n",
       " ('government', 0.098315050573643273),\n",
       " ('election', 0.038259777750699074),\n",
       " ('campaign', 0.030948562034191236),\n",
       " ('elections', 0.029682305103395073),\n",
       " ('commission', 0.029314992871497855),\n",
       " ('results', 0.024928316671568231),\n",
       " ('city', 0.02132785699829063),\n",
       " ('460', 0.020080548927070931),\n",
       " ('finance', 0.019735216016406935)]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda_52_sixty.show_topic(31)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All three documents have only the tag 'politics\" to idenfity them. As we can see, our corpus transformation identifies document 31 as composing .5 of all of them. Document 31 is a politics and elections topic. Looks good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21807"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus_lda)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "corpus_lda_list = list(corpus_lda[0:21790]) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For some reason we can't convert our entire corpus into a list. It is truly mind boggling, and the potential issue is beyond my level of knowledge. But since we can convert most of the corpus, we'll make a (slightly abridged) new dataframe and add our topic compositions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "      <th>topic_comp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>9009</th>\n",
       "      <td>9009</td>\n",
       "      <td>Returning Citizen Child Support &amp; Inmate Oblig...</td>\n",
       "      <td>['Data', 'are', 'provided', 'by', 'the', 'Depa...</td>\n",
       "      <td>Department of Human Resources (DHR)</td>\n",
       "      <td>['number_of_released_inmates_who_pay_any_perce...</td>\n",
       "      <td>['Number of released inmates who pay any perce...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Health and Human Services</td>\n",
       "      <td>['inmate obligors', 'prisoner re-entry', 're-e...</td>\n",
       "      <td>official</td>\n",
       "      <td>1433.000</td>\n",
       "      <td>20.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>628.000</td>\n",
       "      <td>9.297</td>\n",
       "      <td>data.maryland.gov</td>\n",
       "      <td>['inmate', 'obligors', 'prisoner', 're-entry',...</td>\n",
       "      <td>['inmate', 'obligors', 'prisoner', 're-entry',...</td>\n",
       "      <td>[(4, 0.210601353097), (6, 0.0182243633613), (1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9010</th>\n",
       "      <td>9010</td>\n",
       "      <td>Breakfast, Lunch, and At-Risk Afterschool Meal...</td>\n",
       "      <td>['This', 'dataset', 'tracks', 'participation',...</td>\n",
       "      <td>Maryland State Department of Education (MSDE)</td>\n",
       "      <td>['at_risk_afterschool_meals_program_average_da...</td>\n",
       "      <td>['At-Risk Afterschool Program Avg. Daily Parti...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['education']</td>\n",
       "      <td>Health and Human Services</td>\n",
       "      <td>['meal', 'education', 'lunch', 'breakfast', 'c...</td>\n",
       "      <td>official</td>\n",
       "      <td>115.000</td>\n",
       "      <td>26.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>627.000</td>\n",
       "      <td>9.295</td>\n",
       "      <td>data.maryland.gov</td>\n",
       "      <td>['meal', 'education', 'lunch', 'breakfast', 'c...</td>\n",
       "      <td>['meal', 'education', 'lunch', 'breakfast', 'c...</td>\n",
       "      <td>[(1, 0.0776627218935), (2, 0.039201183432), (7...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9011</th>\n",
       "      <td>9011</td>\n",
       "      <td>Total Light Rail Trips Taken by Year: Column C...</td>\n",
       "      <td>['Data', 'are', 'provided', 'by', 'the', 'Mary...</td>\n",
       "      <td>Maryland Transit Authority</td>\n",
       "      <td>['fiscal_year', 'light_rail']</td>\n",
       "      <td>['Fiscal Year', 'Light Rail']</td>\n",
       "      <td>chart</td>\n",
       "      <td>['transportation']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['ridership', 'paratransit', 'mobility', 'rail...</td>\n",
       "      <td>official</td>\n",
       "      <td>181.000</td>\n",
       "      <td>12.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>621.000</td>\n",
       "      <td>9.281</td>\n",
       "      <td>data.maryland.gov</td>\n",
       "      <td>['ridership', 'paratransit', 'mobility', 'rail...</td>\n",
       "      <td>['ridership', 'paratransit', 'mobility', 'rail...</td>\n",
       "      <td>[(0, 0.0156918612058), (2, 0.0263660953474), (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                               name  \\\n",
       "9009        9009  Returning Citizen Child Support & Inmate Oblig...   \n",
       "9010        9010  Breakfast, Lunch, and At-Risk Afterschool Meal...   \n",
       "9011        9011  Total Light Rail Trips Taken by Year: Column C...   \n",
       "\n",
       "                                            description  \\\n",
       "9009  ['Data', 'are', 'provided', 'by', 'the', 'Depa...   \n",
       "9010  ['This', 'dataset', 'tracks', 'participation',...   \n",
       "9011  ['Data', 'are', 'provided', 'by', 'the', 'Mary...   \n",
       "\n",
       "                                        attribution  \\\n",
       "9009            Department of Human Resources (DHR)   \n",
       "9010  Maryland State Department of Education (MSDE)   \n",
       "9011                     Maryland Transit Authority   \n",
       "\n",
       "                                     columns_field_name  \\\n",
       "9009  ['number_of_released_inmates_who_pay_any_perce...   \n",
       "9010  ['at_risk_afterschool_meals_program_average_da...   \n",
       "9011                      ['fiscal_year', 'light_rail']   \n",
       "\n",
       "                                           columns_name     type  \\\n",
       "9009  ['Number of released inmates who pay any perce...  dataset   \n",
       "9010  ['At-Risk Afterschool Program Avg. Daily Parti...    chart   \n",
       "9011                      ['Fiscal Year', 'Light Rail']    chart   \n",
       "\n",
       "              categories            domain_category  \\\n",
       "9009                  []  Health and Human Services   \n",
       "9010       ['education']  Health and Human Services   \n",
       "9011  ['transportation']             Transportation   \n",
       "\n",
       "                                            domain_tags provenance  \\\n",
       "9009  ['inmate obligors', 'prisoner re-entry', 're-e...   official   \n",
       "9010  ['meal', 'education', 'lunch', 'breakfast', 'c...   official   \n",
       "9011  ['ridership', 'paratransit', 'mobility', 'rail...   official   \n",
       "\n",
       "      download_count  page_views_last_month  page_views_last_week  \\\n",
       "9009        1433.000                 20.000                 1.000   \n",
       "9010         115.000                 26.000                 2.000   \n",
       "9011         181.000                 12.000                 1.000   \n",
       "\n",
       "      page_views_total  page_views_total_log             domain  \\\n",
       "9009           628.000                 9.297  data.maryland.gov   \n",
       "9010           627.000                 9.295  data.maryland.gov   \n",
       "9011           621.000                 9.281  data.maryland.gov   \n",
       "\n",
       "                                                   mash  \\\n",
       "9009  ['inmate', 'obligors', 'prisoner', 're-entry',...   \n",
       "9010  ['meal', 'education', 'lunch', 'breakfast', 'c...   \n",
       "9011  ['ridership', 'paratransit', 'mobility', 'rail...   \n",
       "\n",
       "                                               big_mash  \\\n",
       "9009  ['inmate', 'obligors', 'prisoner', 're-entry',...   \n",
       "9010  ['meal', 'education', 'lunch', 'breakfast', 'c...   \n",
       "9011  ['ridership', 'paratransit', 'mobility', 'rail...   \n",
       "\n",
       "                                             topic_comp  \n",
       "9009  [(4, 0.210601353097), (6, 0.0182243633613), (1...  \n",
       "9010  [(1, 0.0776627218935), (2, 0.039201183432), (7...  \n",
       "9011  [(0, 0.0156918612058), (2, 0.0263660953474), (...  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = big_mash.copy()\n",
    "stats = stats.iloc[0:21790, :]\n",
    "stats = stats.assign(topic_comp = corpus_lda_list)\n",
    "stats.iloc[9009:9012, :] #random slice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's poke around to see how our model's calculated topic compositions do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"['police', 'traffic', 'fatality', 'fatalities', 'public', 'safety', 'Dataset', 'of', 'traffic', 'fatalities', 'January', '1st', 'December', '31st', '2015', 'The', 'Austin', 'Police', 'Department', 'Fatality', 'database', 'contains', 'only', 'those', 'crashes', 'investigated', 'by', 'APD', 'and', 'is', 'continuously', 'being', 'updated', 'due', 'to', 'on', 'going', 'investigations', 'The', 'data', 'provided', 'here', 'represents', 'a', 'snapshot', 'of', 'Traffic', 'Fatality', 'information', 'at', 'a', 'specific', 'point', 'in', 'time', 'and', 'may', 'change', 'Due', 'to', 'the', 'long', 'processing', 'times', 'for', 'toxicology', 'testing', 'impairment', 'and', 'suspected', 'impairment', 'statistics', 'are', 'based', 'on', 'the', 'initial', 'assessment', 'of', 'the', 'Detectives', 'and', 'Medical', 'Examiner']\""
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[900]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset is about traffic fatalities and crashes as recorded by the police department."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.5480149120022656\t Topic: 0.071*\"public\" + 0.069*\"safety\" + 0.048*\"fire\" + 0.021*\"month\" + 0.017*\"calls\" + 0.017*\"police\" + 0.017*\"emergency\" + 0.016*\"response\" + 0.015*\"discharge\" + 0.014*\"incident\" + 0.014*\"department\" + 0.011*\"incidents\" + 0.011*\"time\" + 0.011*\"discharges\" + 0.011*\"medical\" \n",
      "\n",
      "Score: 0.3004543236048098\t Topic: 0.046*\"information\" + 0.025*\"tab\" + 0.024*\"projects\" + 0.018*\"concerning\" + 0.018*\"may\" + 0.017*\"contains\" + 0.016*\"project\" + 0.014*\"provided\" + 0.014*\"additional\" + 0.012*\"improvement\" + 0.010*\"capital\" + 0.009*\"totals\" + 0.009*\"available\" + 0.009*\"release\" + 0.008*\"due\" \n",
      "\n",
      "Score: 0.051468371213253206\t Topic: 0.096*\"water\" + 0.048*\"environment\" + 0.021*\"protection\" + 0.020*\"tx\" + 0.020*\"filed\" + 0.019*\"waste\" + 0.019*\"site\" + 0.019*\"quality\" + 0.017*\"environmental\" + 0.016*\"monitoring\" + 0.015*\"activity\" + 0.013*\"facility\" + 0.012*\"recycling\" + 0.012*\"activities\" + 0.011*\"modified\" \n",
      "\n",
      "Score: 0.025782322551716974\t Topic: 0.168*\"public\" + 0.140*\"safety\" + 0.063*\"crime\" + 0.036*\"police\" + 0.019*\"department\" + 0.015*\"illinois\" + 0.013*\"race\" + 0.012*\"reported\" + 0.012*\"crimes\" + 0.011*\"criminal\" + 0.010*\"statistics\" + 0.010*\"statistical\" + 0.010*\"part\" + 0.009*\"arrests\" + 0.009*\"enforcement\" \n",
      "\n",
      "Score: 0.019940912786328306\t Topic: 0.070*\"locations\" + 0.026*\"bacteria\" + 0.026*\"hours\" + 0.025*\"culture\" + 0.024*\"county\" + 0.022*\"levels\" + 0.021*\"contact\" + 0.020*\"directory\" + 0.019*\"e\" + 0.019*\"contains\" + 0.016*\"including\" + 0.014*\"found\" + 0.014*\"information\" + 0.014*\"area\" + 0.013*\"posted\" \n",
      "\n",
      "Score: 0.01929159654447894\t Topic: 0.092*\"measures\" + 0.078*\"quality\" + 0.069*\"performance\" + 0.050*\"government\" + 0.026*\"life\" + 0.025*\"developed\" + 0.016*\"fair\" + 0.015*\"measurement\" + 0.014*\"definitions\" + 0.013*\"index\" + 0.012*\"key\" + 0.011*\"hospitalizations\" + 0.011*\"current\" + 0.011*\"livable\" + 0.010*\"countystat_measures\" \n",
      "\n",
      "Score: 0.01902192027150731\t Topic: 0.060*\"gas\" + 0.038*\"natural\" + 0.035*\"food\" + 0.027*\"greenhouse\" + 0.026*\"environment\" + 0.023*\"emissions\" + 0.021*\"costs\" + 0.021*\"resources\" + 0.021*\"program\" + 0.016*\"fuel\" + 0.014*\"agriculture\" + 0.013*\"information\" + 0.013*\"affordable\" + 0.013*\"plus\" + 0.013*\"total\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, score in sorted(lda_52_sixty[corpus[900]], key=lambda tup: -1*tup[1]): \n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) #15 word topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And indeed, our highest topic affinity comes from a public safety calls/emergency response and medical topic. That's a good match.\n",
    "\n",
    "However, we can see some potential pitfalls here. Several other topics register, including \"government performance measurements\" tag. This speaks to the need to set a threshold for awarding a topic \"credit\" -- i.e. it composes more than .2 of a topic, or it's the largest topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"['benefits', 'veterans', 'services', 'veterans', 'measure', 'a', 'infrastructure', 'health', '&', 'human', 'services', 'Quarterly', 'Measure', 'A', 'dashboard', 'data', 'for', 'Veterans', 'Services', 'Assessment', 'and', 'Staffing', 'initiative', 'Claims', 'data', 'sourced', 'from', 'VetPro', 'DVS', '19', 'reports;', 'Office', 'contacts', 'sources', 'from', 'Daily', 'Appointment', 'Logs']\""
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[20789]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.19399944234439653\t Topic: 0.080*\"business\" + 0.032*\"license\" + 0.022*\"job\" + 0.022*\"contracts\" + 0.020*\"businesses\" + 0.018*\"vendor\" + 0.018*\"owned\" + 0.018*\"contract\" + 0.017*\"licenses\" + 0.017*\"certified\" + 0.016*\"economy\" + 0.016*\"city\" + 0.016*\"list\" + 0.015*\"quantities\" + 0.015*\"vendors\" \n",
      "\n",
      "Score: 0.13777744063768205\t Topic: 0.049*\"transportation\" + 0.046*\"plans\" + 0.037*\"iowa\" + 0.032*\"area\" + 0.030*\"transit\" + 0.030*\"operations\" + 0.029*\"region\" + 0.024*\"bus\" + 0.020*\"priority\" + 0.016*\"people\" + 0.014*\"plan\" + 0.013*\"areas\" + 0.013*\"(including\" + 0.012*\"two\" + 0.012*\"dot\" \n",
      "\n",
      "Score: 0.13194214859348494\t Topic: 0.176*\"development\" + 0.152*\"housing\" + 0.059*\"economic\" + 0.049*\"community\" + 0.019*\"infrastructure\" + 0.018*\"managed\" + 0.018*\"medicaid\" + 0.014*\"department\" + 0.014*\"buildings\" + 0.012*\"economy\" + 0.012*\"assistance\" + 0.011*\"program\" + 0.010*\"family\" + 0.008*\"benefits\" + 0.008*\"addressing\" \n",
      "\n",
      "Score: 0.10411140583554371\t Topic: 0.305*\"health\" + 0.061*\"san\" + 0.059*\"county\" + 0.028*\"care\" + 0.020*\"counties\" + 0.016*\"system\" + 0.014*\"facilities\" + 0.014*\"network\" + 0.011*\"home\" + 0.010*\"francisco\" + 0.010*\"information\" + 0.009*\"providers\" + 0.009*\"immunization\" + 0.008*\"alone\" + 0.008*\"public\" \n",
      "\n",
      "Score: 0.06962864721485408\t Topic: 0.058*\"cook\" + 0.058*\"high\" + 0.039*\"cases\" + 0.032*\"youth\" + 0.028*\"case\" + 0.027*\"court\" + 0.018*\"enrollment\" + 0.018*\"college\" + 0.017*\"wa\" + 0.017*\"gis/maps\" + 0.014*\"non\" + 0.014*\"graduation\" + 0.011*\"less\" + 0.011*\"year\" + 0.011*\"four\" \n",
      "\n",
      "Score: 0.042889359016288285\t Topic: 0.025*\"information\" + 0.022*\"list\" + 0.017*\"code\" + 0.017*\"required\" + 0.014*\"political\" + 0.014*\"utah\" + 0.013*\"public\" + 0.013*\"city\" + 0.012*\"years\" + 0.012*\"see\" + 0.012*\"office\" + 0.012*\"field\" + 0.011*\"section\" + 0.011*\"file\" + 0.011*\"must\" \n",
      "\n",
      "Score: 0.04096265752525667\t Topic: 0.117*\"tax\" + 0.058*\"revenue\" + 0.039*\"sales\" + 0.035*\"taxes\" + 0.033*\"local\" + 0.031*\"colorado\" + 0.030*\"government\" + 0.022*\"state\" + 0.019*\"department\" + 0.015*\"sale\" + 0.015*\"information\" + 0.013*\"runs\" + 0.013*\"credits\" + 0.013*\"cities\" + 0.012*\"unincorporated\" \n",
      "\n",
      "Score: 0.037909374257837074\t Topic: 0.033*\"program\" + 0.029*\"employees\" + 0.024*\"adult\" + 0.024*\"employee\" + 0.020*\"annual\" + 0.018*\"names\" + 0.018*\"full\" + 0.016*\"abuse\" + 0.016*\"nutrition\" + 0.014*\"beginning\" + 0.014*\"table\" + 0.013*\"practices\" + 0.012*\"administration\" + 0.011*\"number\" + 0.011*\"received\" \n",
      "\n",
      "Score: 0.0374932028853666\t Topic: 0.110*\"care\" + 0.049*\"finance\" + 0.044*\"provider\" + 0.042*\"child\" + 0.035*\"cost\" + 0.031*\"plan\" + 0.029*\"day\" + 0.025*\"bay\" + 0.024*\"government\" + 0.018*\"fee\" + 0.018*\"franklin\" + 0.018*\"act\" + 0.017*\"effective\" + 0.013*\"administrative\" + 0.012*\"efficient\" \n",
      "\n",
      "Score: 0.036687104502038055\t Topic: 0.131*\"property\" + 0.052*\"consumption\" + 0.046*\"real\" + 0.031*\"value\" + 0.025*\"properties\" + 0.022*\"significant\" + 0.022*\"easy\" + 0.020*\"usage\" + 0.020*\"assessment\" + 0.015*\"company\" + 0.015*\"percentages\" + 0.015*\"city\" + 0.014*\"estate\" + 0.014*\"values\" + 0.012*\"government\" \n",
      "\n",
      "Score: 0.03596261241271573\t Topic: 0.079*\"census\" + 0.077*\"indicators\" + 0.048*\"survey\" + 0.045*\"includes\" + 0.037*\"indicator\" + 0.033*\"community\" + 0.028*\"demographics\" + 0.024*\"bureau\" + 0.021*\"u\" + 0.019*\"level\" + 0.018*\"estimates\" + 0.017*\"retirement\" + 0.013*\"count\" + 0.013*\"us\" + 0.012*\"poverty\" \n",
      "\n",
      "Score: 0.03514588859416444\t Topic: 0.132*\"population\" + 0.092*\"county\" + 0.084*\"demographics\" + 0.043*\"age\" + 0.028*\"total\" + 0.019*\"king\" + 0.019*\"washington\" + 0.018*\"montgomery\" + 0.018*\"number\" + 0.017*\"years\" + 0.015*\"ratio\" + 0.014*\"center\" + 0.014*\"gender\" + 0.012*\"governor's\" + 0.011*\"department\" \n",
      "\n",
      "Score: 0.03514588859416444\t Topic: 0.075*\"economy\" + 0.059*\"employment\" + 0.038*\"labor\" + 0.030*\"unemployment\" + 0.023*\"insurance\" + 0.022*\"credit\" + 0.021*\"workforce\" + 0.018*\"economic\" + 0.017*\"training\" + 0.016*\"federal\" + 0.015*\"industry\" + 0.014*\"prosperous\" + 0.014*\"wages\" + 0.013*\"work\" + 0.012*\"compensation\" \n",
      "\n",
      "Score: 0.03514588859416444\t Topic: 0.071*\"finance\" + 0.061*\"summary\" + 0.057*\"dashboard\" + 0.046*\"ethics\" + 0.040*\"form\" + 0.033*\"management\" + 0.023*\"board\" + 0.017*\"pension\" + 0.016*\"includes\" + 0.016*\"last\" + 0.016*\"city\" + 0.014*\"see\" + 0.012*\"system\" + 0.012*\"period\" + 0.012*\"reduction\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, score in sorted(lda_52_sixty[corpus[20789]], key=lambda tup: -1*tup[1]): \n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's a great example of **INHERENT** uncertainty in our model. A human eye can barely tell what the metadata on this dataset is saying -- it looks like veterans services, specifically health care -- as indicated by \"Daily Appointment Logs\". It also contains a random infrastructure tag and just several words that don't add much real life meaning. \n",
    "\n",
    "Our model tries to match it with a business licenses topic. I wouldn't say that's right - but look at the composition score. It's not even .2. **This speaks to the need to set a \"cut off\" to get credit for a topic's views. **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"['finance', 'health', 'Insurance', 'plan', 'premiums', 'available', 'to', 'Iowa', 'individuals', 'for', '2017', 'under', 'the', 'Affordable', 'Care', 'Act']\""
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[6718]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.3175787077909651\t Topic: 0.110*\"care\" + 0.049*\"finance\" + 0.044*\"provider\" + 0.042*\"child\" + 0.035*\"cost\" + 0.031*\"plan\" + 0.029*\"day\" + 0.025*\"bay\" + 0.024*\"government\" + 0.018*\"fee\" + 0.018*\"franklin\" + 0.018*\"act\" + 0.017*\"effective\" + 0.013*\"administrative\" + 0.012*\"efficient\" \n",
      "\n",
      "Score: 0.2573984964913682\t Topic: 0.075*\"economy\" + 0.059*\"employment\" + 0.038*\"labor\" + 0.030*\"unemployment\" + 0.023*\"insurance\" + 0.022*\"credit\" + 0.021*\"workforce\" + 0.018*\"economic\" + 0.017*\"training\" + 0.016*\"federal\" + 0.015*\"industry\" + 0.014*\"prosperous\" + 0.014*\"wages\" + 0.013*\"work\" + 0.012*\"compensation\" \n",
      "\n",
      "Score: 0.09159134428015532\t Topic: 0.049*\"transportation\" + 0.046*\"plans\" + 0.037*\"iowa\" + 0.032*\"area\" + 0.030*\"transit\" + 0.030*\"operations\" + 0.029*\"region\" + 0.024*\"bus\" + 0.020*\"priority\" + 0.016*\"people\" + 0.014*\"plan\" + 0.013*\"areas\" + 0.013*\"(including\" + 0.012*\"two\" + 0.012*\"dot\" \n",
      "\n",
      "Score: 0.08984170784776785\t Topic: 0.176*\"development\" + 0.152*\"housing\" + 0.059*\"economic\" + 0.049*\"community\" + 0.019*\"infrastructure\" + 0.018*\"managed\" + 0.018*\"medicaid\" + 0.014*\"department\" + 0.014*\"buildings\" + 0.012*\"economy\" + 0.012*\"assistance\" + 0.011*\"program\" + 0.010*\"family\" + 0.008*\"benefits\" + 0.008*\"addressing\" \n",
      "\n",
      "Score: 0.08493589743589744\t Topic: 0.132*\"population\" + 0.092*\"county\" + 0.084*\"demographics\" + 0.043*\"age\" + 0.028*\"total\" + 0.019*\"king\" + 0.019*\"washington\" + 0.018*\"montgomery\" + 0.018*\"number\" + 0.017*\"years\" + 0.015*\"ratio\" + 0.014*\"center\" + 0.014*\"gender\" + 0.012*\"governor's\" + 0.011*\"department\" \n",
      "\n",
      "Score: 0.08493589743589744\t Topic: 0.060*\"gas\" + 0.038*\"natural\" + 0.035*\"food\" + 0.027*\"greenhouse\" + 0.026*\"environment\" + 0.023*\"emissions\" + 0.021*\"costs\" + 0.021*\"resources\" + 0.021*\"program\" + 0.016*\"fuel\" + 0.014*\"agriculture\" + 0.013*\"information\" + 0.013*\"affordable\" + 0.013*\"plus\" + 0.013*\"total\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, score in sorted(lda_52_sixty[corpus[6718]], key=lambda tup: -1*tup[1]): \n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We haven't really created a \"health insurance\" topic tag. However, the top topic for this metadata is topic that tries to match provider information and finances - it's top words \"care, finance, provider\" -- actually match a real life dataset about ACA insurance plans pretty well! **This is a good example of the model not being perfect, but still adding value.**\n",
    "\n",
    "**Also note** our second highest affinity comes from an economy and unemployment topic, which includes \"insurance\". However, I bet this is in the context of unemployment insurance!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"['housing', 'Office', 'of', 'Housing', 'Data']\""
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "big_mash.big_mash[18]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score: 0.5048076923076926\t Topic: 0.176*\"development\" + 0.152*\"housing\" + 0.059*\"economic\" + 0.049*\"community\" + 0.019*\"infrastructure\" + 0.018*\"managed\" + 0.018*\"medicaid\" + 0.014*\"department\" + 0.014*\"buildings\" + 0.012*\"economy\" + 0.012*\"assistance\" + 0.011*\"program\" + 0.010*\"family\" + 0.008*\"benefits\" + 0.008*\"addressing\" \n",
      "\n",
      "Score: 0.25480769230769246\t Topic: 0.048*\"death\" + 0.042*\"maryland\" + 0.029*\"database\" + 0.027*\"deaths\" + 0.027*\"office\" + 0.024*\"department\" + 0.021*\"new\" + 0.020*\"cdc\" + 0.019*\"metadata\" + 0.018*\"citizens\" + 0.017*\"among\" + 0.017*\"vital\" + 0.015*\"state\" + 0.014*\"infants\" + 0.014*\"national\" \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for index, score in sorted(lda_52_sixty[corpus[18]], key=lambda tup: -1*tup[1]): \n",
    "    print(\"Score: {}\\t Topic: {} \\n\".format(score, lda_52_sixty.print_topic(index, 15))) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Solid match to a housing and community assistance topic."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now let's calculate the popularity of each topic across all datasets\n",
    "- We will need some way to give a topic \"credit\" for the views and downloads of a dataset that it matches."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Winner take all\" popularity metric:**\n",
    "- **Scoring Rules:**\n",
    "    - Only the topic that composes the largest share of a document scores \"points\" for its \"Adjusted Popularity\" total.\n",
    "    - If a topic composes the largest share of that document, its \"points\" are its composition score times that dataset's combined log views and downloads.\n",
    "        - We're using log views so as not to overweigh datasets that just have alot of eyeballs on them (like ones from NYC). \n",
    "    - So, in our example above, the \"development, housing, economic, etc...\" dataset would score .504 times that dataset's (log views + downloads)\n",
    "    - Remember, each \"document\" is just a list of words from one of 28000 datasets' metadata."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First **let's remove healthcare.gov data from our dataset**; it inadvertently crept in there."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "topics = stats[stats.domain != 'data.healthcare.gov']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def winner_take_all_pop(df):\n",
    "    results_dict = {}\n",
    "    df = df.fillna(0) #have to fill NaNs or you'll get wonky results\n",
    "\n",
    "    for row_num in df.index:\n",
    "        tup_list = df.topic_comp[row_num] #list of (topic, doc composition) tuples\n",
    "        \n",
    "        #neat little trick to return only the tuple w/highest index[1] value\n",
    "        winner_tuple = max(tup_list, key=lambda item:item[1])  \n",
    "            \n",
    "        if not winner_tuple[0] in results_dict: #if not in dict, add it with its score\n",
    "            #must use .loc, treating row_num like a label and not an integer\n",
    "            results_dict[winner_tuple[0]] = (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "        else:\n",
    "            pass\n",
    "        \n",
    "        if winner_tuple[0] in results_dict: #if in dict, increment that key's value with score\n",
    "            results_dict[winner_tuple[0]] += (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "    \n",
    "    return results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "winners_dict = winner_take_all_pop(topics)\n",
    "#winners_dict #can uncomment just to double check dict numbers align with df topic number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make a quick and dirty function to replace these topic numbers with names in a sorted df for easy analysis:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def dict_to_df(d):\n",
    "    df = pd.DataFrame.from_dict(d, orient='index')\n",
    "    df = df.rename(index=str, columns={0:\"Adjusted_Popularity\"})\n",
    "    df = df.reset_index() #DON'T drop; our index carries real value as it is our topic numbers\n",
    "    df = df.rename(columns={'index': 'Topic Number'}) #when you reset_index with drop=False, you get orig index as col\n",
    "    df['Topic Number'] = pd.to_numeric(df['Topic Number'])\n",
    "\n",
    "    df = df.sort_values(by='Topic Number', ascending=True) \n",
    "\n",
    "    words_in_topics = [tup[1] for tup in lda_52_sixty.show_topics(num_topics=52, formatted=False)]\n",
    "    df['Topic'] = words_in_topics\n",
    "\n",
    "    df[[\"topic1\", \"topic2\", \"topic3\", \"topic4\", \n",
    "           \"topic5\", \"topic6\", \"topic7\", \"topic8\", \"topic9\", \"topic10\"]] = df.Topic.apply(pd.Series)\n",
    "\n",
    "    df = df.sort_values(by='Adjusted_Popularity', ascending=False)\n",
    "    df = df.reset_index(drop=True) #this time we want to get rid of old index, which means nothing post sorting\n",
    "    \n",
    "    return df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Number</th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28</td>\n",
       "      <td>16465623.054</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "      <td>(transportation, 0.0493422015093)</td>\n",
       "      <td>(plans, 0.0463408207807)</td>\n",
       "      <td>(iowa, 0.0372400214617)</td>\n",
       "      <td>(area, 0.031583902009)</td>\n",
       "      <td>(transit, 0.0297466527633)</td>\n",
       "      <td>(operations, 0.029650998681)</td>\n",
       "      <td>(region, 0.0286308933708)</td>\n",
       "      <td>(bus, 0.0239404049245)</td>\n",
       "      <td>(priority, 0.0197652573751)</td>\n",
       "      <td>(people, 0.0157596265395)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>526719.440</td>\n",
       "      <td>[(service, 0.0872627144419), (requests, 0.0350...</td>\n",
       "      <td>(service, 0.0872627144419)</td>\n",
       "      <td>(requests, 0.0350316945429)</td>\n",
       "      <td>(inspection, 0.0326789351929)</td>\n",
       "      <td>(inspections, 0.0313503518264)</td>\n",
       "      <td>(311, 0.0263119431044)</td>\n",
       "      <td>(public, 0.025968462717)</td>\n",
       "      <td>(violations, 0.0237687923195)</td>\n",
       "      <td>(request, 0.022587674832)</td>\n",
       "      <td>(complaints, 0.0193704153109)</td>\n",
       "      <td>(days, 0.0180026003213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>494236.068</td>\n",
       "      <td>[(police, 0.0495996521003), (user, 0.032265367...</td>\n",
       "      <td>(police, 0.0495996521003)</td>\n",
       "      <td>(user, 0.0322653671005)</td>\n",
       "      <td>(information, 0.02962646057)</td>\n",
       "      <td>(page, 0.0244358789937)</td>\n",
       "      <td>(department, 0.0237288839327)</td>\n",
       "      <td>(use, 0.023314363269)</td>\n",
       "      <td>(orleans, 0.0183127709217)</td>\n",
       "      <td>(injury, 0.0174977535732)</td>\n",
       "      <td>(may, 0.0166580734835)</td>\n",
       "      <td>(records, 0.0131997374277)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>391551.864</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "      <td>(transportation, 0.164542015423)</td>\n",
       "      <td>(traffic, 0.0382494150922)</td>\n",
       "      <td>(street, 0.0297088546702)</td>\n",
       "      <td>(parking, 0.0260484479646)</td>\n",
       "      <td>(infrastructure, 0.024499661836)</td>\n",
       "      <td>(city, 0.0242484880067)</td>\n",
       "      <td>(safe, 0.0239084069945)</td>\n",
       "      <td>(vehicle, 0.0231351100665)</td>\n",
       "      <td>(streets, 0.0219852292388)</td>\n",
       "      <td>(bike, 0.0155102577616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>26</td>\n",
       "      <td>230235.304</td>\n",
       "      <td>[(public, 0.168381768411), (safety, 0.13998228...</td>\n",
       "      <td>(public, 0.168381768411)</td>\n",
       "      <td>(safety, 0.139982283853)</td>\n",
       "      <td>(crime, 0.0631808672948)</td>\n",
       "      <td>(police, 0.03597164994)</td>\n",
       "      <td>(department, 0.0193424493246)</td>\n",
       "      <td>(illinois, 0.0152118075546)</td>\n",
       "      <td>(race, 0.0133652012852)</td>\n",
       "      <td>(reported, 0.011776909211)</td>\n",
       "      <td>(crimes, 0.0115504690759)</td>\n",
       "      <td>(criminal, 0.0105105785255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>43</td>\n",
       "      <td>220462.909</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "      <td>(state, 0.175638296746)</td>\n",
       "      <td>(new, 0.133403517998)</td>\n",
       "      <td>(michigan, 0.10266572561)</td>\n",
       "      <td>(york, 0.0777270773871)</td>\n",
       "      <td>(information, 0.0292838745522)</td>\n",
       "      <td>(check, 0.0203220904165)</td>\n",
       "      <td>(measurements, 0.0197080431536)</td>\n",
       "      <td>(jobs, 0.0170402721275)</td>\n",
       "      <td>(licensing, 0.0169768940674)</td>\n",
       "      <td>(ny, 0.0167776417369)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>18</td>\n",
       "      <td>189346.054</td>\n",
       "      <td>[(column, 0.0634273088776), (update, 0.0466752...</td>\n",
       "      <td>(column, 0.0634273088776)</td>\n",
       "      <td>(update, 0.0466752365795)</td>\n",
       "      <td>(homeless, 0.0416639506669)</td>\n",
       "      <td>(annually, 0.0300723765516)</td>\n",
       "      <td>(frequency, 0.0292689424963)</td>\n",
       "      <td>(position, 0.021127522302)</td>\n",
       "      <td>(city, 0.019841539727)</td>\n",
       "      <td>(commercial, 0.0184651409887)</td>\n",
       "      <td>(daily, 0.0168690536946)</td>\n",
       "      <td>(salaries, 0.0155892433715)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>48</td>\n",
       "      <td>188011.679</td>\n",
       "      <td>[(business, 0.0803271304597), (license, 0.0323...</td>\n",
       "      <td>(business, 0.0803271304597)</td>\n",
       "      <td>(license, 0.032302648157)</td>\n",
       "      <td>(job, 0.0220026249885)</td>\n",
       "      <td>(contracts, 0.0219341654552)</td>\n",
       "      <td>(businesses, 0.0197708503312)</td>\n",
       "      <td>(vendor, 0.0178329754117)</td>\n",
       "      <td>(owned, 0.0175281633585)</td>\n",
       "      <td>(contract, 0.0175169967387)</td>\n",
       "      <td>(licenses, 0.0169698891758)</td>\n",
       "      <td>(certified, 0.0166884614561)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>24</td>\n",
       "      <td>175960.006</td>\n",
       "      <td>[(official, 0.0610942909105), (account, 0.0497...</td>\n",
       "      <td>(official, 0.0610942909105)</td>\n",
       "      <td>(account, 0.0497660475395)</td>\n",
       "      <td>(accounts, 0.0483014586496)</td>\n",
       "      <td>((openmichigan@michigan, 0.0408207146959)</td>\n",
       "      <td>(special, 0.0291639985893)</td>\n",
       "      <td>(nursing, 0.028770388416)</td>\n",
       "      <td>(software, 0.0268065464462)</td>\n",
       "      <td>(gis, 0.023974220834)</td>\n",
       "      <td>(use, 0.0225286064837)</td>\n",
       "      <td>(consumer, 0.0224473950606)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>29</td>\n",
       "      <td>172425.614</td>\n",
       "      <td>[(planning, 0.0609328856062), (county, 0.05924...</td>\n",
       "      <td>(planning, 0.0609328856062)</td>\n",
       "      <td>(county, 0.0592460462606)</td>\n",
       "      <td>(district, 0.0590822346049)</td>\n",
       "      <td>(districts, 0.0460914886485)</td>\n",
       "      <td>(boundaries, 0.0372114905902)</td>\n",
       "      <td>(city, 0.029524721059)</td>\n",
       "      <td>(areas, 0.0270950065527)</td>\n",
       "      <td>(zoning, 0.0266346423368)</td>\n",
       "      <td>(gis, 0.0244396763289)</td>\n",
       "      <td>(council, 0.0230826924938)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>31</td>\n",
       "      <td>166547.463</td>\n",
       "      <td>[(politics, 0.110698260148), (government, 0.09...</td>\n",
       "      <td>(politics, 0.110698260148)</td>\n",
       "      <td>(government, 0.0983150505736)</td>\n",
       "      <td>(election, 0.0382597777507)</td>\n",
       "      <td>(campaign, 0.0309485620342)</td>\n",
       "      <td>(elections, 0.0296823051034)</td>\n",
       "      <td>(commission, 0.0293149928715)</td>\n",
       "      <td>(results, 0.0249283166716)</td>\n",
       "      <td>(city, 0.0213278569983)</td>\n",
       "      <td>(460, 0.0200805489271)</td>\n",
       "      <td>(finance, 0.0197352160164)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>39</td>\n",
       "      <td>157506.251</td>\n",
       "      <td>[(health, 0.305311895849), (san, 0.06110177203...</td>\n",
       "      <td>(health, 0.305311895849)</td>\n",
       "      <td>(san, 0.0611017720363)</td>\n",
       "      <td>(county, 0.0586392706518)</td>\n",
       "      <td>(care, 0.0280055059751)</td>\n",
       "      <td>(counties, 0.0199253952569)</td>\n",
       "      <td>(system, 0.0163770295818)</td>\n",
       "      <td>(facilities, 0.0140076033687)</td>\n",
       "      <td>(network, 0.0135614573066)</td>\n",
       "      <td>(home, 0.0112061646845)</td>\n",
       "      <td>(francisco, 0.0103104720032)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>46</td>\n",
       "      <td>142933.852</td>\n",
       "      <td>[(address, 0.0425142976586), (name, 0.02889526...</td>\n",
       "      <td>(address, 0.0425142976586)</td>\n",
       "      <td>(name, 0.028895265436)</td>\n",
       "      <td>(number, 0.0260959266214)</td>\n",
       "      <td>(facility, 0.0211950547561)</td>\n",
       "      <td>(patient, 0.0187283290738)</td>\n",
       "      <td>(location, 0.0173565095563)</td>\n",
       "      <td>(contains, 0.0147027345652)</td>\n",
       "      <td>(type, 0.0140782421165)</td>\n",
       "      <td>(g, 0.013900974383)</td>\n",
       "      <td>(transfer, 0.0138993318674)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>50</td>\n",
       "      <td>128287.970</td>\n",
       "      <td>[(program, 0.032511463571), (employees, 0.0289...</td>\n",
       "      <td>(program, 0.032511463571)</td>\n",
       "      <td>(employees, 0.0289751722757)</td>\n",
       "      <td>(adult, 0.0244085054991)</td>\n",
       "      <td>(employee, 0.0242968020528)</td>\n",
       "      <td>(annual, 0.0204986126128)</td>\n",
       "      <td>(names, 0.0182179413533)</td>\n",
       "      <td>(full, 0.0179091183965)</td>\n",
       "      <td>(abuse, 0.0162096834129)</td>\n",
       "      <td>(nutrition, 0.0157677124596)</td>\n",
       "      <td>(beginning, 0.0142137708217)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>37</td>\n",
       "      <td>127633.700</td>\n",
       "      <td>[(financial, 0.0959611006916), (permits, 0.066...</td>\n",
       "      <td>(financial, 0.0959611006916)</td>\n",
       "      <td>(permits, 0.0668632739602)</td>\n",
       "      <td>(building, 0.0631953900546)</td>\n",
       "      <td>(permit, 0.054456521697)</td>\n",
       "      <td>(expenditures, 0.0429247389481)</td>\n",
       "      <td>(guide, 0.0250580534259)</td>\n",
       "      <td>(issued, 0.0232434329642)</td>\n",
       "      <td>(construction, 0.0228846174877)</td>\n",
       "      <td>(information, 0.0227968624179)</td>\n",
       "      <td>(filter, 0.021954851646)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Topic Number  Adjusted_Popularity  \\\n",
       "0             28         16465623.054   \n",
       "1             41           526719.440   \n",
       "2             23           494236.068   \n",
       "3              3           391551.864   \n",
       "4             26           230235.304   \n",
       "5             43           220462.909   \n",
       "6             18           189346.054   \n",
       "7             48           188011.679   \n",
       "8             24           175960.006   \n",
       "9             29           172425.614   \n",
       "10            31           166547.463   \n",
       "11            39           157506.251   \n",
       "12            46           142933.852   \n",
       "13            50           128287.970   \n",
       "14            37           127633.700   \n",
       "\n",
       "                                                Topic  \\\n",
       "0   [(transportation, 0.0493422015093), (plans, 0....   \n",
       "1   [(service, 0.0872627144419), (requests, 0.0350...   \n",
       "2   [(police, 0.0495996521003), (user, 0.032265367...   \n",
       "3   [(transportation, 0.164542015423), (traffic, 0...   \n",
       "4   [(public, 0.168381768411), (safety, 0.13998228...   \n",
       "5   [(state, 0.175638296746), (new, 0.133403517998...   \n",
       "6   [(column, 0.0634273088776), (update, 0.0466752...   \n",
       "7   [(business, 0.0803271304597), (license, 0.0323...   \n",
       "8   [(official, 0.0610942909105), (account, 0.0497...   \n",
       "9   [(planning, 0.0609328856062), (county, 0.05924...   \n",
       "10  [(politics, 0.110698260148), (government, 0.09...   \n",
       "11  [(health, 0.305311895849), (san, 0.06110177203...   \n",
       "12  [(address, 0.0425142976586), (name, 0.02889526...   \n",
       "13  [(program, 0.032511463571), (employees, 0.0289...   \n",
       "14  [(financial, 0.0959611006916), (permits, 0.066...   \n",
       "\n",
       "                               topic1                         topic2  \\\n",
       "0   (transportation, 0.0493422015093)       (plans, 0.0463408207807)   \n",
       "1          (service, 0.0872627144419)    (requests, 0.0350316945429)   \n",
       "2           (police, 0.0495996521003)        (user, 0.0322653671005)   \n",
       "3    (transportation, 0.164542015423)     (traffic, 0.0382494150922)   \n",
       "4            (public, 0.168381768411)       (safety, 0.139982283853)   \n",
       "5             (state, 0.175638296746)          (new, 0.133403517998)   \n",
       "6           (column, 0.0634273088776)      (update, 0.0466752365795)   \n",
       "7         (business, 0.0803271304597)      (license, 0.032302648157)   \n",
       "8         (official, 0.0610942909105)     (account, 0.0497660475395)   \n",
       "9         (planning, 0.0609328856062)      (county, 0.0592460462606)   \n",
       "10         (politics, 0.110698260148)  (government, 0.0983150505736)   \n",
       "11           (health, 0.305311895849)         (san, 0.0611017720363)   \n",
       "12         (address, 0.0425142976586)         (name, 0.028895265436)   \n",
       "13          (program, 0.032511463571)   (employees, 0.0289751722757)   \n",
       "14       (financial, 0.0959611006916)     (permits, 0.0668632739602)   \n",
       "\n",
       "                           topic3                                     topic4  \\\n",
       "0         (iowa, 0.0372400214617)                     (area, 0.031583902009)   \n",
       "1   (inspection, 0.0326789351929)             (inspections, 0.0313503518264)   \n",
       "2    (information, 0.02962646057)                    (page, 0.0244358789937)   \n",
       "3       (street, 0.0297088546702)                 (parking, 0.0260484479646)   \n",
       "4        (crime, 0.0631808672948)                    (police, 0.03597164994)   \n",
       "5       (michigan, 0.10266572561)                    (york, 0.0777270773871)   \n",
       "6     (homeless, 0.0416639506669)                (annually, 0.0300723765516)   \n",
       "7          (job, 0.0220026249885)               (contracts, 0.0219341654552)   \n",
       "8     (accounts, 0.0483014586496)  ((openmichigan@michigan, 0.0408207146959)   \n",
       "9     (district, 0.0590822346049)               (districts, 0.0460914886485)   \n",
       "10    (election, 0.0382597777507)                (campaign, 0.0309485620342)   \n",
       "11      (county, 0.0586392706518)                    (care, 0.0280055059751)   \n",
       "12      (number, 0.0260959266214)                (facility, 0.0211950547561)   \n",
       "13       (adult, 0.0244085054991)                (employee, 0.0242968020528)   \n",
       "14    (building, 0.0631953900546)                   (permit, 0.054456521697)   \n",
       "\n",
       "                              topic5                         topic6  \\\n",
       "0         (transit, 0.0297466527633)   (operations, 0.029650998681)   \n",
       "1             (311, 0.0263119431044)       (public, 0.025968462717)   \n",
       "2      (department, 0.0237288839327)          (use, 0.023314363269)   \n",
       "3   (infrastructure, 0.024499661836)        (city, 0.0242484880067)   \n",
       "4      (department, 0.0193424493246)    (illinois, 0.0152118075546)   \n",
       "5     (information, 0.0292838745522)       (check, 0.0203220904165)   \n",
       "6       (frequency, 0.0292689424963)     (position, 0.021127522302)   \n",
       "7      (businesses, 0.0197708503312)      (vendor, 0.0178329754117)   \n",
       "8         (special, 0.0291639985893)      (nursing, 0.028770388416)   \n",
       "9      (boundaries, 0.0372114905902)         (city, 0.029524721059)   \n",
       "10      (elections, 0.0296823051034)  (commission, 0.0293149928715)   \n",
       "11       (counties, 0.0199253952569)      (system, 0.0163770295818)   \n",
       "12        (patient, 0.0187283290738)    (location, 0.0173565095563)   \n",
       "13         (annual, 0.0204986126128)       (names, 0.0182179413533)   \n",
       "14   (expenditures, 0.0429247389481)       (guide, 0.0250580534259)   \n",
       "\n",
       "                             topic7                           topic8  \\\n",
       "0         (region, 0.0286308933708)           (bus, 0.0239404049245)   \n",
       "1     (violations, 0.0237687923195)        (request, 0.022587674832)   \n",
       "2        (orleans, 0.0183127709217)        (injury, 0.0174977535732)   \n",
       "3           (safe, 0.0239084069945)       (vehicle, 0.0231351100665)   \n",
       "4           (race, 0.0133652012852)       (reported, 0.011776909211)   \n",
       "5   (measurements, 0.0197080431536)          (jobs, 0.0170402721275)   \n",
       "6            (city, 0.019841539727)    (commercial, 0.0184651409887)   \n",
       "7          (owned, 0.0175281633585)      (contract, 0.0175169967387)   \n",
       "8       (software, 0.0268065464462)            (gis, 0.023974220834)   \n",
       "9          (areas, 0.0270950065527)        (zoning, 0.0266346423368)   \n",
       "10       (results, 0.0249283166716)          (city, 0.0213278569983)   \n",
       "11    (facilities, 0.0140076033687)       (network, 0.0135614573066)   \n",
       "12      (contains, 0.0147027345652)          (type, 0.0140782421165)   \n",
       "13          (full, 0.0179091183965)         (abuse, 0.0162096834129)   \n",
       "14        (issued, 0.0232434329642)  (construction, 0.0228846174877)   \n",
       "\n",
       "                            topic9                       topic10  \n",
       "0      (priority, 0.0197652573751)     (people, 0.0157596265395)  \n",
       "1    (complaints, 0.0193704153109)       (days, 0.0180026003213)  \n",
       "2           (may, 0.0166580734835)    (records, 0.0131997374277)  \n",
       "3       (streets, 0.0219852292388)       (bike, 0.0155102577616)  \n",
       "4        (crimes, 0.0115504690759)   (criminal, 0.0105105785255)  \n",
       "5     (licensing, 0.0169768940674)         (ny, 0.0167776417369)  \n",
       "6         (daily, 0.0168690536946)   (salaries, 0.0155892433715)  \n",
       "7      (licenses, 0.0169698891758)  (certified, 0.0166884614561)  \n",
       "8           (use, 0.0225286064837)   (consumer, 0.0224473950606)  \n",
       "9           (gis, 0.0244396763289)    (council, 0.0230826924938)  \n",
       "10          (460, 0.0200805489271)    (finance, 0.0197352160164)  \n",
       "11         (home, 0.0112061646845)  (francisco, 0.0103104720032)  \n",
       "12             (g, 0.013900974383)   (transfer, 0.0138993318674)  \n",
       "13    (nutrition, 0.0157677124596)  (beginning, 0.0142137708217)  \n",
       "14  (information, 0.0227968624179)      (filter, 0.021954851646)  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "winners_df = dict_to_df(winners_dict)\n",
    "winners_df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results:\n",
    "- \"Transportation Planning and Operations\" predominates. Another transportation dataset -- something that I would call \"Personal Transportation\" -- parking, traffic, vehicle, and even bike -- is also the fourth most popular by this metric. Between these two topics, it's safe to conclude transportation datasets are very popular.\n",
    "- Likewise with a \"Police Information and Records\" and \"Crime Reports\" topics, which place 3 and 5 respectively.\n",
    "- The metrics for rankings 2, 3, and 4 are very close, and could easily change based on how we tweak our metrics. Let's see."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## \"Winner Take All with Thresholds\" Rules:\n",
    "- **Scoring Rules:**\n",
    "    - Same as \"Winner Take All\", except a winning topic must compose at least a certain threshold of a document to get any points.\n",
    "    - We'll try 0.2 (low) and 0.5 (high) thresholds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def winner_take_all_thresholds(df, thresh):\n",
    "    results_dict = {}\n",
    "    df = df.fillna(0) #make sure we have a clean df\n",
    "    \n",
    "    if thresh == \"low\":\n",
    "        for row_num in df.index:\n",
    "            tup_list = df.topic_comp[row_num] #list of (topic, doc composition) tuples\n",
    "        \n",
    "            #neat little trick to return only the tuple w/highest index[1] value\n",
    "            winner_tuple = max(tup_list, key=lambda item:item[1])  \n",
    "            \n",
    "            if not winner_tuple[0] in results_dict: #if not in dict, add it with its score\n",
    "                if winner_tuple[1] > 0.2:\n",
    "                    results_dict[winner_tuple[0]] = (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "            else:\n",
    "                pass\n",
    "        \n",
    "            if winner_tuple[0] in results_dict: #if in dict, increment that key's value with score\n",
    "                if winner_tuple[1] > 0.2:\n",
    "                    results_dict[winner_tuple[0]] += (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "            else:\n",
    "                pass\n",
    "                \n",
    "    if thresh == \"high\":\n",
    "        for row_num in df.index:\n",
    "            tup_list = df.topic_comp[row_num] #list of (topic, doc composition) tuples\n",
    "        \n",
    "            #neat little trick to return only the tuple w/highest index[1] value\n",
    "            winner_tuple = max(tup_list, key=lambda item:item[1])  \n",
    "            \n",
    "            if not winner_tuple[0] in results_dict: #if not in dict, add it with its score\n",
    "                if winner_tuple[1] > 0.5:\n",
    "                    results_dict[winner_tuple[0]] = (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "            else:\n",
    "                pass\n",
    "        \n",
    "            if winner_tuple[0] in results_dict: #if in dict, increment that key's value with score\n",
    "                if winner_tuple[1] > 0.5:\n",
    "                    results_dict[winner_tuple[0]] += (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "            else:\n",
    "                pass\n",
    "\n",
    "    return results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "low = winner_take_all_thresholds(topics, thresh='low')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "low_thresh_ranks = dict_to_df(low)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Number</th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28</td>\n",
       "      <td>16461078.655</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "      <td>(transportation, 0.0493422015093)</td>\n",
       "      <td>(plans, 0.0463408207807)</td>\n",
       "      <td>(iowa, 0.0372400214617)</td>\n",
       "      <td>(area, 0.031583902009)</td>\n",
       "      <td>(transit, 0.0297466527633)</td>\n",
       "      <td>(operations, 0.029650998681)</td>\n",
       "      <td>(region, 0.0286308933708)</td>\n",
       "      <td>(bus, 0.0239404049245)</td>\n",
       "      <td>(priority, 0.0197652573751)</td>\n",
       "      <td>(people, 0.0157596265395)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>518910.864</td>\n",
       "      <td>[(service, 0.0872627144419), (requests, 0.0350...</td>\n",
       "      <td>(service, 0.0872627144419)</td>\n",
       "      <td>(requests, 0.0350316945429)</td>\n",
       "      <td>(inspection, 0.0326789351929)</td>\n",
       "      <td>(inspections, 0.0313503518264)</td>\n",
       "      <td>(311, 0.0263119431044)</td>\n",
       "      <td>(public, 0.025968462717)</td>\n",
       "      <td>(violations, 0.0237687923195)</td>\n",
       "      <td>(request, 0.022587674832)</td>\n",
       "      <td>(complaints, 0.0193704153109)</td>\n",
       "      <td>(days, 0.0180026003213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>493161.782</td>\n",
       "      <td>[(police, 0.0495996521003), (user, 0.032265367...</td>\n",
       "      <td>(police, 0.0495996521003)</td>\n",
       "      <td>(user, 0.0322653671005)</td>\n",
       "      <td>(information, 0.02962646057)</td>\n",
       "      <td>(page, 0.0244358789937)</td>\n",
       "      <td>(department, 0.0237288839327)</td>\n",
       "      <td>(use, 0.023314363269)</td>\n",
       "      <td>(orleans, 0.0183127709217)</td>\n",
       "      <td>(injury, 0.0174977535732)</td>\n",
       "      <td>(may, 0.0166580734835)</td>\n",
       "      <td>(records, 0.0131997374277)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>325695.778</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "      <td>(transportation, 0.164542015423)</td>\n",
       "      <td>(traffic, 0.0382494150922)</td>\n",
       "      <td>(street, 0.0297088546702)</td>\n",
       "      <td>(parking, 0.0260484479646)</td>\n",
       "      <td>(infrastructure, 0.024499661836)</td>\n",
       "      <td>(city, 0.0242484880067)</td>\n",
       "      <td>(safe, 0.0239084069945)</td>\n",
       "      <td>(vehicle, 0.0231351100665)</td>\n",
       "      <td>(streets, 0.0219852292388)</td>\n",
       "      <td>(bike, 0.0155102577616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>26</td>\n",
       "      <td>219560.730</td>\n",
       "      <td>[(public, 0.168381768411), (safety, 0.13998228...</td>\n",
       "      <td>(public, 0.168381768411)</td>\n",
       "      <td>(safety, 0.139982283853)</td>\n",
       "      <td>(crime, 0.0631808672948)</td>\n",
       "      <td>(police, 0.03597164994)</td>\n",
       "      <td>(department, 0.0193424493246)</td>\n",
       "      <td>(illinois, 0.0152118075546)</td>\n",
       "      <td>(race, 0.0133652012852)</td>\n",
       "      <td>(reported, 0.011776909211)</td>\n",
       "      <td>(crimes, 0.0115504690759)</td>\n",
       "      <td>(criminal, 0.0105105785255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>43</td>\n",
       "      <td>208198.844</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "      <td>(state, 0.175638296746)</td>\n",
       "      <td>(new, 0.133403517998)</td>\n",
       "      <td>(michigan, 0.10266572561)</td>\n",
       "      <td>(york, 0.0777270773871)</td>\n",
       "      <td>(information, 0.0292838745522)</td>\n",
       "      <td>(check, 0.0203220904165)</td>\n",
       "      <td>(measurements, 0.0197080431536)</td>\n",
       "      <td>(jobs, 0.0170402721275)</td>\n",
       "      <td>(licensing, 0.0169768940674)</td>\n",
       "      <td>(ny, 0.0167776417369)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>18</td>\n",
       "      <td>188072.809</td>\n",
       "      <td>[(column, 0.0634273088776), (update, 0.0466752...</td>\n",
       "      <td>(column, 0.0634273088776)</td>\n",
       "      <td>(update, 0.0466752365795)</td>\n",
       "      <td>(homeless, 0.0416639506669)</td>\n",
       "      <td>(annually, 0.0300723765516)</td>\n",
       "      <td>(frequency, 0.0292689424963)</td>\n",
       "      <td>(position, 0.021127522302)</td>\n",
       "      <td>(city, 0.019841539727)</td>\n",
       "      <td>(commercial, 0.0184651409887)</td>\n",
       "      <td>(daily, 0.0168690536946)</td>\n",
       "      <td>(salaries, 0.0155892433715)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>48</td>\n",
       "      <td>181698.149</td>\n",
       "      <td>[(business, 0.0803271304597), (license, 0.0323...</td>\n",
       "      <td>(business, 0.0803271304597)</td>\n",
       "      <td>(license, 0.032302648157)</td>\n",
       "      <td>(job, 0.0220026249885)</td>\n",
       "      <td>(contracts, 0.0219341654552)</td>\n",
       "      <td>(businesses, 0.0197708503312)</td>\n",
       "      <td>(vendor, 0.0178329754117)</td>\n",
       "      <td>(owned, 0.0175281633585)</td>\n",
       "      <td>(contract, 0.0175169967387)</td>\n",
       "      <td>(licenses, 0.0169698891758)</td>\n",
       "      <td>(certified, 0.0166884614561)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>24</td>\n",
       "      <td>172835.501</td>\n",
       "      <td>[(official, 0.0610942909105), (account, 0.0497...</td>\n",
       "      <td>(official, 0.0610942909105)</td>\n",
       "      <td>(account, 0.0497660475395)</td>\n",
       "      <td>(accounts, 0.0483014586496)</td>\n",
       "      <td>((openmichigan@michigan, 0.0408207146959)</td>\n",
       "      <td>(special, 0.0291639985893)</td>\n",
       "      <td>(nursing, 0.028770388416)</td>\n",
       "      <td>(software, 0.0268065464462)</td>\n",
       "      <td>(gis, 0.023974220834)</td>\n",
       "      <td>(use, 0.0225286064837)</td>\n",
       "      <td>(consumer, 0.0224473950606)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>29</td>\n",
       "      <td>170112.755</td>\n",
       "      <td>[(planning, 0.0609328856062), (county, 0.05924...</td>\n",
       "      <td>(planning, 0.0609328856062)</td>\n",
       "      <td>(county, 0.0592460462606)</td>\n",
       "      <td>(district, 0.0590822346049)</td>\n",
       "      <td>(districts, 0.0460914886485)</td>\n",
       "      <td>(boundaries, 0.0372114905902)</td>\n",
       "      <td>(city, 0.029524721059)</td>\n",
       "      <td>(areas, 0.0270950065527)</td>\n",
       "      <td>(zoning, 0.0266346423368)</td>\n",
       "      <td>(gis, 0.0244396763289)</td>\n",
       "      <td>(council, 0.0230826924938)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Topic Number  Adjusted_Popularity  \\\n",
       "0            28         16461078.655   \n",
       "1            41           518910.864   \n",
       "2            23           493161.782   \n",
       "3             3           325695.778   \n",
       "4            26           219560.730   \n",
       "5            43           208198.844   \n",
       "6            18           188072.809   \n",
       "7            48           181698.149   \n",
       "8            24           172835.501   \n",
       "9            29           170112.755   \n",
       "\n",
       "                                               Topic  \\\n",
       "0  [(transportation, 0.0493422015093), (plans, 0....   \n",
       "1  [(service, 0.0872627144419), (requests, 0.0350...   \n",
       "2  [(police, 0.0495996521003), (user, 0.032265367...   \n",
       "3  [(transportation, 0.164542015423), (traffic, 0...   \n",
       "4  [(public, 0.168381768411), (safety, 0.13998228...   \n",
       "5  [(state, 0.175638296746), (new, 0.133403517998...   \n",
       "6  [(column, 0.0634273088776), (update, 0.0466752...   \n",
       "7  [(business, 0.0803271304597), (license, 0.0323...   \n",
       "8  [(official, 0.0610942909105), (account, 0.0497...   \n",
       "9  [(planning, 0.0609328856062), (county, 0.05924...   \n",
       "\n",
       "                              topic1                       topic2  \\\n",
       "0  (transportation, 0.0493422015093)     (plans, 0.0463408207807)   \n",
       "1         (service, 0.0872627144419)  (requests, 0.0350316945429)   \n",
       "2          (police, 0.0495996521003)      (user, 0.0322653671005)   \n",
       "3   (transportation, 0.164542015423)   (traffic, 0.0382494150922)   \n",
       "4           (public, 0.168381768411)     (safety, 0.139982283853)   \n",
       "5            (state, 0.175638296746)        (new, 0.133403517998)   \n",
       "6          (column, 0.0634273088776)    (update, 0.0466752365795)   \n",
       "7        (business, 0.0803271304597)    (license, 0.032302648157)   \n",
       "8        (official, 0.0610942909105)   (account, 0.0497660475395)   \n",
       "9        (planning, 0.0609328856062)    (county, 0.0592460462606)   \n",
       "\n",
       "                          topic3                                     topic4  \\\n",
       "0        (iowa, 0.0372400214617)                     (area, 0.031583902009)   \n",
       "1  (inspection, 0.0326789351929)             (inspections, 0.0313503518264)   \n",
       "2   (information, 0.02962646057)                    (page, 0.0244358789937)   \n",
       "3      (street, 0.0297088546702)                 (parking, 0.0260484479646)   \n",
       "4       (crime, 0.0631808672948)                    (police, 0.03597164994)   \n",
       "5      (michigan, 0.10266572561)                    (york, 0.0777270773871)   \n",
       "6    (homeless, 0.0416639506669)                (annually, 0.0300723765516)   \n",
       "7         (job, 0.0220026249885)               (contracts, 0.0219341654552)   \n",
       "8    (accounts, 0.0483014586496)  ((openmichigan@michigan, 0.0408207146959)   \n",
       "9    (district, 0.0590822346049)               (districts, 0.0460914886485)   \n",
       "\n",
       "                             topic5                        topic6  \\\n",
       "0        (transit, 0.0297466527633)  (operations, 0.029650998681)   \n",
       "1            (311, 0.0263119431044)      (public, 0.025968462717)   \n",
       "2     (department, 0.0237288839327)         (use, 0.023314363269)   \n",
       "3  (infrastructure, 0.024499661836)       (city, 0.0242484880067)   \n",
       "4     (department, 0.0193424493246)   (illinois, 0.0152118075546)   \n",
       "5    (information, 0.0292838745522)      (check, 0.0203220904165)   \n",
       "6      (frequency, 0.0292689424963)    (position, 0.021127522302)   \n",
       "7     (businesses, 0.0197708503312)     (vendor, 0.0178329754117)   \n",
       "8        (special, 0.0291639985893)     (nursing, 0.028770388416)   \n",
       "9     (boundaries, 0.0372114905902)        (city, 0.029524721059)   \n",
       "\n",
       "                            topic7                         topic8  \\\n",
       "0        (region, 0.0286308933708)         (bus, 0.0239404049245)   \n",
       "1    (violations, 0.0237687923195)      (request, 0.022587674832)   \n",
       "2       (orleans, 0.0183127709217)      (injury, 0.0174977535732)   \n",
       "3          (safe, 0.0239084069945)     (vehicle, 0.0231351100665)   \n",
       "4          (race, 0.0133652012852)     (reported, 0.011776909211)   \n",
       "5  (measurements, 0.0197080431536)        (jobs, 0.0170402721275)   \n",
       "6           (city, 0.019841539727)  (commercial, 0.0184651409887)   \n",
       "7         (owned, 0.0175281633585)    (contract, 0.0175169967387)   \n",
       "8      (software, 0.0268065464462)          (gis, 0.023974220834)   \n",
       "9         (areas, 0.0270950065527)      (zoning, 0.0266346423368)   \n",
       "\n",
       "                          topic9                       topic10  \n",
       "0    (priority, 0.0197652573751)     (people, 0.0157596265395)  \n",
       "1  (complaints, 0.0193704153109)       (days, 0.0180026003213)  \n",
       "2         (may, 0.0166580734835)    (records, 0.0131997374277)  \n",
       "3     (streets, 0.0219852292388)       (bike, 0.0155102577616)  \n",
       "4      (crimes, 0.0115504690759)   (criminal, 0.0105105785255)  \n",
       "5   (licensing, 0.0169768940674)         (ny, 0.0167776417369)  \n",
       "6       (daily, 0.0168690536946)   (salaries, 0.0155892433715)  \n",
       "7    (licenses, 0.0169698891758)  (certified, 0.0166884614561)  \n",
       "8         (use, 0.0225286064837)   (consumer, 0.0224473950606)  \n",
       "9         (gis, 0.0244396763289)    (council, 0.0230826924938)  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "low_thresh_ranks.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interesting. The calculated scores haven't changed that much, but they have moved by a few thousand in many cases. \n",
    "\n",
    "We can see from this that how you score matters. Let's see how much by setting a very high affinity threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "high = winner_take_all_thresholds(topics, thresh='high')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "high_thresh_ranks = dict_to_df(high)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Number</th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28</td>\n",
       "      <td>15956210.504</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "      <td>(transportation, 0.0493422015093)</td>\n",
       "      <td>(plans, 0.0463408207807)</td>\n",
       "      <td>(iowa, 0.0372400214617)</td>\n",
       "      <td>(area, 0.031583902009)</td>\n",
       "      <td>(transit, 0.0297466527633)</td>\n",
       "      <td>(operations, 0.029650998681)</td>\n",
       "      <td>(region, 0.0286308933708)</td>\n",
       "      <td>(bus, 0.0239404049245)</td>\n",
       "      <td>(priority, 0.0197652573751)</td>\n",
       "      <td>(people, 0.0157596265395)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23</td>\n",
       "      <td>449272.292</td>\n",
       "      <td>[(police, 0.0495996521003), (user, 0.032265367...</td>\n",
       "      <td>(police, 0.0495996521003)</td>\n",
       "      <td>(user, 0.0322653671005)</td>\n",
       "      <td>(information, 0.02962646057)</td>\n",
       "      <td>(page, 0.0244358789937)</td>\n",
       "      <td>(department, 0.0237288839327)</td>\n",
       "      <td>(use, 0.023314363269)</td>\n",
       "      <td>(orleans, 0.0183127709217)</td>\n",
       "      <td>(injury, 0.0174977535732)</td>\n",
       "      <td>(may, 0.0166580734835)</td>\n",
       "      <td>(records, 0.0131997374277)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>41</td>\n",
       "      <td>275563.717</td>\n",
       "      <td>[(service, 0.0872627144419), (requests, 0.0350...</td>\n",
       "      <td>(service, 0.0872627144419)</td>\n",
       "      <td>(requests, 0.0350316945429)</td>\n",
       "      <td>(inspection, 0.0326789351929)</td>\n",
       "      <td>(inspections, 0.0313503518264)</td>\n",
       "      <td>(311, 0.0263119431044)</td>\n",
       "      <td>(public, 0.025968462717)</td>\n",
       "      <td>(violations, 0.0237687923195)</td>\n",
       "      <td>(request, 0.022587674832)</td>\n",
       "      <td>(complaints, 0.0193704153109)</td>\n",
       "      <td>(days, 0.0180026003213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>24</td>\n",
       "      <td>136515.744</td>\n",
       "      <td>[(official, 0.0610942909105), (account, 0.0497...</td>\n",
       "      <td>(official, 0.0610942909105)</td>\n",
       "      <td>(account, 0.0497660475395)</td>\n",
       "      <td>(accounts, 0.0483014586496)</td>\n",
       "      <td>((openmichigan@michigan, 0.0408207146959)</td>\n",
       "      <td>(special, 0.0291639985893)</td>\n",
       "      <td>(nursing, 0.028770388416)</td>\n",
       "      <td>(software, 0.0268065464462)</td>\n",
       "      <td>(gis, 0.023974220834)</td>\n",
       "      <td>(use, 0.0225286064837)</td>\n",
       "      <td>(consumer, 0.0224473950606)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>43</td>\n",
       "      <td>102284.078</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "      <td>(state, 0.175638296746)</td>\n",
       "      <td>(new, 0.133403517998)</td>\n",
       "      <td>(michigan, 0.10266572561)</td>\n",
       "      <td>(york, 0.0777270773871)</td>\n",
       "      <td>(information, 0.0292838745522)</td>\n",
       "      <td>(check, 0.0203220904165)</td>\n",
       "      <td>(measurements, 0.0197080431536)</td>\n",
       "      <td>(jobs, 0.0170402721275)</td>\n",
       "      <td>(licensing, 0.0169768940674)</td>\n",
       "      <td>(ny, 0.0167776417369)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>31</td>\n",
       "      <td>92203.533</td>\n",
       "      <td>[(politics, 0.110698260148), (government, 0.09...</td>\n",
       "      <td>(politics, 0.110698260148)</td>\n",
       "      <td>(government, 0.0983150505736)</td>\n",
       "      <td>(election, 0.0382597777507)</td>\n",
       "      <td>(campaign, 0.0309485620342)</td>\n",
       "      <td>(elections, 0.0296823051034)</td>\n",
       "      <td>(commission, 0.0293149928715)</td>\n",
       "      <td>(results, 0.0249283166716)</td>\n",
       "      <td>(city, 0.0213278569983)</td>\n",
       "      <td>(460, 0.0200805489271)</td>\n",
       "      <td>(finance, 0.0197352160164)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>46</td>\n",
       "      <td>70784.204</td>\n",
       "      <td>[(address, 0.0425142976586), (name, 0.02889526...</td>\n",
       "      <td>(address, 0.0425142976586)</td>\n",
       "      <td>(name, 0.028895265436)</td>\n",
       "      <td>(number, 0.0260959266214)</td>\n",
       "      <td>(facility, 0.0211950547561)</td>\n",
       "      <td>(patient, 0.0187283290738)</td>\n",
       "      <td>(location, 0.0173565095563)</td>\n",
       "      <td>(contains, 0.0147027345652)</td>\n",
       "      <td>(type, 0.0140782421165)</td>\n",
       "      <td>(g, 0.013900974383)</td>\n",
       "      <td>(transfer, 0.0138993318674)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>26</td>\n",
       "      <td>63222.136</td>\n",
       "      <td>[(public, 0.168381768411), (safety, 0.13998228...</td>\n",
       "      <td>(public, 0.168381768411)</td>\n",
       "      <td>(safety, 0.139982283853)</td>\n",
       "      <td>(crime, 0.0631808672948)</td>\n",
       "      <td>(police, 0.03597164994)</td>\n",
       "      <td>(department, 0.0193424493246)</td>\n",
       "      <td>(illinois, 0.0152118075546)</td>\n",
       "      <td>(race, 0.0133652012852)</td>\n",
       "      <td>(reported, 0.011776909211)</td>\n",
       "      <td>(crimes, 0.0115504690759)</td>\n",
       "      <td>(criminal, 0.0105105785255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>32</td>\n",
       "      <td>56775.321</td>\n",
       "      <td>[(open, 0.111361606245), (created, 0.066633337...</td>\n",
       "      <td>(open, 0.111361606245)</td>\n",
       "      <td>(created, 0.0666333370863)</td>\n",
       "      <td>(items, 0.0469751568036)</td>\n",
       "      <td>(clear, 0.0249716560821)</td>\n",
       "      <td>(volume, 0.0175690284296)</td>\n",
       "      <td>(please, 0.0162081895116)</td>\n",
       "      <td>(presented, 0.015509271243)</td>\n",
       "      <td>(difficult, 0.0141524865491)</td>\n",
       "      <td>(potential, 0.013902418106)</td>\n",
       "      <td>(note, 0.0129587647759)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>50</td>\n",
       "      <td>56295.097</td>\n",
       "      <td>[(program, 0.032511463571), (employees, 0.0289...</td>\n",
       "      <td>(program, 0.032511463571)</td>\n",
       "      <td>(employees, 0.0289751722757)</td>\n",
       "      <td>(adult, 0.0244085054991)</td>\n",
       "      <td>(employee, 0.0242968020528)</td>\n",
       "      <td>(annual, 0.0204986126128)</td>\n",
       "      <td>(names, 0.0182179413533)</td>\n",
       "      <td>(full, 0.0179091183965)</td>\n",
       "      <td>(abuse, 0.0162096834129)</td>\n",
       "      <td>(nutrition, 0.0157677124596)</td>\n",
       "      <td>(beginning, 0.0142137708217)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Topic Number  Adjusted_Popularity  \\\n",
       "0            28         15956210.504   \n",
       "1            23           449272.292   \n",
       "2            41           275563.717   \n",
       "3            24           136515.744   \n",
       "4            43           102284.078   \n",
       "5            31            92203.533   \n",
       "6            46            70784.204   \n",
       "7            26            63222.136   \n",
       "8            32            56775.321   \n",
       "9            50            56295.097   \n",
       "\n",
       "                                               Topic  \\\n",
       "0  [(transportation, 0.0493422015093), (plans, 0....   \n",
       "1  [(police, 0.0495996521003), (user, 0.032265367...   \n",
       "2  [(service, 0.0872627144419), (requests, 0.0350...   \n",
       "3  [(official, 0.0610942909105), (account, 0.0497...   \n",
       "4  [(state, 0.175638296746), (new, 0.133403517998...   \n",
       "5  [(politics, 0.110698260148), (government, 0.09...   \n",
       "6  [(address, 0.0425142976586), (name, 0.02889526...   \n",
       "7  [(public, 0.168381768411), (safety, 0.13998228...   \n",
       "8  [(open, 0.111361606245), (created, 0.066633337...   \n",
       "9  [(program, 0.032511463571), (employees, 0.0289...   \n",
       "\n",
       "                              topic1                         topic2  \\\n",
       "0  (transportation, 0.0493422015093)       (plans, 0.0463408207807)   \n",
       "1          (police, 0.0495996521003)        (user, 0.0322653671005)   \n",
       "2         (service, 0.0872627144419)    (requests, 0.0350316945429)   \n",
       "3        (official, 0.0610942909105)     (account, 0.0497660475395)   \n",
       "4            (state, 0.175638296746)          (new, 0.133403517998)   \n",
       "5         (politics, 0.110698260148)  (government, 0.0983150505736)   \n",
       "6         (address, 0.0425142976586)         (name, 0.028895265436)   \n",
       "7           (public, 0.168381768411)       (safety, 0.139982283853)   \n",
       "8             (open, 0.111361606245)     (created, 0.0666333370863)   \n",
       "9          (program, 0.032511463571)   (employees, 0.0289751722757)   \n",
       "\n",
       "                          topic3                                     topic4  \\\n",
       "0        (iowa, 0.0372400214617)                     (area, 0.031583902009)   \n",
       "1   (information, 0.02962646057)                    (page, 0.0244358789937)   \n",
       "2  (inspection, 0.0326789351929)             (inspections, 0.0313503518264)   \n",
       "3    (accounts, 0.0483014586496)  ((openmichigan@michigan, 0.0408207146959)   \n",
       "4      (michigan, 0.10266572561)                    (york, 0.0777270773871)   \n",
       "5    (election, 0.0382597777507)                (campaign, 0.0309485620342)   \n",
       "6      (number, 0.0260959266214)                (facility, 0.0211950547561)   \n",
       "7       (crime, 0.0631808672948)                    (police, 0.03597164994)   \n",
       "8       (items, 0.0469751568036)                   (clear, 0.0249716560821)   \n",
       "9       (adult, 0.0244085054991)                (employee, 0.0242968020528)   \n",
       "\n",
       "                           topic5                         topic6  \\\n",
       "0      (transit, 0.0297466527633)   (operations, 0.029650998681)   \n",
       "1   (department, 0.0237288839327)          (use, 0.023314363269)   \n",
       "2          (311, 0.0263119431044)       (public, 0.025968462717)   \n",
       "3      (special, 0.0291639985893)      (nursing, 0.028770388416)   \n",
       "4  (information, 0.0292838745522)       (check, 0.0203220904165)   \n",
       "5    (elections, 0.0296823051034)  (commission, 0.0293149928715)   \n",
       "6      (patient, 0.0187283290738)    (location, 0.0173565095563)   \n",
       "7   (department, 0.0193424493246)    (illinois, 0.0152118075546)   \n",
       "8       (volume, 0.0175690284296)      (please, 0.0162081895116)   \n",
       "9       (annual, 0.0204986126128)       (names, 0.0182179413533)   \n",
       "\n",
       "                            topic7                        topic8  \\\n",
       "0        (region, 0.0286308933708)        (bus, 0.0239404049245)   \n",
       "1       (orleans, 0.0183127709217)     (injury, 0.0174977535732)   \n",
       "2    (violations, 0.0237687923195)     (request, 0.022587674832)   \n",
       "3      (software, 0.0268065464462)         (gis, 0.023974220834)   \n",
       "4  (measurements, 0.0197080431536)       (jobs, 0.0170402721275)   \n",
       "5       (results, 0.0249283166716)       (city, 0.0213278569983)   \n",
       "6      (contains, 0.0147027345652)       (type, 0.0140782421165)   \n",
       "7          (race, 0.0133652012852)    (reported, 0.011776909211)   \n",
       "8      (presented, 0.015509271243)  (difficult, 0.0141524865491)   \n",
       "9          (full, 0.0179091183965)      (abuse, 0.0162096834129)   \n",
       "\n",
       "                          topic9                       topic10  \n",
       "0    (priority, 0.0197652573751)     (people, 0.0157596265395)  \n",
       "1         (may, 0.0166580734835)    (records, 0.0131997374277)  \n",
       "2  (complaints, 0.0193704153109)       (days, 0.0180026003213)  \n",
       "3         (use, 0.0225286064837)   (consumer, 0.0224473950606)  \n",
       "4   (licensing, 0.0169768940674)         (ny, 0.0167776417369)  \n",
       "5         (460, 0.0200805489271)    (finance, 0.0197352160164)  \n",
       "6            (g, 0.013900974383)   (transfer, 0.0138993318674)  \n",
       "7      (crimes, 0.0115504690759)   (criminal, 0.0105105785255)  \n",
       "8    (potential, 0.013902418106)       (note, 0.0129587647759)  \n",
       "9   (nutrition, 0.0157677124596)  (beginning, 0.0142137708217)  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "high_thresh_ranks.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**This is very interesting. Our main transportation tag still reigns, but the other transportation tag has fallen completely out of the top 10! Our 311 topic tag still hangs in there, but falls to third versus police records. We can draw a few conclusions from this:**\n",
    "- Transportation, 311/service calls, and police records are resilient; they remain popular across calculations.\n",
    "- Other tags shift rankings significantly. We should be careful about drawing sweeping conclusions from ranks 4-10. \n",
    "    - To begin with, we've sacrificed interpretability to reduce variance in our adjusted popularity calculation. We don't want total views to make NYC datasets the \"most popular\". But by summing log views and downloads, and taking a portion of those views, we've lost some ability to interpret absolute differences in our calculations. A human knows the difference between 1000 views and 10 views; what's the difference between 100,095 adjusted popularity and 92,203? It's not intuitive.\n",
    "    - These results don't mean \"release this open dataset 3rd, then this 4th, etc. It's just a guide to a snapshot of Socrata data at a certain point in time. **As noted, however, it is safe to conclude that transportation plans and operations, along with police records and 311 data, are datasets the public accesses frequently.**\n",
    "- Our \"catch all\" tags seem to get more popular with a stricter cut off. I think it's possible that they match parochial datasets very well, but match fewer datasets as a winner overall. Topics that actually tell us something but are more broad may match datasets less well, but match more (at a 0.3-0.4 threshold), therefore losing some credit with a stricter threshold. This is just speculation, however."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's a method of calculating wherein a dataset can get \"partial credit\", as long as it's above a certain threshold.\n",
    "- Datasets get their share of combined log views + downloads, as long as they compose at least 0.2 of a document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def calc_proportional_pop(df):\n",
    "    df = df.fillna(0)\n",
    "    results_dict = {}\n",
    "    \n",
    "    for row_num in df.index:\n",
    "        for tup in df.topic_comp[row_num]: #no need to pick a winner here...\n",
    "            if not tup[0] in results_dict:\n",
    "                if tup[1] >= 0.2: #.loc interprets as the label of the index, not int position\n",
    "                    results_dict[tup[0]] = (tup[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "                else:\n",
    "                    pass\n",
    "            if tup[0] in results_dict:\n",
    "                if tup[1] >= 0.2:\n",
    "                    results_dict[tup[0]] += (tup[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total_log))\n",
    "    return results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "proportions = calc_proportional_pop(topics) #pretty sure .loc makes this slower than using .iloc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "proportional_pop_df = dict_to_df(proportions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Number</th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>28</td>\n",
       "      <td>16478738.222</td>\n",
       "      <td>[(transportation, 0.0493422015093), (plans, 0....</td>\n",
       "      <td>(transportation, 0.0493422015093)</td>\n",
       "      <td>(plans, 0.0463408207807)</td>\n",
       "      <td>(iowa, 0.0372400214617)</td>\n",
       "      <td>(area, 0.031583902009)</td>\n",
       "      <td>(transit, 0.0297466527633)</td>\n",
       "      <td>(operations, 0.029650998681)</td>\n",
       "      <td>(region, 0.0286308933708)</td>\n",
       "      <td>(bus, 0.0239404049245)</td>\n",
       "      <td>(priority, 0.0197652573751)</td>\n",
       "      <td>(people, 0.0157596265395)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>41</td>\n",
       "      <td>586779.754</td>\n",
       "      <td>[(service, 0.0872627144419), (requests, 0.0350...</td>\n",
       "      <td>(service, 0.0872627144419)</td>\n",
       "      <td>(requests, 0.0350316945429)</td>\n",
       "      <td>(inspection, 0.0326789351929)</td>\n",
       "      <td>(inspections, 0.0313503518264)</td>\n",
       "      <td>(311, 0.0263119431044)</td>\n",
       "      <td>(public, 0.025968462717)</td>\n",
       "      <td>(violations, 0.0237687923195)</td>\n",
       "      <td>(request, 0.022587674832)</td>\n",
       "      <td>(complaints, 0.0193704153109)</td>\n",
       "      <td>(days, 0.0180026003213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>23</td>\n",
       "      <td>502600.191</td>\n",
       "      <td>[(police, 0.0495996521003), (user, 0.032265367...</td>\n",
       "      <td>(police, 0.0495996521003)</td>\n",
       "      <td>(user, 0.0322653671005)</td>\n",
       "      <td>(information, 0.02962646057)</td>\n",
       "      <td>(page, 0.0244358789937)</td>\n",
       "      <td>(department, 0.0237288839327)</td>\n",
       "      <td>(use, 0.023314363269)</td>\n",
       "      <td>(orleans, 0.0183127709217)</td>\n",
       "      <td>(injury, 0.0174977535732)</td>\n",
       "      <td>(may, 0.0166580734835)</td>\n",
       "      <td>(records, 0.0131997374277)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>359881.938</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "      <td>(transportation, 0.164542015423)</td>\n",
       "      <td>(traffic, 0.0382494150922)</td>\n",
       "      <td>(street, 0.0297088546702)</td>\n",
       "      <td>(parking, 0.0260484479646)</td>\n",
       "      <td>(infrastructure, 0.024499661836)</td>\n",
       "      <td>(city, 0.0242484880067)</td>\n",
       "      <td>(safe, 0.0239084069945)</td>\n",
       "      <td>(vehicle, 0.0231351100665)</td>\n",
       "      <td>(streets, 0.0219852292388)</td>\n",
       "      <td>(bike, 0.0155102577616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>31</td>\n",
       "      <td>251066.127</td>\n",
       "      <td>[(politics, 0.110698260148), (government, 0.09...</td>\n",
       "      <td>(politics, 0.110698260148)</td>\n",
       "      <td>(government, 0.0983150505736)</td>\n",
       "      <td>(election, 0.0382597777507)</td>\n",
       "      <td>(campaign, 0.0309485620342)</td>\n",
       "      <td>(elections, 0.0296823051034)</td>\n",
       "      <td>(commission, 0.0293149928715)</td>\n",
       "      <td>(results, 0.0249283166716)</td>\n",
       "      <td>(city, 0.0213278569983)</td>\n",
       "      <td>(460, 0.0200805489271)</td>\n",
       "      <td>(finance, 0.0197352160164)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>26</td>\n",
       "      <td>249091.564</td>\n",
       "      <td>[(public, 0.168381768411), (safety, 0.13998228...</td>\n",
       "      <td>(public, 0.168381768411)</td>\n",
       "      <td>(safety, 0.139982283853)</td>\n",
       "      <td>(crime, 0.0631808672948)</td>\n",
       "      <td>(police, 0.03597164994)</td>\n",
       "      <td>(department, 0.0193424493246)</td>\n",
       "      <td>(illinois, 0.0152118075546)</td>\n",
       "      <td>(race, 0.0133652012852)</td>\n",
       "      <td>(reported, 0.011776909211)</td>\n",
       "      <td>(crimes, 0.0115504690759)</td>\n",
       "      <td>(criminal, 0.0105105785255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>43</td>\n",
       "      <td>241684.562</td>\n",
       "      <td>[(state, 0.175638296746), (new, 0.133403517998...</td>\n",
       "      <td>(state, 0.175638296746)</td>\n",
       "      <td>(new, 0.133403517998)</td>\n",
       "      <td>(michigan, 0.10266572561)</td>\n",
       "      <td>(york, 0.0777270773871)</td>\n",
       "      <td>(information, 0.0292838745522)</td>\n",
       "      <td>(check, 0.0203220904165)</td>\n",
       "      <td>(measurements, 0.0197080431536)</td>\n",
       "      <td>(jobs, 0.0170402721275)</td>\n",
       "      <td>(licensing, 0.0169768940674)</td>\n",
       "      <td>(ny, 0.0167776417369)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>29</td>\n",
       "      <td>233415.749</td>\n",
       "      <td>[(planning, 0.0609328856062), (county, 0.05924...</td>\n",
       "      <td>(planning, 0.0609328856062)</td>\n",
       "      <td>(county, 0.0592460462606)</td>\n",
       "      <td>(district, 0.0590822346049)</td>\n",
       "      <td>(districts, 0.0460914886485)</td>\n",
       "      <td>(boundaries, 0.0372114905902)</td>\n",
       "      <td>(city, 0.029524721059)</td>\n",
       "      <td>(areas, 0.0270950065527)</td>\n",
       "      <td>(zoning, 0.0266346423368)</td>\n",
       "      <td>(gis, 0.0244396763289)</td>\n",
       "      <td>(council, 0.0230826924938)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>24</td>\n",
       "      <td>222915.660</td>\n",
       "      <td>[(official, 0.0610942909105), (account, 0.0497...</td>\n",
       "      <td>(official, 0.0610942909105)</td>\n",
       "      <td>(account, 0.0497660475395)</td>\n",
       "      <td>(accounts, 0.0483014586496)</td>\n",
       "      <td>((openmichigan@michigan, 0.0408207146959)</td>\n",
       "      <td>(special, 0.0291639985893)</td>\n",
       "      <td>(nursing, 0.028770388416)</td>\n",
       "      <td>(software, 0.0268065464462)</td>\n",
       "      <td>(gis, 0.023974220834)</td>\n",
       "      <td>(use, 0.0225286064837)</td>\n",
       "      <td>(consumer, 0.0224473950606)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>48</td>\n",
       "      <td>204355.994</td>\n",
       "      <td>[(business, 0.0803271304597), (license, 0.0323...</td>\n",
       "      <td>(business, 0.0803271304597)</td>\n",
       "      <td>(license, 0.032302648157)</td>\n",
       "      <td>(job, 0.0220026249885)</td>\n",
       "      <td>(contracts, 0.0219341654552)</td>\n",
       "      <td>(businesses, 0.0197708503312)</td>\n",
       "      <td>(vendor, 0.0178329754117)</td>\n",
       "      <td>(owned, 0.0175281633585)</td>\n",
       "      <td>(contract, 0.0175169967387)</td>\n",
       "      <td>(licenses, 0.0169698891758)</td>\n",
       "      <td>(certified, 0.0166884614561)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>18</td>\n",
       "      <td>194450.466</td>\n",
       "      <td>[(column, 0.0634273088776), (update, 0.0466752...</td>\n",
       "      <td>(column, 0.0634273088776)</td>\n",
       "      <td>(update, 0.0466752365795)</td>\n",
       "      <td>(homeless, 0.0416639506669)</td>\n",
       "      <td>(annually, 0.0300723765516)</td>\n",
       "      <td>(frequency, 0.0292689424963)</td>\n",
       "      <td>(position, 0.021127522302)</td>\n",
       "      <td>(city, 0.019841539727)</td>\n",
       "      <td>(commercial, 0.0184651409887)</td>\n",
       "      <td>(daily, 0.0168690536946)</td>\n",
       "      <td>(salaries, 0.0155892433715)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>34</td>\n",
       "      <td>179703.511</td>\n",
       "      <td>[(education, 0.181504757587), (school, 0.08185...</td>\n",
       "      <td>(education, 0.181504757587)</td>\n",
       "      <td>(school, 0.0818576925692)</td>\n",
       "      <td>(schools, 0.0317766682176)</td>\n",
       "      <td>(students, 0.0295716293995)</td>\n",
       "      <td>(public, 0.0171734235818)</td>\n",
       "      <td>(governance, 0.0167833173256)</td>\n",
       "      <td>(state, 0.0165785793709)</td>\n",
       "      <td>(year, 0.0161596047936)</td>\n",
       "      <td>(student, 0.0161397342546)</td>\n",
       "      <td>(district, 0.0160036278832)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>37</td>\n",
       "      <td>172488.703</td>\n",
       "      <td>[(financial, 0.0959611006916), (permits, 0.066...</td>\n",
       "      <td>(financial, 0.0959611006916)</td>\n",
       "      <td>(permits, 0.0668632739602)</td>\n",
       "      <td>(building, 0.0631953900546)</td>\n",
       "      <td>(permit, 0.054456521697)</td>\n",
       "      <td>(expenditures, 0.0429247389481)</td>\n",
       "      <td>(guide, 0.0250580534259)</td>\n",
       "      <td>(issued, 0.0232434329642)</td>\n",
       "      <td>(construction, 0.0228846174877)</td>\n",
       "      <td>(information, 0.0227968624179)</td>\n",
       "      <td>(filter, 0.021954851646)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>49</td>\n",
       "      <td>149059.120</td>\n",
       "      <td>[(development, 0.176027756751), (housing, 0.15...</td>\n",
       "      <td>(development, 0.176027756751)</td>\n",
       "      <td>(housing, 0.152212657649)</td>\n",
       "      <td>(economic, 0.0588541094432)</td>\n",
       "      <td>(community, 0.0485919430535)</td>\n",
       "      <td>(infrastructure, 0.0190401252932)</td>\n",
       "      <td>(managed, 0.0183046960354)</td>\n",
       "      <td>(medicaid, 0.0178764063515)</td>\n",
       "      <td>(department, 0.0140238290886)</td>\n",
       "      <td>(buildings, 0.0137754059922)</td>\n",
       "      <td>(economy, 0.0123473943134)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>46</td>\n",
       "      <td>148932.095</td>\n",
       "      <td>[(address, 0.0425142976586), (name, 0.02889526...</td>\n",
       "      <td>(address, 0.0425142976586)</td>\n",
       "      <td>(name, 0.028895265436)</td>\n",
       "      <td>(number, 0.0260959266214)</td>\n",
       "      <td>(facility, 0.0211950547561)</td>\n",
       "      <td>(patient, 0.0187283290738)</td>\n",
       "      <td>(location, 0.0173565095563)</td>\n",
       "      <td>(contains, 0.0147027345652)</td>\n",
       "      <td>(type, 0.0140782421165)</td>\n",
       "      <td>(g, 0.013900974383)</td>\n",
       "      <td>(transfer, 0.0138993318674)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Topic Number  Adjusted_Popularity  \\\n",
       "0             28         16478738.222   \n",
       "1             41           586779.754   \n",
       "2             23           502600.191   \n",
       "3              3           359881.938   \n",
       "4             31           251066.127   \n",
       "5             26           249091.564   \n",
       "6             43           241684.562   \n",
       "7             29           233415.749   \n",
       "8             24           222915.660   \n",
       "9             48           204355.994   \n",
       "10            18           194450.466   \n",
       "11            34           179703.511   \n",
       "12            37           172488.703   \n",
       "13            49           149059.120   \n",
       "14            46           148932.095   \n",
       "\n",
       "                                                Topic  \\\n",
       "0   [(transportation, 0.0493422015093), (plans, 0....   \n",
       "1   [(service, 0.0872627144419), (requests, 0.0350...   \n",
       "2   [(police, 0.0495996521003), (user, 0.032265367...   \n",
       "3   [(transportation, 0.164542015423), (traffic, 0...   \n",
       "4   [(politics, 0.110698260148), (government, 0.09...   \n",
       "5   [(public, 0.168381768411), (safety, 0.13998228...   \n",
       "6   [(state, 0.175638296746), (new, 0.133403517998...   \n",
       "7   [(planning, 0.0609328856062), (county, 0.05924...   \n",
       "8   [(official, 0.0610942909105), (account, 0.0497...   \n",
       "9   [(business, 0.0803271304597), (license, 0.0323...   \n",
       "10  [(column, 0.0634273088776), (update, 0.0466752...   \n",
       "11  [(education, 0.181504757587), (school, 0.08185...   \n",
       "12  [(financial, 0.0959611006916), (permits, 0.066...   \n",
       "13  [(development, 0.176027756751), (housing, 0.15...   \n",
       "14  [(address, 0.0425142976586), (name, 0.02889526...   \n",
       "\n",
       "                               topic1                         topic2  \\\n",
       "0   (transportation, 0.0493422015093)       (plans, 0.0463408207807)   \n",
       "1          (service, 0.0872627144419)    (requests, 0.0350316945429)   \n",
       "2           (police, 0.0495996521003)        (user, 0.0322653671005)   \n",
       "3    (transportation, 0.164542015423)     (traffic, 0.0382494150922)   \n",
       "4          (politics, 0.110698260148)  (government, 0.0983150505736)   \n",
       "5            (public, 0.168381768411)       (safety, 0.139982283853)   \n",
       "6             (state, 0.175638296746)          (new, 0.133403517998)   \n",
       "7         (planning, 0.0609328856062)      (county, 0.0592460462606)   \n",
       "8         (official, 0.0610942909105)     (account, 0.0497660475395)   \n",
       "9         (business, 0.0803271304597)      (license, 0.032302648157)   \n",
       "10          (column, 0.0634273088776)      (update, 0.0466752365795)   \n",
       "11        (education, 0.181504757587)      (school, 0.0818576925692)   \n",
       "12       (financial, 0.0959611006916)     (permits, 0.0668632739602)   \n",
       "13      (development, 0.176027756751)      (housing, 0.152212657649)   \n",
       "14         (address, 0.0425142976586)         (name, 0.028895265436)   \n",
       "\n",
       "                           topic3                                     topic4  \\\n",
       "0         (iowa, 0.0372400214617)                     (area, 0.031583902009)   \n",
       "1   (inspection, 0.0326789351929)             (inspections, 0.0313503518264)   \n",
       "2    (information, 0.02962646057)                    (page, 0.0244358789937)   \n",
       "3       (street, 0.0297088546702)                 (parking, 0.0260484479646)   \n",
       "4     (election, 0.0382597777507)                (campaign, 0.0309485620342)   \n",
       "5        (crime, 0.0631808672948)                    (police, 0.03597164994)   \n",
       "6       (michigan, 0.10266572561)                    (york, 0.0777270773871)   \n",
       "7     (district, 0.0590822346049)               (districts, 0.0460914886485)   \n",
       "8     (accounts, 0.0483014586496)  ((openmichigan@michigan, 0.0408207146959)   \n",
       "9          (job, 0.0220026249885)               (contracts, 0.0219341654552)   \n",
       "10    (homeless, 0.0416639506669)                (annually, 0.0300723765516)   \n",
       "11     (schools, 0.0317766682176)                (students, 0.0295716293995)   \n",
       "12    (building, 0.0631953900546)                   (permit, 0.054456521697)   \n",
       "13    (economic, 0.0588541094432)               (community, 0.0485919430535)   \n",
       "14      (number, 0.0260959266214)                (facility, 0.0211950547561)   \n",
       "\n",
       "                               topic5                         topic6  \\\n",
       "0          (transit, 0.0297466527633)   (operations, 0.029650998681)   \n",
       "1              (311, 0.0263119431044)       (public, 0.025968462717)   \n",
       "2       (department, 0.0237288839327)          (use, 0.023314363269)   \n",
       "3    (infrastructure, 0.024499661836)        (city, 0.0242484880067)   \n",
       "4        (elections, 0.0296823051034)  (commission, 0.0293149928715)   \n",
       "5       (department, 0.0193424493246)    (illinois, 0.0152118075546)   \n",
       "6      (information, 0.0292838745522)       (check, 0.0203220904165)   \n",
       "7       (boundaries, 0.0372114905902)         (city, 0.029524721059)   \n",
       "8          (special, 0.0291639985893)      (nursing, 0.028770388416)   \n",
       "9       (businesses, 0.0197708503312)      (vendor, 0.0178329754117)   \n",
       "10       (frequency, 0.0292689424963)     (position, 0.021127522302)   \n",
       "11          (public, 0.0171734235818)  (governance, 0.0167833173256)   \n",
       "12    (expenditures, 0.0429247389481)       (guide, 0.0250580534259)   \n",
       "13  (infrastructure, 0.0190401252932)     (managed, 0.0183046960354)   \n",
       "14         (patient, 0.0187283290738)    (location, 0.0173565095563)   \n",
       "\n",
       "                             topic7                           topic8  \\\n",
       "0         (region, 0.0286308933708)           (bus, 0.0239404049245)   \n",
       "1     (violations, 0.0237687923195)        (request, 0.022587674832)   \n",
       "2        (orleans, 0.0183127709217)        (injury, 0.0174977535732)   \n",
       "3           (safe, 0.0239084069945)       (vehicle, 0.0231351100665)   \n",
       "4        (results, 0.0249283166716)          (city, 0.0213278569983)   \n",
       "5           (race, 0.0133652012852)       (reported, 0.011776909211)   \n",
       "6   (measurements, 0.0197080431536)          (jobs, 0.0170402721275)   \n",
       "7          (areas, 0.0270950065527)        (zoning, 0.0266346423368)   \n",
       "8       (software, 0.0268065464462)            (gis, 0.023974220834)   \n",
       "9          (owned, 0.0175281633585)      (contract, 0.0175169967387)   \n",
       "10           (city, 0.019841539727)    (commercial, 0.0184651409887)   \n",
       "11         (state, 0.0165785793709)          (year, 0.0161596047936)   \n",
       "12        (issued, 0.0232434329642)  (construction, 0.0228846174877)   \n",
       "13      (medicaid, 0.0178764063515)    (department, 0.0140238290886)   \n",
       "14      (contains, 0.0147027345652)          (type, 0.0140782421165)   \n",
       "\n",
       "                            topic9                       topic10  \n",
       "0      (priority, 0.0197652573751)     (people, 0.0157596265395)  \n",
       "1    (complaints, 0.0193704153109)       (days, 0.0180026003213)  \n",
       "2           (may, 0.0166580734835)    (records, 0.0131997374277)  \n",
       "3       (streets, 0.0219852292388)       (bike, 0.0155102577616)  \n",
       "4           (460, 0.0200805489271)    (finance, 0.0197352160164)  \n",
       "5        (crimes, 0.0115504690759)   (criminal, 0.0105105785255)  \n",
       "6     (licensing, 0.0169768940674)         (ny, 0.0167776417369)  \n",
       "7           (gis, 0.0244396763289)    (council, 0.0230826924938)  \n",
       "8           (use, 0.0225286064837)   (consumer, 0.0224473950606)  \n",
       "9      (licenses, 0.0169698891758)  (certified, 0.0166884614561)  \n",
       "10        (daily, 0.0168690536946)   (salaries, 0.0155892433715)  \n",
       "11      (student, 0.0161397342546)   (district, 0.0160036278832)  \n",
       "12  (information, 0.0227968624179)      (filter, 0.021954851646)  \n",
       "13    (buildings, 0.0137754059922)    (economy, 0.0123473943134)  \n",
       "14             (g, 0.013900974383)   (transfer, 0.0138993318674)  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "proportional_pop_df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### These are intuitive results. Our top 6 topics are clean and interpretable, and pass a gut-check in terms of what citizens might search and view.\n",
    "- However, there is really not much separating topics 5-10 (again, small differences in magnitude in our adjusted popularity metric don't tell us much).\n",
    "- Transportation, police records, and 311 are still atop the heap. **However, our second transportation category comes all the way back to #4;** I'd bet that this is because it's often second place to the more popular transportation topic in the winner-take-all format.\n",
    "- I would argue this is the best format for scoring; **some datasets really do, to the human eye, fit into two (or three or four!) topic tags**. This also dilutes model error in which the \"winner\" doesn't really match the dataset according to the human eye; a dataset that does can still get partial credit.\n",
    "- However, you can make an argument for only winner topics of high affinity. This type of probabilistic analysis is always going to have assumptions - it's just important to note them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Top Topic Tags by Domain\n",
    "- Let's see if the most popular topics vary by city/state. We can use \"domain\" -- the web address of an open data portal -- to sort them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def popularity_by_domain(df, domain_name):\n",
    "    \n",
    "    domain_df = df[df.domain == domain_name]\n",
    "    popularity_dict = calc_proportional_pop(domain_df) #we're still using our proportional scoring here\n",
    "    \n",
    "    df_pop = pd.DataFrame.from_dict(popularity_dict, orient='index') #dict is in df but index numbers (aka topic numbers) are random\n",
    "    df_pop = df_pop.rename(index=int, columns={0:\"Adjusted_Popularity\"})\n",
    "    \n",
    "    topic_words_list = []\n",
    "    for topic_num in df_pop.index: #this iterates in the right order - index numbers have real meaning\n",
    "        topic_tups = lda_52_sixty.show_topic(topic_num)\n",
    "        raw_words = list(i[0] for i in lda_52_sixty.show_topic(topic_num))\n",
    "        string = \", \".join(raw_words)\n",
    "        topic_words_list.append(string) #plug in names by index numbers\n",
    "    \n",
    "    df_pop['Topic_Words'] = topic_words_list\n",
    "    \n",
    "    df_pop = df_pop.sort_values(by='Adjusted_Popularity', ascending=False)\n",
    "    \n",
    "    #put the domain in each row - kind of an ugly way to preserve info about domain but it's what we've got\n",
    "    domain_string = domain_name\n",
    "    df_pop['Domain'] = [domain_string for number in range(len(df_pop.index))]\n",
    "    \n",
    "    return df_pop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "madison = popularity_by_domain(topics, \"data.cityofmadison.com\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic_Words</th>\n",
       "      <th>Domain</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>3925.956</td>\n",
       "      <td>politics, government, election, campaign, elec...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>3087.734</td>\n",
       "      <td>public, safety, fire, month, calls, police, em...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>2661.438</td>\n",
       "      <td>public, safety, crime, police, department, ill...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>2649.683</td>\n",
       "      <td>development, housing, economic, community, inf...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1713.837</td>\n",
       "      <td>prevention, hospital, ny, healthy, statewide, ...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1133.078</td>\n",
       "      <td>map, \"about\", shows, details, tracking, click,...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>832.791</td>\n",
       "      <td>information, list, code, required, political, ...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>661.705</td>\n",
       "      <td>chart, children, income, historic, home, assis...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>559.380</td>\n",
       "      <td>recreation, parks, jersey, park, center, neigh...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>391.758</td>\n",
       "      <td>service, requests, inspection, inspections, 31...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>347.438</td>\n",
       "      <td>arts, grants, live, county, includes, library,...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>332.834</td>\n",
       "      <td>transportation, traffic, street, parking, infr...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>268.104</td>\n",
       "      <td>address, name, number, facility, patient, loca...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>258.333</td>\n",
       "      <td>water, environment, protection, tx, filed, was...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>245.204</td>\n",
       "      <td>planning, county, district, districts, boundar...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>225.303</td>\n",
       "      <td>services, social, human, dfps, information, pr...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>225.303</td>\n",
       "      <td>health, san, county, care, counties, system, f...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>209.611</td>\n",
       "      <td>report, performance, city, goal, resultsnola, ...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>181.872</td>\n",
       "      <td>mateo, city, gis, land, county, parcel, map, p...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>174.766</td>\n",
       "      <td>tax, revenue, sales, taxes, local, colorado, g...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>156.966</td>\n",
       "      <td>locations, bacteria, hours, culture, county, l...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>97.956</td>\n",
       "      <td>transportation, plans, iowa, area, transit, op...</td>\n",
       "      <td>data.cityofmadison.com</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Adjusted_Popularity                                        Topic_Words  \\\n",
       "31             3925.956  politics, government, election, campaign, elec...   \n",
       "21             3087.734  public, safety, fire, month, calls, police, em...   \n",
       "26             2661.438  public, safety, crime, police, department, ill...   \n",
       "49             2649.683  development, housing, economic, community, inf...   \n",
       "0              1713.837  prevention, hospital, ny, healthy, statewide, ...   \n",
       "12             1133.078  map, \"about\", shows, details, tracking, click,...   \n",
       "42              832.791  information, list, code, required, political, ...   \n",
       "1               661.705  chart, children, income, historic, home, assis...   \n",
       "9               559.380  recreation, parks, jersey, park, center, neigh...   \n",
       "41              391.758  service, requests, inspection, inspections, 31...   \n",
       "22              347.438  arts, grants, live, county, includes, library,...   \n",
       "3               332.834  transportation, traffic, street, parking, infr...   \n",
       "46              268.104  address, name, number, facility, patient, loca...   \n",
       "27              258.333  water, environment, protection, tx, filed, was...   \n",
       "29              245.204  planning, county, district, districts, boundar...   \n",
       "17              225.303  services, social, human, dfps, information, pr...   \n",
       "39              225.303  health, san, county, care, counties, system, f...   \n",
       "45              209.611  report, performance, city, goal, resultsnola, ...   \n",
       "30              181.872  mateo, city, gis, land, county, parcel, map, p...   \n",
       "25              174.766  tax, revenue, sales, taxes, local, colorado, g...   \n",
       "5               156.966  locations, bacteria, hours, culture, county, l...   \n",
       "28               97.956  transportation, plans, iowa, area, transit, op...   \n",
       "\n",
       "                    Domain  \n",
       "31  data.cityofmadison.com  \n",
       "21  data.cityofmadison.com  \n",
       "26  data.cityofmadison.com  \n",
       "49  data.cityofmadison.com  \n",
       "0   data.cityofmadison.com  \n",
       "12  data.cityofmadison.com  \n",
       "42  data.cityofmadison.com  \n",
       "1   data.cityofmadison.com  \n",
       "9   data.cityofmadison.com  \n",
       "41  data.cityofmadison.com  \n",
       "22  data.cityofmadison.com  \n",
       "3   data.cityofmadison.com  \n",
       "46  data.cityofmadison.com  \n",
       "27  data.cityofmadison.com  \n",
       "29  data.cityofmadison.com  \n",
       "17  data.cityofmadison.com  \n",
       "39  data.cityofmadison.com  \n",
       "45  data.cityofmadison.com  \n",
       "30  data.cityofmadison.com  \n",
       "25  data.cityofmadison.com  \n",
       "5   data.cityofmadison.com  \n",
       "28  data.cityofmadison.com  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "madison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Interesting! By our log measures, politics is the most popular topic in Madison. Makes sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally, by this scoring, let's return all topics that formed a top topic and the city's they were top in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def top_topics_by_domain(df, domains_series):\n",
    "    \n",
    "    domains_dict = {}\n",
    "    for domain in domains_series:\n",
    "        \n",
    "        domain_df = popularity_by_domain(df, domain) #this is a df like the Madison example\n",
    "        Topic = domain_df.iloc[0].Topic_Words #string\n",
    "        Domain = domain_df.iloc[0].Domain.split() #have to remember to split(), which really just gives us one word\n",
    "        \n",
    "        if Topic in domains_dict: \n",
    "            domains_dict[Topic] += Domain\n",
    "        else:\n",
    "            domains_dict[Topic] = list(Domain)\n",
    "    \n",
    "    return domains_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(This takes a bit to run since we are slicing df's for 140+ domains and iterating through them. There's probably a better way.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_domains = topics.domain.unique()\n",
    "top_topic_domains = top_topics_by_domain(topics, domains_series=all_domains)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities': ['data.smcgov.org',\n",
       "  'data.tompsc.com',\n",
       "  'data.topeka.org',\n",
       "  'data.vbgov.com',\n",
       "  'information.stpaul.gov',\n",
       "  'performance.ci.janesville.wi.us'],\n",
       " 'business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified': ['data.auburnwa.gov',\n",
       "  'data.cityofgainesville.org',\n",
       "  'data.culvercity.org',\n",
       "  'data.hampton.gov',\n",
       "  'data.oregon.gov',\n",
       "  'data.oxnard.org',\n",
       "  'opendata.cityofhenderson.com',\n",
       "  'opendata.lasvegasnevada.gov'],\n",
       " 'care, finance, provider, child, cost, plan, day, bay, government, fee': ['data.detroitmi.gov'],\n",
       " 'census, indicators, survey, includes, indicator, community, demographics, bureau, u, level': ['data.livewellsd.org'],\n",
       " 'chart, children, income, historic, home, assistance, homes, families, low, pay': ['data.richmondgov.com'],\n",
       " 'column, update, homeless, annually, frequency, position, city, commercial, daily, salaries': ['data.cityofwestsacramento.org',\n",
       "  'data.douglas.co.us'],\n",
       " 'death, maryland, database, deaths, office, department, new, cdc, metadata, citizens': ['data.maryland.gov'],\n",
       " 'development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy': ['dashboard.alexandriava.gov',\n",
       "  'data.maine.gov',\n",
       "  'insights.cincinnati-oh.gov'],\n",
       " 'economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal': ['data.datamontana.us',\n",
       "  'data.hawaii.gov',\n",
       "  'data.iowa.gov'],\n",
       " 'education, school, schools, students, public, governance, state, year, student, district': ['dashboard.hawaii.gov',\n",
       "  'dashboard.slco.org',\n",
       "  'data.mass.gov',\n",
       "  'data.pa.gov',\n",
       "  'data.vermont.gov',\n",
       "  'impact.stlouisco.com'],\n",
       " 'energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities': ['data.monroecounty-fl.gov',\n",
       "  'data.results.wa.gov',\n",
       "  'data.sustainablesm.org',\n",
       "  'performance.providenceri.gov'],\n",
       " 'finance, year, state, monthly, fiscal, payments, government, june, july, report': ['data.cstx.gov',\n",
       "  'data.nj.gov',\n",
       "  'data.oaklandnet.com',\n",
       "  'datacatalog.cookcountyil.gov',\n",
       "  'opencheckbook.data.somervillema.gov',\n",
       "  'performance.cookcountyil.gov',\n",
       "  'transparency.michigan.gov'],\n",
       " 'financial, permits, building, permit, expenditures, guide, issued, construction, information, filter': ['data.brla.gov',\n",
       "  'data.burlingtonvt.gov',\n",
       "  'data.menlopark.org',\n",
       "  'data.montgomeryal.gov',\n",
       "  'data.nashville.gov',\n",
       "  'data.seattle.gov',\n",
       "  'data.tuscaloosa.com'],\n",
       " 'health, disease, 000, per, rate, rates, chronic, risk, agency, due': ['healthdata.nj.gov'],\n",
       " 'health, san, county, care, counties, system, facilities, network, home, francisco': ['data.mo.gov',\n",
       "  'health.data.ny.gov'],\n",
       " 'information, tab, projects, concerning, may, contains, project, provided, additional, improvement': ['data.countyofriverside.us',\n",
       "  'data.lcb.wa.gov',\n",
       "  'data.qac.org'],\n",
       " 'infrastructure, city, public, maps, works, dallas, los, angeles, types, way': ['data.albanyny.gov',\n",
       "  'data.cityofboston.gov',\n",
       "  'data.cityofevanston.org',\n",
       "  'data.jacksonms.gov',\n",
       "  'data.weho.org'],\n",
       " 'mateo, city, gis, land, county, parcel, map, point, survey, feature': ['data.austintexas.gov'],\n",
       " 'measures, quality, performance, government, life, developed, fair, measurement, definitions, index': ['performance.fultoncountyga.gov',\n",
       "  'performance.smcgov.org',\n",
       "  'performance.topeka.org'],\n",
       " 'plan, strategic, master, community, fewer, limitations, registry, cause, years, information': ['data.cincinnati-oh.gov'],\n",
       " 'planning, county, district, districts, boundaries, city, areas, zoning, gis, council': ['data.acgov.org',\n",
       "  'data.lacity.org',\n",
       "  'performance.franklintn.gov',\n",
       "  'tni.princegeorgescountymd.gov'],\n",
       " 'police, user, information, page, department, use, orleans, injury, may, records': ['data.cityofchicago.org',\n",
       "  'data.hartford.gov',\n",
       "  'data.nola.gov',\n",
       "  'data.srcity.org',\n",
       "  'data.urbanaillinois.us',\n",
       "  'stat.cityofgainesville.org',\n",
       "  'www.dallasopendata.com'],\n",
       " 'politics, government, election, campaign, elections, commission, results, city, 460, finance': ['dashboard.plano.gov',\n",
       "  'data.cityofmadison.com',\n",
       "  'data.kingcounty.gov',\n",
       "  'data.macoupincountyil.gov',\n",
       "  'data.redmond.gov',\n",
       "  'datapoints.dallascityhall.com',\n",
       "  'electionsdata.kingcounty.gov',\n",
       "  'opendata.cityofmesquite.com',\n",
       "  'performance.adcogov.org',\n",
       "  'performance.glendaleca.gov',\n",
       "  'performance.princegeorgescountymd.gov',\n",
       "  'performance.seattle.gov',\n",
       "  'performance.vbgov.com',\n",
       "  'policeview.johnscreekga.gov'],\n",
       " 'population, county, demographics, age, total, king, washington, montgomery, number, years': ['data.illinois.gov',\n",
       "  'performance.carson.org'],\n",
       " 'public, safety, crime, police, department, illinois, race, reported, crimes, criminal': ['data.fortworthtexas.gov',\n",
       "  'data.greensboro-nc.gov',\n",
       "  'data.honolulu.gov',\n",
       "  'data.lacounty.gov',\n",
       "  'data.littlerock.gov',\n",
       "  'data.providenceri.gov',\n",
       "  'data.readingpa.gov',\n",
       "  'data.roseville.ca.us',\n",
       "  'data.sonomacounty.ca.gov',\n",
       "  'data.wa.gov',\n",
       "  'kcstat.kcmo.org',\n",
       "  'opencity.fcgov.com',\n",
       "  'opendata.miamidade.gov',\n",
       "  'opendata.utah.gov',\n",
       "  'performance.cityofdavenportiowa.com',\n",
       "  'performance.danvilleva.gov'],\n",
       " 'public, safety, fire, month, calls, police, emergency, response, discharge, incident': ['data.marincounty.org',\n",
       "  'data.montgomerycountymd.gov',\n",
       "  'data.naperville.il.us',\n",
       "  'data.raleighnc.gov',\n",
       "  'opendata.ci.richmond.ca.us',\n",
       "  'opendata.howardcountymd.gov',\n",
       "  'performance.cityofrc.us',\n",
       "  'stats.cityofcarrollton.com'],\n",
       " 'rates, calculated, rate, year, counts, average, seattle, percent, mortality, number': ['outcomestat.baltimorecity.gov'],\n",
       " 'recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access': ['performance.chattanooga.gov'],\n",
       " \"report, performance, city, goal, resultsnola, administration, city's, management, key, topeka\": ['performance.austintexas.gov',\n",
       "  'performance.williamsburgva.gov'],\n",
       " 'service, requests, inspection, inspections, 311, public, violations, request, complaints, days': ['data.cityofnewyork.us',\n",
       "  'data.delaware.gov',\n",
       "  'data.kcmo.org',\n",
       "  'data.princegeorgescountymd.gov'],\n",
       " 'state, new, michigan, york, information, check, measurements, jobs, licensing, ny': ['data.baltimorecity.gov',\n",
       "  'data.michigan.gov',\n",
       "  'data.ny.gov',\n",
       "  'midashboard.michigan.gov'],\n",
       " 'tax, revenue, sales, taxes, local, colorado, government, state, department, sale': ['data.colorado.gov',\n",
       "  'data.ct.gov'],\n",
       " 'transportation, plans, iowa, area, transit, operations, region, bus, priority, people': ['data.cityofgp.com',\n",
       "  'data.texas.gov',\n",
       "  'data.vta.org'],\n",
       " 'transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike': ['data.cambridgema.gov',\n",
       "  'data.cityoftacoma.org',\n",
       "  'data.muni.org',\n",
       "  'data.rutlandcitypolice.com',\n",
       "  'data.sfgov.org',\n",
       "  'data.somervillema.gov',\n",
       "  'reports.data.montgomerycountymd.gov']}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_topic_domains"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have some additional insights into our top rankings. **Notice that our top topic, \"transportation planning and operations\" is only the most popular dataset in Texas and a California transit agency (as well as the smaller City of Grand Prairie).** This makes sense -- people in big populated places probably care alot about transportation projects. \n",
    "\n",
    "This topic is also definitely getting some weight from the fact that Texas and California have high populations and presumably high open data access rates. However, this is why we took the log of views; to dilute the effects of high population.\n",
    "\n",
    "### Still, this has us thinking -- let's see if we get different rankings if we sum a topics popularity within *each* city/state, and then log this sum.\n",
    "- This will further reduce the weight of city/state open data portals that just get a high number of views and downloads (presumably) by virtue of being a high-population area."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One more scoring setting: Normalize Metrics within a City/State \n",
    "## (Dampened Popularity):\n",
    "- Each city/state/organization (as identified by domain) has topics that \"win\" for each dataset.\n",
    "- Add up each's topics \"spoils\" (total views and downloads) within a given city. THEN take the log of that.\n",
    "- Then add up all log totals for each topic across cities.\n",
    "- This is an extra control for big cities with lots of users skewing our metrics. **We'll call it dampened popularity.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from collections import Counter #need this to sync dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def popularity_normalized_by_area(df, domains_series):\n",
    "    \n",
    "    df = df.fillna(0) #there are 369 datasets with NaN download counts (no missing view counts)\n",
    "    #let's assume they're NaN because there were no downloads\n",
    "    \n",
    "    list_of_domain_dicts = []\n",
    "    popularity_dict = {}\n",
    "    \n",
    "    for domain in domains_series:\n",
    "        \n",
    "        results_dict = {}\n",
    "        \n",
    "        #get our df only of rows from a given city/state domain\n",
    "        domain_df = df[df.domain == domain]\n",
    "        \n",
    "        for row_num in domain_df.index:\n",
    "            tup_list = domain_df.topic_comp[row_num] #list of (topic, doc composition) tuples\n",
    "        \n",
    "            #neat little trick to return only the tuple w/highest index[1] value\n",
    "            winner_tuple = max(tup_list, key=lambda item:item[1])  \n",
    "            \n",
    "            if not winner_tuple[0] in results_dict: #if not in dict, add it with its TOTAL VIEWS score\n",
    "                if winner_tuple[1] > 0.2:\n",
    "                    results_dict[winner_tuple[0]] = (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total))\n",
    "            else:\n",
    "                pass\n",
    "        \n",
    "            if winner_tuple[0] in results_dict: #if in dict, increment that key's value with score\n",
    "                if winner_tuple[1] > 0.2:\n",
    "                    results_dict[winner_tuple[0]] += (winner_tuple[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total))\n",
    "            else:\n",
    "                pass\n",
    "            \n",
    "        \n",
    "        #when loop of domain_df is finished, take log of all keys in dict\n",
    "        log_dict = {}\n",
    "        for k,v in results_dict.items():\n",
    "            if v != 0: #have to do this since you can't take a log of 0; some \"winners\" get datasets with 0 views\n",
    "                log_dict[k] = np.log(v)\n",
    "        \n",
    "        #now we have a polished dict of topic numbers as keys and log of all views/DLs as values; append it to list\n",
    "        list_of_domain_dicts.append(log_dict)\n",
    "    \n",
    "    #use Counter() object to sync our dictionaries\n",
    "    c = Counter()\n",
    "    for d in list_of_domain_dicts:\n",
    "        c.update(d)\n",
    "    \n",
    "    popularity_dict = dict(c)\n",
    "    \n",
    "    return popularity_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "all_domains = topics.domain.unique()\n",
    "pop_norm_area_dict = popularity_normalized_by_area(topics, all_domains)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#pop_norm_area_dict #uncomment just to ensure topic numbers align with topic tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#lda_52_sixty.show_topics(num_topics=52, formatted=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Number</th>\n",
       "      <th>Adjusted_Popularity</th>\n",
       "      <th>Topic</th>\n",
       "      <th>topic1</th>\n",
       "      <th>topic2</th>\n",
       "      <th>topic3</th>\n",
       "      <th>topic4</th>\n",
       "      <th>topic5</th>\n",
       "      <th>topic6</th>\n",
       "      <th>topic7</th>\n",
       "      <th>topic8</th>\n",
       "      <th>topic9</th>\n",
       "      <th>topic10</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>26</td>\n",
       "      <td>799.492</td>\n",
       "      <td>[(public, 0.168381768411), (safety, 0.13998228...</td>\n",
       "      <td>(public, 0.168381768411)</td>\n",
       "      <td>(safety, 0.139982283853)</td>\n",
       "      <td>(crime, 0.0631808672948)</td>\n",
       "      <td>(police, 0.03597164994)</td>\n",
       "      <td>(department, 0.0193424493246)</td>\n",
       "      <td>(illinois, 0.0152118075546)</td>\n",
       "      <td>(race, 0.0133652012852)</td>\n",
       "      <td>(reported, 0.011776909211)</td>\n",
       "      <td>(crimes, 0.0115504690759)</td>\n",
       "      <td>(criminal, 0.0105105785255)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>692.631</td>\n",
       "      <td>[(transportation, 0.164542015423), (traffic, 0...</td>\n",
       "      <td>(transportation, 0.164542015423)</td>\n",
       "      <td>(traffic, 0.0382494150922)</td>\n",
       "      <td>(street, 0.0297088546702)</td>\n",
       "      <td>(parking, 0.0260484479646)</td>\n",
       "      <td>(infrastructure, 0.024499661836)</td>\n",
       "      <td>(city, 0.0242484880067)</td>\n",
       "      <td>(safe, 0.0239084069945)</td>\n",
       "      <td>(vehicle, 0.0231351100665)</td>\n",
       "      <td>(streets, 0.0219852292388)</td>\n",
       "      <td>(bike, 0.0155102577616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21</td>\n",
       "      <td>670.051</td>\n",
       "      <td>[(public, 0.0713314310777), (safety, 0.0690276...</td>\n",
       "      <td>(public, 0.0713314310777)</td>\n",
       "      <td>(safety, 0.0690276616473)</td>\n",
       "      <td>(fire, 0.0477503790148)</td>\n",
       "      <td>(month, 0.0205819878206)</td>\n",
       "      <td>(calls, 0.0173756495075)</td>\n",
       "      <td>(police, 0.0170781244666)</td>\n",
       "      <td>(emergency, 0.0165221049163)</td>\n",
       "      <td>(response, 0.0157891037516)</td>\n",
       "      <td>(discharge, 0.0150688301524)</td>\n",
       "      <td>(incident, 0.0140341527688)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>49</td>\n",
       "      <td>623.113</td>\n",
       "      <td>[(development, 0.176027756751), (housing, 0.15...</td>\n",
       "      <td>(development, 0.176027756751)</td>\n",
       "      <td>(housing, 0.152212657649)</td>\n",
       "      <td>(economic, 0.0588541094432)</td>\n",
       "      <td>(community, 0.0485919430535)</td>\n",
       "      <td>(infrastructure, 0.0190401252932)</td>\n",
       "      <td>(managed, 0.0183046960354)</td>\n",
       "      <td>(medicaid, 0.0178764063515)</td>\n",
       "      <td>(department, 0.0140238290886)</td>\n",
       "      <td>(buildings, 0.0137754059922)</td>\n",
       "      <td>(economy, 0.0123473943134)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>37</td>\n",
       "      <td>584.586</td>\n",
       "      <td>[(financial, 0.0959611006916), (permits, 0.066...</td>\n",
       "      <td>(financial, 0.0959611006916)</td>\n",
       "      <td>(permits, 0.0668632739602)</td>\n",
       "      <td>(building, 0.0631953900546)</td>\n",
       "      <td>(permit, 0.054456521697)</td>\n",
       "      <td>(expenditures, 0.0429247389481)</td>\n",
       "      <td>(guide, 0.0250580534259)</td>\n",
       "      <td>(issued, 0.0232434329642)</td>\n",
       "      <td>(construction, 0.0228846174877)</td>\n",
       "      <td>(information, 0.0227968624179)</td>\n",
       "      <td>(filter, 0.021954851646)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>40</td>\n",
       "      <td>582.037</td>\n",
       "      <td>[(finance, 0.164742098166), (year, 0.062366577...</td>\n",
       "      <td>(finance, 0.164742098166)</td>\n",
       "      <td>(year, 0.062366577351)</td>\n",
       "      <td>(state, 0.0533822539548)</td>\n",
       "      <td>(monthly, 0.0506962481995)</td>\n",
       "      <td>(fiscal, 0.0476222221357)</td>\n",
       "      <td>(payments, 0.0357199197181)</td>\n",
       "      <td>(government, 0.0264540232587)</td>\n",
       "      <td>(june, 0.0225770287471)</td>\n",
       "      <td>(july, 0.0216759115561)</td>\n",
       "      <td>(report, 0.0215900104843)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>31</td>\n",
       "      <td>572.482</td>\n",
       "      <td>[(politics, 0.110698260148), (government, 0.09...</td>\n",
       "      <td>(politics, 0.110698260148)</td>\n",
       "      <td>(government, 0.0983150505736)</td>\n",
       "      <td>(election, 0.0382597777507)</td>\n",
       "      <td>(campaign, 0.0309485620342)</td>\n",
       "      <td>(elections, 0.0296823051034)</td>\n",
       "      <td>(commission, 0.0293149928715)</td>\n",
       "      <td>(results, 0.0249283166716)</td>\n",
       "      <td>(city, 0.0213278569983)</td>\n",
       "      <td>(460, 0.0200805489271)</td>\n",
       "      <td>(finance, 0.0197352160164)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>48</td>\n",
       "      <td>560.589</td>\n",
       "      <td>[(business, 0.0803271304597), (license, 0.0323...</td>\n",
       "      <td>(business, 0.0803271304597)</td>\n",
       "      <td>(license, 0.032302648157)</td>\n",
       "      <td>(job, 0.0220026249885)</td>\n",
       "      <td>(contracts, 0.0219341654552)</td>\n",
       "      <td>(businesses, 0.0197708503312)</td>\n",
       "      <td>(vendor, 0.0178329754117)</td>\n",
       "      <td>(owned, 0.0175281633585)</td>\n",
       "      <td>(contract, 0.0175169967387)</td>\n",
       "      <td>(licenses, 0.0169698891758)</td>\n",
       "      <td>(certified, 0.0166884614561)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>41</td>\n",
       "      <td>499.286</td>\n",
       "      <td>[(service, 0.0872627144419), (requests, 0.0350...</td>\n",
       "      <td>(service, 0.0872627144419)</td>\n",
       "      <td>(requests, 0.0350316945429)</td>\n",
       "      <td>(inspection, 0.0326789351929)</td>\n",
       "      <td>(inspections, 0.0313503518264)</td>\n",
       "      <td>(311, 0.0263119431044)</td>\n",
       "      <td>(public, 0.025968462717)</td>\n",
       "      <td>(violations, 0.0237687923195)</td>\n",
       "      <td>(request, 0.022587674832)</td>\n",
       "      <td>(complaints, 0.0193704153109)</td>\n",
       "      <td>(days, 0.0180026003213)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>34</td>\n",
       "      <td>496.677</td>\n",
       "      <td>[(education, 0.181504757587), (school, 0.08185...</td>\n",
       "      <td>(education, 0.181504757587)</td>\n",
       "      <td>(school, 0.0818576925692)</td>\n",
       "      <td>(schools, 0.0317766682176)</td>\n",
       "      <td>(students, 0.0295716293995)</td>\n",
       "      <td>(public, 0.0171734235818)</td>\n",
       "      <td>(governance, 0.0167833173256)</td>\n",
       "      <td>(state, 0.0165785793709)</td>\n",
       "      <td>(year, 0.0161596047936)</td>\n",
       "      <td>(student, 0.0161397342546)</td>\n",
       "      <td>(district, 0.0160036278832)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>9</td>\n",
       "      <td>491.945</td>\n",
       "      <td>[(recreation, 0.131286229681), (parks, 0.06452...</td>\n",
       "      <td>(recreation, 0.131286229681)</td>\n",
       "      <td>(parks, 0.0645285967363)</td>\n",
       "      <td>(jersey, 0.0605939846627)</td>\n",
       "      <td>(park, 0.0400138091412)</td>\n",
       "      <td>(center, 0.029877716132)</td>\n",
       "      <td>(neighborhoods, 0.0258798720295)</td>\n",
       "      <td>(centers, 0.0221407272983)</td>\n",
       "      <td>(facilities, 0.0197678830443)</td>\n",
       "      <td>(environment, 0.0172267599485)</td>\n",
       "      <td>(access, 0.0157062019872)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>15</td>\n",
       "      <td>485.066</td>\n",
       "      <td>[(budget, 0.141547682714), (finance, 0.0616298...</td>\n",
       "      <td>(budget, 0.141547682714)</td>\n",
       "      <td>(finance, 0.0616298478188)</td>\n",
       "      <td>(fund, 0.0528667079259)</td>\n",
       "      <td>(year, 0.0273615283825)</td>\n",
       "      <td>(operating, 0.0267145486731)</td>\n",
       "      <td>(city, 0.0261515403541)</td>\n",
       "      <td>(fiscal, 0.0235877561714)</td>\n",
       "      <td>(funds, 0.0201535441911)</td>\n",
       "      <td>(calendar, 0.0182912645876)</td>\n",
       "      <td>(communities, 0.0168132558204)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>29</td>\n",
       "      <td>484.310</td>\n",
       "      <td>[(planning, 0.0609328856062), (county, 0.05924...</td>\n",
       "      <td>(planning, 0.0609328856062)</td>\n",
       "      <td>(county, 0.0592460462606)</td>\n",
       "      <td>(district, 0.0590822346049)</td>\n",
       "      <td>(districts, 0.0460914886485)</td>\n",
       "      <td>(boundaries, 0.0372114905902)</td>\n",
       "      <td>(city, 0.029524721059)</td>\n",
       "      <td>(areas, 0.0270950065527)</td>\n",
       "      <td>(zoning, 0.0266346423368)</td>\n",
       "      <td>(gis, 0.0244396763289)</td>\n",
       "      <td>(council, 0.0230826924938)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>35</td>\n",
       "      <td>467.623</td>\n",
       "      <td>[(infrastructure, 0.0824452424325), (city, 0.0...</td>\n",
       "      <td>(infrastructure, 0.0824452424325)</td>\n",
       "      <td>(city, 0.0658933761572)</td>\n",
       "      <td>(public, 0.0328457699722)</td>\n",
       "      <td>(maps, 0.0305195329364)</td>\n",
       "      <td>(works, 0.0279350117897)</td>\n",
       "      <td>(dallas, 0.0224431406282)</td>\n",
       "      <td>(los, 0.0198827044081)</td>\n",
       "      <td>(angeles, 0.0191921754348)</td>\n",
       "      <td>(types, 0.0151131207127)</td>\n",
       "      <td>(way, 0.0138871508278)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>30</td>\n",
       "      <td>467.407</td>\n",
       "      <td>[(mateo, 0.0418424874983), (city, 0.0239850798...</td>\n",
       "      <td>(mateo, 0.0418424874983)</td>\n",
       "      <td>(city, 0.0239850798125)</td>\n",
       "      <td>(gis, 0.0198053782604)</td>\n",
       "      <td>(land, 0.0192192818496)</td>\n",
       "      <td>(county, 0.0188704602627)</td>\n",
       "      <td>(parcel, 0.0173183551841)</td>\n",
       "      <td>(map, 0.0172935426724)</td>\n",
       "      <td>(point, 0.0172170884678)</td>\n",
       "      <td>(survey, 0.0164839101622)</td>\n",
       "      <td>(feature, 0.0160484881569)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>4</td>\n",
       "      <td>464.917</td>\n",
       "      <td>[(information, 0.0456295107206), (tab, 0.02519...</td>\n",
       "      <td>(information, 0.0456295107206)</td>\n",
       "      <td>(tab, 0.0251926721763)</td>\n",
       "      <td>(projects, 0.0242891655476)</td>\n",
       "      <td>(concerning, 0.0182745676272)</td>\n",
       "      <td>(may, 0.0177403684305)</td>\n",
       "      <td>(contains, 0.0171133718182)</td>\n",
       "      <td>(project, 0.0158542813828)</td>\n",
       "      <td>(provided, 0.0138395187013)</td>\n",
       "      <td>(additional, 0.0136923822487)</td>\n",
       "      <td>(improvement, 0.0124520027276)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>2</td>\n",
       "      <td>428.610</td>\n",
       "      <td>[(population, 0.131515203217), (county, 0.0919...</td>\n",
       "      <td>(population, 0.131515203217)</td>\n",
       "      <td>(county, 0.0919604340497)</td>\n",
       "      <td>(demographics, 0.0842750050866)</td>\n",
       "      <td>(age, 0.0434468663432)</td>\n",
       "      <td>(total, 0.0282215586982)</td>\n",
       "      <td>(king, 0.0194688109256)</td>\n",
       "      <td>(washington, 0.0192761369915)</td>\n",
       "      <td>(montgomery, 0.0182610800118)</td>\n",
       "      <td>(number, 0.0179005714425)</td>\n",
       "      <td>(years, 0.0169604724905)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>36</td>\n",
       "      <td>412.971</td>\n",
       "      <td>[(census, 0.0788049808377), (indicators, 0.077...</td>\n",
       "      <td>(census, 0.0788049808377)</td>\n",
       "      <td>(indicators, 0.0773738300794)</td>\n",
       "      <td>(survey, 0.0480351597869)</td>\n",
       "      <td>(includes, 0.0448351738525)</td>\n",
       "      <td>(indicator, 0.0369245239412)</td>\n",
       "      <td>(community, 0.0332211165998)</td>\n",
       "      <td>(demographics, 0.0282093059688)</td>\n",
       "      <td>(bureau, 0.0238720417508)</td>\n",
       "      <td>(u, 0.0211392894776)</td>\n",
       "      <td>(level, 0.0192043130616)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>50</td>\n",
       "      <td>409.631</td>\n",
       "      <td>[(program, 0.032511463571), (employees, 0.0289...</td>\n",
       "      <td>(program, 0.032511463571)</td>\n",
       "      <td>(employees, 0.0289751722757)</td>\n",
       "      <td>(adult, 0.0244085054991)</td>\n",
       "      <td>(employee, 0.0242968020528)</td>\n",
       "      <td>(annual, 0.0204986126128)</td>\n",
       "      <td>(names, 0.0182179413533)</td>\n",
       "      <td>(full, 0.0179091183965)</td>\n",
       "      <td>(abuse, 0.0162096834129)</td>\n",
       "      <td>(nutrition, 0.0157677124596)</td>\n",
       "      <td>(beginning, 0.0142137708217)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>25</td>\n",
       "      <td>393.180</td>\n",
       "      <td>[(tax, 0.116949417443), (revenue, 0.0582816749...</td>\n",
       "      <td>(tax, 0.116949417443)</td>\n",
       "      <td>(revenue, 0.0582816749296)</td>\n",
       "      <td>(sales, 0.0386343839808)</td>\n",
       "      <td>(taxes, 0.0347130105104)</td>\n",
       "      <td>(local, 0.0332054161494)</td>\n",
       "      <td>(colorado, 0.0306870467969)</td>\n",
       "      <td>(government, 0.0297069172677)</td>\n",
       "      <td>(state, 0.0219961180005)</td>\n",
       "      <td>(department, 0.0191541789639)</td>\n",
       "      <td>(sale, 0.0152112241357)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Topic Number  Adjusted_Popularity  \\\n",
       "0             26              799.492   \n",
       "1              3              692.631   \n",
       "2             21              670.051   \n",
       "3             49              623.113   \n",
       "4             37              584.586   \n",
       "5             40              582.037   \n",
       "6             31              572.482   \n",
       "7             48              560.589   \n",
       "8             41              499.286   \n",
       "9             34              496.677   \n",
       "10             9              491.945   \n",
       "11            15              485.066   \n",
       "12            29              484.310   \n",
       "13            35              467.623   \n",
       "14            30              467.407   \n",
       "15             4              464.917   \n",
       "16             2              428.610   \n",
       "17            36              412.971   \n",
       "18            50              409.631   \n",
       "19            25              393.180   \n",
       "\n",
       "                                                Topic  \\\n",
       "0   [(public, 0.168381768411), (safety, 0.13998228...   \n",
       "1   [(transportation, 0.164542015423), (traffic, 0...   \n",
       "2   [(public, 0.0713314310777), (safety, 0.0690276...   \n",
       "3   [(development, 0.176027756751), (housing, 0.15...   \n",
       "4   [(financial, 0.0959611006916), (permits, 0.066...   \n",
       "5   [(finance, 0.164742098166), (year, 0.062366577...   \n",
       "6   [(politics, 0.110698260148), (government, 0.09...   \n",
       "7   [(business, 0.0803271304597), (license, 0.0323...   \n",
       "8   [(service, 0.0872627144419), (requests, 0.0350...   \n",
       "9   [(education, 0.181504757587), (school, 0.08185...   \n",
       "10  [(recreation, 0.131286229681), (parks, 0.06452...   \n",
       "11  [(budget, 0.141547682714), (finance, 0.0616298...   \n",
       "12  [(planning, 0.0609328856062), (county, 0.05924...   \n",
       "13  [(infrastructure, 0.0824452424325), (city, 0.0...   \n",
       "14  [(mateo, 0.0418424874983), (city, 0.0239850798...   \n",
       "15  [(information, 0.0456295107206), (tab, 0.02519...   \n",
       "16  [(population, 0.131515203217), (county, 0.0919...   \n",
       "17  [(census, 0.0788049808377), (indicators, 0.077...   \n",
       "18  [(program, 0.032511463571), (employees, 0.0289...   \n",
       "19  [(tax, 0.116949417443), (revenue, 0.0582816749...   \n",
       "\n",
       "                               topic1                         topic2  \\\n",
       "0            (public, 0.168381768411)       (safety, 0.139982283853)   \n",
       "1    (transportation, 0.164542015423)     (traffic, 0.0382494150922)   \n",
       "2           (public, 0.0713314310777)      (safety, 0.0690276616473)   \n",
       "3       (development, 0.176027756751)      (housing, 0.152212657649)   \n",
       "4        (financial, 0.0959611006916)     (permits, 0.0668632739602)   \n",
       "5           (finance, 0.164742098166)         (year, 0.062366577351)   \n",
       "6          (politics, 0.110698260148)  (government, 0.0983150505736)   \n",
       "7         (business, 0.0803271304597)      (license, 0.032302648157)   \n",
       "8          (service, 0.0872627144419)    (requests, 0.0350316945429)   \n",
       "9         (education, 0.181504757587)      (school, 0.0818576925692)   \n",
       "10       (recreation, 0.131286229681)       (parks, 0.0645285967363)   \n",
       "11           (budget, 0.141547682714)     (finance, 0.0616298478188)   \n",
       "12        (planning, 0.0609328856062)      (county, 0.0592460462606)   \n",
       "13  (infrastructure, 0.0824452424325)        (city, 0.0658933761572)   \n",
       "14           (mateo, 0.0418424874983)        (city, 0.0239850798125)   \n",
       "15     (information, 0.0456295107206)         (tab, 0.0251926721763)   \n",
       "16       (population, 0.131515203217)      (county, 0.0919604340497)   \n",
       "17          (census, 0.0788049808377)  (indicators, 0.0773738300794)   \n",
       "18          (program, 0.032511463571)   (employees, 0.0289751722757)   \n",
       "19              (tax, 0.116949417443)     (revenue, 0.0582816749296)   \n",
       "\n",
       "                             topic3                          topic4  \\\n",
       "0          (crime, 0.0631808672948)         (police, 0.03597164994)   \n",
       "1         (street, 0.0297088546702)      (parking, 0.0260484479646)   \n",
       "2           (fire, 0.0477503790148)        (month, 0.0205819878206)   \n",
       "3       (economic, 0.0588541094432)    (community, 0.0485919430535)   \n",
       "4       (building, 0.0631953900546)        (permit, 0.054456521697)   \n",
       "5          (state, 0.0533822539548)      (monthly, 0.0506962481995)   \n",
       "6       (election, 0.0382597777507)     (campaign, 0.0309485620342)   \n",
       "7            (job, 0.0220026249885)    (contracts, 0.0219341654552)   \n",
       "8     (inspection, 0.0326789351929)  (inspections, 0.0313503518264)   \n",
       "9        (schools, 0.0317766682176)     (students, 0.0295716293995)   \n",
       "10        (jersey, 0.0605939846627)         (park, 0.0400138091412)   \n",
       "11          (fund, 0.0528667079259)         (year, 0.0273615283825)   \n",
       "12      (district, 0.0590822346049)    (districts, 0.0460914886485)   \n",
       "13        (public, 0.0328457699722)         (maps, 0.0305195329364)   \n",
       "14           (gis, 0.0198053782604)         (land, 0.0192192818496)   \n",
       "15      (projects, 0.0242891655476)   (concerning, 0.0182745676272)   \n",
       "16  (demographics, 0.0842750050866)          (age, 0.0434468663432)   \n",
       "17        (survey, 0.0480351597869)     (includes, 0.0448351738525)   \n",
       "18         (adult, 0.0244085054991)     (employee, 0.0242968020528)   \n",
       "19         (sales, 0.0386343839808)        (taxes, 0.0347130105104)   \n",
       "\n",
       "                               topic5                            topic6  \\\n",
       "0       (department, 0.0193424493246)       (illinois, 0.0152118075546)   \n",
       "1    (infrastructure, 0.024499661836)           (city, 0.0242484880067)   \n",
       "2            (calls, 0.0173756495075)         (police, 0.0170781244666)   \n",
       "3   (infrastructure, 0.0190401252932)        (managed, 0.0183046960354)   \n",
       "4     (expenditures, 0.0429247389481)          (guide, 0.0250580534259)   \n",
       "5           (fiscal, 0.0476222221357)       (payments, 0.0357199197181)   \n",
       "6        (elections, 0.0296823051034)     (commission, 0.0293149928715)   \n",
       "7       (businesses, 0.0197708503312)         (vendor, 0.0178329754117)   \n",
       "8              (311, 0.0263119431044)          (public, 0.025968462717)   \n",
       "9           (public, 0.0171734235818)     (governance, 0.0167833173256)   \n",
       "10           (center, 0.029877716132)  (neighborhoods, 0.0258798720295)   \n",
       "11       (operating, 0.0267145486731)           (city, 0.0261515403541)   \n",
       "12      (boundaries, 0.0372114905902)            (city, 0.029524721059)   \n",
       "13           (works, 0.0279350117897)         (dallas, 0.0224431406282)   \n",
       "14          (county, 0.0188704602627)         (parcel, 0.0173183551841)   \n",
       "15             (may, 0.0177403684305)       (contains, 0.0171133718182)   \n",
       "16           (total, 0.0282215586982)           (king, 0.0194688109256)   \n",
       "17       (indicator, 0.0369245239412)      (community, 0.0332211165998)   \n",
       "18          (annual, 0.0204986126128)          (names, 0.0182179413533)   \n",
       "19           (local, 0.0332054161494)       (colorado, 0.0306870467969)   \n",
       "\n",
       "                             topic7                           topic8  \\\n",
       "0           (race, 0.0133652012852)       (reported, 0.011776909211)   \n",
       "1           (safe, 0.0239084069945)       (vehicle, 0.0231351100665)   \n",
       "2      (emergency, 0.0165221049163)      (response, 0.0157891037516)   \n",
       "3       (medicaid, 0.0178764063515)    (department, 0.0140238290886)   \n",
       "4         (issued, 0.0232434329642)  (construction, 0.0228846174877)   \n",
       "5     (government, 0.0264540232587)          (june, 0.0225770287471)   \n",
       "6        (results, 0.0249283166716)          (city, 0.0213278569983)   \n",
       "7          (owned, 0.0175281633585)      (contract, 0.0175169967387)   \n",
       "8     (violations, 0.0237687923195)        (request, 0.022587674832)   \n",
       "9          (state, 0.0165785793709)          (year, 0.0161596047936)   \n",
       "10       (centers, 0.0221407272983)    (facilities, 0.0197678830443)   \n",
       "11        (fiscal, 0.0235877561714)         (funds, 0.0201535441911)   \n",
       "12         (areas, 0.0270950065527)        (zoning, 0.0266346423368)   \n",
       "13           (los, 0.0198827044081)       (angeles, 0.0191921754348)   \n",
       "14           (map, 0.0172935426724)         (point, 0.0172170884678)   \n",
       "15       (project, 0.0158542813828)      (provided, 0.0138395187013)   \n",
       "16    (washington, 0.0192761369915)    (montgomery, 0.0182610800118)   \n",
       "17  (demographics, 0.0282093059688)        (bureau, 0.0238720417508)   \n",
       "18          (full, 0.0179091183965)         (abuse, 0.0162096834129)   \n",
       "19    (government, 0.0297069172677)         (state, 0.0219961180005)   \n",
       "\n",
       "                            topic9                         topic10  \n",
       "0        (crimes, 0.0115504690759)     (criminal, 0.0105105785255)  \n",
       "1       (streets, 0.0219852292388)         (bike, 0.0155102577616)  \n",
       "2     (discharge, 0.0150688301524)     (incident, 0.0140341527688)  \n",
       "3     (buildings, 0.0137754059922)      (economy, 0.0123473943134)  \n",
       "4   (information, 0.0227968624179)        (filter, 0.021954851646)  \n",
       "5          (july, 0.0216759115561)       (report, 0.0215900104843)  \n",
       "6           (460, 0.0200805489271)      (finance, 0.0197352160164)  \n",
       "7      (licenses, 0.0169698891758)    (certified, 0.0166884614561)  \n",
       "8    (complaints, 0.0193704153109)         (days, 0.0180026003213)  \n",
       "9       (student, 0.0161397342546)     (district, 0.0160036278832)  \n",
       "10  (environment, 0.0172267599485)       (access, 0.0157062019872)  \n",
       "11     (calendar, 0.0182912645876)  (communities, 0.0168132558204)  \n",
       "12          (gis, 0.0244396763289)      (council, 0.0230826924938)  \n",
       "13        (types, 0.0151131207127)          (way, 0.0138871508278)  \n",
       "14       (survey, 0.0164839101622)      (feature, 0.0160484881569)  \n",
       "15   (additional, 0.0136923822487)  (improvement, 0.0124520027276)  \n",
       "16       (number, 0.0179005714425)        (years, 0.0169604724905)  \n",
       "17            (u, 0.0211392894776)        (level, 0.0192043130616)  \n",
       "18    (nutrition, 0.0157677124596)    (beginning, 0.0142137708217)  \n",
       "19   (department, 0.0191541789639)         (sale, 0.0152112241357)  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "norm_by_areas_df = dict_to_df(pop_norm_area_dict)\n",
    "norm_by_areas_df.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#a little cleaner view for display online:\n",
    "def display_view(df):\n",
    "    \n",
    "    pd.options.display.max_colwidth = 120 #so we can see our whole topic\n",
    "    \n",
    "    clean_topics = []\n",
    "    for topic_tup in df.Topic: \n",
    "        raw_words = list(i[0] for i in topic_tup)\n",
    "        string = \", \".join(raw_words)\n",
    "        clean_topics.append(string)\n",
    "    \n",
    "    clean_df = df.iloc[:, 0:2] #take off the old topic cols\n",
    "    clean_df['Topic'] = clean_topics #plug in clean topic strings list to new df\n",
    "    \n",
    "    clean_df.columns = ['Topic ID', 'Adjusted Popularity Score', 'Topic Content'] #rename for web\n",
    "    \n",
    "    clean_df.index = np.arange(1, len(clean_df.index) + 1) #we'll start index at 1 so it's easier to view ranks\n",
    "    \n",
    "    return clean_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic ID</th>\n",
       "      <th>Adjusted Popularity Score</th>\n",
       "      <th>Topic Content</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>26</td>\n",
       "      <td>799.492</td>\n",
       "      <td>public, safety, crime, police, department, illinois, race, reported, crimes, criminal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>692.631</td>\n",
       "      <td>transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21</td>\n",
       "      <td>670.051</td>\n",
       "      <td>public, safety, fire, month, calls, police, emergency, response, discharge, incident</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49</td>\n",
       "      <td>623.113</td>\n",
       "      <td>development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>37</td>\n",
       "      <td>584.586</td>\n",
       "      <td>financial, permits, building, permit, expenditures, guide, issued, construction, information, filter</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>40</td>\n",
       "      <td>582.037</td>\n",
       "      <td>finance, year, state, monthly, fiscal, payments, government, june, july, report</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>31</td>\n",
       "      <td>572.482</td>\n",
       "      <td>politics, government, election, campaign, elections, commission, results, city, 460, finance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>48</td>\n",
       "      <td>560.589</td>\n",
       "      <td>business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>41</td>\n",
       "      <td>499.286</td>\n",
       "      <td>service, requests, inspection, inspections, 311, public, violations, request, complaints, days</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>34</td>\n",
       "      <td>496.677</td>\n",
       "      <td>education, school, schools, students, public, governance, state, year, student, district</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>9</td>\n",
       "      <td>491.945</td>\n",
       "      <td>recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>15</td>\n",
       "      <td>485.066</td>\n",
       "      <td>budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>29</td>\n",
       "      <td>484.310</td>\n",
       "      <td>planning, county, district, districts, boundaries, city, areas, zoning, gis, council</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>35</td>\n",
       "      <td>467.623</td>\n",
       "      <td>infrastructure, city, public, maps, works, dallas, los, angeles, types, way</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>30</td>\n",
       "      <td>467.407</td>\n",
       "      <td>mateo, city, gis, land, county, parcel, map, point, survey, feature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>4</td>\n",
       "      <td>464.917</td>\n",
       "      <td>information, tab, projects, concerning, may, contains, project, provided, additional, improvement</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>2</td>\n",
       "      <td>428.610</td>\n",
       "      <td>population, county, demographics, age, total, king, washington, montgomery, number, years</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>36</td>\n",
       "      <td>412.971</td>\n",
       "      <td>census, indicators, survey, includes, indicator, community, demographics, bureau, u, level</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>50</td>\n",
       "      <td>409.631</td>\n",
       "      <td>program, employees, adult, employee, annual, names, full, abuse, nutrition, beginning</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>25</td>\n",
       "      <td>393.180</td>\n",
       "      <td>tax, revenue, sales, taxes, local, colorado, government, state, department, sale</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>27</td>\n",
       "      <td>387.617</td>\n",
       "      <td>water, environment, protection, tx, filed, waste, site, quality, environmental, monitoring</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>42</td>\n",
       "      <td>357.764</td>\n",
       "      <td>information, list, code, required, political, utah, public, city, years, see</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>28</td>\n",
       "      <td>356.698</td>\n",
       "      <td>transportation, plans, iowa, area, transit, operations, region, bus, priority, people</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>11</td>\n",
       "      <td>346.489</td>\n",
       "      <td>energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>13</td>\n",
       "      <td>346.136</td>\n",
       "      <td>gas, natural, food, greenhouse, environment, emissions, costs, resources, program, fuel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>23</td>\n",
       "      <td>345.531</td>\n",
       "      <td>police, user, information, page, department, use, orleans, injury, may, records</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>39</td>\n",
       "      <td>341.116</td>\n",
       "      <td>health, san, county, care, counties, system, facilities, network, home, francisco</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>6</td>\n",
       "      <td>337.260</td>\n",
       "      <td>property, consumption, real, value, properties, significant, easy, usage, assessment, company</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>17</td>\n",
       "      <td>321.496</td>\n",
       "      <td>services, social, human, dfps, information, programs, us, definition, visit, state</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>1</td>\n",
       "      <td>320.423</td>\n",
       "      <td>chart, children, income, historic, home, assistance, homes, families, low, pay</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>16</td>\n",
       "      <td>310.315</td>\n",
       "      <td>economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>5</td>\n",
       "      <td>274.758</td>\n",
       "      <td>locations, bacteria, hours, culture, county, levels, contact, directory, e, contains</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>12</td>\n",
       "      <td>266.153</td>\n",
       "      <td>map, \"about\", shows, details, tracking, click, view, additional, weekly, information</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>47</td>\n",
       "      <td>256.439</td>\n",
       "      <td>care, finance, provider, child, cost, plan, day, bay, government, fee</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>43</td>\n",
       "      <td>248.788</td>\n",
       "      <td>state, new, michigan, york, information, check, measurements, jobs, licensing, ny</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>0</td>\n",
       "      <td>247.484</td>\n",
       "      <td>prevention, hospital, ny, healthy, statewide, hospitals, inpatient, healthcare, api, quality-safety-costs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>37</th>\n",
       "      <td>45</td>\n",
       "      <td>242.566</td>\n",
       "      <td>report, performance, city, goal, resultsnola, administration, city's, management, key, topeka</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>38</th>\n",
       "      <td>33</td>\n",
       "      <td>226.347</td>\n",
       "      <td>finance, summary, dashboard, ethics, form, management, board, pension, includes, last</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>39</th>\n",
       "      <td>24</td>\n",
       "      <td>224.813</td>\n",
       "      <td>official, account, accounts, (openmichigan@michigan, special, nursing, software, gis, use, consumer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40</th>\n",
       "      <td>8</td>\n",
       "      <td>223.512</td>\n",
       "      <td>health, disease, 000, per, rate, rates, chronic, risk, agency, due</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>41</th>\n",
       "      <td>44</td>\n",
       "      <td>218.401</td>\n",
       "      <td>book, lead, stations, databook, station, closed, ecology, solar, power, residential</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42</th>\n",
       "      <td>22</td>\n",
       "      <td>218.129</td>\n",
       "      <td>arts, grants, live, county, includes, library, clearance, grant, inventory, cps</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>18</td>\n",
       "      <td>217.918</td>\n",
       "      <td>column, update, homeless, annually, frequency, position, city, commercial, daily, salaries</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44</th>\n",
       "      <td>38</td>\n",
       "      <td>216.225</td>\n",
       "      <td>plan, strategic, master, community, fewer, limitations, registry, cause, years, information</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>7</td>\n",
       "      <td>214.701</td>\n",
       "      <td>rates, calculated, rate, year, counts, average, seattle, percent, mortality, number</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>32</td>\n",
       "      <td>197.366</td>\n",
       "      <td>open, created, items, clear, volume, please, presented, difficult, potential, note</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>47</th>\n",
       "      <td>46</td>\n",
       "      <td>195.113</td>\n",
       "      <td>address, name, number, facility, patient, location, contains, type, g, transfer</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48</th>\n",
       "      <td>10</td>\n",
       "      <td>185.467</td>\n",
       "      <td>measures, quality, performance, government, life, developed, fair, measurement, definitions, index</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49</th>\n",
       "      <td>14</td>\n",
       "      <td>171.853</td>\n",
       "      <td>health, code, statistics, county, codes, zip, public, diego, harbison, crest</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>51</td>\n",
       "      <td>171.328</td>\n",
       "      <td>cook, high, cases, youth, case, court, enrollment, college, wa, gis/maps</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>51</th>\n",
       "      <td>19</td>\n",
       "      <td>170.426</td>\n",
       "      <td>death, maryland, database, deaths, office, department, new, cdc, metadata, citizens</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52</th>\n",
       "      <td>20</td>\n",
       "      <td>108.422</td>\n",
       "      <td>expected, dept, observed, every, 300, pacific, physical, performed, contribution, std</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    Topic ID  Adjusted Popularity Score  \\\n",
       "1         26                    799.492   \n",
       "2          3                    692.631   \n",
       "3         21                    670.051   \n",
       "4         49                    623.113   \n",
       "5         37                    584.586   \n",
       "6         40                    582.037   \n",
       "7         31                    572.482   \n",
       "8         48                    560.589   \n",
       "9         41                    499.286   \n",
       "10        34                    496.677   \n",
       "11         9                    491.945   \n",
       "12        15                    485.066   \n",
       "13        29                    484.310   \n",
       "14        35                    467.623   \n",
       "15        30                    467.407   \n",
       "16         4                    464.917   \n",
       "17         2                    428.610   \n",
       "18        36                    412.971   \n",
       "19        50                    409.631   \n",
       "20        25                    393.180   \n",
       "21        27                    387.617   \n",
       "22        42                    357.764   \n",
       "23        28                    356.698   \n",
       "24        11                    346.489   \n",
       "25        13                    346.136   \n",
       "26        23                    345.531   \n",
       "27        39                    341.116   \n",
       "28         6                    337.260   \n",
       "29        17                    321.496   \n",
       "30         1                    320.423   \n",
       "31        16                    310.315   \n",
       "32         5                    274.758   \n",
       "33        12                    266.153   \n",
       "34        47                    256.439   \n",
       "35        43                    248.788   \n",
       "36         0                    247.484   \n",
       "37        45                    242.566   \n",
       "38        33                    226.347   \n",
       "39        24                    224.813   \n",
       "40         8                    223.512   \n",
       "41        44                    218.401   \n",
       "42        22                    218.129   \n",
       "43        18                    217.918   \n",
       "44        38                    216.225   \n",
       "45         7                    214.701   \n",
       "46        32                    197.366   \n",
       "47        46                    195.113   \n",
       "48        10                    185.467   \n",
       "49        14                    171.853   \n",
       "50        51                    171.328   \n",
       "51        19                    170.426   \n",
       "52        20                    108.422   \n",
       "\n",
       "                                                                                                   Topic Content  \n",
       "1                          public, safety, crime, police, department, illinois, race, reported, crimes, criminal  \n",
       "2                   transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike  \n",
       "3                           public, safety, fire, month, calls, police, emergency, response, discharge, incident  \n",
       "4   development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy  \n",
       "5           financial, permits, building, permit, expenditures, guide, issued, construction, information, filter  \n",
       "6                                finance, year, state, monthly, fiscal, payments, government, june, july, report  \n",
       "7                   politics, government, election, campaign, elections, commission, results, city, 460, finance  \n",
       "8                    business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified  \n",
       "9                 service, requests, inspection, inspections, 311, public, violations, request, complaints, days  \n",
       "10                      education, school, schools, students, public, governance, state, year, student, district  \n",
       "11              recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access  \n",
       "12                            budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities  \n",
       "13                          planning, county, district, districts, boundaries, city, areas, zoning, gis, council  \n",
       "14                                   infrastructure, city, public, maps, works, dallas, los, angeles, types, way  \n",
       "15                                           mateo, city, gis, land, county, parcel, map, point, survey, feature  \n",
       "16             information, tab, projects, concerning, may, contains, project, provided, additional, improvement  \n",
       "17                     population, county, demographics, age, total, king, washington, montgomery, number, years  \n",
       "18                    census, indicators, survey, includes, indicator, community, demographics, bureau, u, level  \n",
       "19                         program, employees, adult, employee, annual, names, full, abuse, nutrition, beginning  \n",
       "20                              tax, revenue, sales, taxes, local, colorado, government, state, department, sale  \n",
       "21                    water, environment, protection, tx, filed, waste, site, quality, environmental, monitoring  \n",
       "22                                  information, list, code, required, political, utah, public, city, years, see  \n",
       "23                         transportation, plans, iowa, area, transit, operations, region, bus, priority, people  \n",
       "24                  energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities  \n",
       "25                       gas, natural, food, greenhouse, environment, emissions, costs, resources, program, fuel  \n",
       "26                               police, user, information, page, department, use, orleans, injury, may, records  \n",
       "27                             health, san, county, care, counties, system, facilities, network, home, francisco  \n",
       "28                 property, consumption, real, value, properties, significant, easy, usage, assessment, company  \n",
       "29                            services, social, human, dfps, information, programs, us, definition, visit, state  \n",
       "30                                chart, children, income, historic, home, assistance, homes, families, low, pay  \n",
       "31           economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal  \n",
       "32                          locations, bacteria, hours, culture, county, levels, contact, directory, e, contains  \n",
       "33                          map, \"about\", shows, details, tracking, click, view, additional, weekly, information  \n",
       "34                                         care, finance, provider, child, cost, plan, day, bay, government, fee  \n",
       "35                             state, new, michigan, york, information, check, measurements, jobs, licensing, ny  \n",
       "36     prevention, hospital, ny, healthy, statewide, hospitals, inpatient, healthcare, api, quality-safety-costs  \n",
       "37                 report, performance, city, goal, resultsnola, administration, city's, management, key, topeka  \n",
       "38                         finance, summary, dashboard, ethics, form, management, board, pension, includes, last  \n",
       "39           official, account, accounts, (openmichigan@michigan, special, nursing, software, gis, use, consumer  \n",
       "40                                            health, disease, 000, per, rate, rates, chronic, risk, agency, due  \n",
       "41                           book, lead, stations, databook, station, closed, ecology, solar, power, residential  \n",
       "42                               arts, grants, live, county, includes, library, clearance, grant, inventory, cps  \n",
       "43                    column, update, homeless, annually, frequency, position, city, commercial, daily, salaries  \n",
       "44                   plan, strategic, master, community, fewer, limitations, registry, cause, years, information  \n",
       "45                           rates, calculated, rate, year, counts, average, seattle, percent, mortality, number  \n",
       "46                            open, created, items, clear, volume, please, presented, difficult, potential, note  \n",
       "47                               address, name, number, facility, patient, location, contains, type, g, transfer  \n",
       "48            measures, quality, performance, government, life, developed, fair, measurement, definitions, index  \n",
       "49                                  health, code, statistics, county, codes, zip, public, diego, harbison, crest  \n",
       "50                                      cook, high, cases, youth, case, court, enrollment, college, wa, gis/maps  \n",
       "51                           death, maryland, database, deaths, office, department, new, cdc, metadata, citizens  \n",
       "52                         expected, dept, observed, every, 300, pacific, physical, performed, contribution, std  "
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_view = display_view(norm_by_areas_df)\n",
    "display_view"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#display_view.to_csv('final_topic_ranks.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dampened Popularity Results:\n",
    "Interesting! Our top transportation tag has fallen out of the Top 10. Perhaps California and Texas really were giving it too much of a boost, even with log_views. \n",
    "- We can see that two public safety tags -- I'd call them \"reported crimes\" \"public safety calls\" have taken the top spot. \n",
    "- Our \"Personal Transportation\" topic is #2. The real life topic of transportation information is still a popular topic.\n",
    "- Interestingly, \"annual budget and fiscal year data\" comes in at #6 for the first time.\n",
    "- 311 calls has fallen all the way to #9! **Unsurprisingly, 311 was a top topic in New York City.**\n",
    "- Our parochial datasets have fallen out of the top ranks. This is good.\n",
    "- **The separation between popularity metrics is much smaller and less significant. We should take care in making sweeping conclusions.** Again, we set a 0.2 threshold for this scoring category in winner-take_all. Different scoring could change this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finally, let's determine most popular topics by domain using *raw totals of dataset views and downloads*. \n",
    "- Previously, we looked at the top topics for each domain using log-adjusted scoring.\n",
    "- But there's really no need to control for big cities by using a logarithm on our totals when we are comparing WITHIN one city."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#first let's modify our old proportional_popularity function to just count TOTAL views and downloads; \n",
    "#no need to take the log to control for big cities if we are ONLY calculating what's most popular in one city\n",
    "\n",
    "def calc_raw_pop(df):\n",
    "    df = df.fillna(0)\n",
    "    results_dict = {}\n",
    "    \n",
    "    for row_num in df.index:\n",
    "        for tup in df.topic_comp[row_num]: #no need to pick a winner here...\n",
    "            if not tup[0] in results_dict:\n",
    "                if tup[1] >= 0.2: #.loc interprets as the label of the index, not int position\n",
    "                    results_dict[tup[0]] = (tup[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total))\n",
    "                else:\n",
    "                    pass\n",
    "            if tup[0] in results_dict:\n",
    "                if tup[1] >= 0.2:\n",
    "                    results_dict[tup[0]] += (tup[1] * (df.loc[row_num].download_count + \n",
    "                                             df.loc[row_num].page_views_total))\n",
    "    return results_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#next we need to modify our specific domain popularity function to call calc_raw_pop, not calc_proportional_pop\n",
    "def raw_popularity_by_domain(df, domain_name):\n",
    "    \n",
    "    domain_df = df[df.domain == domain_name]\n",
    "    popularity_dict = calc_raw_pop(domain_df) #we're still using our proportional scoring here\n",
    "    \n",
    "    df_pop = pd.DataFrame.from_dict(popularity_dict, orient='index') #dict is in df but index numbers (aka topic numbers) are random\n",
    "    df_pop = df_pop.rename(index=int, columns={0:\"Adjusted_Popularity\"})\n",
    "    \n",
    "    topic_words_list = []\n",
    "    for topic_num in df_pop.index: #this iterates in the right order - index numbers have real meaning\n",
    "        topic_tups = lda_52_sixty.show_topic(topic_num)\n",
    "        raw_words = list(i[0] for i in lda_52_sixty.show_topic(topic_num))\n",
    "        string = \", \".join(raw_words)\n",
    "        topic_words_list.append(string) #plug in names by index numbers\n",
    "    \n",
    "    df_pop['Topic_Words'] = topic_words_list\n",
    "    \n",
    "    df_pop = df_pop.sort_values(by='Adjusted_Popularity', ascending=False)\n",
    "    \n",
    "    #put the domain in each row - kind of an ugly way to preserve info about domain but it's what we've got\n",
    "    domain_string = domain_name\n",
    "    df_pop['Domain'] = [domain_string for number in range(len(df_pop.index))]\n",
    "    \n",
    "    return df_pop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally, we'll modify our top_topics_by_domain() function to use this raw metric of scoring**\n",
    "- I really should have restructured how I wrote these and used @decorators. Oh well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def raw_top_topics_by_domain(df, domains_series):\n",
    "    \n",
    "    domains_dict = {}\n",
    "    for domain in domains_series:\n",
    "        \n",
    "        domain_df = raw_popularity_by_domain(df, domain) #this is a df like the Madison example\n",
    "        Topic = domain_df.iloc[0].Topic_Words #string\n",
    "        Domain = domain_df.iloc[0].Domain.split() #have to remember to split(), which really just gives us one word\n",
    "        \n",
    "        if Topic in domains_dict: \n",
    "            domains_dict[Topic] += Domain\n",
    "        else:\n",
    "            domains_dict[Topic] = list(Domain)\n",
    "    \n",
    "    return domains_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "raw_most_popular = raw_top_topics_by_domain(topics, topics.domain.unique()) \n",
    "#again, should have just put this second arg in the functio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities': ['data.macoupincountyil.gov',\n",
       "  'data.tompsc.com',\n",
       "  'data.topeka.org',\n",
       "  'information.stpaul.gov',\n",
       "  'performance.danvilleva.gov',\n",
       "  'performance.franklintn.gov'],\n",
       " 'business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified': ['data.cityofgainesville.org',\n",
       "  'data.culvercity.org',\n",
       "  'data.oxnard.org',\n",
       "  'opendata.cityofhenderson.com'],\n",
       " 'care, finance, provider, child, cost, plan, day, bay, government, fee': ['data.datamontana.us'],\n",
       " 'census, indicators, survey, includes, indicator, community, demographics, bureau, u, level': ['data.livewellsd.org',\n",
       "  'opendata.cityofmesquite.com'],\n",
       " 'development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy': ['dashboard.alexandriava.gov',\n",
       "  'data.countyofriverside.us',\n",
       "  'data.hampton.gov',\n",
       "  'data.maine.gov',\n",
       "  'stat.cityofgainesville.org'],\n",
       " 'economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal': ['data.iowa.gov'],\n",
       " 'education, school, schools, students, public, governance, state, year, student, district': ['dashboard.hawaii.gov',\n",
       "  'data.mass.gov',\n",
       "  'data.vermont.gov'],\n",
       " 'energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities': ['data.results.wa.gov',\n",
       "  'data.sustainablesm.org',\n",
       "  'performance.providenceri.gov'],\n",
       " 'finance, year, state, monthly, fiscal, payments, government, june, july, report': ['data.maryland.gov',\n",
       "  'data.nj.gov',\n",
       "  'impact.stlouisco.com',\n",
       "  'opencheckbook.data.somervillema.gov',\n",
       "  'opendata.lasvegasnevada.gov',\n",
       "  'performance.cookcountyil.gov',\n",
       "  'transparency.michigan.gov'],\n",
       " 'financial, permits, building, permit, expenditures, guide, issued, construction, information, filter': ['data.burlingtonvt.gov',\n",
       "  'data.cityofboston.gov',\n",
       "  'data.cityofevanston.org',\n",
       "  'data.cstx.gov',\n",
       "  'data.fortworthtexas.gov',\n",
       "  'data.lacity.org',\n",
       "  'data.montgomeryal.gov',\n",
       "  'data.nashville.gov',\n",
       "  'data.tuscaloosa.com'],\n",
       " 'gas, natural, food, greenhouse, environment, emissions, costs, resources, program, fuel': ['dashboard.slco.org',\n",
       "  'opendata.utah.gov'],\n",
       " 'health, code, statistics, county, codes, zip, public, diego, harbison, crest': ['data.mo.gov'],\n",
       " 'health, disease, 000, per, rate, rates, chronic, risk, agency, due': ['healthdata.nj.gov'],\n",
       " 'health, san, county, care, counties, system, facilities, network, home, francisco': ['data.smcgov.org',\n",
       "  'health.data.ny.gov'],\n",
       " 'information, tab, projects, concerning, may, contains, project, provided, additional, improvement': ['data.austintexas.gov',\n",
       "  'data.lcb.wa.gov',\n",
       "  'data.pa.gov'],\n",
       " 'infrastructure, city, public, maps, works, dallas, los, angeles, types, way': ['data.albanyny.gov',\n",
       "  'data.jacksonms.gov',\n",
       "  'data.wa.gov',\n",
       "  'opencity.fcgov.com'],\n",
       " 'map, \"about\", shows, details, tracking, click, view, additional, weekly, information': ['data.sonomacounty.ca.gov',\n",
       "  'data.vta.org'],\n",
       " 'mateo, city, gis, land, county, parcel, map, point, survey, feature': ['data.cityofwestsacramento.org',\n",
       "  'data.detroitmi.gov'],\n",
       " 'measures, quality, performance, government, life, developed, fair, measurement, definitions, index': ['performance.smcgov.org',\n",
       "  'performance.topeka.org'],\n",
       " 'open, created, items, clear, volume, please, presented, difficult, potential, note': ['data.baltimorecity.gov'],\n",
       " 'plan, strategic, master, community, fewer, limitations, registry, cause, years, information': ['data.cincinnati-oh.gov',\n",
       "  'data.monroecounty-fl.gov'],\n",
       " 'planning, county, district, districts, boundaries, city, areas, zoning, gis, council': ['data.acgov.org',\n",
       "  'data.kcmo.org'],\n",
       " 'police, user, information, page, department, use, orleans, injury, may, records': ['data.cityofchicago.org',\n",
       "  'data.hartford.gov',\n",
       "  'data.menlopark.org',\n",
       "  'data.nola.gov',\n",
       "  'data.urbanaillinois.us'],\n",
       " 'politics, government, election, campaign, elections, commission, results, city, 460, finance': ['dashboard.plano.gov',\n",
       "  'data.ct.gov',\n",
       "  'data.hawaii.gov',\n",
       "  'data.kingcounty.gov',\n",
       "  'data.ny.gov',\n",
       "  'data.redmond.gov',\n",
       "  'datapoints.dallascityhall.com',\n",
       "  'electionsdata.kingcounty.gov',\n",
       "  'performance.adcogov.org',\n",
       "  'performance.glendaleca.gov',\n",
       "  'performance.seattle.gov',\n",
       "  'performance.vbgov.com',\n",
       "  'policeview.johnscreekga.gov'],\n",
       " 'population, county, demographics, age, total, king, washington, montgomery, number, years': ['data.cityofnewyork.us',\n",
       "  'data.illinois.gov',\n",
       "  'performance.carson.org',\n",
       "  'performance.princegeorgescountymd.gov'],\n",
       " 'program, employees, adult, employee, annual, names, full, abuse, nutrition, beginning': ['data.lacounty.gov',\n",
       "  'datacatalog.cookcountyil.gov'],\n",
       " 'property, consumption, real, value, properties, significant, easy, usage, assessment, company': ['data.douglas.co.us',\n",
       "  'data.qac.org',\n",
       "  'data.richmondgov.com'],\n",
       " 'public, safety, crime, police, department, illinois, race, reported, crimes, criminal': ['data.auburnwa.gov',\n",
       "  'data.greensboro-nc.gov',\n",
       "  'data.honolulu.gov',\n",
       "  'data.littlerock.gov',\n",
       "  'data.naperville.il.us',\n",
       "  'data.oaklandnet.com',\n",
       "  'data.providenceri.gov',\n",
       "  'data.readingpa.gov',\n",
       "  'data.roseville.ca.us',\n",
       "  'data.somervillema.gov',\n",
       "  'data.vbgov.com',\n",
       "  'opendata.miamidade.gov',\n",
       "  'performance.ci.janesville.wi.us',\n",
       "  'performance.cityofdavenportiowa.com',\n",
       "  'performance.fultoncountyga.gov',\n",
       "  'tni.princegeorgescountymd.gov'],\n",
       " 'public, safety, fire, month, calls, police, emergency, response, discharge, incident': ['data.brla.gov',\n",
       "  'data.cityofmadison.com',\n",
       "  'data.marincounty.org',\n",
       "  'data.raleighnc.gov',\n",
       "  'data.seattle.gov',\n",
       "  'kcstat.kcmo.org',\n",
       "  'opendata.ci.richmond.ca.us',\n",
       "  'opendata.howardcountymd.gov',\n",
       "  'performance.cityofrc.us',\n",
       "  'stats.cityofcarrollton.com',\n",
       "  'www.dallasopendata.com'],\n",
       " 'rates, calculated, rate, year, counts, average, seattle, percent, mortality, number': ['outcomestat.baltimorecity.gov'],\n",
       " 'recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access': ['data.srcity.org',\n",
       "  'performance.williamsburgva.gov'],\n",
       " \"report, performance, city, goal, resultsnola, administration, city's, management, key, topeka\": ['performance.austintexas.gov'],\n",
       " 'service, requests, inspection, inspections, 311, public, violations, request, complaints, days': ['data.delaware.gov',\n",
       "  'data.muni.org',\n",
       "  'data.princegeorgescountymd.gov'],\n",
       " 'state, new, michigan, york, information, check, measurements, jobs, licensing, ny': ['data.michigan.gov',\n",
       "  'data.oregon.gov',\n",
       "  'midashboard.michigan.gov'],\n",
       " 'transportation, plans, iowa, area, transit, operations, region, bus, priority, people': ['data.cityofgp.com',\n",
       "  'data.texas.gov'],\n",
       " 'transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike': ['data.cambridgema.gov',\n",
       "  'data.cityoftacoma.org',\n",
       "  'data.montgomerycountymd.gov',\n",
       "  'data.rutlandcitypolice.com',\n",
       "  'data.sfgov.org',\n",
       "  'data.weho.org',\n",
       "  'insights.cincinnati-oh.gov',\n",
       "  'performance.chattanooga.gov',\n",
       "  'reports.data.montgomerycountymd.gov'],\n",
       " 'water, environment, protection, tx, filed, waste, site, quality, environmental, monitoring': ['data.colorado.gov']}"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "raw_most_popular"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#turn this into a DataFrame for export\n",
    "pop_by_portal = pd.DataFrame([raw_most_popular])\n",
    "pop_by_portal = pop_by_portal.T\n",
    "pop_by_portal = pop_by_portal.reset_index()\n",
    "pop_by_portal.columns = ['Topic Content', 'Domains Where Topic Most Popular']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Topic Content</th>\n",
       "      <th>Domains Where Topic Most Popular</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities</td>\n",
       "      <td>[data.macoupincountyil.gov, data.tompsc.com, data.topeka.org, information.stpaul.gov, performance.danvilleva.gov, performance.franklintn.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified</td>\n",
       "      <td>[data.cityofgainesville.org, data.culvercity.org, data.oxnard.org, opendata.cityofhenderson.com]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>care, finance, provider, child, cost, plan, day, bay, government, fee</td>\n",
       "      <td>[data.datamontana.us]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>census, indicators, survey, includes, indicator, community, demographics, bureau, u, level</td>\n",
       "      <td>[data.livewellsd.org, opendata.cityofmesquite.com]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy</td>\n",
       "      <td>[dashboard.alexandriava.gov, data.countyofriverside.us, data.hampton.gov, data.maine.gov, stat.cityofgainesville.org]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal</td>\n",
       "      <td>[data.iowa.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>education, school, schools, students, public, governance, state, year, student, district</td>\n",
       "      <td>[dashboard.hawaii.gov, data.mass.gov, data.vermont.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities</td>\n",
       "      <td>[data.results.wa.gov, data.sustainablesm.org, performance.providenceri.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>finance, year, state, monthly, fiscal, payments, government, june, july, report</td>\n",
       "      <td>[data.maryland.gov, data.nj.gov, impact.stlouisco.com, opencheckbook.data.somervillema.gov, opendata.lasvegasnevada.gov, performance.cookcountyil.gov, transparency.michigan.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>financial, permits, building, permit, expenditures, guide, issued, construction, information, filter</td>\n",
       "      <td>[data.burlingtonvt.gov, data.cityofboston.gov, data.cityofevanston.org, data.cstx.gov, data.fortworthtexas.gov, data.lacity.org, data.montgomeryal.gov, data.nashville.gov, data.tuscaloosa.com]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>gas, natural, food, greenhouse, environment, emissions, costs, resources, program, fuel</td>\n",
       "      <td>[dashboard.slco.org, opendata.utah.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>health, code, statistics, county, codes, zip, public, diego, harbison, crest</td>\n",
       "      <td>[data.mo.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>health, disease, 000, per, rate, rates, chronic, risk, agency, due</td>\n",
       "      <td>[healthdata.nj.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>health, san, county, care, counties, system, facilities, network, home, francisco</td>\n",
       "      <td>[data.smcgov.org, health.data.ny.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>information, tab, projects, concerning, may, contains, project, provided, additional, improvement</td>\n",
       "      <td>[data.austintexas.gov, data.lcb.wa.gov, data.pa.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>infrastructure, city, public, maps, works, dallas, los, angeles, types, way</td>\n",
       "      <td>[data.albanyny.gov, data.jacksonms.gov, data.wa.gov, opencity.fcgov.com]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>map, \"about\", shows, details, tracking, click, view, additional, weekly, information</td>\n",
       "      <td>[data.sonomacounty.ca.gov, data.vta.org]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>mateo, city, gis, land, county, parcel, map, point, survey, feature</td>\n",
       "      <td>[data.cityofwestsacramento.org, data.detroitmi.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>measures, quality, performance, government, life, developed, fair, measurement, definitions, index</td>\n",
       "      <td>[performance.smcgov.org, performance.topeka.org]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>open, created, items, clear, volume, please, presented, difficult, potential, note</td>\n",
       "      <td>[data.baltimorecity.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>plan, strategic, master, community, fewer, limitations, registry, cause, years, information</td>\n",
       "      <td>[data.cincinnati-oh.gov, data.monroecounty-fl.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>planning, county, district, districts, boundaries, city, areas, zoning, gis, council</td>\n",
       "      <td>[data.acgov.org, data.kcmo.org]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>police, user, information, page, department, use, orleans, injury, may, records</td>\n",
       "      <td>[data.cityofchicago.org, data.hartford.gov, data.menlopark.org, data.nola.gov, data.urbanaillinois.us]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>politics, government, election, campaign, elections, commission, results, city, 460, finance</td>\n",
       "      <td>[dashboard.plano.gov, data.ct.gov, data.hawaii.gov, data.kingcounty.gov, data.ny.gov, data.redmond.gov, datapoints.dallascityhall.com, electionsdata.kingcounty.gov, performance.adcogov.org, performance.glendaleca.gov, performance.seattle.gov, per...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>population, county, demographics, age, total, king, washington, montgomery, number, years</td>\n",
       "      <td>[data.cityofnewyork.us, data.illinois.gov, performance.carson.org, performance.princegeorgescountymd.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>program, employees, adult, employee, annual, names, full, abuse, nutrition, beginning</td>\n",
       "      <td>[data.lacounty.gov, datacatalog.cookcountyil.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>property, consumption, real, value, properties, significant, easy, usage, assessment, company</td>\n",
       "      <td>[data.douglas.co.us, data.qac.org, data.richmondgov.com]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>public, safety, crime, police, department, illinois, race, reported, crimes, criminal</td>\n",
       "      <td>[data.auburnwa.gov, data.greensboro-nc.gov, data.honolulu.gov, data.littlerock.gov, data.naperville.il.us, data.oaklandnet.com, data.providenceri.gov, data.readingpa.gov, data.roseville.ca.us, data.somervillema.gov, data.vbgov.com, opendata.miami...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>public, safety, fire, month, calls, police, emergency, response, discharge, incident</td>\n",
       "      <td>[data.brla.gov, data.cityofmadison.com, data.marincounty.org, data.raleighnc.gov, data.seattle.gov, kcstat.kcmo.org, opendata.ci.richmond.ca.us, opendata.howardcountymd.gov, performance.cityofrc.us, stats.cityofcarrollton.com, www.dallasopendata....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>rates, calculated, rate, year, counts, average, seattle, percent, mortality, number</td>\n",
       "      <td>[outcomestat.baltimorecity.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>30</th>\n",
       "      <td>recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access</td>\n",
       "      <td>[data.srcity.org, performance.williamsburgva.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>report, performance, city, goal, resultsnola, administration, city's, management, key, topeka</td>\n",
       "      <td>[performance.austintexas.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>service, requests, inspection, inspections, 311, public, violations, request, complaints, days</td>\n",
       "      <td>[data.delaware.gov, data.muni.org, data.princegeorgescountymd.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>state, new, michigan, york, information, check, measurements, jobs, licensing, ny</td>\n",
       "      <td>[data.michigan.gov, data.oregon.gov, midashboard.michigan.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>34</th>\n",
       "      <td>transportation, plans, iowa, area, transit, operations, region, bus, priority, people</td>\n",
       "      <td>[data.cityofgp.com, data.texas.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>35</th>\n",
       "      <td>transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike</td>\n",
       "      <td>[data.cambridgema.gov, data.cityoftacoma.org, data.montgomerycountymd.gov, data.rutlandcitypolice.com, data.sfgov.org, data.weho.org, insights.cincinnati-oh.gov, performance.chattanooga.gov, reports.data.montgomerycountymd.gov]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>36</th>\n",
       "      <td>water, environment, protection, tx, filed, waste, site, quality, environmental, monitoring</td>\n",
       "      <td>[data.colorado.gov]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                   Topic Content  \\\n",
       "0                             budget, finance, fund, year, operating, city, fiscal, funds, calendar, communities   \n",
       "1                    business, license, job, contracts, businesses, vendor, owned, contract, licenses, certified   \n",
       "2                                          care, finance, provider, child, cost, plan, day, bay, government, fee   \n",
       "3                     census, indicators, survey, includes, indicator, community, demographics, bureau, u, level   \n",
       "4   development, housing, economic, community, infrastructure, managed, medicaid, department, buildings, economy   \n",
       "5            economy, employment, labor, unemployment, insurance, credit, workforce, economic, training, federal   \n",
       "6                       education, school, schools, students, public, governance, state, year, student, district   \n",
       "7                   energy, environment, electricity, air, sustainable, action, climate, city, clean, facilities   \n",
       "8                                finance, year, state, monthly, fiscal, payments, government, june, july, report   \n",
       "9           financial, permits, building, permit, expenditures, guide, issued, construction, information, filter   \n",
       "10                       gas, natural, food, greenhouse, environment, emissions, costs, resources, program, fuel   \n",
       "11                                  health, code, statistics, county, codes, zip, public, diego, harbison, crest   \n",
       "12                                            health, disease, 000, per, rate, rates, chronic, risk, agency, due   \n",
       "13                             health, san, county, care, counties, system, facilities, network, home, francisco   \n",
       "14             information, tab, projects, concerning, may, contains, project, provided, additional, improvement   \n",
       "15                                   infrastructure, city, public, maps, works, dallas, los, angeles, types, way   \n",
       "16                          map, \"about\", shows, details, tracking, click, view, additional, weekly, information   \n",
       "17                                           mateo, city, gis, land, county, parcel, map, point, survey, feature   \n",
       "18            measures, quality, performance, government, life, developed, fair, measurement, definitions, index   \n",
       "19                            open, created, items, clear, volume, please, presented, difficult, potential, note   \n",
       "20                   plan, strategic, master, community, fewer, limitations, registry, cause, years, information   \n",
       "21                          planning, county, district, districts, boundaries, city, areas, zoning, gis, council   \n",
       "22                               police, user, information, page, department, use, orleans, injury, may, records   \n",
       "23                  politics, government, election, campaign, elections, commission, results, city, 460, finance   \n",
       "24                     population, county, demographics, age, total, king, washington, montgomery, number, years   \n",
       "25                         program, employees, adult, employee, annual, names, full, abuse, nutrition, beginning   \n",
       "26                 property, consumption, real, value, properties, significant, easy, usage, assessment, company   \n",
       "27                         public, safety, crime, police, department, illinois, race, reported, crimes, criminal   \n",
       "28                          public, safety, fire, month, calls, police, emergency, response, discharge, incident   \n",
       "29                           rates, calculated, rate, year, counts, average, seattle, percent, mortality, number   \n",
       "30              recreation, parks, jersey, park, center, neighborhoods, centers, facilities, environment, access   \n",
       "31                 report, performance, city, goal, resultsnola, administration, city's, management, key, topeka   \n",
       "32                service, requests, inspection, inspections, 311, public, violations, request, complaints, days   \n",
       "33                             state, new, michigan, york, information, check, measurements, jobs, licensing, ny   \n",
       "34                         transportation, plans, iowa, area, transit, operations, region, bus, priority, people   \n",
       "35                  transportation, traffic, street, parking, infrastructure, city, safe, vehicle, streets, bike   \n",
       "36                    water, environment, protection, tx, filed, waste, site, quality, environmental, monitoring   \n",
       "\n",
       "                                                                                                                                                                                                                             Domains Where Topic Most Popular  \n",
       "0                                                                                                               [data.macoupincountyil.gov, data.tompsc.com, data.topeka.org, information.stpaul.gov, performance.danvilleva.gov, performance.franklintn.gov]  \n",
       "1                                                                                                                                                            [data.cityofgainesville.org, data.culvercity.org, data.oxnard.org, opendata.cityofhenderson.com]  \n",
       "2                                                                                                                                                                                                                                       [data.datamontana.us]  \n",
       "3                                                                                                                                                                                                          [data.livewellsd.org, opendata.cityofmesquite.com]  \n",
       "4                                                                                                                                       [dashboard.alexandriava.gov, data.countyofriverside.us, data.hampton.gov, data.maine.gov, stat.cityofgainesville.org]  \n",
       "5                                                                                                                                                                                                                                             [data.iowa.gov]  \n",
       "6                                                                                                                                                                                                     [dashboard.hawaii.gov, data.mass.gov, data.vermont.gov]  \n",
       "7                                                                                                                                                                                 [data.results.wa.gov, data.sustainablesm.org, performance.providenceri.gov]  \n",
       "8                                                                           [data.maryland.gov, data.nj.gov, impact.stlouisco.com, opencheckbook.data.somervillema.gov, opendata.lasvegasnevada.gov, performance.cookcountyil.gov, transparency.michigan.gov]  \n",
       "9                                                            [data.burlingtonvt.gov, data.cityofboston.gov, data.cityofevanston.org, data.cstx.gov, data.fortworthtexas.gov, data.lacity.org, data.montgomeryal.gov, data.nashville.gov, data.tuscaloosa.com]  \n",
       "10                                                                                                                                                                                                                    [dashboard.slco.org, opendata.utah.gov]  \n",
       "11                                                                                                                                                                                                                                              [data.mo.gov]  \n",
       "12                                                                                                                                                                                                                                        [healthdata.nj.gov]  \n",
       "13                                                                                                                                                                                                                      [data.smcgov.org, health.data.ny.gov]  \n",
       "14                                                                                                                                                                                                       [data.austintexas.gov, data.lcb.wa.gov, data.pa.gov]  \n",
       "15                                                                                                                                                                                   [data.albanyny.gov, data.jacksonms.gov, data.wa.gov, opencity.fcgov.com]  \n",
       "16                                                                                                                                                                                                                   [data.sonomacounty.ca.gov, data.vta.org]  \n",
       "17                                                                                                                                                                                                        [data.cityofwestsacramento.org, data.detroitmi.gov]  \n",
       "18                                                                                                                                                                                                           [performance.smcgov.org, performance.topeka.org]  \n",
       "19                                                                                                                                                                                                                                   [data.baltimorecity.gov]  \n",
       "20                                                                                                                                                                                                         [data.cincinnati-oh.gov, data.monroecounty-fl.gov]  \n",
       "21                                                                                                                                                                                                                            [data.acgov.org, data.kcmo.org]  \n",
       "22                                                                                                                                                     [data.cityofchicago.org, data.hartford.gov, data.menlopark.org, data.nola.gov, data.urbanaillinois.us]  \n",
       "23  [dashboard.plano.gov, data.ct.gov, data.hawaii.gov, data.kingcounty.gov, data.ny.gov, data.redmond.gov, datapoints.dallascityhall.com, electionsdata.kingcounty.gov, performance.adcogov.org, performance.glendaleca.gov, performance.seattle.gov, per...  \n",
       "24                                                                                                                                                  [data.cityofnewyork.us, data.illinois.gov, performance.carson.org, performance.princegeorgescountymd.gov]  \n",
       "25                                                                                                                                                                                                          [data.lacounty.gov, datacatalog.cookcountyil.gov]  \n",
       "26                                                                                                                                                                                                   [data.douglas.co.us, data.qac.org, data.richmondgov.com]  \n",
       "27  [data.auburnwa.gov, data.greensboro-nc.gov, data.honolulu.gov, data.littlerock.gov, data.naperville.il.us, data.oaklandnet.com, data.providenceri.gov, data.readingpa.gov, data.roseville.ca.us, data.somervillema.gov, data.vbgov.com, opendata.miami...  \n",
       "28  [data.brla.gov, data.cityofmadison.com, data.marincounty.org, data.raleighnc.gov, data.seattle.gov, kcstat.kcmo.org, opendata.ci.richmond.ca.us, opendata.howardcountymd.gov, performance.cityofrc.us, stats.cityofcarrollton.com, www.dallasopendata....  \n",
       "29                                                                                                                                                                                                                            [outcomestat.baltimorecity.gov]  \n",
       "30                                                                                                                                                                                                          [data.srcity.org, performance.williamsburgva.gov]  \n",
       "31                                                                                                                                                                                                                              [performance.austintexas.gov]  \n",
       "32                                                                                                                                                                                         [data.delaware.gov, data.muni.org, data.princegeorgescountymd.gov]  \n",
       "33                                                                                                                                                                                             [data.michigan.gov, data.oregon.gov, midashboard.michigan.gov]  \n",
       "34                                                                                                                                                                                                                        [data.cityofgp.com, data.texas.gov]  \n",
       "35                        [data.cambridgema.gov, data.cityoftacoma.org, data.montgomerycountymd.gov, data.rutlandcitypolice.com, data.sfgov.org, data.weho.org, insights.cincinnati-oh.gov, performance.chattanooga.gov, reports.data.montgomerycountymd.gov]  \n",
       "36                                                                                                                                                                                                                                        [data.colorado.gov]  "
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.set_option('display.max_colwidth', 250)\n",
    "pop_by_portal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This can provide some interesting insights, but beware of using it for portals that have only a handful of open datasets. Using the raw total views and downloads can be misleading!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#pop_by_portal.to_csv('most_popular_topics_by_portal_by_raw_totals.csv') #export"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A function to return rows with a given topic number present:\n",
    "- We can use this to view datasets where a given topic wins."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def return_data_by_top_topic(df, topic_num):\n",
    "    index_list = []\n",
    "    for index, tup_list in enumerate(df.topic_comp):\n",
    "        if (any(topic_num in tup for tup in tup_list)): #if a topic num is in topic_comp\n",
    "            win_tuple = max(tup_list, key=lambda item:item[1]) #call winning tuple\n",
    "            if win_tuple[0] == topic_num: #if winning tuple is the topic we want\n",
    "                index_list.append(index) #append index number to list\n",
    "    \n",
    "    df = df.iloc[index_list] #then just select with this list\n",
    "    return df  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.reset_option('display.max_colwidth') #reset our display to default"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>name</th>\n",
       "      <th>description</th>\n",
       "      <th>attribution</th>\n",
       "      <th>columns_field_name</th>\n",
       "      <th>columns_name</th>\n",
       "      <th>type</th>\n",
       "      <th>categories</th>\n",
       "      <th>domain_category</th>\n",
       "      <th>domain_tags</th>\n",
       "      <th>provenance</th>\n",
       "      <th>download_count</th>\n",
       "      <th>page_views_last_month</th>\n",
       "      <th>page_views_last_week</th>\n",
       "      <th>page_views_total</th>\n",
       "      <th>page_views_total_log</th>\n",
       "      <th>domain</th>\n",
       "      <th>mash</th>\n",
       "      <th>big_mash</th>\n",
       "      <th>topic_comp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>21</td>\n",
       "      <td>DCHS_Safety_DVSA Hotline Calls</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual_assault_hotline_calls', 'other_servic...</td>\n",
       "      <td>['Sexual Assault Hotline Calls', 'Other Servic...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>6.000</td>\n",
       "      <td>27.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>190.000</td>\n",
       "      <td>7.577</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.1616786051), (26, 0.431318681319), (50...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>22</td>\n",
       "      <td>Exit Plan for Leaving Shelter</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual_assault_hotline_calls', 'percent_of_c...</td>\n",
       "      <td>['Sexual Assault Hotline Calls', 'Percent of c...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>4.000</td>\n",
       "      <td>18.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>189.000</td>\n",
       "      <td>7.570</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.161766956351), (26, 0.431318681319), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>88</td>\n",
       "      <td>Total People Reached Through Community Engagem...</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['percent_of_participants_reported_that_they_l...</td>\n",
       "      <td>['Percent of participants reported that they l...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>official</td>\n",
       "      <td>3.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>67.000</td>\n",
       "      <td>6.087</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>[(16, 0.203846153846), (26, 0.403846153846), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>101</th>\n",
       "      <td>101</td>\n",
       "      <td>Length of Stay at Shelter</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['unknown', 'left_area', 'residence_of_friend_...</td>\n",
       "      <td>['Unknown', 'Left Area', 'Residence of friend/...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>11.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>58.000</td>\n",
       "      <td>5.883</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.162007823798), (26, 0.431318681319), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>160</td>\n",
       "      <td>Adults and Children Sheltered due to Domestic ...</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['residence_of_friend_relative', 'transitional...</td>\n",
       "      <td>['Residence of friend/relative', 'Transitional...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>29.000</td>\n",
       "      <td>4.907</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.161854234981), (26, 0.431318681319), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>161</th>\n",
       "      <td>161</td>\n",
       "      <td>DCHS_SA_Domestic Violence Shelter Data</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['over_90_days', 'length_of_stay_less_than_24_...</td>\n",
       "      <td>['Over 90 days', 'Less than 24 hours', 'Reside...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>29.000</td>\n",
       "      <td>4.907</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.161685334512), (26, 0.431318681319), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>231</th>\n",
       "      <td>231</td>\n",
       "      <td>DCHS_Safety_DVSA Hotline Calls by Type</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['children_sheltered_due_to_domestic_violence'...</td>\n",
       "      <td>['Children sheltered due to Domestic Violence'...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['sexual assault', 'domestic violence', 'safet...</td>\n",
       "      <td>official</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>10.000</td>\n",
       "      <td>3.459</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>['sexual', 'assault', 'domestic', 'violence', ...</td>\n",
       "      <td>[(16, 0.161934869861), (26, 0.431318681319), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>239</th>\n",
       "      <td>239</td>\n",
       "      <td>DCHS_SA_Domestic Violence And Sexual Assault C...</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['total_people_reached', 'total_hours_of_activ...</td>\n",
       "      <td>['Total People Reached', 'Total Hours of Activ...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>official</td>\n",
       "      <td>2.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>3.322</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>[(16, 0.203846153846), (26, 0.403846153846), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>252</th>\n",
       "      <td>252</td>\n",
       "      <td>Percent of Participants who Report that they L...</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['race_of_participants_caucasian', 'race_of_pa...</td>\n",
       "      <td>['Race of Participants - Caucasian', 'Race of ...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>official</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>7.000</td>\n",
       "      <td>3.000</td>\n",
       "      <td>dashboard.alexandriava.gov</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>['safety', 'sa', 'dv', 'dchs']</td>\n",
       "      <td>[(16, 0.203846153846), (26, 0.403846153846), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>272</th>\n",
       "      <td>272</td>\n",
       "      <td>OHA Baseline Target - Viable Land Base</td>\n",
       "      <td>['OHA', 'Strategic', 'Results', '(2010', '2018...</td>\n",
       "      <td>Field Note: Baseline Hawai'i State, Office of ...</td>\n",
       "      <td>['annual_target_population', 'fiscal_year', 'y...</td>\n",
       "      <td>['Annual Target Population ', 'Fiscal Year', '...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>Individual Rights</td>\n",
       "      <td>['baseline', 'land', 'oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>90.000</td>\n",
       "      <td>118.000</td>\n",
       "      <td>23.000</td>\n",
       "      <td>9494.000</td>\n",
       "      <td>13.213</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['baseline', 'land', 'oha', 'individual', 'rig...</td>\n",
       "      <td>['baseline', 'land', 'oha', 'individual', 'rig...</td>\n",
       "      <td>[(4, 0.0445430353893), (9, 0.13570023278), (21...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>275</th>\n",
       "      <td>275</td>\n",
       "      <td>OHA Baseline - Value History &amp; Culture</td>\n",
       "      <td>['85%', 'of', 'Hawaiʻi', 'residents', 'appreci...</td>\n",
       "      <td>OHA</td>\n",
       "      <td>['annual_target_hi_adult_resident', 'oha_targe...</td>\n",
       "      <td>['Annual Target (HI adult resident)', 'OHA Tar...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>Culture and Recreation</td>\n",
       "      <td>['culture', 'hawaiian', 'history', 'oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>51.000</td>\n",
       "      <td>24.000</td>\n",
       "      <td>3.000</td>\n",
       "      <td>4640.000</td>\n",
       "      <td>12.180</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['culture', 'hawaiian', 'history', 'oha', 'cul...</td>\n",
       "      <td>['culture', 'hawaiian', 'history', 'oha', 'cul...</td>\n",
       "      <td>[(1, 0.137036692987), (5, 0.188701923077), (6,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>286</th>\n",
       "      <td>286</td>\n",
       "      <td>Hawai'i Homeless Counts by Island, 2010-2013</td>\n",
       "      <td>[]</td>\n",
       "      <td>Source: PIT, 2013.</td>\n",
       "      <td>['homeless_count', 'county', 'year', 'hawai_i'...</td>\n",
       "      <td>['Maui', \"O'ahu\", 'Year', \"Hawai'i\", \"Kaua'i\"]</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>Social Services</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>34.000</td>\n",
       "      <td>30.000</td>\n",
       "      <td>3.000</td>\n",
       "      <td>344.000</td>\n",
       "      <td>8.430</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha', 'so...</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha', 'so...</td>\n",
       "      <td>[(17, 0.145604395604), (18, 0.145604395604), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>316</th>\n",
       "      <td>316</td>\n",
       "      <td>State of Hawaiʻi Population by Race/Ethnicity:...</td>\n",
       "      <td>[]</td>\n",
       "      <td>Source. Hawaiʻi, Department of Health, Hawaiʻi...</td>\n",
       "      <td>['population_percent', 'population_number', 'r...</td>\n",
       "      <td>['Population (Percent)', 'Population (Number)'...</td>\n",
       "      <td>chart</td>\n",
       "      <td>[]</td>\n",
       "      <td>Other</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>official</td>\n",
       "      <td>47.000</td>\n",
       "      <td>21.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>257.000</td>\n",
       "      <td>8.011</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>[(2, 0.169871794872), (26, 0.503205128204), (4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>340</th>\n",
       "      <td>340</td>\n",
       "      <td>Decibel Level of Civil Defense Sirens</td>\n",
       "      <td>['Includes', 'decibel', 'levels', 'and', 'prec...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['decibel', 'location', 'unique_id', 'location...</td>\n",
       "      <td>['DECIBEL', 'LOCATION', 'UNIQUE_ID', 'Location...</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>21.000</td>\n",
       "      <td>21.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>219.000</td>\n",
       "      <td>7.781</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>['public', 'safety', 'Includes', 'decibel', 'l...</td>\n",
       "      <td>[(5, 0.33547008547), (26, 0.557692307692)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>358</th>\n",
       "      <td>358</td>\n",
       "      <td>TANF Caseload (By Selected Months)</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['of_employed_households', 'gross_earned_incom...</td>\n",
       "      <td>['# of Employed Households', 'Gross Earned Inc...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>45.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>201.000</td>\n",
       "      <td>7.658</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>[(26, 0.673076923077)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>394</th>\n",
       "      <td>394</td>\n",
       "      <td>Health - Obesity (OHA)</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['date', 'hp2020_benchmark', 'oha_benchmark', ...</td>\n",
       "      <td>['Date', 'HP2020 Benchmark', 'OHA Benchmark', ...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>49.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>170.000</td>\n",
       "      <td>7.418</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>[(26, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>431</th>\n",
       "      <td>431</td>\n",
       "      <td>OHA Baseline Target - Viable Land Base</td>\n",
       "      <td>['OHA', 'Strategic', 'Results', '(2010', '2018...</td>\n",
       "      <td>Field Note: Baseline Hawai'i State, Office of ...</td>\n",
       "      <td>['year', 'annual_target_population', 'fiscal_y...</td>\n",
       "      <td>['Date', 'Annual Target Population ', 'Fiscal ...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Individual Rights</td>\n",
       "      <td>['baseline', 'land', 'oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>51.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>148.000</td>\n",
       "      <td>7.219</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['baseline', 'land', 'oha', 'individual', 'rig...</td>\n",
       "      <td>['baseline', 'land', 'oha', 'individual', 'rig...</td>\n",
       "      <td>[(4, 0.0444289989737), (9, 0.13576160706), (21...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>437</th>\n",
       "      <td>437</td>\n",
       "      <td>Public Safety Budget Data</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['total_operating', 'fiscal_year', 'total_capi...</td>\n",
       "      <td>['Total Operating', 'Fiscal Year', 'Total Capi...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>146.000</td>\n",
       "      <td>7.200</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>[(26, 0.673076923077)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>444</th>\n",
       "      <td>444</td>\n",
       "      <td>Map Of Civil Defense Sirens</td>\n",
       "      <td>['Includes', 'decibel', 'levels', 'and', 'prec...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['location', 'decibel', 'county', 'location_1'...</td>\n",
       "      <td>['LOCATION', 'DECIBEL', 'COUNTY', 'Location 1'...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>26.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>143.000</td>\n",
       "      <td>7.170</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['public', 'safety']</td>\n",
       "      <td>['public', 'safety', 'Includes', 'decibel', 'l...</td>\n",
       "      <td>[(5, 0.33547008547), (26, 0.557692307692)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>459</th>\n",
       "      <td>459</td>\n",
       "      <td>WIC Clinic Participation 2010-2013</td>\n",
       "      <td>[]</td>\n",
       "      <td>Kayla Rosenfeld, DHS; WIC services in Hawaii, ...</td>\n",
       "      <td>['total_served', 'infants', 'women', 'children...</td>\n",
       "      <td>['Total Served', 'Infants', 'Women', 'Children...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>A New Day for: Safety Net, Homelessness, Publi...</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>136.000</td>\n",
       "      <td>7.098</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['a', 'new', 'day', 'for:', 'safety', 'net', '...</td>\n",
       "      <td>['a', 'new', 'day', 'for:', 'safety', 'net', '...</td>\n",
       "      <td>[(26, 0.33547008547), (29, 0.113247863248), (4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>503</th>\n",
       "      <td>503</td>\n",
       "      <td>Hawai'i Homeless Counts by Island, 2010-2013</td>\n",
       "      <td>[]</td>\n",
       "      <td>Source: PIT, 2013.</td>\n",
       "      <td>['homeless_count', 'kaua_i', 'year', 'hawai_i'...</td>\n",
       "      <td>['Maui', \"Kaua'i\", 'Year', \"Hawai'i\", \"O'ahu\"]</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Social Services</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>22.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>120.000</td>\n",
       "      <td>6.919</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha', 'so...</td>\n",
       "      <td>['hawaiian', 'homeless', 'housing', 'oha', 'so...</td>\n",
       "      <td>[(17, 0.145604395604), (18, 0.145604395604), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>517</th>\n",
       "      <td>517</td>\n",
       "      <td>Women's Health Clinics</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['clinic', 'phone', 'location_1']</td>\n",
       "      <td>['Clinic', 'Phone', 'Location 1']</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>A New Day for: Safety Net, Homelessness, Publi...</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>24.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>117.000</td>\n",
       "      <td>6.883</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['a', 'new', 'day', 'for:', 'safety', 'net', '...</td>\n",
       "      <td>['a', 'new', 'day', 'for:', 'safety', 'net', '...</td>\n",
       "      <td>[(26, 0.33547008547), (29, 0.113247863248), (4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>523</th>\n",
       "      <td>523</td>\n",
       "      <td>Health - Prenatal (OHA)</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['unknown_n', 'oha_benchmark', 'hp2020_benchma...</td>\n",
       "      <td>['Unknown (N)', 'OHA Benchmark', 'HP2020 Bench...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>official</td>\n",
       "      <td>28.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>113.000</td>\n",
       "      <td>6.833</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>['oha']</td>\n",
       "      <td>[(26, 0.509615384615)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>534</th>\n",
       "      <td>534</td>\n",
       "      <td>State of Hawaiʻi Population by Race/Ethnicity:...</td>\n",
       "      <td>[]</td>\n",
       "      <td>Source. Hawaiʻi, Department of Health, Hawaiʻi...</td>\n",
       "      <td>['population_number', 'race_ethnicity', 'popul...</td>\n",
       "      <td>['Population (Number)', 'Race/Ethnicity ', 'Po...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Other</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>official</td>\n",
       "      <td>24.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>109.000</td>\n",
       "      <td>6.781</td>\n",
       "      <td>dashboard.hawaii.gov</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>['hawaiian', 'oha', 'population', 'race', 'sta...</td>\n",
       "      <td>[(2, 0.169871794872), (26, 0.503205128205), (4...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>568</th>\n",
       "      <td>568</td>\n",
       "      <td>Fire / EMS Dispatches</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['med_inj_processing', 'fire_ems_disp_avg', 'f...</td>\n",
       "      <td>['PFR Med/Inj Processing 90th', 'Fire / EMS Di...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>38.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>142.000</td>\n",
       "      <td>7.160</td>\n",
       "      <td>dashboard.plano.gov</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>[(2, 0.254807692308), (26, 0.504807692308)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>575</th>\n",
       "      <td>575</td>\n",
       "      <td>EMS Dispatches</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['fire', '_911_via_telematics', 'voip', '_911_...</td>\n",
       "      <td>['% Fire', '911 via Telematics', '% VoiP', '91...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>33.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>108.000</td>\n",
       "      <td>6.768</td>\n",
       "      <td>dashboard.plano.gov</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>[(2, 0.254807692308), (26, 0.504807692308)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>583</th>\n",
       "      <td>583</td>\n",
       "      <td>EMS Dispatches</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['fire', '_911_via_telematics', '_911_via_voip...</td>\n",
       "      <td>['% Fire', '911 via Telematics', '911 via VoiP...</td>\n",
       "      <td>chart</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>25.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>99.000</td>\n",
       "      <td>6.644</td>\n",
       "      <td>dashboard.plano.gov</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>['public', 'safety', 'health']</td>\n",
       "      <td>[(2, 0.254807692308), (26, 0.504807692308)]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>614</th>\n",
       "      <td>614</td>\n",
       "      <td>Index Crime Rate</td>\n",
       "      <td>['Salt', 'Lake', 'County', 'crime', 'rates', '...</td>\n",
       "      <td>Utah Department of Public Safety</td>\n",
       "      <td>['year2', 'crime_rate']</td>\n",
       "      <td>['Year2', 'CRIME RATE']</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['index crime', 'crime rate', 'rate', 'crime']</td>\n",
       "      <td>official</td>\n",
       "      <td>602.000</td>\n",
       "      <td>7.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>27150.000</td>\n",
       "      <td>14.729</td>\n",
       "      <td>dashboard.slco.org</td>\n",
       "      <td>['index', 'crime', 'crime', 'rate', 'rate', 'c...</td>\n",
       "      <td>['index', 'crime', 'crime', 'rate', 'rate', 'c...</td>\n",
       "      <td>[(0, 0.0728021978022), (2, 0.135665761033), (7...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>632</th>\n",
       "      <td>632</td>\n",
       "      <td>State Index Crime Rate</td>\n",
       "      <td>['State', 'of', 'Utah', 'crime', 'rates', 'sta...</td>\n",
       "      <td>Utah Department of Public Safety</td>\n",
       "      <td>['date', 'state_crime_rate']</td>\n",
       "      <td>['DATE', 'STATE CRIME RATE']</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>['crime rate', 'index crimes', 'utah']</td>\n",
       "      <td>official</td>\n",
       "      <td>583.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>26585.000</td>\n",
       "      <td>14.698</td>\n",
       "      <td>dashboard.slco.org</td>\n",
       "      <td>['crime', 'rate', 'index', 'crimes', 'utah']</td>\n",
       "      <td>['crime', 'rate', 'index', 'crimes', 'utah', '...</td>\n",
       "      <td>[(2, 0.0849358974359), (7, 0.168269230769), (2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>655</th>\n",
       "      <td>655</td>\n",
       "      <td>Alameda County Sheriff Crime Reports</td>\n",
       "      <td>['Alameda', 'County', 'Sheriff', 'Crime', 'Rep...</td>\n",
       "      <td>Alameda County Sheriff's Office</td>\n",
       "      <td>['crimedescription', 'location_1', 'datetime',...</td>\n",
       "      <td>['CrimeDescription', 'Location 1', 'DateTime',...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>1111.000</td>\n",
       "      <td>1395.000</td>\n",
       "      <td>310.000</td>\n",
       "      <td>20065.000</td>\n",
       "      <td>14.292</td>\n",
       "      <td>data.acgov.org</td>\n",
       "      <td>['health', 'public', 'safety']</td>\n",
       "      <td>['health', 'public', 'safety', 'Alameda', 'Cou...</td>\n",
       "      <td>[(2, 0.131321120447), (23, 0.106439139191), (2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4262</th>\n",
       "      <td>4262</td>\n",
       "      <td>Uniform Crime Reporting System Offenses by Dep...</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['robbery_gas_service_station_value_stolen', '...</td>\n",
       "      <td>['Robbery Gas/Service_Station Value_Stolen', '...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>738.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>494.000</td>\n",
       "      <td>8.951</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630686980655), (8, 0.0202852809573), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4286</th>\n",
       "      <td>4286</td>\n",
       "      <td>Uniform Crime Reporting System Arrests in 2012</td>\n",
       "      <td>['Arrests', 'are', 'the', 'number', 'of', 'per...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['colchester_csp', 'torrington_pd', 'tolland_c...</td>\n",
       "      <td>['Colchester CSP', 'Torrington PD', 'Tolland C...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['arrests', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>580.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>453.000</td>\n",
       "      <td>8.827</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>[(9, 0.101876589899), (23, 0.0512549258423), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4287</th>\n",
       "      <td>4287</td>\n",
       "      <td>Police Departments</td>\n",
       "      <td>['A', 'listing', 'of', 'State', 'Municipal', '...</td>\n",
       "      <td>Department of Emergency Service and Public Pro...</td>\n",
       "      <td>['telephone', 'address', 'location', 'zip', 'c...</td>\n",
       "      <td>['TELEPHONE', 'ADDRESS', 'Location', 'ZIP', 'C...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['crime', 'departments', 'police', 'public saf...</td>\n",
       "      <td>official</td>\n",
       "      <td>878.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>453.000</td>\n",
       "      <td>8.827</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['crime', 'departments', 'police', 'public', '...</td>\n",
       "      <td>['crime', 'departments', 'police', 'public', '...</td>\n",
       "      <td>[(25, 0.0533695146241), (26, 0.374136972605), ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4290</th>\n",
       "      <td>4290</td>\n",
       "      <td>Uniform Crime Reporting System Offenses 2011</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['milford_pd', 'hamden_pd', 'westbrook_csp', '...</td>\n",
       "      <td>['Milford PD', 'Hamden PD', 'Westbrook CSP', '...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>674.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>450.000</td>\n",
       "      <td>8.817</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630712893518), (8, 0.020285294677), (1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4292</th>\n",
       "      <td>4292</td>\n",
       "      <td>Uniform Crime Reporting System Offenses By Dep...</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['arson_other_value_damaged', 'smasslt_offense...</td>\n",
       "      <td>['Arson Other Value_Damaged', 'SmAsslt Offense...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['2013', 'offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>577.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>446.000</td>\n",
       "      <td>8.804</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['2013', 'offenses', 'crime', 'ucr', 'public',...</td>\n",
       "      <td>['2013', 'offenses', 'crime', 'ucr', 'public',...</td>\n",
       "      <td>[(5, 0.0630705946552), (8, 0.0202853012731), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4294</th>\n",
       "      <td>4294</td>\n",
       "      <td>Uniform Crime Reporting System Arrests in  2010</td>\n",
       "      <td>['Arrests', 'are', 'the', 'number', 'of', 'per...</td>\n",
       "      <td>Department of Emegerncy Service and Public Pro...</td>\n",
       "      <td>['winchester_pd', 'colchester_csp', 'woodbridg...</td>\n",
       "      <td>['Winchester PD', 'Colchester CSP', 'Woodbridg...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['arrests', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>553.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>445.000</td>\n",
       "      <td>8.801</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>[(9, 0.101877187909), (23, 0.0512551214156), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4302</th>\n",
       "      <td>4302</td>\n",
       "      <td>Number of Law Enforcement Personnel by Departm...</td>\n",
       "      <td>['The', 'number', 'of', 'sworn', 'and', 'civil...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['department', 'sworn_male', 'civilian_female'...</td>\n",
       "      <td>['Department', 'Sworn Male ', 'Civilian Female...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['public safety', 'personnel', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>584.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>433.000</td>\n",
       "      <td>8.762</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['public', 'safety', 'personnel', 'police', 'p...</td>\n",
       "      <td>['public', 'safety', 'personnel', 'police', 'p...</td>\n",
       "      <td>[(15, 0.179307951581), (16, 0.0578863990146), ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4329</th>\n",
       "      <td>4329</td>\n",
       "      <td>Uniform Crime Reporting System Offenses in 2010</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['enfield_pd', 'csp_total', 'foxwoods_casino_c...</td>\n",
       "      <td>['Enfield PD', 'CSP Total', 'Foxwoods Casino C...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>702.000</td>\n",
       "      <td>9.000</td>\n",
       "      <td>3.000</td>\n",
       "      <td>381.000</td>\n",
       "      <td>8.577</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630710354293), (8, 0.0202852968573), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4362</th>\n",
       "      <td>4362</td>\n",
       "      <td>CHART : LICENSE ENDORSEMENTS as of NOV 2013</td>\n",
       "      <td>[]</td>\n",
       "      <td>DEPARTMENT OF MOTOR VEHICLE</td>\n",
       "      <td>['total', 'license_endosrements', 'year']</td>\n",
       "      <td>['TOTAL', 'LICENSE ENDOSREMENTS', 'YEAR']</td>\n",
       "      <td>chart</td>\n",
       "      <td>['economy', 'public safety']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['dmv']</td>\n",
       "      <td>official</td>\n",
       "      <td>77.000</td>\n",
       "      <td>13.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>329.000</td>\n",
       "      <td>8.366</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>[(2, 0.169871794872), (3, 0.169871794872), (16...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4388</th>\n",
       "      <td>4388</td>\n",
       "      <td>CHART : LICENSE ENDORSEMENTS as of NOV 2014</td>\n",
       "      <td>[]</td>\n",
       "      <td>DEPARTMENT OF MOTOR VEHICLE</td>\n",
       "      <td>['year', 'total', 'license_endosrements']</td>\n",
       "      <td>['YEAR', 'TOTAL', 'LICENSE ENDOSREMENTS']</td>\n",
       "      <td>chart</td>\n",
       "      <td>['economy', 'public safety']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['dmv']</td>\n",
       "      <td>official</td>\n",
       "      <td>69.000</td>\n",
       "      <td>14.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>291.000</td>\n",
       "      <td>8.190</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>[(2, 0.169871794872), (3, 0.169871794872), (16...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4399</th>\n",
       "      <td>4399</td>\n",
       "      <td>Uniform Crime Reporting System Offenses By Dep...</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['ngmansl_clearances', 'murder_relationship_st...</td>\n",
       "      <td>['NgMansl Clearances', 'Murder Relationship St...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>661.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>278.000</td>\n",
       "      <td>8.124</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630705851969), (8, 0.0202852865935), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4407</th>\n",
       "      <td>4407</td>\n",
       "      <td>Uniform Crime Reporting System Offenses 2012</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['brooklyn_csp', 'stonington_pd', 'stamford_pd...</td>\n",
       "      <td>['Brooklyn CSP', 'Stonington PD', 'Stamford PD...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['crime', 'offenses', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>644.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>268.000</td>\n",
       "      <td>8.071</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['crime', 'offenses', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['crime', 'offenses', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630721336619), (8, 0.0202853058285), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4431</th>\n",
       "      <td>4431</td>\n",
       "      <td>City of New Haven Police Stations</td>\n",
       "      <td>['Locations', 'of', 'Police', 'Stations', 'in'...</td>\n",
       "      <td>City of New Haven</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['police', 'law enforcement', 'new haven', 'nhv']</td>\n",
       "      <td>official</td>\n",
       "      <td>365.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>230.000</td>\n",
       "      <td>7.852</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['police', 'law', 'enforcement', 'new', 'haven...</td>\n",
       "      <td>['police', 'law', 'enforcement', 'new', 'haven...</td>\n",
       "      <td>[(5, 0.0784023668639), (26, 0.539940828397), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4439</th>\n",
       "      <td>4439</td>\n",
       "      <td>Vehicle Crash Data Repository - CT CRASH</td>\n",
       "      <td>['The', 'Connecticut', 'Crash', 'Data', 'Repos...</td>\n",
       "      <td>Department of Transportation</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>href</td>\n",
       "      <td>['transportation', 'social services']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['accidents', 'crash', 'usodcensus', 'vehicle']</td>\n",
       "      <td>official</td>\n",
       "      <td>36.000</td>\n",
       "      <td>12.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>223.000</td>\n",
       "      <td>7.807</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['accidents', 'crash', 'usodcensus', 'vehicle'...</td>\n",
       "      <td>['accidents', 'crash', 'usodcensus', 'vehicle'...</td>\n",
       "      <td>[(3, 0.0701899402726), (7, 0.0131257766537), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4451</th>\n",
       "      <td>4451</td>\n",
       "      <td>CHART : LICENSE ENDORSEMENTS as of NOV 2012</td>\n",
       "      <td>[]</td>\n",
       "      <td>DEPARTMENT OF MOTOR VEHICLE</td>\n",
       "      <td>['year', 'total', 'license_endosrements']</td>\n",
       "      <td>['YEAR', 'TOTAL', 'LICENSE ENDOSREMENTS']</td>\n",
       "      <td>chart</td>\n",
       "      <td>['economy', 'public safety']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['dmv']</td>\n",
       "      <td>official</td>\n",
       "      <td>35.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>1.000</td>\n",
       "      <td>210.000</td>\n",
       "      <td>7.721</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>['dmv', 'economy', 'public', 'safety', 'transp...</td>\n",
       "      <td>[(2, 0.169871794872), (3, 0.169871794872), (16...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4478</th>\n",
       "      <td>4478</td>\n",
       "      <td>Commercial Vehicle Safety Division- Consumer C...</td>\n",
       "      <td>['CVSD', 'Consumer', 'Complaint', 'Cases', 'be...</td>\n",
       "      <td>Department of Motor Vehicles</td>\n",
       "      <td>['cases_closed', 'year', 'month', 'cases_open'...</td>\n",
       "      <td>['Cases Closed', 'Year', 'Month', 'Cases Open'...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>Transportation</td>\n",
       "      <td>['dmv']</td>\n",
       "      <td>official</td>\n",
       "      <td>649.000</td>\n",
       "      <td>7.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>147.000</td>\n",
       "      <td>7.209</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['dmv', 'public', 'safety', 'transportation']</td>\n",
       "      <td>['dmv', 'public', 'safety', 'transportation', ...</td>\n",
       "      <td>[(2, 0.0926573426573), (3, 0.0926573426573), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4524</th>\n",
       "      <td>4524</td>\n",
       "      <td>Uniform Crime Reporting System Arrests 2015</td>\n",
       "      <td>['Arrests', 'are', 'the', 'number', 'of', 'per...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['urban', 'tolland_county', 'w_c_s_u', 's_c_s_...</td>\n",
       "      <td>['Urban', 'Tolland County', 'W.C.S.U.', 'S.C.S...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['arrests', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>327.000</td>\n",
       "      <td>14.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>59.000</td>\n",
       "      <td>5.907</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>['arrests', 'crime', 'ucr', 'public', 'safety'...</td>\n",
       "      <td>[(2, 0.0674580972245), (8, 0.0351770567299), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4528</th>\n",
       "      <td>4528</td>\n",
       "      <td>Uniform Crime Reporting System Offenses By Dep...</td>\n",
       "      <td>['The', 'Uniform', 'Crime', 'Reporting', 'Prog...</td>\n",
       "      <td>Department of Emergency Services and Public Pr...</td>\n",
       "      <td>['murder_relationship_live_in_male', 'murder_r...</td>\n",
       "      <td>['Murder Relationship Live-In, Male', 'Murder ...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety', 'health']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['offenses', 'crime', 'ucr']</td>\n",
       "      <td>official</td>\n",
       "      <td>369.000</td>\n",
       "      <td>11.000</td>\n",
       "      <td>4.000</td>\n",
       "      <td>55.000</td>\n",
       "      <td>5.807</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>['offenses', 'crime', 'ucr', 'public', 'safety...</td>\n",
       "      <td>[(5, 0.0630711277688), (8, 0.0202853015264), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4529</th>\n",
       "      <td>4529</td>\n",
       "      <td>Public Safety Answering Points (PSAP)</td>\n",
       "      <td>['Connecticut', 'Public', 'Safety', 'Answering...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>['infrastructure', 'public safety']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>15.000</td>\n",
       "      <td>10.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>54.000</td>\n",
       "      <td>5.781</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['infrastructure', 'public', 'safety']</td>\n",
       "      <td>['infrastructure', 'public', 'safety', 'Connec...</td>\n",
       "      <td>[(1, 0.0104482683654), (2, 0.0100505055654), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4539</th>\n",
       "      <td>4539</td>\n",
       "      <td>PSAP Districts</td>\n",
       "      <td>['Connecticut', 'Public', 'Safety', 'Answering...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>['infrastructure', 'public safety']</td>\n",
       "      <td>NaN</td>\n",
       "      <td>[]</td>\n",
       "      <td>official</td>\n",
       "      <td>17.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>45.000</td>\n",
       "      <td>5.524</td>\n",
       "      <td>data.ct.gov</td>\n",
       "      <td>['infrastructure', 'public', 'safety']</td>\n",
       "      <td>['infrastructure', 'public', 'safety', 'Connec...</td>\n",
       "      <td>[(2, 0.0458611394719), (3, 0.0111117542859), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4812</th>\n",
       "      <td>4812</td>\n",
       "      <td>DPD: Reported Major Crimes, 2011-2014</td>\n",
       "      <td>['Reported', 'major', 'crime', 'offenses', 'fr...</td>\n",
       "      <td>DPD</td>\n",
       "      <td>['category', 'rownum', 'hour', 'location', 'ce...</td>\n",
       "      <td>['CATEGORY', 'ROWNUM', 'HOUR', 'LOCATION', 'CE...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>['public safety']</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['crime', 'detroit police', 'offenses']</td>\n",
       "      <td>official</td>\n",
       "      <td>534.000</td>\n",
       "      <td>118.000</td>\n",
       "      <td>17.000</td>\n",
       "      <td>5440.000</td>\n",
       "      <td>12.410</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['crime', 'detroit', 'police', 'offenses', 'pu...</td>\n",
       "      <td>['crime', 'detroit', 'police', 'offenses', 'pu...</td>\n",
       "      <td>[(8, 0.0613390263294), (20, 0.0599547511312), ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4828</th>\n",
       "      <td>4828</td>\n",
       "      <td>DPD: 2015 Homicide Offenses - Victims (Confirmed)</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>['y', 'x', 'state', 'day', 'fid', 'report_no',...</td>\n",
       "      <td>['Y', 'X', 'STATE', 'DAY', 'FID', 'REPORT_NO',...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'homicide', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>197.000</td>\n",
       "      <td>66.000</td>\n",
       "      <td>19.000</td>\n",
       "      <td>1559.000</td>\n",
       "      <td>10.607</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'homicide', 'police', 'public...</td>\n",
       "      <td>['victim-based', 'homicide', 'police', 'public...</td>\n",
       "      <td>[(4, 0.235396672376), (23, 0.110301885217), (2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4839</th>\n",
       "      <td>4839</td>\n",
       "      <td>DPD Precincts</td>\n",
       "      <td>['Detroit', 'Police', 'Department', 'precinct'...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['police', 'dpd', 'precincts']</td>\n",
       "      <td>official</td>\n",
       "      <td>992.000</td>\n",
       "      <td>35.000</td>\n",
       "      <td>11.000</td>\n",
       "      <td>1078.000</td>\n",
       "      <td>10.075</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['police', 'dpd', 'precincts', 'public', 'safe...</td>\n",
       "      <td>['police', 'dpd', 'precincts', 'public', 'safe...</td>\n",
       "      <td>[(23, 0.213674846029), (26, 0.331946920678), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4859</th>\n",
       "      <td>4859</td>\n",
       "      <td>DPD: 2015 Non-Fatal Shooting Offenses - Victim...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>['state', 'day', 'location_1', 'precinct', 'fi...</td>\n",
       "      <td>['STATE', 'DAY', 'Location_1', 'PRECINCT', 'FI...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'non-fatal shooting', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>99.000</td>\n",
       "      <td>19.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>666.000</td>\n",
       "      <td>9.382</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>[(4, 0.208449561457), (14, 0.0478343078921), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4867</th>\n",
       "      <td>4867</td>\n",
       "      <td>DPD: 2015 Carjacking Offenses - Victims (Confi...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>['y', 'state', 'day', 'location_1', 'time', 'r...</td>\n",
       "      <td>['Y', 'STATE', 'DAY', 'Location_1', 'TIME', 'R...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'carjacking', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>103.000</td>\n",
       "      <td>18.000</td>\n",
       "      <td>7.000</td>\n",
       "      <td>532.000</td>\n",
       "      <td>9.058</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'carjacking', 'police', 'publ...</td>\n",
       "      <td>['victim-based', 'carjacking', 'police', 'publ...</td>\n",
       "      <td>[(4, 0.239361338926), (22, 0.0441467380176), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4871</th>\n",
       "      <td>4871</td>\n",
       "      <td>DPD: 2014 Homicide Offenses - Victims (Confirmed)</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>['day', 'date', 'fid', 'precinct', 'report_no'...</td>\n",
       "      <td>['DAY', 'DATE', 'FID', 'PRECINCT', 'REPORT_NO'...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['2014', 'victim-based', 'homicide', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>92.000</td>\n",
       "      <td>6.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>474.000</td>\n",
       "      <td>8.892</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['2014', 'victim-based', 'homicide', 'police',...</td>\n",
       "      <td>['2014', 'victim-based', 'homicide', 'police',...</td>\n",
       "      <td>[(4, 0.235266934147), (23, 0.110335410239), (2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4873</th>\n",
       "      <td>4873</td>\n",
       "      <td>DPD: 2015 Homicide Offenses - Victims (Confirm...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'homicide', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>84.000</td>\n",
       "      <td>15.000</td>\n",
       "      <td>5.000</td>\n",
       "      <td>455.000</td>\n",
       "      <td>8.833</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'homicide', 'police', 'public...</td>\n",
       "      <td>['victim-based', 'homicide', 'police', 'public...</td>\n",
       "      <td>[(4, 0.235313755179), (23, 0.110276662358), (2...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4875</th>\n",
       "      <td>4875</td>\n",
       "      <td>DPD: 2014 Non-Fatal Shooting Offenses - Victim...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>['city', 'fid', 'x', 'state', 'precinct', 'dat...</td>\n",
       "      <td>['CITY', 'FID', 'X', 'STATE', 'PRECINCT', 'DAT...</td>\n",
       "      <td>dataset</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'non-fatal shooting', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>80.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>432.000</td>\n",
       "      <td>8.758</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>[(4, 0.208435187859), (14, 0.0478343282382), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4880</th>\n",
       "      <td>4880</td>\n",
       "      <td>DPD: 2015 Carjacking Offenses - Victims (Confi...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'carjacking', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>48.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>359.000</td>\n",
       "      <td>8.492</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'carjacking', 'police', 'publ...</td>\n",
       "      <td>['victim-based', 'carjacking', 'police', 'publ...</td>\n",
       "      <td>[(4, 0.239419303997), (22, 0.0441419106655), (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4889</th>\n",
       "      <td>4889</td>\n",
       "      <td>DPD: 2015 Non-Fatal Shooting Offenses - Victim...</td>\n",
       "      <td>['Year', 'end', 'victim', 'based', 'listing', ...</td>\n",
       "      <td>Detroit Police Department</td>\n",
       "      <td>[]</td>\n",
       "      <td>[]</td>\n",
       "      <td>map</td>\n",
       "      <td>[]</td>\n",
       "      <td>Public Safety</td>\n",
       "      <td>['victim-based', 'non-fatal shooting', 'police']</td>\n",
       "      <td>official</td>\n",
       "      <td>60.000</td>\n",
       "      <td>8.000</td>\n",
       "      <td>2.000</td>\n",
       "      <td>243.000</td>\n",
       "      <td>7.931</td>\n",
       "      <td>data.detroitmi.gov</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>['victim-based', 'non-fatal', 'shooting', 'pol...</td>\n",
       "      <td>[(4, 0.208349233919), (14, 0.0478344892368), (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200 rows × 20 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Unnamed: 0                                               name  \\\n",
       "21            21                     DCHS_Safety_DVSA Hotline Calls   \n",
       "22            22                      Exit Plan for Leaving Shelter   \n",
       "88            88  Total People Reached Through Community Engagem...   \n",
       "101          101                          Length of Stay at Shelter   \n",
       "160          160  Adults and Children Sheltered due to Domestic ...   \n",
       "161          161             DCHS_SA_Domestic Violence Shelter Data   \n",
       "231          231             DCHS_Safety_DVSA Hotline Calls by Type   \n",
       "239          239  DCHS_SA_Domestic Violence And Sexual Assault C...   \n",
       "252          252  Percent of Participants who Report that they L...   \n",
       "272          272             OHA Baseline Target - Viable Land Base   \n",
       "275          275             OHA Baseline - Value History & Culture   \n",
       "286          286       Hawai'i Homeless Counts by Island, 2010-2013   \n",
       "316          316  State of Hawaiʻi Population by Race/Ethnicity:...   \n",
       "340          340              Decibel Level of Civil Defense Sirens   \n",
       "358          358                 TANF Caseload (By Selected Months)   \n",
       "394          394                             Health - Obesity (OHA)   \n",
       "431          431             OHA Baseline Target - Viable Land Base   \n",
       "437          437                          Public Safety Budget Data   \n",
       "444          444                        Map Of Civil Defense Sirens   \n",
       "459          459                 WIC Clinic Participation 2010-2013   \n",
       "503          503       Hawai'i Homeless Counts by Island, 2010-2013   \n",
       "517          517                             Women's Health Clinics   \n",
       "523          523                            Health - Prenatal (OHA)   \n",
       "534          534  State of Hawaiʻi Population by Race/Ethnicity:...   \n",
       "568          568                              Fire / EMS Dispatches   \n",
       "575          575                                     EMS Dispatches   \n",
       "583          583                                     EMS Dispatches   \n",
       "614          614                                   Index Crime Rate   \n",
       "632          632                             State Index Crime Rate   \n",
       "655          655               Alameda County Sheriff Crime Reports   \n",
       "...          ...                                                ...   \n",
       "4262        4262  Uniform Crime Reporting System Offenses by Dep...   \n",
       "4286        4286     Uniform Crime Reporting System Arrests in 2012   \n",
       "4287        4287                                 Police Departments   \n",
       "4290        4290       Uniform Crime Reporting System Offenses 2011   \n",
       "4292        4292  Uniform Crime Reporting System Offenses By Dep...   \n",
       "4294        4294    Uniform Crime Reporting System Arrests in  2010   \n",
       "4302        4302  Number of Law Enforcement Personnel by Departm...   \n",
       "4329        4329    Uniform Crime Reporting System Offenses in 2010   \n",
       "4362        4362        CHART : LICENSE ENDORSEMENTS as of NOV 2013   \n",
       "4388        4388        CHART : LICENSE ENDORSEMENTS as of NOV 2014   \n",
       "4399        4399  Uniform Crime Reporting System Offenses By Dep...   \n",
       "4407        4407       Uniform Crime Reporting System Offenses 2012   \n",
       "4431        4431                  City of New Haven Police Stations   \n",
       "4439        4439           Vehicle Crash Data Repository - CT CRASH   \n",
       "4451        4451        CHART : LICENSE ENDORSEMENTS as of NOV 2012   \n",
       "4478        4478  Commercial Vehicle Safety Division- Consumer C...   \n",
       "4524        4524        Uniform Crime Reporting System Arrests 2015   \n",
       "4528        4528  Uniform Crime Reporting System Offenses By Dep...   \n",
       "4529        4529              Public Safety Answering Points (PSAP)   \n",
       "4539        4539                                     PSAP Districts   \n",
       "4812        4812              DPD: Reported Major Crimes, 2011-2014   \n",
       "4828        4828  DPD: 2015 Homicide Offenses - Victims (Confirmed)   \n",
       "4839        4839                                      DPD Precincts   \n",
       "4859        4859  DPD: 2015 Non-Fatal Shooting Offenses - Victim...   \n",
       "4867        4867  DPD: 2015 Carjacking Offenses - Victims (Confi...   \n",
       "4871        4871  DPD: 2014 Homicide Offenses - Victims (Confirmed)   \n",
       "4873        4873  DPD: 2015 Homicide Offenses - Victims (Confirm...   \n",
       "4875        4875  DPD: 2014 Non-Fatal Shooting Offenses - Victim...   \n",
       "4880        4880  DPD: 2015 Carjacking Offenses - Victims (Confi...   \n",
       "4889        4889  DPD: 2015 Non-Fatal Shooting Offenses - Victim...   \n",
       "\n",
       "                                            description  \\\n",
       "21                                                   []   \n",
       "22                                                   []   \n",
       "88                                                   []   \n",
       "101                                                  []   \n",
       "160                                                  []   \n",
       "161                                                  []   \n",
       "231                                                  []   \n",
       "239                                                  []   \n",
       "252                                                  []   \n",
       "272   ['OHA', 'Strategic', 'Results', '(2010', '2018...   \n",
       "275   ['85%', 'of', 'Hawaiʻi', 'residents', 'appreci...   \n",
       "286                                                  []   \n",
       "316                                                  []   \n",
       "340   ['Includes', 'decibel', 'levels', 'and', 'prec...   \n",
       "358                                                  []   \n",
       "394                                                  []   \n",
       "431   ['OHA', 'Strategic', 'Results', '(2010', '2018...   \n",
       "437                                                  []   \n",
       "444   ['Includes', 'decibel', 'levels', 'and', 'prec...   \n",
       "459                                                  []   \n",
       "503                                                  []   \n",
       "517                                                  []   \n",
       "523                                                  []   \n",
       "534                                                  []   \n",
       "568                                                  []   \n",
       "575                                                  []   \n",
       "583                                                  []   \n",
       "614   ['Salt', 'Lake', 'County', 'crime', 'rates', '...   \n",
       "632   ['State', 'of', 'Utah', 'crime', 'rates', 'sta...   \n",
       "655   ['Alameda', 'County', 'Sheriff', 'Crime', 'Rep...   \n",
       "...                                                 ...   \n",
       "4262  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4286  ['Arrests', 'are', 'the', 'number', 'of', 'per...   \n",
       "4287  ['A', 'listing', 'of', 'State', 'Municipal', '...   \n",
       "4290  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4292  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4294  ['Arrests', 'are', 'the', 'number', 'of', 'per...   \n",
       "4302  ['The', 'number', 'of', 'sworn', 'and', 'civil...   \n",
       "4329  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4362                                                 []   \n",
       "4388                                                 []   \n",
       "4399  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4407  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4431  ['Locations', 'of', 'Police', 'Stations', 'in'...   \n",
       "4439  ['The', 'Connecticut', 'Crash', 'Data', 'Repos...   \n",
       "4451                                                 []   \n",
       "4478  ['CVSD', 'Consumer', 'Complaint', 'Cases', 'be...   \n",
       "4524  ['Arrests', 'are', 'the', 'number', 'of', 'per...   \n",
       "4528  ['The', 'Uniform', 'Crime', 'Reporting', 'Prog...   \n",
       "4529  ['Connecticut', 'Public', 'Safety', 'Answering...   \n",
       "4539  ['Connecticut', 'Public', 'Safety', 'Answering...   \n",
       "4812  ['Reported', 'major', 'crime', 'offenses', 'fr...   \n",
       "4828  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4839  ['Detroit', 'Police', 'Department', 'precinct'...   \n",
       "4859  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4867  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4871  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4873  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4875  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4880  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "4889  ['Year', 'end', 'victim', 'based', 'listing', ...   \n",
       "\n",
       "                                            attribution  \\\n",
       "21                                                  NaN   \n",
       "22                                                  NaN   \n",
       "88                                                  NaN   \n",
       "101                                                 NaN   \n",
       "160                                                 NaN   \n",
       "161                                                 NaN   \n",
       "231                                                 NaN   \n",
       "239                                                 NaN   \n",
       "252                                                 NaN   \n",
       "272   Field Note: Baseline Hawai'i State, Office of ...   \n",
       "275                                                 OHA   \n",
       "286                                  Source: PIT, 2013.   \n",
       "316   Source. Hawaiʻi, Department of Health, Hawaiʻi...   \n",
       "340                                                 NaN   \n",
       "358                                                 NaN   \n",
       "394                                                 NaN   \n",
       "431   Field Note: Baseline Hawai'i State, Office of ...   \n",
       "437                                                 NaN   \n",
       "444                                                 NaN   \n",
       "459   Kayla Rosenfeld, DHS; WIC services in Hawaii, ...   \n",
       "503                                  Source: PIT, 2013.   \n",
       "517                                                 NaN   \n",
       "523                                                 NaN   \n",
       "534   Source. Hawaiʻi, Department of Health, Hawaiʻi...   \n",
       "568                                                 NaN   \n",
       "575                                                 NaN   \n",
       "583                                                 NaN   \n",
       "614                    Utah Department of Public Safety   \n",
       "632                    Utah Department of Public Safety   \n",
       "655                     Alameda County Sheriff's Office   \n",
       "...                                                 ...   \n",
       "4262  Department of Emergency Services and Public Pr...   \n",
       "4286  Department of Emergency Services and Public Pr...   \n",
       "4287  Department of Emergency Service and Public Pro...   \n",
       "4290  Department of Emergency Services and Public Pr...   \n",
       "4292  Department of Emergency Services and Public Pr...   \n",
       "4294  Department of Emegerncy Service and Public Pro...   \n",
       "4302  Department of Emergency Services and Public Pr...   \n",
       "4329  Department of Emergency Services and Public Pr...   \n",
       "4362                        DEPARTMENT OF MOTOR VEHICLE   \n",
       "4388                        DEPARTMENT OF MOTOR VEHICLE   \n",
       "4399  Department of Emergency Services and Public Pr...   \n",
       "4407  Department of Emergency Services and Public Pr...   \n",
       "4431                                  City of New Haven   \n",
       "4439                       Department of Transportation   \n",
       "4451                        DEPARTMENT OF MOTOR VEHICLE   \n",
       "4478                       Department of Motor Vehicles   \n",
       "4524  Department of Emergency Services and Public Pr...   \n",
       "4528  Department of Emergency Services and Public Pr...   \n",
       "4529                                                NaN   \n",
       "4539                                                NaN   \n",
       "4812                                                DPD   \n",
       "4828                          Detroit Police Department   \n",
       "4839                          Detroit Police Department   \n",
       "4859                          Detroit Police Department   \n",
       "4867                          Detroit Police Department   \n",
       "4871                          Detroit Police Department   \n",
       "4873                          Detroit Police Department   \n",
       "4875                          Detroit Police Department   \n",
       "4880                          Detroit Police Department   \n",
       "4889                          Detroit Police Department   \n",
       "\n",
       "                                     columns_field_name  \\\n",
       "21    ['sexual_assault_hotline_calls', 'other_servic...   \n",
       "22    ['sexual_assault_hotline_calls', 'percent_of_c...   \n",
       "88    ['percent_of_participants_reported_that_they_l...   \n",
       "101   ['unknown', 'left_area', 'residence_of_friend_...   \n",
       "160   ['residence_of_friend_relative', 'transitional...   \n",
       "161   ['over_90_days', 'length_of_stay_less_than_24_...   \n",
       "231   ['children_sheltered_due_to_domestic_violence'...   \n",
       "239   ['total_people_reached', 'total_hours_of_activ...   \n",
       "252   ['race_of_participants_caucasian', 'race_of_pa...   \n",
       "272   ['annual_target_population', 'fiscal_year', 'y...   \n",
       "275   ['annual_target_hi_adult_resident', 'oha_targe...   \n",
       "286   ['homeless_count', 'county', 'year', 'hawai_i'...   \n",
       "316   ['population_percent', 'population_number', 'r...   \n",
       "340   ['decibel', 'location', 'unique_id', 'location...   \n",
       "358   ['of_employed_households', 'gross_earned_incom...   \n",
       "394   ['date', 'hp2020_benchmark', 'oha_benchmark', ...   \n",
       "431   ['year', 'annual_target_population', 'fiscal_y...   \n",
       "437   ['total_operating', 'fiscal_year', 'total_capi...   \n",
       "444   ['location', 'decibel', 'county', 'location_1'...   \n",
       "459   ['total_served', 'infants', 'women', 'children...   \n",
       "503   ['homeless_count', 'kaua_i', 'year', 'hawai_i'...   \n",
       "517                   ['clinic', 'phone', 'location_1']   \n",
       "523   ['unknown_n', 'oha_benchmark', 'hp2020_benchma...   \n",
       "534   ['population_number', 'race_ethnicity', 'popul...   \n",
       "568   ['med_inj_processing', 'fire_ems_disp_avg', 'f...   \n",
       "575   ['fire', '_911_via_telematics', 'voip', '_911_...   \n",
       "583   ['fire', '_911_via_telematics', '_911_via_voip...   \n",
       "614                             ['year2', 'crime_rate']   \n",
       "632                        ['date', 'state_crime_rate']   \n",
       "655   ['crimedescription', 'location_1', 'datetime',...   \n",
       "...                                                 ...   \n",
       "4262  ['robbery_gas_service_station_value_stolen', '...   \n",
       "4286  ['colchester_csp', 'torrington_pd', 'tolland_c...   \n",
       "4287  ['telephone', 'address', 'location', 'zip', 'c...   \n",
       "4290  ['milford_pd', 'hamden_pd', 'westbrook_csp', '...   \n",
       "4292  ['arson_other_value_damaged', 'smasslt_offense...   \n",
       "4294  ['winchester_pd', 'colchester_csp', 'woodbridg...   \n",
       "4302  ['department', 'sworn_male', 'civilian_female'...   \n",
       "4329  ['enfield_pd', 'csp_total', 'foxwoods_casino_c...   \n",
       "4362          ['total', 'license_endosrements', 'year']   \n",
       "4388          ['year', 'total', 'license_endosrements']   \n",
       "4399  ['ngmansl_clearances', 'murder_relationship_st...   \n",
       "4407  ['brooklyn_csp', 'stonington_pd', 'stamford_pd...   \n",
       "4431                                                 []   \n",
       "4439                                                 []   \n",
       "4451          ['year', 'total', 'license_endosrements']   \n",
       "4478  ['cases_closed', 'year', 'month', 'cases_open'...   \n",
       "4524  ['urban', 'tolland_county', 'w_c_s_u', 's_c_s_...   \n",
       "4528  ['murder_relationship_live_in_male', 'murder_r...   \n",
       "4529                                                 []   \n",
       "4539                                                 []   \n",
       "4812  ['category', 'rownum', 'hour', 'location', 'ce...   \n",
       "4828  ['y', 'x', 'state', 'day', 'fid', 'report_no',...   \n",
       "4839                                                 []   \n",
       "4859  ['state', 'day', 'location_1', 'precinct', 'fi...   \n",
       "4867  ['y', 'state', 'day', 'location_1', 'time', 'r...   \n",
       "4871  ['day', 'date', 'fid', 'precinct', 'report_no'...   \n",
       "4873                                                 []   \n",
       "4875  ['city', 'fid', 'x', 'state', 'precinct', 'dat...   \n",
       "4880                                                 []   \n",
       "4889                                                 []   \n",
       "\n",
       "                                           columns_name     type  \\\n",
       "21    ['Sexual Assault Hotline Calls', 'Other Servic...    chart   \n",
       "22    ['Sexual Assault Hotline Calls', 'Percent of c...    chart   \n",
       "88    ['Percent of participants reported that they l...    chart   \n",
       "101   ['Unknown', 'Left Area', 'Residence of friend/...    chart   \n",
       "160   ['Residence of friend/relative', 'Transitional...    chart   \n",
       "161   ['Over 90 days', 'Less than 24 hours', 'Reside...  dataset   \n",
       "231   ['Children sheltered due to Domestic Violence'...    chart   \n",
       "239   ['Total People Reached', 'Total Hours of Activ...  dataset   \n",
       "252   ['Race of Participants - Caucasian', 'Race of ...    chart   \n",
       "272   ['Annual Target Population ', 'Fiscal Year', '...    chart   \n",
       "275   ['Annual Target (HI adult resident)', 'OHA Tar...    chart   \n",
       "286      ['Maui', \"O'ahu\", 'Year', \"Hawai'i\", \"Kaua'i\"]    chart   \n",
       "316   ['Population (Percent)', 'Population (Number)'...    chart   \n",
       "340   ['DECIBEL', 'LOCATION', 'UNIQUE_ID', 'Location...      map   \n",
       "358   ['# of Employed Households', 'Gross Earned Inc...    chart   \n",
       "394   ['Date', 'HP2020 Benchmark', 'OHA Benchmark', ...  dataset   \n",
       "431   ['Date', 'Annual Target Population ', 'Fiscal ...  dataset   \n",
       "437   ['Total Operating', 'Fiscal Year', 'Total Capi...  dataset   \n",
       "444   ['LOCATION', 'DECIBEL', 'COUNTY', 'Location 1'...  dataset   \n",
       "459   ['Total Served', 'Infants', 'Women', 'Children...  dataset   \n",
       "503      ['Maui', \"Kaua'i\", 'Year', \"Hawai'i\", \"O'ahu\"]  dataset   \n",
       "517                   ['Clinic', 'Phone', 'Location 1']  dataset   \n",
       "523   ['Unknown (N)', 'OHA Benchmark', 'HP2020 Bench...  dataset   \n",
       "534   ['Population (Number)', 'Race/Ethnicity ', 'Po...  dataset   \n",
       "568   ['PFR Med/Inj Processing 90th', 'Fire / EMS Di...    chart   \n",
       "575   ['% Fire', '911 via Telematics', '% VoiP', '91...    chart   \n",
       "583   ['% Fire', '911 via Telematics', '911 via VoiP...    chart   \n",
       "614                             ['Year2', 'CRIME RATE']  dataset   \n",
       "632                        ['DATE', 'STATE CRIME RATE']  dataset   \n",
       "655   ['CrimeDescription', 'Location 1', 'DateTime',...  dataset   \n",
       "...                                                 ...      ...   \n",
       "4262  ['Robbery Gas/Service_Station Value_Stolen', '...  dataset   \n",
       "4286  ['Colchester CSP', 'Torrington PD', 'Tolland C...  dataset   \n",
       "4287  ['TELEPHONE', 'ADDRESS', 'Location', 'ZIP', 'C...  dataset   \n",
       "4290  ['Milford PD', 'Hamden PD', 'Westbrook CSP', '...  dataset   \n",
       "4292  ['Arson Other Value_Damaged', 'SmAsslt Offense...  dataset   \n",
       "4294  ['Winchester PD', 'Colchester CSP', 'Woodbridg...  dataset   \n",
       "4302  ['Department', 'Sworn Male ', 'Civilian Female...  dataset   \n",
       "4329  ['Enfield PD', 'CSP Total', 'Foxwoods Casino C...  dataset   \n",
       "4362          ['TOTAL', 'LICENSE ENDOSREMENTS', 'YEAR']    chart   \n",
       "4388          ['YEAR', 'TOTAL', 'LICENSE ENDOSREMENTS']    chart   \n",
       "4399  ['NgMansl Clearances', 'Murder Relationship St...  dataset   \n",
       "4407  ['Brooklyn CSP', 'Stonington PD', 'Stamford PD...  dataset   \n",
       "4431                                                 []      map   \n",
       "4439                                                 []     href   \n",
       "4451          ['YEAR', 'TOTAL', 'LICENSE ENDOSREMENTS']    chart   \n",
       "4478  ['Cases Closed', 'Year', 'Month', 'Cases Open'...  dataset   \n",
       "4524  ['Urban', 'Tolland County', 'W.C.S.U.', 'S.C.S...  dataset   \n",
       "4528  ['Murder Relationship Live-In, Male', 'Murder ...  dataset   \n",
       "4529                                                 []      map   \n",
       "4539                                                 []      map   \n",
       "4812  ['CATEGORY', 'ROWNUM', 'HOUR', 'LOCATION', 'CE...  dataset   \n",
       "4828  ['Y', 'X', 'STATE', 'DAY', 'FID', 'REPORT_NO',...  dataset   \n",
       "4839                                                 []      map   \n",
       "4859  ['STATE', 'DAY', 'Location_1', 'PRECINCT', 'FI...  dataset   \n",
       "4867  ['Y', 'STATE', 'DAY', 'Location_1', 'TIME', 'R...  dataset   \n",
       "4871  ['DAY', 'DATE', 'FID', 'PRECINCT', 'REPORT_NO'...  dataset   \n",
       "4873                                                 []      map   \n",
       "4875  ['CITY', 'FID', 'X', 'STATE', 'PRECINCT', 'DAT...  dataset   \n",
       "4880                                                 []      map   \n",
       "4889                                                 []      map   \n",
       "\n",
       "                                 categories  \\\n",
       "21                                       []   \n",
       "22                                       []   \n",
       "88                                       []   \n",
       "101                                      []   \n",
       "160                                      []   \n",
       "161                                      []   \n",
       "231                                      []   \n",
       "239                                      []   \n",
       "252                                      []   \n",
       "272                                      []   \n",
       "275                                      []   \n",
       "286                                      []   \n",
       "316                                      []   \n",
       "340                                      []   \n",
       "358                       ['public safety']   \n",
       "394                                      []   \n",
       "431                                      []   \n",
       "437                                      []   \n",
       "444                                      []   \n",
       "459                                      []   \n",
       "503                                      []   \n",
       "517                                      []   \n",
       "523                                      []   \n",
       "534                                      []   \n",
       "568             ['public safety', 'health']   \n",
       "575             ['public safety', 'health']   \n",
       "583             ['public safety', 'health']   \n",
       "614                                      []   \n",
       "632                                      []   \n",
       "655                              ['health']   \n",
       "...                                     ...   \n",
       "4262            ['public safety', 'health']   \n",
       "4286                      ['public safety']   \n",
       "4287                                     []   \n",
       "4290            ['public safety', 'health']   \n",
       "4292            ['public safety', 'health']   \n",
       "4294                      ['public safety']   \n",
       "4302                                     []   \n",
       "4329            ['public safety', 'health']   \n",
       "4362           ['economy', 'public safety']   \n",
       "4388           ['economy', 'public safety']   \n",
       "4399            ['public safety', 'health']   \n",
       "4407            ['public safety', 'health']   \n",
       "4431                                     []   \n",
       "4439  ['transportation', 'social services']   \n",
       "4451           ['economy', 'public safety']   \n",
       "4478                      ['public safety']   \n",
       "4524                      ['public safety']   \n",
       "4528            ['public safety', 'health']   \n",
       "4529    ['infrastructure', 'public safety']   \n",
       "4539    ['infrastructure', 'public safety']   \n",
       "4812                      ['public safety']   \n",
       "4828                                     []   \n",
       "4839                                     []   \n",
       "4859                                     []   \n",
       "4867                                     []   \n",
       "4871                                     []   \n",
       "4873                                     []   \n",
       "4875                                     []   \n",
       "4880                                     []   \n",
       "4889                                     []   \n",
       "\n",
       "                                        domain_category  \\\n",
       "21                                                  NaN   \n",
       "22                                                  NaN   \n",
       "88                                                  NaN   \n",
       "101                                                 NaN   \n",
       "160                                                 NaN   \n",
       "161                                                 NaN   \n",
       "231                                                 NaN   \n",
       "239                                                 NaN   \n",
       "252                                                 NaN   \n",
       "272                                   Individual Rights   \n",
       "275                              Culture and Recreation   \n",
       "286                                     Social Services   \n",
       "316                                               Other   \n",
       "340                                       Public Safety   \n",
       "358                                                 NaN   \n",
       "394                                                 NaN   \n",
       "431                                   Individual Rights   \n",
       "437                                       Public Safety   \n",
       "444                                       Public Safety   \n",
       "459   A New Day for: Safety Net, Homelessness, Publi...   \n",
       "503                                     Social Services   \n",
       "517   A New Day for: Safety Net, Homelessness, Publi...   \n",
       "523                                                 NaN   \n",
       "534                                               Other   \n",
       "568                                                 NaN   \n",
       "575                                                 NaN   \n",
       "583                                                 NaN   \n",
       "614                                                 NaN   \n",
       "632                                                 NaN   \n",
       "655                                       Public Safety   \n",
       "...                                                 ...   \n",
       "4262                                      Public Safety   \n",
       "4286                                      Public Safety   \n",
       "4287                                      Public Safety   \n",
       "4290                                      Public Safety   \n",
       "4292                                      Public Safety   \n",
       "4294                                      Public Safety   \n",
       "4302                                      Public Safety   \n",
       "4329                                      Public Safety   \n",
       "4362                                     Transportation   \n",
       "4388                                     Transportation   \n",
       "4399                                      Public Safety   \n",
       "4407                                      Public Safety   \n",
       "4431                                      Public Safety   \n",
       "4439                                     Transportation   \n",
       "4451                                     Transportation   \n",
       "4478                                     Transportation   \n",
       "4524                                      Public Safety   \n",
       "4528                                      Public Safety   \n",
       "4529                                                NaN   \n",
       "4539                                                NaN   \n",
       "4812                                      Public Safety   \n",
       "4828                                      Public Safety   \n",
       "4839                                      Public Safety   \n",
       "4859                                      Public Safety   \n",
       "4867                                      Public Safety   \n",
       "4871                                      Public Safety   \n",
       "4873                                      Public Safety   \n",
       "4875                                      Public Safety   \n",
       "4880                                      Public Safety   \n",
       "4889                                      Public Safety   \n",
       "\n",
       "                                            domain_tags provenance  \\\n",
       "21    ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "22    ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "88                       ['safety', 'sa', 'dv', 'dchs']   official   \n",
       "101   ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "160   ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "161   ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "231   ['sexual assault', 'domestic violence', 'safet...   official   \n",
       "239                      ['safety', 'sa', 'dv', 'dchs']   official   \n",
       "252                      ['safety', 'sa', 'dv', 'dchs']   official   \n",
       "272                         ['baseline', 'land', 'oha']   official   \n",
       "275           ['culture', 'hawaiian', 'history', 'oha']   official   \n",
       "286          ['hawaiian', 'homeless', 'housing', 'oha']   official   \n",
       "316   ['hawaiian', 'oha', 'population', 'race', 'sta...   official   \n",
       "340                                                  []   official   \n",
       "358                                                  []   official   \n",
       "394                                             ['oha']   official   \n",
       "431                         ['baseline', 'land', 'oha']   official   \n",
       "437                                                  []   official   \n",
       "444                                                  []   official   \n",
       "459                                                  []   official   \n",
       "503          ['hawaiian', 'homeless', 'housing', 'oha']   official   \n",
       "517                                                  []   official   \n",
       "523                                             ['oha']   official   \n",
       "534   ['hawaiian', 'oha', 'population', 'race', 'sta...   official   \n",
       "568                                                  []   official   \n",
       "575                                                  []   official   \n",
       "583                                                  []   official   \n",
       "614      ['index crime', 'crime rate', 'rate', 'crime']   official   \n",
       "632              ['crime rate', 'index crimes', 'utah']   official   \n",
       "655                                                  []   official   \n",
       "...                                                 ...        ...   \n",
       "4262                       ['offenses', 'crime', 'ucr']   official   \n",
       "4286                        ['arrests', 'crime', 'ucr']   official   \n",
       "4287  ['crime', 'departments', 'police', 'public saf...   official   \n",
       "4290                       ['offenses', 'crime', 'ucr']   official   \n",
       "4292               ['2013', 'offenses', 'crime', 'ucr']   official   \n",
       "4294                        ['arrests', 'crime', 'ucr']   official   \n",
       "4302           ['public safety', 'personnel', 'police']   official   \n",
       "4329                       ['offenses', 'crime', 'ucr']   official   \n",
       "4362                                            ['dmv']   official   \n",
       "4388                                            ['dmv']   official   \n",
       "4399                       ['offenses', 'crime', 'ucr']   official   \n",
       "4407                       ['crime', 'offenses', 'ucr']   official   \n",
       "4431  ['police', 'law enforcement', 'new haven', 'nhv']   official   \n",
       "4439    ['accidents', 'crash', 'usodcensus', 'vehicle']   official   \n",
       "4451                                            ['dmv']   official   \n",
       "4478                                            ['dmv']   official   \n",
       "4524                        ['arrests', 'crime', 'ucr']   official   \n",
       "4528                       ['offenses', 'crime', 'ucr']   official   \n",
       "4529                                                 []   official   \n",
       "4539                                                 []   official   \n",
       "4812            ['crime', 'detroit police', 'offenses']   official   \n",
       "4828             ['victim-based', 'homicide', 'police']   official   \n",
       "4839                     ['police', 'dpd', 'precincts']   official   \n",
       "4859   ['victim-based', 'non-fatal shooting', 'police']   official   \n",
       "4867           ['victim-based', 'carjacking', 'police']   official   \n",
       "4871     ['2014', 'victim-based', 'homicide', 'police']   official   \n",
       "4873             ['victim-based', 'homicide', 'police']   official   \n",
       "4875   ['victim-based', 'non-fatal shooting', 'police']   official   \n",
       "4880           ['victim-based', 'carjacking', 'police']   official   \n",
       "4889   ['victim-based', 'non-fatal shooting', 'police']   official   \n",
       "\n",
       "      download_count  page_views_last_month  page_views_last_week  \\\n",
       "21             6.000                 27.000                 0.000   \n",
       "22             4.000                 18.000                 0.000   \n",
       "88             3.000                  9.000                 1.000   \n",
       "101           11.000                  8.000                 0.000   \n",
       "160            0.000                  0.000                 0.000   \n",
       "161            1.000                  0.000                 0.000   \n",
       "231            0.000                  0.000                 0.000   \n",
       "239            2.000                  0.000                 0.000   \n",
       "252            0.000                  0.000                 0.000   \n",
       "272           90.000                118.000                23.000   \n",
       "275           51.000                 24.000                 3.000   \n",
       "286           34.000                 30.000                 3.000   \n",
       "316           47.000                 21.000                 4.000   \n",
       "340           21.000                 21.000                 0.000   \n",
       "358           17.000                 45.000                 6.000   \n",
       "394           49.000                  1.000                 0.000   \n",
       "431           51.000                  0.000                 0.000   \n",
       "437           17.000                  0.000                 0.000   \n",
       "444           26.000                  2.000                 0.000   \n",
       "459           17.000                  0.000                 0.000   \n",
       "503           22.000                  0.000                 0.000   \n",
       "517           24.000                  0.000                 0.000   \n",
       "523           28.000                  1.000                 0.000   \n",
       "534           24.000                  1.000                 0.000   \n",
       "568           38.000                  0.000                 0.000   \n",
       "575           33.000                  4.000                 1.000   \n",
       "583           25.000                  0.000                 0.000   \n",
       "614          602.000                  7.000                 1.000   \n",
       "632          583.000                  5.000                 0.000   \n",
       "655         1111.000               1395.000               310.000   \n",
       "...              ...                    ...                   ...   \n",
       "4262         738.000                  0.000                 0.000   \n",
       "4286         580.000                  0.000                 0.000   \n",
       "4287         878.000                  9.000                 8.000   \n",
       "4290         674.000                  5.000                 2.000   \n",
       "4292         577.000                  2.000                 0.000   \n",
       "4294         553.000                  6.000                 1.000   \n",
       "4302         584.000                  1.000                 0.000   \n",
       "4329         702.000                  9.000                 3.000   \n",
       "4362          77.000                 13.000                 1.000   \n",
       "4388          69.000                 14.000                 2.000   \n",
       "4399         661.000                  1.000                 0.000   \n",
       "4407         644.000                  5.000                 0.000   \n",
       "4431         365.000                 17.000                 6.000   \n",
       "4439          36.000                 12.000                 2.000   \n",
       "4451          35.000                  4.000                 1.000   \n",
       "4478         649.000                  7.000                 4.000   \n",
       "4524         327.000                 14.000                 5.000   \n",
       "4528         369.000                 11.000                 4.000   \n",
       "4529          15.000                 10.000                 0.000   \n",
       "4539          17.000                  5.000                 0.000   \n",
       "4812         534.000                118.000                17.000   \n",
       "4828         197.000                 66.000                19.000   \n",
       "4839         992.000                 35.000                11.000   \n",
       "4859          99.000                 19.000                 5.000   \n",
       "4867         103.000                 18.000                 7.000   \n",
       "4871          92.000                  6.000                 2.000   \n",
       "4873          84.000                 15.000                 5.000   \n",
       "4875          80.000                  8.000                 0.000   \n",
       "4880          48.000                  8.000                 2.000   \n",
       "4889          60.000                  8.000                 2.000   \n",
       "\n",
       "      page_views_total  page_views_total_log                      domain  \\\n",
       "21             190.000                 7.577  dashboard.alexandriava.gov   \n",
       "22             189.000                 7.570  dashboard.alexandriava.gov   \n",
       "88              67.000                 6.087  dashboard.alexandriava.gov   \n",
       "101             58.000                 5.883  dashboard.alexandriava.gov   \n",
       "160             29.000                 4.907  dashboard.alexandriava.gov   \n",
       "161             29.000                 4.907  dashboard.alexandriava.gov   \n",
       "231             10.000                 3.459  dashboard.alexandriava.gov   \n",
       "239              9.000                 3.322  dashboard.alexandriava.gov   \n",
       "252              7.000                 3.000  dashboard.alexandriava.gov   \n",
       "272           9494.000                13.213        dashboard.hawaii.gov   \n",
       "275           4640.000                12.180        dashboard.hawaii.gov   \n",
       "286            344.000                 8.430        dashboard.hawaii.gov   \n",
       "316            257.000                 8.011        dashboard.hawaii.gov   \n",
       "340            219.000                 7.781        dashboard.hawaii.gov   \n",
       "358            201.000                 7.658        dashboard.hawaii.gov   \n",
       "394            170.000                 7.418        dashboard.hawaii.gov   \n",
       "431            148.000                 7.219        dashboard.hawaii.gov   \n",
       "437            146.000                 7.200        dashboard.hawaii.gov   \n",
       "444            143.000                 7.170        dashboard.hawaii.gov   \n",
       "459            136.000                 7.098        dashboard.hawaii.gov   \n",
       "503            120.000                 6.919        dashboard.hawaii.gov   \n",
       "517            117.000                 6.883        dashboard.hawaii.gov   \n",
       "523            113.000                 6.833        dashboard.hawaii.gov   \n",
       "534            109.000                 6.781        dashboard.hawaii.gov   \n",
       "568            142.000                 7.160         dashboard.plano.gov   \n",
       "575            108.000                 6.768         dashboard.plano.gov   \n",
       "583             99.000                 6.644         dashboard.plano.gov   \n",
       "614          27150.000                14.729          dashboard.slco.org   \n",
       "632          26585.000                14.698          dashboard.slco.org   \n",
       "655          20065.000                14.292              data.acgov.org   \n",
       "...                ...                   ...                         ...   \n",
       "4262           494.000                 8.951                 data.ct.gov   \n",
       "4286           453.000                 8.827                 data.ct.gov   \n",
       "4287           453.000                 8.827                 data.ct.gov   \n",
       "4290           450.000                 8.817                 data.ct.gov   \n",
       "4292           446.000                 8.804                 data.ct.gov   \n",
       "4294           445.000                 8.801                 data.ct.gov   \n",
       "4302           433.000                 8.762                 data.ct.gov   \n",
       "4329           381.000                 8.577                 data.ct.gov   \n",
       "4362           329.000                 8.366                 data.ct.gov   \n",
       "4388           291.000                 8.190                 data.ct.gov   \n",
       "4399           278.000                 8.124                 data.ct.gov   \n",
       "4407           268.000                 8.071                 data.ct.gov   \n",
       "4431           230.000                 7.852                 data.ct.gov   \n",
       "4439           223.000                 7.807                 data.ct.gov   \n",
       "4451           210.000                 7.721                 data.ct.gov   \n",
       "4478           147.000                 7.209                 data.ct.gov   \n",
       "4524            59.000                 5.907                 data.ct.gov   \n",
       "4528            55.000                 5.807                 data.ct.gov   \n",
       "4529            54.000                 5.781                 data.ct.gov   \n",
       "4539            45.000                 5.524                 data.ct.gov   \n",
       "4812          5440.000                12.410          data.detroitmi.gov   \n",
       "4828          1559.000                10.607          data.detroitmi.gov   \n",
       "4839          1078.000                10.075          data.detroitmi.gov   \n",
       "4859           666.000                 9.382          data.detroitmi.gov   \n",
       "4867           532.000                 9.058          data.detroitmi.gov   \n",
       "4871           474.000                 8.892          data.detroitmi.gov   \n",
       "4873           455.000                 8.833          data.detroitmi.gov   \n",
       "4875           432.000                 8.758          data.detroitmi.gov   \n",
       "4880           359.000                 8.492          data.detroitmi.gov   \n",
       "4889           243.000                 7.931          data.detroitmi.gov   \n",
       "\n",
       "                                                   mash  \\\n",
       "21    ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "22    ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "88                       ['safety', 'sa', 'dv', 'dchs']   \n",
       "101   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "160   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "161   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "231   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "239                      ['safety', 'sa', 'dv', 'dchs']   \n",
       "252                      ['safety', 'sa', 'dv', 'dchs']   \n",
       "272   ['baseline', 'land', 'oha', 'individual', 'rig...   \n",
       "275   ['culture', 'hawaiian', 'history', 'oha', 'cul...   \n",
       "286   ['hawaiian', 'homeless', 'housing', 'oha', 'so...   \n",
       "316   ['hawaiian', 'oha', 'population', 'race', 'sta...   \n",
       "340                                ['public', 'safety']   \n",
       "358                                ['public', 'safety']   \n",
       "394                                             ['oha']   \n",
       "431   ['baseline', 'land', 'oha', 'individual', 'rig...   \n",
       "437                                ['public', 'safety']   \n",
       "444                                ['public', 'safety']   \n",
       "459   ['a', 'new', 'day', 'for:', 'safety', 'net', '...   \n",
       "503   ['hawaiian', 'homeless', 'housing', 'oha', 'so...   \n",
       "517   ['a', 'new', 'day', 'for:', 'safety', 'net', '...   \n",
       "523                                             ['oha']   \n",
       "534   ['hawaiian', 'oha', 'population', 'race', 'sta...   \n",
       "568                      ['public', 'safety', 'health']   \n",
       "575                      ['public', 'safety', 'health']   \n",
       "583                      ['public', 'safety', 'health']   \n",
       "614   ['index', 'crime', 'crime', 'rate', 'rate', 'c...   \n",
       "632        ['crime', 'rate', 'index', 'crimes', 'utah']   \n",
       "655                      ['health', 'public', 'safety']   \n",
       "...                                                 ...   \n",
       "4262  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4286  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4287  ['crime', 'departments', 'police', 'public', '...   \n",
       "4290  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4292  ['2013', 'offenses', 'crime', 'ucr', 'public',...   \n",
       "4294  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4302  ['public', 'safety', 'personnel', 'police', 'p...   \n",
       "4329  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4362  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4388  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4399  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4407  ['crime', 'offenses', 'ucr', 'public', 'safety...   \n",
       "4431  ['police', 'law', 'enforcement', 'new', 'haven...   \n",
       "4439  ['accidents', 'crash', 'usodcensus', 'vehicle'...   \n",
       "4451  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4478      ['dmv', 'public', 'safety', 'transportation']   \n",
       "4524  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4528  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4529             ['infrastructure', 'public', 'safety']   \n",
       "4539             ['infrastructure', 'public', 'safety']   \n",
       "4812  ['crime', 'detroit', 'police', 'offenses', 'pu...   \n",
       "4828  ['victim-based', 'homicide', 'police', 'public...   \n",
       "4839  ['police', 'dpd', 'precincts', 'public', 'safe...   \n",
       "4859  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "4867  ['victim-based', 'carjacking', 'police', 'publ...   \n",
       "4871  ['2014', 'victim-based', 'homicide', 'police',...   \n",
       "4873  ['victim-based', 'homicide', 'police', 'public...   \n",
       "4875  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "4880  ['victim-based', 'carjacking', 'police', 'publ...   \n",
       "4889  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "\n",
       "                                               big_mash  \\\n",
       "21    ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "22    ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "88                       ['safety', 'sa', 'dv', 'dchs']   \n",
       "101   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "160   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "161   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "231   ['sexual', 'assault', 'domestic', 'violence', ...   \n",
       "239                      ['safety', 'sa', 'dv', 'dchs']   \n",
       "252                      ['safety', 'sa', 'dv', 'dchs']   \n",
       "272   ['baseline', 'land', 'oha', 'individual', 'rig...   \n",
       "275   ['culture', 'hawaiian', 'history', 'oha', 'cul...   \n",
       "286   ['hawaiian', 'homeless', 'housing', 'oha', 'so...   \n",
       "316   ['hawaiian', 'oha', 'population', 'race', 'sta...   \n",
       "340   ['public', 'safety', 'Includes', 'decibel', 'l...   \n",
       "358                                ['public', 'safety']   \n",
       "394                                             ['oha']   \n",
       "431   ['baseline', 'land', 'oha', 'individual', 'rig...   \n",
       "437                                ['public', 'safety']   \n",
       "444   ['public', 'safety', 'Includes', 'decibel', 'l...   \n",
       "459   ['a', 'new', 'day', 'for:', 'safety', 'net', '...   \n",
       "503   ['hawaiian', 'homeless', 'housing', 'oha', 'so...   \n",
       "517   ['a', 'new', 'day', 'for:', 'safety', 'net', '...   \n",
       "523                                             ['oha']   \n",
       "534   ['hawaiian', 'oha', 'population', 'race', 'sta...   \n",
       "568                      ['public', 'safety', 'health']   \n",
       "575                      ['public', 'safety', 'health']   \n",
       "583                      ['public', 'safety', 'health']   \n",
       "614   ['index', 'crime', 'crime', 'rate', 'rate', 'c...   \n",
       "632   ['crime', 'rate', 'index', 'crimes', 'utah', '...   \n",
       "655   ['health', 'public', 'safety', 'Alameda', 'Cou...   \n",
       "...                                                 ...   \n",
       "4262  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4286  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4287  ['crime', 'departments', 'police', 'public', '...   \n",
       "4290  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4292  ['2013', 'offenses', 'crime', 'ucr', 'public',...   \n",
       "4294  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4302  ['public', 'safety', 'personnel', 'police', 'p...   \n",
       "4329  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4362  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4388  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4399  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4407  ['crime', 'offenses', 'ucr', 'public', 'safety...   \n",
       "4431  ['police', 'law', 'enforcement', 'new', 'haven...   \n",
       "4439  ['accidents', 'crash', 'usodcensus', 'vehicle'...   \n",
       "4451  ['dmv', 'economy', 'public', 'safety', 'transp...   \n",
       "4478  ['dmv', 'public', 'safety', 'transportation', ...   \n",
       "4524  ['arrests', 'crime', 'ucr', 'public', 'safety'...   \n",
       "4528  ['offenses', 'crime', 'ucr', 'public', 'safety...   \n",
       "4529  ['infrastructure', 'public', 'safety', 'Connec...   \n",
       "4539  ['infrastructure', 'public', 'safety', 'Connec...   \n",
       "4812  ['crime', 'detroit', 'police', 'offenses', 'pu...   \n",
       "4828  ['victim-based', 'homicide', 'police', 'public...   \n",
       "4839  ['police', 'dpd', 'precincts', 'public', 'safe...   \n",
       "4859  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "4867  ['victim-based', 'carjacking', 'police', 'publ...   \n",
       "4871  ['2014', 'victim-based', 'homicide', 'police',...   \n",
       "4873  ['victim-based', 'homicide', 'police', 'public...   \n",
       "4875  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "4880  ['victim-based', 'carjacking', 'police', 'publ...   \n",
       "4889  ['victim-based', 'non-fatal', 'shooting', 'pol...   \n",
       "\n",
       "                                             topic_comp  \n",
       "21    [(16, 0.1616786051), (26, 0.431318681319), (50...  \n",
       "22    [(16, 0.161766956351), (26, 0.431318681319), (...  \n",
       "88    [(16, 0.203846153846), (26, 0.403846153846), (...  \n",
       "101   [(16, 0.162007823798), (26, 0.431318681319), (...  \n",
       "160   [(16, 0.161854234981), (26, 0.431318681319), (...  \n",
       "161   [(16, 0.161685334512), (26, 0.431318681319), (...  \n",
       "231   [(16, 0.161934869861), (26, 0.431318681319), (...  \n",
       "239   [(16, 0.203846153846), (26, 0.403846153846), (...  \n",
       "252   [(16, 0.203846153846), (26, 0.403846153846), (...  \n",
       "272   [(4, 0.0445430353893), (9, 0.13570023278), (21...  \n",
       "275   [(1, 0.137036692987), (5, 0.188701923077), (6,...  \n",
       "286   [(17, 0.145604395604), (18, 0.145604395604), (...  \n",
       "316   [(2, 0.169871794872), (26, 0.503205128204), (4...  \n",
       "340          [(5, 0.33547008547), (26, 0.557692307692)]  \n",
       "358                              [(26, 0.673076923077)]  \n",
       "394                              [(26, 0.509615384615)]  \n",
       "431   [(4, 0.0444289989737), (9, 0.13576160706), (21...  \n",
       "437                              [(26, 0.673076923077)]  \n",
       "444          [(5, 0.33547008547), (26, 0.557692307692)]  \n",
       "459   [(26, 0.33547008547), (29, 0.113247863248), (4...  \n",
       "503   [(17, 0.145604395604), (18, 0.145604395604), (...  \n",
       "517   [(26, 0.33547008547), (29, 0.113247863248), (4...  \n",
       "523                              [(26, 0.509615384615)]  \n",
       "534   [(2, 0.169871794872), (26, 0.503205128205), (4...  \n",
       "568         [(2, 0.254807692308), (26, 0.504807692308)]  \n",
       "575         [(2, 0.254807692308), (26, 0.504807692308)]  \n",
       "583         [(2, 0.254807692308), (26, 0.504807692308)]  \n",
       "614   [(0, 0.0728021978022), (2, 0.135665761033), (7...  \n",
       "632   [(2, 0.0849358974359), (7, 0.168269230769), (2...  \n",
       "655   [(2, 0.131321120447), (23, 0.106439139191), (2...  \n",
       "...                                                 ...  \n",
       "4262  [(5, 0.0630686980655), (8, 0.0202852809573), (...  \n",
       "4286  [(9, 0.101876589899), (23, 0.0512549258423), (...  \n",
       "4287  [(25, 0.0533695146241), (26, 0.374136972605), ...  \n",
       "4290  [(5, 0.0630712893518), (8, 0.020285294677), (1...  \n",
       "4292  [(5, 0.0630705946552), (8, 0.0202853012731), (...  \n",
       "4294  [(9, 0.101877187909), (23, 0.0512551214156), (...  \n",
       "4302  [(15, 0.179307951581), (16, 0.0578863990146), ...  \n",
       "4329  [(5, 0.0630710354293), (8, 0.0202852968573), (...  \n",
       "4362  [(2, 0.169871794872), (3, 0.169871794872), (16...  \n",
       "4388  [(2, 0.169871794872), (3, 0.169871794872), (16...  \n",
       "4399  [(5, 0.0630705851969), (8, 0.0202852865935), (...  \n",
       "4407  [(5, 0.0630721336619), (8, 0.0202853058285), (...  \n",
       "4431  [(5, 0.0784023668639), (26, 0.539940828397), (...  \n",
       "4439  [(3, 0.0701899402726), (7, 0.0131257766537), (...  \n",
       "4451  [(2, 0.169871794872), (3, 0.169871794872), (16...  \n",
       "4478  [(2, 0.0926573426573), (3, 0.0926573426573), (...  \n",
       "4524  [(2, 0.0674580972245), (8, 0.0351770567299), (...  \n",
       "4528  [(5, 0.0630711277688), (8, 0.0202853015264), (...  \n",
       "4529  [(1, 0.0104482683654), (2, 0.0100505055654), (...  \n",
       "4539  [(2, 0.0458611394719), (3, 0.0111117542859), (...  \n",
       "4812  [(8, 0.0613390263294), (20, 0.0599547511312), ...  \n",
       "4828  [(4, 0.235396672376), (23, 0.110301885217), (2...  \n",
       "4839  [(23, 0.213674846029), (26, 0.331946920678), (...  \n",
       "4859  [(4, 0.208449561457), (14, 0.0478343078921), (...  \n",
       "4867  [(4, 0.239361338926), (22, 0.0441467380176), (...  \n",
       "4871  [(4, 0.235266934147), (23, 0.110335410239), (2...  \n",
       "4873  [(4, 0.235313755179), (23, 0.110276662358), (2...  \n",
       "4875  [(4, 0.208435187859), (14, 0.0478343282382), (...  \n",
       "4880  [(4, 0.239419303997), (22, 0.0441419106655), (...  \n",
       "4889  [(4, 0.208349233919), (14, 0.0478344892368), (...  \n",
       "\n",
       "[200 rows x 20 columns]"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "twenty_six = return_data_by_top_topic(topics, 26)\n",
    "twenty_six[0:200]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions:\n",
    "**How you count \"popularity\" matters.**\n",
    "- Nevertheless, **transportation and crime/public safety dataset popularity is resilient** across different measures of popularity that we've defined.\n",
    "\n",
    "**When you \"dampen\" the effects of cities/states whose open data portals get a high volume of traffic, popularity rankings change somewhat**. \n",
    "- This suggests that \"popularity\" (as we've defined it) in very large and/or populated areas is different than other areas. \n",
    "- This may be because of a genuine difference in citizen preference, a difference in what data these governments choose to display prominently (everyone care about traffic, but California *really* cares about traffic), or a skew from users across the country and world accessing this data (NYC, for instance, is a popular domain for datasets for data science research projects).\n",
    "\n",
    "**We argue that \"dampened\" popularity is the most appropriate metric for wide applicability to a variety of government organizations**. \n",
    " - It appears to truly reduce the influence of large CA, NYC, and TX datasets. Not that there's anything wrong with these places or portals! But we want this information to be useful to smaller cities as well.\n",
    " - **Using dampened popularity, the following topics show notable popularity:**\n",
    "    - Crime Reports and Public Safety Response\n",
    "    - Personal Transportation and Traffic\n",
    "    - Community Economic Development & Housing\n",
    "    - Business Licensing\n",
    "    - Project Permitting and Financing\n",
    "\n",
    "**Popularity is *always* going to be affected by what a city or state chooses to display prominently on its web portal.**\n",
    "- For example -- using our old metric of popularity, ‘energy environment electricity air sustainable action climate city clean facilities’ was the most popular topic in Providence, RI. Providence, RI displays a link to sustainability info right on its front page. It is hard to tell what is cause and what is effect. \n",
    "- **This is not a controlled experiment. We hope that a sample of 141 open data portals controls enough for individual variation in what datasets are easiest for users to access in a given portal**.\n",
    "\n",
    "**No meta analysis alone can tell a city what kind of open data will be most popular, let alone most urgent to the public interest.**\n",
    "- As we saw from our undampened popularity measures, cities have a variety of most popular topics. This is influenced by both local preference and local decisions about what datasets to display prominently.\n",
    "- Just because a dataset is popular doesn't mean it serves the most urgent or broadest public purpose.\n",
    "    - Open data users are not a cross section of voters or the general public. Permitting information is important and useful for developers. It should be open and accessible for concerned citizens as well. But other types of information may be more useful to the average citizen. \n",
    "\n",
    "**These results are only applicable to domains hosted by Socrata. We'd expect that a wider range of cities/states wouldn't alter results that much, but we can't be sure.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Caveats\n",
    "### Errors, uncertainty, and additional notes:\n",
    "**Again, not every dataset is tagged thoroughy, accurately, or appropriately.** \n",
    "  - Some cities/portals just give their datasets weird names or use stock descriptions for every single category of open data. That's why we've formed a bland or vague topic or two.\n",
    "    \n",
    "**Proper names obviously skew results somewhat; a proper name doesn't really tell us about the content of a dataset.**\n",
    "  - However, it's just not feasible to remove every proper name as a stopword; at least not without extensive trial and error\n",
    "  - \"Dampened Popularity\" best avoids this effect by requiring a cut off (0.2), only counting \"winning\" topics, and normalizing popularity scores *within* a city.\n",
    " \n",
    "**We are only displaying the top 10 words by affinity for each topic. The topics go much deeper, picking up some marginal affinity from words that may not make as much sense to the human eye.**\n",
    "   - This is also a weakness of our model. Real life, human eye topics are of varying length in terms of words. But our model forms topics of equal sizes.\n",
    "   \n",
    "**Why not KMeans or some other form of clustering to group datasets?**\n",
    "- Those with strong backgrounds in data science may ask why we didn't turn each dataset's \"mash\" into a tf-idf vector and then \"cluster\" datasets based on their distance in mathematical space. We tried! However, the data formed by the tf-idf vectorizer was too sparse. KMeans' explicit number of clusters forced dissimilar datasets together too often. DBSCAN's ability to create clusters of varying density and assign data points to an 'outlier' cluster produced highly specific clusters that didn't cut across cities as much as we'd like. And too often a DBSCAN assigned over half of our datasets to an outlier category.\n",
    "- **We'd encourage those with strong backgrounds in data science and machine learning to take a cut at this. It's possible clustering could still produce useful formations, and it would be a more stable formation our rather reverse process using LDA topic modeling.**\n",
    "\n",
    "**The human brain interprets our clusters at the end. It's up to us to determine what our \"topic\" should really be named in human English. Two LDA topics may be very similar real-life topics (we can see this with our two transportation datasets - probably because that's such a prevalent real-life category).** \n",
    "  - Then again, all of this is labeled by humans. Back to our first point, there is always room for disagreement/debate in what \"subject\" a dataset is about, and how narrow to make subjects."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}