1. Introduction - Python & Open Collections API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Introduction - Python & Open Collections API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to Jupyter, through this interface I will be showing you the following:\n",
    "\n",
    "  - Python - A programming language that lets you work quickly. - [Documentation](https://docs.python.org/3/)\n",
    "  - NLTK - Natural Languge Toolkit - a Python Library for working with written language data. -  [Documentation](http://www.nltk.org/book/)\n",
    "  - Open Collections API - Our \"Application Programming Interface\" which will allow you to import full text. - [Documentation](https://open.library.ubc.ca/docs)\n",
    "  \n",
    "Python is a great language for data analysis, more experienced programmers might want to use R, but Python is a nice entry point for everyone.\n",
    "\n",
    "**If you don't know Python, or any programming for that matter, please remain calm you won't need to do any programming** throughout this talk, however if you do know Python you can feel free to edit any of the code and have your notebook update accordingly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## URLS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ocUrl = 'https://open.library.ubc.ca/'\n",
    "ocApiUrl = 'https://oc-index.library.ubc.ca' # APPY URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting API Key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can get your own API key at https://open.library.ubc.ca/research"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "apiKey = 'ac40e6c2cb345593ed1691e0a8b601bba398e42d85f81f893c5ab709cec63c6c'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choosing a collection & item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "collection = 'bcbooks'\n",
    "itemId = '1.0222552'\n",
    "print('\\n Using item: '+ocUrl+'collections/'+collection+'/items/'+itemId)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import json, requests\n",
    "itemUrl = ocApiUrl+'/collections/'+collection+'/items/'+itemId+'?apiKey='+apiKey\n",
    "apiResponse = requests.get(itemUrl).json()\n",
    "item = apiResponse['data']\n",
    "print(json.dumps(item, sort_keys=True, indent=4, separators=(',', ': ')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting the full text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fullText = item['FullText'][0]['value']\n",
    "print(fullText)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clean the full text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order for better results from our analysis later we need to clean up the full text. \n",
    "\n",
    "This could be a project within itself and will differ item to item so for the intial run I have just set the full text to get lowered so 'Canada' and 'canada' aren't considered two different words.\n",
    "\n",
    "I've also provided a basic regex that you can uncomment to strip everything other than words from the full text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "# Lower case full text\n",
    "cleanFullText = fullText.lower()\n",
    "# Remove everything but words\n",
    "pattern = re.compile('[\\W_]+')\n",
    "cleanFullText = pattern.sub(' ', cleanFullText)\n",
    "\n",
    "print(cleanFullText)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use full text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could now post the clean full text into a tool like [Voyant](http://voyant-tools.org/) to explore it further."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}