11 - Lesson - Natural Language Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ba959ea8",
   "metadata": {},
   "source": [
    "# Natural Language Processing\n",
    "### Goal of lesson\n",
    "- How the simple syntax of language can be parsed\n",
    "- What Context-Free Grammar (CFG) is\n",
    "- Use it to parse text\n",
    "- Understand text in trigrams\n",
    "- See how it can be used to generate predictions\n",
    "\n",
    "### What is Natural Language Processing\n",
    "- Automatic computational processing of human languages\n",
    "- Includes \n",
    "    - Algorithms that take human written language as input\n",
    "    - Algorithms that produce natural text\n",
    "\n",
    "- Examples include\n",
    "    - Automatic summarization\n",
    "    - Language identification\n",
    "    - Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88cafa61",
   "metadata": {},
   "source": [
    "### Syntax\n",
    "- One basic description of a language's syntax is the sequence in which the subject, verb, and object usually appear in sentences.\n",
    "\n",
    "### Formal Grammar\n",
    "- A system of rules for genrating sentences in a language\n",
    "- A grammar is usually thought of as a language generator ([wiki](https://en.wikipedia.org/wiki/Formal_grammar))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "338af3bc",
   "metadata": {},
   "source": [
    "### Context-Free Grammar (CFG)\n",
    "- A formal grammar is \"context free\" if its production rules can be applied regardless of the context of a nonterminal ([wiki](https://en.wikipedia.org/wiki/Context-free_grammar))."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec8e4f6",
   "metadata": {},
   "source": [
    "> #### Programming Notes:\n",
    "> - Libraries used\n",
    ">     - [**nltk**](https://www.nltk.org) - Natural Language Toolkit\n",
    ">     - [**os**](https://docs.python.org/3/library/os.html) Miscellaneous operating system interfaces\n",
    ">     - [**collections**](https://docs.python.org/3/library/collections.html) Container datatypes\n",
    ">     - [**markovify**](https://pypi.org/project/markovify/) A simple, extensible Markov chain generato\n",
    "> - Functionality and concepts used\n",
    ">     - [**ChartParser**](https://tedboy.github.io/nlps/generated/generated/nltk.ChartParser.html) generic chart parser\n",
    ">     - **List Comprehension** to convert data ([Lecture on **List Comprehension**](https://youtu.be/vCYEvtfXdig))\n",
    ">     - [**Counter**](https://docs.python.org/3/library/collections.html#collections.Counter) a dict subclass for counting hashable objects\n",
    ">     - [**markovify.Text**](https://pypi.org/project/markovify/) to create your Markov Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ca06c48",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "769c40c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "147dfc18",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "she saw a car\n",
      "         S             \n",
      "  _______|___           \n",
      " |           VP        \n",
      " |    _______|___       \n",
      " NP  |           NP    \n",
      " |   |        ___|___   \n",
      " N   V       D       N \n",
      " |   |       |       |  \n",
      "she saw      a      car\n",
      "\n"
     ]
    }
   ],
   "source": [
    "grammar = nltk.CFG.fromstring(\"\"\"\n",
    "    S -> NP VP\n",
    "\n",
    "    NP -> D N | N\n",
    "    VP -> V | V NP\n",
    "\n",
    "    D -> \"the\" | \"a\"\n",
    "    N -> \"she\" | \"city\" | \"car\"\n",
    "    V -> \"saw\" | \"walked\"    \n",
    "\"\"\")\n",
    "\n",
    "parser = nltk.ChartParser(grammar)\n",
    "\n",
    "sentence = input().split()\n",
    "\n",
    "for tree in parser.parse(sentence):\n",
    "    tree.pretty_print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6fa8235",
   "metadata": {},
   "source": [
    "### Challenge with CFG\n",
    "- You need to encode all possibilities\n",
    "\n",
    "### Idea\n",
    "- Understand text in small subsets\n",
    "- **$n$-gram**\n",
    "    - a contiguous sequence of $n$ items from a sample text\n",
    "- **Word $n$-gram**\n",
    "    - a contiguous sequence of $n$ words from a sample text\n",
    "- **unigram**\n",
    "    - 1 items in sequence\n",
    "- **bigram**\n",
    "    - 2 items in sequence\n",
    "- **trigram**\n",
    "    - 3 items in sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30889d25",
   "metadata": {},
   "source": [
    "### Word Tokenization\n",
    "- the task of splitting a sequence of words into tokens\n",
    "\n",
    "- Considerations: comma, punktuation, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4312a067",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1ade98c4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /Users/rune/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# You need to download this\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "14cc1cb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "content = []\n",
    "for filename in os.listdir('files/holmes/'):\n",
    "    with open(f'files/holmes/{filename}') as f:\n",
    "        content.append(f.read())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "1495276e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "21"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "767c2288",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = []\n",
    "for item in content:\n",
    "    corpus.extend([word.lower() for word in nltk.word_tokenize(item) if any(c.isalpha() for c in word)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "f5297b15",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178205"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e55e7bf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "ngrams = Counter(nltk.ngrams(corpus, 3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c4cfafa6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80: ('it', 'was', 'a')\n",
      "71: ('one', 'of', 'the')\n",
      "65: ('i', 'think', 'that')\n",
      "59: ('out', 'of', 'the')\n",
      "55: ('that', 'it', 'was')\n",
      "55: ('that', 'he', 'had')\n",
      "55: ('there', 'was', 'a')\n",
      "55: ('that', 'he', 'was')\n",
      "52: ('it', 'is', 'a')\n",
      "49: ('i', 'can', 'not')\n"
     ]
    }
   ],
   "source": [
    "for ngram, freq in ngrams.most_common(10):\n",
    "    print(f'{freq}: {ngram}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1373e8d6",
   "metadata": {},
   "source": [
    "### Markov Model\n",
    "- A Markov chain is a stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous even ([wiki](https://en.wikipedia.org/wiki/Markov_chain))\n",
    "- Or as the example above:\n",
    "    - Given any two words -> you have probabilities for the next word"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d47ff960",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install markovify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "27420682",
   "metadata": {},
   "outputs": [],
   "source": [
    "import markovify"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "4325b338",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('files/shakespeare.txt') as f:\n",
    "    text = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5b20b5c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = markovify.Text(text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4828ee83",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'In the wars; defeat thy favor with an ordinary pitch, Who else but I, his forlorn duchess, Was made much poorer by it; but first, how get hence.'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.make_sentence()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "78bbd0ca",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}