api_example_SA_Hansard.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "262c5bce-b55c-4fa9-9d47-83526a50784f",
   "metadata": {},
   "source": [
    "# Working with South Australian Hansard API\n",
    "\n",
    "This notebook provides some examples for how to interact with the API for the proceedings of the South Australian Parliament. This API let's you retrieve parts of the proceedings by date and house, and also allows you (with more effort) to identify all speeches, questions and answers by particular members.\n",
    "\n",
    "The first part of this notebook shows some examples of interacting with the API and the returned data.\n",
    "\n",
    "The second part of this notebook retrieves all speeches in the current Parliamentary session (since 2022) by the current South Australian Premier and counts the words used across his speeches.\n",
    "\n",
    "## Reference Links:\n",
    "\n",
    "- [South Australian Government Hansard API Documentation](https://hansardsearch.parliament.sa.gov.au/docs/api/index.html)\n",
    "- [South Australian Government Hansard Web Search](https://hansardsearch.parliament.sa.gov.au/search)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e10ab15c-8d0b-42e9-b947-d7b0c6e6903b",
   "metadata": {},
   "source": [
    "## Our First API Request\n",
    "\n",
    "This first examples shows how to retrieve one of the handy pieces of information we can request from the API - a requests for the list of chambers.\n",
    "\n",
    "This also shows an example of using the [JSON data format](https://en.wikipedia.org/wiki/JSON) returned by the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2f9e844a-85fa-483d-8aaf-7f2a25626adf",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw response: [{'id': '2310c894d85048eb8051105ca3d348a8', 'name': 'Legislative Council'}, {'id': '8758cca8bb634275820b3047e39bfe8b', 'name': 'Estimates Committee A - Answers to Questions'}, {'id': 'c2ff907f907046a8bb61467a5e2ccc9f', 'name': 'Estimates Committee B'}, {'id': 'd35e3c30ea624513990bacfe6654dfbe', 'name': 'Estimates Committee B - Answers to Questions'}, {'id': 'b2669775c14346e7b73e1eaa25470874', 'name': 'Estimates Committee A'}, {'id': 'd6d71d2191444d70be863757822ef005', 'name': 'House of Assembly'}]\n",
      "\n",
      "{'id': '2310c894d85048eb8051105ca3d348a8', 'name': 'Legislative Council'}\n",
      "{'id': '8758cca8bb634275820b3047e39bfe8b', 'name': 'Estimates Committee A - Answers to Questions'}\n",
      "{'id': 'c2ff907f907046a8bb61467a5e2ccc9f', 'name': 'Estimates Committee B'}\n",
      "{'id': 'd35e3c30ea624513990bacfe6654dfbe', 'name': 'Estimates Committee B - Answers to Questions'}\n",
      "{'id': 'b2669775c14346e7b73e1eaa25470874', 'name': 'Estimates Committee A'}\n",
      "{'id': 'd6d71d2191444d70be863757822ef005', 'name': 'House of Assembly'}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "# We'll need to refer to this domain a lot, but will be changing the path at the end of \n",
    "# to work with different API 'endpoints'. This assigns a label api_url to a list of letters \n",
    "# that represent the domain (everything inside the quote ' characters).\n",
    "api_url = 'https://hansardsearch.parliament.sa.gov.au'\n",
    "\n",
    "# First request - get current chambers:\n",
    "# requests.get retrieves the URL, and we save it in the variable 'response'\n",
    "# Note that in python using + with strings puts them together - 'a' + 'b' = 'ab'\n",
    "response = requests.get(api_url + '/api/chamber')\n",
    "\n",
    "# print is a Python function to explicitly display output.\n",
    "# response.json() parses the data returned by the API so we can manipulate it in Python.\n",
    "print('Raw response:', response.json())\n",
    "print()\n",
    "\n",
    "# The raw response is a little hard to read\n",
    "# But since it starts with a '[' character, we can recognise that it's a list of items\n",
    "# and we can display it more nicely by looping through with a for loop.\n",
    "for chamber in response.json():\n",
    "    print(chamber)\n",
    "\n",
    "# This response shows that there are six 'chambers' for the organisation of the business of the South Australian government - \n",
    "# The expecting House of Assembly and Legislative Council, and four extra 'chambers' relating to estimates committees."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52ef6dae-9c19-4de6-a247-332e0ecb3687",
   "metadata": {},
   "source": [
    "## Search for 'events' in a time period\n",
    "\n",
    "Parliaments do not sit all the time - sometimes we need to be able to figure out when parliaments actually sat so we can start retrieving the right information for us. For example, if you're looking for parliamentary speeches about a particular event, you might want to find the first sitting days after that event."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bee37866-c002-4eb5-8456-9f1a88d1d0c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'events': [{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-26/pdf', 'subjectCount': 54, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-26/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-26/pdf', 'subjectCount': 111, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-26/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-25/pdf', 'subjectCount': 119, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-25/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-25/pdf', 'subjectCount': 66, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-25/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-24/pdf', 'subjectCount': 66, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-24/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-24/pdf', 'subjectCount': 130, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-24/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-19', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-19/pdf', 'subjectCount': 119, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-19/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-19', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-19/pdf', 'subjectCount': 55, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-19/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-18', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-18/pdf', 'subjectCount': 129, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-18/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-18', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-18/pdf', 'subjectCount': 70, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-18/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-17', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-17/pdf', 'subjectCount': 110, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-17/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-17', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-17/pdf', 'subjectCount': 56, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-17/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-26/pdf', 'subjectCount': 50, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-26/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-26/pdf', 'subjectCount': 155, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-26/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-25/pdf', 'subjectCount': 82, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-25/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-25/pdf', 'subjectCount': 120, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-25/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/pdf', 'subjectCount': 49, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-24/pdf', 'subjectCount': 89, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-24/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-12', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-12/pdf', 'subjectCount': 60, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-12/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-12', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-12/pdf', 'subjectCount': 85, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-12/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-11', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-11/pdf', 'subjectCount': 89, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-11/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-11', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-11/pdf', 'subjectCount': 88, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-11/toc'}, {'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-10', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-10/pdf', 'subjectCount': 87, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-10/toc'}, {'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-10', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-10/pdf', 'subjectCount': 100, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-10/toc'}], 'total': 24}\n"
     ]
    }
   ],
   "source": [
    "# Let's use the events endpoint to see what happened in the first three months of 2015\n",
    "# This API takes a start data and an end date as params.\n",
    "# If you've seen a URL like www.example.com/path?x=1&y=2\n",
    "# The x=1&y=2 are parameters, representing x = 1 and y = 2 respectively.\n",
    "# The params argument to the followign requests call handles turning\n",
    "# the dates we have into the right URL.\n",
    "response = requests.get(api_url + '/api/hansard/events/', params={'startDate': '2015-01-01', 'endDate': '2015-04-01'})\n",
    "\n",
    "print(response.json())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e842fa68-11bf-4417-84d8-95bb6e9707bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-26/pdf', 'subjectCount': 54, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-26/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-26/pdf', 'subjectCount': 111, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-26/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-25/pdf', 'subjectCount': 119, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-25/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-25/pdf', 'subjectCount': 66, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-25/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-24/pdf', 'subjectCount': 66, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-24/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-24/pdf', 'subjectCount': 130, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-24/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-19', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-19/pdf', 'subjectCount': 119, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-19/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-19', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-19/pdf', 'subjectCount': 55, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-19/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-18', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-18/pdf', 'subjectCount': 129, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-18/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-18', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-18/pdf', 'subjectCount': 70, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-18/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-03-17', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-17/pdf', 'subjectCount': 110, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-03-17/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-03-17', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-17/pdf', 'subjectCount': 56, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-03-17/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-26/pdf', 'subjectCount': 50, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-26/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-26', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-26/pdf', 'subjectCount': 155, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-26/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-25/pdf', 'subjectCount': 82, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-25/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-25', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-25/pdf', 'subjectCount': 120, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-25/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/pdf', 'subjectCount': 49, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-24', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-24/pdf', 'subjectCount': 89, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-24/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-12', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-12/pdf', 'subjectCount': 60, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-12/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-12', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-12/pdf', 'subjectCount': 85, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-12/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-11', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-11/pdf', 'subjectCount': 89, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-11/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-11', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-11/pdf', 'subjectCount': 88, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-11/toc'}\n",
      "{'houseName': 'House of Assembly', 'houseCode': 'lh', 'date': '2015-02-10', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-10/pdf', 'subjectCount': 87, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/lh/2015-02-10/toc'}\n",
      "{'houseName': 'Legislative Council', 'houseCode': 'uh', 'date': '2015-02-10', 'pdfUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-10/pdf', 'subjectCount': 100, 'tocUrl': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-10/toc'}\n"
     ]
    }
   ],
   "source": [
    "# The raw response of this output is hard to read - there's a lot.\n",
    "# The output starts with '{', indicating that this is a Python dictionary\n",
    "# This is map between a 'key', to values and can be arbitrarily nested.\n",
    "# We can access the 'events' key in Python like below to format this data more readably\n",
    "for event in response.json()['events']:\n",
    "    print(event)\n",
    "\n",
    "# Now we can see there are lots of events in this time window - and the returned data indicates the house, date, and links to\n",
    "# specific parts of the proceedings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e18028d0-4cba-4347-93ca-bc4251579461",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'Commencement', 'subjects': [{'type': 'subject', 'names': ['Commencement'], 'index': 1, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/1?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/1?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/1?contentType=text%2Fjson'}}]}\n",
      "{'name': 'Parliamentary Procedure', 'subjects': [{'type': 'subject', 'names': ['Papers'], 'index': 2, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/2?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/2?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/2?contentType=text%2Fjson'}}]}\n",
      "{'name': 'Ministerial Statement', 'subjects': [{'type': 'subject', 'names': ['Greyhound Racing'], 'index': 3, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/3?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/3?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/3?contentType=text%2Fjson'}, 'speeches': [{'type': 'speech', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:20:06', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Site Contamination, Clovelly Park and Mitchell Park'], 'index': 4, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/4?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/4?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/4?contentType=text%2Fjson'}, 'speeches': [{'type': 'speech', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:22:00', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Defence Shipbuilding'], 'index': 5, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/5?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/5?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/5?contentType=text%2Fjson'}, 'speeches': [{'type': 'speech', 'speaker': {'speaker': '4697', 'name': 'The Hon. K.J. MAHER', 'talktime': '2015-02-24T14:25:38', 'electorate': 'Minister for Manufacturing and Innovation, Minister for Automotive Transformation, Minister for Aboriginal Affairs and Reconciliation'}}]}]}\n",
      "{'name': 'Question Time', 'subjects': [{'type': 'subject', 'names': [\"Members' Conduct\"], 'index': 6, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/6?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/6?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/6?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '1820', 'name': 'The Hon. D.W. RIDGWAY', 'talktime': '2015-02-24T14:26:32', 'electorate': 'Leader of the Opposition'}}, {'type': 'answer', 'speaker': {'speaker': '3125', 'name': 'The President', 'talktime': '2015-02-24T14:27:14'}}]}, {'type': 'subject', 'names': ['APY Lands, Governance'], 'index': 7, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/7?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/7?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/7?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '1819', 'name': 'The Hon. T.J. STEPHENS', 'talktime': '2015-02-24T14:27:23'}}, {'type': 'answer', 'speaker': {'speaker': '4697', 'name': 'The Hon. K.J. MAHER', 'talktime': '2015-02-24T14:29:33', 'electorate': 'Minister for Manufacturing and Innovation, Minister for Automotive Transformation, Minister for Aboriginal Affairs and Reconciliation'}}]}, {'type': 'subject', 'names': ['APY Lands, Governance'], 'index': 8, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/8?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/8?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/8?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '1819', 'name': 'The Hon. T.J. STEPHENS', 'talktime': '2015-02-24T14:32:18'}}, {'type': 'answer', 'speaker': {'speaker': '4697', 'name': 'The Hon. K.J. MAHER', 'talktime': '2015-02-24T14:32:40', 'electorate': 'Minister for Manufacturing and Innovation, Minister for Automotive Transformation, Minister for Aboriginal Affairs and Reconciliation'}}]}, {'type': 'subject', 'names': ['Site Contamination, Clovelly Park and Mitchell Park'], 'index': 9, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/9?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/9?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/9?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '2742', 'name': 'The Hon. J.M.A. LENSINK', 'talktime': '2015-02-24T14:32:53'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:33:36', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Site Contamination, Clovelly Park and Mitchell Park'], 'index': 10, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/10?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/10?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/10?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '2742', 'name': 'The Hon. J.M.A. LENSINK', 'talktime': '2015-02-24T14:36:48'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:37:06', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Site Contamination, Clovelly Park and Mitchell Park'], 'index': 11, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/11?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/11?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/11?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '2742', 'name': 'The Hon. J.M.A. LENSINK', 'talktime': '2015-02-24T14:38:03'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:38:18', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Higher Education'], 'index': 12, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/12?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/12?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/12?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '1822', 'name': 'The Hon. J.M. GAZZOLA', 'talktime': '2015-02-24T14:38:32'}}, {'type': 'answer', 'speaker': {'speaker': '1821', 'name': 'The Hon. G.E. GAGO', 'talktime': '2015-02-24T14:39:15', 'electorate': 'Minister for Employment, Higher Education and Skills, Minister for Science and Information Economy, Minister for the Status of Women, Minister for Business Services and Consumers'}}]}, {'type': 'subject', 'names': ['Greyhound Racing'], 'index': 13, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/13?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/13?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/13?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '4363', 'name': 'The Hon. T.A. FRANKS', 'talktime': '2015-02-24T14:43:30'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:45:58', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Para Wirra Recreation Park'], 'index': 14, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/14?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/14?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/14?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '599', 'name': 'The Hon. J.S.L. DAWKINS', 'talktime': '2015-02-24T14:48:29'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:50:50', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Para Wirra Recreation Park'], 'index': 15, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/15?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/15?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/15?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '599', 'name': 'The Hon. J.S.L. DAWKINS', 'talktime': '2015-02-24T14:58:07'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:58:33', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Para Wirra Recreation Park'], 'index': 16, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/16?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/16?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/16?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '599', 'name': 'The Hon. J.S.L. DAWKINS', 'talktime': '2015-02-24T14:58:48'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T14:59:18', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Para Wirra Recreation Park'], 'index': 17, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/17?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/17?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/17?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '599', 'name': 'The Hon. J.S.L. DAWKINS', 'talktime': '2015-02-24T15:00:25'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T15:00:32', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Goyder Institute Climate Projections'], 'index': 18, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/18?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/18?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/18?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '4564', 'name': 'The Hon. G.A. KANDELAARS', 'talktime': '2015-02-24T15:01:02'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T15:01:22', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Marryatville High School Library'], 'index': 19, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/19?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/19?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/19?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '4364', 'name': 'The Hon. K.L. VINCENT', 'talktime': '2015-02-24T15:07:05'}}, {'type': 'answer', 'speaker': {'speaker': '3122', 'name': 'The Hon. I.K. HUNTER', 'talktime': '2015-02-24T15:08:10', 'electorate': 'Minister for Sustainability, Environment and Conservation, Minister for Water and the River Murray, Minister for Climate Change'}}]}, {'type': 'subject', 'names': ['Business and Consumer Confidence'], 'index': 20, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/20?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/20?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/20?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '4362', 'name': 'The Hon. J.S. LEE', 'talktime': '2015-02-24T15:08:30'}}, {'type': 'answer', 'speaker': {'speaker': '1821', 'name': 'The Hon. G.E. GAGO', 'talktime': '2015-02-24T15:09:59', 'electorate': 'Minister for Employment, Higher Education and Skills, Minister for Science and Information Economy, Minister for the Status of Women, Minister for Business Services and Consumers'}}]}, {'type': 'subject', 'names': ['Skills for All'], 'index': 21, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/21?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/21?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/21?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '3489', 'name': 'The Hon. R.L. BROKENSHIRE', 'talktime': '2015-02-24T15:18:04'}}, {'type': 'answer', 'speaker': {'speaker': '1821', 'name': 'The Hon. G.E. GAGO', 'talktime': '2015-02-24T15:20:19', 'electorate': 'Minister for Employment, Higher Education and Skills, Minister for Science and Information Economy, Minister for the Status of Women, Minister for Business Services and Consumers'}}]}, {'type': 'subject', 'names': ['Far West Mojo'], 'index': 22, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/22?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/22?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/22?contentType=text%2Fjson'}, 'speeches': [{'type': 'question', 'speaker': {'speaker': '4867', 'name': 'The Hon. T.T. NGO', 'talktime': '2015-02-24T15:24:55'}}, {'type': 'answer', 'speaker': {'speaker': '4697', 'name': 'The Hon. K.J. MAHER', 'talktime': '2015-02-24T15:25:34', 'electorate': 'Minister for Manufacturing and Innovation, Minister for Automotive Transformation, Minister for Aboriginal Affairs and Reconciliation'}}]}]}\n",
      "{'name': 'Address in Reply', 'subjects': [{'type': 'subject', 'names': ['Address in Reply'], 'index': 23, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/23?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/23?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/23?contentType=text%2Fjson'}, 'speeches': [{'type': 'speech', 'speaker': {'speaker': '1820', 'name': 'The Hon. D.W. RIDGWAY', 'talktime': '2015-02-24T15:30:04', 'electorate': 'Leader of the Opposition'}}, {'type': 'speech', 'speaker': {'speaker': '2742', 'name': 'The Hon. J.M.A. LENSINK', 'talktime': '2015-02-24T15:51:00'}}, {'type': 'speech', 'speaker': {'speaker': '3130', 'name': 'The Hon. M.C. PARNELL', 'talktime': '2015-02-24T16:07:16'}}, {'type': 'speech', 'speaker': {'speaker': '1819', 'name': 'The Hon. T.J. STEPHENS', 'talktime': '2015-02-24T16:36:20'}}, {'type': 'speech', 'speaker': {'speaker': '599', 'name': 'The Hon. J.S.L. DAWKINS', 'talktime': '2015-02-24T16:55:24'}}]}]}\n",
      "{'name': 'Bills', 'subjects': [{'type': 'subject', 'names': ['Fair Work (Miscellaneous) Amendment Bill'], 'index': 24, 'links': {'html': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/24?contentType=text%2Fhtml', 'xml': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/24?contentType=text%2Fxml', 'json': 'https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/subject/24?contentType=text%2Fjson'}, 'subproceedings': [{'type': 'subproceeding', 'name': 'Second Reading', 'speech': [{'type': 'speech', 'speaker': {'speaker': '605', 'name': 'The Hon. R.I. LUCAS', 'talktime': '2015-02-24T17:24:12'}}, {'type': 'speech', 'speaker': {'speaker': '4564', 'name': 'The Hon. G.A. KANDELAARS', 'talktime': '2015-02-24T17:38:15'}}, {'type': 'speech', 'speaker': {'speaker': '1821', 'name': 'The Hon. G.E. GAGO', 'talktime': '2015-02-24T17:45:08', 'electorate': 'Minister for Employment, Higher Education and Skills, Minister for Science and Information Economy, Minister for the Status of Women, Minister for Business Services and Consumers'}}]}, {'type': 'subproceeding', 'name': 'Committee Stage'}]}]}\n"
     ]
    }
   ],
   "source": [
    "# get the table of contents for a date of the SA upper house - we could use this to retrieve everything said\n",
    "# in a particular house on a particular day.\n",
    "response = requests.get('https://hansardsearch.parliament.sa.gov.au/api/hansard/uh/2015-02-24/toc')\n",
    "\n",
    "# We can inspect and pull apart this response to see all the entries and their associated links for each part of the \n",
    "# table of contents.\n",
    "table_of_contents = response.json()\n",
    "for entry in table_of_contents['proceedings']:\n",
    "    print(entry)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ea770af-fded-456b-b929-672c3f6d2ac5",
   "metadata": {},
   "source": [
    "## All of the Current SA Premier's Speeches for the Current Parliament\n",
    "\n",
    "This example is intended to show a more complex data collection: retrieving all of the current premier of South Australia's speeches for the current session of SA Parliament.\n",
    "\n",
    "This example identifies all speeches that the Premier has said something in and downloads them to a separate directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "44e17165-2c12-437b-83dd-9db7422fc742",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Retrieving: lh-20221201-51\n",
      "Retrieving: lh-20230614-6\n",
      "Retrieving: lh-20230706-6\n",
      "Retrieving: lh-20230706-10\n",
      "Retrieving: lh-20230614-8\n",
      "Retrieving: lh-20230614-36\n",
      "Retrieving: lh-20230530-6\n",
      "Retrieving: lh-20230518-6\n",
      "Retrieving: lh-20230503-18\n",
      "Retrieving: lh-20230326-3\n",
      "Retrieving: lh-20230223-37\n",
      "Retrieving: lh-20230208-9\n",
      "Retrieving: lh-20221101-10\n",
      "Retrieving: lh-20221020-9\n",
      "Retrieving: lh-20220928-9\n",
      "Retrieving: lh-20220920-4\n",
      "Retrieving: lh-20220906-3\n",
      "Retrieving: lh-20220706-11\n",
      "Retrieving: lh-20220531-34\n",
      "Retrieving: lh-20220505-3\n",
      "Retrieving: lh-20220505-4\n",
      "Retrieving: lh-20220503-4\n",
      "Retrieving: lh-20230913-46\n",
      "Retrieving: lh-20231017-3\n",
      "Retrieving: lh-20240220-8\n",
      "Retrieving: lh-20240222-9\n",
      "Retrieving: lh-20240409-12\n",
      "Retrieving: lh-20240409-16\n",
      "Retrieving: lh-20240627-10\n",
      "Retrieving: lh-20240627-11\n",
      "Retrieving: lh-20240627-14\n",
      "Retrieving: lh-20241029-31\n",
      "Retrieving: lh-20241126-13\n",
      "Retrieving: lh-20241126-14\n",
      "Retrieving: lh-20241127-52\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import time\n",
    "from datetime import datetime\n",
    "\n",
    "try:\n",
    "    os.mkdir('data')\n",
    "except FileExistsError:\n",
    "    pass\n",
    "\n",
    "# All members and the speeches they were part of for the latest SA government\n",
    "response = requests.get('https://hansardsearch.parliament.sa.gov.au/api/hansard/indicies/lh/55/1/members', params={'contentType': 'text/json'})\n",
    "\n",
    "# Parse the JSON so we can work turn the text into a Python data structure\n",
    "member_index = response.json()\n",
    "all_members = member_index['members']\n",
    "\n",
    "# Go through the members one at a time until we find the current Premier's speeches.\n",
    "for member in all_members:\n",
    "    if member['name'] == 'MALINAUSKAS, Peter Bryden':\n",
    "        speeches = member['speeches']\n",
    "\n",
    "\n",
    "for speech in speeches[0]['topic']:\n",
    "    # Get the information about this speech from this index data structure.\n",
    "    # It is a complex nested structure, so we need to spend some time unpacking it.\n",
    "    speech_locator = speech['date'][0]['fragment'][0]['uid']\n",
    "\n",
    "    print('Retrieving:', speech_locator)\n",
    "\n",
    "    # Pull apart their locator\n",
    "    house, date, index = speech_locator.split('-')\n",
    "    date_in_api_format = datetime.strptime(date, '%Y%m%d').strftime('%Y-%m-%d')\n",
    "\n",
    "    speech_url = 'https://hansardsearch.parliament.sa.gov.au/api/hansard/{}/{}/subject/{}'.format(\n",
    "        house, date_in_api_format, index\n",
    "    )\n",
    "\n",
    "    response = requests.get(speech_url, params={'contentType': 'text/html'})\n",
    "\n",
    "    file_loc = os.path.join('data', f\"{speech_locator}.html\")\n",
    "    with open(file_loc, 'w') as f:\n",
    "        f.write(response.text)\n",
    "\n",
    "    # Wait 2 seconds in between API calls - it's polite to space out your requests.\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "88284a6a-ea34-49f3-b803-4e4798913ac0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob\n",
    "\n",
    "import inscriptis\n",
    "\n",
    "\n",
    "# Identify all the html files in the folder\n",
    "data_files = glob.glob(os.path.join('data', '*.html'))\n",
    "\n",
    "# Extract the text from each of the HTML files as plaintext\n",
    "# using the inscriptis library, then place that text into\n",
    "# a list.\n",
    "entry_text = []\n",
    "\n",
    "for data_file in data_files:\n",
    "    with open(data_file, 'r') as f:\n",
    "        html_content = f.read()\n",
    "        text = inscriptis.get_text(html_content)\n",
    "        entry_text.append(text)\n",
    "\n",
    "# Write out the concatenated text files so we can discuss the bulk of the data.\n",
    "with open('concatenated_speeches.txt', 'w') as w:\n",
    "    for entry in entry_text:\n",
    "        w.write(entry)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3d1a7eff-c455-4d39-bff9-5c209502284e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total tokens: 152018\n"
     ]
    }
   ],
   "source": [
    "import collections\n",
    "# We'll use the NLTK library to break up the text into words.\n",
    "from nltk.tokenize import wordpunct_tokenize\n",
    "\n",
    "tokenised = [wordpunct_tokenize(t.lower()) for t in entry_text]\n",
    "\n",
    "# Now we'll count the number of times a word is used,\n",
    "# and the number of speeches a word is used in.\n",
    "word_counts = collections.Counter()\n",
    "speech_counts = collections.Counter()\n",
    "\n",
    "for tokens in tokenised:\n",
    "    for token in tokens:\n",
    "        word_counts[token] += 1\n",
    "\n",
    "    for token in set(tokens):\n",
    "        speech_counts[token] += 1\n",
    "\n",
    "print('Total tokens:', sum(word_counts.values()))\n",
    "\n",
    "with open('word_counts.csv', 'w') as f:\n",
    "    f.write(\"token,word_count,speech_count\\n\")\n",
    "    for word, word_count in word_counts.most_common(1000):\n",
    "        f.write(f\"{word},{word_count},{speech_counts[word]}\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}