openai_gym_taxi_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/xhxuciedu/CS175/blob/master/openai_gym_taxi_tutorial.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A8lDQuQEnS8_"
      },
      "source": [
        "# Tutorial: An Introduction to Reinforcement Learning Using OpenAI Gym's Taxi 🚕"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ziMM6qgmncE5"
      },
      "source": [
        "In this introductory tutorial, we'll apply reinforcement learning (RL) to train an agent to solve the ['Taxi' environment from OpenAI Gym](https://gym.openai.com/envs/Taxi-v3/). \n",
        "\n",
        "We'll cover:\n",
        "\n",
        "-   A basic introduction to RL\n",
        "-   Setting up OpenAI Gym & Taxi\n",
        "-   Using the Q-learning algorithm to train our Taxi agent"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HfYSvJF6ocqk"
      },
      "source": [
        "# Before we start, what's 'Taxi'?\n",
        "\n",
        "Taxi is one of many environments available on OpenAI Gym. These environments are used to develop and benchmark reinforcement learning algorithms.\n",
        "\n",
        "The goal of Taxi is to pick-up passengers and drop them off at the destination in the least amount of moves. In this tutorial, you'll start with a taxi agent that takes actions randomly:\n",
        "\n",
        "![random agent](https://drive.google.com/uc?id=1l0XizDh9eGP3gVNCjJHrC0M3DeCWI8Fj)\n",
        "\n",
        "…and successfully apply reinforcement learning to solve the environment:\n",
        "\n",
        "![trained agent](https://drive.google.com/uc?id=1a-OeLhXi3W-kvQuhGRyJ1dOSw4vrIBxr)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aZUF3oE-o889"
      },
      "source": [
        "# 💡 An introduction to Reinforcement Learning\n",
        "\n",
        "Think about how you might teach a dog a new trick, like telling it to sit:\n",
        "\n",
        "-   If it performs the trick correctly (it sits), you'll reward it with a treat (positive feedback) ✔️\n",
        "-   If it doesn't sit correctly, it doesn't get a treat (negative feedback) ❌\n",
        "\n",
        "By continuing to do things that lead to positive outcomes, the dog will learn to sit when it hears the command in order to get its treat. Reinforcement learning is a subdomain of machine learning which involves training an 'agent' (the dog) to learn the correct sequences of actions to take (sitting) on its environment (in response to the command 'sit') in order to maximise its reward (getting a treat). This can be illustrated more formally as:\n",
        "\n",
        "![sutton barto rl](https://www.gocoder.one/static/RL-diagram-b3654cd3d5cc0e07a61a214977038f01.png \"Reinforcement Learning diagram\")  \n",
        "Source: [Sutton & Barto](http://incompleteideas.net/book/bookdraft2017nov5.pdf)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2aWkaLM_o2pH"
      },
      "source": [
        "# 🏋️ Installing OpenAI Gym and Taxi\n",
        "\n",
        "We'll be using the 'Taxi-v3' environment for this tutorial. To install gym (and numpy for later), run the cell below:\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cyjebRSHnK1F",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "df32fc8c-32e0-4a25-bb3e-27a2eb7d24f5"
      },
      "source": [
        "!pip install gym\n",
        "!pip install numpy"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: gym in /usr/local/lib/python3.8/dist-packages (0.25.2)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from gym) (2.2.1)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.8/dist-packages (from gym) (1.21.6)\n",
            "Requirement already satisfied: importlib-metadata>=4.8.0 in /usr/local/lib/python3.8/dist-packages (from gym) (6.0.0)\n",
            "Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.8/dist-packages (from gym) (0.0.8)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.8.0->gym) (3.12.1)\n",
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.8/dist-packages (1.21.6)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dAJK0BtorfEO"
      },
      "source": [
        "Next, import gym (and additional libraries that will be useful for later):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "m7As7qh4navx"
      },
      "source": [
        "import gym                                          \n",
        "import numpy as np                                  \n",
        "import random\n",
        "                   \n",
        "# used to help with visualizing in Colab                 \n",
        "from IPython.display import display, clear_output   \n",
        "from time import sleep                              "
      ],
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TyMhgh3RsLWk"
      },
      "source": [
        "Gym contains a large library of different environments. Let's create the Taxi-v3 environment:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VpYHWA95y_QJ",
        "outputId": "19af1186-7c0f-42c3-b9c2-220c46f0ef72",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# create Taxi environment\n",
        "env = gym.make('Taxi-v3')"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.8/dist-packages/gym/core.py:317: DeprecationWarning: \u001b[33mWARN: Initializing wrapper in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n",
            "/usr/local/lib/python3.8/dist-packages/gym/wrappers/step_api_compatibility.py:39: DeprecationWarning: \u001b[33mWARN: Initializing environment in old step API which returns one bool instead of two. It is recommended to set `new_step_api=True` to use new step API. This will be the default behaviour in future.\u001b[0m\n",
            "  deprecation(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AWMjVABSsnWT"
      },
      "source": [
        "# 🎲 Random Agent\n",
        "\n",
        "We'll start by implementing an agent that doesn't learn at all. Instead, it will select actions at random. This will be our baseline.\n",
        "\n",
        "The first step is to give our agent the initial observation of the state. A state tells our agent what the current environment looks like. In Taxi, a state defines the current positions of the taxi, passenger, and pick-up and drop-off locations. Below are examples of three different states for Taxi:\n",
        "\n",
        "![taxi states](https://www.gocoder.one/static/taxi-states-0aad1b011cf3fe07b571712f2123335c.png \"Different Taxi states\")\n",
        "\n",
        "Note: Yellow = taxi, Blue letter = pickup location, Purple letter = drop-off destination\n",
        "\n",
        "To get the initial state:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mNlV-YvdnlOH",
        "outputId": "a1f55463-80b7-4940-9d66-461f88b69fc7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "source": [
        "# create a new instance of taxi, and get the initial state\n",
        "state = env.reset()\n",
        "\n",
        "print(f\"Initial state: {state}\")"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Initial state: 228\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WM7GRNHnvRaH"
      },
      "source": [
        "Next, we'll run a for-loop to cycle through the game. At each iteration, our agent will:\n",
        "\n",
        "1. Make a random action from the action space (0 - south, 1 - north, 2 - east, 3 - west, 4 - pick-up, 5 - drop-off)\n",
        "2. Receive the new state\n",
        "\n",
        "Here's our random agent:"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install gym[toy_text]"
      ],
      "metadata": {
        "id": "tfh7y2NRQDMe",
        "outputId": "c2ea6fca-2102-493f-9c34-ad68bc37f536",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Requirement already satisfied: gym[toy_text] in /usr/local/lib/python3.8/dist-packages (0.25.2)\n",
            "Requirement already satisfied: cloudpickle>=1.2.0 in /usr/local/lib/python3.8/dist-packages (from gym[toy_text]) (2.2.1)\n",
            "Requirement already satisfied: importlib-metadata>=4.8.0 in /usr/local/lib/python3.8/dist-packages (from gym[toy_text]) (6.0.0)\n",
            "Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.8/dist-packages (from gym[toy_text]) (0.0.8)\n",
            "Requirement already satisfied: numpy>=1.18.0 in /usr/local/lib/python3.8/dist-packages (from gym[toy_text]) (1.21.6)\n",
            "Collecting pygame==2.1.0\n",
            "  Downloading pygame-2.1.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.3/18.3 MB\u001b[0m \u001b[31m66.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.8/dist-packages (from importlib-metadata>=4.8.0->gym[toy_text]) (3.12.1)\n",
            "Installing collected packages: pygame\n",
            "Successfully installed pygame-2.1.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "os.environ[\"SDL_VIDEODRIVER\"] = \"dummy\"\n",
        "#import os\n",
        "#os.environ['SDL_VIDEODRIVER']='windib'"
      ],
      "metadata": {
        "id": "suHc8N1RQjV4"
      },
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# import pygame\n",
        "# pygame.init()\n",
        "# pygame.display.list_modes()"
      ],
      "metadata": {
        "id": "hp-bNfkFQ9Z0",
        "outputId": "54868e64-f97d-448d-9c56-52ece4100b40",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 201
        }
      },
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "error",
          "ename": "error",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31merror\u001b[0m                                     Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-12-30975b609c9a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mpygame\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mpygame\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdisplay\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlist_modes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31merror\u001b[0m: video system not initialized"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aycxOXzQnoLU",
        "outputId": "213ec613-58a4-4078-d042-4bae051b203c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 398
        }
      },
      "source": [
        "num_steps = 99\n",
        "for s in range(num_steps+1):\n",
        "\n",
        "    clear_output(wait=True) \n",
        "\n",
        "    print(f\"step: {s} out of {num_steps}\")\n",
        "\n",
        "    # sample a random action from the list of available actions\n",
        "    action = env.action_space.sample()\n",
        "\n",
        "    # perform this action on the environment\n",
        "    env.step(action)\n",
        "\n",
        "    # print the new state\n",
        "    env.render()\n",
        "\n",
        "    sleep(0.2)\n",
        "\n",
        "# end this instance of the taxi environment\n",
        "env.close()"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "step: 0 out of 99\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "error",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31merror\u001b[0m                                     Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-16-0434df4b21eb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m     \u001b[0;31m# print the new state\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m     \u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m     \u001b[0msleep\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0.2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m             )\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m     ) -> Optional[Union[RenderFrame, List[RenderFrame]]]:\n\u001b[1;32m    420\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 421\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    423\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m             )\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/wrappers/order_enforcing.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     49\u001b[0m                 \u001b[0;34m\"set `disable_render_order_enforcing=True` on the OrderEnforcer wrapper.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     50\u001b[0m             )\n\u001b[0;32m---> 51\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     52\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mproperty\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m             )\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m     ) -> Optional[Union[RenderFrame, List[RenderFrame]]]:\n\u001b[1;32m    420\u001b[0m         \u001b[0;34m\"\"\"Renders the environment.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 421\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    423\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m             )\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/wrappers/env_checker.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     53\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0menv_render_passive_checker\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     54\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 55\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0menv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrender\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/core.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m             )\n\u001b[1;32m     57\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mrender_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mrender\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/envs/toy_text/taxi.py\u001b[0m in \u001b[0;36mrender\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    285\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrenderer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_renders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    286\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 287\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    288\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    289\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_render\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/envs/toy_text/taxi.py\u001b[0m in \u001b[0;36m_render\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    292\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_render_text\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    293\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m\"human\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"rgb_array\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"single_rgb_array\"\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 294\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_render_gui\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    295\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    296\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_render_gui\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.8/dist-packages/gym/envs/toy_text/taxi.py\u001b[0m in \u001b[0;36m_render_gui\u001b[0;34m(self, mode)\u001b[0m\n\u001b[1;32m    368\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mx\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdesc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    369\u001b[0m                 \u001b[0mcell\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mx\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcell_size\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcell_size\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 370\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackground_img\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    371\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mdesc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34mb\"|\"\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mdesc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0my\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mx\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34mb\"|\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    372\u001b[0m                     \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwindow\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mblit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmedian_vert\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31merror\u001b[0m: display Surface quit"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8I-Whw1WxDra"
      },
      "source": [
        "When you run the cell above, you should see your agent making random moves. Not super exciting, but hopefully this helped you get familiar with the OpenAI Gym toolkit.\n",
        "\n",
        "Next, we'll implement the Q-learning algorithm that will enable our agent to learn from rewards."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xcmS8OwLyDL5"
      },
      "source": [
        "# 📖 Q-Learning Agent\n",
        "\n",
        "Q-learning is a reinforcement learning algorithm that seeks to find the best possible next action given its current state, in order to maximise the reward it receives (the 'Q' in Q-learning stands for quality - i.e. how valuable an action is).\n",
        "\n",
        "Let's take the following starting state:\n",
        "\n",
        "![taxi state](https://www.gocoder.one/static/start-state-6a115a72f07cea072c28503d3abf9819.png \"An example Taxi state\")\n",
        "\n",
        "Which action (up, down, left, right, pick-up or drop-off) should it take in order to maximise its reward? (_Note: blue = pick-up location and purple= drop-off destination_)\n",
        "\n",
        "First, let's take a look at how our agent is 'rewarded' for its actions. **Remember in reinforcement learning, we want our agent to take actions that will maximise the possible rewards it receives from its environment.**\n",
        "\n",
        "## 'Taxi' reward system\n",
        "\n",
        "According to the [Taxi documentation](https://gym.openai.com/envs/Taxi-v3/):\n",
        "\n",
        "> _\"…you receive +20 points for a successful drop-off, and lose 1 point for every timestep it takes. There is also a 10 point penalty for illegal pick-up and drop-off actions.\"_\n",
        "\n",
        "Looking back at our original state, the possible actions it can take and the corresponding rewards it will receive are shown below:\n",
        "\n",
        "![taxi rewards](https://www.gocoder.one/static/state-rewards-62ab43a53e07062b531b3199a8bab5b3.png \"Taxi rewards\")\n",
        "\n",
        "In the image above, the agent loses 1 point per timestep it takes. It will also lose 10 points if it uses the pick-up or drop-off action here.\n",
        "\n",
        "We want our agent to go North towards the pick-up location denoted by a blue R- **but how will it know which action to take if they are all equally punishing?**\n",
        "\n",
        "## Exploration\n",
        "\n",
        "Our agent currently has no way of knowing which action will lead it closest to the blue R. This is where trial-and-error comes in - we'll have our agent take random actions, and observe what rewards it gets (i.e. our agent will **explore**).\n",
        "\n",
        "Over many iterations, our agent will have observed that certain sequences of actions will be more rewarding than others. Along the way, our agent will need to keep track of which actions led to what rewards.\n",
        "\n",
        "## Introducing… Q-tables\n",
        "\n",
        "A Q-table is simply a look-up table storing values representing the maximum expected future rewards our agent can expect for a certain action in a certain state (_known as Q-values_). It will tell our agent that when it encounters a certain state, some actions are more likely than others to lead to higher rewards. It becomes a 'cheatsheet' telling our agent what the best action to take is.\n",
        "\n",
        "The image below illustrates what our 'Q-table' will look like:\n",
        "\n",
        "-   Each row corresponds to a unique state in the 'Taxi' environment\n",
        "-   Each column corresponds to an action our agent can take\n",
        "-   Each cell corresponds to the Q-value for that state-action pair - a higher Q-value means a higher maximum reward our agent can expect to get if it takes that action in that state.\n",
        "\n",
        "![Q table](https://www.gocoder.one/static/q-table-9461cc903f50b78d757ea30aeb3eb8bc.png \"Q table\")\n",
        "\n",
        "Before we begin training our agent, we'll need to initialize our Q-table as so:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BVvgYgquUKBg"
      },
      "source": [
        "state_size = env.observation_space.n  # total number of states (S)\n",
        "action_size = env.action_space.n      # total number of actions (A)\n",
        "\n",
        "# initialize a qtable with 0's for all Q-values\n",
        "qtable = np.zeros((state_size, action_size))\n",
        "\n",
        "print(f\"Q table: {qtable}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F0uvTrFXmJk7"
      },
      "source": [
        "As our agent explores, it will update the Q-table with the Q-values it finds. To calculate our Q-values, we'll introduce the Q-learning algorithm."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mdpsBFdJm9ve"
      },
      "source": [
        "# Q-Learning Algorithm\n",
        "\n",
        "The Q-learning algorithm is given below. We won't go into details, but you can read more about it in [Ch 6 of Sutton & Barto (2018)](http://www.incompleteideas.net/book/RLbook2018trimmed.pdf).\n",
        "\n",
        "![Q learning algorithm](https://www.gocoder.one/static/q-learning-algorithm-84b84bb5dc16ba8097e31aff7ea42748.png \"The Q learning algorithm\")\n",
        "\n",
        "The Q-learning algorithm will help our agent **update the current Q-value ($Q(S_t,A_t)$) with its observations after taking an action.** I.e. increase Q if it encountered a positive reward, or decrease Q if it encountered a negative one.\n",
        "\n",
        "Note that in Taxi, our agent doesn't receive a positive reward until it successfully drops off a passenger (_+20 points_). Hence even if our agent is heading in the correct direction, there will be a delay in the positive reward it should receive. The following term in the Q-learning equation addresses this:\n",
        "\n",
        "![maximum q](https://www.gocoder.one/static/max-q-e593ddcec76cda87ed189c31d60837b6.png \"Max Q value\")\n",
        "\n",
        "This term adjusts our current Q-value to include a portion of the rewards it may receive sometime in the future (St+1). The 'a' term refers to all the possible actions available for that state. The equation also contains two hyperparameters which we can specify:\n",
        "\n",
        "1. Learning rate (α): how easily the agent should accept new information over previously learnt information\n",
        "2. Discount factor (γ): how much the agent should take into consideration the rewards it could receive in the future versus its immediate reward\n",
        "\n",
        "Here's our implementation of the Q-learning algorithm:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XsOWYTX4VsDz"
      },
      "source": [
        "# hyperparameters to tune\n",
        "learning_rate = 0.9\n",
        "discount_rate = 0.8\n",
        "\n",
        "# dummy variables\n",
        "reward = 10 # R_(t+1)\n",
        "state = env.observation_space.sample()      # S_t\n",
        "action = env.action_space.sample()          # A_t\n",
        "new_state = env.observation_space.sample()  # S_(t+1)\n",
        "\n",
        "# Qlearning algorithm: Q(s,a) := Q(s,a) + learning_rate * (reward + discount_rate * max Q(s',a') - Q(s,a))\n",
        "qtable[state, action] += learning_rate * (reward + discount_rate * np.max(qtable[new_state,:]) - qtable[state,action])\n",
        "\n",
        "print(f\"Q-value for (state, action) pair ({state}, {action}): {qtable[state,action]}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bx5IMA1idW9X"
      },
      "source": [
        "## Exploration vs Exploitation Trade-off\n",
        "\n",
        "We can let our agent explore to update our Q-table using the Q-learning algorithm. As our agent learns more about the environment, we can let it use this knowledge to take more optimal actions and converge faster - known as **exploitation**.\n",
        "\n",
        "During exploitation, our agent will look at its Q-table and select the action with the highest Q-value (instead of a random action). Over time, our agent will need to explore less, and start exploiting what it knows instead.\n",
        "\n",
        "Here's our implementation of an exploration-exploitation strategy:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3aorBEvYdSLr"
      },
      "source": [
        "# dummy variables\n",
        "episode = random.randint(0,500)\n",
        "qtable = np.random.randn(env.observation_space.sample(), env.action_space.sample())\n",
        "\n",
        "# hyperparameters\n",
        "epsilon = 1.0     # probability that our agent will explore\n",
        "decay_rate = 0.01 # of epsilon\n",
        "\n",
        "if random.uniform(0,1) < epsilon:\n",
        "    # explore\n",
        "    action = env.action_space.sample()\n",
        "else:\n",
        "    # exploit\n",
        "    action = np.argmax(qtable[state,:])\n",
        "\n",
        "# epsilon decreases exponentially --> our agent will explore less and less\n",
        "epsilon = np.exp(-decay_rate*episode)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "G_zL2Vrd1yLv"
      },
      "source": [
        "In the example above, we set some value `epsilon` between 0 and 1. If `epsilon` is 0.7, there is a 70% chance that on this step our agent will explore instead of exploit. `epsilon` exponentially decays with each step, so that our agent explores less and less over time."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s5mA3SflarKs"
      },
      "source": [
        "# Bringing it all together\n",
        "\n",
        "We're done with all the building blocks needed for our reinforcement learning agent. The process for training our agent will look like:\n",
        "\n",
        "1. Initialising our Q-table with 0's for all Q-values\n",
        "2. Let our agent play Taxi over a large number of games\n",
        "3. Continuously update the Q-table using the Q-learning algorithm and an exploration-exploitation strategy\n",
        "\n",
        "Here's the full implementation:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SiXAK2OdpteR"
      },
      "source": [
        "class bcolors:\n",
        "    RED= '\\u001b[31m'\n",
        "    GREEN= '\\u001b[32m'\n",
        "    RESET= '\\u001b[0m'\n",
        "\n",
        "# create Taxi environment\n",
        "env = gym.make('Taxi-v3')\n",
        "\n",
        "# initialize q-table\n",
        "state_size = env.observation_space.n\n",
        "action_size = env.action_space.n\n",
        "qtable = np.zeros((state_size, action_size))\n",
        "\n",
        "# hyperparameters\n",
        "learning_rate = 0.9\n",
        "discount_rate = 0.8\n",
        "epsilon = 1.0\n",
        "decay_rate= 0.005\n",
        "\n",
        "# training variables\n",
        "num_episodes = 2000\n",
        "max_steps = 99 # per episode\n",
        "\n",
        "print(\"AGENT IS TRAINING...\")\n",
        "\n",
        "for episode in range(num_episodes):\n",
        "\n",
        "\t# Reset the environment\n",
        "\tstate = env.reset()\n",
        "\tstep = 0\n",
        "\tdone = False\n",
        "    \n",
        "\tfor step in range(max_steps):\n",
        "\n",
        "\t\t# Exploration-exploitation tradeoff\n",
        "\t\tif random.uniform(0,1) < epsilon:\n",
        "\t\t\t# Explore\n",
        "\t\t\taction = env.action_space.sample()\n",
        "\t\telse:\n",
        "\t\t\t# Exploit\n",
        "\t\t\taction = np.argmax(qtable[state,:])\n",
        "\n",
        "\t\t# Take an action and observe the reward\n",
        "\t\tnew_state, reward, done, info = env.step(action)\n",
        "\n",
        "\t\t# Q-learning algorithm\n",
        "\t\tqtable[state,action] = qtable[state,action] + learning_rate * (reward + discount_rate * np.max(qtable[new_state,:])-qtable[state,action])\n",
        "\n",
        "\t\t# Update to our new state\n",
        "\t\tstate = new_state\n",
        "\n",
        "\t\t# if done, finish episode\n",
        "\t\tif done == True:\n",
        "\t\t\tbreak\n",
        "\n",
        "\t# Decrease epsilon\n",
        "\tepsilon = np.exp(-decay_rate*episode)\n",
        "\t\n",
        "# Get ready to watch our trained agent\n",
        "clear_output()\n",
        "print(f\"Our Q-table: {qtable}\")\n",
        "print(f\"Training completed over {num_episodes} episodes\")\n",
        "input(\"Press Enter to see our trained taxi agent\")\n",
        "sleep(1)\n",
        "clear_output()  \n",
        "\n",
        "episodes_to_preview = 3\n",
        "for episode in range(episodes_to_preview):\n",
        "\n",
        "\t# Reset the environment\n",
        "\tstate = env.reset()\n",
        "\tstep = 0\n",
        "\tdone = False\n",
        "\tepisode_rewards = 0\n",
        "\n",
        "\tfor step in range(num_steps):\n",
        "\t\t# clear screen\n",
        "\t\tclear_output(wait=True)\n",
        "\n",
        "\t\tprint(f\"TRAINED AGENT\")\n",
        "\t\tprint(f\"+++++EPISODE {episode+1}+++++\")\n",
        "\t\tprint(f\"Step {step+1}\")\n",
        "\n",
        "\t\t# Exploit\n",
        "\t\taction = np.argmax(qtable[state,:])\n",
        "\n",
        "\t\t# Take an action and observe the reward\n",
        "\t\tnew_state, reward, done, info = env.step(action)\n",
        "\n",
        "\t\t# Accumulate our rewards    \n",
        "\t\tepisode_rewards += reward\n",
        "\n",
        "\t\tenv.render()\n",
        "\t\tprint(\"\")\n",
        "\t\tif episode_rewards < 0:\n",
        "\t\t\tprint(f\"Score: {bcolors.RED}{episode_rewards}{bcolors.RESET}\")\n",
        "\t\telse:\n",
        "\t\t\tprint(f\"Score: {bcolors.GREEN}{episode_rewards}{bcolors.RESET}\")\n",
        "\t\tsleep(0.5)   \n",
        "\t\t    \n",
        "\t\t# Update to our new state\n",
        "\t\tstate = new_state\n",
        "\n",
        "\t\t# if done, finish episode\n",
        "\t\tif done == True:\n",
        "\t\t\tbreak  \n",
        "\n",
        "# Close the Taxi environment\n",
        "env.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m_tmtN9koho8"
      },
      "source": [
        "# 👏 What's next?\n",
        "\n",
        "There are many other environments available on OpenAI Gym for you to try (e.g. [Frozen Lake](https://gym.openai.com/envs/FrozenLake-v0/)). You can also try optimising the implementation above to solve Taxi in fewer episodes.\n",
        "\n",
        "Some other useful resources include:\n",
        "-   [DeepMind x UCL Reinforcement Learning Lecture Series [2021]](https://deepmind.com/learning-resources/reinforcement-learning-series-2021)\n",
        "-   [A (Long) Peek into Reinforcement Learning](https://lilianweng.github.io/lil-log/2018/02/19/a-long-peek-into-reinforcement-learning.html) by Lilian Weng\n",
        "-   [A good article on RL and its real-world applications](https://www.altexsoft.com/blog/datascience/reinforcement-learning-explained-overview-comparisons-and-applications-in-business/)\n",
        "-   [AlphaGo full documentary](https://www.youtube.com/watch?v=WXuK6gekU1Y) (on Youtube)\n",
        "-   [Reinforcement Learning](http://www.incompleteideas.net/book/RLbook2018trimmed.pdf) by Sutton and Barto\n",
        "-   [Hands-on introduction to deep reinforcement learning](https://www.gocoder.one/blog/hands-on-introduction-to-deep-reinforcement-learning)"
      ]
    }
  ]
}