Chapter 6, done!

Author: dellaert

S65_driving_planning.ipynb

@@ -5,7 +5,7 @@
    "id": "UcRF7OziizF1",
    "metadata": {},
    "source": [
-    "# Planning for Autonomous Driving."
+    "# Planning for Autonomous Driving"
    ]
   },
   {
@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "tLBxSLGeWPV0",
    "metadata": {
     "tags": [
@@ -32,12 +32,12 @@
    },
    "outputs": [],
    "source": [
-    "%pip install -q -U gtbook\n"
+    "%pip install -q -U gtbook"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 87,
+   "execution_count": null,
    "id": "ewrl5k4_akQV",
    "metadata": {
     "tags": [
@@ -46,7 +46,7 @@
    },
    "outputs": [],
    "source": [
-    "# no imports (yet)\n"
+    "# no imports (yet)"
    ]
   },
   {
@@ -114,15 +114,13 @@
    "id": "R0sQPSK681cf",
    "metadata": {},
    "source": [
-    "```{index} motion primitives\n",
-    "```\n",
     "## Motion Primitives\n",
     "\n",
     "Consider a car traveling in reverse that wishes to suddenly change its orientation\n",
     "by completing a rapid 180-degree turn (a favorite maneuver for drivers like James Bond and Steve McQueen).\n",
     "How would we go about implementing this type of maneuver in an autonomous vehicle?\n",
     "\n",
-    "This two approaches we have considered before can be very inefficient for planning trajectories that have such well-defined\n",
+    "The two approaches we have considered before can be very inefficient for planning trajectories that have such well-defined\n",
     "characteristics.\n",
     "For all of our probabilistic methods, we used a discrete time formulation and considered\n",
     "the effects of executing an action (e.g., move forward, move left) for a small duration of time, $\\Delta t$.\n",
@@ -132,6 +130,8 @@
     "In each case, the language of path segments is very simple, and in each case,\n",
     "a full plan will consist of many sequential steps.\n",
     "\n",
+    "```{index} motion primitives\n",
+    "```\n",
     "Instead, the U-turn maneuver could be achieved by a predefined\n",
     "sequence of steps: after achieving a reasonable speed, remove your foot from the gas pedal;\n",
     "turn left sharply and hit the breaks; at the perfect moment, release the breaks\n",
@@ -146,8 +146,7 @@
    "id": "53y_6iTD1Ptz",
    "metadata": {},
    "source": [
-    "This idea is illustrated in the figure below, which shows four motion primitives\n",
-    "for a car.\n",
+    "This idea is illustrated in the Figure [1](#fig:MotionPrimitives), which shows four motion primitives for a car.\n",
     "The primitive $P_1$ corresponds to driving forward, while motion primitives $P_2$, $P_3$, and $P_4$ correspond to veering\n",
     "to the left at increasingly sharp angles."
    ]
@@ -157,14 +156,13 @@
    "id": "y_GGvtQc94pI",
    "metadata": {},
    "source": [
-    "```{index} polynomial trajectories, splines\n",
-    "```\n",
-    "\n",
     "<figure id=\"fig:MotionPrimitives\">\n",
     "<img src=\"https://github.com/gtbook/robotics/blob/main/Figures6/motion-primitives.png?raw=1\" style=\"width:18cm\" alt=\"\">\n",
     "<figcaption>Four motion primitives for a car veering to its left. </figcaption>\n",
     "</figure>\n",
     "\n",
+    "```{index} polynomial trajectories, splines\n",
+    "```\n",
     "Motion primitives can be defined in numerous ways.\n",
     "The figure above illustrates four fixed motion primitives, but it would not be difficult to generalize each of these\n",
     "to a class of motions by using parametric descriptions. \n",
@@ -208,7 +206,7 @@
     "For example, the traffic in rural Georgia is irrelevant when leaving downtown Atlanta on\n",
     "a trip to Boston.\n",
     "In this case, immediate driving decisions depend on the car just ahead, and the nearby\n",
-    "cars in adjacent lanes.\n"
+    "cars in adjacent lanes."
    ]
   },
   {
@@ -218,7 +216,7 @@
    "source": [
     "##  Polynomial Trajectories\n",
     "\n",
-    "Let’s begin with the simple problem of changing lanes along a straight stretch of highway. The situation is illustrated in the figure below.\n",
+    "Let’s begin with the simple problem of changing lanes along a straight stretch of highway. The situation is illustrated in Figure [2](#fig:LaneChange).\n",
     "\n",
     "<figure id=\"fig:LaneChange\">\n",
     "<img src=\"https://github.com/gtbook/robotics/blob/main/Figures6/lane-change.png?raw=1\" style=\"width:18cm\" alt=\"\">\n",
@@ -239,7 +237,7 @@
     "\\end{equation}\n",
     "At the start of the maneuver, $s=0$, which matches the initial condition $d(0)=0$, and at $ s = s_\\mathrm{g}$\n",
     "we match the end condition $d(s_\\mathrm{g}) = d_\\mathrm{g}$.\n",
-    "This trajectory is illustrated in the figure below.\n",
+    "This trajectory is illustrated in Figure [3](#fig:LinearLaneChange).\n",
     "\n",
     "<figure id=\"fig:LinearLaneChange\">\n",
     "<img src=\"https://github.com/gtbook/robotics/blob/main/Figures6/linear-lane-change.png?raw=1\" style=\"width:18cm\" alt=\"\">\n",
@@ -305,7 +303,7 @@
     "\\end{aligned}\n",
     "\\end{equation}\n",
     "Note that these six equations are all linear in the parameters $\\alpha_i$, so it is a simple matter to solve\n",
-    "these."
+    "them."
    ]
   },
   {
@@ -320,13 +318,13 @@
     "While the derivation above produced a single polynomial trajectory,\n",
     "it is a simple matter to extend this formalism to construct trajectories\n",
     "that are composed of multiple consecutive polynomial segments.\n",
-    "Such trajectores belong to the more general class of **splines**.\n",
+    "Such trajectories belong to the more general class of **splines**.\n",
     "In general, a spline is a continuous, piecewise polynomial curve, and we are not\n",
     "necessarily given the specific values for the transition points between adjacent\n",
     "segments.\n",
     "\n",
-    "In fact, we have actually done exactly this in the above derviation,\n",
-    "if we consider that for $s < 0$ and for $s  > s_\\mathrm{g}$ the trajectory $d(s)$ is linear and pararallel to the $s$-axis,\n",
+    "In fact, we have actually done exactly this in the above derivation,\n",
+    "if we consider that for $s < 0$ and for $s  > s_\\mathrm{g}$ the trajectory $d(s)$ is linear and parallel to the $s$-axis,\n",
     "i.e., we have solved for a special case of three polynomial segments with two of those\n",
     "segments being linear and one quintic.\n",
     "\n",
@@ -408,11 +406,11 @@
     "trajectory, becomes an important problem. In this section, we address the problem\n",
     "of following such a trajectory.\n",
     "\n",
-    "The figure below illustrates the situation.\n",
-    "We denote by $\\gamma(s)$ the desired trjactory of the car, where $s$, an arc length\n",
+    "Figure [4](#fig:FrenetFrame) illustrates the situation.\n",
+    "We denote by $\\gamma(s)$ the desired trajectory of the car, where $s$, an arc length\n",
     "parameter, is a function of time, and therefore the instantaneous desired speed of \n",
     "the car is $\\dot{s}(t)$.\n",
-    "Since the goal is to keep the car on the deisred trajectory, it is convenient\n",
+    "Since the goal is to keep the car on the desired trajectory, it is convenient\n",
     "to represent the state of the car in a coordinate frame that is local to the trajectory.\n",
     "To do so, for each point along $\\gamma$, we define a frame with origin $\\gamma(s)$,\n",
     "with axes $t_\\gamma(s)$ and $n_\\gamma(s)$, the tangent and normal vectors\n",
@@ -480,15 +478,15 @@
     "want the maneuver to take too long), and the comfort of the human passenger.\n",
     "As mentioned above, humans are sensitive to acceleration changes in the lateral direction,\n",
     "therefore, we might wish to minimize the overall effect of such changes.\n",
-    "Mathematically, the instantaneous change in lateral cceleration is given by the third\n",
+    "Mathematically, the instantaneous change in lateral acceleration is given by the third\n",
     "derivative of $d$, which is known as the **jerk**.\n",
     "For a given trajectory $d(t)$, defined on the interval $[0,T]$, the following\n",
-    "cost functional penalizes aggregate jerk and total exectution time\n",
+    "cost functional penalizes aggregate jerk and total execution time\n",
     "\\begin{equation}\n",
     "J(d) = \\int_0^T \\left(\\frac{d}{dt}\\ddot{d}(\\tau)\\right)^2  d\\tau + \\beta T\n",
     "\\end{equation}\n",
     "In general, it may not be possible to solve this optimization problem in real time.\n",
-    "In such cases, rather than using $J$ to solve find the optimal $d$, we\n",
+    "In such cases, rather than using $J$ to find the optimal $d$, we\n",
     "can use a generate-and-test approach.\n",
     "With such an approach, several values of $T$ are proposed, and the corresponding quintic trajectories\n",
     "are computed for each of these. It is then a simple matter to evaluate the cost of each of\n",

S66_driving_DRL.ipynb

@@ -23,7 +23,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 1,
+   "execution_count": null,
    "id": "pVeijfbAiYRG",
    "metadata": {
     "tags": [
@@ -40,7 +40,7 @@
     }
    ],
    "source": [
-    "%pip install -q -U gtbook\n"
+    "%pip install -q -U gtbook"
    ]
   },
   {
@@ -94,7 +94,6 @@
    "source": [
     "```{index} pair: deep reinforcement learning; DRL\n",
     "```\n",
-    "\n",
     "Deep reinforcement learning (DRL) applies the power of deep learning to bring reinforcement learning to much more complex domains than what we were able to tackle with the Markov Decision Processes and RL concepts introduced in Chapter 3. The use of large, expressive neural networks has allowed researchers and practitioners alike to work with high bandwidth sensors such as video streams and LIDAR, and bring the promise of RL into real-world domains such as autonomous driving. This is still a field of active discovery and research, however, and we can give but a brief introduction here about what is a vast literature and problem space."
    ]
   },
@@ -139,11 +138,10 @@
    "id": "0xi2Y6T5YAtY",
    "metadata": {},
    "source": [
-    "```{index} deep reinforcement learning; DQN\n",
-    "```\n",
-    "\n",
     "## Deep Q-Learning\n",
     "\n",
+    "```{index} deep reinforcement learning; DQN\n",
+    "```\n",
     "> DQN is an early deep learning RL method akin to Q-learning.\n",
     "\n",
     "Recall from Section 3.6 that we can define a policy in terms of **Q-values**, sometimes also called state-action values, and that we can define the optimal policy as \n",
@@ -166,14 +164,16 @@
    "id": "pCKixeLwsh2Z",
    "metadata": {},
    "source": [
-    "```{index} execution phase, experience replay\n",
+    "```{index} pair: deep Q-network; DQN\n",
     "```\n",
     "In the **deep Q-network** or DQN method we use a *supervised learning* approach to Q-learning. We train a neural network, parameterized by $\\theta$, to approximate the optimal Q-values:\n",
     "\\begin{equation}\n",
     "Q^*(x,a) \\approx \\hat{Q}(x,a; \\theta)\n",
     "\\end{equation}\n",
     "It might be worthwhile at this point to re-visit Section 5.6, where we introduced neural networks and how to train them using stochastic gradient descent (SGD).\n",
     "\n",
+    "```{index} execution phase, experience replay\n",
+    "```\n",
     "In the context of RL, the DQN method uses two additional ideas that are crucial in making the training converge to something sensible in difficult problems. The first is splitting the training into *execution* and *experience replay* phases:\n",
     "\n",
     "- during the **execution phase**, the policy is executed (possibly with some degree of randomness) and the experiences $(x,a,r,x')$, with $r$ the reward, are stored in a dataset $D$;\n",
@@ -189,6 +189,8 @@
     "\\mathcal{L}_{\\text{DQN}}(\\theta; D) \\doteq \\sum_{(x,a,r,x')\\in D} [\\text{target}(x,a,x') - Q(x,a; \\theta)]^2\n",
     "\\end{equation}\n",
     "\n",
+    "```{index} off-policy RL\n",
+    "```\n",
     "With this basic scheme, a team from DeepMind was able to achieve human or super-human performance on about 50 Atari 2600 games in 2015 {cite:p}`Mnih15nature_dqn`.\n",
     "DQN is a so-called **off-policy** method, in that each execution phase uses the best policy we computed so far, but we can still replay earlier experiences gathered with \"lesser\" policies. Nothing in the experience replay phase references the policy: every experience leads to a valid Q-value backup and a valid supervised learning signal."
    ]
@@ -198,19 +200,22 @@
    "id": "D6PHabNMU4OO",
    "metadata": {},
    "source": [
-    "```{index} stochastic policy, deep reinforcement learning; policy optimization\n",
-    "```\n",
-    "\n",
     "## Policy Optimization\n",
     "\n",
+    "```{index} deep reinforcement learning; policy optimization\n",
+    "```\n",
     "> Policy optimization takes a black box optimization approach to a deep policy.\n",
     "\n",
+    "```{index} stochastic policy\n",
+    "```\n",
     "Whereas the above gets at an optimal policy indirectly, via deep Q-learning, a different and very popular idea is to directly parameterize the policy using a neural network, with weights $\\theta$. It is common to make this a **stochastic policy**,\n",
     "\\begin{equation}\n",
     "\\pi(a|x; \\theta)\n",
     "\\end{equation}\n",
     "where $a \\in {\\cal A}$ is an action, $x \\in {\\cal X}$ is a state, and the policy outputs a *probability* for each action $a$ based on the state $x$. One of the reasons to prefer stochastic policies is that they are differentiable, as they output continuous values rather than discrete actions. This allows us to optimize for them via gradient descent, as we explore in the next section.\n",
     "\n",
+    "```{index} cross-entropy\n",
+    "```\n",
     "In Chapter 5 we used *supervised* learning to train neural networks, and we just applied this for learning Q-values in DQN. It is useful to consider how this might work for training a *policy*. Recall from Section 5.6 that we defined the empirical cross-entropy loss as\n",
     "\\begin{equation}\n",
     "\\mathcal{L}_{\\text{CE}}(\\theta; D) \\doteq - \\sum_{(x,y=c)\\in D} \\sum_c \\log p_c(x;\\theta)\n",
@@ -231,7 +236,9 @@
    "id": "So1rSw4zS-C5",
    "metadata": {},
    "source": [
-    "In **policy optimization** we gather data by rolling out a set of trajectories $\\tau_i$. In supervised learning we have a dataset $D$ and labels $y_c$, but we have to proceed a bit differently in a reinforcement learning setting. In particular, for *on-policy* RL we gather data by executing our current best guess for the policy for some rollout length or horizon $H$, and we do this many different times, each time obtaining a *trajectory* $\\tau_i$.\n",
+    "```{index} policy optimization, off-policy RL\n",
+    "```\n",
+    "In **policy optimization** we gather data by rolling out a set of trajectories $\\tau_i$. In supervised learning we have a dataset $D$ and labels $y_c$, but we have to proceed a bit differently in a reinforcement learning setting. In particular, for **on-policy** RL we gather data by executing our current best guess for the policy for some rollout length or horizon $H$, and we do this many different times, each time obtaining a *trajectory* $\\tau_i$.\n",
     "That still leaves the training signal: where does that come from? \n",
     "The key idea is to estimate how good a particular action is by estimating the state-action values $Q$ from the rollout rewards.\n",
     "In detail, we estimate the expected discounted reward starting at $x_{it}$, and taking action $a_{it}$, as\n",
@@ -257,9 +264,9 @@
     "- Initialize $\\theta$\n",
     "- Until convergence:\n",
     "    1. roll out a number of trajectories $\\tau_i$ using the current policy $\\pi(a;x,\\theta)$\n",
-    "    2. try and change the parameters $\\theta$ as to decrease the surrogate loss function $\\mathcal{L}(\\theta)$\n",
+    "    2. try to change the parameters $\\theta$ as to decrease the surrogate loss function $\\mathcal{L}(\\theta)$\n",
     "    \n",
-    "A simple, gradient-free approach for step 2 is simple hill-climbing aka stochastic search:\n",
+    "A simple, gradient-free approach for step 2 is simple hill-climbing, aka stochastic search:\n",
     "\n",
     " - perturb $\\theta$ to $\\theta'$\n",
     " - set $\\theta \\leftarrow \\theta'$ *iff* $\\mathcal{L}(\\theta') < \\mathcal{L}(\\theta)$\n",
@@ -273,18 +280,23 @@
    "id": "-sLUpvmQ2sNd",
    "metadata": {},
    "source": [
-    "```{index} deep reinforcement learning; policy gradient methods\n",
-    "```\n",
-    "\n",
     "## Policy Gradient Methods\n",
     "\n",
+    "```{index} deep reinforcement learning; policy gradient methods\n",
+    "```\n",
     "> Policy gradient methods are akin to policy iteration, with a neural flavor.\n",
     "\n",
+    "```{index} softmax function, logit\n",
+    "```\n",
     "In a nutshell, policy gradient methods calculate the *gradient* of the surrogate loss $\\mathcal{L}(\\theta)$ defined above with respect to the policy parameters $\\theta$:\n",
     "\\begin{equation}\n",
     "\\nabla_\\theta \\mathcal{L}(\\theta) \\leftarrow - \\sum_i \\sum_{t=1}^H \\hat{Q}(x_{it},a_{it}) \\nabla_\\theta \\log \\pi(a_{it}|x_{it}, \\theta),\n",
     "\\end{equation}\n",
-    "where $\\nabla_\\theta \\log \\pi(a_{it}|x_{it}, \\theta)$ is the gradient of the logarithm of the stochastic policy. This is easily obtained via back-propagation using any neural network framework of choice. In the case that actions are discrete, as in our example above, a stochastic policy network typically has a \"softmax\" function at the end. Then $\\nabla_\\theta \\log \\pi(a_{it}|x_{it}, \\theta)$ is the derivative of the \"logit\" layer right before the softmax function.\n",
+    "where $\\nabla_\\theta \\log \\pi(a_{it}|x_{it}, \\theta)$ is the gradient of the logarithm of the stochastic policy. This is easily obtained via back-propagation using any neural network framework of choice. In the case that actions are discrete, as in our example above, a stochastic policy network typically ends with a *softmax* function. Recall that if the network outputs a vector of raw scores (the *logits*) $z \\in \\mathbb{R}^K$, the softmax is defined as\n",
+    "\\begin{equation}\n",
+    "\\mathrm{softmax}(z)_i = \\frac{e^{z_i}}{\\sum_{j=1}^K e^{z_j}}, \\quad i = 1,\\dots,K.\n",
+    "\\end{equation}\n",
+    "Thus, the logits are the raw outputs before applying the softmax, and $\\nabla_\\theta \\log \\pi(a_{it}|x_{it}, \\theta)$ is computed with respect to *these* values.\n",
     "We then use gradient descent to update the policy parameters:\n",
     "\\begin{equation}\n",
     "\\theta \\leftarrow \\theta - \\alpha \\nabla_\\theta \\mathcal{L}(\\theta)\n",
@@ -294,12 +306,6 @@
     "The algorithm above, using the estimated Q-values, is almost identical to the REINFORCE method {cite:p}`Williams92ml_reinforce`. That algorithm further improves on performance by not using the raw Q-values but rather the difference between the Q-values and some baseline policy. This has the effect of reducing the variance in the estimated Q-values due to using only a finite amount of data.\n",
     "The REINFORCE algorithm was introduced in 1992 and hence pre-dates the deep-learning revolution by about 20 years. It should also be said that in DRL, the neural networks that are used are typically not very deep. Several modern methods, such as \"proximal policy optimization\" (PPO) {cite:p}`Schulman17_PPO` apply a number of techniques to improve this basic method even further and make it more sample-efficient. PPO is now one of the most often-used DRL methods."
    ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "xtNoiDaqfViL",
-   "metadata": {},
-   "source": []
   }
  ],
  "metadata": {