Gaussians.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to TensorFlow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "View this tutorial at: [https://nbviewer.jupyter.org/github/berkeley-cocosci/tensorflow-tutorial/blob/master/Gaussians.ipynb](https://nbviewer.jupyter.org/github/berkeley-cocosci/tensorflow-tutorial/blob/master/Gaussians.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll walk through the basics of TensorFlow and go through a simple example of fitting a Gaussian distribution to some data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I would recommend installing Tensorflow inside a conda environment. If you don't already have conda or Anaconda installed, you can install it quickly using `miniconda`: http://conda.pydata.org/miniconda.html\n",
    "\n",
    "After you have conda installed, run the following commands:\n",
    "\n",
    "```bash\n",
    "# create and activate the conda environment\n",
    "conda create -n tensorflow python=3 numpy scipy matplotlib ipython\n",
    "source activate tensorflow\n",
    "\n",
    "# install the IPython kernel for the Jupyter notebook\n",
    "ipython kernel install --user --name=tensorflow --display-name=tensorflow\n",
    "\n",
    "# install tensorflow\n",
    "conda install -c conda-forge tensorflow\n",
    "```\n",
    "\n",
    "You should now be able to open this notebook in Jupyter. Make sure you are running the right kernel by selecting \"Kernel > Change kernel > tensorflow\" from the menubar.\n",
    "\n",
    "Note that this will not include GPU support, but you don't need that anyway unless you're running a machine that has a good GPU. For our lab, that will mean `theano`, though it doesn't have a GPU yet."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TensorFlow basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we'll start off by defining some of the basic terminology surrounding TensorFlow:\n",
    "\n",
    "**Tensors** -- these are like arrays in NumPy, but they are internal to the TensorFlow graph.\n",
    "\n",
    "**Ops** -- these are operations that take tensors, manipulate them some how, and produce a new tensor. For example, the `mul` op would take two tensors and multiply them together, producing a third tensor.\n",
    "\n",
    "**Variables** -- variables are a special type of tensor that have values that are independent of the rest of the graph (in other words, they are not the product of an operation). Variables are what we care about in terms of being able to learn parameters -- the parameters that we want to learn are stored in variables, and we compute gradients with respect to variables.\n",
    "\n",
    "**Graph** -- the graph is what defines the set of computations that will occur: which ops operate over which tensors and variables, which tensors are produced, etc. We we create a tensor or use an operation, those pieces are automatically added to the graph definition; however, *they are not computed, they are only defined*. You can think of the graph like a file containing code: the code is only the definition of the computation that will be performed.\n",
    "\n",
    "\n",
    "**Session** -- the session is what actually performs the computation. You can think of it like a programming language interpreter (e.g. the Python interpreter). You tell the session to run operations by calling `session.run()`. Like an interpreter, the session maintains state, so if the values of any variables change, they will persist across different calls to `session.run()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructing the graph"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import tfutils\n",
    "import scipy.stats\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For our first example, we'll construct a TensorFlow graph that computes the log probability of some data under a Gaussian distribution. To start, we can construct a custom op that computes the Gaussian log probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def gaussian_logpdf(X, mu, sigma, name=None):\n",
    "    \"\"\"Construct a tensorflow op for a Gaussian probability.\"\"\"\n",
    "    with tf.op_scope([X, mu, sigma], name, \"gaussian_logpdf\"):\n",
    "        Z = -0.5 * tf.log(2 * np.pi * sigma ** 2)\n",
    "        e = -(X - mu) ** 2 / (2 * sigma ** 2)\n",
    "        logpdf = Z + e\n",
    "        return logpdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need some initial starting values for our estimate of those parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial: mu=0.6965, sigma=0.2861\n"
     ]
    }
   ],
   "source": [
    "np.random.seed(123)\n",
    "initial_mu = np.random.rand()\n",
    "initial_sigma = np.random.rand()\n",
    "print(\"Initial: mu={:.4f}, sigma={:.4f}\".format(initial_mu, initial_sigma))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to start constructing the TensorFlow graph! We are going to create (1) a placeholder tensor for the data, (2) variables to hold the parameters mu and sigma, (3) the op to compute the log probability of the data given the parameters, and (4) a special TensorFlow operation to initialize the values of the variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# create a placeholder for the input to our Gaussian distribution. a\n",
    "# placeholder is a special type of Tensor that is only given a value\n",
    "# at runtime by the user (i.e., it is not the product of an op).\n",
    "X = tf.placeholder(tf.float32, shape=(None, 1), name=\"X\")\n",
    "\n",
    "# create trainable variables for the parameters of the Gaussian distribution\n",
    "mu = tf.Variable(initial_value=initial_mu, name=\"mu\")\n",
    "sigma = tf.Variable(initial_value=initial_sigma, name=\"sigma\")\n",
    "\n",
    "# compute the log probability of X given mu and sigma\n",
    "logp = tf.reduce_sum(gaussian_logpdf(X, mu, sigma), name=\"logp\")\n",
    "\n",
    "# when we run this op, it will actually assign initial values to all the variables\n",
    "init = tf.initialize_all_variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TensorFlow comes with some really nice visualization capabilites. Here, we can interactively peek at what the graph looks like. Notice that the graph includes variables for `mu` and `sigma`, and that those variables go into the op for `gaussian_logpdf`, which also takes the input tensor of `X`. Try double clicking on `gaussian_logpdf` and you'll be able to see the flow of operations taking place within that scope!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.2268514535642031&quot;).pbtxt = 'node {\\n  name: &quot;X&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.6964691877365112\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;mu&quot;\\n  input: &quot;mu/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;mu&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.2861393392086029\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;sigma&quot;\\n  input: &quot;sigma/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;sigma&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 6.2831854820251465\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;gaussian_logpdf/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: -0.5\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;X&quot;\\n  input: &quot;mu/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/truediv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;gaussian_logpdf/mul_1&quot;\\n  input: &quot;gaussian_logpdf/truediv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Rank&quot;\\n  op: &quot;Rank&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/start&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/delta&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range&quot;\\n  op: &quot;Range&quot;\\n  input: &quot;range/start&quot;\\n  input: &quot;Rank&quot;\\n  input: &quot;range/delta&quot;\\n}\\nnode {\\n  name: &quot;logp&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  input: &quot;range&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;init&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^mu/Assign&quot;\\n  input: &quot;^sigma/Assign&quot;\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.2268514535642031&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfutils.show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Doing a forward pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now that we've defined our graph, let's actually do some computation with it. Remember that we have so far only defined a placeholder variable for our data -- so we need to externally generate some data first:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True:    mu=6.9647, sigma=2.8614\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[ 11.9388723 ],\n",
       "       [  8.12772775],\n",
       "       [  7.8863743 ],\n",
       "       [  6.81728719],\n",
       "       [  6.38039936],\n",
       "       [ 12.62840761],\n",
       "       [  2.33123479],\n",
       "       [  3.77720222],\n",
       "       [  5.68439312],\n",
       "       [ 11.73866437]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "true_mu = 6.9647\n",
    "true_sigma = 2.8614\n",
    "print(\"True:    mu={:.4f}, sigma={:.4f}\".format(true_mu, true_sigma))\n",
    "\n",
    "def sample_X():\n",
    "    X_vals = np.random.normal(true_mu, true_sigma, (1000, 1))\n",
    "    return X_vals\n",
    "\n",
    "X_vals = sample_X()\n",
    "X_vals[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To evaluate the log probablity of these values, we create a `Session` and then call the `run()` method. Note that we pass in to the `feed_dict` keyword argument a dictionary where the keys are the actual tensors, and the values are the values we want to use for those tensors:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-275808.19"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    logp_val = sess.run(logp, feed_dict={X: X_vals})\n",
    "\n",
    "logp_val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can verify that we're getting the correct values from TensorFlow by comparing them to what Scipy says the log probability is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-275808.22153962194"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scipy.stats.norm.logpdf(X_vals, initial_mu, initial_sigma).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the really awesome things about Tensorflow is that it knows how to compute the gradient for (almost) every one of its ops. Thus, if we want to compute the gradient of some quantity with respect to variables in the graph, all we have to do is create a single op called `gradients`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<tf.Tensor 'gradients/gaussian_logpdf/sub_grad/Reshape_1:0' shape=() dtype=float32>,\n",
       " <tf.Tensor 'gradients/AddN:0' shape=() dtype=float32>]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compute gradients of the negative log probability with respect to the parameters.\n",
    "# note that we are using negative log probability here since when we do optimization\n",
    "# it will be doing minimization, rather than maximization\n",
    "grads = tf.gradients(-logp, [mu, sigma])\n",
    "grads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, when we inspect the graph, you'll notice that there is a new `gradients` op. If you open it up, you'll see what is almost a mirror image of our original graph, but where all the ops have names ending in `_grad`. Importantly, a key thing to take away here is that *computing gradients is just another operation*! There is absolutely nothing special about it; it is just that TensorFlow makes it incredibly easy to do because it knows about the derivatives of all its ops:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.7278572444454114&quot;).pbtxt = 'node {\\n  name: &quot;X&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.6964691877365112\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;mu&quot;\\n  input: &quot;mu/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;mu&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.2861393392086029\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;sigma&quot;\\n  input: &quot;sigma/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;sigma&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 6.2831854820251465\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;gaussian_logpdf/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: -0.5\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;X&quot;\\n  input: &quot;mu/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/truediv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;gaussian_logpdf/mul_1&quot;\\n  input: &quot;gaussian_logpdf/truediv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Rank&quot;\\n  op: &quot;Rank&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/start&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/delta&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range&quot;\\n  op: &quot;Range&quot;\\n  input: &quot;range/start&quot;\\n  input: &quot;Rank&quot;\\n  input: &quot;range/delta&quot;\\n}\\nnode {\\n  name: &quot;logp&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  input: &quot;range&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;init&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^mu/Assign&quot;\\n  input: &quot;^sigma/Assign&quot;\\n}\\nnode {\\n  name: &quot;Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;logp&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Const&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Fill&quot;\\n  op: &quot;Fill&quot;\\n  input: &quot;gradients/Shape&quot;\\n  input: &quot;gradients/Const&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Neg_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/Fill&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Size&quot;\\n  op: &quot;Size&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;range&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/mod&quot;\\n  op: &quot;Mod&quot;\\n  input: &quot;gradients/logp_grad/add&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gradients/logp_grad/mod&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range/start&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range/delta&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range&quot;\\n  op: &quot;Range&quot;\\n  input: &quot;gradients/logp_grad/range/start&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  input: &quot;gradients/logp_grad/range/delta&quot;\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Fill/value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Fill&quot;\\n  op: &quot;Fill&quot;\\n  input: &quot;gradients/logp_grad/Shape_1&quot;\\n  input: &quot;gradients/logp_grad/Fill/value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  op: &quot;DynamicStitch&quot;\\n  input: &quot;gradients/logp_grad/range&quot;\\n  input: &quot;gradients/logp_grad/mod&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  input: &quot;gradients/logp_grad/Fill&quot;\\n  attr {\\n    key: &quot;N&quot;\\n    value {\\n      i: 2\\n    }\\n  }\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Maximum/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Maximum&quot;\\n  op: &quot;Maximum&quot;\\n  input: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  input: &quot;gradients/logp_grad/Maximum/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/floordiv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  input: &quot;gradients/logp_grad/Maximum&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/Neg_grad/Neg&quot;\\n  input: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Tile&quot;\\n  op: &quot;Tile&quot;\\n  input: &quot;gradients/logp_grad/Reshape&quot;\\n  input: &quot;gradients/logp_grad/floordiv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/truediv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/logp_grad/Tile&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/logp_grad/Tile&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/truediv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/truediv&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Square&quot;\\n  op: &quot;Square&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/truediv_1&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Square&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/truediv_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Log_grad/Inv&quot;\\n  op: &quot;Inv&quot;\\n  input: &quot;gaussian_logpdf/mul&quot;\\n  input: &quot;^gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/Inv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;X&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;mu/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Sum_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/AddN&quot;\\n  op: &quot;AddN&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Reshape&quot;\\n  attr {\\n    key: &quot;N&quot;\\n    value {\\n      i: 2\\n    }\\n  }\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.7278572444454114&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfutils.show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Continuing with the point that computing gradients is just another operation, we can also compute the gradients and look at their values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True:      mu=6.9647, sigma=2.8614\n",
      "Initial:   mu=0.6965, sigma=0.2861\n",
      "Gradients: mu=-74570.2734, sigma=-1926617.8750\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    grad_vals = sess.run(grads, feed_dict={X: X_vals})\n",
    "\n",
    "print(\"True:      mu={:.4f}, sigma={:.4f}\".format(true_mu, true_sigma))\n",
    "print(\"Initial:   mu={:.4f}, sigma={:.4f}\".format(initial_mu, initial_sigma))\n",
    "print(\"Gradients: mu={:.4f}, sigma={:.4f}\".format(*grad_vals))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, we are now ready to try to optimize the parameters of our Gaussian. We'll use the `AdamOptimizer` here as it is much more efficient than regular gradient descent (if you want to try it out yourself, change the optimizer to `tf.train.GradientDescentOptimizer` and set the learning rate to something small, like 0.0001):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "optimizer = tf.train.AdamOptimizer(0.1)\n",
    "train = optimizer.apply_gradients(zip(*[grads, [mu, sigma]]))\n",
    "\n",
    "# we need to recreate the initializer, because the Adam optimizer creates\n",
    "# additional variables\n",
    "init = tf.initialize_all_variables()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like computing gradients, the optimizer is just another set of operations in our graph. You should now see a new node called `Adam`, along with a couple of new variables (`beta1_power` and `beta2_power`) that the Adam optimizer uses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.5434046245690786&quot;).pbtxt = 'node {\\n  name: &quot;X&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.6964691877365112\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;mu&quot;\\n  input: &quot;mu/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;mu&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.2861393392086029\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;sigma&quot;\\n  input: &quot;sigma/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;sigma&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 6.2831854820251465\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;gaussian_logpdf/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: -0.5\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;X&quot;\\n  input: &quot;mu/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_1&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/pow_2&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2/x&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/truediv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gaussian_logpdf/add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;gaussian_logpdf/mul_1&quot;\\n  input: &quot;gaussian_logpdf/truediv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Rank&quot;\\n  op: &quot;Rank&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/start&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range/delta&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;range&quot;\\n  op: &quot;Range&quot;\\n  input: &quot;range/start&quot;\\n  input: &quot;Rank&quot;\\n  input: &quot;range/delta&quot;\\n}\\nnode {\\n  name: &quot;logp&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  input: &quot;range&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;init&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^mu/Assign&quot;\\n  input: &quot;^sigma/Assign&quot;\\n}\\nnode {\\n  name: &quot;Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;logp&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Const&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Fill&quot;\\n  op: &quot;Fill&quot;\\n  input: &quot;gradients/Shape&quot;\\n  input: &quot;gradients/Const&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Neg_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/Fill&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/add&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Size&quot;\\n  op: &quot;Size&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/add&quot;\\n  op: &quot;Add&quot;\\n  input: &quot;range&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/mod&quot;\\n  op: &quot;Mod&quot;\\n  input: &quot;gradients/logp_grad/add&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gradients/logp_grad/mod&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range/start&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range/delta&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/range&quot;\\n  op: &quot;Range&quot;\\n  input: &quot;gradients/logp_grad/range/start&quot;\\n  input: &quot;gradients/logp_grad/Size&quot;\\n  input: &quot;gradients/logp_grad/range/delta&quot;\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Fill/value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Fill&quot;\\n  op: &quot;Fill&quot;\\n  input: &quot;gradients/logp_grad/Shape_1&quot;\\n  input: &quot;gradients/logp_grad/Fill/value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  op: &quot;DynamicStitch&quot;\\n  input: &quot;gradients/logp_grad/range&quot;\\n  input: &quot;gradients/logp_grad/mod&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  input: &quot;gradients/logp_grad/Fill&quot;\\n  attr {\\n    key: &quot;N&quot;\\n    value {\\n      i: 2\\n    }\\n  }\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Maximum/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n        }\\n        int_val: 1\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Maximum&quot;\\n  op: &quot;Maximum&quot;\\n  input: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  input: &quot;gradients/logp_grad/Maximum/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/floordiv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/logp_grad/Shape&quot;\\n  input: &quot;gradients/logp_grad/Maximum&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/Neg_grad/Neg&quot;\\n  input: &quot;gradients/logp_grad/DynamicStitch&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/logp_grad/Tile&quot;\\n  op: &quot;Tile&quot;\\n  input: &quot;gradients/logp_grad/Reshape&quot;\\n  input: &quot;gradients/logp_grad/floordiv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/truediv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/logp_grad/Tile&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/logp_grad/Tile&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  input: &quot;gaussian_logpdf/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_1/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/truediv&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/truediv&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gaussian_logpdf/Neg&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Square&quot;\\n  op: &quot;Square&quot;\\n  input: &quot;gaussian_logpdf/mul_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/truediv_1&quot;\\n  op: &quot;Div&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Square&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/add_grad/Reshape_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/truediv_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Log_grad/Inv&quot;\\n  op: &quot;Inv&quot;\\n  input: &quot;gaussian_logpdf/mul&quot;\\n  input: &quot;^gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_1_grad/Reshape_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/Inv&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul_2/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/truediv_grad/Reshape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gaussian_logpdf/mul/x&quot;\\n  input: &quot;gradients/gaussian_logpdf/Log_grad/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow_1/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/Neg_grad/Neg&quot;\\n  input: &quot;gaussian_logpdf/pow_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;gaussian_logpdf/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow_2/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_2_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;gaussian_logpdf/pow/y&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sigma/read&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/mul_grad/Reshape_1&quot;\\n  input: &quot;gaussian_logpdf/pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;sigma/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_2&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Log&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/mul_3&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/pow_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Sum_1&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;X&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n  op: &quot;Shape&quot;\\n  input: &quot;mu/read&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_1_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Sum_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/gaussian_logpdf/sub_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Neg&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/AddN&quot;\\n  op: &quot;AddN&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_2_grad/Reshape&quot;\\n  input: &quot;gradients/gaussian_logpdf/pow_grad/Reshape&quot;\\n  attr {\\n    key: &quot;N&quot;\\n    value {\\n      i: 2\\n    }\\n  }\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta1_power/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.8999999761581421\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta1_power&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta1_power/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;beta1_power&quot;\\n  input: &quot;beta1_power/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta1_power/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;beta1_power&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta2_power/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.9990000128746033\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta2_power&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta2_power/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;beta2_power&quot;\\n  input: &quot;beta2_power/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;beta2_power/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;beta2_power&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;zeros&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;mu/Adam&quot;\\n  input: &quot;zeros&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;mu/Adam&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;zeros_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam_1&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam_1/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;mu/Adam_1&quot;\\n  input: &quot;zeros_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mu/Adam_1/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;mu/Adam_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;zeros_2&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;sigma/Adam&quot;\\n  input: &quot;zeros_2&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;sigma/Adam&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;zeros_3&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam_1&quot;\\n  op: &quot;Variable&quot;\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam_1/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;sigma/Adam_1&quot;\\n  input: &quot;zeros_3&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sigma/Adam_1/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;sigma/Adam_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/learning_rate&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.10000000149011612\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/beta1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.8999999761581421\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/beta2&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.9990000128746033\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/epsilon&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 9.99999993922529e-09\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/update_mu/ApplyAdam&quot;\\n  op: &quot;ApplyAdam&quot;\\n  input: &quot;mu&quot;\\n  input: &quot;mu/Adam&quot;\\n  input: &quot;mu/Adam_1&quot;\\n  input: &quot;beta1_power/read&quot;\\n  input: &quot;beta2_power/read&quot;\\n  input: &quot;Adam/learning_rate&quot;\\n  input: &quot;Adam/beta1&quot;\\n  input: &quot;Adam/beta2&quot;\\n  input: &quot;Adam/epsilon&quot;\\n  input: &quot;gradients/gaussian_logpdf/sub_grad/Reshape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/update_sigma/ApplyAdam&quot;\\n  op: &quot;ApplyAdam&quot;\\n  input: &quot;sigma&quot;\\n  input: &quot;sigma/Adam&quot;\\n  input: &quot;sigma/Adam_1&quot;\\n  input: &quot;beta1_power/read&quot;\\n  input: &quot;beta2_power/read&quot;\\n  input: &quot;Adam/learning_rate&quot;\\n  input: &quot;Adam/beta1&quot;\\n  input: &quot;Adam/beta2&quot;\\n  input: &quot;Adam/epsilon&quot;\\n  input: &quot;gradients/AddN&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@sigma&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;beta1_power/read&quot;\\n  input: &quot;Adam/beta1&quot;\\n  input: &quot;^Adam/update_mu/ApplyAdam&quot;\\n  input: &quot;^Adam/update_sigma/ApplyAdam&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;beta1_power&quot;\\n  input: &quot;Adam/mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;beta2_power/read&quot;\\n  input: &quot;Adam/beta2&quot;\\n  input: &quot;^Adam/update_mu/ApplyAdam&quot;\\n  input: &quot;^Adam/update_sigma/ApplyAdam&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam/Assign_1&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;beta2_power&quot;\\n  input: &quot;Adam/mul_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@mu&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;Adam&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^Adam/update_mu/ApplyAdam&quot;\\n  input: &quot;^Adam/update_sigma/ApplyAdam&quot;\\n  input: &quot;^Adam/Assign&quot;\\n  input: &quot;^Adam/Assign_1&quot;\\n}\\nnode {\\n  name: &quot;init_1&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^mu/Assign&quot;\\n  input: &quot;^sigma/Assign&quot;\\n  input: &quot;^beta1_power/Assign&quot;\\n  input: &quot;^beta2_power/Assign&quot;\\n  input: &quot;^mu/Adam/Assign&quot;\\n  input: &quot;^mu/Adam_1/Assign&quot;\\n  input: &quot;^sigma/Adam/Assign&quot;\\n  input: &quot;^sigma/Adam_1/Assign&quot;\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.5434046245690786&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tfutils.show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now construct our training loop. We'll run for 1000 optimization steps, and compute at each step the log probability of the data, the current value of `mu`, the current value of `sigma`, and the actual training operation. We'll print out the status of the training after every 100 steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log p(X | mu=0.7965, sigma=0.3861) = -294427.0000\n",
      "log p(X | mu=3.4322, sigma=1.6038) = -5360.1357\n",
      "log p(X | mu=4.7522, sigma=1.7714) = -3619.7363\n",
      "log p(X | mu=5.6646, sigma=1.8733) = -2916.5569\n",
      "log p(X | mu=6.2374, sigma=1.9476) = -2746.0251\n",
      "log p(X | mu=6.5817, sigma=2.0106) = -2660.7000\n",
      "log p(X | mu=6.7746, sigma=2.0696) = -2646.9377\n",
      "log p(X | mu=6.8673, sigma=2.1254) = -2564.0066\n",
      "log p(X | mu=6.9186, sigma=2.1784) = -2563.6431\n",
      "log p(X | mu=6.9543, sigma=2.2289) = -2568.0984\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for i in range(1000):\n",
    "        ops_to_run = {\n",
    "            \"logp\": logp,\n",
    "            \"mu\": mu,\n",
    "            \"sigma\": sigma,\n",
    "            \"train\": train\n",
    "        }\n",
    "        results = sess.run(ops_to_run, feed_dict={X: sample_X()})\n",
    "        if (i % 100) == 0:\n",
    "            print(\"log p(X | mu={mu:.4f}, sigma={sigma:.4f}) = {logp:.4f}\".format(**results))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's compare this to what we'd get if we computed the log probability under the true values of mu and sigma:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "log p(X | mu=6.9647, sigma=2.8614) = -2449.9692\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    logp_val = sess.run(logp, feed_dict={X: sample_X(), mu: true_mu, sigma: true_sigma})\n",
    "    print(\"log p(X | mu={mu:.4f}, sigma={sigma:.4f}) = {logp:.4f}\".format(\n",
    "            mu=true_mu, sigma=true_sigma, logp=logp_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note on running operations in the session\n",
    "\n",
    "When we tell the session to run something, it will *only* run what we tell it to, plus whatever operations the given ops depend on. So, if we told the session to just run our `logp` op, *it wouldn't run the training step or compute gradients* because `logp` doesn't depend on those operations. So, that's why we have to explicitly plass the `train` operation to the session if we want it to update the variables."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tensorflow",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}