Skip to content

Commit dc18dca

Browse files
MarkDaoustMarkDaoust
committed
Import eager notebooks.
From https://github.com/tensorflow/tensorflow/ tensorflow/contrib/eager/python/examples/notebooks @64f191cdc0121bbcb322c3b11b160d638c2f4af9 I was unable to import the file history.
1 parent a12f4cd commit dc18dca

File tree

5 files changed

+1744
-0
lines changed

5 files changed

+1744
-0
lines changed

‎site/en/tutorials/eager/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
## Research and experimentation
2+
3+
Eager execution provides an imperative, define-by-run interface for advanced
4+
operations. Write custom layers, forward passes, and training loops with auto
5+
differentiation. Start with these notebooks, then read the
6+
[eager execution guide](https://www.tensorflow.org/guide/eager).
7+
8+
1. [Eager execution basics](./eager_basics.ipynb)
9+
2. [Automatic differentiation and gradient tapes](./automatic_differentiation.ipynb)
10+
3. [Custom training: basics](./custom_training.ipynb)
11+
4. [Custom layers](./custom_layers.ipynb)

‎site/en/tutorials/eager/automatic_differentiation.ipynb

Lines changed: 366 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,366 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"colab_type": "text",
7+
"id": "t09eeeR5prIJ"
8+
},
9+
"source": [
10+
"##### Copyright 2018 The TensorFlow Authors."
11+
]
12+
},
13+
{
14+
"cell_type": "code",
15+
"execution_count": 0,
16+
"metadata": {
17+
"cellView": "form",
18+
"colab": {
19+
"autoexec": {
20+
"startup": false,
21+
"wait_interval": 0
22+
}
23+
},
24+
"colab_type": "code",
25+
"id": "GCCk8_dHpuNf"
26+
},
27+
"outputs": [],
28+
"source": [
29+
"#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
30+
"# you may not use this file except in compliance with the License.\n",
31+
"# You may obtain a copy of the License at\n",
32+
"#\n",
33+
"# https://www.apache.org/licenses/LICENSE-2.0\n",
34+
"#\n",
35+
"# Unless required by applicable law or agreed to in writing, software\n",
36+
"# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
37+
"# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
38+
"# See the License for the specific language governing permissions and\n",
39+
"# limitations under the License."
40+
]
41+
},
42+
{
43+
"cell_type": "markdown",
44+
"metadata": {
45+
"colab_type": "text",
46+
"id": "xh8WkEwWpnm7"
47+
},
48+
"source": [
49+
"# Automatic differentiation and gradient tape"
50+
]
51+
},
52+
{
53+
"cell_type": "markdown",
54+
"metadata": {
55+
"colab_type": "text",
56+
"id": "idv0bPeCp325"
57+
},
58+
"source": [
59+
"\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\u003ctd\u003e\n",
60+
"\u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/examples/notebooks/automatic_differentiation.ipynb\"\u003e\n",
61+
" \u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003eRun in Google Colab\u003c/a\u003e\n",
62+
"\u003c/td\u003e\u003ctd\u003e\n",
63+
"\u003ca target=\"_blank\" href=\"https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/eager/python/examples/notebooks/automatic_differentiation.ipynb\"\u003e\u003cimg width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003eView source on GitHub\u003c/a\u003e\u003c/td\u003e\u003c/table\u003e"
64+
]
65+
},
66+
{
67+
"cell_type": "markdown",
68+
"metadata": {
69+
"colab_type": "text",
70+
"id": "vDJ4XzMqodTy"
71+
},
72+
"source": [
73+
"In the previous tutorial we introduced `Tensor`s and operations on them. In this tutorial we will cover [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), a key technique for optimizing machine learning models."
74+
]
75+
},
76+
{
77+
"cell_type": "markdown",
78+
"metadata": {
79+
"colab_type": "text",
80+
"id": "GQJysDM__Qb0"
81+
},
82+
"source": [
83+
"## Setup\n"
84+
]
85+
},
86+
{
87+
"cell_type": "code",
88+
"execution_count": 0,
89+
"metadata": {
90+
"colab": {
91+
"autoexec": {
92+
"startup": false,
93+
"wait_interval": 0
94+
}
95+
},
96+
"colab_type": "code",
97+
"id": "OiMPZStlibBv"
98+
},
99+
"outputs": [],
100+
"source": [
101+
"import tensorflow as tf\n",
102+
"tf.enable_eager_execution()\n",
103+
"\n",
104+
"tfe = tf.contrib.eager # Shorthand for some symbols"
105+
]
106+
},
107+
{
108+
"cell_type": "markdown",
109+
"metadata": {
110+
"colab_type": "text",
111+
"id": "1CLWJl0QliB0"
112+
},
113+
"source": [
114+
"## Derivatives of a function\n",
115+
"\n",
116+
"TensorFlow provides APIs for automatic differentiation - computing the derivative of a function. The way that more closely mimics the math is to encapsulate the computation in a Python function, say `f`, and use `tfe.gradients_function` to create a function that computes the derivatives of `f` with respect to its arguments. If you're familiar with [autograd](https://github.com/HIPS/autograd) for differentiating numpy functions, this will be familiar. For example: "
117+
]
118+
},
119+
{
120+
"cell_type": "code",
121+
"execution_count": 0,
122+
"metadata": {
123+
"colab": {
124+
"autoexec": {
125+
"startup": false,
126+
"wait_interval": 0
127+
}
128+
},
129+
"colab_type": "code",
130+
"id": "9FViq92UX7P8"
131+
},
132+
"outputs": [],
133+
"source": [
134+
"from math import pi\n",
135+
"\n",
136+
"def f(x):\n",
137+
" return tf.square(tf.sin(x))\n",
138+
"\n",
139+
"assert f(pi/2).numpy() == 1.0\n",
140+
"\n",
141+
"\n",
142+
"# grad_f will return a list of derivatives of f\n",
143+
"# with respect to its arguments. Since f() has a single argument,\n",
144+
"# grad_f will return a list with a single element.\n",
145+
"grad_f = tfe.gradients_function(f)\n",
146+
"assert tf.abs(grad_f(pi/2)[0]).numpy() \u003c 1e-7"
147+
]
148+
},
149+
{
150+
"cell_type": "markdown",
151+
"metadata": {
152+
"colab_type": "text",
153+
"id": "v9fPs8RyopCf"
154+
},
155+
"source": [
156+
"### Higher-order gradients\n",
157+
"\n",
158+
"The same API can be used to differentiate as many times as you like:\n"
159+
]
160+
},
161+
{
162+
"cell_type": "code",
163+
"execution_count": 0,
164+
"metadata": {
165+
"colab": {
166+
"autoexec": {
167+
"startup": false,
168+
"wait_interval": 0
169+
}
170+
},
171+
"colab_type": "code",
172+
"id": "3D0ZvnGYo0rW"
173+
},
174+
"outputs": [],
175+
"source": [
176+
"def f(x):\n",
177+
" return tf.square(tf.sin(x))\n",
178+
"\n",
179+
"def grad(f):\n",
180+
" return lambda x: tfe.gradients_function(f)(x)[0]\n",
181+
"\n",
182+
"x = tf.lin_space(-2*pi, 2*pi, 100) # 100 points between -2π and +2π\n",
183+
"\n",
184+
"import matplotlib.pyplot as plt\n",
185+
"\n",
186+
"plt.plot(x, f(x), label=\"f\")\n",
187+
"plt.plot(x, grad(f)(x), label=\"first derivative\")\n",
188+
"plt.plot(x, grad(grad(f))(x), label=\"second derivative\")\n",
189+
"plt.plot(x, grad(grad(grad(f)))(x), label=\"third derivative\")\n",
190+
"plt.legend()\n",
191+
"plt.show()"
192+
]
193+
},
194+
{
195+
"cell_type": "markdown",
196+
"metadata": {
197+
"colab_type": "text",
198+
"id": "-39gouo7mtgu"
199+
},
200+
"source": [
201+
"## Gradient tapes\n",
202+
"\n",
203+
"Every differentiable TensorFlow operation has an associated gradient function. For example, the gradient function of `tf.square(x)` would be a function that returns `2.0 * x`. To compute the gradient of a user-defined function (like `f(x)` in the example above), TensorFlow first \"records\" all the operations applied to compute the output of the function. We call this record a \"tape\". It then uses that tape and the gradients functions associated with each primitive operation to compute the gradients of the user-defined function using [reverse mode differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation).\n",
204+
"\n",
205+
"Since operations are recorded as they are executed, Python control flow (using `if`s and `while`s for example) is naturally handled:\n",
206+
"\n"
207+
]
208+
},
209+
{
210+
"cell_type": "code",
211+
"execution_count": 0,
212+
"metadata": {
213+
"colab": {
214+
"autoexec": {
215+
"startup": false,
216+
"wait_interval": 0
217+
}
218+
},
219+
"colab_type": "code",
220+
"id": "MH0UfjympWf7"
221+
},
222+
"outputs": [],
223+
"source": [
224+
"def f(x, y):\n",
225+
" output = 1\n",
226+
" # Must use range(int(y)) instead of range(y) in Python 3 when\n",
227+
" # using TensorFlow 1.10 and earlier. Can use range(y) in 1.11+\n",
228+
" for i in range(int(y)):\n",
229+
" output = tf.multiply(output, x)\n",
230+
" return output\n",
231+
"\n",
232+
"def g(x, y):\n",
233+
" # Return the gradient of `f` with respect to it's first parameter\n",
234+
" return tfe.gradients_function(f)(x, y)[0]\n",
235+
"\n",
236+
"assert f(3.0, 2).numpy() == 9.0 # f(x, 2) is essentially x * x\n",
237+
"assert g(3.0, 2).numpy() == 6.0 # And its gradient will be 2 * x\n",
238+
"assert f(4.0, 3).numpy() == 64.0 # f(x, 3) is essentially x * x * x\n",
239+
"assert g(4.0, 3).numpy() == 48.0 # And its gradient will be 3 * x * x"
240+
]
241+
},
242+
{
243+
"cell_type": "markdown",
244+
"metadata": {
245+
"colab_type": "text",
246+
"id": "aNmR5-jhpX2t"
247+
},
248+
"source": [
249+
"At times it may be inconvenient to encapsulate computation of interest into a function. For example, if you want the gradient of the output with respect to intermediate values computed in the function. In such cases, the slightly more verbose but explicit [tf.GradientTape](https://www.tensorflow.org/api_docs/python/tf/GradientTape) context is useful. All computation inside the context of a `tf.GradientTape` is \"recorded\".\n",
250+
"\n",
251+
"For example:"
252+
]
253+
},
254+
{
255+
"cell_type": "code",
256+
"execution_count": 0,
257+
"metadata": {
258+
"colab": {
259+
"autoexec": {
260+
"startup": false,
261+
"wait_interval": 0
262+
}
263+
},
264+
"colab_type": "code",
265+
"id": "bAFeIE8EuVIq"
266+
},
267+
"outputs": [],
268+
"source": [
269+
"x = tf.ones((2, 2))\n",
270+
" \n",
271+
"# TODO(b/78880779): Remove the 'persistent=True' argument and use\n",
272+
"# a single t.gradient() call when the bug is resolved.\n",
273+
"with tf.GradientTape(persistent=True) as t:\n",
274+
" # TODO(ashankar): Explain with \"watch\" argument better?\n",
275+
" t.watch(x)\n",
276+
" y = tf.reduce_sum(x)\n",
277+
" z = tf.multiply(y, y)\n",
278+
"\n",
279+
"# Use the same tape to compute the derivative of z with respect to the\n",
280+
"# intermediate value y.\n",
281+
"dz_dy = t.gradient(z, y)\n",
282+
"assert dz_dy.numpy() == 8.0\n",
283+
"\n",
284+
"# Derivative of z with respect to the original input tensor x\n",
285+
"dz_dx = t.gradient(z, x)\n",
286+
"for i in [0, 1]:\n",
287+
" for j in [0, 1]:\n",
288+
" assert dz_dx[i][j].numpy() == 8.0"
289+
]
290+
},
291+
{
292+
"cell_type": "markdown",
293+
"metadata": {
294+
"colab_type": "text",
295+
"id": "DK05KXrAAld3"
296+
},
297+
"source": [
298+
"### Higher-order gradients\n",
299+
"\n",
300+
"Operations inside of the `GradientTape` context manager are recorded for automatic differentiation. If gradients are computed in that context, then the gradient computation is recorded as well. As a result, the exact same API works for higher-order gradients as well. For example:"
301+
]
302+
},
303+
{
304+
"cell_type": "code",
305+
"execution_count": 0,
306+
"metadata": {
307+
"colab": {
308+
"autoexec": {
309+
"startup": false,
310+
"wait_interval": 0
311+
}
312+
},
313+
"colab_type": "code",
314+
"id": "cPQgthZ7ugRJ"
315+
},
316+
"outputs": [],
317+
"source": [
318+
"# TODO(ashankar): Should we use the persistent tape here instead? Follow up on Tom and Alex's discussion\n",
319+
"\n",
320+
"x = tf.constant(1.0) # Convert the Python 1.0 to a Tensor object\n",
321+
"\n",
322+
"with tf.GradientTape() as t:\n",
323+
" with tf.GradientTape() as t2:\n",
324+
" t2.watch(x)\n",
325+
" y = x * x * x\n",
326+
" # Compute the gradient inside the 't' context manager\n",
327+
" # which means the gradient computation is differentiable as well.\n",
328+
" dy_dx = t2.gradient(y, x)\n",
329+
"d2y_dx2 = t.gradient(dy_dx, x)\n",
330+
"\n",
331+
"assert dy_dx.numpy() == 3.0\n",
332+
"assert d2y_dx2.numpy() == 6.0"
333+
]
334+
},
335+
{
336+
"cell_type": "markdown",
337+
"metadata": {
338+
"colab_type": "text",
339+
"id": "4U1KKzUpNl58"
340+
},
341+
"source": [
342+
"## Next Steps\n",
343+
"\n",
344+
"In this tutorial we covered gradient computation in TensorFlow. With that we have enough of the primitives required to build an train neural networks, which we will cover in the [next tutorial](https://github.com/tensorflow/models/tree/master/official/contrib/eager/python/examples/notebooks/3_neural_networks.ipynb)."
345+
]
346+
}
347+
],
348+
"metadata": {
349+
"colab": {
350+
"collapsed_sections": [],
351+
"default_view": {},
352+
"name": "automatic_differentiation.ipynb",
353+
"private_outputs": true,
354+
"provenance": [],
355+
"toc_visible": true,
356+
"version": "0.3.2",
357+
"views": {}
358+
},
359+
"kernelspec": {
360+
"display_name": "Python 3",
361+
"name": "python3"
362+
}
363+
},
364+
"nbformat": 4,
365+
"nbformat_minor": 0
366+
}

0 commit comments

Comments
 (0)