Skip to content

Commit b4c9ee6

Browse files
atcemgilatcemgil
committed
Automatic Differentiation Exercise added
1 parent 4f3c728 commit b4c9ee6

6 files changed

+496
-136
lines changed

‎Automatic Differentiation.ipynb

+256
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,256 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Automatic differentiation\n",
8+
"\n",
9+
"Consider the function\n",
10+
"$$\n",
11+
"f(x) = \\cos(x)\\sin(x)\n",
12+
"$$\n",
13+
"\n",
14+
"When we want to evaluate the function numerically at a specific $x$, say $x=1$ we can implement a computer program like\n",
15+
"\n",
16+
"~~~\n",
17+
"x = 1\n",
18+
"f = cos(x)*sin(x)\n",
19+
"~~~\n",
20+
"\n",
21+
"Now suppose we need the derivative as well. For this example, it is a simple exercise to calculate the derivative as \n",
22+
"$$\n",
23+
"f'(x) = \\cos(x)\\cos(x) -\\sin(x)\\sin(x) = \\cos(x)^2 - \\sin(x)^2\n",
24+
"$$\n",
25+
"\n",
26+
"~~~\n",
27+
"x = 1\n",
28+
"df_dx = cos(x)*cos(x) - sin(x)*sin(x)\n",
29+
"~~~\n",
30+
"\n",
31+
"But could we have calculated the derivative without coding it up explicitely, that is without symbolically evaluating it a priori by hand?\n",
32+
"The answer turns out to be yes and it is a quite fascinating subject called __automatic differentiation__. Interestingly, this algorithm, known also as __backpropagation__, is in the core of todays artificial intelligence systems, programs that learn how to program themselves from input and output examples. See https://www.youtube.com/watch?v=aircAruvnKk for an introduction to a particular type of model, known as a __neural network__.\n",
33+
"\n",
34+
"To symbolically evaluate the derivative, we can use the chain rule which says when \n",
35+
"$$\n",
36+
"f(x) = g(h(x))\n",
37+
"$$\n",
38+
"we have\n",
39+
"$$\n",
40+
"f'(x) = g'(h(x)) h'(x)\n",
41+
"$$\n",
42+
"\n",
43+
"When we have functions of two variables, such as\n",
44+
"$$\n",
45+
"f(x) = g(h_1(x), h_2(x)) \n",
46+
"$$\n",
47+
"we have to adopt the partial derivative notation\n",
48+
"$$\n",
49+
"\\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial g} ( \\frac{\\partial g}{\\partial h_1} \\frac{\\partial h_1}{\\partial x} + \\frac{\\partial g}{\\partial h_2} \\frac{\\partial h_2}{\\partial x})\n",
50+
"$$\n",
51+
"\n",
52+
"\n",
53+
"To see a concrete example, consider the function\n",
54+
"$$\n",
55+
"f(x) = \\sin(x)\\cos(x)\n",
56+
"$$\n",
57+
"\n",
58+
"\n",
59+
"We define \n",
60+
"\\begin{align}\n",
61+
"h_1(x) & = c = \\cos(x) \\\\\n",
62+
"h_2(x) & = s = \\sin(x) \\\\\n",
63+
"g(c,s) & = g = c \\times s \\\\\n",
64+
"f & = g(c,s)\n",
65+
"\\end{align}\n",
66+
"\n",
67+
"that is equivalent to the following program\n",
68+
"~~~\n",
69+
"x = 1\n",
70+
"c = cos(x)\n",
71+
"s = sin(x)\n",
72+
"g = c * s\n",
73+
"f = g\n",
74+
"~~~\n",
75+
"\n",
76+
"This program can be represented by the following directed computation graph:\n",
77+
"<img src=\"../latex_figures/cos_sin.png\" width=\"300\">\n",
78+
"\n",
79+
"The function can be evaluated by traversing the variable nodes of the directed graph from the inputs to the outputs in the topological order. At each variable node, we merely evaluate the incoming function. Topological order guarantees that the inputs for the function are already calculated. \n",
80+
"\n",
81+
"It is not obvious, but the derivatives can also be calculated easily. By the chain rule, we have \n",
82+
"\\begin{eqnarray}\n",
83+
"\\frac{\\partial f}{\\partial x} &=& \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c} \\frac{\\partial c}{\\partial x} + \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial s} \\frac{\\partial s}{\\partial x} \\\\\n",
84+
"&=& 1 \\cdot s \\cdot \\sin(x) + 1 \\cdot c \\cdot (-\\cos(x)) \\\\\n",
85+
"&=& 1 \\cdot \\sin(x) \\cdot \\sin(x) + 1 \\cdot \\cos(x) \\cdot (-\\cos(x)) \\\\\n",
86+
"\\end{eqnarray}\n",
87+
"\n",
88+
"The derivative could have been calculated numerically by the following program\n",
89+
"\n",
90+
"~~~\n",
91+
"df_dx = 0, df_ds = 0, df_dc = 0, df_dg = 0 \n",
92+
"df_df = 1\n",
93+
"\n",
94+
"df_dg += df_df // df/dg = 1\n",
95+
"df_dc += s * df_dg // dg/dc = s\n",
96+
"df_ds += c * df_dg // dg/ds = c\n",
97+
"df_dx += cos(x) * df_ds // ds/dx = cos(x)\n",
98+
"df_dx += -sin(x) * df_dc // dc/dx = -sin(x)\n",
99+
"~~~\n",
100+
"\n",
101+
"Note that the total derivative consists of sums of several terms. Each term is the product of the derivatives along the path leading from $f$ to $x$. In the above example, there are only two paths: \n",
102+
"\n",
103+
"- $f,g,c,x$\n",
104+
"- $f,g,s,x$\n",
105+
"\n",
106+
"$$\n",
107+
"\\frac{\\partial f}{\\partial x} = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c} \\frac{\\partial c}{\\partial x} + \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial s} \\frac{\\partial s}{\\partial x}\n",
108+
"$$\n",
109+
"\n",
110+
"It is not obvious in this simple example but the fact that we are propagating backwards makes us save computation by storing the intermediate variables.\n",
111+
"\n",
112+
"This program can be represented by the following directed computation graph:\n",
113+
"<img src=\"../latex_figures/cos_sin_with_df.png\" width=\"400\">\n",
114+
"\n",
115+
"Note that during the backward pass, if we traverse variable nodes in the reverse topological order, we only need the derivatives already computed and perhaps some of the variables that are connected to the function node that are computed during the forward pass. As an example, consider\n",
116+
"\n",
117+
"$$\n",
118+
"\\frac{\\partial f}{\\partial c} = \\frac{\\partial f}{\\partial g} \\frac{\\partial g}{\\partial c}\n",
119+
"$$\n",
120+
"The first term is already available during the backward pass. The second term needs to be programmed by calculating the partial derivative of $g(s,c) = sc$ with respect to $c$. It has a simple form, namely $s$. More importantly, the numerical value is also immediately available, as it is calculated during the forward pass. For each function type, this calculation will be different but is nevertheless straightforward for all basic functions, including the binary arithmetic operators $+,-,\\times$ and $\\div$.\n",
121+
"\n",
122+
"\n",
123+
"# Programming Assignment\n",
124+
"\n",
125+
"In this exercise, you will write a program that gets two input files. The first file will be the function definition file, the second will be the input values for which the function and its derivative is to be calculated\n",
126+
"\n",
127+
"## Function Definition File\n",
128+
"The function definition file will have the following format:\n",
129+
"~~~\n",
130+
"<inputs>\n",
131+
"<output>\n",
132+
"<assignments>\n",
133+
"~~~\n",
134+
"\n",
135+
"An example is the following for the function\n",
136+
"$$\n",
137+
"f(x_1, x_2) = \\sin(2x_1)\\cos(x_1 x_2)\n",
138+
"$$\n",
139+
"\n",
140+
"~~~\n",
141+
"input x_1\n",
142+
"input x_2\n",
143+
"output f\n",
144+
"t_0 = mult 2 x_1\n",
145+
"t_1 = sin t_0\n",
146+
"t_2 = mult x_1 x_2\n",
147+
"t_3 = cos t_2\n",
148+
"t_4 = mult t_1 t_3\n",
149+
"f = t_4 \n",
150+
"~~~\n",
151+
"\n",
152+
"## Input Values File\n",
153+
"\n",
154+
"~~~\n",
155+
"<input 1><input 2>...<input n>\n",
156+
"<value 1><value 2>...<value n>\n",
157+
"<value 1><value 2>...<value n>\n",
158+
"<value 1><value 2>...<value n>\n",
159+
"...\n",
160+
"~~~\n",
161+
"\n",
162+
"If we want to evaluate the function on values $(x_1,x_2) = \\{(0,0),(1.2,-3),(5,5)\\}$, the example file format will be\n",
163+
"~~~\n",
164+
"x_1 x_2\n",
165+
"0 0\n",
166+
"1.2 -3\n",
167+
"5 5\n",
168+
"~~~\n",
169+
"\n",
170+
"Your program will have two output files: Function values and the derivatives (upto 5 digits of precision)\n",
171+
"\n",
172+
"~~~\n",
173+
"f\n",
174+
"0.0\n",
175+
"-0.60573\n",
176+
"-0.53924\n",
177+
"~~~\n",
178+
"\n",
179+
"The partial derivatives are\n",
180+
"\n",
181+
"$$\n",
182+
"\\frac{\\partial f}{\\partial x_1} = 2\\cos(2x_1)\\cos(x_1 x_2) - \\sin(2x_1)\\sin(x_1 x_2) x_2\n",
183+
"$$\n",
184+
"\n",
185+
"$$\n",
186+
"\\frac{\\partial f}{\\partial x_2} = -\\sin(2x_1)\\sin(x_1 x_2)x_1 \n",
187+
"$$\n",
188+
"\n",
189+
"\n",
190+
"Hence, the output file will be\n",
191+
"~~~\n",
192+
"df/dx_1 df/dx_2\n",
193+
"2.0 -0.0\n",
194+
"0.24682 -0.35869\n",
195+
"0.20232 -0.36001\n",
196+
"~~~\n",
197+
"\n",
198+
"The basic binary arithmetic operations\n",
199+
"~~~\n",
200+
"add\n",
201+
"subs\n",
202+
"mult\n",
203+
"divide\n",
204+
"~~~\n",
205+
"\n",
206+
"The following functions that are a subset of the cmath library must also be supported\n",
207+
"~~~\n",
208+
"cos\n",
209+
"sin\n",
210+
"tan\n",
211+
"acos\n",
212+
"asin\n",
213+
"atan\n",
214+
"exp\n",
215+
"log\n",
216+
"log10\n",
217+
"pow\n",
218+
"sqrt\n",
219+
"~~~\n",
220+
"\n",
221+
"\n",
222+
"The algorithm can be sketched as follows:\n",
223+
"\n",
224+
"- Construct a directed graph of variable and function nodes\n",
225+
"- Check if the graph is acyclic. If cyclic, generate an error message. The function is invalid\n",
226+
"- Forward pass: Evaluate the function graph in topological order to find the temporary variables. Write the function output.\n",
227+
"- Backward pass: Evaluate the derivatives by initializing $\\partial f/\\partial f = 1$ and propagate in the reverse topological order for finding the derivatives. To implement the chain rule, proceed by multiplication of the derivative at the output variable with the partial derivative across the immediate neighbors of a function node.\n",
228+
"\n",
229+
"\n",
230+
"\n",
231+
"\n"
232+
]
233+
}
234+
],
235+
"metadata": {
236+
"kernelspec": {
237+
"display_name": "Python [conda env:py36]",
238+
"language": "python",
239+
"name": "conda-env-py36-py"
240+
},
241+
"language_info": {
242+
"codemirror_mode": {
243+
"name": "ipython",
244+
"version": 3
245+
},
246+
"file_extension": ".py",
247+
"mimetype": "text/x-python",
248+
"name": "python",
249+
"nbconvert_exporter": "python",
250+
"pygments_lexer": "ipython3",
251+
"version": "3.6.1"
252+
}
253+
},
254+
"nbformat": 4,
255+
"nbformat_minor": 2
256+
}

‎ChangeOfVariables.ipynb

+53-6
Original file line numberDiff line numberDiff line change
@@ -974,8 +974,10 @@
974974
},
975975
{
976976
"cell_type": "code",
977-
"execution_count": 4,
978-
"metadata": {},
977+
"execution_count": 2,
978+
"metadata": {
979+
"scrolled": true
980+
},
979981
"outputs": [
980982
{
981983
"data": {
@@ -996,17 +998,17 @@
996998
" <iframe\n",
997999
" width=\"350\"\n",
9981000
" height=\"100\"\n",
999-
" src=\"../feed_forward2.pdf\"\n",
1001+
" src=\"../latex_figures/feed_forward2.pdf\"\n",
10001002
" frameborder=\"0\"\n",
10011003
" allowfullscreen\n",
10021004
" ></iframe>\n",
10031005
" "
10041006
],
10051007
"text/plain": [
1006-
"<IPython.lib.display.IFrame at 0x1062ee828>"
1008+
"<IPython.lib.display.IFrame at 0x10284fc88>"
10071009
]
10081010
},
1009-
"execution_count": 4,
1011+
"execution_count": 2,
10101012
"metadata": {},
10111013
"output_type": "execute_result"
10121014
}
@@ -1017,7 +1019,52 @@
10171019
"\n",
10181020
"display(HTML('<img src=\"../feed_forward2.png\" width=\"250\">'))\n",
10191021
"\n",
1020-
"IFrame(\"../feed_forward2.pdf\", width=350, height=100)\n"
1022+
"IFrame(\"../latex_figures/feed_forward2.pdf\", width=350, height=100)\n"
1023+
]
1024+
},
1025+
{
1026+
"cell_type": "markdown",
1027+
"metadata": {},
1028+
"source": []
1029+
},
1030+
{
1031+
"cell_type": "code",
1032+
"execution_count": 10,
1033+
"metadata": {},
1034+
"outputs": [
1035+
{
1036+
"name": "stdout",
1037+
"output_type": "stream",
1038+
"text": [
1039+
"0.0\n",
1040+
"-0.605727292083\n",
1041+
"-0.539235254827\n",
1042+
"[ 2. 0.24682407 0.20232278]\n",
1043+
"[-0. -0.35868752 -0.36001073]\n",
1044+
"2.0 -0.0\n",
1045+
"0.246824066159 -0.358687519304\n",
1046+
"0.202322781119 -0.360010730582\n"
1047+
]
1048+
}
1049+
],
1050+
"source": [
1051+
"import numpy as np\n",
1052+
"x_1 = np.array([0,1.2,5])\n",
1053+
"x_2 = np.array([0,-3,5])\n",
1054+
"\n",
1055+
"for x,y in zip(x_1,x_2):\n",
1056+
" print(np.sin(2*x)*np.cos(x*y))\n",
1057+
"\n",
1058+
" \n",
1059+
"dfdx1 = -np.sin(2*x_1)*np.sin(x_1*x_2)*x_2 + 2*np.cos(x_1)*np.cos(x_1*x_2)\n",
1060+
"dfdx2 = -np.sin(2*x_1)*np.sin(x_1*x_2)*x_1\n",
1061+
"\n",
1062+
"print(dfdx1)\n",
1063+
"\n",
1064+
"print(dfdx2)\n",
1065+
"\n",
1066+
"for i in range(3):\n",
1067+
" print(dfdx1[i], dfdx2[i])"
10211068
]
10221069
}
10231070
],

0 commit comments

Comments
 (0)