diff --git a/lab1/GeneralUser.sf2 b/lab1/GeneralUser.sf2 new file mode 100644 index 00000000..0450c056 Binary files /dev/null and b/lab1/GeneralUser.sf2 differ diff --git a/lab1/PT_Part1_Intro.ipynb b/lab1/PT_Part1_Intro.ipynb index db97d067..17da79e1 100644 --- a/lab1/PT_Part1_Intro.ipynb +++ b/lab1/PT_Part1_Intro.ipynb @@ -1,701 +1,822 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "WBk0ZDWY-ff8" - }, - "source": [ - "\n", - " \n", - " \n", - " \n", - "
\n", - " \n", - " Visit MIT Deep Learning\n", - " Run in Google Colab\n", - " View Source on GitHub
\n", - "\n", - "# Copyright Information\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "3eI6DUic-6jo" - }, - "outputs": [], - "source": [ - "# Copyright 2025 MIT Introduction to Deep Learning. All Rights Reserved.\n", - "#\n", - "# Licensed under the MIT License. You may not use this file except in compliance\n", - "# with the License. Use and/or modification of this code outside of MIT Introduction\n", - "# to Deep Learning must reference:\n", - "#\n", - "# © MIT Introduction to Deep Learning\n", - "# http://introtodeeplearning.com\n", - "#" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "57knM8jrYZ2t" - }, - "source": [ - "# Lab 1: Intro to PyTorch and Music Generation with RNNs\n", - "\n", - "In this lab, you'll get exposure to using PyTorch and learn how it can be used for deep learning. Go through the code and run each cell. Along the way, you'll encounter several ***TODO*** blocks -- follow the instructions to fill them out before running those cells and continuing.\n", - "\n", - "\n", - "# Part 1: Intro to PyTorch\n", - "\n", - "## 0.1 Install PyTorch\n", - "\n", - "[PyTorch](https://pytorch.org/) is a popular deep learning library known for its flexibility and ease of use. Here we'll learn how computations are represented and how to define a simple neural network in PyTorch. For all the labs in Introduction to Deep Learning 2025, there will be a PyTorch version available.\n", - "\n", - "Let's install PyTorch and a couple of dependencies." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "LkaimNJfYZ2w" - }, - "outputs": [], - "source": [ - "import torch\n", - "import torch.nn as nn\n", - "\n", - "# Download and import the MIT Introduction to Deep Learning package\n", - "!pip install mitdeeplearning --quiet\n", - "import mitdeeplearning as mdl\n", - "\n", - "import numpy as np\n", - "import matplotlib.pyplot as plt" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "2QNMcdP4m3Vs" - }, - "source": [ - "## 1.1 What is PyTorch?\n", - "\n", - "PyTorch is a machine learning library, like TensorFlow. At its core, PyTorch provides an interface for creating and manipulating [tensors](https://pytorch.org/docs/stable/tensors.html), which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base datatypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions. PyTorch provides the ability to perform computation on these tensors, define neural networks, and train them efficiently.\n", - "\n", - "The [```shape```](https://pytorch.org/docs/stable/generated/torch.Tensor.shape.html#torch.Tensor.shape) of a PyTorch tensor defines its number of dimensions and the size of each dimension. The `ndim` or [```dim```](https://pytorch.org/docs/stable/generated/torch.Tensor.dim.html#torch.Tensor.dim) of a PyTorch tensor provides the number of dimensions (n-dimensions) -- this is equivalent to the tensor's rank (as is used in TensorFlow), and you can also think of this as the tensor's order or degree.\n", - "\n", - "Let’s start by creating some tensors and inspecting their properties:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "tFxztZQInlAB" - }, - "outputs": [], - "source": [ - "integer = torch.tensor(1234)\n", - "decimal = torch.tensor(3.14159265359)\n", - "\n", - "print(f\"`integer` is a {integer.ndim}-d Tensor: {integer}\")\n", - "print(f\"`decimal` is a {decimal.ndim}-d Tensor: {decimal}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-dljcPUcoJZ6" - }, - "source": [ - "Vectors and lists can be used to create 1-d tensors:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "oaHXABe8oPcO" - }, - "outputs": [], - "source": [ - "fibonacci = torch.tensor([1, 1, 2, 3, 5, 8])\n", - "count_to_100 = torch.tensor(range(100))\n", - "\n", - "print(f\"`fibonacci` is a {fibonacci.ndim}-d Tensor with shape: {fibonacci.shape}\")\n", - "print(f\"`count_to_100` is a {count_to_100.ndim}-d Tensor with shape: {count_to_100.shape}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "gvffwkvtodLP" - }, - "source": [ - "Next, let’s create 2-d (i.e., matrices) and higher-rank tensors. In image processing and computer vision, we will use 4-d Tensors with dimensions corresponding to batch size, number of color channels, image height, and image width." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "tFeBBe1IouS3" - }, - "outputs": [], - "source": [ - "### Defining higher-order Tensors ###\n", - "\n", - "'''TODO: Define a 2-d Tensor'''\n", - "matrix = # TODO\n", - "\n", - "assert isinstance(matrix, torch.Tensor), \"matrix must be a torch Tensor object\"\n", - "assert matrix.ndim == 2\n", - "\n", - "'''TODO: Define a 4-d Tensor.'''\n", - "# Use torch.zeros to initialize a 4-d Tensor of zeros with size 10 x 3 x 256 x 256.\n", - "# You can think of this as 10 images where each image is RGB 256 x 256.\n", - "images = # TODO\n", - "\n", - "assert isinstance(images, torch.Tensor), \"images must be a torch Tensor object\"\n", - "assert images.ndim == 4, \"images must have 4 dimensions\"\n", - "assert images.shape == (10, 3, 256, 256), \"images is incorrect shape\"\n", - "print(f\"images is a {images.ndim}-d Tensor with shape: {images.shape}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "wkaCDOGapMyl" - }, - "source": [ - "As you have seen, the `shape` of a tensor provides the number of elements in each tensor dimension. The `shape` is quite useful, and we'll use it often. You can also use slicing to access subtensors within a higher-rank tensor:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "FhaufyObuLEG" - }, - "outputs": [], - "source": [ - "row_vector = matrix[1]\n", - "column_vector = matrix[:, 1]\n", - "scalar = matrix[0, 1]\n", - "\n", - "print(f\"`row_vector`: {row_vector}\")\n", - "print(f\"`column_vector`: {column_vector}\")\n", - "print(f\"`scalar`: {scalar}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "iD3VO-LZYZ2z" - }, - "source": [ - "## 1.2 Computations on Tensors\n", - "\n", - "A convenient way to think about and visualize computations in a machine learning framework like PyTorch is in terms of graphs. We can define this graph in terms of tensors, which hold data, and the mathematical operations that act on these tensors in some order. Let's look at a simple example, and define this computation using PyTorch:\n", - "\n", - "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/add-graph.png)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "X_YJrZsxYZ2z" - }, - "outputs": [], - "source": [ - "# Create the nodes in the graph and initialize values\n", - "a = torch.tensor(15)\n", - "b = torch.tensor(61)\n", - "\n", - "# Add them!\n", - "c1 = torch.add(a, b)\n", - "c2 = a + b # PyTorch overrides the \"+\" operation so that it is able to act on Tensors\n", - "print(f\"c1: {c1}\")\n", - "print(f\"c2: {c2}\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Mbfv_QOiYZ23" - }, - "source": [ - "Notice how we've created a computation graph consisting of PyTorch operations, and how the output is a tensor with value 76 -- we've just created a computation graph consisting of operations, and it's executed them and given us back the result.\n", - "\n", - "Now let's consider a slightly more complicated example:\n", - "\n", - "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/computation-graph.png)\n", - "\n", - "Here, we take two inputs, `a, b`, and compute an output `e`. Each node in the graph represents an operation that takes some input, does some computation, and passes its output to another node.\n", - "\n", - "Let's define a simple function in PyTorch to construct this computation function:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "PJnfzpWyYZ23", - "scrolled": true - }, - "outputs": [], - "source": [ - "### Defining Tensor computations ###\n", - "\n", - "# Construct a simple computation function\n", - "def func(a, b):\n", - " '''TODO: Define the operation for c, d, e.'''\n", - " c = # TODO\n", - " d = # TODO\n", - " e = # TODO\n", - " return e\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "AwrRfDMS2-oy" - }, - "source": [ - "Now, we can call this function to execute the computation graph given some inputs `a,b`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "pnwsf8w2uF7p" - }, - "outputs": [], - "source": [ - "# Consider example values for a,b\n", - "a, b = 1.5, 2.5\n", - "# Execute the computation\n", - "e_out = func(a, b)\n", - "print(f\"e_out: {e_out}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "6HqgUIUhYZ29" - }, - "source": [ - "Notice how our output is a tensor with value defined by the output of the computation, and that the output has no shape as it is a single scalar value." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "1h4o9Bb0YZ29" - }, - "source": [ - "## 1.3 Neural networks in PyTorch\n", - "We can also define neural networks in PyTorch. PyTorch uses [``torch.nn.Module``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), which serves as a base class for all neural network modules in PyTorch and thus provides a framework for building and training neural networks.\n", - "\n", - "Let's consider the example of a simple perceptron defined by just one dense (aka fully-connected or linear) layer: $ y = \\sigma(Wx + b) $, where $W$ represents a matrix of weights, $b$ is a bias, $x$ is the input, $\\sigma$ is the sigmoid activation function, and $y$ is the output.\n", - "\n", - "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/computation-graph-2.png)\n", - "\n", - "We will use `torch.nn.Module` to define layers -- the building blocks of neural networks. Layers implement common neural networks operations. In PyTorch, when we implement a layer, we subclass `nn.Module` and define the parameters of the layer as attributes of our new class. We also define and override a function [``forward``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward), which will define the forward pass computation that is performed at every step. All classes subclassing `nn.Module` should override the `forward` function.\n", - "\n", - "Let's write a dense layer class to implement a perceptron defined above." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "HutbJk-1kHPh" - }, - "outputs": [], - "source": [ - "### Defining a dense layer ###\n", - "\n", - "# num_inputs: number of input nodes\n", - "# num_outputs: number of output nodes\n", - "# x: input to the layer\n", - "\n", - "class OurDenseLayer(torch.nn.Module):\n", - " def __init__(self, num_inputs, num_outputs):\n", - " super(OurDenseLayer, self).__init__()\n", - " # Define and initialize parameters: a weight matrix W and bias b\n", - " # Note that the parameter initialize is random!\n", - " self.W = torch.nn.Parameter(torch.randn(num_inputs, num_outputs))\n", - " self.bias = torch.nn.Parameter(torch.randn(num_outputs))\n", - "\n", - " def forward(self, x):\n", - " '''TODO: define the operation for z (hint: use torch.matmul).'''\n", - " z = # TODO\n", - "\n", - " '''TODO: define the operation for out (hint: use torch.sigmoid).'''\n", - " y = # TODO\n", - " return y\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "GqeEbn959hV_" - }, - "source": [ - "Now, let's test the output of our layer." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "2yxjCPa69hV_" - }, - "outputs": [], - "source": [ - "# Define a layer and test the output!\n", - "num_inputs = 2\n", - "num_outputs = 3\n", - "layer = OurDenseLayer(num_inputs, num_outputs)\n", - "x_input = torch.tensor([[1, 2.]])\n", - "y = layer(x_input)\n", - "\n", - "print(f\"input shape: {x_input.shape}\")\n", - "print(f\"output shape: {y.shape}\")\n", - "print(f\"output result: {y}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Jt1FgM7qYZ3D" - }, - "source": [ - "Conveniently, PyTorch has defined a number of ```nn.Modules``` (or Layers) that are commonly used in neural networks, for example a [```nn.Linear```](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) or [`nn.Sigmoid`](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) module.\n", - "\n", - "Now, instead of using a single ```Module``` to define our simple neural network, we'll use the [`nn.Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) module from PyTorch and a single [`nn.Linear` ](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer to define our network. With the `Sequential` API, you can readily create neural networks by stacking together layers like building blocks." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "7WXTpmoL6TDz" - }, - "outputs": [], - "source": [ - "### Defining a neural network using the PyTorch Sequential API ###\n", - "\n", - "# define the number of inputs and outputs\n", - "n_input_nodes = 2\n", - "n_output_nodes = 3\n", - "\n", - "# Define the model\n", - "'''TODO: Use the Sequential API to define a neural network with a\n", - " single linear (dense!) layer, followed by non-linearity to compute z'''\n", - "model = nn.Sequential( ''' TODO ''' )\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HDGcwYfUyR-U" - }, - "source": [ - "We've defined our model using the Sequential API. Now, we can test it out using an example input:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "zKhp6XqCFFa0" - }, - "outputs": [], - "source": [ - "# Test the model with example input\n", - "x_input = torch.tensor([[1, 2.]])\n", - "model_output = model(x_input)\n", - "print(f\"input shape: {x_input.shape}\")\n", - "print(f\"output shape: {y.shape}\")\n", - "print(f\"output result: {y}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "596NvsOOtr9F" - }, - "source": [ - "With PyTorch, we can create more flexible models by subclassing [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). The `nn.Module` class allows us to group layers together flexibly to define new architectures.\n", - "\n", - "As we saw earlier with `OurDenseLayer`, we can subclass `nn.Module` to create a class for our model, and then define the forward pass through the network using the `forward` function. Subclassing affords the flexibility to define custom layers, custom training loops, custom activation functions, and custom models. Let's define the same neural network model as above (i.e., Linear layer with an activation function after it), now using subclassing and using PyTorch's built in linear layer from `nn.Linear`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "K4aCflPVyViD" - }, - "outputs": [], - "source": [ - "### Defining a model using subclassing ###\n", - "\n", - "class LinearWithSigmoidActivation(nn.Module):\n", - " def __init__(self, num_inputs, num_outputs):\n", - " super(LinearWithSigmoidActivation, self).__init__()\n", - " '''TODO: define a model with a single Linear layer and sigmoid activation.'''\n", - " self.linear = '''TODO: linear layer'''\n", - " self.activation = '''TODO: sigmoid activation'''\n", - "\n", - " def forward(self, inputs):\n", - " linear_output = self.linear(inputs)\n", - " output = self.activation(linear_output)\n", - " return output\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "goKCQ9dEGzRn" - }, - "source": [ - "Let's test out our new model, using an example input, setting `n_input_nodes=2` and `n_output_nodes=3` as before." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "V-eNhSyRG6hl" - }, - "outputs": [], - "source": [ - "n_input_nodes = 2\n", - "n_output_nodes = 3\n", - "model = LinearWithSigmoidActivation(n_input_nodes, n_output_nodes)\n", - "x_input = torch.tensor([[1, 2.]])\n", - "y = model(x_input)\n", - "print(f\"input shape: {x_input.shape}\")\n", - "print(f\"output shape: {y.shape}\")\n", - "print(f\"output result: {y}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "HTIFMJLAzsyE" - }, - "source": [ - "Importantly, `nn.Module` affords us a lot of flexibility to define custom models. For example, we can use boolean arguments in the `forward` function to specify different network behaviors, for example different behaviors during training and inference. Let's suppose under some instances we want our network to simply output the input, without any perturbation. We define a boolean argument `isidentity` to control this behavior:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "P7jzGX5D1xT5" - }, - "outputs": [], - "source": [ - "### Custom behavior with subclassing nn.Module ###\n", - "\n", - "class LinearButSometimesIdentity(nn.Module):\n", - " def __init__(self, num_inputs, num_outputs):\n", - " super(LinearButSometimesIdentity, self).__init__()\n", - " self.linear = nn.Linear(num_inputs, num_outputs)\n", - "\n", - " '''TODO: Implement the behavior where the network outputs the input, unchanged,\n", - " under control of the isidentity argument.'''\n", - " def forward(self, inputs, isidentity=False):\n", - " ''' TODO '''\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "Ku4rcCGx5T3y" - }, - "source": [ - "Let's test this behavior:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "NzC0mgbk5dp2" - }, - "outputs": [], - "source": [ - "# Test the IdentityModel\n", - "model = LinearButSometimesIdentity(num_inputs=2, num_outputs=3)\n", - "x_input = torch.tensor([[1, 2.]])\n", - "\n", - "'''TODO: pass the input into the model and call with and without the input identity option.'''\n", - "out_with_linear = # TODO\n", - "\n", - "out_with_identity = # TODO\n", - "\n", - "print(f\"input: {x_input}\")\n", - "print(\"Network linear output: {}; network identity output: {}\".format(out_with_linear, out_with_identity))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7V1dEqdk6VI5" - }, - "source": [ - "Now that we have learned how to define layers and models in PyTorch using both the Sequential API and subclassing `nn.Module`, we're ready to turn our attention to how to actually implement network training with backpropagation." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "dQwDhKn8kbO2" - }, - "source": [ - "## 1.4 Automatic Differentiation in PyTorch\n", - "\n", - "In PyTorch, [`torch.autograd`](https://pytorch.org/docs/stable/autograd.html) is used for [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), which is critical for training deep learning models with [backpropagation](https://en.wikipedia.org/wiki/Backpropagation).\n", - "\n", - "We will use the PyTorch [`.backward()`](https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html) method to trace operations for computing gradients. On a tensor, the [`requires_grad`](https://pytorch.org/docs/stable/generated/torch.Tensor.requires_grad_.html) attribute controls whether autograd should record operations on that tensor. When a forward pass is made through the network, PyTorch builds a computational graph dynamically; then, to compute the gradient, the `backward()` method is called to perform backpropagation.\n", - "\n", - "Let's compute the gradient of $ y = x^2 $:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "tdkqk8pw5yJM" - }, - "outputs": [], - "source": [ - "### Gradient computation ###\n", - "\n", - "# y = x^2\n", - "# Example: x = 3.0\n", - "x = torch.tensor(3.0, requires_grad=True)\n", - "y = x ** 2\n", - "y.backward() # Compute the gradient\n", - "\n", - "dy_dx = x.grad\n", - "print(\"dy_dx of y=x^2 at x=3.0 is: \", dy_dx)\n", - "assert dy_dx == 6.0\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "JhU5metS5xF3" - }, - "source": [ - "In training neural networks, we use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Now that we have a sense of how PyTorch's autograd can be used to compute and access derivatives, we will look at an example where we use automatic differentiation and SGD to find the minimum of $ L=(x-x_f)^2 $. Here $x_f$ is a variable for a desired value we are trying to optimize for; $L$ represents a loss that we are trying to minimize. While we can clearly solve this problem analytically ($ x_{min}=x_f $), considering how we can compute this using PyTorch's autograd sets us up nicely for future labs where we use gradient descent to optimize entire neural network losses." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "attributes": { - "classes": [ - "py" - ], - "id": "" - }, - "id": "7g1yWiSXqEf-" - }, - "outputs": [], - "source": [ - "### Function minimization with autograd and gradient descent ###\n", - "\n", - "# Initialize a random value for our intial x\n", - "x = torch.randn(1)\n", - "print(f\"Initializing x={x.item()}\")\n", - "\n", - "learning_rate = 1e-2 # Learning rate\n", - "history = []\n", - "x_f = 4 # Target value\n", - "\n", - "\n", - "# We will run gradient descent for a number of iterations. At each iteration, we compute the loss,\n", - "# compute the derivative of the loss with respect to x, and perform the update.\n", - "for i in range(500):\n", - " x = torch.tensor([x], requires_grad=True)\n", - "\n", - " # TODO: Compute the loss as the square of the difference between x and x_f\n", - " loss = # TODO\n", - "\n", - " # Backpropagate through the loss to compute gradients\n", - " loss.backward()\n", - "\n", - " # Update x with gradient descent\n", - " x = x.item() - learning_rate * x.grad\n", - "\n", - " history.append(x.item())\n", - "\n", - "# Plot the evolution of x as we optimize toward x_f!\n", - "plt.plot(history)\n", - "plt.plot([0, 500], [x_f, x_f])\n", - "plt.legend(('Predicted', 'True'))\n", - "plt.xlabel('Iteration')\n", - "plt.ylabel('x value')\n", - "plt.show()\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "pC7czCwk3ceH" - }, - "source": [ - "Now, we have covered the fundamental concepts of PyTorch -- tensors, operations, neural networks, and automatic differentiation. Fire!!\n" - ] + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "WBk0ZDWY-ff8" + }, + "source": [ + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " Visit MIT Deep Learning\n", + " Run in Google Colab\n", + " View Source on GitHub
\n", + "\n", + "# Copyright Information\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "3eI6DUic-6jo" + }, + "outputs": [], + "source": [ + "# Copyright 2025 MIT Introduction to Deep Learning. All Rights Reserved.\n", + "#\n", + "# Licensed under the MIT License. You may not use this file except in compliance\n", + "# with the License. Use and/or modification of this code outside of MIT Introduction\n", + "# to Deep Learning must reference:\n", + "#\n", + "# © MIT Introduction to Deep Learning\n", + "# http://introtodeeplearning.com\n", + "#" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "57knM8jrYZ2t" + }, + "source": [ + "# Lab 1: Intro to PyTorch and Music Generation with RNNs\n", + "\n", + "In this lab, you'll get exposure to using PyTorch and learn how it can be used for deep learning. Go through the code and run each cell. Along the way, you'll encounter several ***TODO*** blocks -- follow the instructions to fill them out before running those cells and continuing.\n", + "\n", + "\n", + "# Part 1: Intro to PyTorch\n", + "\n", + "## 0.1 Install PyTorch\n", + "\n", + "[PyTorch](https://pytorch.org/) is a popular deep learning library known for its flexibility and ease of use. Here we'll learn how computations are represented and how to define a simple neural network in PyTorch. For all the labs in Introduction to Deep Learning 2025, there will be a PyTorch version available.\n", + "\n", + "Let's install PyTorch and a couple of dependencies." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "LkaimNJfYZ2w" + }, + "outputs": [], + "source": [ + "import torch\n", + "import torch.nn as nn\n", + "\n", + "# Download and import the MIT Introduction to Deep Learning package\n", + "!pip install mitdeeplearning --quiet\n", + "import mitdeeplearning as mdl\n", + "\n", + "import numpy as np\n", + "import matplotlib.pyplot as plt" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2QNMcdP4m3Vs" + }, + "source": [ + "## 1.1 What is PyTorch?\n", + "\n", + "PyTorch is a machine learning library, like TensorFlow. At its core, PyTorch provides an interface for creating and manipulating [tensors](https://pytorch.org/docs/stable/tensors.html), which are data structures that you can think of as multi-dimensional arrays. Tensors are represented as n-dimensional arrays of base datatypes such as a string or integer -- they provide a way to generalize vectors and matrices to higher dimensions. PyTorch provides the ability to perform computation on these tensors, define neural networks, and train them efficiently.\n", + "\n", + "The [```shape```](https://pytorch.org/docs/stable/generated/torch.Tensor.shape.html#torch.Tensor.shape) of a PyTorch tensor defines its number of dimensions and the size of each dimension. The `ndim` or [```dim```](https://pytorch.org/docs/stable/generated/torch.Tensor.dim.html#torch.Tensor.dim) of a PyTorch tensor provides the number of dimensions (n-dimensions) -- this is equivalent to the tensor's rank (as is used in TensorFlow), and you can also think of this as the tensor's order or degree.\n", + "\n", + "Let’s start by creating some tensors and inspecting their properties:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "tFxztZQInlAB" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "`integer` is a 0-d Tensor: 1234\n", + "`decimal` is a 0-d Tensor: 3.1415927410125732\n" + ] } - ], - "metadata": { - "accelerator": "GPU", - "colab": { - "collapsed_sections": [ - "WBk0ZDWY-ff8" - ], - "name": "PT_Part1_Intro.ipynb", - "provenance": [] - }, - "kernelspec": { - "display_name": "Python 3", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.6" - }, - "vscode": { - "interpreter": { - "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" - } + ], + "source": [ + "integer = torch.tensor(1234)\n", + "decimal = torch.tensor(3.14159265359)\n", + "\n", + "print(f\"`integer` is a {integer.ndim}-d Tensor: {integer}\")\n", + "print(f\"`decimal` is a {decimal.ndim}-d Tensor: {decimal}\")\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-dljcPUcoJZ6" + }, + "source": [ + "Vectors and lists can be used to create 1-d tensors:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "oaHXABe8oPcO" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "`fibonacci` is a 1-d Tensor with shape: torch.Size([6])\n", + "`count_to_100` is a 1-d Tensor with shape: torch.Size([100])\n" + ] + } + ], + "source": [ + "fibonacci = torch.tensor([1, 1, 2, 3, 5, 8])\n", + "count_to_100 = torch.tensor(range(100))\n", + "\n", + "print(f\"`fibonacci` is a {fibonacci.ndim}-d Tensor with shape: {fibonacci.shape}\")\n", + "print(f\"`count_to_100` is a {count_to_100.ndim}-d Tensor with shape: {count_to_100.shape}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gvffwkvtodLP" + }, + "source": [ + "Next, let’s create 2-d (i.e., matrices) and higher-rank tensors. In image processing and computer vision, we will use 4-d Tensors with dimensions corresponding to batch size, number of color channels, image height, and image width." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "tFeBBe1IouS3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "images is a 4-d Tensor with shape: torch.Size([10, 3, 256, 256])\n" + ] + } + ], + "source": [ + "### Defining higher-order Tensors ###\n", + "\n", + "'''TODO: Define a 2-d Tensor'''\n", + "matrix = torch.rand(2, 3) # TODO\n", + "\n", + "assert isinstance(matrix, torch.Tensor), \"matrix must be a torch Tensor object\"\n", + "assert matrix.ndim == 2\n", + "\n", + "'''TODO: Define a 4-d Tensor.'''\n", + "# Use torch.zeros to initialize a 4-d Tensor of zeros with size 10 x 3 x 256 x 256.\n", + "# You can think of this as 10 images where each image is RGB 256 x 256.\n", + "images = torch.zeros(10, 3, 256, 256) # TODO\n", + "\n", + "assert isinstance(images, torch.Tensor), \"images must be a torch Tensor object\"\n", + "assert images.ndim == 4, \"images must have 4 dimensions\"\n", + "assert images.shape == (10, 3, 256, 256), \"images is incorrect shape\"\n", + "print(f\"images is a {images.ndim}-d Tensor with shape: {images.shape}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wkaCDOGapMyl" + }, + "source": [ + "As you have seen, the `shape` of a tensor provides the number of elements in each tensor dimension. The `shape` is quite useful, and we'll use it often. You can also use slicing to access subtensors within a higher-rank tensor:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "FhaufyObuLEG" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "`row_vector`: tensor([0.0550, 0.5976, 0.5626])\n", + "`column_vector`: tensor([0.7651, 0.5976])\n", + "`scalar`: 0.7651234865188599\n" + ] + } + ], + "source": [ + "row_vector = matrix[1]\n", + "column_vector = matrix[:, 1]\n", + "scalar = matrix[0, 1]\n", + "\n", + "print(f\"`row_vector`: {row_vector}\")\n", + "print(f\"`column_vector`: {column_vector}\")\n", + "print(f\"`scalar`: {scalar}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "iD3VO-LZYZ2z" + }, + "source": [ + "## 1.2 Computations on Tensors\n", + "\n", + "A convenient way to think about and visualize computations in a machine learning framework like PyTorch is in terms of graphs. We can define this graph in terms of tensors, which hold data, and the mathematical operations that act on these tensors in some order. Let's look at a simple example, and define this computation using PyTorch:\n", + "\n", + "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/add-graph.png)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "X_YJrZsxYZ2z" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "c1: 76\n", + "c2: 76\n" + ] + } + ], + "source": [ + "# Create the nodes in the graph and initialize values\n", + "a = torch.tensor(15)\n", + "b = torch.tensor(61)\n", + "\n", + "# Add them!\n", + "c1 = torch.add(a, b)\n", + "c2 = a + b # PyTorch overrides the \"+\" operation so that it is able to act on Tensors\n", + "print(f\"c1: {c1}\")\n", + "print(f\"c2: {c2}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Mbfv_QOiYZ23" + }, + "source": [ + "Notice how we've created a computation graph consisting of PyTorch operations, and how the output is a tensor with value 76 -- we've just created a computation graph consisting of operations, and it's executed them and given us back the result.\n", + "\n", + "Now let's consider a slightly more complicated example:\n", + "\n", + "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/computation-graph.png)\n", + "\n", + "Here, we take two inputs, `a, b`, and compute an output `e`. Each node in the graph represents an operation that takes some input, does some computation, and passes its output to another node.\n", + "\n", + "Let's define a simple function in PyTorch to construct this computation function:" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "PJnfzpWyYZ23", + "scrolled": true + }, + "outputs": [], + "source": [ + "### Defining Tensor computations ###\n", + "\n", + "# Construct a simple computation function\n", + "def func(a, b):\n", + " '''TODO: Define the operation for c, d, e.'''\n", + " c = torch.add(a, b) # TODO\n", + " d = torch.subtract(b, 1)\n", + " e = torch.mul(c, d)\n", + " return e" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "AwrRfDMS2-oy" + }, + "source": [ + "Now, we can call this function to execute the computation graph given some inputs `a,b`:" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "pnwsf8w2uF7p" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "e_out: 6.0\n" + ] + } + ], + "source": [ + "# Consider example values for a,b\n", + "a, b = 1.5, 2.5\n", + "# Execute the computation\n", + "e_out = func(a, b)\n", + "print(f\"e_out: {e_out}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "6HqgUIUhYZ29" + }, + "source": [ + "Notice how our output is a tensor with value defined by the output of the computation, and that the output has no shape as it is a single scalar value." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1h4o9Bb0YZ29" + }, + "source": [ + "## 1.3 Neural networks in PyTorch\n", + "We can also define neural networks in PyTorch. PyTorch uses [``torch.nn.Module``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html), which serves as a base class for all neural network modules in PyTorch and thus provides a framework for building and training neural networks.\n", + "\n", + "Let's consider the example of a simple perceptron defined by just one dense (aka fully-connected or linear) layer: $ y = \\sigma(Wx + b) $, where $W$ represents a matrix of weights, $b$ is a bias, $x$ is the input, $\\sigma$ is the sigmoid activation function, and $y$ is the output.\n", + "\n", + "![alt text](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2025/lab1/img/computation-graph-2.png)\n", + "\n", + "We will use `torch.nn.Module` to define layers -- the building blocks of neural networks. Layers implement common neural networks operations. In PyTorch, when we implement a layer, we subclass `nn.Module` and define the parameters of the layer as attributes of our new class. We also define and override a function [``forward``](https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward), which will define the forward pass computation that is performed at every step. All classes subclassing `nn.Module` should override the `forward` function.\n", + "\n", + "Let's write a dense layer class to implement a perceptron defined above." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "HutbJk-1kHPh" + }, + "outputs": [], + "source": [ + "### Defining a dense layer ###\n", + "\n", + "# num_inputs: number of input nodes\n", + "# num_outputs: number of output nodes\n", + "# x: input to the layer\n", + "\n", + "class OurDenseLayer(torch.nn.Module):\n", + " def __init__(self, num_inputs, num_outputs):\n", + " super(OurDenseLayer, self).__init__()\n", + " # Define and initialize parameters: a weight matrix W and bias b\n", + " # Note that the parameter initialize is random!\n", + " self.W = torch.nn.Parameter(torch.randn(num_inputs, num_outputs))\n", + " self.bias = torch.nn.Parameter(torch.randn(num_outputs))\n", + "\n", + " def forward(self, x):\n", + " '''TODO: define the operation for z (hint: use torch.matmul).'''\n", + " z = torch.matmul(x, self.W) + self.bias # TODO\n", + "\n", + " '''TODO: define the operation for out (hint: use torch.sigmoid).'''\n", + " y = torch.sigmoid(z) # TODO\n", + " return y" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "GqeEbn959hV_" + }, + "source": [ + "Now, let's test the output of our layer." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "2yxjCPa69hV_" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input shape: torch.Size([1, 2])\n", + "output shape: torch.Size([1, 3])\n", + "output result: tensor([[0.1816, 0.6963, 0.5102]], grad_fn=)\n" + ] + } + ], + "source": [ + "# Define a layer and test the output!\n", + "num_inputs = 2\n", + "num_outputs = 3\n", + "layer = OurDenseLayer(num_inputs, num_outputs)\n", + "x_input = torch.tensor([[1, 2.]])\n", + "y = layer(x_input)\n", + "\n", + "print(f\"input shape: {x_input.shape}\")\n", + "print(f\"output shape: {y.shape}\")\n", + "print(f\"output result: {y}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Jt1FgM7qYZ3D" + }, + "source": [ + "Conveniently, PyTorch has defined a number of ```nn.Modules``` (or Layers) that are commonly used in neural networks, for example a [```nn.Linear```](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) or [`nn.Sigmoid`](https://pytorch.org/docs/stable/generated/torch.nn.Sigmoid.html) module.\n", + "\n", + "Now, instead of using a single ```Module``` to define our simple neural network, we'll use the [`nn.Sequential`](https://pytorch.org/docs/stable/generated/torch.nn.Sequential.html) module from PyTorch and a single [`nn.Linear` ](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer to define our network. With the `Sequential` API, you can readily create neural networks by stacking together layers like building blocks." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "7WXTpmoL6TDz" + }, + "outputs": [], + "source": [ + "### Defining a neural network using the PyTorch Sequential API ###\n", + "\n", + "# define the number of inputs and outputs\n", + "n_input_nodes = 2\n", + "n_output_nodes = 3\n", + "\n", + "# Define the model\n", + "'''TODO: Use the Sequential API to define a neural network with a\n", + " single linear (dense!) layer, followed by non-linearity to compute z'''\n", + "model = nn.Sequential(nn.Linear(n_input_nodes, n_output_nodes))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HDGcwYfUyR-U" + }, + "source": [ + "We've defined our model using the Sequential API. Now, we can test it out using an example input:" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "zKhp6XqCFFa0" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input shape: torch.Size([1, 2])\n", + "output shape: torch.Size([1, 3])\n", + "output result: tensor([[0.1816, 0.6963, 0.5102]], grad_fn=)\n" + ] + } + ], + "source": [ + "# Test the model with example input\n", + "x_input = torch.tensor([[1, 2.]])\n", + "model_output = model(x_input)\n", + "print(f\"input shape: {x_input.shape}\")\n", + "print(f\"output shape: {y.shape}\")\n", + "print(f\"output result: {y}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "596NvsOOtr9F" + }, + "source": [ + "With PyTorch, we can create more flexible models by subclassing [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html). The `nn.Module` class allows us to group layers together flexibly to define new architectures.\n", + "\n", + "As we saw earlier with `OurDenseLayer`, we can subclass `nn.Module` to create a class for our model, and then define the forward pass through the network using the `forward` function. Subclassing affords the flexibility to define custom layers, custom training loops, custom activation functions, and custom models. Let's define the same neural network model as above (i.e., Linear layer with an activation function after it), now using subclassing and using PyTorch's built in linear layer from `nn.Linear`." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "K4aCflPVyViD" + }, + "outputs": [], + "source": [ + "### Defining a model using subclassing ###\n", + "\n", + "class LinearWithSigmoidActivation(nn.Module):\n", + " def __init__(self, num_inputs, num_outputs):\n", + " super(LinearWithSigmoidActivation, self).__init__()\n", + " '''TODO: define a model with a single Linear layer and sigmoid activation.'''\n", + " self.linear = nn.Linear(num_inputs, num_outputs) # TODO\n", + " self.activation = nn.Sigmoid()\n", + "\n", + " def forward(self, inputs):\n", + " linear_output = self.linear(inputs)\n", + " output = self.activation(linear_output)\n", + " return output" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "goKCQ9dEGzRn" + }, + "source": [ + "Let's test out our new model, using an example input, setting `n_input_nodes=2` and `n_output_nodes=3` as before." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "V-eNhSyRG6hl" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input shape: torch.Size([1, 2])\n", + "output shape: torch.Size([1, 3])\n", + "output result: tensor([[0.1600, 0.7861, 0.5941]], grad_fn=)\n" + ] + } + ], + "source": [ + "n_input_nodes = 2\n", + "n_output_nodes = 3\n", + "model = LinearWithSigmoidActivation(n_input_nodes, n_output_nodes)\n", + "x_input = torch.tensor([[1, 2.]])\n", + "y = model(x_input)\n", + "print(f\"input shape: {x_input.shape}\")\n", + "print(f\"output shape: {y.shape}\")\n", + "print(f\"output result: {y}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HTIFMJLAzsyE" + }, + "source": [ + "Importantly, `nn.Module` affords us a lot of flexibility to define custom models. For example, we can use boolean arguments in the `forward` function to specify different network behaviors, for example different behaviors during training and inference. Let's suppose under some instances we want our network to simply output the input, without any perturbation. We define a boolean argument `isidentity` to control this behavior:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "P7jzGX5D1xT5" + }, + "outputs": [], + "source": [ + "### Custom behavior with subclassing nn.Module ###\n", + "\n", + "class LinearButSometimesIdentity(nn.Module):\n", + " def __init__(self, num_inputs, num_outputs):\n", + " super(LinearButSometimesIdentity, self).__init__()\n", + " self.linear = nn.Linear(num_inputs, num_outputs)\n", + "\n", + " '''TODO: Implement the behavior where the network outputs the input, unchanged,\n", + " under control of the isidentity argument.'''\n", + " def forward(self, inputs, isidentity=False): # TODO\n", + " if isidentity:\n", + " return inputs\n", + " else:\n", + " return self.linear(inputs)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Ku4rcCGx5T3y" + }, + "source": [ + "Let's test this behavior:" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "NzC0mgbk5dp2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "input: tensor([[1., 2.]])\n", + "Network linear output: tensor([[ 1.0846, -0.3116, 1.1746]], grad_fn=); network identity output: tensor([[1., 2.]])\n" + ] + } + ], + "source": [ + "# Test the IdentityModel\n", + "model = LinearButSometimesIdentity(num_inputs=2, num_outputs=3)\n", + "x_input = torch.tensor([[1, 2.]])\n", + "\n", + "'''TODO: pass the input into the model and call with and without the input identity option.'''\n", + "out_with_linear = model(x_input) # TODO\n", + "\n", + "out_with_identity = model(x_input, isidentity=True)\n", + "\n", + "print(f\"input: {x_input}\")\n", + "print(\"Network linear output: {}; network identity output: {}\".format(out_with_linear, out_with_identity))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7V1dEqdk6VI5" + }, + "source": [ + "Now that we have learned how to define layers and models in PyTorch using both the Sequential API and subclassing `nn.Module`, we're ready to turn our attention to how to actually implement network training with backpropagation." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "dQwDhKn8kbO2" + }, + "source": [ + "## 1.4 Automatic Differentiation in PyTorch\n", + "\n", + "In PyTorch, [`torch.autograd`](https://pytorch.org/docs/stable/autograd.html) is used for [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation), which is critical for training deep learning models with [backpropagation](https://en.wikipedia.org/wiki/Backpropagation).\n", + "\n", + "We will use the PyTorch [`.backward()`](https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html) method to trace operations for computing gradients. On a tensor, the [`requires_grad`](https://pytorch.org/docs/stable/generated/torch.Tensor.requires_grad_.html) attribute controls whether autograd should record operations on that tensor. When a forward pass is made through the network, PyTorch builds a computational graph dynamically; then, to compute the gradient, the `backward()` method is called to perform backpropagation.\n", + "\n", + "Let's compute the gradient of $ y = x^2 $:" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "id": "tdkqk8pw5yJM" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "dy_dx of y=x^2 at x=3.0 is: 6.0\n" + ] } + ], + "source": [ + "### Gradient computation ###\n", + "\n", + "# y = x^2\n", + "# Example: x = 3.0\n", + "x = torch.tensor(3.0, requires_grad=True)\n", + "y = x ** 2\n", + "y.backward() # Compute the gradient\n", + "\n", + "dy_dx = x.grad\n", + "print(\"dy_dx of y=x^2 at x=3.0 is:\", dy_dx.item())\n", + "assert dy_dx == 6.0" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JhU5metS5xF3" + }, + "source": [ + "In training neural networks, we use differentiation and stochastic gradient descent (SGD) to optimize a loss function. Now that we have a sense of how PyTorch's autograd can be used to compute and access derivatives, we will look at an example where we use automatic differentiation and SGD to find the minimum of $ L=(x-x_f)^2 $. Here $x_f$ is a variable for a desired value we are trying to optimize for; $L$ represents a loss that we are trying to minimize. While we can clearly solve this problem analytically ($ x_{min}=x_f $), considering how we can compute this using PyTorch's autograd sets us up nicely for future labs where we use gradient descent to optimize entire neural network losses." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "attributes": { + "classes": [ + "py" + ], + "id": "" + }, + "id": "7g1yWiSXqEf-" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Initializing x=1.4513012170791626\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjAsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvlHJYcgAAAAlwSFlzAAAPYQAAD2EBqD+naQAAQzJJREFUeJzt3Xl8VPW9//H3TPZ1QiArBBNkD4RdjVWhIlBRK1cft1i5ihteWrRYpFrwd8V7bW+wVS/gArgAglVqGxcqQsHK4oZsQcIqSyABEkJAspJMlvP7I2QwsmWZmTMzeT0fj3kkc+Y7M5+cRObtdzlfi2EYhgAAAHyE1ewCAAAAnIlwAwAAfArhBgAA+BTCDQAA8CmEGwAA4FMINwAAwKcQbgAAgE/xN7sAd6urq9OxY8cUEREhi8VidjkAAKAJDMNQaWmpEhMTZbVeum+mzYWbY8eOKSkpyewyAABAC+Tl5alTp06XbNPmwk1ERISk+pMTGRlpcjUAAKApSkpKlJSU5Pgcv5Q2F24ahqIiIyMJNwAAeJmmTClhQjEAAPAphBsAAOBTCDcAAMCnEG4AAIBPIdwAAACfQrgBAAA+hXADAAB8CuEGAAD4FMINAADwKYQbAADgUzwm3GRkZMhiseixxx67ZLt169Zp0KBBCg4OVpcuXTRv3jz3FAgAALyCR4SbTZs26bXXXlNaWtol2+Xk5Gj06NG6/vrrlZWVpenTp+s3v/mNMjMz3VQpAADwdKZvnFlWVqZx48bp9ddf1x/+8IdLtp03b546d+6sWbNmSZJ69eqlzZs36/nnn9edd97phmovwTCk6gpzawAAXJZhGGe/nr3/4+OO+w2PGz947o9fq3Gbi73mhV5Xl3nuBWu6+I91Xq2XaNSah5vQQIqLDJYCQqUmbHLpCqaHm0mTJumWW27RTTfddNlw8/XXX2vkyJGNjo0aNUpvvvmmqqurFRAQcN5zqqqqVFVV5bhfUlLinMJ/rLpC+t9E17w2AMBpLD/6CheZfkwKDDPlrU0dllq6dKm2bt2qjIyMJrUvKChQXFxco2NxcXGqqalRUVHRBZ+TkZEhm83muCUlJbW6bgAA4LlM67nJy8vT5MmTtWrVKgUHBzf5eZYfdXE1dNn9+HiDadOmacqUKY77JSUlrgk4AaH1KRVAixmGoQp7rcqqalRSWaOyymqVVNaotLJapZU1Kq2qUemZs/erzh2vsNeqwl6jM/ZaVdhrdaa61uwfRVaL5O9nVYDVIn8/q/ytVgX4WeTvZ1HA2fv+fvWPBVotZ+/rbNv6dlaLRX5Wi/wsFlmt9fetVov8LHIc87c2HDvXxs9af6v/Xj94nfp/K3/4mg2PWy0WWSz131tU385ytn39/frH9IM2Dc+xSLLoB8+31P/8kkXWRq/R8Jr17a3Ws18tDaMX59pbLede02I5N7rh6HX5wT/5FjV+0HLugUaP//BT4tzrWH50/yKv2ej9mvbcC30sXazNxV6zURuThnhaLCDUtLc2Ldxs2bJFhYWFGjRokONYbW2t1q9fr5dffllVVVXy8/Nr9Jz4+HgVFBQ0OlZYWCh/f3+1b9/+gu8TFBSkoKAg5/8AP2axmNb9Bniqmto6nSy360Rplb6vsOtUuV3fl9t1qqL67Nez98/evq+wq7q2CQP6F2U9ezs3RB0a6Hf25t/o+5BAP4UE+Ck4wKog/3Nfg/ytCrrY9/5+Z++fOxbob1WAX0Nwqf8aYLXKavWyDyLAh5gWboYPH67s7OxGx+6//3717NlTTz755HnBRpLS09P1j3/8o9GxVatWafDgwRecbwPANQzDUFGZXcdOn9HxkkoVllapsOFraZXj2MmyKtW1IKv4Wy2KCPZXZEhA/dfg+q8RwQGO78895q/woACFBPopLMhPoQH+ju+D/f0IGUAbZFq4iYiIUJ8+fRodCwsLU/v27R3Hp02bpqNHj2rx4sWSpIkTJ+rll1/WlClTNGHCBH399dd688039e6777q9fsCX1dUZOl5aqaPfn9GR78/o6OkzOvJ9Rf33Z+9X1dQ16bWsFik6LEgdwgPVLjRQ0WGBahcWoOjQQLULO3v/7PHosEBFhQYoJMDP+7rgAXgM01dLXUp+fr5yc3Md91NSUvTJJ5/ot7/9rV555RUlJiZqzpw55i8DB7yQYRg6VW7XoZPlOniiXDlFjW+XCy8WixQbEaS4yGDFRgQrNjJIcWe/njsepPbhQfKj9wSAG1kM48er9n1bSUmJbDabiouLFRkZaXY5gFsUV1RrT0GJ9h4v1Z6CUu0tKNW+46Uqqay56HP8rRYlRAWrY1SIOkaFqlO7EHVsF6JOUfVfE2whCvT3iOuAAmgDmvP57dE9NwCaxzAM5RdXavuR0/r2SLF2HSvR3oJSFZRUXvQ5HaNClNIh7NwtJkxdOoSpY1SI/P0ILwC8D+EG8GLfl9u17chpbc8rdgSaorKqC7btGBWiHvER6hEfoZ7xEeoeF6Hk9mEKCTx/8j4AeDPCDeBF8ovPaGPOKcdtX2HZeW38rBZ1j4tQv0429eloqw8y8RGKDGZFIYC2gXADeLCisip9vu+Evth3UhsPnVTeqTPntenSIUxpnWxK6xSlfkk29U6w0RsDoE0j3AAexF5Tp82HT+nzfUVa/90J7TzWeC80q0Xq09Gmq5KjNSQlWkOSoxUdFmhStQDgmQg3gMlOV9j1r92FWrWrQJ/vK1KFvfHWAamJkbq+W4yuvbK9Bl7RTuFB/GcLAJfCv5KACQqKK7VqV4H+ubNAGw6eUu0PLuPbITxQ13eL0Q3dO+i6rjGKiXDD9iEA4EMIN4CbFJ+p1orsfH247ai+yTmlH15hqmd8hEb2jtPI1Hj1TohkywAAaAXCDeBCVTW1WrOnUB9mHdNnewplrz131d+BnaP0sz7xGtk7Xskd2HQVAJyFcAO4wKGicr27MVd/23JEp8rtjuPd48I1ZkBH/bxfojq1CzWxQgDwXYQbwEmqa+u0etdxvfNNrr7YX+Q4HhcZpDH9O+r2/h3VKyGCDSEBwMUIN0Arna6w6y/f5Oqtrw6psLT+6sAWizS0e4zuvqqzbuwZyzYGAOBGhBughQ6fLNeCL3L03uYjOlNdv3w7JiJIYwcnaeyQJCVFM+wEAGYg3ADNtL+wVHP+tV8fbz+mhhXcvRIiNeH6FN2alshO2QBgMsIN0ET7jpdqzmf1oaZhGffQ7jF6+IYuuvbK9sylAQAPQbgBLiP3ZIX+vGpvo1AzKjVOvxneTamJNnOLAwCch3ADXMT35Xa99Nl+LdlwSNW19amGUAMAno9wA/xIVU2tFn15SK+s2a+SyhpJ0vXdOujJn/VUn46EGgDwdIQb4Ac+33dCT3+0UzlF5ZLqt0WYPrqXbugeY3JlAICmItwAqt/I8tnlu7R8e74kKTYiSL8b1UN3DOwkP/Z5AgCvQrhBm1ZbZ2jRV4f04qq9KrfXymqR7rs2Rb8d0U0RwQFmlwcAaAHCDdqsnKJy/e5v32rz4e8l1W9k+eyYPkwWBgAvR7hBm1NXZ+itrw/puZV7VFldp/Agf00f3Ut3DUmSlSEoAPB6hBu0KXmnKjT1b9/qm5xTkqSfdG2v5+5MY4duAPAhhBu0Gf/cWaDf/e1blVTWKDTQT9NG99J/XN2ZKwsDgI8h3MDnVdXUKuOTPVr01SFJ0oDOUZo9doA6t6e3BgB8EeEGPu3wyXI98k6Wso8WS5IevqGLfjeqhwL82NwSAHwV4QY+a83eQv3m3SyVVtYoKjRAL/6in27sGWd2WQAAFyPcwOcYhqHX1h/UzJV7ZBjSoCva6aVfDlBiVIjZpQEA3IBwA59SWV2r32du14fbjkmSfnlVkv77530U6M8wFAC0FYQb+IxT5XY9+NYmZeWelp/Vohm39dY911zBaigAaGMIN/AJh0+W676Fm5RTVC5bSIDm/sdAXXtlB7PLAgCYgHADr/dt3mk9+NYmFZXZ1TEqRG89cJW6xoabXRYAwCSEG3i1z/ed0MOLt+hMda1SEyO18P4hio0INrssAICJCDfwWp/uOq5f/2Wr7LV1ur5bB839j0EKD+JPGgDaOj4J4JWWb8/X5KVZqqkz9LPUeM355QBWRAEAJBFu4IXe33pEU//2reoM6fb+iXrh3/vJnysOAwDOItzAq3y07age/9u3MgzpriFJ+uO/9ZWflaXeAIBzCDfwGit3FGjKe/XB5u6rO+sPt/eRlWADAPgR+vLhFdbsLdSj725VbZ2hOwZ2JNgAAC6KcAOPt+HgSU1cskXVtYZuSUvQn+5MI9gAAC6KcAOPtqegRBMWb1ZVTZ1u6hWrWWP7M3kYAHBJfErAY+UXn9F9CzaptLJGQ5Lb6eW7ByqAYAMAuAw+KeCRis9U674Fm1RQUqmuseF6/d7BCg7wM7ssAIAXINzA49hr6vSfSzZr7/FSxUYEadH9QxQVGmh2WQAAL0G4gUcxDEMzlu3QhoOnFB7kr4X3D1GndqFmlwUA8CKEG3iUJRsO692NebJYpJd+OUCpiTazSwIAeBnCDTzGV/uL9N//2CVJevJnPfXTnrEmVwQA8EaEG3iE3JMV+vU79RfpG9M/Uf95QxezSwIAeCnCDUxXWV2r/3x7i05XVCutk00z70yTxcJF+gAALUO4gen++x+7tDu/RO3DAjX/nkEs+QYAtArhBqb6MOuo3t2YK4tFmnVXfyXYQswuCQDg5Qg3MM3+wjJN/yBbkvTojd10fbcYkysCAPgCwg1MccZeq0l/2aoKe62uvbK9Jg/vZnZJAAAfQbiBKTJW7Nbe46WKiQjSrLv6y49dvgEATkK4gdut3VuoxV8fliS9+It+io0INrkiAIAvIdzArb4vt+uJv2+XJN13bTLzbAAATmdquJk7d67S0tIUGRmpyMhIpaena8WKFRdtv3btWlkslvNue/bscWPVaCnDMDT9g2wVllbpypgw/f7mnmaXBADwQf5mvnmnTp00c+ZMde3aVZL01ltv6fbbb1dWVpZSU1Mv+ry9e/cqMjLScT8mhv/79wYfZB3Vih0F8rdaNPuuAVzPBgDgEqaGm9tuu63R/T/+8Y+aO3euNmzYcMlwExsbq6ioqCa9R1VVlaqqqhz3S0pKWlQrWud4SaVmLNspSfrtiO7q05ENMQEAruExc25qa2u1dOlSlZeXKz09/ZJtBwwYoISEBA0fPlxr1qy5ZNuMjAzZbDbHLSkpyZllo4me/miHSitr1K+TjX2jAAAuZXq4yc7OVnh4uIKCgjRx4kR98MEH6t279wXbJiQk6LXXXlNmZqbef/999ejRQ8OHD9f69esv+vrTpk1TcXGx45aXl+eqHwUXsSI7X//ceVz+Votm3pkmfz/T/+wAAD7MYhiGYWYBdrtdubm5On36tDIzM/XGG29o3bp1Fw04P3bbbbfJYrFo2bJlTWpfUlIim82m4uLiRvN24BrFFdW66f/W6URplR69saseH9nD7JIAAF6oOZ/fpv8vdGBgoLp27arBgwcrIyND/fr10+zZs5v8/GuuuUb79u1zYYVojYwVu3WitEpdYsI06addzS4HANAGmB5ufswwjEYTgC8nKytLCQkJLqwILfX1gZNauql+GHDmHWmsjgIAuIWpq6WmT5+um2++WUlJSSotLdXSpUu1du1arVy5UlL9fJmjR49q8eLFkqRZs2YpOTlZqampstvtevvtt5WZmanMzEwzfwxcQHVtnf7rox2SpHFXd9ZVKdEmVwQAaCtMDTfHjx/XPffco/z8fNlsNqWlpWnlypUaMWKEJCk/P1+5ubmO9na7XVOnTtXRo0cVEhKi1NRULV++XKNHjzbrR8BFvPXVIe0vLFP7sEA98TMu1gcAcB/TJxS7GxOKXa+wpFI3vrBOZVU1eu7Ovho7pLPZJQEAvJxXTSiG75m5Yo/KquqvafPvg7iuEADAvQg3cKrNh07p/ayjslik/7m9j6xWi9klAQDaGMINnKa2ztDTH9VvsfCLQUnqlxRlbkEAgDaJcAOnydx6RLvySxQR7K/f/YyL9QEAzEG4gVOcsdfqxVXfSZJ+c2M3dQgPMrkiAEBbRbiBUyz4MkcFJZXqGBWie9KvMLscAEAbRrhBq50qt2ve2gOSpKmjunMlYgCAqQg3aLWXPtun0qoa9U6I1O39OppdDgCgjSPcoFUOnyzX2xsOS5Kmj+7F0m8AgOkIN2iV51d9p+paQzd0j9F13TqYXQ4AAIQbtNx3x0v18fZjkqQnWfoNAPAQhBu02OxP98kwpJv7xCs10WZ2OQAASCLcoIX2FJRoeXa+JGnyTd1MrgYAgHMIN2iR2Z/ukyTd0jdBPePZXR0A4DkIN2i2XcdKtGJHgSwWem0AAJ6HcINmm/2v+m0WbumboO5xESZXAwBAY4QbNMvOY8X6587j9b02w+m1AQB4HsINmmXu2W0Wbk1LVDd6bQAAHohwgyY7fLJcn5xdIfWroVeaXA0AABdGuEGTvf75QdUZ0tDuMeqdyAopAIBnItygSYrKqvS3zUckSRPptQEAeDDCDZpk0ZeHVFVTp35JUbqmS7TZ5QAAcFGEG1xWWVWNFn99SJL0q6FdZLGw8zcAwHMRbnBZSzfmqqSyRl06hGlE73izywEA4JIIN7ik6to6vflFjiTp4Ru6yM9Krw0AwLMRbnBJ/9xZoPziSnUID9K/DexodjkAAFwW4QaXtOjLQ5Kku6/urCB/P3OLAQCgCQg3uKgdR4u1+fD38rda9B9Xdza7HAAAmoRwg4ta9NUhSdItaQmKjQw2txgAAJqIcIMLOllWpWXfHpMkjb822dxiAABoBsINLmjppjzZa+rUr5NNA5KizC4HAIAmI9zgPNW1dVry9WFJ0n0/SeaifQAAr0K4wXn+ubNABSX1y79H900wuxwAAJqFcIPzvL2hvtfm7quSWP4NAPA6hBs0klNUrg0HT8lqke66iuXfAADvQ7hBI0s35UqShnaPUWJUiMnVAADQfIQbONhr6vT3zUck0WsDAPBehBs4fLr7uE6W2xUbEaQbe8aaXQ4AAC1CuIHDuxvrh6T+fXAnBfjxpwEA8E58gkGSlHeqQp/vK5IkjR3MkBQAwHsRbiBJ+uumPEnSdV07qHP7UJOrAQCg5Qg3UE1tnd7bXB9ufslEYgCAlyPcQOv3nVBhaZWiwwI1onec2eUAANAqhBsoc+tRSdLt/RMV6M+fBADAu/FJ1saVVFZr9a7jkqQ7BnQyuRoAAFqPcNPGrcjOl72mTt1iw9WnY6TZ5QAA0GqEmzbu/bNDUv82sKMsFovJ1QAA0HqEmzYs71SFvsk5JYtFGtO/o9nlAADgFISbNuyjbfW9Nuld2rNJJgDAZxBu2ijDMPR+1tkhqQH02gAAfAfhpo3afqRYB0+UKzjAqpv7JphdDgAATkO4aaPe33pEkjQqNV7hQf4mVwMAgPMQbtqgmto6Lc/OlySNYUgKAOBjCDdt0MacUyoqs6tdaICu69rB7HIAAHAqwk0b9PHZXpuf9YlXgB9/AgAA38InWxtTU1unlTsKJEm39E00uRoAAJzP1HAzd+5cpaWlKTIyUpGRkUpPT9eKFSsu+Zx169Zp0KBBCg4OVpcuXTRv3jw3Vesbvj54UqfK7YoOC9Q1XaLNLgcAAKczNdx06tRJM2fO1ObNm7V582bdeOONuv3227Vz584Lts/JydHo0aN1/fXXKysrS9OnT9dvfvMbZWZmurly77V8+7khKX+GpAAAPshiGIZhdhE/FB0drT//+c968MEHz3vsySef1LJly7R7927HsYkTJ+rbb7/V119/fcHXq6qqUlVVleN+SUmJkpKSVFxcrMjItrVRZHVtnYb88VOdrqjWOw9drWuZTAwA8BIlJSWy2WxN+vz2mP91r62t1dKlS1VeXq709PQLtvn66681cuTIRsdGjRqlzZs3q7q6+oLPycjIkM1mc9ySkpKcXru3+OrASZ2uqFaH8EBdlcKQFADAN5kebrKzsxUeHq6goCBNnDhRH3zwgXr37n3BtgUFBYqLi2t0LC4uTjU1NSoqKrrgc6ZNm6bi4mLHLS8vz+k/g7dYvv2YJIakAAC+zfRL0/bo0UPbtm3T6dOnlZmZqfHjx2vdunUXDTgWi6XR/YZRtR8fbxAUFKSgoCDnFu2F7DWskgIAtA2mh5vAwEB17dpVkjR48GBt2rRJs2fP1vz5889rGx8fr4KCgkbHCgsL5e/vr/bt27ulXm/15f4ilVTWKCYiiCEpAIBP87ixCcMwGk0A/qH09HStXr260bFVq1Zp8ODBCggIcEd5XuufO+tD4ajUOPlZL9zLBQCALzA13EyfPl2ff/65Dh06pOzsbD311FNau3atxo0bJ6l+vsy9997raD9x4kQdPnxYU6ZM0e7du7VgwQK9+eabmjp1qlk/gleorTP06e7jkuo3ygQAwJeZOix1/Phx3XPPPcrPz5fNZlNaWppWrlypESNGSJLy8/OVm5vraJ+SkqJPPvlEv/3tb/XKK68oMTFRc+bM0Z133mnWj+AVtuV9r6IyuyKC/HV1CsN3AADf5nHXuXG15qyT9xUZK3Zr/rqD+nm/RM355QCzywEAoNm88jo3cJ3VO+uHpEb0jrtMSwAAvB/hxsftLyzTwaJyBfhZNKxHjNnlAADgcoQbH7d6V32vTfqVHRQRzIoyAIDva1G4WbJkiX7yk58oMTFRhw8fliTNmjVLH330kVOLQ+ut2lW/BHwkQ1IAgDai2eFm7ty5mjJlikaPHq3Tp0+rtrZWkhQVFaVZs2Y5uz60QmFJpbblnZbEfBsAQNvR7HDz0ksv6fXXX9dTTz0lPz8/x/HBgwcrOzvbqcWhdT7dXSjDkPp1sikuMtjscgAAcItmh5ucnBwNGHD+cuKgoCCVl5c7pSg4x+qGISku3AcAaEOaHW5SUlK0bdu2846vWLHioptdwv0q7DX68sBJSQxJAQDalmZfofh3v/udJk2apMrKShmGoY0bN+rdd99VRkaG3njjDVfUiBb4av9J2Wvq1KldiLrFhptdDgAAbtPscHP//ferpqZGTzzxhCoqKnT33XerY8eOmj17tu666y5X1IgWWPtdoSTppz1iZbGwUSYAoO1o0d5SEyZM0IQJE1RUVKS6ujrFxsY6uy60gmEYWrPnhCRx4T4AQJvTqo0zO3To4Kw64EQHTpTp6OkzCvSzKv1KNsoEALQtzQ43KSkplxzmOHjwYKsKQuut3Vvfa3N1l2iFBpq68TsAAG7X7E++xx57rNH96upqZWVlaeXKlfrd737nrLrQCg3hZlgPhgsBAG1Ps8PN5MmTL3j8lVde0ebNm1tdEFqnvKpGG3NOSWK+DQCgbXLaxpk333yzMjMznfVyaKGvDpyUvbZOSdEh6tIhzOxyAABwO6eFm7///e+Kjo521suhhdbuZQk4AKBta/aw1IABAxp9aBqGoYKCAp04cUKvvvqqU4tD8xiG8YP5NgxJAQDapmaHmzFjxjS6b7VaFRMTo2HDhqlnz57OqgstsL/w7BJwf6vSu7BMHwDQNjU73MyYMcMVdcAJGnptrunSXiGBfpdpDQCAb2pSuCkpKWnyC0ZGRra4GLTO+n314WZod4akAABtV5PCTVRU1GUnpxqGIYvFotraWqcUhuapqqnVpkP1S8Cv78aQFACg7WpSuFmzZo2r60ArbT18WpXVdYqJCGIXcABAm9akcDN06FBX14FW+nJ/kSTpJ1e2Zwk4AKBNa/HGQxUVFcrNzZXdbm90PC0trdVFofm+aAg3XRmSAgC0bc0ONydOnND999+vFStWXPBx5ty4X0lltbYfOS2JcAMAQLOvUPzYY4/p+++/14YNGxQSEqKVK1fqrbfeUrdu3bRs2TJX1IjL2HDgpOoMqUuHMCVGhZhdDgAApmp2z81nn32mjz76SEOGDJHVatUVV1yhESNGKDIyUhkZGbrllltcUScu4asDJyVJ13Ztb3IlAACYr9k9N+Xl5YqNjZUkRUdH68SJ+mur9O3bV1u3bnVudWiShvk21zEkBQBA88NNjx49tHfvXklS//79NX/+fB09elTz5s1TQkKC0wvEpR0vqdT+wjJZLPVXJgYAoK1r9rDUY489pvz8fEn1WzGMGjVKf/nLXxQYGKhFixY5uz5cRsMS8L4dbYoKDTS5GgAAzNfscDNu3DjH9wMGDNChQ4e0Z88ede7cWR06MCzibiwBBwCgsWYPS61bt67R/dDQUA0cOJBgYwLDMPTV/vrJxD+5kvMPAIDUgnAzYsQIde7cWb///e+1Y8cOV9SEJjpwolwFJZUK9LdqcHI7s8sBAMAjNDvcHDt2TE888YQ+//xzpaWlKS0tTX/605905MgRV9SHS/gmp77XZmDnKAUH+JlcDQAAnqHZ4aZDhw565JFH9OWXX+rAgQMaO3asFi9erOTkZN14442uqBEXsTGnfhfwq1NYJQUAQINmh5sfSklJ0e9//3vNnDlTffv2PW8+DlzHMAx9c7Ah3ESbXA0AAJ6jxeHmyy+/1K9//WslJCTo7rvvVmpqqj7++GNn1oZLyDt1RgUllQrws2hAZ+bbAADQoNlLwadPn653331Xx44d00033aRZs2ZpzJgxCg0NdUV9uIgNZ+fbpHWKUkgg820AAGjQ7HCzdu1aTZ06VWPHjmX5t4ka5ttcxZAUAACNNDvcfPXVV66oA810bjIx4QYAgB9q1YRimCO/+IxyT1XIapEGXcF8GwAAfohw44Uaem1SE22KCA4wuRoAADwL4cYLfcOQFAAAF0W48ULfHKxfKcVkYgAAztfscPPpp59e9LH58+e3qhhcXlFZlQ6cKJckDUkm3AAA8GPNDje33HKLHn/8cdntdsexEydO6LbbbtO0adOcWhzOt+nskFTP+Ai1Cws0uRoAADxPs8PN+vXr9Y9//ENDhgzRzp07tXz5cvXp00dlZWX69ttvXVEjfuAbrm8DAMAlNTvcXH311crKylJaWpoGDRqkf/u3f9Pjjz+uzz77TElJSa6oET9AuAEA4NJaNKF479692rRpkzp16iR/f3/t2bNHFRUVzq4NP1JWVaO9BSWSmG8DAMDFNDvczJw5U+np6RoxYoR27NihTZs2OXpyvv76a1fUiLO+zTutOkPqGBWiuMhgs8sBAMAjNTvczJ49Wx9++KFeeuklBQcHKzU1VRs3btQdd9yhYcOGuaBENNhy+HtJ0kCuSgwAwEU1e2+p7Ozs8zbMDAgI0J///GfdeuutTisM59uaWx9uBnWOMrcQAAA8WLN7bi61E/jQoUNbVQwurq7O0FZ6bgAAuCyuUOwlDhaVqaSyRsEBVvVKiDS7HAAAPBbhxks0zLdJ6xSlAD9+bQAAXIypn5IZGRkaMmSIIiIiFBsbqzFjxmjv3r2XfM7atWtlsVjOu+3Zs8dNVZtj6+HTkqRBDEkBAHBJpoabdevWadKkSdqwYYNWr16tmpoajRw5UuXl5Zd97t69e5Wfn++4devWzQ0Vm2fL2cnEAzsTbgAAuJRmr5ZyppUrVza6v3DhQsXGxmrLli264YYbLvnc2NhYRUVFubA6z1FcUa39hWWSpIGslAIA4JI8avJGcXGxJCk6+vJX3x0wYIASEhI0fPhwrVmz5qLtqqqqVFJS0ujmbbLy6nttktuHqn14kMnVAADg2Twm3BiGoSlTpui6665Tnz59LtouISFBr732mjIzM/X++++rR48eGj58uNavX3/B9hkZGbLZbI6bN+5/xRJwAACazmIYhmF2EZI0adIkLV++XF988YU6derUrOfedtttslgsWrZs2XmPVVVVqaqqynG/pKRESUlJKi4uVmSkdyyp/o83vtEX+4v0hzF99B/XXGF2OQAAuF1JSYlsNluTPr89oufm0Ucf1bJly7RmzZpmBxtJuuaaa7Rv374LPhYUFKTIyMhGN29SW2coq+HKxPTcAABwWaZOKDYMQ48++qg++OADrV27VikpKS16naysLCUkJDi5Os/w3fFSldtrFR7kr+5xEWaXAwCAxzM13EyaNEnvvPOOPvroI0VERKigoECSZLPZFBISIkmaNm2ajh49qsWLF0uSZs2apeTkZKWmpsput+vtt99WZmamMjMzTfs5XKlhP6n+SVHys1pMrgYAAM9nariZO3euJJ23m/jChQt13333SZLy8/OVm5vreMxut2vq1Kk6evSoQkJClJqaquXLl2v06NHuKtutvs07Lak+3AAAgMvzmAnF7tKcCUme4Gez1mtPQaleu2eQRqbGm10OAACm8LoJxbiwCnuNvjteKknqR88NAABNQrjxYDuPlajOkOIigxQXGWx2OQAAeAXCjQdrmG+T1inK1DoAAPAmhBsPtv1I/XYU/TrZTK4EAADvQbjxYNuPnJZEzw0AAM1BuPFQpyvsOnSyQpKURs8NAABNRrjxUA1DUle0D1VUaKDJ1QAA4D0INx6KISkAAFqGcOOhvmUyMQAALUK48VD03AAA0DKEGw90vKRSx0uqZLVIfTp6/hYRAAB4EsKNB2q4eF+32AiFBpq6tykAAF6HcOOBGlZKsQQcAIDmI9x4oG8b5tuwWSYAAM1GuPEwhmEo+ygrpQAAaCnCjYc58v0Zna6oVoCfRT3jmUwMAEBzEW48zM5j9b023eMiFOjPrwcAgObi09PD7DxWIklKTaTXBgCAliDceJhz4Yb5NgAAtAThxsM0DEvRcwMAQMsQbjxIUVmVjpdUyWKReiYQbgAAaAnCjQfZdXZIKrl9mMKDuDIxAAAtQbjxIA3zbXozJAUAQIsRbjwI820AAGg9wo0H2cVKKQAAWo1w4yHKq2qUc7JcEj03AAC0BuHGQ+zOL5FhSHGRQeoQHmR2OQAAeC3CjYfg4n0AADgH4cZDMJkYAADnINx4CPaUAgDAOQg3HsBeU6fvjpdKYlgKAIDWItx4gH2FpaquNRQZ7K9O7ULMLgcAAK9GuPEAP7wyscViMbkaAAC8G+HGA3DxPgAAnIdw4wF25deHm17sBA4AQKsRbkxmGIb2FtRPJu4ZH2FyNQAAeD/CjcmOl1Sp+Ey1/KwWdY0NN7scAAC8HuHGZHsK6oekktuHKjjAz+RqAADwfoQbk50bkmK+DQAAzkC4MVlDuOnBfBsAAJyCcGOyPUwmBgDAqQg3JqqprdP+wjJJDEsBAOAshBsT5RSVy15bp9BAP7ZdAADASQg3JmoYkuoeFyGrlW0XAABwBsKNibh4HwAAzke4MdEeVkoBAOB0hBsT7T1efwE/wg0AAM5DuDFJWVWN8k6dkcRKKQAAnIlwY5LvjtcPScVGBCk6LNDkagAA8B2EG5NwZWIAAFyDcGMSVkoBAOAahBuTNOwG3oP5NgAAOBXhxgSGYbCnFAAALkK4MUFhaZVOV1TLapG6xoabXQ4AAD6FcGOChvk2yR3CFBzgZ3I1AAD4FsKNCfad3Qm8eyxDUgAAOJup4SYjI0NDhgxRRESEYmNjNWbMGO3du/eyz1u3bp0GDRqk4OBgdenSRfPmzXNDtc6z/2y46RbHkBQAAM5marhZt26dJk2apA0bNmj16tWqqanRyJEjVV5eftHn5OTkaPTo0br++uuVlZWl6dOn6ze/+Y0yMzPdWHnr7C+sH5Zivg0AAM7nb+abr1y5stH9hQsXKjY2Vlu2bNENN9xwwefMmzdPnTt31qxZsyRJvXr10ubNm/X888/rzjvvdHXJrWYYhmNYinADAIDzedScm+LiYklSdHT0Rdt8/fXXGjlyZKNjo0aN0ubNm1VdXX1e+6qqKpWUlDS6melkuV2nK6plsUhXxhBuAABwNo8JN4ZhaMqUKbruuuvUp0+fi7YrKChQXFxco2NxcXGqqalRUVHRee0zMjJks9kct6SkJKfX3hz7jtf32iS1C2WlFAAALuAx4eaRRx7R9u3b9e677162rcViaXTfMIwLHpekadOmqbi42HHLy8tzTsEt1DDfphtDUgAAuISpc24aPProo1q2bJnWr1+vTp06XbJtfHy8CgoKGh0rLCyUv7+/2rdvf177oKAgBQUFObXe1tjPfBsAAFzK1J4bwzD0yCOP6P3339dnn32mlJSUyz4nPT1dq1evbnRs1apVGjx4sAICAlxVqtMwmRgAANcyNdxMmjRJb7/9tt555x1FRESooKBABQUFOnPmjKPNtGnTdO+99zruT5w4UYcPH9aUKVO0e/duLViwQG+++aamTp1qxo/QbOeuccMF/AAAcAVTw83cuXNVXFysYcOGKSEhwXH761//6miTn5+v3Nxcx/2UlBR98sknWrt2rfr3769nn31Wc+bM8Ypl4MUV1SosrZIkXRkTZnI1AAD4JlPn3DRMBL6URYsWnXds6NCh2rp1qwsqcq39J+onEyfYghUR7PlDaAAAeCOPWS3VFjCZGAAA1yPcuFHDNW4INwAAuA7hxo0aVkp1YzdwAABchnDjRgxLAQDgeoQbNymvqtHR0/VL3Lk6MQAArkO4cZODJ8olSR3CA9UuLNDkagAA8F2EGzfZd3ZPKXYCBwDAtQg3buKYTBxHuAEAwJUIN27imExMzw0AAC5FuHGTcyulWAYOAIArEW7cwF5Tp9xTFZJYBg4AgKsRbtwg7/sK1dYZCg30U1xkkNnlAADg0wg3bpBzdhl4SocwWSwWk6sBAMC3EW7c4GBR/XyblA5hJlcCAIDvI9y4QU5Rfc9NF8INAAAuR7hxgwNnh6W6sAwcAACXI9y4QUPPDcNSAAC4HuHGxUorq3WitEqSlBJDuAEAwNUINy7W0GvTITxIkcEBJlcDAIDvI9y4GJOJAQBwL8KNix10TCYm3AAA4A6EGxc7yGRiAADcinDjYjlcwA8AALfyN7sAX2YYhmPrBa5xAwCuV1tbq+rqarPLQAsFBgbKam19vwvhxoUKS6tUbq+Vn9WiztGhZpcDAD7LMAwVFBTo9OnTZpeCVrBarUpJSVFgYGCrXodw40INk4mT2oUo0J8RQABwlYZgExsbq9DQUDYp9kJ1dXU6duyY8vPz1blz51b9Dgk3LsSGmQDgerW1tY5g0759e7PLQSvExMTo2LFjqqmpUUBAy68NR3eCCzXMt0npwHwbAHCVhjk2oaEM/3u7huGo2traVr0O4caFGpaBc40bAHA9hqK8n7N+h4QbF+LqxAAAuB/hxkWqa+uUe6pCEhtmAgDM9cwzz6h///6O+/fdd5/GjBnj9joOHToki8Wibdu2ufR9CDcuknuqQrV1hkIC/BQfGWx2OQAAD3TffffJYrHIYrEoICBAXbp00dSpU1VeXu7S9509e7YWLVrUpLbuCiTOxGopFzk3mTiMcWAAwEX97Gc/08KFC1VdXa3PP/9cDz30kMrLyzV37txG7aqrq1u1guiHbDabU17HU9Fz4yIN820YkgIA9zMMQxX2GrffDMNodq1BQUGKj49XUlKS7r77bo0bN04ffvihYyhpwYIF6tKli4KCgmQYhoqLi/Xwww8rNjZWkZGRuvHGG/Xtt982es2ZM2cqLi5OERERevDBB1VZWdno8R8PS9XV1em5555T165dFRQUpM6dO+uPf/yjJCklJUWSNGDAAFksFg0bNszxvIULF6pXr14KDg5Wz5499eqrrzZ6n40bN2rAgAEKDg7W4MGDlZWV1ezz0xL03LhIzsmz4aY94QYA3O1Mda16P/1Pt7/vrv8ZpdDA1n20hoSEOJa379+/X++9954yMzPl5+cnSbrlllsUHR2tTz75RDabTfPnz9fw4cP13XffKTo6Wu+9955mzJihV155Rddff72WLFmiOXPmqEuXLhd9z2nTpun111/X//3f/+m6665Tfn6+9uzZI6k+oFx11VX69NNPlZqa6liu/frrr2vGjBl6+eWXNWDAAGVlZWnChAkKCwvT+PHjVV5erltvvVU33nij3n77beXk5Gjy5MmtOjdNRbhxkdyT9ZOJr2jPdRcAAE2zceNGvfPOOxo+fLgkyW63a8mSJYqJiZEkffbZZ8rOzlZhYaGCgoIkSc8//7w+/PBD/f3vf9fDDz+sWbNm6YEHHtBDDz0kSfrDH/6gTz/99LzemwalpaWaPXu2Xn75ZY0fP16SdOWVV+q6666TJMd7t2/fXvHx8Y7nPfvss3rhhRd0xx13SKrv4dm1a5fmz5+v8ePH6y9/+Ytqa2u1YMEChYaGKjU1VUeOHNGvfvUrZ5+28xBuXOTwqfqemyvouQEAtwsJ8NOu/xllyvs218cff6zw8HDV1NSourpat99+u1566SW9+uqruuKKKxzhQpK2bNmisrKy867EfObMGR04cECStHv3bk2cOLHR4+np6VqzZs0F33/37t2qqqpyBKqmOHHihPLy8vTggw9qwoQJjuM1NTWO+Ty7d+9Wv379Gl1cMT09vcnv0RqEGxew19Tp6PdnJEnJ9NwAgNtZLJZWDw+5y09/+lPNnTtXAQEBSkxMbDRpOCys8f8g19XVKSEhQWvXrj3vdaKiolr0/iEhIc1+Tl1dnaT6oamrr7660WMNw2ctmX/kLN7xm/cyR0+fUZ1Rn+BjIoLMLgcA4MHCwsLUtWvXJrUdOHCgCgoK5O/vr+Tk5Au26dWrlzZs2KB7773XcWzDhg0Xfc1u3bopJCRE//rXvxxDWT90oS0R4uLi1LFjRx08eFDjxo274Ov27t1bS5Ys0ZkzZxwB6lJ1OBOrpVzg0NnJxJ2j2ZkWAOA8N910k9LT0zVmzBj985//1KFDh/TVV1/p//2//6fNmzdLkiZPnqwFCxZowYIF+u677zRjxgzt3Lnzoq8ZHBysJ598Uk888YQWL16sAwcOaMOGDXrzzTclSbGxsQoJCdHKlSt1/PhxFRcXS6q/MGBGRoZmz56t7777TtnZ2Vq4cKFefPFFSdLdd98tq9WqBx98ULt27dInn3yi559/3sVnqB7hxgWYTAwAcAWLxaJPPvlEN9xwgx544AF1795dd911lw4dOqS4uDhJ0tixY/X000/rySef1KBBg3T48OHLTuL9r//6Lz3++ON6+umn1atXL40dO1aFhYWSJH9/f82ZM0fz589XYmKibr/9dknSQw89pDfeeEOLFi1S3759NXToUC1atMixdDw8PFz/+Mc/tGvXLg0YMEBPPfWUnnvuOReenXMshpmDYiYoKSmRzWZTcXGxIiMjXfIe//OPXVrwZY4mXJ+ip27p7ZL3AADUq6ysVE5OjlJSUhQczBXhvdmlfpfN+fym58YFDp9kpRQAAGYh3LjA4VMMSwEAYBbCjZPV1RmO3cCviKbnBgAAdyPcOFlBSaXsNXXyt1qUGMXYLwAA7ka4cbLDZ1dKdWoXIn8/Ti8AAO7Gp6+TMZkYAABzEW6cjMnEAACYi3DjZOcu4EfPDQAAZiDcOFnD1gtXRNNzAwCAGQg3TmQYBlsvAABgMsKNE50qt6u0qkYWi5REzw0A4CIsFsslb/fdd5/ZJXo1f7ML8CUNk4njI4MVHOBncjUAAE+Vn5/v+P6vf/2rnn76ae3du9dxLCQkpFH76upqBQQEuK0+b0fPjRMxJAUAHsIwJHu5+29N3Is6Pj7ecbPZbLJYLI77lZWVioqK0nvvvadhw4YpODhYb7/9tp555hn179+/0evMmjVLycnJjY4tXLhQvXr1UnBwsHr27KlXX33VSSfVe9Bz40TnJhOzUgoATFVdIf1vovvfd/oxKdA5nwFPPvmkXnjhBS1cuFBBQUF67bXXLvuc119/XTNmzNDLL7+sAQMGKCsrSxMmTFBYWJjGjx/vlLq8AeHGiRp6bjrTcwMAaKXHHntMd9xxR7Oe8+yzz+qFF15wPC8lJUW7du3S/PnzCTfusn79ev35z3/Wli1blJ+frw8++EBjxoy5aPu1a9fqpz/96XnHd+/erZ49e7qw0qZp6LlJ5ho3AGCugND6XhQz3tdJBg8e3Kz2J06cUF5enh588EFNmDDBcbympkY2m81pdXkDU8NNeXm5+vXrp/vvv1933nlnk5+3d+9eRUZGOu7HxMS4orxmy+XqxADgGSwWpw0PmSUsrHH9VqtVxo/m9FRXVzu+r6urk1Q/NHX11Vc3aufn17YWuZgabm6++WbdfPPNzX5ebGysoqKimtS2qqpKVVVVjvslJSXNfr+mKKuqUVGZXRLDUgAA54uJiVFBQYEMw5DFYpEkbdu2zfF4XFycOnbsqIMHD2rcuHEmVekZvHK11IABA5SQkKDhw4drzZo1l2ybkZEhm83muCUlJbmkphOlVWofFqjosEBFBrNcDwDgXMOGDdOJEyf0pz/9SQcOHNArr7yiFStWNGrzzDPPKCMjQ7Nnz9Z3332n7OxsLVy4UC+++KJJVZvDq8JNQkKCXnvtNWVmZur9999Xjx49NHz4cK1fv/6iz5k2bZqKi4sdt7y8PJfUltIhTFv+a4S++v2NLnl9AEDb1qtXL7366qt65ZVX1K9fP23cuFFTp05t1Oahhx7SG2+8oUWLFqlv374aOnSoFi1apJSUFJOqNofF+PEAnkksFstlJxRfyG233SaLxaJly5Y1qX1JSYlsNpuKi4sbzdsBAHinyspK5eTkKCUlRcHBwWaXg1a41O+yOZ/fXtVzcyHXXHON9u3bZ3YZAADAQ3h9uMnKylJCQoLZZQAAAA9h6mqpsrIy7d+/33E/JydH27ZtU3R0tDp37qxp06bp6NGjWrx4saRzl5lOTU2V3W7X22+/rczMTGVmZpr1IwAAAA9jarjZvHlzo4vyTZkyRZI0fvx4LVq0SPn5+crNzXU8brfbNXXqVB09elQhISFKTU3V8uXLNXr0aLfXDgAAPJPHTCh2FyYUA4BvaZiEmpycfN5u2vAuZ86c0aFDh5hQDABo2wIC6q8tVlFRYXIlaC27vf5iuK29ojIbZwIAvJqfn5+ioqJUWFgoSQoNDXVcwRfeo66uTidOnFBoaKj8/VsXTwg3AACvFx8fL0mOgAPvZLVa1blz51aHU8INAMDrWSwWJSQkKDY2ttFmkvAugYGBslpbP2OGcAMA8Bl+fn5tbgdsnI8JxQAAwKcQbgAAgE8h3AAAAJ/S5ubcNFyzsKSkxORKAABAUzV8bjfl2sNtLtyUlpZKkpKSkkyuBAAANFdpaalsNtsl27S57Rfq6up07NgxRUREOP0iTyUlJUpKSlJeXh5bO7gQ59k9OM/uwXl2H861e7jqPBuGodLSUiUmJl52uXib67mxWq3q1KmTS98jMjKS/3DcgPPsHpxn9+A8uw/n2j1ccZ4v12PTgAnFAADApxBuAACATyHcOFFQUJBmzJihoKAgs0vxaZxn9+A8uwfn2X041+7hCee5zU0oBgAAvo2eGwAA4FMINwAAwKcQbgAAgE8h3AAAAJ9CuHGSV199VSkpKQoODtagQYP0+eefm12SV1m/fr1uu+02JSYmymKx6MMPP2z0uGEYeuaZZ5SYmKiQkBANGzZMO3fubNSmqqpKjz76qDp06KCwsDD9/Oc/15EjR9z4U3i+jIwMDRkyRBEREYqNjdWYMWO0d+/eRm041603d+5cpaWlOS5ilp6erhUrVjge5xy7RkZGhiwWix577DHHMc61czzzzDOyWCyNbvHx8Y7HPe48G2i1pUuXGgEBAcbrr79u7Nq1y5g8ebIRFhZmHD582OzSvMYnn3xiPPXUU0ZmZqYhyfjggw8aPT5z5kwjIiLCyMzMNLKzs42xY8caCQkJRklJiaPNxIkTjY4dOxqrV682tm7davz0pz81+vXrZ9TU1Lj5p/Fco0aNMhYuXGjs2LHD2LZtm3HLLbcYnTt3NsrKyhxtONett2zZMmP58uXG3r17jb179xrTp083AgICjB07dhiGwTl2hY0bNxrJyclGWlqaMXnyZMdxzrVzzJgxw0hNTTXy8/Mdt8LCQsfjnnaeCTdOcNVVVxkTJ05sdKxnz57G73//e5Mq8m4/Djd1dXVGfHy8MXPmTMexyspKw2azGfPmzTMMwzBOnz5tBAQEGEuXLnW0OXr0qGG1Wo2VK1e6rXZvU1hYaEgy1q1bZxgG59qV2rVrZ7zxxhucYxcoLS01unXrZqxevdoYOnSoI9xwrp1nxowZRr9+/S74mCeeZ4alWslut2vLli0aOXJko+MjR47UV199ZVJVviUnJ0cFBQWNznFQUJCGDh3qOMdbtmxRdXV1ozaJiYnq06cPv4dLKC4uliRFR0dL4ly7Qm1trZYuXary8nKlp6dzjl1g0qRJuuWWW3TTTTc1Os65dq59+/YpMTFRKSkpuuuuu3Tw4EFJnnme29zGmc5WVFSk2tpaxcXFNToeFxengoICk6ryLQ3n8ULn+PDhw442gYGBateu3Xlt+D1cmGEYmjJliq677jr16dNHEufambKzs5Wenq7KykqFh4frgw8+UO/evR3/kHOOnWPp0qXaunWrNm3adN5j/D07z9VXX63Fixere/fuOn78uP7whz/o2muv1c6dOz3yPBNunMRisTS6bxjGecfQOi05x/weLu6RRx7R9u3b9cUXX5z3GOe69Xr06KFt27bp9OnTyszM1Pjx47Vu3TrH45zj1svLy9PkyZO1atUqBQcHX7Qd57r1br75Zsf3ffv2VXp6uq688kq99dZbuuaaayR51nlmWKqVOnToID8/v/OSZ2Fh4XkpFi3TMCP/Uuc4Pj5edrtd33///UXb4JxHH31Uy5Yt05o1a9SpUyfHcc618wQGBqpr164aPHiwMjIy1K9fP82ePZtz7ERbtmxRYWGhBg0aJH9/f/n7+2vdunWaM2eO/P39HeeKc+18YWFh6tu3r/bt2+eRf9OEm1YKDAzUoEGDtHr16kbHV69erWuvvdakqnxLSkqK4uPjG51ju92udevWOc7xoEGDFBAQ0KhNfn6+duzYwe/hBwzD0COPPKL3339fn332mVJSUho9zrl2HcMwVFVVxTl2ouHDhys7O1vbtm1z3AYPHqxx48Zp27Zt6tKlC+faRaqqqrR7924lJCR45t+006cot0ENS8HffPNNY9euXcZjjz1mhIWFGYcOHTK7NK9RWlpqZGVlGVlZWYYk48UXXzSysrIcy+lnzpxp2Gw24/333zeys7ONX/7ylxdcZtipUyfj008/NbZu3WrceOONLOf8kV/96leGzWYz1q5d22hJZ0VFhaMN57r1pk2bZqxfv97Iyckxtm/fbkyfPt2wWq3GqlWrDMPgHLvSD1dLGQbn2lkef/xxY+3atcbBgweNDRs2GLfeeqsRERHh+JzztPNMuHGSV155xbjiiiuMwMBAY+DAgY6ltWiaNWvWGJLOu40fP94wjPqlhjNmzDDi4+ONoKAg44YbbjCys7MbvcaZM2eMRx55xIiOjjZCQkKMW2+91cjNzTXhp/FcFzrHkoyFCxc62nCuW++BBx5w/HsQExNjDB8+3BFsDINz7Eo/Djeca+douG5NQECAkZiYaNxxxx3Gzp07HY972nm2GIZhOL8/CAAAwBzMuQEAAD6FcAMAAHwK4QYAAPgUwg0AAPAphBsAAOBTCDcAAMCnEG4AAIBPIdwAAACfQrgB0OYkJydr1qxZZpcBwEUINwBc6r777tOYMWMkScOGDdNjjz3mtvdetGiRoqKizju+adMmPfzww26rA4B7+ZtdAAA0l91uV2BgYIufHxMT48RqAHgaem4AuMV9992ndevWafbs2bJYLLJYLDp06JAkadeuXRo9erTCw8MVFxene+65R0VFRY7nDhs2TI888oimTJmiDh06aMSIEZKkF198UX379lVYWJiSkpL061//WmVlZZKktWvX6v7771dxcbHj/Z555hlJ5w9L5ebm6vbbb1d4eLgiIyP1i1/8QsePH3c8/swzz6h///5asmSJkpOTZbPZdNddd6m0tNS1Jw1AixBuALjF7NmzlZ6ergkTJig/P1/5+flKSkpSfn6+hg4dqv79+2vz5s1auXKljh8/rl/84heNnv/WW2/J399fX375pebPny9JslqtmjNnjnbs2KG33npLn332mZ544glJ0rXXXqtZs2YpMjLS8X5Tp049ry7DMDRmzBidOnVK69at0+rVq3XgwAGNHTu2UbsDBw7oww8/1Mcff6yPP/5Y69at08yZM110tgC0BsNSANzCZrMpMDBQoaGhio+PdxyfO3euBg4cqP/93/91HFuwYIGSkpL03XffqXv37pKkrl276k9/+lOj1/zh/J2UlBQ9++yz+tWvfqVXX31VgYGBstlsslgsjd7vxz799FNt375dOTk5SkpKkiQtWbJEqamp2rRpk4YMGSJJqqur06JFixQRESFJuueee/Svf/1Lf/zjH1t3YgA4HT03AEy1ZcsWrVmzRuHh4Y5bz549JdX3ljQYPHjwec9ds2aNRowYoY4dOyoiIkL33nuvTp48qfLy8ia//+7du5WUlOQINpLUu3dvRUVFaffu3Y5jycnJjmAjSQkJCSosLGzWzwrAPei5AWCquro63XbbbXruuefOeywhIcHxfVhYWKPHDh8+rNGjR2vixIl69tlnFR0drS+++EIPPvigqqurm/z+hmHIYrFc9nhAQECjxy0Wi+rq6pr8PgDch3ADwG0CAwNVW1vb6NjAgQOVmZmp5ORk+fs3/Z+kzZs3q6amRi+88IKs1vpO6Pfee++y7/djvXv3Vm5urvLy8hy9N7t27VJxcbF69erV5HoAeA6GpQC4TXJysr755hsdOnRIRUVFqqur06RJk3Tq1Cn98pe/1MaNG3Xw4EGtWrVKDzzwwCWDyZVXXqmamhq99NJLOnjwoJYsWaJ58+ad935lZWX617/+paKiIlVUVJz3OjfddJPS0tI0btw4bd26VRs3btS9996roUOHXnAoDIDnI9wAcJupU6fKz89PvXv3VkxMjHJzc5WYmKgvv/xStbW1GjVqlPr06aPJkyfLZrM5emQupH///nrxxRf13HPPqU+fPvrLX/6ijIyMRm2uvfZaTZw4UWPHjlVMTMx5E5Kl+uGlDz/8UO3atdMNN9ygm266SV26dNFf//pXp//8ANzDYhiGYXYRAAAAzkLPDQAA8CmEGwAA4FMINwAAwKcQbgAAgE8h3AAAAJ9CuAEAAD6FcAMAAHwK4QYAAPgUwg0AAPAphBsAAOBTCDcAAMCn/H9Yf7DWeMkDcQAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "### Function minimization with autograd and gradient descent ###\n", + "\n", + "# Initialize a random value for our intial x\n", + "x = torch.randn(1)\n", + "print(f\"Initializing x={x.item()}\")\n", + "\n", + "learning_rate = 1e-2 # Learning rate\n", + "history = []\n", + "x_f = 4 # Target value\n", + "\n", + "\n", + "# We will run gradient descent for a number of iterations. At each iteration, we compute the loss,\n", + "# compute the derivative of the loss with respect to x, and perform the update.\n", + "for i in range(500):\n", + " x = torch.tensor([x], requires_grad=True)\n", + "\n", + " # TODO: Compute the loss as the square of the difference between x and x_f\n", + " loss = (x - x_f) ** 2\n", + "\n", + " # Backpropagate through the loss to compute gradients\n", + " loss.backward()\n", + "\n", + " # Update x with gradient descent\n", + " x = x.item() - learning_rate * x.grad\n", + "\n", + " history.append(x.item())\n", + "\n", + "# Plot the evolution of x as we optimize toward x_f!\n", + "plt.plot(history)\n", + "plt.plot([0, 500], [x_f, x_f])\n", + "plt.legend(('Predicted', 'True'))\n", + "plt.xlabel('Iteration')\n", + "plt.ylabel('x value')\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pC7czCwk3ceH" + }, + "source": [ + "Now, we have covered the fundamental concepts of PyTorch -- tensors, operations, neural networks, and automatic differentiation. Fire!!\n" + ] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "collapsed_sections": [ + "WBk0ZDWY-ff8" + ], + "name": "PT_Part1_Intro.ipynb", + "provenance": [] + }, + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.7" }, - "nbformat": 4, - "nbformat_minor": 0 + "vscode": { + "interpreter": { + "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6" + } + } + }, + "nbformat": 4, + "nbformat_minor": 4 } diff --git a/lab1/PT_Part2_Music_Generation.ipynb b/lab1/PT_Part2_Music_Generation.ipynb index a99bca7d..4dfcfe9a 100644 --- a/lab1/PT_Part2_Music_Generation.ipynb +++ b/lab1/PT_Part2_Music_Generation.ipynb @@ -1,1052 +1,1404 @@ { - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "id": "uoJsVjtCMunI" - }, - "source": [ - "\n", - " \n", - " \n", - " \n", - "
\n", - " \n", - " Visit MIT Deep Learning\n", - " Run in Google Colab\n", - " View Source on GitHub
\n", - "\n", - "# Copyright Information" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "bUik05YqMyCH" - }, - "outputs": [], - "source": [ - "# Copyright 2025 MIT Introduction to Deep Learning. All Rights Reserved.\n", - "#\n", - "# Licensed under the MIT License. You may not use this file except in compliance\n", - "# with the License. Use and/or modification of this code outside of MIT Introduction\n", - "# to Deep Learning must reference:\n", - "#\n", - "# © MIT Introduction to Deep Learning\n", - "# http://introtodeeplearning.com\n", - "#" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "O-97SDET3JG-" - }, - "source": [ - "# Lab 1: Intro to PyTorch and Music Generation with RNNs\n", - "\n", - "# Part 2: Music Generation with RNNs\n", - "\n", - "In this portion of the lab, we will explore building a Recurrent Neural Network (RNN) for music generation using PyTorch. We will train a model to learn the patterns in raw sheet music in [ABC notation](https://en.wikipedia.org/wiki/ABC_notation) and then use this model to generate new music." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rsvlBQYCrE4I" - }, - "source": [ - "## 2.1 Dependencies\n", - "First, let's download the course repository, install dependencies, and import the relevant packages we'll need for this lab.\n", - "\n", - "We will be using [Comet ML](https://www.comet.com/docs/v2/) to track our model development and training runs. First, sign up for a Comet account [at this link](https://www.comet.com/signup?utm_source=mit_dl&utm_medium=partner&utm_content=github\n", - ") (you can use your Google or Github account). You will need to generate a new personal API Key, which you can find either in the first 'Get Started with Comet' page, under your account settings, or by pressing the '?' in the top right corner and then 'Quickstart Guide'. Enter this API key as the global variable `COMET_API_KEY`." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "riVZCVK65QTH" - }, - "outputs": [], - "source": [ - "!pip install comet_ml > /dev/null 2>&1\n", - "import comet_ml\n", - "# TODO: ENTER YOUR API KEY HERE!! instructions above\n", - "COMET_API_KEY = \"\"\n", - "\n", - "# Import PyTorch and other relevant libraries\n", - "import torch\n", - "import torch.nn as nn\n", - "import torch.optim as optim\n", - "\n", - "# Download and import the MIT Introduction to Deep Learning package\n", - "!pip install mitdeeplearning --quiet\n", - "import mitdeeplearning as mdl\n", - "\n", - "# Import all remaining packages\n", - "import numpy as np\n", - "import os\n", - "import time\n", - "import functools\n", - "from IPython import display as ipythondisplay\n", - "from tqdm import tqdm\n", - "from scipy.io.wavfile import write\n", - "!apt-get install abcmidi timidity > /dev/null 2>&1\n", - "\n", - "\n", - "# Check that we are using a GPU, if not switch runtimes\n", - "# using Runtime > Change Runtime Type > GPU\n", - "assert torch.cuda.is_available(), \"Please enable GPU from runtime settings\"\n", - "assert COMET_API_KEY != \"\", \"Please insert your Comet API Key\"\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_ajvp0No4qDm" - }, - "source": [ - "## 2.2 Dataset\n", - "\n", - "![Let's Dance!](http://33.media.tumblr.com/3d223954ad0a77f4e98a7b87136aa395/tumblr_nlct5lFVbF1qhu7oio1_500.gif)\n", - "\n", - "We've gathered a dataset of thousands of Irish folk songs, represented in the ABC notation. Let's download the dataset and inspect it:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "P7dFnP5q3Jve" - }, - "outputs": [], - "source": [ - "# Download the dataset\n", - "songs = mdl.lab1.load_training_data()\n", - "\n", - "# Print one of the songs to inspect it in greater detail!\n", - "example_song = songs[0]\n", - "print(\"\\nExample song: \")\n", - "print(example_song)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hKF3EHJlCAj2" - }, - "source": [ - "We can easily convert a song in ABC notation to an audio waveform and play it back. Be patient for this conversion to run, it can take some time." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "11toYzhEEKDz" - }, - "outputs": [], - "source": [ - "# Convert the ABC notation to audio file and listen to it\n", - "mdl.lab1.play_song(example_song)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "7vH24yyquwKQ" - }, - "source": [ - "One important thing to think about is that this notation of music does not simply contain information on the notes being played, but additionally there is meta information such as the song title, key, and tempo. How does the number of different characters that are present in the text file impact the complexity of the learning problem? This will become important soon, when we generate a numerical representation for the text data." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "IlCgQBRVymwR" - }, - "outputs": [], - "source": [ - "# Join our list of song strings into a single string containing all songs\n", - "songs_joined = \"\\n\\n\".join(songs)\n", - "\n", - "# Find all unique characters in the joined string\n", - "vocab = sorted(set(songs_joined))\n", - "print(\"There are\", len(vocab), \"unique characters in the dataset\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rNnrKn_lL-IJ" - }, - "source": [ - "## 2.3 Process the dataset for the learning task\n", - "\n", - "Let's take a step back and consider our prediction task. We're trying to train an RNN model to learn patterns in ABC music, and then use this model to generate (i.e., predict) a new piece of music based on this learned information.\n", - "\n", - "Breaking this down, what we're really asking the model is: given a character, or a sequence of characters, what is the most probable next character? We'll train the model to perform this task.\n", - "\n", - "To achieve this, we will input a sequence of characters to the model, and train the model to predict the output, that is, the following character at each time step. RNNs maintain an internal state that depends on previously seen elements, so information about all characters seen up until a given moment will be taken into account in generating the prediction." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "LFjSVAlWzf-N" - }, - "source": [ - "### Vectorize the text\n", - "\n", - "Before we begin training our RNN model, we'll need to create a numerical representation of our text-based dataset. To do this, we'll generate two lookup tables: one that maps characters to numbers, and a second that maps numbers back to characters. Recall that we just identified the unique characters present in the text.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "IalZLbvOzf-F" - }, - "outputs": [], - "source": [ - "### Define numerical representation of text ###\n", - "\n", - "# Create a mapping from character to unique index.\n", - "# For example, to get the index of the character \"d\",\n", - "# we can evaluate `char2idx[\"d\"]`.\n", - "char2idx = {u: i for i, u in enumerate(vocab)}\n", - "\n", - "# Create a mapping from indices to characters. This is\n", - "# the inverse of char2idx and allows us to convert back\n", - "# from unique index to the character in our vocabulary.\n", - "idx2char = np.array(vocab)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "tZfqhkYCymwX" - }, - "source": [ - "This gives us an integer representation for each character. Observe that the unique characters (i.e., our vocabulary) in the text are mapped as indices from 0 to `len(unique)`. Let's take a peek at this numerical representation of our dataset:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "FYyNlCNXymwY" - }, - "outputs": [], - "source": [ - "print('{')\n", - "for char, _ in zip(char2idx, range(20)):\n", - " print(' {:4s}: {:3d},'.format(repr(char), char2idx[char]))\n", - "print(' ...\\n}')" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "g-LnKyu4dczc" - }, - "outputs": [], - "source": [ - "### Vectorize the songs string ###\n", - "\n", - "'''TODO: Write a function to convert the all songs string to a vectorized\n", - " (i.e., numeric) representation. Use the appropriate mapping\n", - " above to convert from vocab characters to the corresponding indices.\n", - "\n", - " NOTE: the output of the `vectorize_string` function\n", - " should be a np.array with `N` elements, where `N` is\n", - " the number of characters in the input string\n", - "'''\n", - "def vectorize_string(string):\n", - " '''TODO'''\n", - "\n", - "vectorized_songs = vectorize_string(songs_joined)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "IqxpSuZ1w-ub" - }, - "source": [ - "We can also look at how the first part of the text is mapped to an integer representation:\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "l1VKcQHcymwb" - }, - "outputs": [], - "source": [ - "print ('{} ---- characters mapped to int ----> {}'.format(repr(songs_joined[:10]), vectorized_songs[:10]))\n", - "# check that vectorized_songs is a numpy array\n", - "assert isinstance(vectorized_songs, np.ndarray), \"returned result should be a numpy array\"" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "hgsVvVxnymwf" - }, - "source": [ - "### Create training examples and targets\n", - "\n", - "Our next step is to actually divide the text into example sequences that we'll use during training. Each input sequence that we feed into our RNN will contain `seq_length` characters from the text. We'll also need to define a target sequence for each input sequence, which will be used in training the RNN to predict the next character. For each input, the corresponding target will contain the same length of text, except shifted one character to the right.\n", - "\n", - "To do this, we'll break the text into chunks of `seq_length+1`. Suppose `seq_length` is 4 and our text is \"Hello\". Then, our input sequence is \"Hell\" and the target sequence is \"ello\".\n", - "\n", - "The batch method will then let us convert this stream of character indices to sequences of the desired size.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "LF-N8F7BoDRi" - }, - "outputs": [], - "source": [ - "### Batch definition to create training examples ###\n", - "\n", - "def get_batch(vectorized_songs, seq_length, batch_size):\n", - " # the length of the vectorized songs string\n", - " n = vectorized_songs.shape[0] - 1\n", - " # randomly choose the starting indices for the examples in the training batch\n", - " idx = np.random.choice(n - seq_length, batch_size)\n", - "\n", - " '''TODO: construct a list of input sequences for the training batch'''\n", - " input_batch = # TODO\n", - "\n", - " '''TODO: construct a list of output sequences for the training batch'''\n", - " output_batch = # TODO\n", - "\n", - " # Convert the input and output batches to tensors\n", - " x_batch = torch.tensor(input_batch, dtype=torch.long)\n", - " y_batch = torch.tensor(output_batch, dtype=torch.long)\n", - "\n", - " return x_batch, y_batch\n", - "\n", - "# Perform some simple tests to make sure your batch function is working properly!\n", - "test_args = (vectorized_songs, 10, 2)\n", - "x_batch, y_batch = get_batch(*test_args)\n", - "assert x_batch.shape == (2, 10), \"x_batch shape is incorrect\"\n", - "assert y_batch.shape == (2, 10), \"y_batch shape is incorrect\"\n", - "print(\"Batch function works correctly!\")\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "_33OHL3b84i0" - }, - "source": [ - "For each of these vectors, each index is processed at a single time step. So, for the input at time step 0, the model receives the index for the first character in the sequence, and tries to predict the index of the next character. At the next timestep, it does the same thing, but the RNN considers the information from the previous step, i.e., its updated state, in addition to the current input.\n", - "\n", - "We can make this concrete by taking a look at how this works over the first several characters in our text:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "0eBu9WZG84i0" - }, - "outputs": [], - "source": [ - "x_batch, y_batch = get_batch(vectorized_songs, seq_length=5, batch_size=1)\n", - "\n", - "for i, (input_idx, target_idx) in enumerate(zip(x_batch[0], y_batch[0])):\n", - " print(\"Step {:3d}\".format(i))\n", - " print(\" input: {} ({:s})\".format(input_idx, repr(idx2char[input_idx.item()])))\n", - " print(\" expected output: {} ({:s})\".format(target_idx, repr(idx2char[target_idx.item()])))\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "r6oUuElIMgVx" - }, - "source": [ - "## 2.4 The Recurrent Neural Network (RNN) model" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "m8gPwEjRzf-Z" - }, - "source": [ - "Now we're ready to define and train an RNN model on our ABC music dataset, and then use that trained model to generate a new song. We'll train our RNN using batches of song snippets from our dataset, which we generated in the previous section.\n", - "\n", - "The model is based off the LSTM architecture, where we use a state vector to maintain information about the temporal relationships between consecutive characters. The final output of the LSTM is then fed into a fully connected linear [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer where we'll output a softmax over each character in the vocabulary, and then sample from this distribution to predict the next character.\n", - "\n", - "As we introduced in the first portion of this lab, we'll be using PyTorch's [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) to define the model. Three components are used to define the model:\n", - "\n", - "* [`nn.Embedding`](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html): This is the input layer, consisting of a trainable lookup table that maps the numbers of each character to a vector with `embedding_dim` dimensions.\n", - "* [`nn.LSTM`](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html): Our LSTM network, with size `hidden_size`.\n", - "* [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html): The output layer, with `vocab_size` outputs.\n", - "\n", - "\"Drawing\"/\n", - "\n", - "\n", - "\n", - "" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "rlaOqndqBmJo" - }, - "source": [ - "### Define the RNN model\n", - "\n", - "Let's define our model as an `nn.Module`. Fill in the `TODOs` to define the RNN model.\n" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "8DsWzojvkbc7" - }, - "outputs": [], - "source": [ - "### Defining the RNN Model ###\n", - "\n", - "'''TODO: Add LSTM and Linear layers to define the RNN model using nn.Module'''\n", - "class LSTMModel(nn.Module):\n", - " def __init__(self, vocab_size, embedding_dim, hidden_size):\n", - " super(LSTMModel, self).__init__()\n", - " self.hidden_size = hidden_size\n", - "\n", - " # Define each of the network layers\n", - " # Layer 1: Embedding layer to transform indices into dense vectors\n", - " # of a fixed embedding size\n", - " self.embedding = nn.Embedding(vocab_size, embedding_dim)\n", - "\n", - " '''TODO: Layer 2: LSTM with hidden_size `hidden_size`. note: number of layers defaults to 1.\n", - " Use the nn.LSTM() module from pytorch.'''\n", - " self.lstm = nn.LSTM('''TODO''') # TODO\n", - "\n", - " '''TODO: Layer 3: Linear (fully-connected) layer that transforms the LSTM output\n", - " # into the vocabulary size.'''\n", - " self.fc = nn.Linear('''TODO''') # TODO\n", - "\n", - " def init_hidden(self, batch_size, device):\n", - " # Initialize hidden state and cell state with zeros\n", - " return (torch.zeros(1, batch_size, self.hidden_size).to(device),\n", - " torch.zeros(1, batch_size, self.hidden_size).to(device))\n", - "\n", - " def forward(self, x, state=None, return_state=False):\n", - " x = self.embedding(x)\n", - "\n", - " if state is None:\n", - " state = self.init_hidden(x.size(0), x.device)\n", - " out, state = self.lstm(x, state)\n", - "\n", - " out = self.fc(out)\n", - " return out if not return_state else (out, state)\n", - "\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "IbWU4dMJmMvq" - }, - "source": [ - "The time has come! Let's instantiate the model!" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "MtCrdfzEI2N0" - }, - "outputs": [], - "source": [ - "# Instantiate the model! Build a simple model with default hyperparameters. You\n", - "# will get the chance to change these later.\n", - "vocab_size = len(vocab)\n", - "embedding_dim = 256\n", - "hidden_size = 1024\n", - "batch_size = 8\n", - "\n", - "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", - "\n", - "model = LSTMModel(vocab_size, embedding_dim, hidden_size).to(device)\n", - "\n", - "# print out a summary of the model\n", - "print(model)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "-ubPo0_9Prjb" - }, - "source": [ - "### Test out the RNN model\n", - "\n", - "It's always a good idea to run a few simple checks on our model to see that it behaves as expected. \n", - "\n", - "We can quickly check the layers in the model, the shape of the output of each of the layers, the batch size, and the dimensionality of the output. Note that the model can be run on inputs of any length." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "C-_70kKAPrPU" - }, - "outputs": [], - "source": [ - "# Test the model with some sample data\n", - "x, y = get_batch(vectorized_songs, seq_length=100, batch_size=32)\n", - "x = x.to(device)\n", - "y = y.to(device)\n", - "\n", - "pred = model(x)\n", - "print(\"Input shape: \", x.shape, \" # (batch_size, sequence_length)\")\n", - "print(\"Prediction shape: \", pred.shape, \"# (batch_size, sequence_length, vocab_size)\")" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "id": "mT1HvFVUGpoE" - }, - "source": [ - "### Predictions from the untrained model\n", - "\n", - "Let's take a look at what our untrained model is predicting.\n", - "\n", - "To get actual predictions from the model, we sample from the output distribution, which is defined by a torch.softmax over our character vocabulary. This will give us actual character indices. This means we are using a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) to sample over the example prediction. This gives a prediction of the next character (specifically its index) at each timestep. [`torch.multinomial`](https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial) samples over a categorical distribution to generate predictions.\n", - "\n", - "Note here that we sample from this probability distribution, as opposed to simply taking the `argmax`, which can cause the model to get stuck in a repetitive loop.\n", - "\n", - "Let's try this sampling out for the first example in the batch." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4V4MfFg0RQJg" - }, - "outputs": [], - "source": [ - "sampled_indices = torch.multinomial(torch.softmax(pred[0], dim=-1), num_samples=1)\n", - "sampled_indices = sampled_indices.squeeze(-1).cpu().numpy()\n", - "sampled_indices" - ] - }, + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "uoJsVjtCMunI" + }, + "source": [ + "\n", + " \n", + " \n", + " \n", + "
\n", + " \n", + " Visit MIT Deep Learning\n", + " Run in Google Colab\n", + " View Source on GitHub
\n", + "\n", + "# Copyright Information" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "bUik05YqMyCH" + }, + "outputs": [], + "source": [ + "# Copyright 2025 MIT Introduction to Deep Learning. All Rights Reserved.\n", + "#\n", + "# Licensed under the MIT License. You may not use this file except in compliance\n", + "# with the License. Use and/or modification of this code outside of MIT Introduction\n", + "# to Deep Learning must reference:\n", + "#\n", + "# © MIT Introduction to Deep Learning\n", + "# http://introtodeeplearning.com\n", + "#" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O-97SDET3JG-" + }, + "source": [ + "# Lab 1: Intro to PyTorch and Music Generation with RNNs\n", + "\n", + "# Part 2: Music Generation with RNNs\n", + "\n", + "In this portion of the lab, we will explore building a Recurrent Neural Network (RNN) for music generation using PyTorch. We will train a model to learn the patterns in raw sheet music in [ABC notation](https://en.wikipedia.org/wiki/ABC_notation) and then use this model to generate new music." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rsvlBQYCrE4I" + }, + "source": [ + "## 2.1 Dependencies\n", + "First, let's download the course repository, install dependencies, and import the relevant packages we'll need for this lab.\n", + "\n", + "We will be using [Comet ML](https://www.comet.com/docs/v2/) to track our model development and training runs. First, sign up for a Comet account [at this link](https://www.comet.com/signup?utm_source=mit_dl&utm_medium=partner&utm_content=github\n", + ") (you can use your Google or Github account). You will need to generate a new personal API Key, which you can find either in the first 'Get Started with Comet' page, under your account settings, or by pressing the '?' in the top right corner and then 'Quickstart Guide'. Enter this API key as the global variable `COMET_API_KEY`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "riVZCVK65QTH" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "LfLtsP3mUhCG" - }, - "source": [ - "We can now decode these to see the text predicted by the untrained model:" - ] + "name": "stderr", + "output_type": "stream", + "text": [ + "‘Ёб⥬Ґ ­Ґ г¤ Ґвбп ­ ©вЁ гЄ § ­­л© Їгвм.\n" + ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "xWcFwPwLSo05" - }, - "outputs": [], - "source": [ - "print(\"Input: \\n\", repr(\"\".join(idx2char[x[0].cpu()])))\n", - "print()\n", - "print(\"Next Char Predictions: \\n\", repr(\"\".join(idx2char[sampled_indices])))" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Requirement already satisfied: music21 in c:\\users\\savic\\anaconda3\\lib\\site-packages (9.5.0)\n", + "Requirement already satisfied: midi2audio in c:\\users\\savic\\anaconda3\\lib\\site-packages (0.1.1)\n", + "Requirement already satisfied: pretty_midi in c:\\users\\savic\\anaconda3\\lib\\site-packages (0.2.10)\n", + "Requirement already satisfied: soundfile in c:\\users\\savic\\anaconda3\\lib\\site-packages (0.13.1)\n", + "Requirement already satisfied: chardet in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (4.0.0)\n", + "Requirement already satisfied: joblib in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (1.4.2)\n", + "Requirement already satisfied: jsonpickle in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (4.0.5)\n", + "Requirement already satisfied: matplotlib in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (3.10.0)\n", + "Requirement already satisfied: more-itertools in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (10.3.0)\n", + "Requirement already satisfied: numpy<2.0.0 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (1.26.4)\n", + "Requirement already satisfied: requests in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (2.32.3)\n", + "Requirement already satisfied: webcolors>=1.5 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from music21) (24.11.1)\n", + "Requirement already satisfied: mido>=1.1.16 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from pretty_midi) (1.3.3)\n", + "Requirement already satisfied: six in c:\\users\\savic\\anaconda3\\lib\\site-packages (from pretty_midi) (1.16.0)\n", + "Requirement already satisfied: cffi>=1.0 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from soundfile) (1.17.1)\n", + "Requirement already satisfied: pycparser in c:\\users\\savic\\anaconda3\\lib\\site-packages (from cffi>=1.0->soundfile) (2.21)\n", + "Requirement already satisfied: packaging in c:\\users\\savic\\anaconda3\\lib\\site-packages (from mido>=1.1.16->pretty_midi) (24.2)\n", + "Requirement already satisfied: contourpy>=1.0.1 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (1.3.1)\n", + "Requirement already satisfied: cycler>=0.10 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (0.11.0)\n", + "Requirement already satisfied: fonttools>=4.22.0 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (4.55.3)\n", + "Requirement already satisfied: kiwisolver>=1.3.1 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (1.4.8)\n", + "Requirement already satisfied: pillow>=8 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (11.1.0)\n", + "Requirement already satisfied: pyparsing>=2.3.1 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (3.2.0)\n", + "Requirement already satisfied: python-dateutil>=2.7 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from matplotlib->music21) (2.9.0.post0)\n", + "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from requests->music21) (3.3.2)\n", + "Requirement already satisfied: idna<4,>=2.5 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from requests->music21) (3.7)\n", + "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from requests->music21) (2.3.0)\n", + "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\savic\\anaconda3\\lib\\site-packages (from requests->music21) (2025.1.31)\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "HEHHcRasIDm9" - }, - "source": [ - "As you can see, the text predicted by the untrained model is pretty nonsensical! How can we do better? Well, we can train the network!" - ] - }, + "name": "stderr", + "output_type": "stream", + "text": [ + "‘Ёб⥬Ґ ­Ґ г¤ Ґвбп ­ ©вЁ гЄ § ­­л© Їгвм.\n" + ] + } + ], + "source": [ + "!pip install comet_ml > /dev/null 2>&1\n", + "import comet_ml\n", + "# TODO: ENTER YOUR API KEY HERE!! instructions above\n", + "COMET_API_KEY = \"tTD6WoX5wfoAVBsmd5rREll2p\"\n", + "\n", + "# Import PyTorch and other relevant libraries\n", + "import torch\n", + "import torch.nn as nn\n", + "import torch.optim as optim\n", + "\n", + "# Download and import the MIT Introduction to Deep Learning package\n", + "!pip install mitdeeplearning --quiet\n", + "import mitdeeplearning as mdl\n", + "\n", + "# Another ones Python libraries for worling via Jupyter Notebook and import them\n", + "!pip install music21 midi2audio pretty_midi soundfile\n", + "from music21 import converter, environment\n", + "import pretty_midi\n", + "import soundfile as sf\n", + "\n", + "# Import all remaining packages\n", + "import numpy as np\n", + "import os\n", + "import time\n", + "import functools\n", + "from IPython import display as ipythondisplay\n", + "from tqdm import tqdm\n", + "from scipy.io.wavfile import write\n", + "!apt-get install abcmidi timidity > /dev/null 2>&1\n", + "\n", + "\n", + "# Check that we are using a GPU, if not switch runtimes\n", + "# using Runtime > Change Runtime Type > GPU\n", + "assert torch.cuda.is_available(), \"Please enable GPU from runtime settings\"\n", + "assert COMET_API_KEY != \"\", \"Please insert your Comet API Key\"\n", + "\n", + "# Preprocessing errors in music21 while saving .wav\n", + "env = environment.Environment()\n", + "env['autoDownload'] = 'allow' # разрешить автоматическую загрузку при необходимости" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_ajvp0No4qDm" + }, + "source": [ + "## 2.2 Dataset\n", + "\n", + "![Let's Dance!](http://33.media.tumblr.com/3d223954ad0a77f4e98a7b87136aa395/tumblr_nlct5lFVbF1qhu7oio1_500.gif)\n", + "\n", + "We've gathered a dataset of thousands of Irish folk songs, represented in the ABC notation. Let's download the dataset and inspect it:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "P7dFnP5q3Jve" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "LJL0Q0YPY6Ee" - }, - "source": [ - "## 2.5 Training the model: loss and training operations\n", - "\n", - "Now it's time to train the model!\n", - "\n", - "At this point, we can think of our next character prediction problem as a standard classification problem. Given the previous state of the RNN, as well as the input at a given time step, we want to predict the class of the next character -- that is, to actually predict the next character.\n", - "\n", - "To train our model on this classification task, we can use a form of the `crossentropy` loss (i.e., negative log likelihood loss). Specifically, we will use PyTorch's [`CrossEntropyLoss`](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html), as it combines the application of a log-softmax ([`LogSoftmax`](https://pytorch.org/docs/stable/generated/torch.nn.LogSoftmax.html#torch.nn.LogSoftmax)) and negative log-likelihood ([`NLLLoss`](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html#torch.nn.NLLLoss) in a single class and accepts integer targets for categorical classification tasks. We will want to compute the loss using the true targets -- the `labels` -- and the predicted targets -- the `logits`.\n", - "\n", - "Let's define a function to compute the loss, and then use that function to compute the loss using our example predictions from the untrained model." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 817 songs in text\n", + "\n", + "Example song: \n", + "X:1\n", + "T:Alexander's\n", + "Z: id:dc-hornpipe-1\n", + "M:C|\n", + "L:1/8\n", + "K:D Major\n", + "(3ABc|dAFA DFAd|fdcd FAdf|gfge fefd|(3efe (3dcB A2 (3ABc|!\n", + "dAFA DFAd|fdcd FAdf|gfge fefd|(3efe dc d2:|!\n", + "AG|FAdA FAdA|GBdB GBdB|Acec Acec|dfaf gecA|!\n", + "FAdA FAdA|GBdB GBdB|Aceg fefd|(3efe dc d2:|!\n" + ] + } + ], + "source": [ + "# Download the dataset\n", + "songs = mdl.lab1.load_training_data()\n", + "\n", + "# Print one of the songs to inspect it in greater detail!\n", + "example_song = songs[0]\n", + "print(\"\\nExample song: \")\n", + "print(example_song)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hKF3EHJlCAj2" + }, + "source": [ + "We can easily convert a song in ABC notation to an audio waveform and play it back. Be patient for this conversion to run, it can take some time." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "11toYzhEEKDz" + }, + "outputs": [], + "source": [ + "# Convert the ABC notation to audio file and listen to it\n", + "mdl.lab1.play_song(example_song)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7vH24yyquwKQ" + }, + "source": [ + "One important thing to think about is that this notation of music does not simply contain information on the notes being played, but additionally there is meta information such as the song title, key, and tempo. How does the number of different characters that are present in the text file impact the complexity of the learning problem? This will become important soon, when we generate a numerical representation for the text data." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "IlCgQBRVymwR" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "4HrXTACTdzY-" - }, - "outputs": [], - "source": [ - "### Defining the loss function ###\n", - "\n", - "# '''TODO: define the compute_loss function to compute and return the loss between\n", - "# the true labels and predictions (logits). '''\n", - "cross_entropy = nn.CrossEntropyLoss() # instantiates the function\n", - "def compute_loss(labels, logits):\n", - " \"\"\"\n", - " Inputs:\n", - " labels: (batch_size, sequence_length)\n", - " logits: (batch_size, sequence_length, vocab_size)\n", - "\n", - " Output:\n", - " loss: scalar cross entropy loss over the batch and sequence length\n", - " \"\"\"\n", - "\n", - " # Batch the labels so that the shape of the labels should be (B * L,)\n", - " batched_labels = labels.view(-1)\n", - "\n", - " ''' TODO: Batch the logits so that the shape of the logits should be (B * L, V) '''\n", - " batched_logits = \"\"\" TODO \"\"\" # TODO\n", - "\n", - " '''TODO: Compute the cross-entropy loss using the batched next characters and predictions'''\n", - " loss = \"\"\" TODO \"\"\" # TODO\n", - " return loss" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "There are 83 unique characters in the dataset\n" + ] + } + ], + "source": [ + "# Join our list of song strings into a single string containing all songs\n", + "songs_joined = \"\\n\\n\".join(songs)\n", + "\n", + "# Find all unique characters in the joined string\n", + "vocab = sorted(set(songs_joined))\n", + "print(\"There are\", len(vocab), \"unique characters in the dataset\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rNnrKn_lL-IJ" + }, + "source": [ + "## 2.3 Process the dataset for the learning task\n", + "\n", + "Let's take a step back and consider our prediction task. We're trying to train an RNN model to learn patterns in ABC music, and then use this model to generate (i.e., predict) a new piece of music based on this learned information.\n", + "\n", + "Breaking this down, what we're really asking the model is: given a character, or a sequence of characters, what is the most probable next character? We'll train the model to perform this task.\n", + "\n", + "To achieve this, we will input a sequence of characters to the model, and train the model to predict the output, that is, the following character at each time step. RNNs maintain an internal state that depends on previously seen elements, so information about all characters seen up until a given moment will be taken into account in generating the prediction." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LFjSVAlWzf-N" + }, + "source": [ + "### Vectorize the text\n", + "\n", + "Before we begin training our RNN model, we'll need to create a numerical representation of our text-based dataset. To do this, we'll generate two lookup tables: one that maps characters to numbers, and a second that maps numbers back to characters. Recall that we just identified the unique characters present in the text.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "id": "IalZLbvOzf-F" + }, + "outputs": [], + "source": [ + "### Define numerical representation of text ###\n", + "\n", + "# Create a mapping from character to unique index.\n", + "# For example, to get the index of the character \"d\",\n", + "# we can evaluate `char2idx[\"d\"]`.\n", + "char2idx = {u: i for i, u in enumerate(vocab)}\n", + "\n", + "# Create a mapping from indices to characters. This is\n", + "# the inverse of char2idx and allows us to convert back\n", + "# from unique index to the character in our vocabulary.\n", + "idx2char = np.array(vocab)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "tZfqhkYCymwX" + }, + "source": [ + "This gives us an integer representation for each character. Observe that the unique characters (i.e., our vocabulary) in the text are mapped as indices from 0 to `len(unique)`. Let's take a peek at this numerical representation of our dataset:" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "FYyNlCNXymwY" + }, + "outputs": [ { - "cell_type": "code", - "source": [ - "### compute the loss on the predictions from the untrained model from earlier. ###\n", - "y.shape # (batch_size, sequence_length)\n", - "pred.shape # (batch_size, sequence_length, vocab_size)\n", - "\n", - "'''TODO: compute the loss using the true next characters from the example batch\n", - " and the predictions from the untrained model several cells above'''\n", - "example_batch_loss = compute_loss('''TODO''', '''TODO''') # TODO\n", - "\n", - "print(f\"Prediction shape: {pred.shape} # (batch_size, sequence_length, vocab_size)\")\n", - "print(f\"scalar_loss: {example_batch_loss.mean().item()}\")" - ], - "metadata": { - "id": "GuGUJB0ZT_Uo" - }, - "execution_count": null, - "outputs": [] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " '\\n': 0,\n", + " ' ' : 1,\n", + " '!' : 2,\n", + " '\"' : 3,\n", + " '#' : 4,\n", + " \"'\" : 5,\n", + " '(' : 6,\n", + " ')' : 7,\n", + " ',' : 8,\n", + " '-' : 9,\n", + " '.' : 10,\n", + " '/' : 11,\n", + " '0' : 12,\n", + " '1' : 13,\n", + " '2' : 14,\n", + " '3' : 15,\n", + " '4' : 16,\n", + " '5' : 17,\n", + " '6' : 18,\n", + " '7' : 19,\n", + " ...\n", + "}\n" + ] + } + ], + "source": [ + "print('{')\n", + "for char, _ in zip(char2idx, range(20)):\n", + " print(' {:4s}: {:3d},'.format(repr(char), char2idx[char]))\n", + "print(' ...\\n}')" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "g-LnKyu4dczc" + }, + "outputs": [], + "source": [ + "### Vectorize the songs string ###\n", + "\n", + "'''TODO: Write a function to convert the all songs string to a vectorized\n", + " (i.e., numeric) representation. Use the appropriate mapping\n", + " above to convert from vocab characters to the corresponding indices.\n", + "\n", + " NOTE: the output of the `vectorize_string` function\n", + " should be a np.array with `N` elements, where `N` is\n", + " the number of characters in the input string\n", + "'''\n", + "def vectorize_string(string):\n", + " return np.array([char2idx.get(i) for i in songs_joined])\n", + "\n", + "vectorized_songs = vectorize_string(songs_joined)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IqxpSuZ1w-ub" + }, + "source": [ + "We can also look at how the first part of the text is mapped to an integer representation:\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "l1VKcQHcymwb" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "0Seh7e6eRqd7" - }, - "source": [ - "Let's start by defining some hyperparameters for training the model. To start, we have provided some reasonable values for some of the parameters. It is up to you to use what we've learned in class to help optimize the parameter selection here!" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "'X:1\\nT:Alex' ---- characters mapped to int ----> [49 22 13 0 45 22 26 67 60 79]\n" + ] + } + ], + "source": [ + "print ('{} ---- characters mapped to int ----> {}'.format(repr(songs_joined[:10]), vectorized_songs[:10]))\n", + "# check that vectorized_songs is a numpy array\n", + "assert isinstance(vectorized_songs, np.ndarray), \"returned result should be a numpy array\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hgsVvVxnymwf" + }, + "source": [ + "### Create training examples and targets\n", + "\n", + "Our next step is to actually divide the text into example sequences that we'll use during training. Each input sequence that we feed into our RNN will contain `seq_length` characters from the text. We'll also need to define a target sequence for each input sequence, which will be used in training the RNN to predict the next character. For each input, the corresponding target will contain the same length of text, except shifted one character to the right.\n", + "\n", + "To do this, we'll break the text into chunks of `seq_length+1`. Suppose `seq_length` is 4 and our text is \"Hello\". Then, our input sequence is \"Hell\" and the target sequence is \"ello\".\n", + "\n", + "The batch method will then let us convert this stream of character indices to sequences of the desired size.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "LF-N8F7BoDRi" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "JQWUUhKotkAY" - }, - "outputs": [], - "source": [ - "### Hyperparameter setting and optimization ###\n", - "\n", - "vocab_size = len(vocab)\n", - "\n", - "# Model parameters:\n", - "params = dict(\n", - " num_training_iterations = 3000, # Increase this to train longer\n", - " batch_size = 8, # Experiment between 1 and 64\n", - " seq_length = 100, # Experiment between 50 and 500\n", - " learning_rate = 5e-3, # Experiment between 1e-5 and 1e-1\n", - " embedding_dim = 256,\n", - " hidden_size = 1024, # Experiment between 1 and 2048\n", - ")\n", - "\n", - "# Checkpoint location:\n", - "checkpoint_dir = './training_checkpoints'\n", - "checkpoint_prefix = os.path.join(checkpoint_dir, \"my_ckpt\")\n", - "os.makedirs(checkpoint_dir, exist_ok=True)" - ] + "name": "stdout", + "output_type": "stream", + "text": [ + "Batch function works correctly!\n" + ] }, { - "cell_type": "markdown", - "metadata": { - "id": "AyLzIPeAIqfg" - }, - "source": [ - "Having defined our hyperparameters we can set up for experiment tracking with Comet. [`Experiment`](https://www.comet.com/docs/v2/api-and-sdk/python-sdk/reference/Experiment/) are the core objects in Comet and will allow us to track training and model development. Here we have written a short function to create a new Comet experiment. Note that in this setup, when hyperparameters change, you can run the `create_experiment()` function to initiate a new experiment. All experiments defined with the same `project_name` will live under that project in your Comet interface.\n", - "\n" - ] - }, + "name": "stderr", + "output_type": "stream", + "text": [ + "C:\\Users\\savic\\AppData\\Local\\Temp\\ipykernel_28864\\1114461848.py:16: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at C:\\actions-runner\\_work\\pytorch\\pytorch\\pytorch\\torch\\csrc\\utils\\tensor_new.cpp:257.)\n", + " x_batch = torch.tensor(input_batch, dtype=torch.long)\n" + ] + } + ], + "source": [ + "### Batch definition to create training examples ###\n", + "\n", + "def get_batch(vectorized_songs, seq_length, batch_size):\n", + " # the length of the vectorized songs string\n", + " n = vectorized_songs.shape[0] - 1\n", + " # randomly choose the starting indices for the examples in the training batch\n", + " idx = np.random.choice(n - seq_length, batch_size)\n", + "\n", + " '''TODO: construct a list of input sequences for the training batch'''\n", + " input_batch = [vectorized_songs[i: i + seq_length] for i in idx]\n", + "\n", + " '''TODO: construct a list of output sequences for the training batch'''\n", + " output_batch = [vectorized_songs[i + 1: i + seq_length + 1] for i in idx]\n", + "\n", + " # Convert the input and output batches to tensors\n", + " x_batch = torch.tensor(input_batch, dtype=torch.long)\n", + " y_batch = torch.tensor(output_batch, dtype=torch.long)\n", + "\n", + " return x_batch, y_batch\n", + "\n", + "# Perform some simple tests to make sure your batch function is working properly!\n", + "test_args = (vectorized_songs, 10, 2)\n", + "x_batch, y_batch = get_batch(*test_args)\n", + "assert x_batch.shape == (2, 10), \"x_batch shape is incorrect\"\n", + "assert y_batch.shape == (2, 10), \"y_batch shape is incorrect\"\n", + "print(\"Batch function works correctly!\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_33OHL3b84i0" + }, + "source": [ + "For each of these vectors, each index is processed at a single time step. So, for the input at time step 0, the model receives the index for the first character in the sequence, and tries to predict the index of the next character. At the next timestep, it does the same thing, but the RNN considers the information from the previous step, i.e., its updated state, in addition to the current input.\n", + "\n", + "We can make this concrete by taking a look at how this works over the first several characters in our text:" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "0eBu9WZG84i0" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "MBsN1vvxInmN" - }, - "outputs": [], - "source": [ - "### Create a Comet experiment to track our training run ###\n", - "\n", - "def create_experiment():\n", - " # end any prior experiments\n", - " if 'experiment' in locals():\n", - " experiment.end()\n", - "\n", - " # initiate the comet experiment for tracking\n", - " experiment = comet_ml.Experiment(\n", - " api_key=COMET_API_KEY,\n", - " project_name=\"6S191_Lab1_Part2\")\n", - " # log our hyperparameters, defined above, to the experiment\n", - " for param, value in params.items():\n", - " experiment.log_parameter(param, value)\n", - " experiment.flush()\n", - "\n", - " return experiment" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Step 0\n", + " input: 60 ('e')\n", + " expected output: 56 ('a')\n", + "Step 1\n", + " input: 56 ('a')\n", + " expected output: 56 ('a')\n", + "Step 2\n", + " input: 56 ('a')\n", + " expected output: 62 ('g')\n", + "Step 3\n", + " input: 62 ('g')\n", + " expected output: 1 (' ')\n", + "Step 4\n", + " input: 1 (' ')\n", + " expected output: 56 ('a')\n" + ] + } + ], + "source": [ + "x_batch, y_batch = get_batch(vectorized_songs, seq_length=5, batch_size=1)\n", + "\n", + "for i, (input_idx, target_idx) in enumerate(zip(np.squeeze(x_batch), np.squeeze(y_batch))):\n", + " print(\"Step {:3d}\".format(i))\n", + " print(\" input: {} ({:s})\".format(input_idx, repr(idx2char[input_idx])))\n", + " print(\" expected output: {} ({:s})\".format(target_idx, repr(idx2char[target_idx])))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "r6oUuElIMgVx" + }, + "source": [ + "## 2.4 The Recurrent Neural Network (RNN) model" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "m8gPwEjRzf-Z" + }, + "source": [ + "Now we're ready to define and train an RNN model on our ABC music dataset, and then use that trained model to generate a new song. We'll train our RNN using batches of song snippets from our dataset, which we generated in the previous section.\n", + "\n", + "The model is based off the LSTM architecture, where we use a state vector to maintain information about the temporal relationships between consecutive characters. The final output of the LSTM is then fed into a fully connected linear [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) layer where we'll output a softmax over each character in the vocabulary, and then sample from this distribution to predict the next character.\n", + "\n", + "As we introduced in the first portion of this lab, we'll be using PyTorch's [`nn.Module`](https://pytorch.org/docs/stable/generated/torch.nn.Module.html) to define the model. Three components are used to define the model:\n", + "\n", + "* [`nn.Embedding`](https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html): This is the input layer, consisting of a trainable lookup table that maps the numbers of each character to a vector with `embedding_dim` dimensions.\n", + "* [`nn.LSTM`](https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html): Our LSTM network, with size `hidden_size`.\n", + "* [`nn.Linear`](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html): The output layer, with `vocab_size` outputs.\n", + "\n", + "\"Drawing\"/\n", + "\n", + "\n", + "\n", + "" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rlaOqndqBmJo" + }, + "source": [ + "### Define the RNN model\n", + "\n", + "Let's define our model as an `nn.Module`. Fill in the `TODOs` to define the RNN model.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "id": "8DsWzojvkbc7" + }, + "outputs": [], + "source": [ + "### Defining the RNN Model ###\n", + "\n", + "'''TODO: Add LSTM and Linear layers to define the RNN model using nn.Module'''\n", + "class LSTMModel(nn.Module):\n", + " def __init__(self, vocab_size, embedding_dim, hidden_size):\n", + " super(LSTMModel, self).__init__()\n", + " self.hidden_size = hidden_size\n", + "\n", + " # Define each of the network layers\n", + " # Layer 1: Embedding layer to transform indices into dense vectors\n", + " # of a fixed embedding size\n", + " self.embedding = nn.Embedding(vocab_size, embedding_dim)\n", + "\n", + " # Layer 2: LSTM with hidden_size `hidden_size`. note: number of layers defaults to 1.\n", + " # TODO: Use the nn.LSTM() module from pytorch.\n", + " self.lstm = nn.LSTM(embedding_dim, hidden_size, batch_first=True)\n", + "\n", + " # Layer 3: Linear (fully-connected) layer that transforms the LSTM output\n", + " # into the vocabulary size.\n", + " # TODO: Add the Linear layer.\n", + " self.fc = nn.Linear(hidden_size, vocab_size)\n", + "\n", + " def init_hidden(self, batch_size, device):\n", + " # Initialize hidden state and cell state with zeros\n", + " return (torch.zeros(1, batch_size, self.hidden_size).to(device),\n", + " torch.zeros(1, batch_size, self.hidden_size).to(device))\n", + "\n", + " def forward(self, x, state=None, return_state=False):\n", + " x = self.embedding(x)\n", + "\n", + " if state is None:\n", + " state = self.init_hidden(x.size(0), x.device)\n", + " out, state = self.lstm(x, state)\n", + "\n", + " out = self.fc(out)\n", + " return out if not return_state else (out, state)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IbWU4dMJmMvq" + }, + "source": [ + "The time has come! Let's instantiate the model!" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "MtCrdfzEI2N0" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "5cu11p1MKYZd" - }, - "source": [ - "Now, we are ready to define our training operation -- the optimizer and duration of training -- and use this function to train the model. You will experiment with the choice of optimizer and the duration for which you train your models, and see how these changes affect the network's output. Some optimizers you may like to try are [`Adam`](https://pytorch.org/docs/stable/generated/torch.optim.Adam.html) and [`Adagrad`](https://pytorch.org/docs/stable/generated/torch.optim.Adagrad.html).\n", - "\n", - "First, we will instantiate a new model and an optimizer, and ready them for training. Then, we will use [`loss.backward()`](https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html), enabled by PyTorch's [autograd](https://pytorch.org/docs/stable/generated/torch.autograd.grad.html) method, to perform the backpropagation. Finally, to update the model's parameters based on the computed gradients, we will utake a step with the optimizer, using [`optimizer.step()`](https://pytorch.org/docs/stable/generated/torch.optim.Optimizer.step.html).\n", - "\n", - "We will also generate a print-out of the model's progress through training, which will help us easily visualize whether or not we are minimizing the loss." - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "LSTMModel(\n", + " (embedding): Embedding(83, 256)\n", + " (lstm): LSTM(256, 1024, batch_first=True)\n", + " (fc): Linear(in_features=1024, out_features=83, bias=True)\n", + ")\n" + ] + } + ], + "source": [ + "# Instantiate the model! Build a simple model with default hyperparameters. You\n", + "# will get the chance to change these later.\n", + "vocab_size = len(vocab)\n", + "embedding_dim = 256\n", + "hidden_size = 1024\n", + "batch_size = 8\n", + "\n", + "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n", + "\n", + "model = LSTMModel(vocab_size, embedding_dim, hidden_size).to(device)\n", + "\n", + "# print out a summary of the model\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-ubPo0_9Prjb" + }, + "source": [ + "### Test out the RNN model\n", + "\n", + "It's always a good idea to run a few simple checks on our model to see that it behaves as expected. \n", + "\n", + "We can quickly check the layers in the model, the shape of the output of each of the layers, the batch size, and the dimensionality of the output. Note that the model can be run on inputs of any length." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "id": "C-_70kKAPrPU" + }, + "outputs": [ { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "F31vzJ_u66cb" - }, - "outputs": [], - "source": [ - "### Define optimizer and training operation ###\n", - "\n", - "'''TODO: instantiate a new LSTMModel model for training using the hyperparameters\n", - " created above.'''\n", - "model = LSTMModel('''TODO: arguments''')\n", - "\n", - "# Move the model to the GPU\n", - "model.to(device)\n", - "\n", - "'''TODO: instantiate an optimizer with its learning rate.\n", - " Checkout the PyTorch website for a list of supported optimizers.\n", - " https://pytorch.org/docs/stable/optim.html\n", - " Try using the Adam optimizer to start.'''\n", - "optimizer = # TODO\n", - "\n", - "def train_step(x, y):\n", - " # Set the model's mode to train\n", - " model.train()\n", - "\n", - " # Zero gradients for every step\n", - " optimizer.zero_grad()\n", - "\n", - " # Forward pass\n", - " '''TODO: feed the current input into the model and generate predictions'''\n", - " y_hat = model('''TODO''')\n", - "\n", - " # Compute the loss\n", - " '''TODO: compute the loss!'''\n", - " loss = compute_loss('''TODO''', '''TODO''')\n", - "\n", - " # Backward pass\n", - " '''TODO: complete the gradient computation and update step.\n", - " Remember that in PyTorch there are two steps to the training loop:\n", - " 1. Backpropagate the loss\n", - " 2. Update the model parameters using the optimizer\n", - " '''\n", - " '''TODO'''\n", - "\n", - " return loss\n", - "\n", - "##################\n", - "# Begin training!#\n", - "##################\n", - "\n", - "history = []\n", - "plotter = mdl.util.PeriodicPlotter(sec=2, xlabel='Iterations', ylabel='Loss')\n", - "experiment = create_experiment()\n", - "\n", - "if hasattr(tqdm, '_instances'): tqdm._instances.clear() # clear if it exists\n", - "for iter in tqdm(range(params[\"num_training_iterations\"])):\n", - "\n", - " # Grab a batch and propagate it through the network\n", - " x_batch, y_batch = get_batch(vectorized_songs, params[\"seq_length\"], params[\"batch_size\"])\n", - "\n", - " # Convert numpy arrays to PyTorch tensors\n", - " x_batch = torch.tensor(x_batch, dtype=torch.long).to(device)\n", - " y_batch = torch.tensor(y_batch, dtype=torch.long).to(device)\n", - "\n", - " # Take a train step\n", - " loss = train_step(x_batch, y_batch)\n", - "\n", - " # Log the loss to the Comet interface\n", - " experiment.log_metric(\"loss\", loss.item(), step=iter)\n", - "\n", - " # Update the progress bar and visualize within notebook\n", - " history.append(loss.item())\n", - " plotter.plot(history)\n", - "\n", - " # Save model checkpoint\n", - " if iter % 100 == 0:\n", - " torch.save(model.state_dict(), checkpoint_prefix)\n", - "\n", - "# Save the final trained model\n", - "torch.save(model.state_dict(), checkpoint_prefix)\n", - "experiment.flush()\n" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Input shape: torch.Size([32, 100]) # (batch_size, sequence_length)\n", + "Prediction shape: torch.Size([32, 100, 83]) # (batch_size, sequence_length, vocab_size)\n" + ] + } + ], + "source": [ + "# Test the model with some sample data\n", + "x, y = get_batch(vectorized_songs, seq_length=100, batch_size=32)\n", + "x = x.to(device)\n", + "y = y.to(device)\n", + "\n", + "pred = model(x)\n", + "print(\"Input shape: \", x.shape, \" # (batch_size, sequence_length)\")\n", + "print(\"Prediction shape: \", pred.shape, \"# (batch_size, sequence_length, vocab_size)\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "mT1HvFVUGpoE" + }, + "source": [ + "### Predictions from the untrained model\n", + "\n", + "Let's take a look at what our untrained model is predicting.\n", + "\n", + "To get actual predictions from the model, we sample from the output distribution, which is defined by a torch.softmax over our character vocabulary. This will give us actual character indices. This means we are using a [categorical distribution](https://en.wikipedia.org/wiki/Categorical_distribution) to sample over the example prediction. This gives a prediction of the next character (specifically its index) at each timestep. [`torch.multinomial`](https://pytorch.org/docs/stable/generated/torch.multinomial.html#torch.multinomial) samples over a categorical distribution to generate predictions.\n", + "\n", + "Note here that we sample from this probability distribution, as opposed to simply taking the `argmax`, which can cause the model to get stuck in a repetitive loop.\n", + "\n", + "Let's try this sampling out for the first example in the batch." + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "id": "4V4MfFg0RQJg" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "kKkD5M6eoSiN" - }, - "source": [ - "## 2.6 Generate music using the RNN model\n", - "\n", - "Now, we can use our trained RNN model to generate some music! When generating music, we'll have to feed the model some sort of seed to get it started (because it can't predict anything without something to start with!).\n", - "\n", - "Once we have a generated seed, we can then iteratively predict each successive character (remember, we are using the ABC representation for our music) using our trained RNN. More specifically, recall that our RNN outputs a `softmax` over possible successive characters. For inference, we iteratively sample from these distributions, and then use our samples to encode a generated song in the ABC format.\n", - "\n", - "Then, all we have to do is write it to a file and listen!" + "data": { + "text/plain": [ + "array([67, 25, 49, 58, 40, 54, 8, 66, 71, 0, 7, 79, 78, 38, 74, 50, 37,\n", + " 48, 4, 21, 20, 13, 67, 42, 63, 25, 55, 75, 36, 58, 52, 46, 12, 40,\n", + " 80, 67, 42, 65, 75, 44, 58, 47, 4, 0, 15, 13, 38, 82, 28, 70, 12,\n", + " 70, 63, 78, 24, 36, 27, 66, 16, 52, 72, 32, 4, 33, 35, 5, 51, 23,\n", + " 59, 34, 29, 61, 32, 22, 68, 38, 65, 55, 71, 70, 3, 35, 14, 42, 20,\n", + " 8, 2, 24, 54, 24, 81, 5, 10, 45, 21, 66, 17, 2, 2, 35],\n", + " dtype=int64)" ] - }, + }, + "execution_count": 15, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sampled_indices = torch.multinomial(torch.softmax(pred[0], dim=-1), num_samples=1)\n", + "sampled_indices = sampled_indices.squeeze(-1).cpu().numpy()\n", + "sampled_indices" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LfLtsP3mUhCG" + }, + "source": [ + "We can now decode these to see the text predicted by the untrained model:" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "id": "xWcFwPwLSo05" + }, + "outputs": [ { - "cell_type": "markdown", - "metadata": { - "id": "DjGz1tDkzf-u" - }, - "source": [ - "### The prediction procedure\n", - "\n", - "Now, we're ready to write the code to generate text in the ABC music format:\n", - "\n", - "* Initialize a \"seed\" start string and the RNN state, and set the number of characters we want to generate.\n", - "\n", - "* Use the start string and the RNN state to obtain the probability distribution over the next predicted character.\n", - "\n", - "* Sample from multinomial distribution to calculate the index of the predicted character. This predicted character is then used as the next input to the model.\n", - "\n", - "* At each time step, the updated RNN state is fed back into the model, so that it now has more context in making the next prediction. After predicting the next character, the updated RNN states are again fed back into the model, which is how it learns sequence dependencies in the data, as it gets more information from the previous predictions.\n", - "\n", - "![LSTM inference](https://raw.githubusercontent.com/MITDeepLearning/introtodeeplearning/2019/lab1/img/lstm_inference.png)\n", - "\n", - "Complete and experiment with this code block (as well as some of the aspects of network definition and training!), and see how the model performs. How do songs generated after training with a small number of epochs compare to those generated after a longer duration of training?" - ] - }, + "name": "stdout", + "output_type": "stream", + "text": [ + "Input: \n", + " 'agf gag|fdf ecA|agf gag|fdf efg|!\\nagf gag|fdf ece|fdf gec|Adc d2:|!\\n\\nX:65\\nT:Humours of Ennistymon\\nZ:'\n", + "\n", + "Next Char Predictions: \n", + " 'l>XcO^,kp\\n)xwMsYLW#981lQh>_tKc[U0OylQjtScV#\\n31M|Co0ohw=KBk4[qG#HJ\\'Z