03_tensors.Rmd

# Tensors

**Learning objectives:**

Learn how to:

- Create and modify tensors
- Access parts of tensors
- Apply operations to tensors

## What are tensors? {-}

From the book:

> "... *tensors* are 'just' multi-dimensional arrays optimized for fast computation - 
not on the CPU only but also on specialized devices such as GPUs and TPUs.

Load libraries:

```{r message = FALSE, warning=FALSE}
library(torch)
library(dplyr)
library(ggplot2)
```

Create a tensor:

```{r}
t1 <- torch_tensor(2)
t1$shape
```

Parameters of `torch_tensor()` (see help file for this function):

* `data`
* `dtype`
* `device`
* `requires_grad`
* `pin_memory`

Look at the attributes of the tensor:

```{r}
t1$dtype
t1$device
t1$shape
summary(t1)
```

Change attributes:

```{r}
# to integer:
t2 <- t1$to(dtype = torch_int())
t2$dtype
```

```{r eval=FALSE}
# utilize GPU:
t2 <- t1$to(device = "cuda")
t2$device
```


## Creating tensors

### Tensors from values

**Defaults**: long integer type and CPU device

Let's test this:

```{r}
torch_tensor(1:2)
```
In the example above, the type is long integer.

```{r}
torch_tensor(c(1,2))
```

In this example, the type is float instead of a long integer. 
My guess is that the difference is due to the difference in data types of `1:2` and `c(1,2)`:

```{r}
class(c(1,2))
class(1:2)
```

Explicitly set the type and device:

```{r eval = FALSE}
torch_tensor(1:5, dtype = torch_float(), device = "cuda")
```

Two-dimensional tensor:

```{r}
torch_tensor(matrix(1:9, ncol = 3, byrow = TRUE))
```

Higher dimensions:

```{r}
torch_tensor(array(1:24, dim = c(4,3,2)))
```

### Tensors from specifications

```{r}
#normal distribution:
torch_randn(3, 3)

#unfiorm distribution
torch_rand(3, 3)
```

```{r}
torch_zeros(2,2)
torch_ones(3,3)
torch_eye(3,3)
torch_diag(c(1,2,3))
```

See full list at https://torch.mlverse.org/docs/reference/#tensor-creation-utilities. 

### Tensors from datasets

We'll look at the `presidential` dataset from the `ggplot2` package:

```{r}
data(presidential) # from ggplot2 
presidential |> glimpse()
```

```{r}
presidential |> 
  mutate(name = as.numeric(factor(name)),
         start = as.numeric(start),
         end = as.numeric(end),
         party = as.numeric(factor(party))) |>
  as.matrix() |>
  torch_tensor() |>
  print(5)
```

If your data contains `NA`'s, you'll need to convert them before training a neural network 
(topic to be covered later).

## Operations on Tensors

Define two tensors:

```{r}
t1 <- torch_tensor(c(1,2))
t2 <- torch_tensor(c(3,4))
```

Add:

```{r}
torch_add(t1, t2) # t1 is NOT modified
t1$add(t2) # t1 is NOT modified
t1$add_(t2) # t1 IS modified
t1
```

In general: 

* underscore appended to operation indicates modification in-place.
* torch does not distinguish between row and column vectors.
* Other examples of operations: `torch_t()`, `torch_dot()`, `torch_matmul()`, and `torch_multiply()`. 
See https://torch.mlverse.org/docs/reference/#mathematical-operations-on-tensors.

Another example:

```{r}
t1 <- torch_tensor(1:3)
t2 <- torch_tensor(4:6)
```

```{r}
t1$dot(t2)
```

### Summary operations

Create a matrix and a tensor using outer products:

```{r}
m <- outer(1:3, 1:6)
t <- torch_outer(torch_tensor(1:3), torch_tensor(1:6))
```

```{r}
apply(m, 1, sum) # row sums (of R matrix)
t$sum(dim = 2) # row sums (of tensor)
```

* In R, we *group* by row (dimension 1) for row summaries and by columns (dimension 2) for column summaries
* In `torch, we *collapse* the columns (dimension 2) for row summaries and the rows (dimension 1) for column summaries.

Time series example: Two features collected three times for four individuals. 

* Dimension 1: Runs over individuals
* Dimension 2: Runs over points in time
* Dimension 3: Runs over features

```{r}
t <- torch_randn(4, 3, 2)
t
```

Obtain averages of features, independent of subject (dimension 1) and time (dimension 2):

```{r}
t$mean(dim = c(1, 2))
```


## Accessing Parts of a Tensor (indexing and slicing)

Indexing in `torch` is 1-based. Very similar to functionality in R.

Example:

```{r}
t <- torch_tensor(matrix(1:9, ncol = 3, byrow = TRUE))
t
t[1, ] # row 1. 
t[1, , drop = FALSE] # Same as above except that dimensionality is preserved
```

Slicing example:

```{r}
t <- torch_rand(3, 3, 3)
t[1:2, 2:3, c(1, 3)]
```

### Beyond R

Access last element:

```{r}
t <- torch_tensor(matrix(1:4, ncol = 2, byrow = TRUE))
t[-1, -1]
```

```{r}
t <- torch_tensor(1:4)
t[-1]
```

Compare to R:

```{r}
# matrix:
m <- matrix(1:4, ncol = 2, byrow = TRUE)
m[-1, -1]

# vector:
m <- 1:4
m[-1]
```

Step pattern:

```{r}
t <- torch_tensor(matrix(1:20, ncol = 10, byrow = TRUE))
t
t[ , 1:8:2] # every other value in columns 1 through 8
```

Use `..` to designate all dimensions not explicitly referenced.

```{r}
t2 <- torch_randn(2, 2, 2)
t2
t2[2, ..] # 2nd element of the 1st dimension
```

## Reshaping Tensors

Two methods:

* `view`: Zero-copy reshape (by changing tensor metadata). Will fail if zero-copy reshape is not possible.
* `reshape`: Uses zero-copy reshape (via metadata) when possible. If not, then will make a copy.

```{r}
t <- torch_tensor(matrix(1:15, nrow = 3, byrow = TRUE))
t
t$stride()
```

`$stride()` tells us the jump necessary to go from one element to the next in a single dimension (e.g. the stride necessary to get from one row to the next).

Next, change shape of `t` using `view()`:

```{r}
t2 <- t$view(c(5,3))
t2
t2$stride()
```


Check memory location:

```{r}
t$storage()$data_ptr()
t2$storage()$data_ptr() # same location
```


`view()` vs `reshape()`:

When two operations that change the stride are done in sequence, this will likely fail.
Below is an example of transpose, `t()` followed by `view()`.

```{r eval = FALSE}
t$t()$view(15) # error
```

However, `reshape()` makes a copy of the underlying data and does not fail.

```{r}
t3 <- t$t()$reshape(15) # no error
t3
```


Now the memory locations of t3 and the original data are different:

```{r}
t$storage()$data_ptr()
t3$storage()$data_ptr() # different location
```

Two additional functions that are zero-copy operations:

* `squeeze()`: Removes singleton dimensions
* `unsqueeze()`: Adds a singleton dimension

```{r}
t <- torch_rand(3) # one dimensional tensor of shape 3
t
t2 <- t$unsqueeze(1) # two dimensional tensor of shape {1,3}
t2
```
Inspect location:

```{r}
t$storage()$data_ptr()
t2$storage()$data_ptr() # same location
```


## Broadcasting

Example: Want to add two tensors of shape 3x7x1 and 1x5

```
t1 shape: 3 7 1
t2 shape:   1 5
```

Broadcast (from right to left):

```
t1 shape: 3 7 5
t2 shape:   7 5
```

Next, virtual expansion:

```
t1 shape: 3 7 5
t2 shape: 1 7 5
```

And, broadcast again:

```
t1 shape: 3 7 5
t2 shape: 3 7 5
```

Let's see it in action:

```{r}
t1 <- torch_ones(3, 7, 1)
t2 <- torch_zeros(1, 5)
torch_add(t1, t2)
```

The following will NOT work:

```{r eval = FALSE}
t1 <- torch_ones(3, 7, 1)
t2 <- torch_zeros(6, 5)
torch_add(t1, t2)
```



## Meeting Videos {-}

### Cohort 1 {-}

`r knitr::include_url("https://www.youtube.com/embed/URL")`

<details>
<summary> Meeting chat log </summary>

```
LOG
```
</details>