DCGAN_pytorch (pclubiitk#18)

* Create README.md

* Update README.md

* Update README.md

* Update README.md

* Update README.md

* Create temp

* Add files via upload

* Create models.py

* Create utils.py

* Create main.py

* Update utils.py

* Update main.py

* Update main.py

* Update main.py

* Delete temp

* Delete generated img.png

* Update main.py

* Update main.py

* Update main.py

* Update README.md

* Update README.md

* name changed to assets

* Update README.md

Author: nakul-jindal

generative_models/DCGAN_pytorch/README.md

@@ -0,0 +1,127 @@
+# DCGAN implementation in pytorch on MNIST
+
+## Usage
+```bash
+$ python3 main.py --num_epochs 10
+```
+> **_NOTE:_** on Colab Notebook use following command:
+```python
+!git clone link-to-repo
+%run main.py --num_epochs 10
+```
+
+## Help log
+```
+usage: main.py [-h] [--num_epochs NUM_EPOCHS]
+               [--batch_size  BATCH_SIZE] 
+               [--channels_noise CHANNELS_NOISE] 
+               [--lr_g LR_G][--lr_d LR_D]
+               [--beta1 BETA1]
+
+optional arguments:
+  -h, --help                           show this help message and exit
+  --num_epochs NUM_EPOCHS              no. of epochs : default=10
+  --batch_size  BATCH_SIZE             batch size : default=128
+  --channels_noise CHANNELS_NOISE      size of noise vector : default=100
+  --lr_g LR_G                          learning rate generator : default=0.0002
+  --lr_d LR_D                          learning rate discriminator : default=0.0002
+  --beta1 BETA1                        bet1 value for adam optimizer
+                        
+  ```
+
+## contributed by : 
+* [Nakul Jindal](https://github.com/nakul-jindal)
+
+## References
+
+* **Title**: UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS
+* **Authors**: Alec Radford, Luke Metz, Soumith Chintala
+* **Link**: https://arxiv.org/pdf/1511.06434.pdf
+* **Tags**: Neural Network, Generative Networks, GANs
+* **Year**: 2015
+
+
+# summary
+
+## Introduction
+* Deep Convolution Generative Adversarial Networks (DCGANs) belong to a set of algorithms called generative models, which are widely used for unupervised learning tasks which aim to learn the underlying structure of the given data. 
+
+* Simple GANs allow you to generate new unseen data that mimic the actual given real data. However, GANs pose problems in training and require carefullly tuned hyperparameters.
+
+* DCGAN aims to solve this problem by explicitly using convolutional and convolutional-transpose layers in the discriminator and generator, respectively. 
+
+* DCGANs basically convert the laplacian pyramid technique (many pairs of G and D to progressively upscale an image) to a single pair of G and D.
+
+## Generator
+ 
+* The generator `G` is designed to map the latent space vector `z` (random noise) to data-space (images same as training images) 
+* involves a series of transpose Conv2d layers, each with BatchNorm2d and relu activation.
+* The output of the generator is fed through a tanh function to return it to the input data range of `[-1,1]`.
+
+## Discriminator
+
+* The discriminator `D` is a binary classification network that takes an image as input and outputs a scalar probability that the input image is real or fake. 
+* `D` involves a series of Conv2d, BatchNorm2d, and LeakyReLU layers.
+* outputs the final probability through a Sigmoid activation function.
+
+
+> The DCGAN paper mentions it is a good practice to use strided convolution rather than pooling to downsample because it lets the network learn its own pooling function. Also batch norm and leaky relu functions promote healthy gradient flow which is critical for the learning process of both `G` and `D`.
+
+
+## Model Architecture
+
+![architecture](assets/architecture.png)
+ 
+## Network Design of DCGAN:
+* Replace all pooling layers with strided convolutions for the downsampling
+* Remove all fully connected layers.
+* Use transposed convolutions for upsampling. 
+* Use Batch Normalization after every layer except after the output layer of the generator and the input layer of the discriminator.
+* Use ReLU non-linearity for each layer in the generator except for output layer use tanh.
+* Use Leaky-ReLU non-linearity for each layer of the disciminator excpet for output layer use sigmoid.
+
+## Hyperparameters for this Implementation
+Hyperparameters are chosen as given in the paper.
+* mini-batch size: 128
+* learning rate: 0.0002
+* momentum term beta1: 0.5
+* slope of leak of LeakyReLU: 0.2
+* For the optimizer Adam (with beta2 = 0.999) has been used instead of SGD as described in the paper.
+
+## MNIST vs Generated images
+
+<table align='center'>
+<tr align='center'>
+<td> MNIST </td>
+<td> DCGAN after 10 epochs </td>
+</tr>
+<tr>
+<td><img src = 'assets/raw_MNIST.png'>
+<td><img src = 'assets/MNIST_DCGAN_10.png'>
+</tr>
+</table>
+
+## Training loss
+
+![Loss](assets/loss.png)
+
+## contributions of the research paper
+
+* proposes and evaluates Deep Convolutional GANs (DCGAN) which are a set of constraints on the architectural topology of Convolutional
+GANs that make them stable to train in most settings.
+
+* use of trained discriminators for image classification tasks, showing competitive performance with other unsupervised algorithms.
+
+* visualize the filters learnt by GANs and empirically show that specific filters have learned to draw specific objects.
+
+* show that the generators have interesting vector arithmetic properties allowing for easy manipulation of many semantic qualities of generated samples.
+
+## Conclusion of research paper
+
+This paper shows how convolutional layers can be used with GANs and provides a series of additional architectural guidelines for doing this. The paper also discusses topics such as Visualizing GAN features, Latent space interpolation, using discriminator features to train classifiers, and evaluating results. The paper contains many examples of images generated by final and intermediate layers of the network.
+
+#### key observations
+* Images in the latent space do not show sharp transitions indicating that network did not memorize images.
+* DCGAN can learn an interesting hierarchy of features.
+* Networks seems to have some success in disentangling image representation from object representation.
+* Vector arithmetic can be performed on the Z vectors corresponding to the face samples to get results like `smiling woman - normal woman + normal man = smiling man` visually.

generative_models/DCGAN_pytorch/assets/MNIST_DCGAN_10.png

@@ -0,0 +1,110 @@
+import torch
+import torchvision
+import torch.nn as nn                       # All neural network modules, nn.Linear, nn.Conv2d, BatchNorm, Loss functions
+import torch.optim as optim                 # For all Optimization algorithms, SGD, Adam, etc.
+import torchvision.datasets as datasets     # Has standard datasets we can import in a nice way
+import torchvision.transforms as transforms # Transformations we can perform on our dataset
+from torch.utils.data import DataLoader     # Gives easier dataset managment and creates mini batches
+
+import argparse
+parser = argparse.ArgumentParser()
+parser.add_argument('--num_epochs', type=int, default=10 , help="no. of epochs : default=10")
+parser.add_argument('--batch_size', type=int, default=128, help="batch size : default=128")
+parser.add_argument('--channels_noise', type=int, default=100, help="size of noise vector : default=100")
+parser.add_argument('--lr_g', type=float, default=0.0002, help="learning rate generator : default=0.0002")
+parser.add_argument('--lr_d', type=float, default=0.0002, help="learning rate discriminator : default=0.0002")
+parser.add_argument('--beta1', type=float, default=0.5, help="bet1 value for adam optimizer" )
+args = parser.parse_args()
+
+lr_g = args.lr_g
+lr_d = args.lr_d
+beta1 = args.beta1
+batch_size = args.batch_size
+channels_noise = args.channels_noise
+num_epochs = args.num_epochs
+
+image_size = 64
+features_d = 128
+features_g = 128
+
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+my_transforms = transforms.Compose([
+    transforms.Resize(image_size),
+    transforms.ToTensor(),
+    transforms.Normalize((0.5,),(0.5,)),
+    ])
+
+dataset = datasets.MNIST(root='dataset/', train=True, transform=my_transforms, download=True)
+dataloader = DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=2)
+
+from models import Generator , Discriminator , weights_init
+netD = Discriminator(channels_img, features_d).to(device)
+netG = Generator(channels_noise, channels_img, features_g).to(device)
+netG=netG.apply(weights_init)
+netD=netD.apply(weights_init)
+
+optimizerD = optim.Adam(netD.parameters(), lr=lr_d, betas=(beta1, 0.999) )
+optimizerG = optim.Adam(netG.parameters(), lr=lr_g, betas=(beta1, 0.999) )
+
+criterion = nn.BCELoss()
+
+real_label = 1
+fake_label = 0
+
+img_list = []
+G_losses = []
+D_losses = []
+
+for epoch in range(num_epochs):
+    for batch_idx, (data, targets) in enumerate(dataloader):
+        
+        data = data.to(device)
+        batch_size = data.shape[0]
+        
+        # Train Discriminator: max log(D(x)) + log(1 - D(G(z)))
+        netD.zero_grad()
+        label = (torch.ones(batch_size)*0.9).to(device)
+        output = netD(data).view(-1)
+        lossD_real = criterion(output, label)
+        D_x = output.mean().item()
+        
+        noise = torch.randn(batch_size, channels_noise, 1, 1).to(device)
+        fake = netG(noise)
+        label = (torch.ones(batch_size)*0.1).to(device)
+        output = netD(fake.detach()).view(-1)
+        lossD_fake = criterion(output, label)
+        
+        lossD = lossD_real + lossD_fake
+        lossD.backward()
+        optimizerD.step()
+        
+        # Train Generator: max log(D(G(z)))
+        netG.zero_grad()
+        label = torch.ones(batch_size).to(device)
+        output = netD(fake).reshape(-1)
+        lossG = criterion(output, label)
+        lossG.backward()
+        optimizerG.step()
+        D_G_x = output.mean().item()
+        
+         
+        if batch_idx % 100 == 0:
+            # Print losses ocassionally
+            print(f'Epoch [{epoch}/{num_epochs}] Batch {batch_idx}/{len(dataloader)}  Loss D: {lossD:.4f}  loss G: {lossG:.4f}  D(x): {D_x:.4f}  D(G(z)): {D_G_x:.4f} ')
+            G_losses.append(lossG.item())
+            D_losses.append(lossD.item())
+            
+            # Check how the generator is doing by saving G's output on fixed_noise
+            with torch.no_grad():
+                fake = netG(fixed_noise).detach().cpu()
+            img_list.append(torchvision.utils.make_grid(fake, padding=2, normalize=True))
+   
+
+
+from utils import compare_img , plot_loss , animation
+plot_loss(G_losses,D_losses)   # visualise losses vs iterations
+compare_img(data,fake)         # compare generated imgs with real mnist images
+animation(img_list)            # visualise generated images on a fixed noise at intervals
+
+

generative_models/DCGAN_pytorch/assets/architecture.png

@@ -0,0 +1,70 @@
+import torch
+import torch.nn as nn
+
+# weight initialisation with mean=0 and stddev=0.02
+
+def weights_init(m):
+    classname = m.__class__.__name__
+    if classname.find('Conv') != -1:
+        nn.init.normal_(m.weight.data, 0.0, 0.02)
+    elif classname.find('BatchNorm') != -1:
+        nn.init.normal_(m.weight.data, 1.0, 0.02)
+        nn.init.constant_(m.bias.data, 0)
+        
+class Generator(nn.Module):
+    def __init__(self, channels_noise, channels_img, features_g):
+        super(Generator, self).__init__()
+        
+        self.net = nn.Sequential(
+            
+            nn.ConvTranspose2d(channels_noise, features_g*8, kernel_size=4, stride=1, padding=0 , bias = False ),
+            nn.BatchNorm2d(features_g*8),
+            nn.ReLU(True),
+            
+            nn.ConvTranspose2d(features_g*8, features_g*4, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.BatchNorm2d(features_g*4),
+              nn.ReLU(True),
+            
+            nn.ConvTranspose2d(features_g*4, features_g*2, kernel_size=4, stride=2, padding=1  , bias = False),
+            nn.BatchNorm2d(features_g*2),
+             nn.ReLU(True),
+            
+            nn.ConvTranspose2d(features_g*2, features_g, kernel_size=4, stride=2, padding=1  , bias = False),
+            nn.BatchNorm2d(features_g),
+            nn.ReLU(True),
+            
+            nn.ConvTranspose2d(features_g, channels_img, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.Tanh()
+            )      
+    
+    def forward(self, x):
+        return self.net(x)        
+        
+class Discriminator(nn.Module):
+    def __init__(self, channels_img, features_d):
+        super(Discriminator, self).__init__()
+        self.net = nn.Sequential(
+            
+            nn.Conv2d(channels_img, features_d, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.LeakyReLU(0.2, inplace=True),
+
+            nn.Conv2d(features_d, features_d*2, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.BatchNorm2d(features_d*2),
+            nn.LeakyReLU(0.2, inplace=True),
+
+            nn.Conv2d(features_d*2, features_d*4, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.BatchNorm2d(features_d*4),
+            nn.LeakyReLU(0.2, inplace=True),
+
+            nn.Conv2d(features_d*4, features_d*8, kernel_size=4, stride=2, padding=1 , bias = False ),
+            nn.BatchNorm2d(features_d*8),
+            nn.LeakyReLU(0.2, inplace=True),
+
+            nn.Conv2d(features_d*8, 1, kernel_size=4, stride=2, padding=0 , bias = False ),
+            nn.Sigmoid()
+            )  
+    
+    def forward(self, x):
+        return self.net(x)
+        
+        

generative_models/DCGAN_pytorch/assets/loss.png

@@ -0,0 +1,30 @@
+import numpy as np
+import torchvision
+import matplotlib.pyplot as plt
+import matplotlib.animation as animation
+from IPython.display import HTML
+
+device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
+
+def animation(img_list):
+  fig = plt.figure(figsize=(8,8))
+  plt.axis("off")
+  ims = [[plt.imshow(np.transpose(i,(1,2,0)), animated=True)] for i in img_list]
+  ani = animation.ArtistAnimation(fig, ims, interval=1000, repeat_delay=1000, blit=True)
+  HTML(ani.to_jshtml())
+  
+def compare_img(data,fake):
+  img_grid_real = torchvision.utils.make_grid(data[:32], normalize=True).cpu()
+  img_grid_fake = torchvision.utils.make_grid(fake[:32], normalize=True).cpu()
+  plt.imshow(img_grid_fake.permute(1, 2, 0))
+  plt.imshow(img_grid_real.permute(1, 2, 0))
+  
+def plot_loss(G_losses,D_losses):
+  plt.figure(figsize=(10,5))
+  plt.title("Generator and Discriminator Loss During Training")
+  plt.plot(G_losses,label="G")
+  plt.plot(D_losses,label="D")
+  plt.xlabel("iterations")
+  plt.ylabel("Loss")
+  plt.legend()
+  plt.show()