Merge pull request ilkarman#81 from ThomasDelteil/gluon_multigpu

Gluon multigpu notebook

Author: Ilia Karmanov

.gitignore

@@ -5,4 +5,6 @@
 cifar-10-batches-py/
 __pycache__
 .DS_Store
-
+notebooks/chestxray
+notebooks/*-0000.params
+notebooks/*-symbol.json

README.md

@@ -1,5 +1,5 @@
 # Deep Learning Framework Examples
-
+   
 <p align="center">
 <img src="support/logo.png" alt="logo" width="50%"/>
 </p>
@@ -59,7 +59,7 @@ This is a work in progress
 | [Keras(TF)](notebooks/Keras_TF_MultiGPU.ipynb)    | 51min                 | 22min                 |
 | [Tensorflow](notebooks/Tensorflow_MultiGPU.ipynb) | 50min                 | 25min                 |
 | [Chainer](notebooks/Chainer_MultiGPU.ipynb)       | 65min                 | ?                     |
-| [MXNet(Gluon)]()                                  | ?                     | ?                     |
+| [MXNet(Gluon)](notebooks/Gluon_MultiGPU.ipynb)    | TBA                   | TBA                   |
 
 **Train w/ synthetic-data**
 
@@ -69,7 +69,7 @@ This is a work in progress
 | [Keras(TF)](notebooks/Keras_TF_MultiGPU.ipynb)    | 18min25s              |
 | [Tensorflow](notebooks/Tensorflow_MultiGPU.ipynb) | 17min6s               |
 | [Chainer]()                                       | ?                     |
-| [MXNet(Gluon)]()                                  | ?                     |
+| [MXNet(Gluon)](notebooks/Gluon_MultiGPU.ipynb)    | TBA                   |
 
 
 Input for this model is 112,120 PNGs of chest X-rays resized to (264, 264). **Note for the notebook to automatically download the data you must install [Azcopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux#download-and-install-azcopy) and increase the size of your OS-Disk in Azure Portal so that you have at-least 45GB of free-space (the Chest X-ray data is large!). The notebooks may take more than 10 minutes to first download the data.** These notebooks train DenseNet-121 and use native data-loaders to pre-process the data and perform the following data-augmentation:  
@@ -188,4 +188,4 @@ The below offers some insights we gained after trying to match test-accuracy acr
 
 1. There are multiple RNN implementations/kernels available for most frameworks (for example [Tensorflow](http://returnn.readthedocs.io/en/latest/tf_lstm_benchmark.html)); once reduced down to the cudnnLSTM/GRU level the execution is the fastest, however this implementation is less flexible (e.g. maybe you want layer normalisation) and may become problematic if inference is run on the CPU at a later stage. At the cudDNN level most of the frameworks' runtimes are very similar. [This](https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/) Nvidia blog-post goes through several interesting cuDNN optimisations for recurrent neural nets e.g. fusing - "combining the computation of many small matrices into that of larger ones and streaming the computation whenever possible, the ratio of computation to memory I/O can be increased, which results in better performance on GPU".
 
-2. It seems that the fastest data-shape for RNNs is TNC - implementing this in [MXNet](notebooks/MXNet_RNN_TNC.ipynb) only gave an improvement of 0.5s so I have chosen to use the sligthly slower shape to remain consistent with other frameworks and to keep the code less complicated
+2. It seems that the fastest data-shape for RNNs is TNC - implementing this in [MXNet](notebooks/MXNet_RNN_TNC.ipynb) only gave an improvement of 0.5s so I have chosen to use the sligthly slower shape to remain consistent with other frameworks and to keep the code less complicated