You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+10-25
Original file line number
Diff line number
Diff line change
@@ -8,8 +8,6 @@
8
8
9
9
**For more details check out our [blog-post](https://blogs.technet.microsoft.com/machinelearning/2018/03/14/comparing-deep-learning-frameworks-a-rosetta-stone-approach/)**
10
10
11
-
We want to extend our gratitude to the CNTK, Pytorch, Chainer, Caffe2, MXNet and Knet teams, and everyone else from the open-source community who contributed to the repo over the past few months.
12
-
13
11
## Goal
14
12
15
13
1. Create a Rosetta Stone of deep-learning frameworks to allow data-scientists to easily leverage their expertise from one framework to another
@@ -21,8 +19,6 @@ We want to extend our gratitude to the CNTK, Pytorch, Chainer, Caffe2, MXNet and
21
19
22
20
The notebooks are executed on an Azure [Deep Learning Virtual Machine](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.dsvm-deep-learning) using both the K80 and the newer P100.
23
21
24
-
Please see these notebooks as examples rather than any formal benchmark (only the multi-GPU examples have potential to be that at some point) - for further details check this [post](https://www.reddit.com/r/MachineLearning/comments/7v3ibo/discussion_stop_benchmark_stupidity_and_improve_it/dtp9hng/).
25
-
26
22
*Accuracies are reported in notebooks, they should match to ensure we have common mode/code*
27
23
28
24
## Results
@@ -33,20 +29,20 @@ Please see these notebooks as examples rather than any formal benchmark (only th
*Note: It is recommended to use higher level APIs where possible; see these notebooks for examples with [Tensorflow](notebooks/Tensorflow_CNN_highAPI.ipynb), [MXNet](notebooks/MXNet_CNN_highAPI.ipynb) and [CNTK](notebooks/CNTK_CNN_highAPI.ipynb). They are not linked in the table to keep the common-structure-for-all approach*
45
+
*Note: It is recommended to use higher level APIs where possible; see these notebooks for examples with [Tensorflow](support/Tensorflow_CNN_highAPI.ipynb), [MXNet](support/MXNet_CNN_highAPI.ipynb) and [CNTK](support/CNTK_CNN_highAPI.ipynb). They are not linked in the table to keep the common-structure-for-all approach*
50
46
51
47
Input for this model is the standard [CIFAR-10 dataset](http://www.cs.toronto.edu/~kriz/cifar.html) containing 50k training images and 10k test images, uniformly split across 10 classes. Each 32 by 32 image is supplied as a tensor of shape (3, 32, 32) with pixel intensity re-scaled from 0-255 to 0-1.
52
48
@@ -62,7 +58,7 @@ Input for this model is the standard [CIFAR-10 dataset](http://www.cs.toronto.ed
Input for this model is 112,120 PNGs of chest X-rays. **Note for the notebook to automatically download the data you must install [Azcopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux#download-and-install-azcopy) and increase the size of your OS-Disk in Azure Portal so that you have at-least 45GB of free-space (the Chest X-ray data is large!). The notebooks may take more than 10 minutes to first download the data.** These notebooks train DenseNet-121 and use native data-loaders to pre-process the data and perform data-augmentation.
@@ -76,11 +72,10 @@ Comparing synthetic data to actual PNG files we can estimate the IO lag for **Py
@@ -112,14 +107,10 @@ The classification model creates an embedding matrix of size (150x125) and then
112
107
113
108
## Lessons Learned
114
109
115
-
The below offer some insights we gained after trying to match test-accuracy across frameworks and from all the GitHub issues/PRs raised.
116
-
117
-
#### Multi-GPU DenseNet
118
-
119
-
1. Data loading and augmentation has the potential to flip the results around and curiously from all of the frameworks it seems by default Keras is most efficient at this. We will try to create openCV-based common data-loading and data-augmentation functions to help standardise the results and let forward+backward training take centre stage
120
-
121
110
#### CNN
122
111
112
+
The below offers some insights I gained after trying to match test-accuracy across frameworks and from all the GitHub issues/PRs raised.
113
+
123
114
1. The above examples (except for Keras), for ease of comparison, try to use the same level of API and so all use the same generator-function. For [MXNet](support/MXNet_CNN_highAPI.ipynb), [Tensorflow](support/Tensorflow_CNN_highAPI.ipynb), and [CNTK](support/CNTK_CNN_highAPI.ipynb) I have experimented with a higher-level API, where I use the framework's training generator function. The speed improvement is negligible in this example because the whole dataset is loaded as NumPy array in RAM and the only processing done each epoch is a shuffle. I suspect the framework's generators perform the shuffle asynchronously. Curiously, it seems that the frameworks shuffle on a batch-level, rather than on an observation level, and thus ever so slightly decreases the test-accuracy (at least after 10 epochs). For scenarios where we have IO activity and perhaps pre-processing and data-augmentation on the fly, custom generators would have a much bigger impact on performance.
124
115
125
116
2. Running on CuDNN we want to use [NCHW] instead of channels-last. Keras finally supports this for Tensorflow (previously it had NHWC hard-coded and would auto-reshape after every batch)
@@ -170,12 +161,6 @@ The below offer some insights we gained after trying to match test-accuracy acro
170
161
make install
171
162
```
172
163
173
-
13. When using MXNet, you should avoid assigning outputs or data to numpy np.array in your training loop. This causes the data to be copied from the GPU to the CPU. You should use mx.nd.array instead, allocated in the right context at the beginning. This can dramatically increase performance.
174
-
175
-
14. When using MXNet, operations are allocated on the queue of the back-end engine and parallelized, try to avoid any blocking operations in your training loop. You can add a nd.waitall(), which will force waiting for all operations to complete at the end of each epoch to avoid filling up your memory.
176
-
177
-
15. With MXNet/Gluon, calling `.hybridize()` on your network will cache the computation graph and you will get performance gains. However that means that you won't be able to step through every calculations anymore. Use it once you are done debugging your network.
178
-
179
164
#### RNN
180
165
181
166
1. There are multiple RNN implementations/kernels available for most frameworks (for example [Tensorflow](http://returnn.readthedocs.io/en/latest/tf_lstm_benchmark.html)); once reduced down to the cudnnLSTM/GRU level the execution is the fastest, however this implementation is less flexible (e.g. maybe you want layer normalisation) and may become problematic if inference is run on the CPU at a later stage. At the cudDNN level most of the frameworks' runtimes are very similar. [This](https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/) Nvidia blog-post goes through several interesting cuDNN optimisations for recurrent neural nets e.g. fusing - "combining the computation of many small matrices into that of larger ones and streaming the computation whenever possible, the ratio of computation to memory I/O can be increased, which results in better performance on GPU".
0 commit comments