Skip to content

Commit fb4aded

Browse files
authored
Merge pull request ilkarman#76 from ilkarman/ilia
multi-gpu example
2 parents 25b9c4c + 2913976 commit fb4aded

File tree

6 files changed

+1212
-446
lines changed

6 files changed

+1212
-446
lines changed

README.md

Lines changed: 24 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -50,22 +50,35 @@ Input for this model is the standard [CIFAR-10 dataset](http://www.cs.toronto.ed
5050

5151
### 2. Training Time: DenseNet-121 on ChestXRay - Image Recognition (Multi-GPU)
5252

53-
**This is a work in progress**
53+
This is a work in progress
5454

55-
**CUDA 9/CuDNN 7.0**
55+
**Train+Val w/ data-loader + data-augmentation**
5656

57-
| DL Library | 1xP100 | 2xP100 | 4xP100 | **4xP100 Synthetic Data** |
58-
| ----------------------------------------------- | :------------------: | :-------------------: | :------------------: | :------------------: |
59-
| [Pytorch](notebooks/PyTorch_MultiGPU.ipynb) | 41min46s | 28min50s | 23min7s | 11min48s |
60-
| [Keras(TF)](notebooks/Keras_TF_MultiGPU.ipynb) | 51min27s | 32min1s | 22min49s | 18min30s |
61-
| [Tensorflow](notebooks/Tensorflow_MultiGPU.ipynb) | 62min8s | 44min13s | 31min4s | 17min10s |
62-
| [Chainer]() | ? | ? | ? | ? |
63-
| [MXNet]() | ? | ? | ? | ? |
57+
| DL Library | 1xP100/CUDA 9/CuDNN 7 | 4xP100/CUDA 9/CuDNN 7 |
58+
| ----------------------------------------------- | :------------------: | :------------------: |
59+
| [Pytorch](notebooks/PyTorch_MultiGPU.ipynb) | 41min | 17min |
60+
| [Keras(TF)](notebooks/Keras_TF_MultiGPU.ipynb) | 51min | 22min |
61+
| [Tensorflow](notebooks/Tensorflow_MultiGPU.ipynb) | 50min | 25min |
62+
| [Chainer](notebooks/Chainer_MultiGPU.ipynb) | 65min | ? |
63+
| [MXNet]() | ? | ? |
6464

65+
**Train w/ synthetic-data**
6566

66-
Input for this model is 112,120 PNGs of chest X-rays. **Note for the notebook to automatically download the data you must install [Azcopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux#download-and-install-azcopy) and increase the size of your OS-Disk in Azure Portal so that you have at-least 45GB of free-space (the Chest X-ray data is large!). The notebooks may take more than 10 minutes to first download the data.** These notebooks train DenseNet-121 and use native data-loaders to pre-process the data and perform data-augmentation.
67+
| DL Library | 4xP100/CUDA 9/CuDNN 7 |
68+
| ----------------------------------------------- | :------------------: |
69+
| [Pytorch](notebooks/PyTorch_MultiGPU.ipynb) | 11min37s |
70+
| [Keras(TF)](notebooks/Keras_TF_MultiGPU.ipynb) | 18min25s |
71+
| [Tensorflow](notebooks/Tensorflow_MultiGPU.ipynb) | 17min6s |
72+
| [Chainer]() | ? |
73+
| [MXNet]() | ? |
6774

68-
Comparing synthetic data to actual PNG files we can estimate the IO lag for **PyTorch (~11min), Keras(TF) (~4min), Tensorflow (~13min)!** We need to investigate this to establish the most performant data-loading pipeline and any **help is appreciated**. The current plan is to write functions in OpenCV (or perhaps use ChainerCV) and share between all frameworks.
75+
76+
Input for this model is 112,120 PNGs of chest X-rays resized to (264, 264). **Note for the notebook to automatically download the data you must install [Azcopy](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-linux#download-and-install-azcopy) and increase the size of your OS-Disk in Azure Portal so that you have at-least 45GB of free-space (the Chest X-ray data is large!). The notebooks may take more than 10 minutes to first download the data.** These notebooks train DenseNet-121 and use native data-loaders to pre-process the data and perform the following data-augmentation:
77+
78+
1. Random crop to from (264, 264) to (224, 224)
79+
2. Randon horizontal flip
80+
81+
**The Chainer example is currently a bit broken and help is greatly appreciated & TF Estimator API wastes a lot of time saving/loading between training/validation**
6982

7083
### 3. Avg Time(s) for 1000 images: ResNet-50 - Feature Extraction
7184

0 commit comments

Comments
 (0)