From c98c2a3f2b259d865b42c9d0efadc191335bcb23 Mon Sep 17 00:00:00 2001 From: Edoardo Holzl Date: Mon, 7 Dec 2020 14:07:39 +0100 Subject: [PATCH 1/2] Add results table --- _layouts/page.html | 4 +++- index.md | 7 ++++--- public/css/hyde.css | 7 +++++++ results/index.md | 37 +++++++++++++++++++++++++++++++++++++ 4 files changed, 51 insertions(+), 4 deletions(-) create mode 100644 results/index.md diff --git a/_layouts/page.html b/_layouts/page.html index 4e0d4eb..72ad5d7 100644 --- a/_layouts/page.html +++ b/_layouts/page.html @@ -3,6 +3,8 @@ ---
-

{{ page.title }}

+ {% if page.notitle != true %} +

{{ page.title }}

+ {% endif %} {{ content }}
diff --git a/index.md b/index.md index e6007fb..6a2fdc1 100644 --- a/index.md +++ b/index.md @@ -5,9 +5,10 @@ title: Home

MLBench: Distributed Machine Learning Benchmark

- - - + + + + Documentation Status diff --git a/public/css/hyde.css b/public/css/hyde.css index 213e232..0ea5ec7 100644 --- a/public/css/hyde.css +++ b/public/css/hyde.css @@ -248,3 +248,10 @@ a.sidebar-nav-item:focus { .theme-base-0f .related-posts li a:hover { color: #8f5536; } + + +table.wide { + max-width: 200%; + width: 200%; + font-size: 15px; +} \ No newline at end of file diff --git a/results/index.md b/results/index.md new file mode 100644 index 0000000..7b97e6e --- /dev/null +++ b/results/index.md @@ -0,0 +1,37 @@ +--- +layout: page +notitle: true +title: Results +datatable: true +--- + +# MLBench Official Training Results +--- + +

+Here we present the official training results obtained by running the benchmark tasks. +It provides a reference and comparison point for different implementations. +

+ + +

+The table below provides a comparison table for results. Times are in seconds. The baseline represents the 1 worker case, and we +only present the results for the best obtained speedups + +

+ +--- + +|Task | Dataset | Model | Aggregation scheme | Metric Name | Metric Goal | Framework | Baseline | Best Speedup | Best Speedup Workers | Description | Implementation | Results | +|:-------------------:|:--------:|:---------------------:|:------------------:|:----------------------:|:-----------:|:------------:|---------------:|:--------------:|:--------------------:|:-----------:|:--------------:|:-------:| +| Image Recognition | CIFAR10 | ResNet20 | All-Reduce | Validation Accuracy | 80% | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-all-reduce) | | +| | | | PyTorch DDP | | | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-distributed-data-parallel) | | +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+-------------+---------| +| Language Modelling | Wikitext2| AWD-LSTM | All-Reduce | Validation Perplexity | 70 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-3-language-modelling) | [code]() | | +| Language Modelling | | BERT +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| +| Machine Translation | WMT16 | LSTM (GNMT) | All-Reduce | Validation BLEU-score | 24 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#a-lstm-wmt16-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt16-gnmt-all-reduce) | | +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| +| Machine Translation | WMT17 | Transformer | All-Reduce | Validation BLEU-score | 25 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#b-transformer-wmt17-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt17-transformer-all-reduce) | | +{:.wide} + From 9a4ef951d10c6ec87f74ff54b7f5687a17048e51 Mon Sep 17 00:00:00 2001 From: Edoardo Holzl Date: Thu, 25 Feb 2021 12:34:27 +0100 Subject: [PATCH 2/2] Update with latest results --- results/index.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/results/index.md b/results/index.md index 7b97e6e..22f3a4b 100644 --- a/results/index.md +++ b/results/index.md @@ -16,22 +16,24 @@ It provides a reference and comparison point for different implementations.

The table below provides a comparison table for results. Times are in seconds. The baseline represents the 1 worker case, and we -only present the results for the best obtained speedups +only present the results for the best obtained speedups. + +For more detailed results for each run, please check the documentation.

--- |Task | Dataset | Model | Aggregation scheme | Metric Name | Metric Goal | Framework | Baseline | Best Speedup | Best Speedup Workers | Description | Implementation | Results | -|:-------------------:|:--------:|:---------------------:|:------------------:|:----------------------:|:-----------:|:------------:|---------------:|:--------------:|:--------------------:|:-----------:|:--------------:|:-------:| -| Image Recognition | CIFAR10 | ResNet20 | All-Reduce | Validation Accuracy | 80% | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-all-reduce) | | -| | | | PyTorch DDP | | | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-distributed-data-parallel) | | +|:-------------------:|:--------:|:---------------------:|:------------------:|:----------------------:|:-----------:|:------------:|:--------------:|:--------------:|:--------------------:|:-----------:|:--------------:|:-------:| +| Image Recognition | CIFAR10 | ResNet20 | All-Reduce | Validation Accuracy | 80% | Torch 1.7.0 | 171.15s | 6.6 (total), 11.9 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id17) | +| | | | PyTorch DDP | | | Torch 1.7.0 | 182.81s | 4.4 | 8 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-distributed-data-parallel) | | |---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+-------------+---------| -| Language Modelling | Wikitext2| AWD-LSTM | All-Reduce | Validation Perplexity | 70 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-3-language-modelling) | [code]() | | +| Language Modelling | Wikitext2| AWD-LSTM | All-Reduce | Validation Perplexity | 70 | Torch 1.7.0 | 87,401.33s | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-3-language-modelling) | [code]() | | | Language Modelling | | BERT |---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| -| Machine Translation | WMT16 | LSTM (GNMT) | All-Reduce | Validation BLEU-score | 24 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#a-lstm-wmt16-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt16-gnmt-all-reduce) | | +| Machine Translation | WMT16 | LSTM (GNMT) | All-Reduce | Validation BLEU-score | 24 | Torch 1.7.0 | 65,206.62s | 2.8 (total), 15.8 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#a-lstm-wmt16-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt16-gnmt-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id30) | |---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| -| Machine Translation | WMT17 | Transformer | All-Reduce | Validation BLEU-score | 25 | Torch 1.7.0 | | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#b-transformer-wmt17-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt17-transformer-all-reduce) | | +| Machine Translation | WMT17 | Transformer | All-Reduce | Validation BLEU-score | 25 | Torch 1.7.0 | 37,594.21s | 3.0 (total), 18.3 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#b-transformer-wmt17-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt17-transformer-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id32) | {:.wide}