Table 3 Distributed deep learning

Data parallelism

[75]

• ResNet110 and AlexNet models on CIFAR10

• Train loss

• Test accuracy

[72]

• Matrix Classification

• MovieLens Avazu-CTR

• Convergence time per epoch

• Disk I/O

• Network communication

[65]

• ResNet-50 on ImageNet dataset

• ALBERT-large on WikiText-103 dataset

• Training time

[71]

• ResNet101 on CIFAR10 dataset

• Convergence

• Robustness

[69]

• LeNet-5 on MNIST dataset

• Accuracy

[70]

• ResNet-50 and Inception-v3 on ImageNet

• LM model on One Billion Word Benchmark

• NMT model on WMT English-German dataset

• Validation error

• Test perplexity

• BLEU

[73]

• Inception V3

• ResNet-101

• VGG-16

• Images processed per second

[68]

• CNN on CIFAR and ImageNet datasets

• Test loss

• Test error

[67]

• ImageNet

• Accuracy

Model parallelism

[77]

• GNN model on OGB-Product, OGB-Paper, UK-2006-05, UK-Union, Facebook datasets

• ROC

[79]

• ResNet and WRN models on CIFAR-10 dataset

• ResNet-18 and MobileNet v2 on Tiny-ImageNet

• Error rate

[76]

• AlexNet, Inception-v3 and ResNet-101 on ImageNet dataset

• RNNTC on Movie Reviews dataset

• RNNLM on Penn Treebank dataset

• NMT on WMT English-German dataser

• Accuracy

[80]

• ResNet on CIFAR

• Accuracy

Pipelining parallelism

[81]

• AmoebaNet-D

• U-Net

• Throughput

• Speed up

[82]

• VGG-16 and ResNet-50 on ImageNet

• AlexNet on Synthetic Data

• GNMT-16 and GNMT-8 on WMT16 EN-De

• AWD LM on Penn Treebank

• S2VT on MSVD

• Accuracy

• Speed up

[83]

• VGG16, ResNet-152, Inception v4 and SNN on CIFAR-10

• Transformer on IMDb Movie Review Sentiment Dataset

• Residual LSTM on IMDb Dataset

• Speed up

[84]

• VGG-A model on ImageNet

• Speed up

Hybrid parallelization

[88]

• MATCHNET, CTRDNN, 2EMB and NCE models

• Scheduling performance

• Throughput

[64]

• 3D-ResAttNet on Alzheimer’s Disease Neuroimaging Initiative (ADNI) database

• Speedup

• Accuracy

• Training time•

[91]

• CosmoFlow and 3D UNet models

• MSE

[86]

• Seq2Seq RNN MT with attention on WMT14 and WMT17 datasets

• BLEU scores

[87]

• SFC, SCONV, Lenet-c, Cifar-c, AlexNet,VGG-A, VGG-B, VGG-C, VGG-D and VGG-E models on MNIST, CIFAR-10 and ImageNet datasets

• Energy efficiency

• Performance

• Total communication

[85]

• AlexNet and VGG models

• Communication Overhead

• Training time

• Speed up

[90]

• CNN on ImageNet LSVRC-2010 dataset

• Error rate

[89]

• ImageNet dataset

• Error rate