Skip to content

Latest commit

 

History

History
108 lines (100 loc) · 12.3 KB

BENCHMARK_MPI.md

File metadata and controls

108 lines (100 loc) · 12.3 KB

Benchmarking Results for MPI-based federated learning

Please visit the following link to check the latest benchmark experimental results: https://app.wandb.ai/automl/fedml/reports/FedML-Benchmark-Experimental-Results--VmlldzoxODE2NTU FedML white paper (https://arxiv.org/pdf/2007.13518.pdf) also summarizes the dataset list and related benchmarks. We refer the hyper-parameters and reproduce results from many top-tier ML conferences. Please check details of our reference hyperparameters as follows.

Linear Models

Data Model Alg Partition #C #C_p bs c_opt lr e #R acc
MNIST LR FedAvg Power Law 1000 10 10 SGD 0.03 1 >100 >75
Federated EMNIST LR FedAvg Power Law 200 10 10 SGD 0.003 1 >200 10~40
Synthetic(α,β） LR FedAvg Power Law 30 10 10 SGD 0.01 1 >200 >60

Note: #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch_size; c_opt = client optimizer; e = epoch; #R = number of rounds; acc = accuracy. For Synthetic(α,β), (α,β) is chosen from (0,0), (0.5,0.5), (1,1)

  • MNIST – Logistic Regression – FedAvg
    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description
    • comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
    • accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
  • Federated EMNIST – Logistic Regression-FedAvg
    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 7, Section 5.1, ‘Real data’
    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • epochs: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 9 description
    • comm_round: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
    • accuracy: ‘Federated optimization in heterogeneous networks’, page 21, Appendix C.3.2 Figure 10
  • Synthetic(α,β) – Logistic Regression -FedAvg
    • Patition Method: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’
    • client_num_in_total: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.1, ‘Synthetic’
    • client_num_per_round: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • batch_size: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • client_optimizer: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Implementation
    • lr: ‘Federated optimization in heterogeneous networks’, page 18, Appendix C.2, ‘Hyperparameters’
    • epochs: ‘Federated optimization in heterogeneous networks’, page 8, Section 5.1, ‘Hyperparameters & evaluation metrics’
    • comm_round: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6
    • accuracy: ‘Federated optimization in heterogeneous networks’, page 19, Appendix C.3.2 Figure 6

Lightweight and shallow neural network models

Task Data Set Model Alogrithm Partition Method Partition Alpha client_num_in_total client_num_per_round batch_size client_optimizer lr wd epochs comm_round accuracy
CV Federated EMNIST CNN (2 Conv + 2 FC) FedAvg Power Law 　 3400 10 20 SGD 0.1 - 1 >1500 84.9
CV CIFAR-100 ResNet-18+group normalization FedAvg Pachinko Allocation 100/500(ex/cli) 500 10 20 SGD 0.1 - 1 >4000 44.7
NLP Shakespeare RNN (2 LSTM + 1 FC) FedAvg realistic patition 　 715 10 4 SGD 1 - 1 >1200 56.9
NLP StackOverflow RNN (1 LSTM + 2 FC) FedAvg Pachinko Allocation 　 342477 50 16 SGD pow(10,-0.5) - 1 >1500 19.5
  • Federated EMNIST-CNN-FedAvg (https://openreview.net/pdf?id=LkFG3lB13U5)
    • Patition Method: ‘Adaptive federated optimization’ (https://openreview.net/pdf?id=LkFG3lB13U5), page 23, Appendix C.2
    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
    • wd (learning rate decay): ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 2
    • epochs: ‘Adaptive federated optimization’, page34, Appendix E.6, Paragraph 1
    • comm_round:‘Adaptive federated optimization’, page28, Appendix E.1, figure 3
    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
  • CIFAR-100 – ResNet18 -FedAvg
    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 3
    • Patition_alpha: ‘Adaptive federated optimization’, page 23, Appendix C.1, Paragraph 2
    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
  • Shakespeare – RNN – FedAvg
    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.3
    • client_num_in_total: ‘Adaptive federated optimization’, page 23, Appendix C Dataset & Models, Table2
    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1
  • StackOverflow – RNN – FedAvg
    • Patition Method: ‘Adaptive federated optimization’, page 23, Appendix C.4, Paragraph 2
    • client_num_in_total: ‘Adaptive federated optimization’, page 25, Appendix C.4, Paragraph 1
    • client_num_per_round: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • batch_size: ‘Adaptive federated optimization’, page 27, Appendix D Experiment Hyperparameters, Table7
    • client_optimizer: ‘Adaptive federated optimization’, page 25, Appendix D.1, Paragraph 1
    • lr: ‘Adaptive federated optimization’, page 27, Appendix D.4, Table8
    • epochs: ‘Adaptive federated optimization’, page 6, Section 4, ‘Optimizer and hyperparameters’
    • comm_round: ‘Adaptive federated optimization’, page 7, Section 4, figure 1
    • accuracy: ‘Adaptive federated optimization’, page 7, Section 5, Table1

Benchmarking using modern DNNs

Data Model Alg # C # C_p bs c_opt lr wd e round IID acc non-IID acc
CIFAR10 ResNet-56 FedAvg 10 10 64 SGD 0.001 0.001 20 100 93.19 87.12
CIFAR100 ResNet-56 FedAvg 10 10 64 SGD 0.001 0.001 20 100 68.91 64.70
CINIC10 ResNet-56 FedAvg 10 10 64 SGD 0.001 0.001 20 100 82.57 73.49
CIFAR10 MobileNet FedAvg 10 10 64 SGD 0.001 0.001 20 100 91.12 86.32
CIFAR100 MobileNet FedAvg 10 10 64 SGD 0.001 0.001 20 100 55.12 53.54
CINIC10 MobileNet FedAvg 10 10 64 SGD 0.001 0.001 20 100 79.95 71.23

Note: Non-IID distribution is set using LDA ( LDA = Latent Dirichlet Allocation) with alpha = 0.5; #C stands for client_num_in_total; #C_p stands for client_num_per_round; bs = batch size; c_opt = client optimizer.