Async simulation and FedBuff aggregation

[ewenw]

Implements the async simulation mode with FedBuff using device traces without needing a constant arrival parameter, following the min-heap method described here.

Benchmarks comparison for Femnist (Sync vs Async with Fedbuff):

Sync / Async

Round: 100 / 180
Virtual clock: 27,618s / 18,913s
Top_5 eval accuracy: 0.914 / 0.92

Params: 5 clients per round, model = resnet18, max_concurrency=10

The results are consistent with the hypothesis that the async scheduling system increases the number of rounds that can be completed within the same amount of virtual clock time, and improves straggler tolerance. They also show the effect of aggregating stale updates from previous rounds, resulting in more rounds before convergence.

Checks

• I've included any doc changes needed for https://fedscale.readthedocs.io/en/latest/
• I've made sure the following tests are passing.
• Testing Configurations
  • Dry Run (20 training rounds & 1 evaluation round)
  • Cifar 10 (20 training rounds & 1 evaluation round)
  • Femnist (20 training rounds & 1 evaluation round)

[fanlai0990]

Thank you Ewen. It looks good to me. @AmberLJC @IKACE Can you please run some test? Thanks.

[IKACE]

Thank you so much for you contribution Ewen! Just one small thing:

• The model_zoo config in benchmark/configs/fedbuff_femnist/conf.yml seems to cause input/output mismatch error in training (see below). I can verify that commenting it out like benchmark/configs/femnist/conf.yml did would solve the issue.

[ewenw]

Thank you so much for you contribution Ewen! Just one small thing:

• The model_zoo config in benchmark/configs/fedbuff_femnist/conf.yml seems to cause input/output mismatch error in training (see below). I can verify that commenting it out like benchmark/configs/femnist/conf.yml did would solve the issue.

Thanks for catching this. I just commented it out.

[IKACE]

Thanks for catching this. I just commented it out.

Great! It looks perfect to me now. Thanks again!!