multi-GPU training

[camillebrianceau]

Training a neural network can pose problems with regards to learning time or memory (model too big). With multi-GPU distribution, we can:

• Parallelize the data to speed up the training by distributing each data batch: the model is replicated on each GPU to process a sub-set of the data.
• Parallelize the model, which involves segmenting the model across multiple GPUs in a sequential way. This does not accelerate learning and is more challenging to implement.
• Both methods can be used at the same time.

To do so , Pytorch Lightning can be used but it will require to add a dependence on ClinicaDL: is it necessary?

[mdiazmel]

If I understand well, the last mentioned approach is frequently called Pipeline Parallelism (It seems that Pytorch support is limited, at this moment). We need to define which of these approaches would be better for ClinicaDL, under the following constraints:

• The organization of the input data.
• The model size (with the current models, sometimes we need to decrease the batch size during the training phase in order to transfer the model to some GPUs, e.g. nvidia V100, 32Gb ).

To do so , Pytorch Lightning can be used but it will require to add a dependence on ClinicaDL: is it necessary?

I think also that we need to correctly evaluate the pros and cons of introducing a new dependency as PL. Probably using native Pytorch tools integrated to our MAPS class can be enough.