[QUESTION] why all rank0 of tp group make datasets？

[ltm920716]

Your question
hi,
I am very confused about the logic bellow. Here, datasets are built for each rank 0 of the TP group. If it is a rank 0 in the TP group of the middle layer, is it necessary to build datasets?
If the tp group is [[0,1,2,3], [4,5,6,7]], pp-group is [[0,4], [1,5], [2,6], [3,7]], world-size=8, that is global rank0 and rank4 make datasets both，that is right？

help please， thanks！
(

Megatron-LM/megatron/training/training.py

Line 1690 in 55622ff

if is_distributed or mpu.get_tensor_model_parallel_rank() == 0:

)