Why do we need to Call "torch.cuda.synchronize()" to synchronize all streams in /megatron/core/inference/communication_utils.py?
It describes as "To protect against race condition when using batch_isend_irecv()". But Event Record/ Event wait already inserted in communication/compute streams to ensure order. Thanks.
Why do we need to Call "torch.cuda.synchronize()" to synchronize all streams in /megatron/core/inference/communication_utils.py?
It describes as "To protect against race condition when using batch_isend_irecv()". But Event Record/ Event wait already inserted in communication/compute streams to ensure order. Thanks.