Skip to content

[QUESTION] The Reason for calling torch.cuda.synchronize() in func recv_from_prev_pipeline_rank_/send_to_next_pipeline_rank #1144

Description

@CCCCarpediem

Why do we need to Call "torch.cuda.synchronize()" to synchronize all streams in /megatron/core/inference/communication_utils.py?
It describes as "To protect against race condition when using batch_isend_irecv()". But Event Record/ Event wait already inserted in communication/compute streams to ensure order. Thanks.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions