Skip to content

Fix NCCL warm-up allreduce count mismatch#69

Open
rwakizaka wants to merge 1 commit into
r-ccs-cms:mainfrom
rwakizaka:fix-w_size-mismatch
Open

Fix NCCL warm-up allreduce count mismatch#69
rwakizaka wants to merge 1 commit into
r-ccs-cms:mainfrom
rwakizaka:fix-w_size-mismatch

Conversation

@rwakizaka

Copy link
Copy Markdown
Contributor

Problem

When the number of determinants is not evenly divisible by bdet_comm_size, the
process reproducibly hangs inside the NCCL warm-up ncclAllReduce call in
sbd/chemistry/tpb/sbdiag.h, before "NCCL communicators have been created." is
printed.

Root cause

After init_nccl_comm, a warm-up nccl_allreduce was called with a buffer sized
W.size():

if (mpi_size_b > 1) {
    init_nccl_comm(&b_nccl_comm, b_comm);
    thrust::device_vector<double> A(W.size(), 0.0);  // ← here
    nccl_allreduce(A, ncclSum, b_nccl_comm);
}

W is the local wavefunction chunk distributed across the β-det communicator. When
the total number of determinants is not evenly divisible by bdet_comm_size, each
rank holds a different-sized slice of W. NCCL's ncclAllReduce requires all
participating ranks to pass the same element count; mismatched counts cause a
protocol deadlock.

Fix

Replace the variable-size warm-up buffers with a constant-size buffer of 1 element
for all four NCCL communicators (h, b, t, a):

if (mpi_size_b > 1) {
    init_nccl_comm(&b_nccl_comm, b_comm);
    thrust::device_vector<double> A(1, 0.0);  // fixed: all ranks agree on count=1
    nccl_allreduce(A, ncclSum, b_nccl_comm);
}

The warm-up exists solely to trigger NCCL channel setup before the main computation;
the data transferred is irrelevant, so a single-element buffer is sufficient.

Files changed

  • include/sbd/chemistry/tpb/sbdiag.h — warm-up allreduce buffer size changed from
    W.size() to 1 for all four NCCL communicators

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant