Pipeline Parallelism Across Nodes Fails with EOFError

[rrutmann]

Bug description

We created this issue for pytorch, but since we're not sure if it fits better to this repo, we link it here for reference:

pytorch/pytorch#165143

Could someone, who successully ran torchtitan with pipeline parallelism on multiple nodes, please provide a working setup here that we could try?

Versions

Component	Version / Info
PyTorch	2.10.0.dev20251008+cu126; 2.10.0.dev20250918+cu128 (nightly); 2.10.0.dev20250924+cu130
TorchTitan	current main branch (as of Oct 2025)
CUDA	12.6; 12.8; 13.0
Python	3.10.12; 3.11.5; 3.12.3
Cluster	2× DGX nodes (tested with and without InfiniBand)
Launcher	torch.distributed.run
Backend	NCCL
Environment vars	Default unless otherwise specified

[tianyu-l]

1. Does this issue happen on latest pytorch nightly, which is the recommended version to run torchtitan.
2. Could you provide a repro instruction -- what configs could we use to repro this issue?

cc @H-Huang

[rrutmann]

1. Does this issue happen on latest pytorch nightly, which is the recommended version to run torchtitan.

   2. Could you provide a repro instruction -- what configs could we use to repro this issue?

cc @H-Huang

Thank you for the quick response. We tested it with (at least) these pytorch versions:
2.10.0.dev20251008+cu126; 2.10.0.dev20250918+cu128; 2.10.0.dev20250924+cu130

We ran the training on different systems, one of them is the HPC Leonardo:
https://docs.hpc.cineca.it/hpc/leonardo.html#leonardo-card

You can find a repro instruction together with the used environment variables in the linked issue. We basically use your example config with pipeline_parallel_degree = 2 on two nodes with one rank each

[BlueCrescent]

The following Dockerfile and run command with the device specific values set in the <...> fields can be used to reproduce the issue:

FROM ghcr.io/pytorch/pytorch-nightly:2.10.0.dev20251012-cuda12.6-cudnn9-runtime

RUN apt-get update && \
    apt-get install -y git && \
    git clone https://github.com/pytorch/torchtitan.git && \
    cd torchtitan && \
    python -m pip install -e . && \
    sed -i 's/pipeline_parallel_degree = 1/pipeline_parallel_degree = 2/' torchtitan/models/llama3/train_configs/debug_model.toml

ENV TORCH_DISTRIBUTED_DEBUG=DETAIL
ENV TORCH_CPP_LOG_LEVEL=INFO
ENV NCCL_DEBUG=INFO
ENV NCCL_DEBUG_SUBSYS=ALL
ENV TORCH_NCCL_ASYNC_ERROR_HANDLING=1
ENV NCCL_IB_DISABLE=1

WORKDIR /workspace/torchtitan

CMD ["python", "-m", "torch.distributed.run", "--nnodes", "2", "--node_rank", "<ZERO OR ONE ON RESPECTIVE NODE>", "--nproc_per_node", "1", "--rdzv-endpoint=<RANK 0 MACHINE IP>:29406", "--rdzv-id=ttmultinode", "-m", "torchtitan.train", "--job.config_file", "torchtitan/models/llama3/train_configs/debug_model.toml"]

docker build --pull --rm -f 'Dockerfile' -t 'torchtitan:latest' '.' 

docker run --rm --gpus '"device=0"' -it --network=host --ipc=host --env NCCL_SOCKET_IFNAME=<SOCKET NAME ON THE SPECIFIC NODE> --env GLOO_SOCKET_IFNAME=<SOCKET NAME ON THE SPECIFIC NODE> torchtitan:latest

[BlueCrescent]

I did some further debugging:
We already saw, that the process on the first stage hangs when accessing the model weights, either while unsharding or when computing the embedding layer if fsdp is not used. Additionally, when trying to access the model weights via VS Code's debugger, the whole debugger will hang. I tried to find the earliest point where accessing the weights becomes impossible.
It seems like accessing any tensor on the cuda device becomes inaccessible after this line:
https://github.com/pytorch/pytorch/blob/74db92b21868b7e9e77cc966e5d57a8246723cbd/torch/distributed/pipelining/schedules.py#L583
Adding this line afterwards on either stage will cause the process to hang:

torch.zeros(1).to(self._stage.device)

When stepping through, I can reach _wait_batch_p2p() where it will break. When running until a breakpoint at the start of _wait_batch_p2p() it will already be broken there.

The warning from the original issue gets already raised at this point (on rank 1):

[2025-10-14 13:19:10] dgx2:1542750:1544071 [0] transport/net_socket.cc:563 NCCL WARN NET/Socket : peer 10.183.75.13<57670> message truncated : receiving 8 bytes instead of 4. If you believe your socket network is in a healthy state, there may be a mismatch in collective sizes or environment settings (e.g. NCCL_PROTO, NCCL_ALGO) between ranks

[BlueCrescent]

I think, the following script reproduces the error without the pipeline parallelism code:

test_nccl_p2pops.py

import torch
import torch.distributed as dist

def main():
    dist.init_process_group(backend="gloo")
    rank = dist.get_rank()
    world = dist.get_world_size()
    if world != 2:
        print("Run with exactly 2 ranks")
        return
    mesh = dist.init_device_mesh("cuda", [2], mesh_dim_names=["pp"])
    print(f"Rank {rank}/{world} initialized process group with mesh: {mesh}")
    group = mesh.get_group("pp")
    print(f"Rank {rank} has pp group: {group}")

    if rank == 0:
        send_tensor = torch.arange(10).to("cuda:0")
        send_p2pop = dist.P2POp(
            dist.isend,
            send_tensor,
            group_peer=1,
            group=group,
        )
        work = dist.batch_isend_irecv([send_p2pop])
        work[0].wait()
        print("Rank 0: sent tensor", send_tensor)
    else:
        recv_tensor = torch.zeros(10).to("cuda:0")
        recv_p2pop = dist.P2POp(
            dist.irecv,
            recv_tensor,
            group_peer=0,
            group=group,
        )
        work = dist.batch_isend_irecv([recv_p2pop])
        work[0].wait()
        print("Rank 1: received tensor", recv_tensor)

if __name__ == "__main__":
    main()

I ran it on the Leonardo HPC System using the following singularity image and slurm script:

Singularity

Bootstrap: docker

From: ghcr.io/pytorch/pytorch-nightly:2.10.0.dev20251012-cuda12.6-cudnn9-runtime

%post

    apt-get update && \
    apt-get install -y git && \
    git clone https://github.com/pytorch/torchtitan.git && \
    cd torchtitan && \
    python -m pip install -e . && \
    sed -i 's/pipeline_parallel_degree = 1/pipeline_parallel_degree = 2/' torchtitan/models/llama3/train_configs/debug_model.toml

%environment

TORCH_DISTRIBUTED_DEBUG=DETAIL
TORCH_CPP_LOG_LEVEL=INFO
NCCL_DEBUG=INFO
NCCL_DEBUG_SUBSYS=ALL
TORCH_NCCL_ASYNC_ERROR_HANDLING=1
<!-- NCCL_IB_DISABLE=1 -->

%runscript

    exec python -m torch.distributed.run

job.sbatch

#!/bin/bash

##SBATCH --exclusive
#SBATCH --qos=boost_qos_dbg
#SBATCH --job-name=test_nccl_p2pops
#SBATCH --output=logs/%x_%j.batch.out.log
#SBATCH --error=logs/%x_%j.batch.err.log
#SBATCH --time=00:03:00
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=32
#SBATCH --nodes=2
#SBATCH --gres=gpu:1
#SBATCH --mem=0

set -x -e

WORKDIR=torch_titan
cd $WORKDIR

SING_IMAGE=images/torchtitan_cuda12.6.img
 
MASTER_ADDR=$(scontrol show hostnames $SLURM_JOB_NODELIST | head -n 1)
MASTER_PORT=6000

module load cuda/12.6

echo "START TIME: $(date)"

srun \
  --output=$WORKDIR/logs/%x_%j.%N.%t.out.log \
  --error=$WORKDIR/logs/%x_%j.%N.%t.err.log \
  singularity exec --nv \
  -B /dev/infiniband:/dev/infiniband \
  -B $WORKDIR/outputs:/torchtitan/outputs \
  --env TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
  --env NCCL_SOCKET_IFNAME=ib0 \
  --env GLOO_SOCKET_IFNAME=ib0 \
  --env NCCL_DEBUG=INFO \
  --env NCCL_DEBUG_SUBSYS=ALL \
  --env NCCL_IB_TIMEOUT=5000 \
  --env NCCL_IB_RETRY_CNT=16 \
  --env UCX_NET_DEVICES=ib0 \
  --env UCX_RC_TIMEOUT=10s \
  --env MASTER_ADDR=$MASTER_ADDR \
  --env MASTER_PORT=$MASTER_PORT \
  --env SLURM_NODEID=$SLURM_NODEID \
  --env SLURM_JOB_NUM_NODES=$SLURM_JOB_NUM_NODES \
  --env LOG_RANK=${LOG_RANK:-0} \
  --env CONFIG_FILE=/torchtitan/torchtitan/models/llama3/train_configs/debug_model.toml \
  --env LOGLEVEL=DEBUG \
  --env FI_PROVIDER="efa" \
  --env NCCL_IB_DISABLE=1 \
  --env NCCL_BUFFSIZE=2097152 \
  --env CUDA_VISIBLE_DEVICES=0 \
  --env CUDA_LAUNCH_BLOCKING=0 \
  --env FI_EFA_SET_CUDA_SYNC_MEMOPS=0 \
  "$SING_IMAGE" bash -lc '
    echo "HOST: $(hostname)"; cat /sys/class/net/ib0/operstate || true; cat /sys/class/net/ib0/mtu || true
    nvidia-smi -L
    nvidia-smi
    pwd
    torchrun \
      --node_rank=$SLURM_NODEID \
      --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT \
      --rdzv-id=101 \
      --nnodes $SLURM_JOB_NUM_NODES \
      --nproc_per_node 1 \
      --rdzv_backend c10d \
      test_nccl_p2pops.py
  '

echo "END TIME: $(date)"
echo "=== FINISHED ==="

The logs on rank 0 look fine here:

stdout

[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 0 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
Rank 0/2 initialized process group with mesh: DeviceMesh((pp=2), 'cuda', stride=(1,))
Rank 0 has pp group: <torch.distributed.distributed_c10d.ProcessGroup object at 0x1551df889c70>
lrdn2741:2973508:2973508 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0
lrdn2741:2973508:2973508 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ib0
lrdn2741:2973508:2973508 [0] NCCL INFO Bootstrap: Using ib0:10.128.48.245<0>
lrdn2741:2973508:2973508 [0] NCCL INFO cudaDriverVersion 12020
lrdn2741:2973508:2973508 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
lrdn2741:2973508:2973508 [0] NCCL INFO init.cc:1798 Cuda Host Alloc Size 4 pointer 0x154fba600000
lrdn2741:2973508:2973508 [0] NCCL INFO Comm config Blocking set to 1
lrdn2741:2973508:2973551 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. 
lrdn2741:2973508:2973551 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
lrdn2741:2973508:2973551 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0
lrdn2741:2973508:2973551 [0] NCCL INFO NET/Socket : Using [0]ib0:10.128.48.245<0>
lrdn2741:2973508:2973551 [0] NCCL INFO Initialized NET plugin Socket
lrdn2741:2973508:2973551 [0] NCCL INFO Assigned NET plugin Socket to comm
lrdn2741:2973508:2973551 [0] NCCL INFO Using network Socket
lrdn2741:2973508:2973551 [0] NCCL INFO ncclCommInitRankConfig comm 0x563676814370 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1d000 commId 0x3dc40b8043af4393 - Init START
lrdn2741:2973508:2973551 [0] NCCL INFO RAS network listening socket at 10.128.48.245<50043>
lrdn2741:2973508:2973551 [0] NCCL INFO RAS client listening socket at ::1<28028>
lrdn2741:2973508:2973551 [0] NCCL INFO Mem Realloc old size 0, new size 256 pointer 0x154f8f715d20
lrdn2741:2973508:2973552 [0] NCCL INFO RAS thread started
lrdn2741:2973508:2973552 [0] NCCL INFO Mem Realloc old size 0, new size 32 pointer 0x154f64004e00
lrdn2741:2973508:2973551 [0] NCCL INFO Bootstrap timings total 0.002859 (create 0.000017, send 0.000063, recv 0.000109, ring 0.000041, delay 0.000000)
lrdn2741:2973508:2973552 [0] NCCL INFO RAS handling local addRanks request (old nRasPeers 0)
lrdn2741:2973508:2973552 [0] NCCL INFO RAS finished local processing of addRanks request (new nRasPeers 2, nRankPeers 2)
lrdn2741:2973508:2973552 [0] NCCL INFO RAS peer 0: socket 10.128.48.245<50043>, pid 2973508, GPU 0 [this process]
lrdn2741:2973508:2973552 [0] NCCL INFO RAS peer 1: socket 10.128.49.17<60805>, pid 1549299, GPU 0
lrdn2741:2973508:2973552 [0] NCCL INFO RAS peersHash 0x72fe89341463d2f
lrdn2741:2973508:2973552 [0] NCCL INFO RAS link 1: opening new primary connection with 10.128.49.17<60805>
lrdn2741:2973508:2973552 [0] NCCL INFO RAS link -1: calculated existing primary connection with 10.128.49.17<60805>
lrdn2741:2973508:2973552 [0] NCCL INFO RAS handling connInitAck from 10.128.49.17<60805> (nack 0)
lrdn2741:2973508:2973552 [0] NCCL INFO RAS established connection with 10.128.49.17<60805> (sendQ empty, experiencingDelays 0, startRetryTime 0.00s)
lrdn2741:2973508:2973551 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
lrdn2741:2973508:2973551 [0] NCCL INFO Retrieving state for Socket
lrdn2741:2973508:2973551 [0] NCCL INFO Initialized state 0 for Socket
lrdn2741:2973508:2973551 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for HCA 0 'ib0'
lrdn2741:2973508:2973551 [0] NCCL INFO ncclTopoPopulateNics : Filled ib0 in topo with pciPath=/sys/devices/pci0000:17/0000:17:02.0/0000:18:00.0/0000:19:00.0/0000:1a:00.0 keep=1 coll=(null)
lrdn2741:2973508:2973551 [0] NCCL INFO === System : maxBw 12.5 totalBw 24.0 ===
lrdn2741:2973508:2973551 [0] NCCL INFO CPU/0-0 (1/1/2)
lrdn2741:2973508:2973551 [0] NCCL INFO + PCI[24.0] - PCI/0-18000 (15b3197600000000)
lrdn2741:2973508:2973551 [0] NCCL INFO               + PCI[24.0] - NIC/0-1a000
lrdn2741:2973508:2973551 [0] NCCL INFO                             + NET[12.5] - NET/0-0 (0/0/0/12.500000)
lrdn2741:2973508:2973551 [0] NCCL INFO               + PCI[24.0] - PCI/0-1b000 (15b3197600000000)
lrdn2741:2973508:2973551 [0] NCCL INFO                             + PCI[24.0] - GPU/0-1d000 (0)
lrdn2741:2973508:2973551 [0] NCCL INFO ==========================================
lrdn2741:2973508:2973551 [0] NCCL INFO GPU/0-1d000 :GPU/0-1d000 (0/5000.0/LOC) CPU/0-0 (3/24.0/PHB) NET/0-0 (6/12.5/PHB) 
lrdn2741:2973508:2973551 [0] NCCL INFO NET/0-0 :GPU/0-1d000 (6/12.5/PHB) CPU/0-0 (3/12.5/PHB) NET/0-0 (0/5000.0/LOC) 
lrdn2741:2973508:2973551 [0] NCCL INFO Setting affinity for GPU 0 to 0-15
lrdn2741:2973508:2973551 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 1, bw 10.000000/10.000000, type LOC/PHB, sameChannels 1
lrdn2741:2973508:2973551 [0] NCCL INFO  0 : NET/0-0 GPU/0-1d000 NET/0-0
lrdn2741:2973508:2973551 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 1, bw 20.000000/10.000000, type LOC/PHB, sameChannels 1
lrdn2741:2973508:2973551 [0] NCCL INFO  0 : NET/0-0 GPU/0-1d000 NET/0-0
lrdn2741:2973508:2973551 [0] NCCL INFO comm 0x563676814370 rank 0 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0
lrdn2741:2973508:2973551 [0] NCCL INFO Tree 0 : -1 -> 0 -> 1/-1/-1
lrdn2741:2973508:2973551 [0] NCCL INFO Tree 1 : 1 -> 0 -> -1/-1/-1
lrdn2741:2973508:2973551 [0] NCCL INFO Channel 00/02 : 0 1
lrdn2741:2973508:2973551 [0] NCCL INFO Channel 01/02 : 0 1
lrdn2741:2973508:2973551 [0] NCCL INFO Ring 00 : 1 -> 0 -> 1
lrdn2741:2973508:2973551 [0] NCCL INFO Ring 01 : 1 -> 0 -> 1
lrdn2741:2973508:2973551 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1
lrdn2741:2973508:2973551 [0] NCCL INFO NCCL_BUFFSIZE set by environment to 2097152.
lrdn2741:2973508:2973551 [0] NCCL INFO P2P Chunksize set to 131072
lrdn2741:2973508:2973551 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
lrdn2741:2973508:2973551 [0] NCCL INFO Check P2P Type isAllDirectP2p 0 directMode 0
lrdn2741:2973508:2973551 [0] NCCL INFO UDS: Creating service thread comm 0x563676814370 rank 0
lrdn2741:2973508:2973551 [0] NCCL INFO misc/utils.cc:233 memory stack hunk malloc(65536)
lrdn2741:2973508:2973553 [0] NCCL INFO [Proxy Service] Device 0 CPU core 1
lrdn2741:2973508:2973554 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 4
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:42 Cuda Alloc Size 1216 pointer 0x154f85c00000
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:45 Cuda Alloc Size 32 pointer 0x154f85e00000
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:56 Cuda Alloc Size 8 pointer 0x154f86e00000
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:42 Cuda Alloc Size 1216 pointer 0x154f87000000
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:45 Cuda Alloc Size 32 pointer 0x154f87200000
lrdn2741:2973508:2973551 [0] NCCL INFO channel.cc:56 Cuda Alloc Size 8 pointer 0x154f87400000
lrdn2741:2973508:2973551 [0] NCCL INFO   Algorithm   |                            Tree                  |                            Ring                  |                   CollNetDirect                  |
lrdn2741:2973508:2973551 [0] NCCL INFO   Protocol    |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |
lrdn2741:2973508:2973551 [0] NCCL INFO  Max NThreads |            512 |            640 |            512 |            512 |            640 |            256 |              0 |              0 |            640 |
lrdn2741:2973508:2973551 [0] NCCL INFO     Broadcast |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     9.3/   5.0 |    18.0/   0.0 |    22.4/  10.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO        Reduce |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     9.3/   5.0 |    18.0/   0.0 |    22.4/  10.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllGather |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    11.6/  10.0 |    22.5/   0.0 |    22.4/  20.0 |     5.0/   0.0 |     8.5/   0.0 |    14.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO ReduceScatter |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    11.6/  10.0 |    22.5/   0.0 |    22.4/  20.0 |     5.0/   0.0 |     8.5/   0.0 |    14.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllReduce |    16.8/   2.1 |    31.0/   0.0 |    36.4/   7.8 |    16.6/   5.0 |    31.0/   0.0 |    36.4/  10.0 |     5.0/   0.0 |     8.5/   0.0 |    14.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO   Algorithm   |                    CollNetChain                  |                            NVLS                  |                        NVLSTree                  |
lrdn2741:2973508:2973551 [0] NCCL INFO   Protocol    |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |
lrdn2741:2973508:2973551 [0] NCCL INFO  Max NThreads |              0 |              0 |            640 |              0 |              0 |            640 |              0 |              0 |            640 |
lrdn2741:2973508:2973551 [0] NCCL INFO     Broadcast |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO        Reduce |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllGather |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    14.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO ReduceScatter |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    14.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllReduce |     5.0/   0.0 |     8.5/   0.0 |    14.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    14.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |    28.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO   Algorithm   |                             PAT                  |
lrdn2741:2973508:2973551 [0] NCCL INFO   Protocol    |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |             LL |          LL128 |         Simple |
lrdn2741:2973508:2973551 [0] NCCL INFO  Max NThreads |              0 |              0 |              0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     Broadcast |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO        Reduce |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllGather |     0.0/   0.0 |     0.0/   0.0 |    17.6/  15.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO ReduceScatter |     0.0/   0.0 |     0.0/   0.0 |    17.6/  15.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO     AllReduce |     0.0/   0.0 |     0.0/   0.0 |     0.0/   0.0 |
lrdn2741:2973508:2973551 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lrdn2741:2973508:2973551 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lrdn2741:2973508:2973551 [0] NCCL INFO init.cc:447 Cuda Alloc Size 23648 pointer 0x154f87600000
lrdn2741:2973508:2973551 [0] NCCL INFO init.cc:449 Cuda Alloc Size 8 pointer 0x154f87800000
lrdn2741:2973508:2973551 [0] NCCL INFO CC Off, workFifoBytes 1048576
lrdn2741:2973508:2973551 [0] NCCL INFO init.cc:491 Cuda Host Alloc Size 1048576 pointer 0x154fba600200
lrdn2741:2973508:2973551 [0] NCCL INFO init.cc:501 Cuda Host Alloc Size 65536 pointer 0x154fba700200
lrdn2741:2973508:2973551 [0] NCCL INFO init.cc:502 Cuda Host Alloc Size 65536 pointer 0x154fba710200
lrdn2741:2973508:2973551 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
lrdn2741:2973508:2973551 [0] NCCL INFO ncclCommInitRankConfig comm 0x563676814370 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1d000 commId 0x3dc40b8043af4393 - Init COMPLETE
lrdn2741:2973508:2973551 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 0 nranks 2 total 0.66 (kernels 0.63, alloc 0.00, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.00, rest 0.00)
lrdn2741:2973508:2973508 [0] NCCL INFO Send: opCount 0 sendbuff (nil) recvbuff 0x154fba400000 count 10 datatype 4 op 0 root 1 comm 0x563676814370 [nranks=2] stream 0x56366f5bb9e0
lrdn2741:2973508:2973508 [0] NCCL INFO misc/utils.cc:233 memory stack hunk malloc(65536)
lrdn2741:2973508:2973553 [0] NCCL INFO Mem Realloc old size 0, new size 8 pointer 0x154f5c004f10
lrdn2741:2973508:2973553 [0] NCCL INFO Allocated 33030532 bytes of shared memory in /dev/shm/nccl-cd6oIV
lrdn2741:2973508:2973553 [0] NCCL INFO New proxy send connection 0 from local rank 0, transport 2
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f65cf00 op.type=1 op.reqBuff=0x154f5c004ed0 op.respSize=16 done
lrdn2741:2973508:2973556 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 5
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f65cf00 matches expected opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO Connected to proxy localRank 0 -> connection 0x154f5c004f30
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Init res=0
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f65cf00 op.type=3 op.reqBuff=0x154f5c008e60 op.respSize=0 done
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f65cf00 matches expected opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO Channel 00/1 : 0[0] -> 1[0] [send] via NET/Socket/0/Shared
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Setup res=0
lrdn2741:2973508:2973553 [0] NCCL INFO New proxy send connection 1 from local rank 0, transport 2
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f65cf00 op.type=1 op.reqBuff=0x154f5c0094d0 op.respSize=16 done
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f65cf00
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Init res=0
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f65cf00 matches expected opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO Connected to proxy localRank 0 -> connection 0x154f5c004fa8
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f65cf00 op.type=3 op.reqBuff=0x154f5c009510 op.respSize=0 done
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f65cf00 matches expected opId=0x154f8f65cf00
lrdn2741:2973508:2973555 [0] NCCL INFO Channel 01/1 : 0[0] -> 1[0] [send] via NET/Socket/0/Shared
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Setup res=0
lrdn2741:2973508:2973555 [0] NCCL INFO sendConnect ncclProxyCallAsync opId=0x154f8f7a2f10
lrdn2741:2973508:2973555 [0] NCCL INFO sendConnect ncclProxyCallAsync opId=0x154f8f7a7930
lrdn2741:2973508:2973553 [0] NCCL INFO transport/net.cc:559 Cuda Host Alloc Size 4194304 pointer 0x154fa2a00000
lrdn2741:2973508:2973553 [0] NCCL INFO transport/net.cc:811 Cuda Host Alloc Size 8192 pointer 0x154fba720200
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f7a2f10 op.type=4 op.reqBuff=0x154f5c009b80 op.respSize=1400 done
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f7a2f10
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f7a2f10 matches expected opId=0x154f8f7a2f10
lrdn2741:2973508:2973555 [0] NCCL INFO sendConnect ncclPollProxyResponse opId=0x154f8f7a2f10
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Connect res=0
lrdn2741:2973508:2973553 [0] NCCL INFO transport/net.cc:811 Cuda Host Alloc Size 8192 pointer 0x154fba722200
lrdn2741:2973508:2973553 [0] NCCL INFO proxyProgressAsync opId=0x154f8f7a7930 op.type=4 op.reqBuff=0x154f5c009c10 op.respSize=1400 done
lrdn2741:2973508:2973555 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x154f8f7a7930
lrdn2741:2973508:2973555 [0] NCCL INFO resp.opId=0x154f8f7a7930 matches expected opId=0x154f8f7a7930
lrdn2741:2973508:2973555 [0] NCCL INFO sendConnect ncclPollProxyResponse opId=0x154f8f7a7930
lrdn2741:2973508:2973553 [0] NCCL INFO Received and initiated operation=Connect res=0
Rank 0: sent tensor tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], device='cuda:0')

Rank 1 does not manage to print out the received tensor and has the same error message as when running torchtitan with pp:

stdout

[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
[Gloo] Rank 1 is connected to 1 peer ranks. Expected number of connected peer ranks is : 1
Rank 1/2 initialized process group with mesh: DeviceMesh((pp=2), 'cuda', stride=(1,))
Rank 1 has pp group: <torch.distributed.distributed_c10d.ProcessGroup object at 0x14746d685970>
lrdn2748:1549299:1549299 [0] NCCL INFO cudaDriverVersion 12020
lrdn2748:1549299:1549299 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0
lrdn2748:1549299:1549299 [0] NCCL INFO NCCL_SOCKET_IFNAME set to ib0
lrdn2748:1549299:1549299 [0] NCCL INFO Bootstrap: Using ib0:10.128.49.17<0>
lrdn2748:1549299:1549299 [0] NCCL INFO NCCL version 2.27.5+cuda12.9
lrdn2748:1549299:1549299 [0] NCCL INFO init.cc:1798 Cuda Host Alloc Size 4 pointer 0x14724a600000
lrdn2748:1549299:1549299 [0] NCCL INFO Comm config Blocking set to 1
lrdn2748:1549299:1549341 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. 
lrdn2748:1549299:1549341 [0] NCCL INFO NCCL_IB_DISABLE set by environment to 1.
lrdn2748:1549299:1549341 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to ib0
lrdn2748:1549299:1549341 [0] NCCL INFO NET/Socket : Using [0]ib0:10.128.49.17<0>
lrdn2748:1549299:1549341 [0] NCCL INFO Initialized NET plugin Socket
lrdn2748:1549299:1549341 [0] NCCL INFO Assigned NET plugin Socket to comm
lrdn2748:1549299:1549341 [0] NCCL INFO Using network Socket
lrdn2748:1549299:1549341 [0] NCCL INFO ncclCommInitRankConfig comm 0x564d2ca0bf30 rank 1 nranks 2 cudaDev 0 nvmlDev 0 busId 1d000 commId 0x3dc40b8043af4393 - Init START
lrdn2748:1549299:1549341 [0] NCCL INFO RAS network listening socket at 10.128.49.17<60805>
lrdn2748:1549299:1549341 [0] NCCL INFO RAS client listening socket at ::1<28028>
lrdn2748:1549299:1549341 [0] NCCL INFO Mem Realloc old size 0, new size 256 pointer 0x14721f715fe0
lrdn2748:1549299:1549342 [0] NCCL INFO RAS thread started
lrdn2748:1549299:1549342 [0] NCCL INFO Mem Realloc old size 0, new size 32 pointer 0x1471f4004e00
lrdn2748:1549299:1549341 [0] NCCL INFO Bootstrap timings total 0.005669 (create 0.000018, send 0.000180, recv 0.002818, ring 0.000024, delay 0.000000)
lrdn2748:1549299:1549342 [0] NCCL INFO RAS handling local addRanks request (old nRasPeers 0)
lrdn2748:1549299:1549342 [0] NCCL INFO RAS finished local processing of addRanks request (new nRasPeers 2, nRankPeers 2)
lrdn2748:1549299:1549342 [0] NCCL INFO RAS link 1: calculated deferred primary connection with 10.128.48.245<50043>
lrdn2748:1549299:1549342 [0] NCCL INFO RAS link -1: calculated deferred primary connection with 10.128.48.245<50043>
lrdn2748:1549299:1549342 [0] NCCL INFO RAS new incoming socket connection from 10.128.48.245<36630>
lrdn2748:1549299:1549342 [0] NCCL INFO RAS handling connInit from 10.128.48.245<36630> (version 22705, listeningAddr 10.128.48.245<50043>, peersHash 0x72fe89341463d2f, deadPeersHash 0x0)
lrdn2748:1549299:1549341 [0] NCCL INFO TOPO/NET : Importing network plugins to topology
lrdn2748:1549299:1549341 [0] NCCL INFO Retrieving state for Socket
lrdn2748:1549299:1549341 [0] NCCL INFO Initialized state 0 for Socket
lrdn2748:1549299:1549341 [0] NCCL INFO NET/Socket : GPU Direct RDMA Disabled for HCA 0 'ib0'
lrdn2748:1549299:1549341 [0] NCCL INFO ncclTopoPopulateNics : Filled ib0 in topo with pciPath=/sys/devices/pci0000:17/0000:17:02.0/0000:18:00.0/0000:19:00.0/0000:1a:00.0 keep=1 coll=(null)
lrdn2748:1549299:1549341 [0] NCCL INFO === System : maxBw 12.5 totalBw 24.0 ===
lrdn2748:1549299:1549341 [0] NCCL INFO CPU/0-0 (1/1/2)
lrdn2748:1549299:1549341 [0] NCCL INFO + PCI[24.0] - PCI/0-18000 (15b3197600000000)
lrdn2748:1549299:1549341 [0] NCCL INFO               + PCI[24.0] - NIC/0-1a000
lrdn2748:1549299:1549341 [0] NCCL INFO                             + NET[12.5] - NET/0-0 (0/0/0/12.500000)
lrdn2748:1549299:1549341 [0] NCCL INFO               + PCI[24.0] - PCI/0-1b000 (15b3197600000000)
lrdn2748:1549299:1549341 [0] NCCL INFO                             + PCI[24.0] - GPU/0-1d000 (1)
lrdn2748:1549299:1549341 [0] NCCL INFO ==========================================
lrdn2748:1549299:1549341 [0] NCCL INFO GPU/0-1d000 :GPU/0-1d000 (0/5000.0/LOC) CPU/0-0 (3/24.0/PHB) NET/0-0 (6/12.5/PHB) 
lrdn2748:1549299:1549341 [0] NCCL INFO NET/0-0 :GPU/0-1d000 (6/12.5/PHB) CPU/0-0 (3/12.5/PHB) NET/0-0 (0/5000.0/LOC) 
lrdn2748:1549299:1549341 [0] NCCL INFO Setting affinity for GPU 0 to 0-15
lrdn2748:1549299:1549341 [0] NCCL INFO Pattern 4, crossNic 0, nChannels 1, bw 10.000000/10.000000, type LOC/PHB, sameChannels 1
lrdn2748:1549299:1549341 [0] NCCL INFO  0 : NET/0-0 GPU/0-1d000 NET/0-0
lrdn2748:1549299:1549341 [0] NCCL INFO Pattern 3, crossNic 0, nChannels 1, bw 20.000000/10.000000, type LOC/PHB, sameChannels 1
lrdn2748:1549299:1549341 [0] NCCL INFO  0 : NET/0-0 GPU/0-1d000 NET/0-0
lrdn2748:1549299:1549341 [0] NCCL INFO comm 0x564d2ca0bf30 rank 1 nRanks 2 nNodes 2 localRanks 1 localRank 0 MNNVL 0
lrdn2748:1549299:1549341 [0] NCCL INFO Tree 0 : 0 -> 1 -> -1/-1/-1
lrdn2748:1549299:1549341 [0] NCCL INFO Tree 1 : -1 -> 1 -> 0/-1/-1
lrdn2748:1549299:1549341 [0] NCCL INFO Ring 00 : 0 -> 1 -> 0
lrdn2748:1549299:1549341 [0] NCCL INFO Ring 01 : 0 -> 1 -> 0
lrdn2748:1549299:1549341 [0] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1
lrdn2748:1549299:1549341 [0] NCCL INFO NCCL_BUFFSIZE set by environment to 2097152.
lrdn2748:1549299:1549341 [0] NCCL INFO P2P Chunksize set to 131072
lrdn2748:1549299:1549341 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. 
lrdn2748:1549299:1549341 [0] NCCL INFO UDS: Creating service thread comm 0x564d2ca0bf30 rank 1
lrdn2748:1549299:1549341 [0] NCCL INFO misc/utils.cc:233 memory stack hunk malloc(65536)
lrdn2748:1549299:1549344 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 3
lrdn2748:1549299:1549343 [0] NCCL INFO [Proxy Service] Device 0 CPU core 1
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:42 Cuda Alloc Size 1216 pointer 0x147215c00000
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:45 Cuda Alloc Size 32 pointer 0x147215e00000
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:56 Cuda Alloc Size 8 pointer 0x147216e00000
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:42 Cuda Alloc Size 1216 pointer 0x147217000000
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:45 Cuda Alloc Size 32 pointer 0x147217200000
lrdn2748:1549299:1549341 [0] NCCL INFO channel.cc:56 Cuda Alloc Size 8 pointer 0x147217400000
lrdn2748:1549299:1549341 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
lrdn2748:1549299:1549341 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
lrdn2748:1549299:1549341 [0] NCCL INFO init.cc:447 Cuda Alloc Size 23648 pointer 0x147217600000
lrdn2748:1549299:1549341 [0] NCCL INFO init.cc:449 Cuda Alloc Size 8 pointer 0x147217800000
lrdn2748:1549299:1549341 [0] NCCL INFO init.cc:491 Cuda Host Alloc Size 1048576 pointer 0x14724a600200
lrdn2748:1549299:1549341 [0] NCCL INFO init.cc:501 Cuda Host Alloc Size 65536 pointer 0x14724a700200
lrdn2748:1549299:1549341 [0] NCCL INFO init.cc:502 Cuda Host Alloc Size 65536 pointer 0x14724a710200
lrdn2748:1549299:1549341 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
lrdn2748:1549299:1549341 [0] NCCL INFO ncclCommInitRankConfig comm 0x564d2ca0bf30 rank 1 nranks 2 cudaDev 0 nvmlDev 0 busId 1d000 commId 0x3dc40b8043af4393 - Init COMPLETE
lrdn2748:1549299:1549341 [0] NCCL INFO Init timings - ncclCommInitRankConfig: rank 1 nranks 2 total 0.65 (kernels 0.62, alloc 0.00, bootstrap 0.01, allgathers 0.00, topo 0.02, graphs 0.00, connections 0.00, rest 0.00)
lrdn2748:1549299:1549299 [0] NCCL INFO Recv: opCount 0 sendbuff (nil) recvbuff 0x14724a400000 count 10 datatype 7 op 0 root 0 comm 0x564d2ca0bf30 [nranks=2] stream 0x564d257b0850
lrdn2748:1549299:1549299 [0] NCCL INFO misc/utils.cc:233 memory stack hunk malloc(65536)
lrdn2748:1549299:1549343 [0] NCCL INFO Mem Realloc old size 0, new size 8 pointer 0x1471f0004f10
lrdn2748:1549299:1549343 [0] NCCL INFO Allocated 33030532 bytes of shared memory in /dev/shm/nccl-3BkkH4
lrdn2748:1549299:1549343 [0] NCCL INFO New proxy recv connection 0 from local rank 0, transport 2
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f65cf00 op.type=1 op.reqBuff=0x1471f0004ed0 op.respSize=16 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f65cf00
lrdn2748:1549299:1549345 [0] NCCL INFO resp.opId=0x14721f65cf00 matches expected opId=0x14721f65cf00
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Init res=0
lrdn2748:1549299:1549345 [0] NCCL INFO Connected to proxy localRank 0 -> connection 0x1471f0004f30
lrdn2748:1549299:1549346 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 4
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f65cf00 op.type=3 op.reqBuff=0x1471f0008e60 op.respSize=128 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f65cf00
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Setup res=0
lrdn2748:1549299:1549345 [0] NCCL INFO resp.opId=0x14721f65cf00 matches expected opId=0x14721f65cf00
lrdn2748:1549299:1549345 [0] NCCL INFO Channel 00/1 : 0[0] -> 1[0] [receive] via NET/Socket/0/Shared
lrdn2748:1549299:1549343 [0] NCCL INFO New proxy recv connection 1 from local rank 0, transport 2
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f65cf00 op.type=1 op.reqBuff=0x1471f0009670 op.respSize=16 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f65cf00
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Init res=0
lrdn2748:1549299:1549345 [0] NCCL INFO resp.opId=0x14721f65cf00 matches expected opId=0x14721f65cf00
lrdn2748:1549299:1549345 [0] NCCL INFO Connected to proxy localRank 0 -> connection 0x1471f0004fa8
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f65cf00 op.type=3 op.reqBuff=0x1471f00096b0 op.respSize=128 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f65cf00
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Setup res=0
lrdn2748:1549299:1549345 [0] NCCL INFO resp.opId=0x14721f65cf00 matches expected opId=0x14721f65cf00
lrdn2748:1549299:1549345 [0] NCCL INFO Channel 01/1 : 0[0] -> 1[0] [receive] via NET/Socket/0/Shared
lrdn2748:1549299:1549345 [0] NCCL INFO recvConnect ncclProxyCallAsync opId=0x14721f7a3018 &recv->proxyConn=0x14721f7a3020 connectInfo=0x14721f7afbb0
lrdn2748:1549299:1549345 [0] NCCL INFO recvConnect ncclProxyCallAsync opId=0x14721f7a7a38 &recv->proxyConn=0x14721f7a7a40 connectInfo=0x14721f7afcb0
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Connect res=0
lrdn2748:1549299:1549343 [0] NCCL INFO Received and initiated operation=Connect res=0
lrdn2748:1549299:1549343 [0] NCCL INFO transport/net.cc:559 Cuda Host Alloc Size 4194304 pointer 0x147232a00000
lrdn2748:1549299:1549343 [0] NCCL INFO transport/net.cc:972 Cuda Host Alloc Size 8192 pointer 0x14724a720200
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f7a3018 op.type=4 op.reqBuff=0x1471f0009e60 op.respSize=1400 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f7a3018
lrdn2748:1549299:1549345 [0] NCCL INFO Queuing opId=0x14721f7a3018 respBuff=0x14721f7b7bc0 respSize=1400
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Dequeued cached opId=0x14721f7a3018
lrdn2748:1549299:1549345 [0] NCCL INFO recvConnect ncclPollProxyResponse opId=0x14721f7a3018
lrdn2748:1549299:1549343 [0] NCCL INFO transport/net.cc:972 Cuda Host Alloc Size 8192 pointer 0x14724a722200
lrdn2748:1549299:1549343 [0] NCCL INFO proxyProgressAsync opId=0x14721f7a7a38 op.type=4 op.reqBuff=0x1471f00114b0 op.respSize=1400 done
lrdn2748:1549299:1549345 [0] NCCL INFO ncclPollProxyResponse Received new opId=0x14721f7a7a38
lrdn2748:1549299:1549345 [0] NCCL INFO resp.opId=0x14721f7a7a38 matches expected opId=0x14721f7a7a38
lrdn2748:1549299:1549345 [0] NCCL INFO recvConnect ncclPollProxyResponse opId=0x14721f7a7a38
Rank 1: received tensor 
[2025-10-14 16:27:59] lrdn2748:1549299:1549346 [0] transport/net_socket.cc:563 NCCL WARN NET/Socket : peer 10.128.48.245<51634> message truncated : receiving 80 bytes instead of 40. If you believe your socket network is in a healthy state, there may be a mismatch in collective sizes or environment settings (e.g. NCCL_PROTO, NCCL_ALGO) between ranks
lrdn2748:1549299:1549346 [0] NCCL INFO transport/net.cc:1393 -> 5

On stderr of rank 1 I see this error, but It seems like this occured afterwards:

stderr

[W1014 16:33:01.369413144 TCPStore.cpp:115] [c10d] recvVector failed on SocketImpl(fd=17, addr=[lrdn2748.leonardo.local]:42538, remote=[lrdn2741.leonardo.local]:6000): Failed to recv, got 0 bytes. Connection was likely closed. Did the remote server shutdown or crash?
Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:697 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x9d (0x148bd457265d in /opt/conda/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: <unknown function> + 0x6b59afd (0x148bc122dafd in /opt/conda/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #2: <unknown function> + 0x6b5baff (0x148bc122faff in /opt/conda/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #3: c10d::TCPStore::compareSet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<unsigned char, std::allocator<unsigned char> > const&, std::vector<unsigned char, std::allocator<unsigned char> > const&) + 0x1d1 (0x148bc1229911 in /opt/conda/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so)
frame #4: <unknown function> + 0xe89d92 (0x148bd0aa6d92 in /opt/conda/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x3fea2a (0x148bd001ba2a in /opt/conda/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: <unknown function> + 0x1f9196 (0x557daf22d196 in /opt/conda/bin/python3.11)
frame #7: _PyObject_MakeTpCall + 0x29b (0x557daf20a6ab in /opt/conda/bin/python3.11)
frame #8: <unknown function> + 0x227408 (0x557daf25b408 in /opt/conda/bin/python3.11)
frame #9: _PyEval_EvalFrameDefault + 0x4a29 (0x557daf21c049 in /opt/conda/bin/python3.11)
frame #10: _PyFunction_Vectorcall + 0x17f (0x557daf23ca4f in /opt/conda/bin/python3.11)
frame #11: _PyEval_EvalFrameDefault + 0x4a29 (0x557daf21c049 in /opt/conda/bin/python3.11)
frame #12: _PyFunction_Vectorcall + 0x17f (0x557daf23ca4f in /opt/conda/bin/python3.11)
frame #13: _PyEval_EvalFrameDefault + 0x4a29 (0x557daf21c049 in /opt/conda/bin/python3.11)
frame #14: <unknown function> + 0x227a92 (0x557daf25ba92 in /opt/conda/bin/python3.11)
frame #15: <unknown function> + 0x227360 (0x557daf25b360 in /opt/conda/bin/python3.11)
frame #16: <unknown function> + 0x2f873f (0x557daf32c73f in /opt/conda/bin/python3.11)
frame #17: <unknown function> + 0x2c5d64 (0x557daf2f9d64 in /opt/conda/bin/python3.11)
frame #18: <unknown function> + 0x94ac3 (0x148bda6f4ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #19: clone + 0x44 (0x148bda785a74 in /lib/x86_64-linux-gnu/libc.so.6)

[BlueCrescent]

I think, I got it. Above script starts working if you set the same dtype in the sent and received tensor. PP in torch titan works when setting the same dtype in PyTorch here:
https://github.com/pytorch/pytorch/blob/5c583e2573f29243742e00b9fa36b266c5c78bb3/torch/distributed/pipelining/stage.py#L935-L936
What I do not understand, why it worked when both processes where on the same node. Because of this, this problem probably also does not appear in the unit tests.
I opened a PR for PyTorch: pytorch/pytorch#165539