Skip to content

Hang with nccl all_reduce_perf on 8 cards RTX 4090 #45

Closed
Closed
Hang with nccl all_reduce_perf on 8 cards RTX 4090#45
Labels
bugSomething isn't working

Description

@cunfate
cunfate
opened on Jun 20, 2025

NVIDIA Open GPU Kernel Modules Version

570.148.08

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

  • I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Ubuntu 22.04.5 LTS

Kernel Release

Linux i-56butc28zq2jtjaluzq7va7p6 5.15.0-136-generic NVIDIA#147-Ubuntu SMP Sat Mar 15 15:53:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

  • I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 4090

Describe the bug

When I run nccl all_reduce_perf, it hanged.
command:
NCCL_P2P_DISABLE=0 NCCL_DEBUG=TRACE ./all_reduce_perf -b 8 -e 2G -w 2 -n 10 -f 2 -g 2

output:

# nThread 1 nGpus 2 minBytes 8 maxBytes 2147483648 step: 2(factor) warmup iters: 2 iters: 10 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   7995 on i-56butc28zq2jtjaluzq7va7p6 device  0 [0000:2c:00] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid   7995 on i-56butc28zq2jtjaluzq7va7p6 device  1 [0000:2d:00] NVIDIA GeForce RTX 4090
i-56butc28zq2jtjaluzq7va7p6:7995:7995 [0] NCCL INFO Bootstrap: Using bond0:10.0.11.5<0>
i-56butc28zq2jtjaluzq7va7p6:7995:7995 [0] NCCL INFO cudaDriverVersion 12080
i-56butc28zq2jtjaluzq7va7p6:7995:7995 [0] NCCL INFO NCCL version 2.26.5+cuda12.9
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal net plugin.
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO NET/IB : Using [0]mlx5_bond_0:1/RoCE [RO]; OOB bond0:10.0.11.5<0>
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Using network IB
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO Using network IB
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO ncclCommInitAll comm 0x563ec98ab6c0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xfeee037c5f86501 - Init START
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO ncclCommInitAll comm 0x563ec979a7c0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 2c000 commId 0xfeee037c5f86501 - Init START
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO Bootstrap timings total 0.000512 (create 0.000023, send 0.000101, recv 0.000144, ring 0.000018, delay 0.000000)
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Bootstrap timings total 0.000473 (create 0.000019, send 0.000085, recv 0.000159, ring 0.000016, delay 0.000000)
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO comm 0x563ec98ab6c0 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO comm 0x563ec979a7c0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO P2P Chunksize set to 131072
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Channel 00/04 : 0 1
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Channel 01/04 : 0 1
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Channel 02/04 : 0 1
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Channel 03/04 : 0 1
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO P2P Chunksize set to 131072
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 1
i-56butc28zq2jtjaluzq7va7p6:7995:8108 [1] NCCL INFO [Proxy Service] Device 1 CPU core 24
i-56butc28zq2jtjaluzq7va7p6:7995:8109 [0] NCCL INFO [Proxy Service] Device 0 CPU core 25
i-56butc28zq2jtjaluzq7va7p6:7995:8110 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 26
i-56butc28zq2jtjaluzq7va7p6:7995:8111 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 27
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO CC Off, workFifoBytes 1048576
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO ncclCommInitAll comm 0x563ec98ab6c0 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2d000 commId 0xfeee037c5f86501 - Init COMPLETE
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO ncclCommInitAll comm 0x563ec979a7c0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 2c000 commId 0xfeee037c5f86501 - Init COMPLETE
i-56butc28zq2jtjaluzq7va7p6:7995:8104 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.31 (kernels 0.21, alloc 0.09, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
i-56butc28zq2jtjaluzq7va7p6:7995:8103 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.31 (kernels 0.21, alloc 0.09, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
i-56butc28zq2jtjaluzq7va7p6:7995:8112 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8113 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8112 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8113 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8112 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8113 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8112 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8113 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/direct pointer
i-56butc28zq2jtjaluzq7va7p6:7995:8113 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
i-56butc28zq2jtjaluzq7va7p6:7995:8112 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
^C^C^C^C^C^C

If p2p disabled, things all going well.
NCCL_P2P_DISABLE=1 NCCL_DEBUG=TRACE ./all_reduce_perf -b 8 -e 2G -w 2 -n 10 -f 2 -g 2

# nThread 1 nGpus 2 minBytes 8 maxBytes 2147483648 step: 2(factor) warmup iters: 2 iters: 10 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid   9822 on i-56butc28zq2jtjaluzq7va7p6 device  0 [0000:2c:00] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid   9822 on i-56butc28zq2jtjaluzq7va7p6 device  1 [0000:2d:00] NVIDIA GeForce RTX 4090
i-56butc28zq2jtjaluzq7va7p6:9822:9822 [0] NCCL INFO Bootstrap: Using bond0:10.0.11.5<0>
i-56butc28zq2jtjaluzq7va7p6:9822:9822 [0] NCCL INFO cudaDriverVersion 12080
i-56butc28zq2jtjaluzq7va7p6:9822:9822 [0] NCCL INFO NCCL version 2.26.5+cuda12.9
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal net plugin.
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO NET/IB : Using [0]mlx5_bond_0:1/RoCE [RO]; OOB bond0:10.0.11.5<0>
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Using network IB
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO Using network IB
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO ncclCommInitAll comm 0x55d905b24190 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2d000 commId 0x2d6e1ab33284f324 - Init START
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO ncclCommInitAll comm 0x55d905a14550 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 2c000 commId 0x2d6e1ab33284f324 - Init START
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO RAS client listening socket at 127.0.0.1<28028>
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Bootstrap timings total 0.000491 (create 0.000019, send 0.000092, recv 0.000164, ring 0.000015, delay 0.000000)
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO Bootstrap timings total 0.000534 (create 0.000021, send 0.000108, recv 0.000145, ring 0.000011, delay 0.000000)
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO NCCL_P2P_DISABLE set by environment to 1
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO Setting affinity for GPU 1 to ffff,ffffffff,00000000,0000ffff,ffffffff
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Setting affinity for GPU 0 to ffff,ffffffff,00000000,0000ffff,ffffffff
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO comm 0x55d905a14550 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO comm 0x55d905b24190 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Channel 00/02 : 0 1
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO P2P Chunksize set to 131072
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Channel 01/02 : 0 1
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO P2P Chunksize set to 131072
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 0 directMode 1
i-56butc28zq2jtjaluzq7va7p6:9822:9934 [0] NCCL INFO [Proxy Service] Device 0 CPU core 106
i-56butc28zq2jtjaluzq7va7p6:9822:9936 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 12
i-56butc28zq2jtjaluzq7va7p6:9822:9935 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 11
i-56butc28zq2jtjaluzq7va7p6:9822:9933 [1] NCCL INFO [Proxy Service] Device 1 CPU core 105
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO CC Off, workFifoBytes 1048576
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so. Using internal tuner plugin.
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO ncclCommInitAll comm 0x55d905b24190 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2d000 commId 0x2d6e1ab33284f324 - Init COMPLETE
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO ncclCommInitAll comm 0x55d905a14550 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 2c000 commId 0x2d6e1ab33284f324 - Init COMPLETE
i-56butc28zq2jtjaluzq7va7p6:9822:9929 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.23 (kernels 0.14, alloc 0.08, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
i-56butc28zq2jtjaluzq7va7p6:9822:9928 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.23 (kernels 0.13, alloc 0.08, bootstrap 0.00, allgathers 0.00, topo 0.00, graphs 0.00, connections 0.00, rest 0.00)
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
i-56butc28zq2jtjaluzq7va7p6:9822:9937 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
i-56butc28zq2jtjaluzq7va7p6:9822:9938 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
i-56butc28zq2jtjaluzq7va7p6:9822:9937 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
i-56butc28zq2jtjaluzq7va7p6:9822:9938 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
i-56butc28zq2jtjaluzq7va7p6:9822:9937 [1] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
i-56butc28zq2jtjaluzq7va7p6:9822:9938 [0] NCCL INFO Connected all rings, use ring PXN 0 GDR 1
           8             2     float     sum      -1     8.49    0.00    0.00      0     8.37    0.00    0.00      0
          16             4     float     sum      -1     8.26    0.00    0.00      0     8.25    0.00    0.00      0
          32             8     float     sum      -1     8.38    0.00    0.00      0     8.38    0.00    0.00      0
          64            16     float     sum      -1     8.56    0.01    0.01      0     8.30    0.01    0.01      0
         128            32     float     sum      -1     8.32    0.02    0.02      0     8.25    0.02    0.02      0
         256            64     float     sum      -1     8.37    0.03    0.03      0     8.25    0.03    0.03      0
         512           128     float     sum      -1     8.57    0.06    0.06      0     8.96    0.06    0.06      0
        1024           256     float     sum      -1     8.41    0.12    0.12      0     8.32    0.12    0.12      0
        2048           512     float     sum      -1     8.38    0.24    0.24      0     8.24    0.25    0.25      0
        4096          1024     float     sum      -1     8.69    0.47    0.47      0     8.39    0.49    0.49      0
        8192          2048     float     sum      -1     9.43    0.87    0.87      0     8.87    0.92    0.92      0
       16384          4096     float     sum      -1    10.03    1.63    1.63      0    10.09    1.62    1.62      0
       32768          8192     float     sum      -1    12.84    2.55    2.55      0    12.70    2.58    2.58      0
       65536         16384     float     sum      -1    18.29    3.58    3.58      0    18.04    3.63    3.63      0
      131072         32768     float     sum      -1    29.02    4.52    4.52      0    28.48    4.60    4.60      0
      262144         65536     float     sum      -1    45.84    5.72    5.72      0    45.22    5.80    5.80      0
      524288        131072     float     sum      -1    58.77    8.92    8.92      0    58.90    8.90    8.90      0
     1048576        262144     float     sum      -1    93.74   11.19   11.19      0    93.56   11.21   11.21      0
     2097152        524288     float     sum      -1    164.6   12.74   12.74      0    167.3   12.54   12.54      0
     4194304       1048576     float     sum      -1    302.6   13.86   13.86      0    303.9   13.80   13.80      0
     8388608       2097152     float     sum      -1    571.1   14.69   14.69      0    570.9   14.69   14.69      0
    16777216       4194304     float     sum      -1   1125.8   14.90   14.90      0   1130.4   14.84   14.84      0
    33554432       8388608     float     sum      -1   2227.0   15.07   15.07      0   2231.4   15.04   15.04      0
    67108864      16777216     float     sum      -1   4434.7   15.13   15.13      0   4421.7   15.18   15.18      0
   134217728      33554432     float     sum      -1   8774.7   15.30   15.30      0   8807.8   15.24   15.24      0
   268435456      67108864     float     sum      -1    17553   15.29   15.29      0    17443   15.39   15.39      0
   536870912     134217728     float     sum      -1    34797   15.43   15.43      0    34785   15.43   15.43      0
  1073741824     268435456     float     sum      -1    68753   15.62   15.62      0    68602   15.65   15.65      0
  2147483648     536870912     float     sum      -1   136671   15.71   15.71      0   136300   15.76   15.76      0

To Reproduce

Build and install p2p driver.
install cuda 11.5
Install nccl
build nccl-tests and run
NCCL_P2P_DISABLE=0 NCCL_DEBUG=TRACE ./all_reduce_perf -b 8 -e 2G -w 2 -n 10 -f 2 -g 2

Bug Incidence

Always

nvidia-bug-report.log.gz

nvidia-bug-report.log.gz

More Info

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions