Enabling P2P capability on 8 RTX 4090 GPUs results in significantly lower performance in NCCL alltoall_perf tests compared to when P2P capability is disabled.

[ZP-AlwaysWin]

NVIDIA Open GPU Kernel Modules Version

NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

Description: Ubuntu 22.04.1 LTS

Kernel Release

5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA GeForce RTX 4090

Describe the bug

Enabling P2P capability on 8 RTX 4090 GPUs results in significantly lower performance in NCCL alltoall_perf tests compared to when P2P capability is disabled.

To Reproduce

Enabling P2P capability on two RTX 4090 GPUs significantly improves performance in the NCCL alltoall_perf tests compared to when P2P is disabled. However, when testing with eight GPUs, the performance gap between enabling and disabling P2P is much larger, with a severe performance drop when P2P is enabled. The relevant test data is as follows:

• P2P capability is enabled.

root@moons:~# nvidia-smi topo -p2p rw
  GPU0  GPU1  GPU2  GPU3  GPU4  GPU5  GPU6  GPU7
 GPU0 X OK  OK  OK  OK  OK  OK  OK
 GPU1 OK  X OK  OK  OK  OK  OK  OK
 GPU2 OK  OK  X OK  OK  OK  OK  OK
 GPU3 OK  OK  OK  X OK  OK  OK  OK
 GPU4 OK  OK  OK  OK  X OK  OK  OK
 GPU5 OK  OK  OK  OK  OK  X OK  OK
 GPU6 OK  OK  OK  OK  OK  OK  X OK
 GPU7 OK  OK  OK  OK  OK  OK  OK  X

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown

• The simpleP2P test passes.

Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 21.11GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Disabling peer access...
Shutting down...
Test passed

The alltoall_perf test data for two GPUs with P2P disabled:

root@moons:~/nccl-tests-master# ./build/alltoall_perf -b 8 -e 128M -f 2 -g 2
# nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 120986 on      moons device  0 [0x41] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 120986 on      moons device  1 [0x42] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             1     float    none      -1    11.87    0.00    0.00      0    11.64    0.00    0.00    N/A
          16             2     float    none      -1    11.79    0.00    0.00      0    11.47    0.00    0.00    N/A
          32             4     float    none      -1    11.92    0.00    0.00      0    11.46    0.00    0.00    N/A
          64             8     float    none      -1    11.27    0.01    0.00      0    11.59    0.01    0.00    N/A
         128            16     float    none      -1    11.45    0.01    0.01      0    11.34    0.01    0.01    N/A
         256            32     float    none      -1    11.39    0.02    0.01      0    11.24    0.02    0.01    N/A
         512            64     float    none      -1    11.24    0.05    0.02      0    11.18    0.05    0.02    N/A
        1024           128     float    none      -1    11.64    0.09    0.04      0    11.32    0.09    0.05    N/A
        2048           256     float    none      -1    11.33    0.18    0.09      0    11.16    0.18    0.09    N/A
        4096           512     float    none      -1    11.33    0.36    0.18      0    11.07    0.37    0.19    N/A
        8192          1024     float    none      -1    11.70    0.70    0.35      0    10.89    0.75    0.38    N/A
       16384          2048     float    none      -1    11.51    1.42    0.71      0    11.66    1.41    0.70    N/A
       32768          4096     float    none      -1    12.96    2.53    1.26      0    12.88    2.54    1.27    N/A
       65536          8192     float    none      -1    18.67    3.51    1.75      0    18.40    3.56    1.78    N/A
      131072         16384     float    none      -1    18.12    7.23    3.62      0    17.79    7.37    3.68    N/A
      262144         32768     float    none      -1    23.19   11.31    5.65      0    22.85   11.47    5.74    N/A
      524288         65536     float    none      -1    34.97   14.99    7.50      0    34.77   15.08    7.54    N/A
     1048576        131072     float    none      -1    56.78   18.47    9.23      0    56.60   18.52    9.26    N/A
     2097152        262144     float    none      -1    101.1   20.74   10.37      0    100.6   20.85   10.42    N/A
     4194304        524288     float    none      -1    188.1   22.29   11.15      0    186.7   22.47   11.23    N/A
     8388608       1048576     float    none      -1    357.0   23.50   11.75      0    353.3   23.74   11.87    N/A
    16777216       2097152     float    none      -1    619.9   27.07   13.53      0    576.4   29.11   14.55    N/A
    33554432       4194304     float    none      -1   1214.1   27.64   13.82      0   1129.3   29.71   14.86    N/A
    67108864       8388608     float    none      -1   2410.7   27.84   13.92      0   2221.7   30.21   15.10    N/A
   134217728      16777216     float    none      -1   4813.9   27.88   13.94      0   4396.9   30.53   15.26    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 4.85887
#

The alltoall_perf test data for two GPUs with P2P enabled:

root@moons:~/nccl-tests-master# NCCL_P2P_LEVEL=SYS ./build/alltoall_perf -b 8 -e 128M -f 2 -g 2
# nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 121058 on      moons device  0 [0x41] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 121058 on      moons device  1 [0x42] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           8             1     float    none      -1    12.28    0.00    0.00      0    11.80    0.00    0.00    N/A
          16             2     float    none      -1    12.03    0.00    0.00      0    11.36    0.00    0.00    N/A
          32             4     float    none      -1    11.90    0.00    0.00      0    11.57    0.00    0.00    N/A
          64             8     float    none      -1    11.52    0.01    0.00      0    11.72    0.01    0.00    N/A
         128            16     float    none      -1    11.59    0.01    0.01      0    11.47    0.01    0.01    N/A
         256            32     float    none      -1    11.63    0.02    0.01      0    11.60    0.02    0.01    N/A
         512            64     float    none      -1    11.78    0.04    0.02      0    11.50    0.04    0.02    N/A
        1024           128     float    none      -1    11.69    0.09    0.04      0    11.29    0.09    0.05    N/A
        2048           256     float    none      -1    11.80    0.17    0.09      0    11.52    0.18    0.09    N/A
        4096           512     float    none      -1    11.83    0.35    0.17      0    12.07    0.34    0.17    N/A
        8192          1024     float    none      -1    11.74    0.70    0.35      0    11.50    0.71    0.36    N/A
       16384          2048     float    none      -1    11.64    1.41    0.70      0    12.08    1.36    0.68    N/A
       32768          4096     float    none      -1    11.83    2.77    1.38      0    11.63    2.82    1.41    N/A
       65536          8192     float    none      -1    12.23    5.36    2.68      0    11.91    5.50    2.75    N/A
      131072         16384     float    none      -1    15.97    8.21    4.10      0    15.68    8.36    4.18    N/A
      262144         32768     float    none      -1    20.26   12.94    6.47      0    20.13   13.03    6.51    N/A
      524288         65536     float    none      -1    30.07   17.44    8.72      0    29.61   17.71    8.85    N/A
     1048576        131072     float    none      -1    42.37   24.75   12.38      0    42.20   24.85   12.42    N/A
     2097152        262144     float    none      -1    69.70   30.09   15.04      0    67.78   30.94   15.47    N/A
     4194304        524288     float    none      -1    123.4   33.99   16.99      0    118.3   35.46   17.73    N/A
     8388608       1048576     float    none      -1    223.1   37.59   18.80      0    222.1   37.77   18.88    N/A
    16777216       2097152     float    none      -1    433.9   38.67   19.33      0    423.5   39.61   19.81    N/A
    33554432       4194304     float    none      -1    849.8   39.49   19.74      0    828.4   40.51   20.25    N/A
    67108864       8388608     float    none      -1   1686.1   39.80   19.90      0   1639.1   40.94   20.47    N/A
   134217728      16777216     float    none      -1   3353.1   40.03   20.01      0   3261.5   41.15   20.58    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 6.75317
#

From the two-GPU test, it's evident that enabling P2P results in a significant performance boost.

The alltoall_perf test data for eight GPUs with P2P disabled:

root@moons:~/nccl-tests-master# ./build/alltoall_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 121126 on      moons device  0 [0x41] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 121126 on      moons device  1 [0x42] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 121126 on      moons device  2 [0x43] NVIDIA GeForce RTX 4090
#  Rank  3 Group  0 Pid 121126 on      moons device  3 [0x44] NVIDIA GeForce RTX 4090
#  Rank  4 Group  0 Pid 121126 on      moons device  4 [0x61] NVIDIA GeForce RTX 4090
#  Rank  5 Group  0 Pid 121126 on      moons device  5 [0x62] NVIDIA GeForce RTX 4090
#  Rank  6 Group  0 Pid 121126 on      moons device  6 [0x63] NVIDIA GeForce RTX 4090
#  Rank  7 Group  0 Pid 121126 on      moons device  7 [0x64] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           0             0     float    none      -1    63.21    0.00    0.00      0    60.02    0.00    0.00    N/A
           0             0     float    none      -1    62.41    0.00    0.00      0    62.10    0.00    0.00    N/A
          32             1     float    none      -1    63.09    0.00    0.00      0    62.15    0.00    0.00    N/A
          64             2     float    none      -1    63.61    0.00    0.00      0    63.87    0.00    0.00    N/A
         128             4     float    none      -1    143.9    0.00    0.00      0    63.67    0.00    0.00    N/A
         256             8     float    none      -1    63.84    0.00    0.00      0    62.88    0.00    0.00    N/A
         512            16     float    none      -1    63.01    0.01    0.01      0    62.95    0.01    0.01    N/A
        1024            32     float    none      -1    63.13    0.02    0.01      0    63.05    0.02    0.01    N/A
        2048            64     float    none      -1    65.01    0.03    0.03      0    63.81    0.03    0.03    N/A
        4096           128     float    none      -1    64.14    0.06    0.06      0    63.38    0.06    0.06    N/A
        8192           256     float    none      -1    62.95    0.13    0.11      0    62.81    0.13    0.11    N/A
       16384           512     float    none      -1    64.39    0.25    0.22      0    63.06    0.26    0.23    N/A
       32768          1024     float    none      -1    63.51    0.52    0.45      0    62.81    0.52    0.46    N/A
       65536          2048     float    none      -1    64.23    1.02    0.89      0    63.03    1.04    0.91    N/A
      131072          4096     float    none      -1    65.07    2.01    1.76      0    64.43    2.03    1.78    N/A
      262144          8192     float    none      -1    75.25    3.48    3.05      0    74.81    3.50    3.07    N/A
      524288         16384     float    none      -1    66.32    7.91    6.92      0    65.05    8.06    7.05    N/A
     1048576         32768     float    none      -1    94.40   11.11    9.72      0    93.75   11.18    9.79    N/A
     2097152         65536     float    none      -1    169.6   12.36   10.82      0    169.4   12.38   10.83    N/A
     4194304        131072     float    none      -1    300.8   13.94   12.20      0    295.4   14.20   12.42    N/A
     8388608        262144     float    none      -1    562.6   14.91   13.05      0    561.0   14.95   13.08    N/A
    16777216        524288     float    none      -1   1058.8   15.85   13.86      0   1060.8   15.82   13.84    N/A
    33554432       1048576     float    none      -1   1982.6   16.92   14.81      0   1988.3   16.88   14.77    N/A
    67108864       2097152     float    none      -1   3902.6   17.20   15.05      0   3901.4   17.20   15.05    N/A
   134217728       4194304     float    none      -1   7447.2   18.02   15.77      0   7472.3   17.96   15.72    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 4.76019
#

The alltoall_perf test data for eight GPUs with P2P enabled:

root@moons:~/nccl-tests-master# NCCL_P2P_LEVEL=SYS ./build/alltoall_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 121524 on      moons device  0 [0x41] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 121524 on      moons device  1 [0x42] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 121524 on      moons device  2 [0x43] NVIDIA GeForce RTX 4090
#  Rank  3 Group  0 Pid 121524 on      moons device  3 [0x44] NVIDIA GeForce RTX 4090
#  Rank  4 Group  0 Pid 121524 on      moons device  4 [0x61] NVIDIA GeForce RTX 4090
#  Rank  5 Group  0 Pid 121524 on      moons device  5 [0x62] NVIDIA GeForce RTX 4090
#  Rank  6 Group  0 Pid 121524 on      moons device  6 [0x63] NVIDIA GeForce RTX 4090
#  Rank  7 Group  0 Pid 121524 on      moons device  7 [0x64] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)
           0             0     float    none      -1    62.06    0.00    0.00      0    60.23    0.00    0.00    N/A
           0             0     float    none      -1    61.34    0.00    0.00      0    59.87    0.00    0.00    N/A
          32             1     float    none      -1    64.00    0.00    0.00      0    62.55    0.00    0.00    N/A
          64             2     float    none      -1    62.70    0.00    0.00      0    62.47    0.00    0.00    N/A
         128             4     float    none      -1    63.38    0.00    0.00      0    61.81    0.00    0.00    N/A
         256             8     float    none      -1    62.82    0.00    0.00      0    62.12    0.00    0.00    N/A
         512            16     float    none      -1    63.87    0.01    0.01      0    62.01    0.01    0.01    N/A
        1024            32     float    none      -1    62.26    0.02    0.01      0    62.37    0.02    0.01    N/A
        2048            64     float    none      -1    63.28    0.03    0.03      0    63.23    0.03    0.03    N/A
        4096           128     float    none      -1    63.83    0.06    0.06      0    62.95    0.07    0.06    N/A
        8192           256     float    none      -1    63.94    0.13    0.11      0    62.09    0.13    0.12    N/A
       16384           512     float    none      -1    63.99    0.26    0.22      0    63.94    0.26    0.22    N/A
       32768          1024     float    none      -1    66.91    0.49    0.43      0    65.51    0.50    0.44    N/A
       65536          2048     float    none      -1    122.3    0.54    0.47      0    120.3    0.54    0.48    N/A
      131072          4096     float    none      -1    237.7    0.55    0.48      0    235.8    0.56    0.49    N/A
      262144          8192     float    none      -1    464.4    0.56    0.49      0    459.7    0.57    0.50    N/A
      524288         16384     float    none      -1    466.4    1.12    0.98      0    460.3    1.14    1.00    N/A
     1048576         32768     float    none      -1    914.1    1.15    1.00      0    913.9    1.15    1.00    N/A
     2097152         65536     float    none      -1   1776.9    1.18    1.03      0   1786.8    1.17    1.03    N/A
     4194304        131072     float    none      -1   3445.6    1.22    1.07      0   3427.3    1.22    1.07    N/A
     8388608        262144     float    none      -1   6377.0    1.32    1.15      0   6258.2    1.34    1.17    N/A
    16777216        524288     float    none      -1    11991    1.40    1.22      0    11809    1.42    1.24    N/A
    33554432       1048576     float    none      -1    23087    1.45    1.27      0    22581    1.49    1.30    N/A
    67108864       2097152     float    none      -1    53267    1.26    1.10      0    53155    1.26    1.10    N/A
   134217728       4194304     float    none      -1   106721    1.26    1.10      0   106371    1.26    1.10    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.492673
#

From the eight-GPU test, it's clear that enabling P2P causes a severe performance drop.

Does anyone have experience in addressing this performance degradation when enabling P2P for eight GPUs?

Bug Incidence

Always

nvidia-bug-report.log.gz

～

More Info

If more information is needed, I can provide it at any time.

[mylesgoose]

I have 8 gpu rtx 4090 550.90.07-p2p working fine and some are on nvme 4x pcie cards

[ZP-AlwaysWin]

I have 8 gpu rtx 4090 550.90.07-p2p working fine and some are on nvme 4x pcie cards

I am using the 550.90.07-p2p driver and can still reproduce the issue where enabling P2P capability results in a performance drop in the alltoall_perf test. Could you please share your test data?

[mylesgoose]

Maybe it don't likey 8 cards :-)

NCCL_P2P_LEVEL=SYS ./alltoall_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 370722 on   ubuntu11 device  0 [0x01] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 370722 on   ubuntu11 device  1 [0x02] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 370722 on   ubuntu11 device  2 [0x2a] NVIDIA GeForce RTX 4090
#  Rank  3 Group  0 Pid 370722 on   ubuntu11 device  3 [0x2c] NVIDIA GeForce RTX 4090
#  Rank  4 Group  0 Pid 370722 on   ubuntu11 device  4 [0x41] NVIDIA GeForce RTX 4090
#  Rank  5 Group  0 Pid 370722 on   ubuntu11 device  5 [0x42] NVIDIA GeForce RTX 4090
#  Rank  6 Group  0 Pid 370722 on   ubuntu11 device  6 [0x61] NVIDIA GeForce RTX 4090
#  Rank  7 Group  0 Pid 370722 on   ubuntu11 device  7 [0x62] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
           0             0     float    none      -1    49.16    0.00    0.00      0    48.85    0.00    0.00    N/A
           0             0     float    none      -1    49.42    0.00    0.00      0    48.91    0.00    0.00    N/A
          32             1     float    none      -1    50.95    0.00    0.00      0    50.04    0.00    0.00    N/A
          64             2     float    none      -1    50.87    0.00    0.00      0    50.45    0.00    0.00    N/A
         128             4     float    none      -1    50.24    0.00    0.00      0    50.41    0.00    0.00    N/A
         256             8     float    none      -1    50.94    0.01    0.00      0    89.63    0.00    0.00    N/A
         512            16     float    none      -1    52.09    0.01    0.01      0    51.60    0.01    0.01    N/A
        1024            32     float    none      -1    51.65    0.02    0.02      0    66.24    0.02    0.01    N/A
        2048            64     float    none      -1    54.57    0.04    0.03      0    51.28    0.04    0.03    N/A
        4096           128     float    none      -1    51.27    0.08    0.07      0    51.27    0.08    0.07    N/A
        8192           256     float    none      -1    53.11    0.15    0.13      0    51.23    0.16    0.14    N/A
       16384           512     float    none      -1    60.79    0.27    0.24      0    56.50    0.29    0.25    N/A
       32768          1024     float    none      -1    113.8    0.29    0.25      0    113.5    0.29    0.25    N/A
       65536          2048     float    none      -1    301.3    0.22    0.19      0    229.7    0.29    0.25    N/A
      131072          4096     float    none      -1    426.3    0.31    0.27      0    427.6    0.31    0.27    N/A
      262144          8192     float    none      -1    774.7    0.34    0.30      0    687.1    0.38    0.33    N/A
      524288         16384     float    none      -1    599.9    0.87    0.76      0    474.4    1.11    0.97    N/A
     1048576         32768     float    none      -1   1159.5    0.90    0.79      0   1056.1    0.99    0.87    N/A
     2097152         65536     float    none      -1   2342.7    0.90    0.78      0   2205.7    0.95    0.83    N/A
     4194304        131072     float    none      -1   4459.3    0.94    0.82      0   4234.0    0.99    0.87    N/A
     8388608        262144     float    none      -1   8618.4    0.97    0.85      0   8355.6    1.00    0.88    N/A
    16777216        524288     float    none      -1    16657    1.01    0.88      0    16446    1.02    0.89    N/A
    33554432       1048576     float    none      -1    32761    1.02    0.90      0    32640    1.03    0.90    N/A
    67108864       2097152     float    none      -1    65425    1.03    0.90      0    65250    1.03    0.90    N/A
   134217728       4194304     float    none      -1   132129    1.02    0.89      0   130992    1.02    0.90    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.374518 
#

myles@ubuntu11:~/nccl-tests/build$ ./alltoall_perf -b 8 -e 128M -f 2 -g 8
# nThread 1 nGpus 8 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 370879 on   ubuntu11 device  0 [0x01] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 370879 on   ubuntu11 device  1 [0x02] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 370879 on   ubuntu11 device  2 [0x2a] NVIDIA GeForce RTX 4090
#  Rank  3 Group  0 Pid 370879 on   ubuntu11 device  3 [0x2c] NVIDIA GeForce RTX 4090
#  Rank  4 Group  0 Pid 370879 on   ubuntu11 device  4 [0x41] NVIDIA GeForce RTX 4090
#  Rank  5 Group  0 Pid 370879 on   ubuntu11 device  5 [0x42] NVIDIA GeForce RTX 4090
#  Rank  6 Group  0 Pid 370879 on   ubuntu11 device  6 [0x61] NVIDIA GeForce RTX 4090
#  Rank  7 Group  0 Pid 370879 on   ubuntu11 device  7 [0x62] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
           0             0     float    none      -1    66.31    0.00    0.00      0    49.37    0.00    0.00    N/A
           0             0     float    none      -1    50.01    0.00    0.00      0    49.44    0.00    0.00    N/A
          32             1     float    none      -1    50.88    0.00    0.00      0    51.28    0.00    0.00    N/A
          64             2     float    none      -1    64.19    0.00    0.00      0    51.21    0.00    0.00    N/A
         128             4     float    none      -1    51.64    0.00    0.00      0    51.11    0.00    0.00    N/A
         256             8     float    none      -1    51.55    0.00    0.00      0    50.90    0.01    0.00    N/A
         512            16     float    none      -1    51.44    0.01    0.01      0    50.86    0.01    0.01    N/A
        1024            32     float    none      -1    50.94    0.02    0.02      0    115.4    0.01    0.01    N/A
        2048            64     float    none      -1    51.44    0.04    0.03      0    50.84    0.04    0.04    N/A
        4096           128     float    none      -1    51.01    0.08    0.07      0    51.03    0.08    0.07    N/A
        8192           256     float    none      -1    51.89    0.16    0.14      0    51.40    0.16    0.14    N/A
       16384           512     float    none      -1    63.17    0.26    0.23      0    56.80    0.29    0.25    N/A
       32768          1024     float    none      -1    110.4    0.30    0.26      0    112.7    0.29    0.25    N/A
       65536          2048     float    none      -1    229.7    0.29    0.25      0    226.8    0.29    0.25    N/A
      131072          4096     float    none      -1    431.9    0.30    0.27      0    427.4    0.31    0.27    N/A
      262144          8192     float    none      -1    790.9    0.33    0.29      0    699.8    0.37    0.33    N/A
      524288         16384     float    none      -1    580.2    0.90    0.79      0    523.1    1.00    0.88    N/A
     1048576         32768     float    none      -1   1197.2    0.88    0.77      0   1132.2    0.93    0.81    N/A
     2097152         65536     float    none      -1   2355.8    0.89    0.78      0   2166.9    0.97    0.85    N/A
     4194304        131072     float    none      -1   4450.6    0.94    0.82      0   4277.7    0.98    0.86    N/A
     8388608        262144     float    none      -1   8625.1    0.97    0.85      0   8311.6    1.01    0.88    N/A
    16777216        524288     float    none      -1    16530    1.01    0.89      0    16491    1.02    0.89    N/A
    33554432       1048576     float    none      -1    32384    1.04    0.91      0    32438    1.03    0.91    N/A
    67108864       2097152     float    none      -1    65691    1.02    0.89      0    65766    1.02    0.89    N/A
   134217728       4194304     float    none      -1   132444    1.01    0.89      0   130140    1.03    0.90    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.372955 
#

myles@ubuntu11:~/nccl-tests/build$ nvidia-smi topo -p2p rw
 	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	
 GPU0	X	OK	OK	OK	OK	OK	OK	OK	
 GPU1	OK	X	OK	OK	OK	OK	OK	OK	
 GPU2	OK	OK	X	OK	OK	OK	OK	OK	
 GPU3	OK	OK	OK	X	OK	OK	OK	OK	
 GPU4	OK	OK	OK	OK	X	OK	OK	OK	
 GPU5	OK	OK	OK	OK	OK	X	OK	OK	
 GPU6	OK	OK	OK	OK	OK	OK	X	OK	
 GPU7	OK	OK	OK	OK	OK	OK	OK	X	

Legend:

  X    = Self
  OK   = Status Ok
  CNS  = Chipset not supported
  GNS  = GPU not supported
  TNS  = Topology not supported
  NS   = Not supported
  U    = Unknown
myles@ubuntu11:~/nccl-tests/build$ 

[mylesgoose]

I think this would be a normal result because with 8 cards would be saturating the bandwidth maybe? Wonder why this is so different betwen our cards. what motherboard do you use how do you connect the 8 cards? i use the pcie riser cable version pcie 5.0 for 7 of them and then a nvme adaptor for the 8th one. Avg bus bandwidth : 0.372955 of mine vs yours Avg bus bandwidth : 0.492673 but i have the cards all limited to 400w of power.

[ZP-AlwaysWin]

I think this would be a normal result because with 8 cards would be saturating the bandwidth maybe? Wonder why this is so different betwen our cards. what motherboard do you use how do you connect the 8 cards? i use the pcie riser cable version pcie 5.0 for 7 of them and then a nvme adaptor for the 8th one. Avg bus bandwidth : 0.372955 of mine vs yours Avg bus bandwidth : 0.492673 but i have the cards all limited to 400w of power.

My GPUs are not power-limited, and I’m using an Intel SPR motherboard with all 8 GPUs directly connected to the CPU, using PCIe 4.0. I don't think it’s related to bandwidth saturation, because even with just 3 GPUs, the performance significantly degrades when P2P is enabled.

[mylesgoose]

have a look it actually says NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct/direct that means its going via the cpu when more than two devices are listed.
and with just two devices shows this
NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/direct pointer

export NCCL_P2P_DISABLE=0
myles@ubuntu11:~/nccl-tests/build$ NCCL_DEBUG=INFO ./all_reduce_perf -g 3
# nThread 1 nGpus 3 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 103631 on   ubuntu11 device  0 [0x01] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 103631 on   ubuntu11 device  1 [0x02] NVIDIA GeForce RTX 4090
#  Rank  2 Group  0 Pid 103631 on   ubuntu11 device  2 [0x2b] NVIDIA GeForce RTX 4090
ubuntu11:103631:103631 [0] NCCL INFO Bootstrap : Using enp37s0f0:192.168.1.32<0>
ubuntu11:103631:103631 [0] NCCL INFO cudaDriverVersion 12060
ubuntu11:103631:103631 [0] NCCL INFO NCCL version 2.23.4+cuda12.4
ubuntu11:103631:103642 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
ubuntu11:103631:103642 [0] NCCL INFO NET/IB : No device found.
ubuntu11:103631:103642 [0] NCCL INFO NET/Socket : Using [0]enp37s0f0:192.168.1.32<0> [1]enp37s0f1:192.168.1.47<0>
ubuntu11:103631:103642 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
ubuntu11:103631:103642 [0] NCCL INFO Using network Socket
ubuntu11:103631:103644 [2] NCCL INFO Using network Socket
ubuntu11:103631:103643 [1] NCCL INFO Using network Socket
ubuntu11:103631:103644 [2] NCCL INFO ncclCommInitAll comm 0x650203e649b0 rank 2 nranks 3 cudaDev 2 nvmlDev 2 busId 2b000 commId 0xfcf1664b18e49f00 - Init START
ubuntu11:103631:103643 [1] NCCL INFO ncclCommInitAll comm 0x650203e24070 rank 1 nranks 3 cudaDev 1 nvmlDev 1 busId 2000 commId 0xfcf1664b18e49f00 - Init START
ubuntu11:103631:103642 [0] NCCL INFO ncclCommInitAll comm 0x650203de37d0 rank 0 nranks 3 cudaDev 0 nvmlDev 0 busId 1000 commId 0xfcf1664b18e49f00 - Init START
ubuntu11:103631:103643 [1] NCCL INFO Bootstrap timings total 0.001324 (create 0.000039, send 0.000161, recv 0.000395, ring 0.000121, delay 0.000000)
ubuntu11:103631:103642 [0] NCCL INFO Bootstrap timings total 0.001267 (create 0.000048, send 0.000227, recv 0.000700, ring 0.000082, delay 0.000000)
ubuntu11:103631:103644 [2] NCCL INFO Bootstrap timings total 0.001342 (create 0.000042, send 0.000175, recv 0.000675, ring 0.000234, delay 0.000001)
ubuntu11:103631:103643 [1] NCCL INFO NVLS multicast support is not available on dev 1
ubuntu11:103631:103644 [2] NCCL INFO NVLS multicast support is not available on dev 2
ubuntu11:103631:103642 [0] NCCL INFO NVLS multicast support is not available on dev 0
ubuntu11:103631:103644 [2] NCCL INFO comm 0x650203e649b0 rank 2 nRanks 3 nNodes 1 localRanks 3 localRank 2 MNNVL 0
ubuntu11:103631:103643 [1] NCCL INFO comm 0x650203e24070 rank 1 nRanks 3 nNodes 1 localRanks 3 localRank 1 MNNVL 0
ubuntu11:103631:103642 [0] NCCL INFO comm 0x650203de37d0 rank 0 nRanks 3 nNodes 1 localRanks 3 localRank 0 MNNVL 0
ubuntu11:103631:103642 [0] NCCL INFO Channel 00/04 : 0 1 2
ubuntu11:103631:103642 [0] NCCL INFO Channel 01/04 : 0 1 2
ubuntu11:103631:103642 [0] NCCL INFO Channel 02/04 : 0 1 2
ubuntu11:103631:103644 [2] NCCL INFO Trees [0] -1/-1/-1->2->1 [1] -1/-1/-1->2->1 [2] -1/-1/-1->2->1 [3] -1/-1/-1->2->1
ubuntu11:103631:103643 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0
ubuntu11:103631:103643 [1] NCCL INFO P2P Chunksize set to 131072
ubuntu11:103631:103642 [0] NCCL INFO Channel 03/04 : 0 1 2
ubuntu11:103631:103642 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1
ubuntu11:103631:103642 [0] NCCL INFO P2P Chunksize set to 131072
ubuntu11:103631:103644 [2] NCCL INFO P2P Chunksize set to 131072
ubuntu11:103631:103647 [1] NCCL INFO [Proxy Service] Device 1 CPU core 118
ubuntu11:103631:103649 [2] NCCL INFO [Proxy Service] Device 2 CPU core 59
ubuntu11:103631:103652 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 7
ubuntu11:103631:103650 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 65
ubuntu11:103631:103648 [0] NCCL INFO [Proxy Service] Device 0 CPU core 39
ubuntu11:103631:103651 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 105
ubuntu11:103631:103643 [1] NCCL INFO Channel 00 : 1[1] -> 2[2] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 00 : 2[2] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103642 [0] NCCL INFO Channel 00 : 0[0] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 01 : 1[1] -> 2[2] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 01 : 2[2] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103642 [0] NCCL INFO Channel 01 : 0[0] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 02 : 1[1] -> 2[2] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 02 : 2[2] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103642 [0] NCCL INFO Channel 02 : 0[0] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 03 : 1[1] -> 2[2] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 03 : 2[2] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103642 [0] NCCL INFO Channel 03 : 0[0] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Connected all rings
ubuntu11:103631:103644 [2] NCCL INFO Connected all rings
ubuntu11:103631:103642 [0] NCCL INFO Connected all rings
ubuntu11:103631:103644 [2] NCCL INFO Channel 00 : 2[2] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 01 : 2[2] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 02 : 2[2] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103644 [2] NCCL INFO Channel 03 : 2[2] -> 1[1] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 00 : 1[1] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 01 : 1[1] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 02 : 1[1] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103643 [1] NCCL INFO Channel 03 : 1[1] -> 0[0] via SHM/direct/direct
ubuntu11:103631:103642 [0] NCCL INFO Connected all trees
ubuntu11:103631:103644 [2] NCCL INFO Connected all trees
ubuntu11:103631:103643 [1] NCCL INFO Connected all trees
ubuntu11:103631:103658 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 80
ubuntu11:103631:103659 [1] NCCL INFO [Proxy Progress] Device 1 CPU core 121
ubuntu11:103631:103643 [1] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
ubuntu11:103631:103643 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ubuntu11:103631:103642 [0] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
ubuntu11:103631:103642 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ubuntu11:103631:103642 [0] NCCL INFO CC Off, Multi-GPU CC Off, workFifoBytes 1048576
ubuntu11:103631:103660 [2] NCCL INFO [Proxy Progress] Device 2 CPU core 97
ubuntu11:103631:103644 [2] NCCL INFO threadThresholds 8/8/64 | 24/8/64 | 512 | 512
ubuntu11:103631:103644 [2] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ubuntu11:103631:103642 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
ubuntu11:103631:103642 [0] NCCL INFO ncclCommInitAll comm 0x650203de37d0 rank 0 nranks 3 cudaDev 0 nvmlDev 0 busId 1000 commId 0xfcf1664b18e49f00 - Init COMPLETE
ubuntu11:103631:103642 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 3 total 0.56 (kernels 0.26, alloc 0.06, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.00, connections 0.21, rest 0.01)
ubuntu11:103631:103643 [1] NCCL INFO ncclCommInitAll comm 0x650203e24070 rank 1 nranks 3 cudaDev 1 nvmlDev 1 busId 2000 commId 0xfcf1664b18e49f00 - Init COMPLETE
ubuntu11:103631:103644 [2] NCCL INFO ncclCommInitAll comm 0x650203e649b0 rank 2 nranks 3 cudaDev 2 nvmlDev 2 busId 2b000 commId 0xfcf1664b18e49f00 - Init COMPLETE
ubuntu11:103631:103643 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 3 total 0.56 (kernels 0.26, alloc 0.06, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.21, rest 0.01)
ubuntu11:103631:103644 [2] NCCL INFO Init timings - ncclCommInitAll: rank 2 nranks 3 total 0.56 (kernels 0.26, alloc 0.06, bootstrap 0.00, allgathers 0.00, topo 0.02, graphs 0.00, connections 0.22, rest 0.00)
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    33554432       8388608     float     sum      -1    12835    2.61    3.49      0    12855    2.61    3.48      0
ubuntu11:103631:103631 [0] NCCL INFO comm 0x650203de37d0 rank 0 nranks 3 cudaDev 0 busId 1000 - Destroy COMPLETE
ubuntu11:103631:103631 [2] NCCL INFO comm 0x650203e649b0 rank 2 nranks 3 cudaDev 2 busId 2b000 - Destroy COMPLETE
ubuntu11:103631:103631 [1] NCCL INFO comm 0x650203e24070 rank 1 nranks 3 cudaDev 1 busId 2000 - Destroy COMPLETE
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 3.48293 
#

myles@ubuntu11:~/nccl-tests/build$ NCCL_DEBUG=INFO ./all_reduce_perf -g 2
# nThread 1 nGpus 2 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid 104198 on   ubuntu11 device  0 [0x01] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid 104198 on   ubuntu11 device  1 [0x02] NVIDIA GeForce RTX 4090
ubuntu11:104198:104198 [0] NCCL INFO Bootstrap : Using enp37s0f0:192.168.1.32<0>
ubuntu11:104198:104198 [0] NCCL INFO cudaDriverVersion 12060
ubuntu11:104198:104198 [0] NCCL INFO NCCL version 2.23.4+cuda12.4
ubuntu11:104198:104212 [0] NCCL INFO NET/Plugin: Could not find: libnccl-net.so. Using internal network plugin.
ubuntu11:104198:104212 [0] NCCL INFO NET/IB : No device found.
ubuntu11:104198:104212 [0] NCCL INFO NET/Socket : Using [0]enp37s0f0:192.168.1.32<0> [1]enp37s0f1:192.168.1.47<0>
ubuntu11:104198:104212 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so.
ubuntu11:104198:104212 [0] NCCL INFO Using network Socket
ubuntu11:104198:104213 [1] NCCL INFO Using network Socket
ubuntu11:104198:104213 [1] NCCL INFO ncclCommInitAll comm 0x64b9672f4710 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2000 commId 0x91379a7e94fa7192 - Init START
ubuntu11:104198:104212 [0] NCCL INFO ncclCommInitAll comm 0x64b9672b4af0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1000 commId 0x91379a7e94fa7192 - Init START
ubuntu11:104198:104213 [1] NCCL INFO Bootstrap timings total 0.001072 (create 0.000054, send 0.000212, recv 0.000337, ring 0.000068, delay 0.000000)
ubuntu11:104198:104212 [0] NCCL INFO Bootstrap timings total 0.001006 (create 0.000046, send 0.000226, recv 0.000458, ring 0.000054, delay 0.000000)
ubuntu11:104198:104213 [1] NCCL INFO comm 0x64b9672f4710 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
ubuntu11:104198:104213 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] 0/-1/-1->1->-1 [2] -1/-1/-1->1->0 [3] 0/-1/-1->1->-1
ubuntu11:104198:104213 [1] NCCL INFO P2P Chunksize set to 131072
ubuntu11:104198:104212 [0] NCCL INFO comm 0x64b9672b4af0 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
ubuntu11:104198:104212 [0] NCCL INFO Channel 00/04 : 0 1
ubuntu11:104198:104212 [0] NCCL INFO Channel 01/04 : 0 1
ubuntu11:104198:104212 [0] NCCL INFO Channel 02/04 : 0 1
ubuntu11:104198:104212 [0] NCCL INFO Channel 03/04 : 0 1
ubuntu11:104198:104212 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] -1/-1/-1->0->1 [2] 1/-1/-1->0->-1 [3] -1/-1/-1->0->1
ubuntu11:104198:104212 [0] NCCL INFO P2P Chunksize set to 131072
ubuntu11:104198:104216 [1] NCCL INFO [Proxy Service] Device 1 CPU core 80
ubuntu11:104198:104217 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 43
ubuntu11:104198:104218 [0] NCCL INFO [Proxy Service] Device 0 CPU core 49
ubuntu11:104198:104219 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 22
ubuntu11:104198:104212 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/direct pointer
ubuntu11:104198:104212 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/direct pointer
ubuntu11:104198:104213 [1] NCCL INFO Channel 00/0 : 1[1] -> 0[0] via P2P/direct pointer
ubuntu11:104198:104212 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/direct pointer
ubuntu11:104198:104213 [1] NCCL INFO Channel 01/0 : 1[1] -> 0[0] via P2P/direct pointer
ubuntu11:104198:104212 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/direct pointer
ubuntu11:104198:104213 [1] NCCL INFO Channel 02/0 : 1[1] -> 0[0] via P2P/direct pointer
ubuntu11:104198:104213 [1] NCCL INFO Channel 03/0 : 1[1] -> 0[0] via P2P/direct pointer
ubuntu11:104198:104213 [1] NCCL INFO Connected all rings
ubuntu11:104198:104213 [1] NCCL INFO Connected all trees
ubuntu11:104198:104212 [0] NCCL INFO Connected all rings
ubuntu11:104198:104212 [0] NCCL INFO Connected all trees
ubuntu11:104198:104220 [1] NCCL INFO [Proxy Progress] Device 1 CPU core 15
ubuntu11:104198:104221 [0] NCCL INFO [Proxy Progress] Device 0 CPU core 115
ubuntu11:104198:104212 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
ubuntu11:104198:104212 [0] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ubuntu11:104198:104212 [0] NCCL INFO CC Off, Multi-GPU CC Off, workFifoBytes 1048576
ubuntu11:104198:104213 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
ubuntu11:104198:104213 [1] NCCL INFO 4 coll channels, 4 collnet channels, 0 nvls channels, 4 p2p channels, 2 p2p channels per peer
ubuntu11:104198:104212 [0] NCCL INFO TUNER/Plugin: Could not find: libnccl-tuner.so libnccl-net.so. Using internal tuner plugin.
ubuntu11:104198:104212 [0] NCCL INFO ncclCommInitAll comm 0x64b9672b4af0 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId 1000 commId 0x91379a7e94fa7192 - Init COMPLETE
ubuntu11:104198:104212 [0] NCCL INFO Init timings - ncclCommInitAll: rank 0 nranks 2 total 0.29 (kernels 0.19, alloc 0.07, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.02, rest 0.00)
ubuntu11:104198:104213 [1] NCCL INFO ncclCommInitAll comm 0x64b9672f4710 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId 2000 commId 0x91379a7e94fa7192 - Init COMPLETE
ubuntu11:104198:104213 [1] NCCL INFO Init timings - ncclCommInitAll: rank 1 nranks 2 total 0.29 (kernels 0.20, alloc 0.06, bootstrap 0.00, allgathers 0.00, topo 0.01, graphs 0.00, connections 0.02, rest 0.00)
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    33554432       8388608     float     sum      -1   1398.2   24.00   24.00      0   1395.1   24.05   24.05      0
ubuntu11:104198:104198 [0] NCCL INFO comm 0x64b9672b4af0 rank 0 nranks 2 cudaDev 0 busId 1000 - Destroy COMPLETE
ubuntu11:104198:104198 [1] NCCL INFO comm 0x64b9672f4710 rank 1 nranks 2 cudaDev 1 busId 2000 - Destroy COMPLETE
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 24.0255 
#

[mylesgoose]

NCCL_P2P_LEVEL=PHB

[mylesgoose]

I think this would be a normal result because with 8 cards would be saturating the bandwidth maybe? Wonder why this is so different betwen our cards. what motherboard do you use how do you connect the 8 cards? i use the pcie riser cable version pcie 5.0 for 7 of them and then a nvme adaptor for the 8th one. Avg bus bandwidth : 0.372955 of mine vs yours Avg bus bandwidth : 0.492673 but i have the cards all limited to 400w of power.

My GPUs are not power-limited, and I’m using an Intel SPR motherboard with all 8 GPUs directly connected to the CPU, using PCIe 4.0. I don't think it’s related to bandwidth saturation, because even with just 3 GPUs, the performance significantly degrades when P2P is enabled.

@ZP-AlwaysWin hey found solution. Nccl is choosing cpu transfer if you says nccl level sys if more than 2 gpu are selected. Export above command and try. I tested on the cuda 12.6 and nvidia driver 560.35.03 and it used p2p for all the transfers. With 7 gpu. I'm going test with 8 gpu in a few min when plug back in another gpu. But this command NCCL_P2P_LEVEL=PHB shows on 7 gpu very high bandwidth 24 gb s.

[ZP-AlwaysWin]

I think this would be a normal result because with 8 cards would be saturating the bandwidth maybe? Wonder why this is so different betwen our cards. what motherboard do you use how do you connect the 8 cards? i use the pcie riser cable version pcie 5.0 for 7 of them and then a nvme adaptor for the 8th one. Avg bus bandwidth : 0.372955 of mine vs yours Avg bus bandwidth : 0.492673 but i have the cards all limited to 400w of power.

My GPUs are not power-limited, and I’m using an Intel SPR motherboard with all 8 GPUs directly connected to the CPU, using PCIe 4.0. I don't think it’s related to bandwidth saturation, because even with just 3 GPUs, the performance significantly degrades when P2P is enabled.

@ZP-AlwaysWin hey found solution. Nccl is choosing cpu transfer if you says nccl level sys if more than 2 gpu are selected. Export above command and try. I tested on the cuda 12.6 and nvidia driver 560.35.03 and it used p2p for all the transfers. With 7 gpu. I'm going test with 8 gpu in a few min when plug back in another gpu. But this command NCCL_P2P_LEVEL=PHB shows on 7 gpu very high bandwidth 24 gb s.

Here is the log output for the command NCCL_DEBUG=INFO NCCL_P2P_LEVEL=SYS ./alltoall_perf -b 8 -e 128M -f 2 -g 8

[ZP-AlwaysWin]

I think this would be a normal result because with 8 cards would be saturating the bandwidth maybe? Wonder why this is so different betwen our cards. what motherboard do you use how do you connect the 8 cards? i use the pcie riser cable version pcie 5.0 for 7 of them and then a nvme adaptor for the 8th one. Avg bus bandwidth : 0.372955 of mine vs yours Avg bus bandwidth : 0.492673 but i have the cards all limited to 400w of power.

My GPUs are not power-limited, and I’m using an Intel SPR motherboard with all 8 GPUs directly connected to the CPU, using PCIe 4.0. I don't think it’s related to bandwidth saturation, because even with just 3 GPUs, the performance significantly degrades when P2P is enabled.

@ZP-AlwaysWin hey found solution. Nccl is choosing cpu transfer if you says nccl level sys if more than 2 gpu are selected. Export above command and try. I tested on the cuda 12.6 and nvidia driver 560.35.03 and it used p2p for all the transfers. With 7 gpu. I'm going test with 8 gpu in a few min when plug back in another gpu. But this command NCCL_P2P_LEVEL=PHB shows on 7 gpu very high bandwidth 24 gb s.

Please provide detailed test logs or leave a contact information for further communication @mylesgoose

[ZP-AlwaysWin]

I just tested the P2P capability using version 560.35.03, and the result is the same as the previous version, with no improvement.

[mylesgoose]

NCCL_P2P_LEVEL=PHB did you try that command? When I tested my one with SYS it failed back to cpu. When I said PHB it went to p2p with 7 cards. And my one was falling back to cpu transfer. At level before. However. If I enabled a card that does not have pcie 16x bandwidth it falls back to cpu copy. If you run with info the command and see. My point is if you have one device that is not on same bandwidth as the rest it falls back to cpu. I cannot get my pc to boot at the moment with 8 cards due to asus wrx80e motherboard pcie16x 4.0 issues. Can you try running with that command and are all your devices pxie16 4.0. @ZP-AlwaysWin try to disable with export cuda visible devices any devices that are not pcie 16.

[ZP-AlwaysWin]

NCCL_P2P_LEVEL=PHB

NCCL_P2P_LEVEL=PHB I tested it, and the result is the same as without enabling P2P capability, which means P2P is effectively not enabled. @mylesgoose

[mylesgoose]

Yeah, but did you try to disable the None pcie16x cards? So for example if card device number 2 is pcie 8x and the rest are 16x. Export cuda visible devices 0,1,3,4,5,6,7 and nccl level=PHB and test if p2p is still enabled with that level as long as all fevices on dame numa node as I have seen it works with 7 cards at 16x just fine . When I did that test with 7 cards only excluding the 8th card on pcie 4x. The system worked with full p2p. If I did the same test with 7 cards and including the 4x card it sent all via the cpu. As is shown in nccl info. If I had 8 cards and p2p enabled with one of cards being 4x it would be cpu. Because it falls back to cpu on 7 cards with one being at 4x and rest at 16x. Yet works fine with 7 cards at p2p PHB as long as the 7th card is not a 4x card. This leads me to think the problem lies with NCCl software. Which we have the source code for and could find out why. Perhaps it does a quick test to see if all devices are pcie 16x 4.0 equivalent. And I can't get my motherboard working so it's being returned, so I can't prove the output works as long as the cards where all at the same speed. Hopefully, I can order a new motherboard soon. But that asus sage wrx80e is playing up with that many cards. My point is this. It is not a driver issue this is an issue with how nccl is handling that specific request depending on your hardware configuration. Maybe you have 8 cards at pcie 16x 4.0. And if you do and you have tried nccl level PHB and still failed to show p2p in the info export. Then I maybe wrong. But i pinpoint the issue to the fact I can replicate this issue with 5 devices or 6 or 7. Depending if one of the specific devices is at a slower speed pcie bandwidth than the other 4,5,6 cards I feel that explains the culprit. And I think we should investigate that path. So if you can test that that idea. Would be good.