Set Up NCCL for Multi-Node GPU Communication

Step 1
Confirm network connectivity

Use NVIDIA Sync Cluster Assistant to connect your nodes. It is the recommended path: it handles the cabling checks, interface configuration, and passwordless SSH for you.

TIP

If Cluster Assistant successfully created your cluster, this step is already complete. Continue below.

If you choose not to use Cluster Assistant, follow the network setup instructions from the Connect multiple nodes for distributed workloads playbook instead.

The manual connection path includes:

Physical QSFP cable connection
Network interface configuration (automatic or manual IP assignment)
Passwordless SSH setup
Network connectivity verification

Quick start (scripts)

If you just want a working setup fast, use the helper scripts. They automate Steps 2–5 below. Complete Step 1 first, then run everything from Node 1 (the launcher), passing each node's management IP (the address you SSH to):

Bash

# 1. Download the helper scripts.
curl -fsSL "https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-nccl/assets/setup.sh" -o setup.sh
curl -fsSL "https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-nccl/assets/launch.sh" -o launch.sh

# 2. Build NCCL v2.31.2-1 and the test suite on both nodes.
bash setup.sh <NODE_2_IP>

# 3. Run the all_gather test across both nodes.
#    Assumes the Ethernet interface enP7s7. On Wi-Fi, prefix the command with
#    MGMT_IFNAME=wlP9s9 (your Wi-Fi interface) and use Wi-Fi IPs. See Step 4.
bash launch.sh --topology direct <NODE_1_IP> <NODE_2_IP>

To understand what the scripts do — or to debug — follow the manual steps below.

Step 2
Build NCCL with Blackwell support

Execute these commands on both nodes to build NCCL from source with Blackwell architecture support:

Bash

# Install dependencies and build NCCL
sudo apt-get update && sudo apt-get install -y libopenmpi-dev
git clone -b v2.31.2-1 https://github.com/NVIDIA/nccl.git ~/nccl/
cd ~/nccl/
make -j src.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"

# Set environment variables
export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

Step 3
Build NCCL test suite

Compile the NCCL test suite on both nodes:

Bash

# Clone and build NCCL tests (pinned: v2.20.0 and later add GIN tests that fail to link with the system OpenMPI)
git clone -b v2.19.7 https://github.com/NVIDIA/nccl-tests.git ~/nccl-tests/
cd ~/nccl-tests/
make MPI=1

Step 4
Confirm interconnect ports and note each node's management IP

Bash

# Check network port status
ibdev2netdev

Example output:

Text

rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)

For the test command you need each node's management IP (the regular Ethernet address you SSH to). Find it on each node with:

Bash

ip addr show enP7s7

Take note of the management IP for both nodes.

NOTE

These steps assume the wired Ethernet management interface (enP7s7) validated on multi-node capable hardware. If your nodes use Wi-Fi instead (no Ethernet), replace enP7s7 with your Wi-Fi interface (e.g. wlP9s9 — confirm the name with ip -o link show) in Step 5, and use each node's Wi-Fi IP as its management IP. All nodes must use the same interface — either enP7s7 on every node or wlP9s9 on every node, not a mix.

Step 5
Run NCCL communication test

NOTE

Full bandwidth can be achieved with just one QSFP cable. When two QSFP cables are connected, all four interfaces must be assigned IP addresses to obtain full bandwidth.

Run these commands on Node 1 (the launcher); mpirun launches the test across all nodes over SSH. Replace the IP addresses and interface names with the ones you found in the previous step.

Bash

# Run the all_gather performance test across both nodes (replace the management IP addresses with the ones you found from the previous step)
mpirun -np 2 -H <management IP for Node 1>:1,<management IP for Node 2>:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  -x UCX_NET_DEVICES=enP7s7 \
  -x NCCL_SOCKET_IFNAME=enP7s7 \
  -x OMPI_MCA_btl_tcp_if_include=enP7s7 \
  $HOME/nccl-tests/build/all_gather_perf

You can also test your NCCL setup with a larger buffer size to use more of your interconnect bandwidth.

Bash

# Run the all_gather performance test across both nodes
mpirun -np 2 -H <management IP for Node 1>:1,<management IP for Node 2>:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  -x UCX_NET_DEVICES=enP7s7 \
  -x NCCL_SOCKET_IFNAME=enP7s7 \
  -x OMPI_MCA_btl_tcp_if_include=enP7s7 \
  $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G

NOTE

The IP addresses in the mpirun command are followed by :1. For example, mpirun -np 2 -H 192.168.0.10:1,192.168.0.20:1

Step 6
Cleanup and rollback

Bash

# Remove NCCL build artifacts (if needed)
rm -rf ~/nccl/
rm -rf ~/nccl-tests/

Step 7
Next steps

Your NCCL environment is ready for multi-node distributed training workloads on multi-node capable hardware. Next, try a larger distributed workload such as TensorRT-LLM or vLLM inference.

Set Up NCCL for Multi-Node GPU Communication

Step 1Confirm network connectivity

Quick start (scripts)

Step 2Build NCCL with Blackwell support

Step 3Build NCCL test suite

Step 4Confirm interconnect ports and note each node's management IP

Step 5Run NCCL communication test

Step 6Cleanup and rollback

Step 7Next steps

Resources

Step 1
Confirm network connectivity

Step 2
Build NCCL with Blackwell support

Step 3
Build NCCL test suite

Step 4
Confirm interconnect ports and note each node's management IP

Step 5
Run NCCL communication test

Step 6
Cleanup and rollback

Step 7
Next steps