---
title: "Set Up NCCL for Multi-Node GPU Communication — Two nodes (direct)"
canonical: "https://build.nvidia.com/playbooks/nccl/two-nodes.md"
---

# Step 1. Confirm network connectivity

Use [NVIDIA Sync Cluster Assistant](https://docs.nvidia.com/sync/latest/cluster-assistant.html) to connect your nodes. It is the recommended path: it handles the cabling checks, interface configuration, and passwordless SSH for you.

> [!TIP]
> If Cluster Assistant successfully created your cluster, this step is already complete. Continue below.

If you choose not to use Cluster Assistant, follow the network setup instructions from the [Connect multiple nodes for distributed workloads](https://build.nvidia.com/playbooks/connect-multiple-sparks) playbook instead.

The manual connection path includes:

- Physical QSFP cable connection
- Network interface configuration (automatic or manual IP assignment)
- Passwordless SSH setup
- Network connectivity verification

# Quick start (scripts)

If you just want a working setup fast, use the helper scripts. They automate Steps 2–5 below. Complete Step 1 first, then run everything from **Node 1** (the launcher), passing each node's **management IP** (the address you SSH to):

```bash
# 1. Download the helper scripts.
curl -fsSL "https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-nccl/assets/setup.sh" -o setup.sh
curl -fsSL "https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-nccl/assets/launch.sh" -o launch.sh

# 2. Build NCCL v2.31.2-1 and the test suite on both nodes.
bash setup.sh <NODE_2_IP>

# 3. Run the all_gather test across both nodes.
#    Assumes the Ethernet interface enP7s7. On Wi-Fi, prefix the command with
#    MGMT_IFNAME=wlP9s9 (your Wi-Fi interface) and use Wi-Fi IPs. See Step 4.
bash launch.sh --topology direct <NODE_1_IP> <NODE_2_IP>
```

To understand what the scripts do — or to debug — follow the manual steps below.

---

# Step 2. Build NCCL with Blackwell support

Execute these commands on both nodes to build NCCL from source with Blackwell architecture support:

```bash
# Install dependencies and build NCCL
sudo apt-get update && sudo apt-get install -y libopenmpi-dev
git clone -b v2.31.2-1 https://github.com/NVIDIA/nccl.git ~/nccl/
cd ~/nccl/
make -j src.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"

# Set environment variables
export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"
```

# Step 3. Build NCCL test suite

Compile the NCCL test suite on **both nodes**:

```bash
# Clone and build NCCL tests (pinned: v2.20.0 and later add GIN tests that fail to link with the system OpenMPI)
git clone -b v2.19.7 https://github.com/NVIDIA/nccl-tests.git ~/nccl-tests/
cd ~/nccl-tests/
make MPI=1
```

# Step 4. Confirm interconnect ports and note each node's management IP

```bash
# Check network port status
ibdev2netdev
```

Example output:

```text
rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Down)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Down)
```

For the test command you need each node's **management IP** (the regular Ethernet address you SSH to). Find it on each node with:

```bash
ip addr show enP7s7
```

Take note of the management IP for **both nodes**.

> [!NOTE]
> These steps assume the wired **Ethernet** management interface (`enP7s7`) validated on multi-node capable hardware. If your nodes use **Wi-Fi** instead (no Ethernet), replace `enP7s7` with your Wi-Fi interface (e.g. `wlP9s9` — confirm the name with `ip -o link show`) in Step 5, and use each node's **Wi-Fi IP** as its management IP. All nodes must use the same interface — either `enP7s7` on every node or `wlP9s9` on every node, not a mix.

# Step 5. Run NCCL communication test

> [!NOTE]
> Full bandwidth can be achieved with just one QSFP cable.
> When two QSFP cables are connected, all four interfaces must be assigned IP addresses to obtain full bandwidth.

Run these commands on **Node 1** (the launcher); `mpirun` launches the test across all nodes over SSH. Replace the IP addresses and interface names with the ones you found in the previous step.

```bash
# Run the all_gather performance test across both nodes (replace the management IP addresses with the ones you found from the previous step)
mpirun -np 2 -H <management IP for Node 1>:1,<management IP for Node 2>:1 \
--mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
-x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
-x UCX_NET_DEVICES=enP7s7 \
-x NCCL_SOCKET_IFNAME=enP7s7 \
-x OMPI_MCA_btl_tcp_if_include=enP7s7 \
$HOME/nccl-tests/build/all_gather_perf
```

You can also test your NCCL setup with a larger buffer size to use more of your interconnect bandwidth.

```bash
# Run the all_gather performance test across both nodes
mpirun -np 2 -H <management IP for Node 1>:1,<management IP for Node 2>:1 \
--mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
-x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
-x UCX_NET_DEVICES=enP7s7 \
-x NCCL_SOCKET_IFNAME=enP7s7 \
-x OMPI_MCA_btl_tcp_if_include=enP7s7 \
$HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G
```

> [!NOTE]
> The IP addresses in the `mpirun` command are followed by `:1`. For example, `mpirun -np 2 -H 192.168.0.10:1,192.168.0.20:1`

# Step 6. Cleanup and rollback

```bash
# Remove NCCL build artifacts (if needed)
rm -rf ~/nccl/
rm -rf ~/nccl-tests/
```

# Step 7. Next steps

Your NCCL environment is ready for multi-node distributed training workloads on multi-node capable hardware. Next, try a larger distributed workload such as TensorRT-LLM or vLLM inference.