---
title: "Set Up NCCL for Multi-Node GPU Communication"
publisher: "nvidia"
type: "playbook"
updated: "2026-10-15T21:25:19.176Z"
description: "Benchmarked interconnect bandwidth for distributed training across nodes"
canonical: "https://build.nvidia.com/playbooks/nccl.md"
---

# Basic idea

NCCL (NVIDIA Collective Communication Library) enables high-performance GPU-to-GPU communication across multiple nodes. This walkthrough sets up NCCL for multi-node distributed training on two, three, or four nodes of multi-node capable hardware. You'll configure networking, build NCCL from source with Blackwell support, and validate communication between nodes.

# What you'll accomplish

You'll have a working multi-node NCCL environment that enables high-bandwidth GPU communication across nodes for distributed training workloads, with validated network performance and proper GPU topology detection.

# What to know before starting

**Required:**

- Working with Linux network configuration and netplan
- Basic understanding of MPI (Message Passing Interface) concepts
- SSH key management and passwordless authentication setup

**Optional:**

- Familiarity with RoCE / high-speed interconnect troubleshooting on multi-node capable hardware

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended defaults, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 64GB or 128GB Unified Memory | Build NCCL from source (`v2.31.2-1`, Blackwell `sm_121`) | ✅ (high-speed interconnect) |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Two, three, or four nodes of multi-node capable hardware
- Multi-node networking and inter-device SSH configured by one of these paths:
- **Recommended for supported two- to four-node DGX Spark clusters:** Complete the [NVIDIA Sync Cluster Assistant](https://docs.nvidia.com/sync/latest/cluster-assistant.html). If Cluster Assistant reports success, the connection prerequisite is complete; do not repeat a manual connection playbook. Continue with the tab for your node count.
- **Manual setup or troubleshooting:** Complete the [Connect multiple nodes for distributed workloads](https://build.nvidia.com/playbooks/connect-multiple-sparks) playbook.

**Software requirements**

- The same username on every node. When using Cluster Assistant, choose to standardize user information when prompted; the NCCL helper scripts assume matching usernames.
- NVIDIA driver installed: `nvidia-smi`
- CUDA toolkit available: `nvcc --version`
- Root/sudo privileges: `sudo whoami`

# Ancillary files

All required assets can be found [in this playbook's assets folder](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/playbook-nccl/assets).

- `setup.sh` — builds NCCL and the NCCL test suite on every node
- `launch.sh` — runs the all_gather performance test for direct, ring, or switch topologies

# Time & risk

- **Estimated time:** 30 MIN for setup and validation
- **Risk level:** Medium
- Involves network configuration changes on multi-node capable hardware
- **Rollback:** Remove the NCCL and NCCL Tests repositories from each node (`~/nccl/`, `~/nccl-tests/`)
- **Last Updated:** 10/14/2026
- Updated the NCCL build to v2.31.2-1 and removed the redundant `-f 2` flag from the `all_gather_perf` command, since `-b` and `-e` are both 16G
- The NCCL test suite build is pinned to nccl-tests v2.19.7 so `make MPI=1` links against the system OpenMPI
- Added NVIDIA Sync Cluster Assistant as the recommended network setup path and clarified that successful Cluster Assistant users should skip the manual connection playbooks

## More

- [Two nodes (direct)](/playbooks/nccl/two-nodes.md)
- [Three nodes (ring)](/playbooks/nccl/three-nodes.md)
- [Four nodes (switch)](/playbooks/nccl/four-nodes.md)
- [Troubleshooting](/playbooks/nccl/troubleshooting.md)