Benchmarked interconnect bandwidth for distributed training across nodes
NCCL (NVIDIA Collective Communication Library) enables high-performance GPU-to-GPU communication across multiple nodes. This walkthrough sets up NCCL for multi-node distributed training on two, three, or four nodes of multi-node capable hardware. You'll configure networking, build NCCL from source with Blackwell support, and validate communication between nodes.
You'll have a working multi-node NCCL environment that enables high-bandwidth GPU communication across nodes for distributed training workloads, with validated network performance and proper GPU topology detection.
Required:
Optional:
Use the matrix below to confirm your hardware platform, recommended defaults, and whether multi-node applies.
| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
|---|---|---|---|---|
| DGX Spark | DGX OS (Linux) | 128 GB Unified Memory | Build NCCL from source (v2.31.2-1, Blackwell sm_121) | ✅ (high-speed interconnect) |
Hardware requirements
Software requirements
nvidia-sminvcc --versionsudo whoamiAll required assets can be found in this playbook's assets folder.
setup.sh — builds NCCL and the NCCL test suite on every nodelaunch.sh — runs the all_gather performance test for direct, ring, or switch topologies~/nccl/, ~/nccl-tests/)-f 2 flag from the all_gather_perf command, since -b and -e are both 16Gmake MPI=1 links against the system OpenMPI