Skip to main content
NVIDIA
Explore
Models
Skills
Blueprints
GPUs
Docs
Help Center
Getting Started
  1. Create and verify your account to unlock full access to NVIDIA NIM APIs.
ResourcesDeveloper ForumsContact Support
FAQs
  • View All Playbooks
    View All Playbooks

    onboarding

    • Set Up Local Network Access
    • Open WebUI with Ollama

    data science

    • Single-cell RNA Sequencing
    • Portfolio Optimization
    • CUDA-X Data Science
    • Build Knowledge Graphs with txt2kg
    • Optimized JAX

    tools

    • DGX Dashboard
    • RAG Application in AI Workbench
    • Set up Tailscale on Your Spark
    • VS Code
    • Connect Three DGX Spark in a Ring Topology
    • Connect Multiple DGX Spark through a Switch

    fine tuning

    • FLUX.1 Dreambooth LoRA Fine-tuning
    • LLaMA Factory
    • Fine-tune with NeMo
    • Fine-tune with Pytorch
    • Unsloth on DGX Spark

    use case

    • Run Hermes Agent with a Local LLM
    • cuTile Kernels
    • CLI Coding Agent
    • Run NemoClaw with a Local LLM
    • 🦞 Set Up Example NemoClaw Agents 🦞
    • Live VLM WebUI
    • Install and Use Isaac Sim and Isaac Lab
    • Vibe Coding in VS Code
    • Build and Deploy a Multi-Agent Chatbot
    • Connect Two Sparks
    • NCCL for Multiple Sparks
    • Build a Video Search and Summarization (VSS) Agent
    • Spark & Reachy Photo Booth
    • Secure AI Agents with OpenShell
    • Run OpenClaw with a Local LLM

    inference

    • Generate Images and Videos with ComfyUI
    • Serve LLMs with vLLM
    • Speculative Decoding
    • Run models with llama.cpp on DGX Spark
    • Nemotron Model Family on DGX Spark
    • Serve LLMs with SGLang
    • TRT LLM for Inference
    • Quantize Models to NVFP4 with NVIDIA Model Optimizer
    • Multi-modal Inference
    • NIM on Spark
    • LM Studio on DGX Spark
    Terms of Use
    Privacy Policy
    Your Privacy Choices
    Contact

    Copyright © 2026 NVIDIA Corporation

    NCCL for Multiple Sparks

    30 MIN

    Install and test NCCL on two, three, or four Sparks

    • DGX
    • Spark
    OverviewOverviewRun on two SparksRun on two SparksRun on three SparksRun on three SparksRun on four SparksRun on four SparksTroubleshootingTroubleshooting

    Quick start (scripts)

    If you just want a working setup fast, use the helper scripts from GitHub. They automate Steps 2-5 below. Complete Step 1 (network setup) first, then run everything from Node 1 (the launcher), passing each node's management IP (the address you SSH to):

    # 1. Download the helper scripts.
curl -fsSL https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/nccl/assets/setup.sh -o setup.sh
curl -fsSL https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/nccl/assets/launch.sh -o launch.sh

# 2. Build NCCL v2.30.7-1 and the test suite on all three nodes.
bash setup.sh <NODE_2_IP> <NODE_3_IP>

# 3. Run the all_gather test across all three nodes.
#    Assumes the Ethernet interface enP7s7. On Wi-Fi, prefix the command with
#    MGMT_IFNAME=wlP9s9 (your Wi-Fi interface) and use Wi-Fi IPs. See Step 4.
bash launch.sh --topology ring <NODE_1_IP> <NODE_2_IP> <NODE_3_IP>

    To understand what the scripts do — or to debug — follow the manual steps below.

    Step 1
    Configure network connectivity

    Follow the network setup instructions from the Connect three Sparks playbook to establish connectivity between your DGX Spark nodes.

    This includes:

    • Physical QSFP cable connection
    • Network interface configuration (automatic or manual IP assignment)
    • Passwordless SSH setup
    • Network connectivity verification

    Step 2
    Build NCCL with Blackwell support

    Execute these commands on all three nodes to build NCCL from source with Blackwell architecture support:

    # Install dependencies and build NCCL
sudo apt-get update && sudo apt-get install -y libopenmpi-dev
git clone -b v2.30.7-1 https://github.com/NVIDIA/nccl.git ~/nccl/
cd ~/nccl/
make -j src.build NVCC_GENCODE="-gencode=arch=compute_121,code=sm_121"

# Set environment variables
export CUDA_HOME="/usr/local/cuda"
export MPI_HOME="/usr/lib/aarch64-linux-gnu/openmpi"
export NCCL_HOME="$HOME/nccl/build/"
export LD_LIBRARY_PATH="$NCCL_HOME/lib:$CUDA_HOME/lib64/:$MPI_HOME/lib:$LD_LIBRARY_PATH"

    Step 3
    Build NCCL test suite

    Compile the NCCL test suite on all three nodes:

    # Clone and build NCCL tests
git clone https://github.com/NVIDIA/nccl-tests.git ~/nccl-tests/
cd ~/nccl-tests/
make MPI=1

    Step 4
    Confirm the CX-7 ports and note each node's management IP

    # Check network port status
ibdev2netdev

    Example output:

    rocep1s0f0 port 1 ==> enp1s0f0np0 (Up)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Up)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)

    For the test command you need each node's management IP (the regular Ethernet address you SSH to). Find it on each node with:

    ip addr show enP7s7

    Take note of the management IP for all three nodes.

    NOTE

    These steps assume the wired Ethernet management interface (enP7s7). If your nodes use Wi-Fi instead (no Ethernet), replace enP7s7 with your Wi-Fi interface (e.g. wlP9s9 — confirm the name with ip -o link show) in Step 5, and use each node's Wi-Fi IP as its management IP. All nodes must use the same interface — either enP7s7 on every node or wlP9s9 on every node, not a mix.

    Step 5
    Run NCCL communication test

    NOTE

    Full bandwidth can be achieved with just one QSFP cable. When two QSFP cables are connected, all four interfaces must be assigned IP addresses to obtain full bandwidth.

    Run these commands on Node 1 (the launcher); mpirun launches the test across all nodes over SSH. Replace the IP addresses and interface names with the ones you found in the previous step.

    # Set network interface environment variables (use your management network interface)
export UCX_NET_DEVICES=enP7s7
export NCCL_SOCKET_IFNAME=enP7s7
export OMPI_MCA_btl_tcp_if_include=enP7s7

# Ring-specific NCCL settings
export NCCL_IB_SUBNET_AWARE_ROUTING=1
export NCCL_NET_PLUGIN=none

# Run the all_gather performance test across all three nodes (replace the management IP addresses with the ones you found from the previous step)
mpirun -np 3 -H <management IP for Node 1>:1,<management IP for Node 2>:1,<management IP for Node 3>:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf

    You can also test your NCCL setup with a larger buffer size to use more of your 200Gbps bandwidth.

    # Set network interface environment variables (use your management network interface)
export UCX_NET_DEVICES=enP7s7
export NCCL_SOCKET_IFNAME=enP7s7
export OMPI_MCA_btl_tcp_if_include=enP7s7

# Ring-specific NCCL settings
export NCCL_IB_SUBNET_AWARE_ROUTING=1
export NCCL_NET_PLUGIN=none

# Run the all_gather performance test across all three nodes
mpirun -np 3 -H <management IP for Node 1>:1,<management IP for Node 2>:1,<management IP for Node 3>:1 \
  --mca plm_rsh_agent "ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no" \
  -x LD_LIBRARY_PATH=$LD_LIBRARY_PATH \
  $HOME/nccl-tests/build/all_gather_perf -b 16G -e 16G -f 2

    NOTE

    The IP addresses in the mpirun command are followed by :1. For example, mpirun -np 3 -H 192.168.0.10:1,192.168.0.20:1,192.168.0.30:1

    Step 6
    Cleanup and rollback

    # Rollback network configuration (if needed)
rm -rf ~/nccl/
rm -rf ~/nccl-tests/

    Step 7
    Next steps

    Your NCCL environment is ready for multi-node distributed training workloads on DGX Spark. Now you can try running a larger distributed workload such as TRT-LLM or vLLM inference.

    Resources

    • NCCL Documentation
    • DGX Spark Documentation
    • DGX Spark Forum
    • DGX Spark User Performance Guide