Skip to main content
NVIDIA
Explore
Models
Skills
Blueprints
GPUs
Docs
Help Center
Getting Started
  1. Create and verify your account to unlock full access to NVIDIA NIM APIs.
ResourcesDeveloper ForumsContact Support
FAQs
  • View All Playbooks
    View All Playbooks

    onboarding

    • Connect Two DGX Stations for Distributed Workloads
    • MIG on DGX Station

    data science

    • Topic Modeling
    • Build Knowledge Graphs with txt2kg

    tools

    • Quantize Models to NVFP4 with NVIDIA Model Optimizer

    fine tuning

    • Train a Chat Model with NanoChat
    • NVFP4 Pretraining with Megatron Bridge

    use case

    • Run NemoClaw with a Local LLM
    • DGX Station AI Skills and dgx-assist
    • Secure AI Agents with OpenShell
    • Local Coding Agent
    • Profiler-Driven Kernel Optimization for Fine-Tuning
    • Local Healthcare Agent on DGX Station

    inference

    • Serve LLMs with vLLM
    • Generate Images and Videos with ComfyUI
    • Isaac GR00T N1.6 Fine-Tuning
    • Serve LLMs with SGLang

    Connect Two DGX Stations for Distributed Workloads

    60 MIN

    Combined memory and compute over a direct high-speed link

    • DGX Station
    • Multi-Node
    • NCCL
    View on GitHub
    OverviewOverviewInstructionsInstructionsTroubleshootingTroubleshooting
    SymptomCauseFix
    mlx5_0 or mlx5_1 is missing in Step 3 outputCX8 driver not loaded, PCIe issue, or incomplete SOT imageRun lspci | grep -i mellanox and ibdev2netdev on the DUT. Confirm DOCA/OFED and the NVIDIA driver are installed. Contact NVIDIA support if the HCA is not visible.
    Required command missing in Step 1 (other than flint / mlxconfig)SOT software image missing a core RDMA or network toolStop setup. Install or restore the missing package from the SOT image (rdma-core, perftest, ethtool, etc.) or contact NVIDIA support.
    Step 4 reports link down on one or both railsCable not seated, wrong QSFP port, or crisscrossed railsConfirm QSFP0-to-QSFP0 and QSFP1-to-QSFP1 with no crisscross. Reseat both ends on the failed rail and rerun ./04_check_cable_presence.sh.
    Step 4 reports Cable error or High TemperatureQSFP module thermal or cable faultStop software setup. Let the module cool, verify CX8/QSFP airflow, reseat the cable. Swap the cable with the other rail or replace it if the condition returns.
    Step 4 reports speed_ok=no or 200G instead of 400GCable not 400G-capable, degraded negotiation, or port/module issueReplace with 400G-capable QSFP cables. Limited-speed validation (Steps 5–8) is possible, but do not claim 800G pass until both rails report 400G.
    Step 5 or 6 fails with permission errorsMissing sudo access for the SSH userEnsure the nvidia user (or CX8_OS_USER) can run privileged network commands. Enter the DUT OS password at both the SSH and [sudo] prompts.
    Step 7 reports missing rail IP or wrong MTUStep 5 did not complete or interface was resetRerun ./05_configure_rails_runtime.sh and then ./07_validate_setup.sh.
    Step 7 jumbo ping fails on one railMTU mismatch, wrong route, or cable/link issue on that railCheck ip addr and ip route on both DUTs for the failed rail subnet. Rerun Step 4 for link health, then Step 5 for IP/MTU.
    Step 7 reports Link detected: no or Speed: UnknownPhysical link not up despite earlier stepsRerun ./04_check_cable_presence.sh. Inspect cable, port, and module before retrying Step 7.
    Step 7 reports Link detected: no (Overheat), High Temperature, or Cable errorQSFP/CX8 thermal or cable conditionStop software setup. Let the module cool, verify airflow, reseat the cable, and swap/replace the failed rail cable if the condition returns.
    Step 7 yellow warning that rails are below 400GDegraded link speed from cable, port, or negotiationContinue only for limited-speed functional validation. Do not treat Step 8 results as an 800G pass until Step 4 reports 400G on both rails.
    Yellow WARN for cma_roce_tos in Step 6RoCE ToS tool not available or not applicable on this imageContinue to Step 7 for basic RoCE validation. ToS tuning is optional for the basic fabric path.
    nvidia_peermem is not loadedNormal on some DGX Station OS images when CUDA DMA-BUF/Data Direct is the GPUDirect pathContinue. Basic rail/RoCE validation is not blocked. Step 8 --gdr can pass through CUDA DMA-BUF/Data Direct if ib_write_bw supports --use_cuda_dmabuf and --use_data_direct.
    nvidia_peermem fails to load (Invalid argument) after CX8_TRY_NVIDIA_PEERMEM=1The optional peermem-based GPUDirect path is unavailable on this stackDo not treat this as a basic setup failure. Use the CUDA DMA-BUF/Data Direct --gdr path first. Collect Step 6/8 logs only if the --gdr path also fails.
    Step 6 or Step 8 says ib_write_bw does not support --use_cuda_dmabuf or --use_data_directInstalled perftest is too old or was built without CUDA DMA-BUF/Data Direct supportRun host-memory Step 8 first for basic RDMA. If GPU-memory RDMA is required, run ./10_install_perftest_gdr.sh, then rerun Step 6 and Step 8 --gdr.
    Step 8 --gdr reaches CUDA DMA-BUF/Data Direct but fails with mlx5dv_reg_dmabuf_mr / Couldn't allocate MR with error=524ACS/Data Direct/IOMMU prerequisites are not configured for the GPU and ConnectX pathRun sudo rdma_topo check or ./11_configure_acs_grub.sh to confirm. If ACS/IOMMU checks fail and GDR is required, run ./11_configure_acs_grub.sh --apply, reboot both DUTs, then rerun Steps 5, 6, 7, and Step 8 --gdr. Step 3 is only needed if OS/packages changed; Step 4 is only needed if cable/link state is uncertain.
    Step 8 perftest fails or shows very low bandwidthWrong rail selected, firewall, or degraded link speedConfirm Step 7 passed. Run perftest per rail separately. Check Step 4 speed reports. Verify rail IPs and routes match the selected --rail index.
    NCCL shows NET/Socket instead of NET/IB after fabric passesContainer missing RDMA devices, wrong NCCL_IB_HCA, or launcher configExpose /dev/infiniband to the runtime. Set NCCL_IB_HCA=mlx5_0,mlx5_1. Use NCCL_DEBUG=INFO and confirm both HCAs appear in the log.
    rdma_topo check reports ACS/Data Direct prerequisites not metPCIe topology or ACS configuration not set for GPUDirect/Data DirectComplete basic two-rail setup first (Step 7 pass). Run Step 11 dry-run to capture the current state, then use ./11_configure_acs_grub.sh --apply only when GPUDirect/Data Direct validation is required. Reboot before retesting, then restore runtime setup with Steps 5, 6, and 7 before Step 8 --gdr.
    SSH control-socket reset or repeated connection-reuse failureSSH multiplexing is incompatible with the control-host environmentRetry with CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail <0-or-1>. This affects only SSH connection reuse, not RDMA, Data Direct, or perftest behavior. Windows Git Bash/MSYS/Cygwin disables multiplexing automatically.
    Rail settings lost after rebootStep 5 run without --persistExpected for runtime setup. Rerun ./05_configure_rails_runtime.sh after reboot, or use --persist if settings must survive reboot.
    Persistent netplan needs removalStep 5 was run with --persist and you want to undoRun ./99_cleanup_runtime.sh --remove-persist or manually remove /etc/netplan/60-cx8-fabric.yaml on both DUTs and sudo netplan apply.

    Stop conditions

    Stop and inspect before changing anything else if:

    • mlx5_0 or mlx5_1 is missing
    • A rail remains down after the cable is connected
    • Jumbo ping fails on either rail
    • RDMA tools are missing
    • CUDA DMA-BUF/Data Direct is unavailable or Step 8 --gdr fails when GPU-memory RDMA is required
    • rdma_topo check says ACS/Data Direct prerequisites are not met

    Add ACS/Data Direct grub configuration with Step 11 only after the basic two-rail setup is proven and GPUDirect/Data Direct validation requires it.

    Resources

    • NVIDIA DOCA Documentation
    • NVIDIA NCCL Documentation
    • RDMA perftest (ib_write_bw)
    Terms of Use
    Privacy Policy
    Your Privacy Choices
    Contact

    Copyright © 2026 NVIDIA Corporation