Connect Two DGX Stations for Distributed Workloads

Basic idea

Two DGX Stations can be connected directly by two ConnectX-8 (CX8) QSFP cables — one cable per rail — with no switch and no rail bonding. Each rail is an independent 400 Gb/s RoCEv2 path: rail 0 maps to mlx5_0 (QSFP0-to-QSFP0) and rail 1 maps to mlx5_1 (QSFP1-to-QSFP1). Cross-station traffic uses CX8 RoCEv2; there is no inter-station NVLink.

This playbook walks through the manual fabric bring-up from a separate control host (Linux, macOS, or Windows). The control host SSHs into both DGX Stations and runs numbered setup scripts that configure rail IPs, MTU 9000, RoCEv2, and GPUDirect runtime settings, then validate the two-rail fabric before you move on to NCCL, Ray, vLLM, or other distributed workloads.

What you'll accomplish

You'll connect two DGX Stations with two CX8 QSFP rails and prove the fabric in two levels:

• Confirm OS SSH access and prerequisite tools on both DGX Stations.
• Physically cable QSFP0-to-QSFP0 and QSFP1-to-QSFP1 and verify 400 Gb/s link on both rails.
• Assign private rail IPs and MTU 9000 on each CX8 netdev.
• Apply RoCEv2 runtime settings.
• Validate jumbo ping, RDMA port state, and route selection on both rails.
• Run a host-memory RDMA bandwidth smoke test (ib_write_bw) per rail.
• For GPU-performance work, optionally validate the GDR path with Step 8 --gdr.
• If needed, optionally install or build a GDR-capable perftest and apply ACS/Data Direct GRUB settings.

After Step 7 passes, the basic two-rail RoCE fabric is up. After host-memory Step 8 passes, basic RDMA bandwidth is proven. Before using the pair for GPU-performance workloads such as NCCL, Ray, vLLM, or other distributed applications, use Step 8 --gdr to prove the CUDA DMA-BUF/Data Direct GPU-memory RDMA path.

What to know before starting

• Basic Linux command line and SSH usage
• Familiarity with network concepts: IP addresses, MTU, point-to-point links
• Understanding that this playbook configures the fabric layer only — container RDMA exposure, NCCL environment, and distributed launchers are covered as next steps, not in the core setup
• Two people or a clear cable-labeling plan helps avoid crisscrossing QSFP rails

Prerequisites

Hardware Requirements:

• Two NVIDIA DGX Station systems (DUT A and DUT B), each with CX8 adapters exposing mlx5_0 and mlx5_1
• Two 400G-capable QSFP cables (one per rail) for full-speed validation
• A separate control host with a shell, ssh, and tar — Linux, macOS, or Windows

Software Requirements (on both DGX Stations):

Both systems are expected to already have the intended SOT firmware, OS, NVIDIA driver, CUDA, DOCA/OFED, MFT, and RDMA/perftest tools installed. The setup scripts check for required commands; missing core tools (other than optional MFT utilities flint and mlxconfig) indicate an incomplete SOT image.

• OS SSH access for user nvidia (or your configured CX8_OS_USER)
• sudo for privileged network and module commands on the DUTs
• RDMA stack: ibdev2netdev, ibv_devinfo, show_gids, ib_write_bw
• RoCE tuning tools: mlnx_qos, cma_roce_tos (where supported by the installed OFED/DOCA image)

Software Requirements (on the control host):

• ssh and tar (OpenSSH client on Windows if using PowerShell)
• Bash shell scripts on Linux/macOS, or Python 3 with OpenSSH in PATH for the cross-platform cx8_setup.py wrapper

Ancillary files

All required assets can be found in the playbook repository. Run the setup from the playbook assets/ directory.

• 00_env.local.example — Template for the local configuration file; copy to 00_env.local and edit hostnames/IPs before Step 0
• 00_env.local — Your lab-specific configuration (git-ignored; the only file operators should create or edit)
• 01_probe_access.sh through 11_configure_acs_grub.sh — Numbered control-host scripts for each setup step and optional GPUDirect/Data Direct remediation
• 99_cleanup_runtime.sh — Optional runtime cleanup
• cx8_setup.py — Cross-platform Python wrapper (Windows, macOS, Linux) that runs the same steps as the shell scripts
• dut-assets/ — Linux DUT helper scripts copied to both DGX Stations by Step 2

Time & risk

• Estimated time: 45-60 minutes for core bring-up; longer if optional GDR perftest build or ACS/Data Direct reboot is required
• Risk level: Medium
  • Incorrect QSFP cabling (crisscrossed rails) causes hard-to-debug RDMA/NCCL issues
  • Cable thermal or module errors (Cable error, High Temperature) require physical intervention before software setup can continue
  • Runtime rail configuration is temporary unless --persist is used; a reboot clears non-persistent settings
• Rollback: Run 99_cleanup_runtime.sh to remove rail IPs and reset MTU; add --remove-persist if netplan was written; add --down to bring interfaces down
• Last Updated: 07/16/2026
  • First Publication

Key terms