Connect Two Nodes for Distributed Workloads

Basic idea

Two multi-node capable hardware platforms can be connected directly by two ConnectX-8 (CX8) QSFP cables — one cable per rail — with no switch and no rail bonding. Each rail is an independent 400 Gb/s RoCEv2 path: rail 0 maps to mlx5_0 (QSFP0-to-QSFP0) and rail 1 maps to mlx5_1 (QSFP1-to-QSFP1). Cross-node traffic uses CX8 RoCEv2; there is no inter-node NVLink.

This playbook walks through the manual fabric bring-up from a separate control host (Linux, macOS, or Windows). The control host SSHs into both nodes and runs numbered setup scripts that configure rail IPs, MTU 9000, RoCEv2, and GPUDirect runtime settings, then validate the two-rail fabric before you move on to NCCL, Ray, vLLM, or other distributed workloads.

What you'll accomplish

You'll connect two nodes of multi-node capable hardware with two CX8 QSFP rails and prove the fabric in two levels:

• Confirm OS SSH access and prerequisite tools on both nodes.
• Physically cable QSFP0-to-QSFP0 and QSFP1-to-QSFP1 and verify 400 Gb/s link on both rails.
• Assign private rail IPs and MTU 9000 on each CX8 netdev.
• Apply RoCEv2 runtime settings.
• Validate jumbo ping, RDMA port state, and route selection on both rails.
• Run a host-memory RDMA bandwidth smoke test (ib_write_bw) per rail.
• For GPU-performance work, verify the GDR-capable perftest flags and ACS/Data Direct topology before starting a GDR test.
• After those prerequisites pass, validate the GDR path on both rails with ./08_run_perftest_pair.sh --rail <0-or-1> --gdr.

After ./07_validate_setup.sh passes, the basic two-rail RoCE fabric is up. After the host-memory ./08_run_perftest_pair.sh tests pass, basic RDMA bandwidth is proven. Before using the pair for GPU-performance workloads such as NCCL, Ray, vLLM, or other distributed applications, require the perftest-capability and ACS/Data Direct checks to pass on both DUTs, then use the GDR variant on both rails to prove the CUDA DMA-BUF/Data Direct GPU-memory RDMA path.

What to know before starting

Required:

• Basic Linux command line and SSH usage
• Familiarity with network concepts: IP addresses, MTU, point-to-point links
• Understanding that this playbook configures the fabric layer only — container RDMA exposure, NCCL environment, and distributed launchers are covered as next steps, not in the core setup

Optional:

• Two people or a clear cable-labeling plan helps avoid crisscrossing QSFP rails
• Familiarity with InfiniBand/RoCE tooling such as ibdev2netdev and ib_write_bw

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• Two nodes of multi-node capable hardware (DUT A and DUT B), each with CX8 adapters exposing mlx5_0 and mlx5_1
• Two 400G-capable QSFP cables (one per rail) for full-speed validation
• A separate control host with a shell, ssh, and tar — Linux, macOS, or Windows

Software requirements

On both nodes:

• OS SSH access for user nvidia (or your configured CX8_OS_USER)
• sudo for privileged network and module commands on the DUTs
• RDMA stack: ibdev2netdev, ibv_devinfo, show_gids, ib_write_bw
• RoCE tuning tools: mlnx_qos, cma_roce_tos (where supported by the installed OFED/DOCA image)
• Both systems are expected to already have firmware, OS, NVIDIA driver, CUDA, DOCA/OFED, and RDMA/perftest tools installed. The setup scripts check for required commands; missing core tools indicate an incomplete software image. MFT utilities flint and mlxconfig are optional. nmcli is required only when NetworkManager is active; when NetworkManager is inactive or unavailable, the supported ip-based configuration path does not require nmcli.
• For GDR validation, ib_write_bw must advertise both --use_cuda_dmabuf and --use_data_direct, rdma_topo must be installed, and the Script 11 ACS/Data Direct inspection must pass on both DUTs before Script 08 starts a GDR test.

On the control host:

• ssh and tar (OpenSSH client on Windows if using PowerShell)
• Bash shell scripts on Linux/macOS, or Python 3 with OpenSSH in PATH for the cross-platform cx8_setup.py wrapper

Ancillary files

All required assets can be found in the playbook repository. Run the setup from the playbook assets/ directory.

• 00_env.local.example — Template for the local configuration file; copy to 00_env.local and edit hostnames/IPs before running the setup scripts
• 00_env.local — Your lab-specific configuration (git-ignored; the only file operators should create or edit)
• 01_probe_access.sh through 11_configure_acs_grub.sh — Numbered control-host scripts for setup, validation, and required GPUDirect/Data Direct readiness checks
• 99_cleanup_runtime.sh — Optional runtime cleanup
• cx8_setup.py — Cross-platform Python wrapper (Windows, macOS, Linux) that runs the same steps as the shell scripts
• dut-assets/ — Linux DUT helper scripts copied to both nodes by ./02_push_assets.sh

Time & risk

• Estimated time: 60 MIN for core bring-up; longer if optional GDR perftest build or ACS/Data Direct reboot is required
• Risk level: Medium
  • Incorrect QSFP cabling (crisscrossed rails) causes hard-to-debug RDMA/NCCL issues
  • Cable thermal or module errors (Cable error, High Temperature) require physical intervention before software setup can continue
  • Runtime rail configuration is temporary unless --persist is used; a reboot clears non-persistent settings
• Rollback: Run 99_cleanup_runtime.sh to remove rail IPs and reset MTU; add --remove-persist if netplan was written; add --down to bring interfaces down
• Last Updated: 09/04/2026
  • Hardened perftest server cleanup, positive timeout validation, and ib_write_bw selection (NVBug 6725953)
  • Replaced authored numeric step references with action/script names and corrected ib_write_bw capability detection
  • Consolidated two-node CX8 QSFP fabric bring-up with rail IP/MTU, RoCEv2, RDMA validation, and optional GPUDirect/Data Direct checks

Key terms