---
title: "Connect Two Nodes for Distributed Workloads — Overview"
canonical: "https://build.nvidia.com/playbooks/connect-two-stations/overview.md"
---

# Basic idea

Two multi-node capable hardware platforms can be connected directly by two **ConnectX-8 (CX8)** QSFP cables — one cable per rail — with no switch and no rail bonding. Each rail is an independent 400 Gb/s RoCEv2 path: rail 0 maps to `mlx5_0` (QSFP0-to-QSFP0) and rail 1 maps to `mlx5_1` (QSFP1-to-QSFP1). Cross-node traffic uses CX8 RoCEv2; there is no inter-node NVLink.

This playbook walks through the **manual fabric bring-up** from a separate **control host** (Linux, macOS, or Windows). The control host SSHs into both nodes and runs numbered setup scripts that configure rail IPs, MTU 9000, RoCEv2, and GPUDirect runtime settings, then validate the two-rail fabric before you move on to NCCL, Ray, vLLM, or other distributed workloads.

# What you'll accomplish

You'll connect two nodes of multi-node capable hardware with two CX8 QSFP rails and prove the fabric in two levels:

- Confirm OS SSH access and prerequisite tools on both nodes.
- Physically cable QSFP0-to-QSFP0 and QSFP1-to-QSFP1 and verify 400 Gb/s link on both rails.
- Assign private rail IPs and MTU 9000 on each CX8 netdev.
- Apply RoCEv2 runtime settings.
- Validate jumbo ping, RDMA port state, and route selection on both rails.
- Run a host-memory RDMA bandwidth smoke test (`ib_write_bw`) per rail.
- For GPU-performance work, verify the GDR-capable `perftest` flags and ACS/Data Direct topology before starting a GDR test.
- After those prerequisites pass, validate the GDR path on both rails with `./08_run_perftest_pair.sh --rail <0-or-1> --gdr`.

After `./07_validate_setup.sh` passes, the basic two-rail RoCE fabric is up. After the host-memory `./08_run_perftest_pair.sh` tests pass, basic RDMA bandwidth is proven. Before using the pair for GPU-performance workloads such as NCCL, Ray, vLLM, or other distributed applications, require the perftest-capability and ACS/Data Direct checks to pass on both DUTs, then use the GDR variant on both rails to prove the CUDA DMA-BUF/Data Direct GPU-memory RDMA path.

# What to know before starting

**Required:**

- Basic Linux command line and SSH usage
- Familiarity with network concepts: IP addresses, MTU, point-to-point links
- Understanding that this playbook configures the **fabric layer only** — container RDMA exposure, NCCL environment, and distributed launchers are covered as next steps, not in the core setup

**Optional:**

- Two people or a clear cable-labeling plan helps avoid crisscrossing QSFP rails
- Familiarity with InfiniBand/RoCE tooling such as `ibdev2netdev` and `ib_write_bw`

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Station** | DGX OS (Linux) | Large HBM + Grace DRAM | Two CX8 QSFP rails (`mlx5_0` / `mlx5_1`); control-host setup scripts | ✅ (ConnectX-8 QSFP rails) |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Two nodes of multi-node capable hardware (DUT A and DUT B), each with CX8 adapters exposing `mlx5_0` and `mlx5_1`
- Two 400G-capable QSFP cables (one per rail) for full-speed validation
- A separate **control host** with a shell, `ssh`, and `tar` — Linux, macOS, or Windows

**Software requirements**

On both nodes:

- OS SSH access for user `nvidia` (or your configured `CX8_OS_USER`)
- `sudo` for privileged network and module commands on the DUTs
- RDMA stack: `ibdev2netdev`, `ibv_devinfo`, `show_gids`, `ib_write_bw`
- RoCE tuning tools: `mlnx_qos`, `cma_roce_tos` (where supported by the installed OFED/DOCA image)
- Both systems are expected to already have firmware, OS, NVIDIA driver, CUDA, DOCA/OFED, and RDMA/perftest tools installed. The setup scripts check for required commands; missing core tools indicate an incomplete software image. MFT utilities `flint` and `mlxconfig` are optional. `nmcli` is required only when NetworkManager is active; when NetworkManager is inactive or unavailable, the supported `ip`-based configuration path does not require `nmcli`.
- For GDR validation, `ib_write_bw` must advertise both `--use_cuda_dmabuf` and `--use_data_direct`, `rdma_topo` must be installed, and the Script 11 ACS/Data Direct inspection must pass on both DUTs before Script 08 starts a GDR test.

On the control host:

- `ssh` and `tar` (OpenSSH client on Windows if using PowerShell)
- Bash shell scripts on Linux/macOS, **or** Python 3 with OpenSSH in `PATH` for the cross-platform `cx8_setup.py` wrapper

# Ancillary files

All required assets can be found [in the playbook repository](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/playbook-connect-two-stations/assets/). Run the setup from the playbook `assets/` directory.

- `00_env.local.example` — Template for the local configuration file; copy to `00_env.local` and edit hostnames/IPs before running the setup scripts
- `00_env.local` — Your lab-specific configuration (git-ignored; the only file operators should create or edit)
- `01_probe_access.sh` through `11_configure_acs_grub.sh` — Numbered control-host scripts for setup, validation, and required GPUDirect/Data Direct readiness checks
- `99_cleanup_runtime.sh` — Optional runtime cleanup
- `cx8_setup.py` — Cross-platform Python wrapper (Windows, macOS, Linux) that runs the same steps as the shell scripts
- `dut-assets/` — Linux DUT helper scripts copied to both nodes by `./02_push_assets.sh`

# Time & risk

- **Estimated time:** 60 MIN for core bring-up; longer if optional GDR perftest build or ACS/Data Direct reboot is required
- **Risk level:** Medium
- Incorrect QSFP cabling (crisscrossed rails) causes hard-to-debug RDMA/NCCL issues
- Cable thermal or module errors (`Cable error`, `High Temperature`) require physical intervention before software setup can continue
- Runtime rail configuration is temporary unless `--persist` is used; a reboot clears non-persistent settings
- **Rollback:** Run `99_cleanup_runtime.sh` to remove rail IPs and reset MTU; add `--remove-persist` if netplan was written; add `--down` to bring interfaces down
- **Last Updated:** 09/04/2026
- Hardened perftest server cleanup, positive timeout validation, and `ib_write_bw` selection (NVBug 6725953)
- Replaced authored numeric step references with action/script names and corrected `ib_write_bw` capability detection
- Consolidated two-node CX8 QSFP fabric bring-up with rail IP/MTU, RoCEv2, RDMA validation, and optional GPUDirect/Data Direct checks

# Key terms

| Term | Meaning in this setup |
|---|---|
| DUT | Device under test — the two hardware platforms being connected |
| Control host | The separate system where you run this playbook; requires `ssh` and `tar` |
| CX8 | ConnectX-8, the NVIDIA network adapter used for the direct two-node connection |
| Rail | One independent network path between the two nodes (`mlx5_0`/QSFP0 and `mlx5_1`/QSFP1) |
| Rail IP | Private IP assigned to one CX8 rail (for example `192.168.100.1` on DUT A, `192.168.100.2` on DUT B) |
| MTU | Maximum Transmission Unit; this setup uses MTU 9000 for jumbo frames |
| Jumbo ping | Large-packet ping with do-not-fragment set; proves MTU 9000 works end to end |
| RDMA | Remote Direct Memory Access — low-CPU-overhead data movement between systems |
| RoCE | RDMA over Converged Ethernet; this setup uses RoCEv2 over the CX8 rails |
| GPUDirect RDMA | NVIDIA technology for RDMA to access GPU memory directly |
| CUDA DMA-BUF/Data Direct | GPUDirect path used by the optional `--gdr` perftest when supported by `ib_write_bw` |
| ACS/Data Direct GRUB configuration | Boot-time PCIe/ACS configuration written by `rdma_topo` when the GPU and CX8 Data Direct path are not in the required topology for GPUDirect/Data Direct |
| `nvidia_peermem` | Optional legacy/module-based GPUDirect path on some stacks; default OS images may not load it, and this playbook does not require it when CUDA DMA-BUF/Data Direct works |
| MFT | NVIDIA/Mellanox Firmware Tools (`flint`, `mlxconfig`); optional for this basic setup |