--- title: "Connect Two DGX Stations for Distributed Workloads" publisher: "nvidia" type: "playbook" updated: "2026-07-20T20:39:44.213Z" description: "Combined memory and compute over a direct high-speed link" canonical: "https://build.nvidia.com/station/connect-two-stations.md" --- # Basic idea Two DGX Stations can be connected directly by two **ConnectX-8 (CX8)** QSFP cables — one cable per rail — with no switch and no rail bonding. Each rail is an independent 400 Gb/s RoCEv2 path: rail 0 maps to `mlx5_0` (QSFP0-to-QSFP0) and rail 1 maps to `mlx5_1` (QSFP1-to-QSFP1). Cross-station traffic uses CX8 RoCEv2; there is no inter-station NVLink. This playbook walks through the **manual fabric bring-up** from a separate **control host** (Linux, macOS, or Windows). The control host SSHs into both DGX Stations and runs numbered setup scripts that configure rail IPs, MTU 9000, RoCEv2, and GPUDirect runtime settings, then validate the two-rail fabric before you move on to NCCL, Ray, vLLM, or other distributed workloads. # What you'll accomplish You'll connect two DGX Stations with two CX8 QSFP rails and prove the fabric in two levels: - Confirm OS SSH access and prerequisite tools on both DGX Stations. - Physically cable QSFP0-to-QSFP0 and QSFP1-to-QSFP1 and verify 400 Gb/s link on both rails. - Assign private rail IPs and MTU 9000 on each CX8 netdev. - Apply RoCEv2 runtime settings. - Validate jumbo ping, RDMA port state, and route selection on both rails. - Run a host-memory RDMA bandwidth smoke test (`ib_write_bw`) per rail. - For GPU-performance work, optionally validate the GDR path with Step 8 `--gdr`. - If needed, optionally install or build a GDR-capable `perftest` and apply ACS/Data Direct GRUB settings. After Step 7 passes, the basic two-rail RoCE fabric is up. After host-memory Step 8 passes, basic RDMA bandwidth is proven. Before using the pair for GPU-performance workloads such as NCCL, Ray, vLLM, or other distributed applications, use Step 8 `--gdr` to prove the CUDA DMA-BUF/Data Direct GPU-memory RDMA path. # What to know before starting - Basic Linux command line and SSH usage - Familiarity with network concepts: IP addresses, MTU, point-to-point links - Understanding that this playbook configures the **fabric layer only** — container RDMA exposure, NCCL environment, and distributed launchers are covered as next steps, not in the core setup - Two people or a clear cable-labeling plan helps avoid crisscrossing QSFP rails # Prerequisites **Hardware Requirements:** - Two NVIDIA DGX Station systems (DUT A and DUT B), each with CX8 adapters exposing `mlx5_0` and `mlx5_1` - Two 400G-capable QSFP cables (one per rail) for full-speed validation - A separate **control host** with a shell, `ssh`, and `tar` — Linux, macOS, or Windows **Software Requirements (on both DGX Stations):** Both systems are expected to already have the intended SOT firmware, OS, NVIDIA driver, CUDA, DOCA/OFED, MFT, and RDMA/perftest tools installed. The setup scripts check for required commands; missing core tools (other than optional MFT utilities `flint` and `mlxconfig`) indicate an incomplete SOT image. - OS SSH access for user `nvidia` (or your configured `CX8_OS_USER`) - `sudo` for privileged network and module commands on the DUTs - RDMA stack: `ibdev2netdev`, `ibv_devinfo`, `show_gids`, `ib_write_bw` - RoCE tuning tools: `mlnx_qos`, `cma_roce_tos` (where supported by the installed OFED/DOCA image) **Software Requirements (on the control host):** - `ssh` and `tar` (OpenSSH client on Windows if using PowerShell) - Bash shell scripts on Linux/macOS, **or** Python 3 with OpenSSH in `PATH` for the cross-platform `cx8_setup.py` wrapper # Ancillary files All required assets can be found [in the playbook repository](https://github.com/NVIDIA/dgx-spark-playbooks/blob/main/nvidia/station-connect-two-stations/assets/). Run the setup from the playbook `assets/` directory. - `00_env.local.example` — Template for the local configuration file; copy to `00_env.local` and edit hostnames/IPs before Step 0 - `00_env.local` — Your lab-specific configuration (git-ignored; the only file operators should create or edit) - `01_probe_access.sh` through `11_configure_acs_grub.sh` — Numbered control-host scripts for each setup step and optional GPUDirect/Data Direct remediation - `99_cleanup_runtime.sh` — Optional runtime cleanup - `cx8_setup.py` — Cross-platform Python wrapper (Windows, macOS, Linux) that runs the same steps as the shell scripts - `dut-assets/` — Linux DUT helper scripts copied to both DGX Stations by Step 2 # Time & risk * **Estimated time:** 45-60 minutes for core bring-up; longer if optional GDR perftest build or ACS/Data Direct reboot is required * **Risk level:** Medium * Incorrect QSFP cabling (crisscrossed rails) causes hard-to-debug RDMA/NCCL issues * Cable thermal or module errors (`Cable error`, `High Temperature`) require physical intervention before software setup can continue * Runtime rail configuration is temporary unless `--persist` is used; a reboot clears non-persistent settings * **Rollback:** Run `99_cleanup_runtime.sh` to remove rail IPs and reset MTU; add `--remove-persist` if netplan was written; add `--down` to bring interfaces down * **Last Updated:** 07/16/2026 * First Publication # Key terms | Term | Meaning in this setup | |---|---| | DUT | Device under test — the two DGX Station systems being connected | | Control host | The separate system where you run this playbook; requires `ssh` and `tar` | | CX8 | ConnectX-8, the NVIDIA network adapter used for the direct two-system connection | | Rail | One independent network path between the two DGX Stations (`mlx5_0`/QSFP0 and `mlx5_1`/QSFP1) | | Rail IP | Private IP assigned to one CX8 rail (for example `192.168.100.1` on DUT A, `192.168.100.2` on DUT B) | | MTU | Maximum Transmission Unit; this setup uses MTU 9000 for jumbo frames | | Jumbo ping | Large-packet ping with do-not-fragment set; proves MTU 9000 works end to end | | RDMA | Remote Direct Memory Access — low-CPU-overhead data movement between systems | | RoCE | RDMA over Converged Ethernet; DGX Station uses RoCEv2 over the CX8 rails | | GPUDirect RDMA | NVIDIA technology for RDMA to access GPU memory directly | | CUDA DMA-BUF/Data Direct | GPUDirect path used by the optional `--gdr` perftest when supported by `ib_write_bw` | | ACS/Data Direct GRUB configuration | Boot-time PCIe/ACS configuration written by `rdma_topo` when the GPU and CX8 Data Direct path are not in the required topology for GPUDirect/Data Direct | | `nvidia_peermem` | Optional legacy/module-based GPUDirect path on some stacks; default DGX Station OS images may not load it, and this playbook does not require it when CUDA DMA-BUF/Data Direct works | | MFT | NVIDIA/Mellanox Firmware Tools (`flint`, `mlxconfig`); optional for this basic setup | ## More - [Instructions](/station/connect-two-stations/instructions.md) - [Troubleshooting](/station/connect-two-stations/troubleshooting.md)