Connect Two DGX Stations for Distributed Workloads

Get the setup package and create local configuration

Clone the playbook repository on your control host and create the local environment file.

git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-station-playbooks/nvidia/station-connect-two-stations/assets
cp 00_env.local.example 00_env.local

Edit 00_env.local before continuing. Set the Station A/B names and OS hosts for your pair. The OS user and rail CIDRs have defaults; change them only if your setup differs.

Passwords are intentionally not stored in this directory. SSH and sudo will prompt when needed. 00_env.local is ignored by git.

If the default private rail subnets conflict with your lab, change the four STATION_*_RAIL*_CIDR values before Step 5. Keep rail 0 and rail 1 on different subnets.

Overall flow

Use Steps 1-8 without --gdr as the core bring-up path. That proves both rails are cabled, configured, routable, jumbo-ping clean, and RDMA-capable with host memory.

Use Steps 8 --gdr, 10, and 11 only when you need GPU-memory RDMA/Data Direct performance validation. Those steps are intentionally separate because they may require package installation, ACS/Data Direct GRUB configuration, and reboot.

Command style by control host

The DUTs are Linux systems, so the copied dut-assets/*.sh helpers are Bash scripts on the DUTs. The control host can be Windows, macOS, or Linux:

The Python wrapper requires Python 3 and an OpenSSH client in PATH. On Windows, enable or install the Windows OpenSSH client first if ssh is not available.

The Python wrapper uses the same 00_env.local file as the shell scripts. Numeric aliases match the setup scripts:

For unattended lab loops, configure SSH keys and passwordless sudo outside this playbook. The playbook supports that setup but does not create it or store credentials.

Probe OS SSH access

Confirm that both DGX Stations are reachable over OS SSH and that the expected user can run basic shell commands. This step also lists whether the tools needed for CX8 setup are already installed.

./01_probe_access.sh

If SSH reports an unknown host key, verify the DUT identity and answer the OpenSSH prompt. This is intentional: the default SSH policy is StrictHostKeyChecking=ask, not automatic trust-on-first-use. In controlled lab automation only, CX8_SSH_STRICT_HOST_KEY_CHECKING="accept-new" can be set in 00_env.local.

If only flint and mlxconfig are missing, continue. If any other command is missing, stop and contact NVIDIA support — the SOT software image is missing a required tool.

flint and mlxconfig are optional for this basic setup. They are provided by NVIDIA MFT (Mellanox Firmware Tools), commonly through mft or mstflint packages depending on the image, and are used only for CX8 firmware/config visibility. Missing MFT tools do not block the two-rail network setup or validation.

Copy setup scripts to both stations

Copy the reusable setup helpers from your local checkout to both DGX Stations. The script streams helper files through one SSH session per DUT, so each DUT should require only one OS password prompt. The script normalizes helper scripts to Linux LF line endings and uses a clean ustar stream so macOS extended attributes are not sent to the DUT.

./02_push_assets.sh

Run Step 1 before Step 2 so each DUT host key is confirmed before this tar-over-SSH copy step.

This copies the generic setup scripts to:

${CX8_REMOTE_BASE}/assets/

Later steps run those copied scripts from the same remote directory on each system, so both stations execute the same logic.

Run prerequisite and current-state checks

Collect the current CX8/GPU/RDMA state before changing the rail configuration. This step confirms CX8 devices exist, maps mlx5_0 and mlx5_1 to Linux network interfaces, shows link state, checks RoCE GIDs, checks firmware/tool visibility, and records recent CX/RDMA/GPU/PCIe error messages.

./03_prereq_check.sh

The console output is intentionally short. The script saves the full raw log locally and prints a summary for each DUT: required missing commands, optional missing commands, CX8 rail mapping, port state, optional nvidia_peermem state, and RDMA-topology visibility. The terminal summary colors PASS green and FAIL red; the saved summary.md remains plain text. The default check is non-privileged, so it should ask only for the OS SSH password once per DUT.

Expected before cabling:

• mlx5_0 and mlx5_1 exist
• Both ports report Ethernet link layer
• Required tools are present, or missing tools are clearly listed

Expected after cabling:

• Both CX8 ports are active
• Each rail can eventually train at 400 Gb/s

Connect QSFP cables and check link presence

Physically connect the two CX8 ports as two matching rails. Rail 0 stays on QSFP0-to-QSFP0 and rail 1 stays on QSFP1-to-QSFP1. For full-speed validation, use two 400G-capable QSFP cables. Keeping the rails symmetric makes route, RDMA, and NCCL debugging much easier.

Station A QSFP0  <---->  Station B QSFP0
Station A QSFP1  <---->  Station B QSFP1

Do not crisscross the rails.

After connecting both cables, run the software-visible link check:

./04_check_cable_presence.sh

This does not configure IP addresses. It checks whether the OS can see each CX8 rail as physically linked by reading ibv_devinfo, ethtool, carrier, and operstate. It also checks whether each linked rail reports the expected 400G speed for this setup and reports recent CX8 port-module events such as Cable error or High Temperature.

A green PASS means both rails show link at 400G on both DUTs. If Step 4 reports link down, Cable error, or High Temperature, stop the software setup path. Let the QSFP cable/module cool, confirm CX8/QSFP airflow, reseat both cable ends on the failed rail, and rerun Step 4.

If Step 4 reports speed_ok=no, Speed: 200000Mb/s, or 200G, the cable is linked but degraded for this 400G-per-rail / 800G-total setup. Treat that as a cable capability, cable orientation, port, or negotiation issue. You may continue through Steps 5–8 for limited-speed functional validation, but do not claim the 800G setup passed until Step 4 reports 400G on both rails.

Configure rail IP and MTU

Bring up the two CX8 Linux network interfaces as independent point-to-point RoCE rails. The script maps mlx5_0 and mlx5_1 to their netdevs, sets MTU 9000, brings the interfaces up, and assigns one /24 subnet per rail.

Recommended: use the temporary runtime setup for normal bring-up and validation. This changes the CX8 rail IPs/MTU in the current running OS only. If the system reboots, those settings disappear.

./05_configure_rails_runtime.sh

You can rerun this command safely. If Step 7 later reports that a rail is missing its configured /24 address, rerun Step 5 and then rerun Step 7.

This step runs privileged commands on each DUT. Expect up to two prompts per DUT: one for OS SSH login and one for remote sudo. With SSH keys and passwordless sudo configured, those prompts can be reduced, but this playbook does not require storing passwords.

This assigns:

To use a different private address range, set these in 00_env.local before running Step 5:

STATION_A_RAIL0_CIDR="192.168.100.1/24"
STATION_B_RAIL0_CIDR="192.168.100.2/24"
STATION_A_RAIL1_CIDR="192.168.101.1/24"
STATION_B_RAIL1_CIDR="192.168.101.2/24"

Only the CIDR values need to change. The scripts pass the configured values to the remote DUT helpers during Steps 5, 7, and 8.

Use --persist only when this pair should keep the same rail IP/MTU settings after reboot:

./05_configure_rails_runtime.sh --persist

--persist is not required for the normal temporary bring-up flow and should not be treated as a bug workaround. It is the right choice when the validation flow intentionally crosses a reboot, when a longer GDR/Data Direct validation run should keep the rail settings stable, or when this pair is expected to keep the same private CX8 rail addresses for repeated performance runs.

To remove persistent netplan configuration later:

source ./00_env.sh
ssh "${remote_a}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'
ssh "${remote_b}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'

Configure RoCEv2 and GPUDirect runtime settings

Apply the runtime settings needed for RoCE traffic and GPU RDMA validation. The script configures DSCP/ToS handling where the installed tools support it and verifies whether the installed ib_write_bw supports the CUDA DMA-BUF/Data Direct GPUDirect path used by Step 8 --gdr.

nvidia_peermem is optional on the default path. Some DGX Station OS images do not load it, and that is not by itself a setup failure. If you specifically need to test the legacy/module-based peermem path, set CX8_TRY_NVIDIA_PEERMEM=1 in 00_env.local before running this step.

./06_configure_roce_gdr_runtime.sh

This step also runs privileged commands on each DUT, so expect SSH and remote sudo prompts.

If the script prints a yellow WARN for cma_roce_tos, continue to Step 7 for basic RoCE validation. If CX8_TRY_NVIDIA_PEERMEM=1 was set and nvidia_peermem fails with Invalid argument, that means only the peermem-based GPUDirect path is unavailable; it does not block basic rail/RoCE validation or the CUDA DMA-BUF/Data Direct GPUDirect path used by Step 8 --gdr.

If Step 6 says ib_write_bw does not advertise --use_cuda_dmabuf and --use_data_direct, the installed perftest package is not ready for Step 8 --gdr. Continue with Step 7 and the non---gdr host-memory Step 8 test first. If GPU-memory RDMA validation is required, run optional Step 10 to install or build a GDR-capable perftest, then rerun Step 6.

When Step 10 installs ib_write_bw under /usr/local, Step 6 checks that prefix before the OS default path. This avoids a false warning from sudo using an older /usr/bin/ib_write_bw.

Validate basic setup

Prove that the two-rail fabric is usable before running NCCL, Ray, vLLM, or application-level tests. This step checks each rail independently so a single-rail cable, MTU, route, or RDMA issue is caught before higher-level software hides the root cause.

./07_validate_setup.sh

This validates:

• Expected IP and MTU on both rails
• Route lookup uses the correct rail
• Jumbo ping succeeds on both rails
• Both ports report 400 G where the driver exposes speed
• RDMA verbs report Ethernet link layer and PORT_ACTIVE
• optional nvidia_peermem status is reported
• RDMA error counters are visible

ibv_devinfo may not report an active_speed field on this Ethernet/RoCE path. In that case, Step 7 uses ethtool 400G link speed as the speed authority and keeps validating the rail.

Only continue to Step 8 when Step 7 prints PASS: basic CX8 two-station setup validated for both DUTs. If Step 7 prints any red ERROR or FAIL, do not run Step 8; fix the specific failed rail first. If Step 7 prints a yellow warning that one or more rails are below 400G, Step 8 may be used for limited-speed functional validation, but not as an 800G setup pass.

If Step 7 reports Link detected: no, Speed: Unknown, or link is not detected, rerun Step 4 and inspect the reported cable/port before retrying Step 7:

./04_check_cable_presence.sh

If Step 7 reports Link detected: no (Overheat), High Temperature, or Cable error, stop software setup. This is a QSFP/CX8 thermal or cable condition. Let the module cool, verify CX8/QSFP airflow or fan operation, reseat the cable, and swap/replace the failed rail cable if the condition returns.

Optional RDMA bandwidth smoke test

Run a simple RDMA bandwidth test after basic validation passes. The wrapper starts ib_write_bw server-side on Station A, runs the matching client on Station B, then saves both logs locally. Use this to confirm that the fabric is not only linked but also moving RDMA traffic at expected bandwidth.

Prerequisite: Step 7 must pass on both DUTs. Do not use Step 8 to bypass a Step 7 red ERROR or FAIL. If Step 7 passed with yellow speed warnings, Step 8 is allowed only as a limited-speed functional smoke test; expect lower bandwidth and keep the result labeled as degraded, not 800G accepted.

This step opens SSH sessions to both DUTs. On Linux/macOS, the wrapper uses temporary SSH connection reuse during one script run, so expect one password prompt for Station A and one for Station B in the normal case. On Windows Git Bash/MSYS/Cygwin, SSH multiplexing is disabled automatically because OpenSSH control-master sockets can reset there; extra password prompts are expected. Disable multiplexing manually on any host with:

CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail 0

The wrapper waits until the remote server is ready before starting the client. The default wait is 30 seconds; change CX8_PERFTEST_SERVER_READY_TIMEOUT only if the GDR server needs more time on your image.

The non---gdr host-memory test is a functional RDMA smoke test. It uses a simple single-QP configuration and may report less than 400 Gb/s on a 400G rail. Use Step 8 --gdr for GPU-memory/Data Direct performance acceptance.

Rail 0:

./08_run_perftest_pair.sh --rail 0

Rail 1:

./08_run_perftest_pair.sh --rail 1

GPUDirect/Data Direct variant:

./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

The --gdr variant uses CUDA DMA-BUF/Data Direct flags in ib_write_bw. It does not require nvidia_peermem to be loaded on stacks where DMA-BUF/Data Direct is the supported GPUDirect path. If --gdr fails because the perftest binary does not support --use_cuda_dmabuf or --use_data_direct, use the non---gdr host-memory RDMA test first. If GPU-memory RDMA validation is required, run optional Step 10 to install or build a GDR-capable perftest, then rerun Step 6 and Step 8 --gdr. Do not reinstall the OS or driver only because nvidia_peermem is absent; first check whether the CUDA DMA-BUF/Data Direct path passes.

If the DUT has more than one CUDA device, set CX8_GPU_BDF in 00_env.local or pass --gpu-bdf <PCI_BUS_ID> so the GDR smoke test uses the intended GB300 GPU instead of the default CUDA device 0.

Optional collect logs

Archive setup logs from both DUTs and the control host for support or post-run review.

./09_collect_logs.sh

Each script writes logs under {repo_root}/logs/ on the control host. Remote logs are written under ${CX8_REMOTE_BASE}/logs/ on each DUT.

Optional install or upgrade perftest for GPUDirect/Data Direct

Use this only when Step 6 or Step 8 reports that ib_write_bw does not support --use_cuda_dmabuf or --use_data_direct. It first tries the OS package path, then falls back to building linux-rdma/perftest from source with CUDA header support and installs it under /usr/local by default. The fallback source build defaults to pinned upstream tag 26.04.17 rather than a moving branch.

This is not a read-only smoke test. It may install build dependencies or upgrade related OS packages through the DUT package manager. Use --check-only first if you only want to inspect the current ib_write_bw capability.

./10_install_perftest_gdr.sh

After it passes, rerun Step 6 and then rerun Step 8 with --gdr:

./06_configure_roce_gdr_runtime.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

For a dry check that makes no package or source changes:

./10_install_perftest_gdr.sh --check-only

If your validation requires a different perftest ref, set CX8_PERFTEST_REF in 00_env.local or pass --ref <git-ref>.

Optional configure ACS/Data Direct GRUB settings

Use this only when Step 8 --gdr reaches the CUDA DMA-BUF/Data Direct path but fails with memory-registration errors, or when sudo rdma_topo check reports ACS/IOMMU-group failures. This step uses rdma_topo to inspect the current topology and, only with --apply, writes the ACS/Data Direct GRUB configuration needed by GPUDirect/Data Direct.

Step 11 is a selective config_acs= policy generated by rdma_topo; it is not a blanket "disable ACS everywhere" change. See the ACS/Data Direct appendix for an example of the default-vs-expected values this step fixes.

Run Step 11 from the separate control host. If a DUT is used as the control host, rebooting that DUT will interrupt the playbook and can leave the two systems in different states.

First inspect without changing the system:

./11_configure_acs_grub.sh

If the output shows ACS/Data Direct failures and you need GPUDirect/Data Direct performance validation, apply the GRUB configuration:

./11_configure_acs_grub.sh --apply

Before rebooting, confirm both DUTs are idle, no shared workload is active, and the workload owner has approved the downtime. Reboot both DUTs after --apply, then rerun:

./05_configure_rails_runtime.sh --persist
./06_configure_roce_gdr_runtime.sh
./07_validate_setup.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

Step 3 is optional after reboot unless the OS/packages changed. Step 4 is optional unless the cable/link state is uncertain. Step 5 uses --persist here because this advanced GDR path intentionally crosses a reboot and often continues into longer performance validation, so the rail IP/MTU configuration should remain stable without rerunning Step 5 manually.

Optional cleanup

Use this when you want to return the two DUTs to a clean runtime state before rerunning setup or handing the systems back. Cleanup does not reinstall the OS, update firmware, remove drivers, or reboot the DUTs.

Default cleanup removes only the configured rail IP addresses and resets both CX8 rail MTUs to 1500:

./99_cleanup_runtime.sh

If Step 5 was run with --persist, also remove the generated netplan file:

./99_cleanup_runtime.sh --remove-persist

To also bring the two CX8 rail interfaces down:

./99_cleanup_runtime.sh --down

Options can be combined:

./99_cleanup_runtime.sh --remove-persist --down

After cleanup, rerun from Step 3 or Step 5 depending on how much state you want to re-check. If you changed the rail CIDRs in 00_env.local, cleanup removes those configured CIDRs rather than the default example addresses.

Troubleshooting quick reference

Next steps

Use the validation level that matches the next workload:

• Step 7 pass means the two DGX Stations have basic two-rail RoCE connectivity: correct rail IPs, MTU, route selection, link state, jumbo ping, and RDMA port visibility.
• Step 8 without --gdr pass means host-memory RDMA bandwidth works on each rail. It is not the line-rate acceptance bar for GPU workloads.
• Step 8 with --gdr pass means the CUDA DMA-BUF/Data Direct GPU-memory RDMA path works. Use this bar before accepting the setup for GPU-performance work such as NCCL, vLLM, or other distributed AI workloads.

Full NCCL/application validation still requires container RDMA device exposure, NCCL environment selection, and a distributed launcher configuration.

QSFP cables
  -> CX8 link up at 400G x 2
  -> Linux netdevs with rail IPs and MTU 9000
  -> RoCEv2 / libibverbs / RDMA devices mlx5_0 and mlx5_1
  -> Optional Step 8 --gdr CUDA DMA-BUF/Data Direct validation
  -> NCCL NET/IB transport
  -> PyTorch, vLLM, Ray, MPI, or another distributed application

The key NCCL validation signal is the transport line in the NCCL log:

Good: NCCL log shows NET/IB and mlx5_0 / mlx5_1
Bad:  NCCL log shows only NET/Socket, which means TCP fallback

If Step 8 host-memory RDMA passes but NCCL falls back to NET/Socket, the CX8 fabric is likely usable and the issue is usually in the container, NCCL environment, RDMA device visibility, or launcher configuration. If GPU-memory RDMA is required, validate the supported GPUDirect path explicitly with Step 8 --gdr and NCCL NET/IB logs before accepting the setup for performance work.

Add ACS/Data Direct grub configuration with Step 11 only after the basic two-rail setup is proven and GPUDirect/Data Direct validation requires it.

Appendix: ACS/Data Direct reference

The platform default ACS policy may be conservative for normal PCIe isolation but not suitable for GB300 + CX8 Data Direct GPU-memory RDMA. In that failure mode, rdma_topo check reports ACS bit mismatches and the CX8 Data Direct DMA function and GB300 GPU are in different IOMMU groups. Step 11 uses rdma_topo write-grub-acs to write the selective config_acs= policy expected by this topology.

Example failure before Step 11:

FAIL ACS for grace_rp 0009:00:00.0 0011101 != xx111x0, 0x1d != 0x1c
FAIL ACS for cx_switch 0009:02:00.0 0011101 != xx110x1, 0x1d != 0x19
FAIL ACS for cx_switch 0009:02:02.0 0011101 != xx101x1, 0x1d != 0x15
FAIL Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are not equal

Expected success after Step 11 apply and reboot:

OK ACS for grace_rp 0009:00:00.0 has correct values 0011100 = xx111x0
OK ACS for cx_switch 0009:02:00.0 has correct values 0011001 = xx110x1
OK ACS for cx_switch 0009:02:02.0 has correct values 0010101 = xx101x1
OK Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are equal

Tools and commands reference

If flint and mlxconfig are missing, that is acceptable for the basic rail setup path because this playbook is not changing NIC firmware configuration.