Connect Two Nodes for Distributed Workloads

Get the setup package and create local configuration

Clone the playbook repository on your control host and create the local environment file.

git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/playbook-connect-two-stations/assets
cp 00_env.local.example 00_env.local

Edit 00_env.local before continuing. Set the Node A/B names and OS hosts for your pair. The OS user and rail CIDRs have defaults; change them only if your setup differs.

Passwords are intentionally not stored in this directory. SSH and sudo will prompt when needed. 00_env.local is ignored by git.

If the default private rail subnets conflict with your lab, change the four STATION_*_RAIL*_CIDR values before running ./05_configure_rails_runtime.sh. Keep rail 0 and rail 1 on different subnets.

Overall flow

Run the probe, asset-copy, prerequisite, cable, rail, RoCE, validation, and host-memory perftest actions in the order shown as the core bring-up path. That proves both rails are cabled, configured, routable, jumbo-ping clean, and RDMA-capable with host memory.

GPU-memory RDMA/Data Direct validation has an additional mandatory readiness gate. Before the first ./08_run_perftest_pair.sh --gdr command, confirm that ib_write_bw has both required GDR flags and that ./11_configure_acs_grub.sh passes on both DUTs. If the ACS check does not pass, apply its generated GRUB configuration and reboot before attempting GDR. Script 08 repeats these read-only checks and will not launch a GDR server or client unless both DUTs are ready.

Command style by control host

The DUTs are Linux systems, so the copied dut-assets/*.sh helpers are Bash scripts on the DUTs. The control host can be Windows, macOS, or Linux:

The Python wrapper requires Python 3 and an OpenSSH client in PATH. On Windows, enable or install the Windows OpenSSH client first if ssh is not available.

The Python wrapper uses the same 00_env.local file as the shell scripts. Numeric aliases match the setup scripts:

For unattended lab loops, configure SSH keys and passwordless sudo outside this playbook. The playbook supports that setup but does not create it or store credentials.

Probe OS SSH access

Confirm that both nodes are reachable over OS SSH and that the expected user can run basic shell commands. This step also lists whether the tools needed for CX8 setup are already installed.

./01_probe_access.sh

If SSH reports an unknown host key, verify the DUT identity and answer the OpenSSH prompt. This is intentional: the default SSH policy is StrictHostKeyChecking=ask, not automatic trust-on-first-use. In controlled lab automation only, CX8_SSH_STRICT_HOST_KEY_CHECKING="accept-new" can be set in 00_env.local.

Missing flint or mlxconfig is allowed. Missing nmcli is also allowed when NetworkManager is inactive or unavailable; when NetworkManager is active, nmcli is required. If any other required command is missing, stop and contact NVIDIA support — the software image is missing a required tool.

flint and mlxconfig are optional for this basic setup. They are provided by NVIDIA MFT (Mellanox Firmware Tools), commonly through mft or mstflint packages depending on the image, and are used only for CX8 firmware/config visibility. Missing MFT tools do not block the two-rail network setup or validation.

Copy setup scripts to both nodes

Copy the reusable setup helpers from your local checkout to both nodes. The script streams helper files through one SSH session per DUT, so each DUT should require only one OS password prompt. The script normalizes helper scripts to Linux LF line endings and uses a clean ustar stream so macOS extended attributes are not sent to the DUT.

./02_push_assets.sh

Run ./01_probe_access.sh before ./02_push_assets.sh so each DUT host key is confirmed before the tar-over-SSH copy.

This copies the generic setup scripts to:

${CX8_REMOTE_BASE}/assets/

Later steps run those copied scripts from the same remote directory on each system, so both nodes execute the same logic.

Run prerequisite and current-state checks

Collect the current CX8/GPU/RDMA state before changing the rail configuration. This step confirms CX8 devices exist, maps mlx5_0 and mlx5_1 to Linux network interfaces, shows link state, checks RoCE GIDs, checks firmware/tool visibility, and records recent CX/RDMA/GPU/PCIe error messages.

./03_prereq_check.sh

The console output is intentionally short. The script saves the full raw log locally and prints a summary for each DUT: required missing commands, optional missing commands, CX8 rail mapping, port state, optional nvidia_peermem state, and RDMA-topology visibility. The terminal summary colors PASS green and FAIL red; the saved summary.md remains plain text. The default check is non-privileged, so it should ask only for the OS SSH password once per DUT.

Expected before cabling:

• mlx5_0 and mlx5_1 exist
• Both ports report Ethernet link layer
• Required tools are present, or missing tools are clearly listed

Expected after cabling:

• Both CX8 ports are active
• Each rail can eventually train at 400 Gb/s

Connect QSFP cables and check link presence

Physically connect the two CX8 ports as two matching rails. Rail 0 stays on QSFP0-to-QSFP0 and rail 1 stays on QSFP1-to-QSFP1. For full-speed validation, use two 400G-capable QSFP cables. Keeping the rails symmetric makes route, RDMA, and NCCL debugging much easier.

Node A QSFP0  <---->  Node B QSFP0
Node A QSFP1  <---->  Node B QSFP1

Do not crisscross the rails.

After connecting both cables, run the software-visible link check:

./04_check_cable_presence.sh

This does not configure IP addresses. It checks whether the OS can see each CX8 rail as physically linked by reading ibv_devinfo, ethtool, carrier, and operstate. It also checks whether each linked rail reports the expected 400G speed for this setup and reports recent CX8 port-module events such as Cable error or High Temperature.

A green PASS means both rails show link at 400G on both DUTs. If ./04_check_cable_presence.sh reports link down, Cable error, or High Temperature, stop the software setup path. Let the QSFP cable/module cool, confirm CX8/QSFP airflow, reseat both cable ends on the failed rail, and rerun the cable/link presence check.

If ./04_check_cable_presence.sh reports speed_ok=no, Speed: 200000Mb/s, or 200G, the cable is linked but degraded for this 400G-per-rail / 800G-total setup. Treat that as a cable capability, cable orientation, port, or negotiation issue. You may continue through rail configuration, RoCE setup, basic validation, and perftest for limited-speed functional validation, but do not claim the 800G setup passed until the cable/link presence check reports 400G on both rails.

Configure rail IP and MTU

Bring up the two CX8 Linux network interfaces as independent point-to-point RoCE rails. The script maps mlx5_0 and mlx5_1 to their netdevs, sets MTU 9000, brings the interfaces up, and assigns one /24 subnet per rail. When NetworkManager manages a rail, the script activates an in-memory manual-address profile so an autoconnecting DHCP profile cannot remove the address. On systems where NetworkManager does not manage the rail, it uses the equivalent ip commands.

Recommended: use the temporary runtime setup for normal bring-up and validation. This changes the CX8 rail IPs/MTU in the current running OS only. If the system reboots, those settings disappear.

./05_configure_rails_runtime.sh

You can rerun this command safely. The temporary NetworkManager profiles are not written to disk and disappear when NetworkManager restarts or the system reboots. If ./07_validate_setup.sh later reports that a rail is missing its configured /24 address after either event, rerun ./05_configure_rails_runtime.sh and then rerun ./07_validate_setup.sh.

This step runs privileged commands on each DUT. Expect up to two prompts per DUT: one for OS SSH login and one for remote sudo. With SSH keys and passwordless sudo configured, those prompts can be reduced, but this playbook does not require storing passwords.

This assigns:

To use a different private address range, set these in 00_env.local before running ./05_configure_rails_runtime.sh:

STATION_A_RAIL0_CIDR="192.168.100.1/24"
STATION_B_RAIL0_CIDR="192.168.100.2/24"
STATION_A_RAIL1_CIDR="192.168.101.1/24"
STATION_B_RAIL1_CIDR="192.168.101.2/24"

Only the CIDR values need to change. The scripts pass the configured values to the remote DUT helpers during rail configuration, basic validation, and RDMA bandwidth testing.

Use --persist only when this pair should keep the same rail IP/MTU settings after reboot:

./05_configure_rails_runtime.sh --persist

--persist is not required for the normal temporary bring-up flow and should not be treated as a bug workaround. It is the right choice when a longer GDR/Data Direct validation run should keep the rail settings across later reboots, or when this pair is expected to keep the same private CX8 rail addresses for repeated performance runs.

To remove persistent netplan configuration later:

source ./00_env.sh
ssh "${remote_a}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'
ssh "${remote_b}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'

Configure RoCEv2 and GPUDirect runtime settings

Apply the runtime settings needed for RoCE traffic and GPU RDMA validation. The script configures DSCP/ToS handling where the installed tools support it and verifies whether the installed ib_write_bw supports the CUDA DMA-BUF/Data Direct GPUDirect path used by the ./08_run_perftest_pair.sh GDR test.

nvidia_peermem is optional on the default path. Some OS images do not load it, and that is not by itself a setup failure. If you specifically need to test the legacy/module-based peermem path, set CX8_TRY_NVIDIA_PEERMEM=1 in 00_env.local before running this step.

./06_configure_roce_gdr_runtime.sh

This step also runs privileged commands on each DUT, so expect SSH and remote sudo prompts.

If the script prints a yellow WARN for cma_roce_tos, continue with ./07_validate_setup.sh for basic RoCE validation. If CX8_TRY_NVIDIA_PEERMEM=1 was set and nvidia_peermem fails with Invalid argument, that means only the peermem-based GPUDirect path is unavailable; it does not block basic rail/RoCE validation. CUDA DMA-BUF/Data Direct readiness still requires the Script 06 perftest-flag pass and the Script 11 ACS/Data Direct pass before any GDR test.

If ./06_configure_roce_gdr_runtime.sh says ib_write_bw does not advertise --use_cuda_dmabuf and --use_data_direct, the installed perftest package is not ready for the GDR test. Continue with ./07_validate_setup.sh and the non---gdr host-memory ./08_run_perftest_pair.sh test first. If GPU-memory RDMA validation is required, run ./10_install_perftest_gdr.sh to install or build a GDR-capable perftest, then rerun ./06_configure_roce_gdr_runtime.sh.

When ./10_install_perftest_gdr.sh installs ib_write_bw under /usr/local, ./06_configure_roce_gdr_runtime.sh checks that prefix before the OS default path. This avoids a false warning from sudo using an older /usr/bin/ib_write_bw.

Validate basic setup

Prove that the two-rail fabric is usable before running NCCL, Ray, vLLM, or application-level tests. This step checks each rail independently so a single-rail cable, MTU, route, or RDMA issue is caught before higher-level software hides the root cause.

./07_validate_setup.sh

This validates:

• Expected IP and MTU on both rails
• Route lookup uses the correct rail
• Jumbo ping succeeds on both rails
• Both ports report 400 G where the driver exposes speed
• RDMA verbs report Ethernet link layer and PORT_ACTIVE
• optional nvidia_peermem status is reported
• RDMA error counters are visible

ibv_devinfo may not report an active_speed field on this Ethernet/RoCE path. In that case, ./07_validate_setup.sh uses ethtool 400G link speed as the speed authority and keeps validating the rail.

Only continue to ./08_run_perftest_pair.sh when ./07_validate_setup.sh prints PASS: basic CX8 two-station setup validated for both DUTs. If basic validation prints any red ERROR or FAIL, do not run perftest; fix the specific failed rail first. If basic validation prints a yellow warning that one or more rails are below 400G, perftest may be used for limited-speed functional validation, but not as an 800G setup pass.

If ./07_validate_setup.sh reports Link detected: no, Speed: Unknown, or link is not detected, rerun the cable/link presence check and inspect the reported cable/port before retrying basic validation:

./04_check_cable_presence.sh

If ./07_validate_setup.sh reports Link detected: no (Overheat), High Temperature, or Cable error, stop software setup. This is a QSFP/CX8 thermal or cable condition. Let the module cool, verify CX8/QSFP airflow or fan operation, reseat the cable, and swap/replace the failed rail cable if the condition returns.

Optional RDMA bandwidth smoke test

Run a simple RDMA bandwidth test after basic validation passes. The wrapper starts ib_write_bw server-side on Node A, runs the matching client on Node B, then saves both logs locally. Use this to confirm that the fabric is not only linked but also moving RDMA traffic at expected bandwidth.

Prerequisite: ./07_validate_setup.sh must pass on both DUTs. Do not use ./08_run_perftest_pair.sh to bypass a basic-validation red ERROR or FAIL. If basic validation passed with yellow speed warnings, perftest is allowed only as a limited-speed functional smoke test; expect lower bandwidth and keep the result labeled as degraded, not 800G accepted.

This step opens SSH sessions to both DUTs. On Linux/macOS, the wrapper uses temporary SSH connection reuse during one script run, so expect one password prompt for Node A and one for Node B in the normal case. On Windows Git Bash/MSYS/Cygwin, SSH multiplexing is disabled automatically because OpenSSH control-master sockets can reset there; extra password prompts are expected. SSH multiplexing affects only SSH connection reuse and does not change RDMA, Data Direct, or perftest behavior. See the troubleshooting guide only if the control host reports an SSH control-socket error.

The wrapper waits until the remote server is ready before starting the client. The default wait is 30 seconds; change CX8_PERFTEST_SERVER_READY_TIMEOUT only if the GDR server needs more time on your image, and set it to a positive integer. If readiness fails, the wrapper terminates the detached server before returning an error so a retry does not inherit a stale process.

The non---gdr host-memory test is a functional RDMA smoke test. It uses a simple single-QP configuration and may report less than 400 Gb/s on a 400G rail.

Rail 0:

./08_run_perftest_pair.sh --rail 0

Rail 1:

./08_run_perftest_pair.sh --rail 1

Required preparation before the GDR variant

Do not add --gdr to Script 08 until all of the following are true on both DUTs:

• Script 07 passes the basic setup.
• Script 06 prints PASS: ib_write_bw supports the CUDA DMA-BUF and Data Direct flags required by the GDR bandwidth-test action. If it does not, run ./10_install_perftest_gdr.sh, then rerun Script 06.
• ./11_configure_acs_grub.sh passes its ACS/Data Direct inspection on both DUTs.

Run the required ACS/Data Direct inspection before the first GDR test:

./11_configure_acs_grub.sh

Only continue when it prints PASS: both DUTs satisfy the ACS/Data Direct prerequisite for GDR. If either DUT is not ready, do not start a GDR test. Apply the generated configuration:

./11_configure_acs_grub.sh --apply

Script 11 does not reboot either DUT. After it completes successfully, confirm both DUTs are idle and obtain workload-owner approval before rebooting both systems through the approved lab procedure. Then restore and revalidate the runtime state, including a new ACS inspection:

./05_configure_rails_runtime.sh
./06_configure_roce_gdr_runtime.sh
./07_validate_setup.sh
./11_configure_acs_grub.sh

The temporary Script 05 configuration is sufficient for these tests. Use --persist only when the pair must retain the private rail configuration across later reboots or repeated performance runs.

Script 08 repeats the read-only perftest-capability and ACS checks on both DUTs whenever --gdr is specified. It starts neither the GDR server nor client unless every prerequisite passes. After the readiness gate passes, run the GPUDirect/Data Direct variant:

The ACS check requires privileged PCIe configuration readback, so the GDR readiness gate may prompt once for remote sudo on each DUT before it starts either endpoint.

./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

The --gdr variant uses CUDA DMA-BUF/Data Direct flags in ib_write_bw. It does not require nvidia_peermem to be loaded on stacks where DMA-BUF/Data Direct is the supported GPUDirect path. Do not reinstall the OS or driver only because nvidia_peermem is absent.

If the DUT has more than one CUDA device, set CX8_GPU_BDF in 00_env.local or pass --gpu-bdf <PCI_BUS_ID> so the GDR smoke test uses the intended GPU instead of the default CUDA device 0.

Optional collect logs

Archive setup logs from both DUTs and the control host for support or post-run review.

./09_collect_logs.sh

By default, each script writes logs under ./logs/ in the current playbook assets directory on the control host. Remote logs are written under ${CX8_REMOTE_BASE}/logs/ on each DUT.

Optional install or upgrade perftest for GPUDirect/Data Direct

Use this before the first GDR test when Script 06 does not confirm that ib_write_bw supports both --use_cuda_dmabuf and --use_data_direct. Do not attempt GDR with an unqualified perftest binary. This action first tries the OS package path, then falls back to building linux-rdma/perftest from source with CUDA header support and installs it under /usr/local by default. The fallback source build defaults to pinned upstream tag 26.04.17 rather than a moving branch.

This is not a read-only smoke test. It may install build dependencies or upgrade related OS packages through the DUT package manager. Use --check-only first if you only want to inspect the current ib_write_bw capability.

./10_install_perftest_gdr.sh

After it passes, rerun Script 06 and complete the Script 11 ACS/Data Direct readiness gate before either GDR rail test:

./06_configure_roce_gdr_runtime.sh
./11_configure_acs_grub.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

For a dry check that makes no package or source changes:

./10_install_perftest_gdr.sh --check-only

If your validation requires a different perftest ref, set CX8_PERFTEST_REF in 00_env.local or pass --ref <git-ref>.

Verify and, when required, configure ACS/Data Direct GRUB settings

Run this inspection on both DUTs before the first ./08_run_perftest_pair.sh --gdr test. It uses rdma_topo to verify that the GPU and ConnectX-8 Data Direct function satisfy the required ACS/IOMMU topology. Only the --apply form writes the generated ACS/Data Direct GRUB configuration.

The ACS/Data Direct action applies a selective config_acs= policy generated by rdma_topo; it is not a blanket "disable ACS everywhere" change. See the ACS/Data Direct appendix for an example of the default-vs-expected values it fixes.

Run ./11_configure_acs_grub.sh from the separate control host. If a DUT is used as the control host, rebooting that DUT will interrupt the playbook and can leave the two systems in different states.

First inspect without changing either system:

./11_configure_acs_grub.sh

Only proceed to GDR when this inspection passes on both DUTs. If either output shows NOT READY or an ACS/IOMMU failure and you need GPUDirect/Data Direct validation, apply the GRUB configuration before attempting GDR:

./11_configure_acs_grub.sh --apply

Script 11 does not reboot either DUT. Before rebooting, confirm both DUTs are idle, no shared workload is active, and the workload owner has approved the downtime. Reboot both DUTs after --apply, then rerun:

During --apply, a yellow CHANGE REQUIRED line describes the pre-change ACS/IOMMU state that the command is correcting. A red FAIL or ERROR still means the apply did not complete and must be investigated before rebooting.

./05_configure_rails_runtime.sh
./06_configure_roce_gdr_runtime.sh
./07_validate_setup.sh
./11_configure_acs_grub.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr

The prerequisite/current-state check is optional after reboot unless the OS/packages changed. The cable/link presence check is optional unless the cable/link state is uncertain. Use ./05_configure_rails_runtime.sh --persist instead only when the pair must keep the same private rail settings across later reboots or repeated performance runs.

Optional cleanup

Use this when you want to return the two DUTs to a clean runtime state before rerunning setup or handing the systems back. Cleanup does not reinstall the OS, update firmware, remove drivers, or reboot the DUTs.

Default cleanup removes the configured rail IP addresses, deletes any in-memory NetworkManager profiles created by the playbook, and resets both CX8 rail MTUs to 1500:

./99_cleanup_runtime.sh

If ./05_configure_rails_runtime.sh was run with --persist, also remove the generated netplan file:

./99_cleanup_runtime.sh --remove-persist

To also bring the two CX8 rail interfaces down:

./99_cleanup_runtime.sh --down

Options can be combined:

./99_cleanup_runtime.sh --remove-persist --down

After cleanup, rerun ./03_prereq_check.sh for a full state re-check, or resume with ./05_configure_rails_runtime.sh if the prerequisites and cable/link state are still known-good. If you changed the rail CIDRs in 00_env.local, cleanup removes those configured CIDRs rather than the default example addresses.

Troubleshooting quick reference

Next steps

Use the validation level that matches the next workload:

• A ./07_validate_setup.sh pass means the two nodes have basic two-rail RoCE connectivity: correct rail IPs, MTU, route selection, link state, jumbo ping, and RDMA port visibility.
• A ./08_run_perftest_pair.sh pass without --gdr means host-memory RDMA bandwidth works on each rail. It is not the line-rate acceptance bar for GPU workloads.
• A ./08_run_perftest_pair.sh pass with --gdr means the CUDA DMA-BUF/Data Direct GPU-memory RDMA path works. Use this bar before accepting the setup for GPU-performance work such as NCCL, vLLM, or other distributed AI workloads.

Full NCCL/application validation still requires container RDMA device exposure, NCCL environment selection, and a distributed launcher configuration.

QSFP cables
  -> CX8 link up at 400G x 2
  -> Linux netdevs with rail IPs and MTU 9000
  -> RoCEv2 / libibverbs / RDMA devices mlx5_0 and mlx5_1
  -> Optional ./08_run_perftest_pair.sh --gdr CUDA DMA-BUF/Data Direct validation
  -> NCCL NET/IB transport
  -> PyTorch, vLLM, Ray, MPI, or another distributed application

The key NCCL validation signal is the transport line in the NCCL log:

Good: NCCL log shows NET/IB and mlx5_0 / mlx5_1
Bad:  NCCL log shows only NET/Socket, which means TCP fallback

If the ./08_run_perftest_pair.sh host-memory RDMA test passes but NCCL falls back to NET/Socket, the CX8 fabric is likely usable and the issue is usually in the container, NCCL environment, RDMA device visibility, or launcher configuration. If GPU-memory RDMA is required, validate the supported GPUDirect path explicitly with the GDR variant and NCCL NET/IB logs before accepting the setup for performance work.

Inspect ACS/Data Direct readiness with ./11_configure_acs_grub.sh before the first GDR test. Use --apply only after the basic two-rail setup is proven, the inspection is not ready, and GPUDirect/Data Direct validation requires the change.

Appendix: ACS/Data Direct reference

The platform default ACS policy may be conservative for normal PCIe isolation but not suitable for GPU + CX8 Data Direct GPU-memory RDMA. In that failure mode, rdma_topo check reports ACS bit mismatches and the CX8 Data Direct DMA function and GPU are in different IOMMU groups. ./11_configure_acs_grub.sh --apply uses rdma_topo write-grub-acs to write the selective config_acs= policy expected by this topology.

Example failure before applying the ACS/Data Direct GRUB configuration:

FAIL ACS for grace_rp 0009:00:00.0 0011101 != xx111x0, 0x1d != 0x1c
FAIL ACS for cx_switch 0009:02:00.0 0011101 != xx110x1, 0x1d != 0x19
FAIL ACS for cx_switch 0009:02:02.0 0011101 != xx101x1, 0x1d != 0x15
FAIL Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are not equal

Expected success after applying the ACS/Data Direct GRUB configuration and rebooting:

OK ACS for grace_rp 0009:00:00.0 has correct values 0011100 = xx111x0
OK ACS for cx_switch 0009:02:00.0 has correct values 0011001 = xx110x1
OK ACS for cx_switch 0009:02:02.0 has correct values 0010101 = xx101x1
OK Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are equal

Tools and commands reference

If flint and mlxconfig are missing, that is acceptable for the basic rail setup path because this playbook is not changing NIC firmware configuration.