---
title: "Connect Two DGX Stations for Distributed Workloads — Instructions"
canonical: "https://build.nvidia.com/station/connect-two-stations/instructions.md"
---

> [!TIP]
> Run all commands from a **control host** — a separate Linux, macOS, or Windows system with `ssh` and `tar` access to both DGX Stations. Do not run the control-host scripts directly on the DUTs.
> Step 11 may reboot both DUTs; do not run Step 11 from a DUT that is being configured.

> [!NOTE]
> Before starting, the only file you should create or edit is `00_env.local`. Do not edit the tracked scripts for lab-specific hostnames, IP addresses, or user overrides.

# Step 0. Get the setup package and create local configuration

Clone the playbook repository on your control host and create the local environment file.

```bash
git clone https://github.com/NVIDIA/dgx-spark-playbooks dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/station-connect-two-stations/assets
cp 00_env.local.example 00_env.local
```

Edit `00_env.local` before continuing. Set the Station A/B names and OS hosts for your pair. The OS user and rail CIDRs have defaults; change them only if your setup differs.

| Variable | Meaning | Edit needed? | Default/example value |
|---|---|---:|---|
| `CX8_A_NAME` | Friendly name for Station A | Yes | `station-a` |
| `CX8_A_HOST` | Station A OS hostname or IP | Yes | `<station-a-os-host-or-ip>` |
| `CX8_B_NAME` | Friendly name for Station B | Yes | `station-b` |
| `CX8_B_HOST` | Station B OS hostname or IP | Yes | `<station-b-os-host-or-ip>` |
| `CX8_OS_USER` | OS SSH user used for both DUTs | Only if not `nvidia` | `nvidia` |
| `CX8_SSH_STRICT_HOST_KEY_CHECKING` | SSH host-key policy | Only for controlled lab automation | `ask` |
| `CX8_GPU_BDF` | GPU PCI bus ID for Step 8 `--gdr` when the DUT has more than one CUDA device | Only if CUDA device 0 is not the intended GPU | Example: `0009:06:00.0` |
| `CX8_TRY_NVIDIA_PEERMEM` | Opt-in legacy peermem load attempt in Step 6 | Only if you specifically need to test `nvidia_peermem` | `0` |
| `CX8_PERFTEST_REPO` | Perftest git repository used by optional Step 10 source build | Only if a mirror or fork is required | `https://github.com/linux-rdma/perftest.git` |
| `CX8_PERFTEST_REF` | Perftest git ref used by optional Step 10 source build | Only if a different tested ref is required | `26.04.17` |
| `CX8_PERFTEST_PREFIX` | Install prefix used by optional Step 10 source build | Only if `/usr/local` is not desired | `/usr/local` |
| `CX8_PERFTEST_SERVER_READY_TIMEOUT` | Step 8 wait time for the remote `ib_write_bw` server to become ready | Only if GPU-memory server startup is unusually slow | `30` |
| `STATION_A_RAIL0_CIDR` / `STATION_B_RAIL0_CIDR` | Rail 0 private IPs assigned by Step 5 | Only if subnet conflicts | `192.168.100.1/24` / `192.168.100.2/24` |
| `STATION_A_RAIL1_CIDR` / `STATION_B_RAIL1_CIDR` | Rail 1 private IPs assigned by Step 5 | Only if subnet conflicts | `192.168.101.1/24` / `192.168.101.2/24` |

Passwords are intentionally not stored in this directory. SSH and `sudo` will prompt when needed. `00_env.local` is ignored by git.

If the default private rail subnets conflict with your lab, change the four `STATION_*_RAIL*_CIDR` values before Step 5. Keep rail 0 and rail 1 on different subnets.

## Overall flow

Use Steps 1-8 without `--gdr` as the core bring-up path. That proves both rails are cabled, configured, routable, jumbo-ping clean, and RDMA-capable with host memory.

Use Steps 8 `--gdr`, 10, and 11 only when you need GPU-memory RDMA/Data Direct performance validation. Those steps are intentionally separate because they may require package installation, ACS/Data Direct GRUB configuration, and reboot.

## Command style by control host

The DUTs are Linux systems, so the copied `dut-assets/*.sh` helpers are Bash scripts on the DUTs. The control host can be Windows, macOS, or Linux:

| Control host | Recommended command style |
|---|---|
| Linux or macOS with Bash | Use the numbered shell scripts, for example `./01_probe_access.sh` |
| Windows PowerShell or any host without Bash | Use the cross-platform Python wrapper, for example `py -3 .\cx8_setup.py 01` |
| Linux/macOS with Python preferred | Use `python3 ./cx8_setup.py 01` |

The Python wrapper requires Python 3 and an OpenSSH client in `PATH`. On Windows, enable or install the Windows OpenSSH client first if `ssh` is not available.

The Python wrapper uses the same `00_env.local` file as the shell scripts. Numeric aliases match the setup scripts:

| Setup script | Shell command | Python command |
|---|---|---|
| 1 | `./01_probe_access.sh` | `python3 ./cx8_setup.py 01` |
| 2 | `./02_push_assets.sh` | `python3 ./cx8_setup.py 02` |
| 3 | `./03_prereq_check.sh` | `python3 ./cx8_setup.py 03` |
| 4 | `./04_check_cable_presence.sh` | `python3 ./cx8_setup.py 04` |
| 5 | `./05_configure_rails_runtime.sh` | `python3 ./cx8_setup.py 05` |
| 5 persistent | `./05_configure_rails_runtime.sh --persist` | `python3 ./cx8_setup.py 05 --persist` |
| 6 | `./06_configure_roce_gdr_runtime.sh` | `python3 ./cx8_setup.py 06` |
| 7 | `./07_validate_setup.sh` | `python3 ./cx8_setup.py 07` |
| 8 rail 0 | `./08_run_perftest_pair.sh --rail 0` | `python3 ./cx8_setup.py 08 --rail 0` |
| 8 rail 1 | `./08_run_perftest_pair.sh --rail 1` | `python3 ./cx8_setup.py 08 --rail 1` |
| 9 | `./09_collect_logs.sh` | `python3 ./cx8_setup.py 09` |
| 10 | `./10_install_perftest_gdr.sh` | `python3 ./cx8_setup.py 10` |
| 11 | `./11_configure_acs_grub.sh` | `python3 ./cx8_setup.py 11` |
| 11 apply | `./11_configure_acs_grub.sh --apply` | `python3 ./cx8_setup.py 11 --apply` |
| 99 | `./99_cleanup_runtime.sh` | `python3 ./cx8_setup.py 99` |

For unattended lab loops, configure SSH keys and passwordless `sudo` outside this playbook. The playbook supports that setup but does not create it or store credentials.

# Step 1. Probe OS SSH access

Confirm that both DGX Stations are reachable over OS SSH and that the expected user can run basic shell commands. This step also lists whether the tools needed for CX8 setup are already installed.

```bash
./01_probe_access.sh
```

If SSH reports an unknown host key, verify the DUT identity and answer the OpenSSH prompt. This is intentional: the default SSH policy is `StrictHostKeyChecking=ask`, not automatic trust-on-first-use. In controlled lab automation only, `CX8_SSH_STRICT_HOST_KEY_CHECKING="accept-new"` can be set in `00_env.local`.

If only `flint` and `mlxconfig` are missing, continue. If any other command is missing, stop and contact NVIDIA support — the SOT software image is missing a required tool.

`flint` and `mlxconfig` are optional for this basic setup. They are provided by NVIDIA MFT (Mellanox Firmware Tools), commonly through `mft` or `mstflint` packages depending on the image, and are used only for CX8 firmware/config visibility. Missing MFT tools do not block the two-rail network setup or validation.

# Step 2. Copy setup scripts to both stations

Copy the reusable setup helpers from your local checkout to both DGX Stations. The script streams helper files through one SSH session per DUT, so each DUT should require only one OS password prompt. The script normalizes helper scripts to Linux LF line endings and uses a clean `ustar` stream so macOS extended attributes are not sent to the DUT.

```bash
./02_push_assets.sh
```

Run Step 1 before Step 2 so each DUT host key is confirmed before this tar-over-SSH copy step.

This copies the generic setup scripts to:

```text
${CX8_REMOTE_BASE}/assets/
```

Later steps run those copied scripts from the same remote directory on each system, so both stations execute the same logic.

# Step 3. Run prerequisite and current-state checks

Collect the current CX8/GPU/RDMA state before changing the rail configuration. This step confirms CX8 devices exist, maps `mlx5_0` and `mlx5_1` to Linux network interfaces, shows link state, checks RoCE GIDs, checks firmware/tool visibility, and records recent CX/RDMA/GPU/PCIe error messages.

```bash
./03_prereq_check.sh
```

The console output is intentionally short. The script saves the full raw log locally and prints a summary for each DUT: required missing commands, optional missing commands, CX8 rail mapping, port state, optional `nvidia_peermem` state, and RDMA-topology visibility. The terminal summary colors `PASS` green and `FAIL` red; the saved `summary.md` remains plain text. The default check is non-privileged, so it should ask only for the OS SSH password once per DUT.

Expected before cabling:

- `mlx5_0` and `mlx5_1` exist
- Both ports report Ethernet link layer
- Required tools are present, or missing tools are clearly listed

Expected after cabling:

- Both CX8 ports are active
- Each rail can eventually train at 400 Gb/s

# Step 4. Connect QSFP cables and check link presence

Physically connect the two CX8 ports as two matching rails. Rail 0 stays on QSFP0-to-QSFP0 and rail 1 stays on QSFP1-to-QSFP1. For full-speed validation, use two 400G-capable QSFP cables. Keeping the rails symmetric makes route, RDMA, and NCCL debugging much easier.

```text
Station A QSFP0  <---->  Station B QSFP0
Station A QSFP1  <---->  Station B QSFP1
```

Do not crisscross the rails.

After connecting both cables, run the software-visible link check:

```bash
./04_check_cable_presence.sh
```

This does not configure IP addresses. It checks whether the OS can see each CX8 rail as physically linked by reading `ibv_devinfo`, `ethtool`, carrier, and operstate. It also checks whether each linked rail reports the expected 400G speed for this setup and reports recent CX8 port-module events such as `Cable error` or `High Temperature`.

A green `PASS` means both rails show link at 400G on both DUTs. If Step 4 reports link down, `Cable error`, or `High Temperature`, stop the software setup path. Let the QSFP cable/module cool, confirm CX8/QSFP airflow, reseat both cable ends on the failed rail, and rerun Step 4.

If Step 4 reports `speed_ok=no`, `Speed: 200000Mb/s`, or `200G`, the cable is linked but degraded for this 400G-per-rail / 800G-total setup. Treat that as a cable capability, cable orientation, port, or negotiation issue. You may continue through Steps 5–8 for limited-speed functional validation, but do not claim the 800G setup passed until Step 4 reports 400G on both rails.

# Step 5. Configure rail IP and MTU

Bring up the two CX8 Linux network interfaces as independent point-to-point RoCE rails. The script maps `mlx5_0` and `mlx5_1` to their netdevs, sets MTU 9000, brings the interfaces up, and assigns one `/24` subnet per rail.

Recommended: use the temporary runtime setup for normal bring-up and validation. This changes the CX8 rail IPs/MTU in the current running OS only. If the system reboots, those settings disappear.

```bash
./05_configure_rails_runtime.sh
```

You can rerun this command safely. If `./07_validate_setup.sh` later reports that a rail is missing its configured `/24` address, rerun `./05_configure_rails_runtime.sh` and then rerun `./07_validate_setup.sh`.

This step runs privileged commands on each DUT. Expect up to two prompts per DUT: one for OS SSH login and one for remote `sudo`. With SSH keys and passwordless sudo configured, those prompts can be reduced, but this playbook does not require storing passwords.

> [!NOTE]
> Enter the OS SSH password at the first prompt, for example `nvidia@<dut-ip>'s password:`. If a second `[sudo on ...] password for nvidia:` prompt appears, enter the same DUT OS password again. Password input is hidden, so the terminal will not show typed characters.

This assigns:

| Rail | A address | B address |
|---|---:|---:|
| rail0 / `mlx5_0` | `192.168.100.1/24` | `192.168.100.2/24` |
| rail1 / `mlx5_1` | `192.168.101.1/24` | `192.168.101.2/24` |

To use a different private address range, set these in `00_env.local` before running Step 5:

```bash
STATION_A_RAIL0_CIDR="192.168.100.1/24"
STATION_B_RAIL0_CIDR="192.168.100.2/24"
STATION_A_RAIL1_CIDR="192.168.101.1/24"
STATION_B_RAIL1_CIDR="192.168.101.2/24"
```

Only the CIDR values need to change. The scripts pass the configured values to the remote DUT helpers during Steps 5, 7, and 8.

Use `--persist` only when this pair should keep the same rail IP/MTU settings after reboot:

```bash
./05_configure_rails_runtime.sh --persist
```

`--persist` is not required for the normal temporary bring-up flow and should not be treated as a bug workaround. It is the right choice when the validation flow intentionally crosses a reboot, when a longer GDR/Data Direct validation run should keep the rail settings stable, or when this pair is expected to keep the same private CX8 rail addresses for repeated performance runs.

To remove persistent netplan configuration later:

```bash
source ./00_env.sh
ssh "${remote_a}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'
ssh "${remote_b}" 'sudo rm -f /etc/netplan/60-cx8-fabric.yaml && sudo netplan apply'
```

# Step 6. Configure RoCEv2 and GPUDirect runtime settings

Apply the runtime settings needed for RoCE traffic and GPU RDMA validation. The script configures DSCP/ToS handling where the installed tools support it and verifies whether the installed `ib_write_bw` supports the CUDA DMA-BUF/Data Direct GPUDirect path used by Step 8 `--gdr`.

`nvidia_peermem` is optional on the default path. Some DGX Station OS images do not load it, and that is not by itself a setup failure. If you specifically need to test the legacy/module-based peermem path, set `CX8_TRY_NVIDIA_PEERMEM=1` in `00_env.local` before running this step.

```bash
./06_configure_roce_gdr_runtime.sh
```

This step also runs privileged commands on each DUT, so expect SSH and remote `sudo` prompts.

> [!NOTE]
> Enter the OS SSH password at the first prompt. If a second `[sudo on ...] password for nvidia:` prompt appears, enter the same DUT OS password again. Password input is hidden.

If the script prints a yellow `WARN` for `cma_roce_tos`, continue to Step 7 for basic RoCE validation. If `CX8_TRY_NVIDIA_PEERMEM=1` was set and `nvidia_peermem` fails with `Invalid argument`, that means only the peermem-based GPUDirect path is unavailable; it does not block basic rail/RoCE validation or the CUDA DMA-BUF/Data Direct GPUDirect path used by Step 8 `--gdr`.

If `./06_configure_roce_gdr_runtime.sh` says `ib_write_bw` does not advertise `--use_cuda_dmabuf` and `--use_data_direct`, the installed `perftest` package is not ready for Step 8 `--gdr`. Continue with Step 7 and the non-`--gdr` host-memory Step 8 test first. If GPU-memory RDMA validation is required, run optional Step 10 to install or build a GDR-capable perftest, then rerun `./06_configure_roce_gdr_runtime.sh`.

When Step 10 installs `ib_write_bw` under `/usr/local`, `./06_configure_roce_gdr_runtime.sh` checks that prefix before the OS default path. This avoids a false warning from `sudo` using an older `/usr/bin/ib_write_bw`.

# Step 7. Validate basic setup

Prove that the two-rail fabric is usable before running NCCL, Ray, vLLM, or application-level tests. This step checks each rail independently so a single-rail cable, MTU, route, or RDMA issue is caught before higher-level software hides the root cause.

```bash
./07_validate_setup.sh
```

This validates:

- Expected IP and MTU on both rails
- Route lookup uses the correct rail
- Jumbo ping succeeds on both rails
- Both ports report 400 G where the driver exposes speed
- RDMA verbs report Ethernet link layer and `PORT_ACTIVE`
- optional `nvidia_peermem` status is reported
- RDMA error counters are visible

`ibv_devinfo` may not report an `active_speed` field on this Ethernet/RoCE path. In that case, Step 7 uses `ethtool` 400G link speed as the speed authority and keeps validating the rail.

Only continue to Step 8 when Step 7 prints `PASS: basic CX8 two-station setup validated` for both DUTs. If Step 7 prints any red `ERROR` or `FAIL`, do not run Step 8; fix the specific failed rail first. If Step 7 prints a yellow warning that one or more rails are below 400G, Step 8 may be used for limited-speed functional validation, but not as an 800G setup pass.

If Step 7 reports `Link detected: no`, `Speed: Unknown`, or `link is not detected`, rerun Step 4 and inspect the reported cable/port before retrying Step 7:

```bash
./04_check_cable_presence.sh
```

If Step 7 reports `Link detected: no (Overheat)`, `High Temperature`, or `Cable error`, stop software setup. This is a QSFP/CX8 thermal or cable condition. Let the module cool, verify CX8/QSFP airflow or fan operation, reseat the cable, and swap/replace the failed rail cable if the condition returns.

# Step 8. Optional RDMA bandwidth smoke test

Run a simple RDMA bandwidth test after basic validation passes. The wrapper starts `ib_write_bw` server-side on Station A, runs the matching client on Station B, then saves both logs locally. Use this to confirm that the fabric is not only linked but also moving RDMA traffic at expected bandwidth.

Prerequisite: Step 7 must pass on both DUTs. Do not use Step 8 to bypass a Step 7 red `ERROR` or `FAIL`. If Step 7 passed with yellow speed warnings, Step 8 is allowed only as a limited-speed functional smoke test; expect lower bandwidth and keep the result labeled as degraded, not 800G accepted.

This step opens SSH sessions to both DUTs. On Linux/macOS, the wrapper uses temporary SSH connection reuse during one script run, so expect one password prompt for Station A and one for Station B in the normal case. On Windows Git Bash/MSYS/Cygwin, SSH multiplexing is disabled automatically because OpenSSH control-master sockets can reset there; extra password prompts are expected. SSH multiplexing affects only SSH connection reuse and does not change RDMA, Data Direct, or perftest behavior. See the troubleshooting guide only if the control host reports an SSH control-socket error.

The wrapper waits until the remote server is ready before starting the client. The default wait is 30 seconds; change `CX8_PERFTEST_SERVER_READY_TIMEOUT` only if the GDR server needs more time on your image.

The non-`--gdr` host-memory test is a functional RDMA smoke test. It uses a simple single-QP configuration and may report less than 400 Gb/s on a 400G rail. Use Step 8 `--gdr` for GPU-memory/Data Direct performance acceptance.

Rail 0:

```bash
./08_run_perftest_pair.sh --rail 0
```

Rail 1:

```bash
./08_run_perftest_pair.sh --rail 1
```

GPUDirect/Data Direct variant:

```bash
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr
```

The `--gdr` variant uses CUDA DMA-BUF/Data Direct flags in `ib_write_bw`. It does not require `nvidia_peermem` to be loaded on stacks where DMA-BUF/Data Direct is the supported GPUDirect path. If `--gdr` fails because the perftest binary does not support `--use_cuda_dmabuf` or `--use_data_direct`, use the non-`--gdr` host-memory RDMA test first. If GPU-memory RDMA validation is required, run optional Step 10 to install or build a GDR-capable perftest, then rerun Step 6 and Step 8 `--gdr`. Do not reinstall the OS or driver only because `nvidia_peermem` is absent; first check whether the CUDA DMA-BUF/Data Direct path passes.

If the DUT has more than one CUDA device, set `CX8_GPU_BDF` in `00_env.local` or pass `--gpu-bdf <PCI_BUS_ID>` so the GDR smoke test uses the intended GB300 GPU instead of the default CUDA device 0.

# Step 9. Optional collect logs

Archive setup logs from both DUTs and the control host for support or post-run review.

```bash
./09_collect_logs.sh
```

By default, each script writes logs under `./logs/` in the current playbook `assets` directory on the control host. Remote logs are written under `${CX8_REMOTE_BASE}/logs/` on each DUT.

# Step 10. Optional install or upgrade perftest for GPUDirect/Data Direct

Use this only when Step 6 or Step 8 reports that `ib_write_bw` does not support `--use_cuda_dmabuf` or `--use_data_direct`. It first tries the OS package path, then falls back to building `linux-rdma/perftest` from source with CUDA header support and installs it under `/usr/local` by default. The fallback source build defaults to pinned upstream tag `26.04.17` rather than a moving branch.

This is not a read-only smoke test. It may install build dependencies or upgrade related OS packages through the DUT package manager. Use `--check-only` first if you only want to inspect the current `ib_write_bw` capability.

```bash
./10_install_perftest_gdr.sh
```

After it passes, rerun Step 6 and then rerun Step 8 with `--gdr`:

```bash
./06_configure_roce_gdr_runtime.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr
```

For a dry check that makes no package or source changes:

```bash
./10_install_perftest_gdr.sh --check-only
```

If your validation requires a different perftest ref, set `CX8_PERFTEST_REF` in `00_env.local` or pass `--ref <git-ref>`.

# Step 11. Optional configure ACS/Data Direct GRUB settings

Use this only when Step 8 `--gdr` reaches the CUDA DMA-BUF/Data Direct path but fails with memory-registration errors, or when `sudo rdma_topo check` reports ACS/IOMMU-group failures. This step uses `rdma_topo` to inspect the current topology and, only with `--apply`, writes the ACS/Data Direct GRUB configuration needed by GPUDirect/Data Direct.

Step 11 is a selective `config_acs=` policy generated by `rdma_topo`; it is not a blanket "disable ACS everywhere" change. See the ACS/Data Direct appendix for an example of the default-vs-expected values this step fixes.

Run Step 11 from the separate control host. If a DUT is used as the control host, rebooting that DUT will interrupt the playbook and can leave the two systems in different states.

First inspect without changing the system:

```bash
./11_configure_acs_grub.sh
```

If the output shows ACS/Data Direct failures and you need GPUDirect/Data Direct performance validation, apply the GRUB configuration:

```bash
./11_configure_acs_grub.sh --apply
```

Before rebooting, confirm both DUTs are idle, no shared workload is active, and the workload owner has approved the downtime. Reboot both DUTs after `--apply`, then rerun:

```bash
./05_configure_rails_runtime.sh --persist
./06_configure_roce_gdr_runtime.sh
./07_validate_setup.sh
./08_run_perftest_pair.sh --rail 0 --gdr
./08_run_perftest_pair.sh --rail 1 --gdr
```

Step 3 is optional after reboot unless the OS/packages changed. Step 4 is optional unless the cable/link state is uncertain. Step 5 uses `--persist` here because this advanced GDR path intentionally crosses a reboot and often continues into longer performance validation, so the rail IP/MTU configuration should remain stable without rerunning Step 5 manually.

# Step 99. Optional cleanup

Use this when you want to return the two DUTs to a clean runtime state before rerunning setup or handing the systems back. Cleanup does not reinstall the OS, update firmware, remove drivers, or reboot the DUTs.

Default cleanup removes only the configured rail IP addresses and resets both CX8 rail MTUs to 1500:

```bash
./99_cleanup_runtime.sh
```

If Step 5 was run with `--persist`, also remove the generated netplan file:

```bash
./99_cleanup_runtime.sh --remove-persist
```

To also bring the two CX8 rail interfaces down:

```bash
./99_cleanup_runtime.sh --down
```

Options can be combined:

```bash
./99_cleanup_runtime.sh --remove-persist --down
```

After cleanup, rerun from Step 3 or Step 5 depending on how much state you want to re-check. If you changed the rail CIDRs in `00_env.local`, cleanup removes those configured CIDRs rather than the default example addresses.

# Troubleshooting quick reference

| Symptom | Meaning | Action |
|---|---|---|
| Step 7 or Step 8 says a rail IP is missing | The temporary runtime rail IP/MTU setup is not present, commonly after reboot or network service refresh | Rerun `./05_configure_rails_runtime.sh`, then `./06_configure_roce_gdr_runtime.sh` and `./07_validate_setup.sh`. Use `./05_configure_rails_runtime.sh --persist` when the pair must keep rail IPs after reboot or across repeated performance runs. |
| Step 8 host-memory bandwidth is around 220-230 Gb/s on a 400G rail | Single-QP host-memory smoke test is below line-rate; the rail can still be functional | Use the result as basic RDMA proof. For GPU performance acceptance, run Step 8 `--gdr`. |
| Step 8 `--gdr` server exits before the client connects | Perftest server failed during CUDA/GDR setup or rail IP state is stale | Inspect the saved server log. Rerun Step 5/6/7 first if rail IP or GDR prerequisites changed. |
| A killed or interrupted `--gdr` run leaves CUDA/GPU state stuck | CUDA context cleanup may not have completed after interruption | Stop stale `ib_write_bw` processes. Only if the GPU is idle, no shared workload is active, and policy/workload-owner approval allows it, reset the GPU with `sudo nvidia-smi --gpu-reset -i <gpu-index>`, then rerun Step 8. |
| Step 6 says `nvidia_peermem` is not loaded | Default path can use CUDA DMA-BUF/Data Direct instead of legacy peermem | Continue unless you explicitly need peermem. Validate the GDR path with Step 8 `--gdr`. |

# Next steps

Use the validation level that matches the next workload:

- Step 7 pass means the two DGX Stations have basic two-rail RoCE connectivity: correct rail IPs, MTU, route selection, link state, jumbo ping, and RDMA port visibility.
- Step 8 without `--gdr` pass means host-memory RDMA bandwidth works on each rail. It is not the line-rate acceptance bar for GPU workloads.
- Step 8 with `--gdr` pass means the CUDA DMA-BUF/Data Direct GPU-memory RDMA path works. Use this bar before accepting the setup for GPU-performance work such as NCCL, vLLM, or other distributed AI workloads.

Full NCCL/application validation still requires container RDMA device exposure, NCCL environment selection, and a distributed launcher configuration.

```text
QSFP cables
-> CX8 link up at 400G x 2
-> Linux netdevs with rail IPs and MTU 9000
-> RoCEv2 / libibverbs / RDMA devices mlx5_0 and mlx5_1
-> Optional Step 8 --gdr CUDA DMA-BUF/Data Direct validation
-> NCCL NET/IB transport
-> PyTorch, vLLM, Ray, MPI, or another distributed application
```

| Layer | What still needs to be set up | How to tell it is working |
|---|---|---|
| Container/runtime | Expose RDMA devices to the application environment. For Docker this usually means `--network host`, `--ipc host`, `--device /dev/infiniband`, and enough locked-memory limit. | Inside the container, `/dev/infiniband` exists and RDMA/NCCL commands can open `mlx5_0` and `mlx5_1`. |
| NCCL | Install NCCL and select the CX8 rails, for example `NCCL_IB_HCA=mlx5_0,mlx5_1`. Use `NCCL_DEBUG=INFO` for first validation. | NCCL log shows `NET/IB` and both `mlx5_0` / `mlx5_1`. |
| Distributed launcher | Configure MPI, PyTorch distributed, Ray, vLLM, or another launcher so both systems join the same job. | The launcher reports both nodes/ranks as alive and the workload uses both GPUs/systems. |
| GPUDirect RDMA | If GPU-memory RDMA is required, CUDA DMA-BUF/Data Direct or another supported GPUDirect path must work. Step 10 may be required if `ib_write_bw` lacks the Data Direct flags. Step 11 may be required if ACS/Data Direct prerequisites are not yet configured. | Step 8 `--gdr` passes at expected per-rail bandwidth; NCCL can move GPU buffers without host-memory staging. |

The key NCCL validation signal is the transport line in the NCCL log:

```text
Good: NCCL log shows NET/IB and mlx5_0 / mlx5_1
Bad:  NCCL log shows only NET/Socket, which means TCP fallback
```

If Step 8 host-memory RDMA passes but NCCL falls back to `NET/Socket`, the CX8 fabric is likely usable and the issue is usually in the container, NCCL environment, RDMA device visibility, or launcher configuration. If GPU-memory RDMA is required, validate the supported GPUDirect path explicitly with Step 8 `--gdr` and NCCL `NET/IB` logs before accepting the setup for performance work.

Add ACS/Data Direct grub configuration with Step 11 only after the basic two-rail setup is proven and GPUDirect/Data Direct validation requires it.

# Appendix: ACS/Data Direct reference

The platform default ACS policy may be conservative for normal PCIe isolation but not suitable for GB300 + CX8 Data Direct GPU-memory RDMA. In that failure mode, `rdma_topo check` reports ACS bit mismatches and the CX8 Data Direct DMA function and GB300 GPU are in different IOMMU groups. Step 11 uses `rdma_topo write-grub-acs` to write the selective `config_acs=` policy expected by this topology.

Example failure before Step 11:

```text
FAIL ACS for grace_rp 0009:00:00.0 0011101 != xx111x0, 0x1d != 0x1c
FAIL ACS for cx_switch 0009:02:00.0 0011101 != xx110x1, 0x1d != 0x19
FAIL ACS for cx_switch 0009:02:02.0 0011101 != xx101x1, 0x1d != 0x15
FAIL Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are not equal
```

Expected success after Step 11 apply and reboot:

```text
OK ACS for grace_rp 0009:00:00.0 has correct values 0011100 = xx111x0
OK ACS for cx_switch 0009:02:00.0 has correct values 0011001 = xx110x1
OK ACS for cx_switch 0009:02:02.0 has correct values 0010101 = xx101x1
OK Kernel iommu_group for DMA 0009:03:00.0 and GPU 0009:06:00.0 are equal
```

# Tools and commands reference

| Command | Required for basic setup? | Usually provided by | Used in | Purpose |
|---|---:|---|---|---|
| `bash` | Yes | Base OS | All scripts | Runs the setup scripts |
| `ssh` | Yes, on control host | OpenSSH client | All control-host scripts | Connects from control host to each DUT |
| `python3` / `py -3` | Optional control-host alternative | Python 3 | Cross-platform wrapper | Runs `cx8_setup.py` on Windows, macOS, or Linux |
| `sudo` | Yes | Base OS | Steps 5, 6, and 99 | Privileged network, module, and cleanup commands on DUTs |
| `lspci` | Yes | `pciutils` | Steps 1 and 3 | Lists PCIe devices; confirms CX8/NVIDIA visibility |
| `nvidia-smi` | Yes | NVIDIA driver | Steps 1 and 3 | GPU driver visibility, identity, PCI bus IDs |
| `ibdev2netdev` | Yes | DOCA/OFED or `rdma-core` | Steps 1, 3–8 | Maps `mlx5_0` and `mlx5_1` to Linux netdevs |
| `ibv_devinfo` | Yes | `rdma-core` / OFED | Steps 1, 3, 4, and 7 | CX8 port state, link layer, MTU, active speed |
| `ip` | Yes | `iproute2` | Steps 1, 3, 5, and 7 | Rail IP, link state, MTU, route selection |
| `ethtool` | Yes | `ethtool` package | Steps 1, 3–5, and 7 | Ethernet link status and speed per CX8 netdev |
| `show_gids` | Yes for RoCE validation | RDMA/OFED tools | Steps 1 and 3 | RoCE GIDs and IPv4 RoCEv2 GID visibility |
| `rdma_topo` | Yes for GPUDirect/Data Direct | DOCA Host / DOCA-OFED | Steps 1, 3, and 11 | GPU/NIC PCIe topology and ACS/Data Direct prerequisites |
| `mlnx_qos` | Yes for RoCE tuning | Mellanox/NVIDIA OFED | Steps 1 and 6 | DSCP trust and PFC settings for RoCE |
| `cma_roce_tos` | Yes for RoCE ToS setup | Mellanox/NVIDIA OFED | Steps 1 and 6 | RoCE CM ToS/DSCP value for RDMA connection setup |
| `ib_write_bw` | Yes for RDMA bandwidth test | `perftest` package | Steps 1 and 8 | RDMA write bandwidth test between DUTs |
| `ping` | Yes for basic rail validation | `iputils-ping` | Step 7 | Jumbo-frame ICMP probes per rail |
| `lsmod` | Yes for GPUDirect status | `kmod` | Steps 3 and 7 | Reports whether `nvidia_peermem` is loaded |
| `modprobe` | Optional for peermem setup attempt | `kmod` | Step 6 only when `CX8_TRY_NVIDIA_PEERMEM=1` | Attempts to load `nvidia_peermem`; Step 8 `--gdr` can still use CUDA DMA-BUF/Data Direct when supported |
| `dmesg` | Useful for debug | `util-linux` | Step 3 | Recent CX/RDMA/GPU/PCIe error highlights |
| `netplan` | Optional | Ubuntu netplan | Step 5 with `--persist` | Applies persistent rail IP/MTU configuration |
| `flint` | Optional | NVIDIA MFT | Steps 1 and 3 visibility only | CX8 firmware version and product info |
| `mlxconfig` | Optional | NVIDIA MFT | Step 1 visibility only | NIC firmware configuration queries |
| `tar` | Yes on control host and DUTs | Base OS | Step 2 and log packaging | Streams setup helpers to DUTs |
| `sha256sum` or `shasum` | Optional | GNU coreutils or macOS `shasum` | Log packaging | Checksum for the archived log bundle |

If `flint` and `mlxconfig` are missing, that is acceptable for the basic rail setup path because this playbook is not changing NIC firmware configuration.