---
title: "Connect Two Nodes for Distributed Workloads — Troubleshooting"
canonical: "https://build.nvidia.com/playbooks/connect-two-stations/troubleshooting.md"
---

| Symptom | Cause | Fix |
|---------|-------|-----|
| `mlx5_0` or `mlx5_1` is missing in `./03_prereq_check.sh` output | CX8 driver not loaded, PCIe issue, or incomplete software image | Run `lspci \| grep -i mellanox` and `ibdev2netdev` on the DUT. Confirm DOCA/OFED and the NVIDIA driver are installed. Contact NVIDIA support if the HCA is not visible. |
| Required command missing in `./01_probe_access.sh` output (other than `flint` / `mlxconfig`) | Software image missing a core RDMA or network tool | Stop setup. Install or restore the missing package (`rdma-core`, `perftest`, `ethtool`, etc.) or contact NVIDIA support. |
| `./04_check_cable_presence.sh` reports link down on one or both rails | Cable not seated, wrong QSFP port, or crisscrossed rails | Confirm QSFP0-to-QSFP0 and QSFP1-to-QSFP1 with no crisscross. Reseat both ends on the failed rail and rerun the cable/link presence check. |
| `./04_check_cable_presence.sh` reports `Cable error` or `High Temperature` | QSFP module thermal or cable fault | Stop software setup. Let the module cool, verify CX8/QSFP airflow, and reseat the cable. Swap the cable with the other rail or replace it if the condition returns. |
| `./04_check_cable_presence.sh` reports `speed_ok=no` or `200G` instead of `400G` | Cable not 400G-capable, degraded negotiation, or port/module issue | Replace with 400G-capable QSFP cables. Limited-speed rail/RoCE/perftest validation is possible, but do not claim 800G pass until both rails report 400G. |
| Rail or RoCE runtime configuration fails with permission errors | Missing `sudo` access for the SSH user | Ensure the `nvidia` user (or `CX8_OS_USER`) can run privileged network commands. Enter the DUT OS password at both the SSH and `[sudo]` prompts. |
| `./07_validate_setup.sh` reports missing rail IP or wrong MTU | `./05_configure_rails_runtime.sh` did not complete, or NetworkManager/system restart removed the temporary in-memory rail profile | Rerun `./05_configure_rails_runtime.sh` and then `./07_validate_setup.sh`. Do not use a raw `ip addr` assignment on a NetworkManager-managed rail because its DHCP profile can remove that address. |
| `./07_validate_setup.sh` jumbo ping fails on one rail | MTU mismatch, wrong route, or cable/link issue on that rail | Check `ip addr` and `ip route` on both DUTs for the failed rail subnet. Rerun `./04_check_cable_presence.sh` for link health, then `./05_configure_rails_runtime.sh` for IP/MTU. |
| `./07_validate_setup.sh` reports `Link detected: no` or `Speed: Unknown` | Physical link not up despite earlier actions | Rerun `./04_check_cable_presence.sh`. Inspect cable, port, and module before retrying `./07_validate_setup.sh`. |
| `./07_validate_setup.sh` reports `Link detected: no (Overheat)`, `High Temperature`, or `Cable error` | QSFP/CX8 thermal or cable condition | Stop software setup. Let the module cool, verify airflow, reseat the cable, and swap/replace the failed rail cable if the condition returns. |
| `./07_validate_setup.sh` warns that rails are below 400G | Degraded link speed from cable, port, or negotiation | Continue only for limited-speed functional validation. Do not treat perftest results as an 800G pass until `./04_check_cable_presence.sh` reports 400G on both rails. |
| `./06_configure_roce_gdr_runtime.sh` prints a yellow `WARN` for `cma_roce_tos` | RoCE ToS tool not available or not applicable on this image | Continue with `./07_validate_setup.sh` for basic RoCE validation. ToS tuning is optional for the basic fabric path. |
| `nvidia_peermem` is not loaded | Normal on some OS images when CUDA DMA-BUF/Data Direct is the GPUDirect path | Continue. Basic rail/RoCE validation is not blocked. Before GDR, require the Script 06 perftest-flag pass and Script 11 ACS/Data Direct pass. |
| `nvidia_peermem` fails to load (`Invalid argument`) after `CX8_TRY_NVIDIA_PEERMEM=1` | The optional peermem-based GPUDirect path is unavailable on this stack | Do not treat this as a basic setup failure. Use the CUDA DMA-BUF/Data Direct path only after its Script 06 and Script 11 readiness gates pass. |
| Script 06 or the Script 08 GDR prerequisite gate says `ib_write_bw` does not support `--use_cuda_dmabuf` or `--use_data_direct` | Installed `perftest` is too old or was built without CUDA DMA-BUF/Data Direct support | Run host-memory Script 08 first for basic RDMA. If GPU-memory RDMA is required, run Script 10, rerun Script 06, and complete the Script 11 ACS readiness gate before either GDR rail test. |
| The Script 08 GDR prerequisite gate stops before starting a server | Perftest GDR flags or ACS/Data Direct topology did not pass on one or both DUTs | Inspect `*_gdr_preflight.log`. Use Script 10 only for missing flags. Use Script 11 `--apply` plus reboot only for an ACS-not-ready result; then rerun Scripts 05 through 07 and Script 11 before retrying GDR. |
| A GDR test reaches `mlx5dv_reg_dmabuf_mr` and fails with error 524 despite the prerequisite gate | Prerequisite enforcement was bypassed, topology changed after the check, or a new kernel/driver defect exists | Stop and collect the GDR logs plus current `rdma_topo check` and kernel messages. Do not treat this as the normal setup path or repeatedly rerun GDR. |
| `./08_run_perftest_pair.sh` fails or shows very low bandwidth | Wrong rail selected, firewall, or degraded link speed | Confirm `./07_validate_setup.sh` passed. Run perftest per rail separately. Check `./04_check_cable_presence.sh` speed reports. Verify rail IPs and routes match the selected `--rail` index. |
| NCCL shows `NET/Socket` instead of `NET/IB` after fabric passes | Container missing RDMA devices, wrong `NCCL_IB_HCA`, or launcher config | Expose `/dev/infiniband` to the runtime. Set `NCCL_IB_HCA=mlx5_0,mlx5_1`. Use `NCCL_DEBUG=INFO` and confirm both HCAs appear in the log. |
| `rdma_topo check` reports ACS/Data Direct prerequisites not met | PCIe topology or ACS configuration not set for GPUDirect/Data Direct | Complete basic two-rail setup first. Use Script 11 `--apply` only when GDR validation is required, then reboot. Restore runtime setup with Scripts 05 through 07 and require Script 11 to pass before either GDR rail test. |
| SSH control-socket reset or repeated connection-reuse failure | SSH multiplexing is incompatible with the control-host environment | Retry with `CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail <0-or-1>`. This affects only SSH connection reuse, not RDMA, Data Direct, or perftest behavior. Windows Git Bash/MSYS/Cygwin disables multiplexing automatically. |
| Rail settings lost after reboot | `./05_configure_rails_runtime.sh` was run without `--persist` | Expected for runtime setup. Rerun the command after reboot, or use `--persist` if settings must survive reboot. |
| Persistent netplan needs removal | `./05_configure_rails_runtime.sh --persist` was used and you want to undo it | Run `./99_cleanup_runtime.sh --remove-persist` or manually remove `/etc/netplan/60-cx8-fabric.yaml` on both DUTs and `sudo netplan apply`. |

## Stop conditions

Stop and inspect before changing anything else if:

- `mlx5_0` or `mlx5_1` is missing
- A rail remains down after the cable is connected
- Jumbo ping fails on either rail
- RDMA tools are missing
- CUDA DMA-BUF/Data Direct perftest flags are unavailable
- `rdma_topo check` says ACS/Data Direct prerequisites are not met

Inspect ACS/Data Direct readiness before the first GDR test. Add the generated GRUB configuration with Script 11 `--apply` only after the basic two-rail setup is proven, the inspection is not ready, and GPUDirect/Data Direct validation requires it.