---
title: "Connect Two DGX Stations for Distributed Workloads — Troubleshooting"
canonical: "https://build.nvidia.com/station/connect-two-stations/troubleshooting.md"
---

| Symptom | Cause | Fix |
|---------|-------|-----|
| `mlx5_0` or `mlx5_1` is missing in Step 3 output | CX8 driver not loaded, PCIe issue, or incomplete SOT image | Run `lspci \| grep -i mellanox` and `ibdev2netdev` on the DUT. Confirm DOCA/OFED and the NVIDIA driver are installed. Contact NVIDIA support if the HCA is not visible. |
| Required command missing in Step 1 (other than `flint` / `mlxconfig`) | SOT software image missing a core RDMA or network tool | Stop setup. Install or restore the missing package from the SOT image (`rdma-core`, `perftest`, `ethtool`, etc.) or contact NVIDIA support. |
| Step 4 reports link down on one or both rails | Cable not seated, wrong QSFP port, or crisscrossed rails | Confirm QSFP0-to-QSFP0 and QSFP1-to-QSFP1 with no crisscross. Reseat both ends on the failed rail and rerun `./04_check_cable_presence.sh`. |
| Step 4 reports `Cable error` or `High Temperature` | QSFP module thermal or cable fault | Stop software setup. Let the module cool, verify CX8/QSFP airflow, reseat the cable. Swap the cable with the other rail or replace it if the condition returns. |
| Step 4 reports `speed_ok=no` or `200G` instead of `400G` | Cable not 400G-capable, degraded negotiation, or port/module issue | Replace with 400G-capable QSFP cables. Limited-speed validation (Steps 5–8) is possible, but do not claim 800G pass until both rails report 400G. |
| Step 5 or 6 fails with permission errors | Missing `sudo` access for the SSH user | Ensure the `nvidia` user (or `CX8_OS_USER`) can run privileged network commands. Enter the DUT OS password at both the SSH and `[sudo]` prompts. |
| Step 7 reports missing rail IP or wrong MTU | Step 5 did not complete or interface was reset | Rerun `./05_configure_rails_runtime.sh` and then `./07_validate_setup.sh`. |
| Step 7 jumbo ping fails on one rail | MTU mismatch, wrong route, or cable/link issue on that rail | Check `ip addr` and `ip route` on both DUTs for the failed rail subnet. Rerun Step 4 for link health, then Step 5 for IP/MTU. |
| Step 7 reports `Link detected: no` or `Speed: Unknown` | Physical link not up despite earlier steps | Rerun `./04_check_cable_presence.sh`. Inspect cable, port, and module before retrying Step 7. |
| Step 7 reports `Link detected: no (Overheat)`, `High Temperature`, or `Cable error` | QSFP/CX8 thermal or cable condition | Stop software setup. Let the module cool, verify airflow, reseat the cable, and swap/replace the failed rail cable if the condition returns. |
| Step 7 yellow warning that rails are below 400G | Degraded link speed from cable, port, or negotiation | Continue only for limited-speed functional validation. Do not treat Step 8 results as an 800G pass until Step 4 reports 400G on both rails. |
| Yellow `WARN` for `cma_roce_tos` in Step 6 | RoCE ToS tool not available or not applicable on this image | Continue to Step 7 for basic RoCE validation. ToS tuning is optional for the basic fabric path. |
| `nvidia_peermem` is not loaded | Normal on some DGX Station OS images when CUDA DMA-BUF/Data Direct is the GPUDirect path | Continue. Basic rail/RoCE validation is not blocked. Step 8 `--gdr` can pass through CUDA DMA-BUF/Data Direct if `ib_write_bw` supports `--use_cuda_dmabuf` and `--use_data_direct`. |
| `nvidia_peermem` fails to load (`Invalid argument`) after `CX8_TRY_NVIDIA_PEERMEM=1` | The optional peermem-based GPUDirect path is unavailable on this stack | Do not treat this as a basic setup failure. Use the CUDA DMA-BUF/Data Direct `--gdr` path first. Collect Step 6/8 logs only if the `--gdr` path also fails. |
| Step 6 or Step 8 says `ib_write_bw` does not support `--use_cuda_dmabuf` or `--use_data_direct` | Installed `perftest` is too old or was built without CUDA DMA-BUF/Data Direct support | Run host-memory Step 8 first for basic RDMA. If GPU-memory RDMA is required, run `./10_install_perftest_gdr.sh`, then rerun Step 6 and Step 8 `--gdr`. |
| Step 8 `--gdr` reaches CUDA DMA-BUF/Data Direct but fails with `mlx5dv_reg_dmabuf_mr` / `Couldn't allocate MR with error=524` | ACS/Data Direct/IOMMU prerequisites are not configured for the GPU and ConnectX path | Run `sudo rdma_topo check` or `./11_configure_acs_grub.sh` to confirm. If ACS/IOMMU checks fail and GDR is required, run `./11_configure_acs_grub.sh --apply`, reboot both DUTs, then rerun Steps 5, 6, 7, and Step 8 `--gdr`. Step 3 is only needed if OS/packages changed; Step 4 is only needed if cable/link state is uncertain. |
| Step 8 perftest fails or shows very low bandwidth | Wrong rail selected, firewall, or degraded link speed | Confirm Step 7 passed. Run perftest per rail separately. Check Step 4 speed reports. Verify rail IPs and routes match the selected `--rail` index. |
| NCCL shows `NET/Socket` instead of `NET/IB` after fabric passes | Container missing RDMA devices, wrong `NCCL_IB_HCA`, or launcher config | Expose `/dev/infiniband` to the runtime. Set `NCCL_IB_HCA=mlx5_0,mlx5_1`. Use `NCCL_DEBUG=INFO` and confirm both HCAs appear in the log. |
| `rdma_topo check` reports ACS/Data Direct prerequisites not met | PCIe topology or ACS configuration not set for GPUDirect/Data Direct | Complete basic two-rail setup first (Step 7 pass). Run Step 11 dry-run to capture the current state, then use `./11_configure_acs_grub.sh --apply` only when GPUDirect/Data Direct validation is required. Reboot before retesting, then restore runtime setup with Steps 5, 6, and 7 before Step 8 `--gdr`. |
| SSH control-socket reset or repeated connection-reuse failure | SSH multiplexing is incompatible with the control-host environment | Retry with `CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail <0-or-1>`. This affects only SSH connection reuse, not RDMA, Data Direct, or perftest behavior. Windows Git Bash/MSYS/Cygwin disables multiplexing automatically. |
| Rail settings lost after reboot | Step 5 run without `--persist` | Expected for runtime setup. Rerun `./05_configure_rails_runtime.sh` after reboot, or use `--persist` if settings must survive reboot. |
| Persistent netplan needs removal | Step 5 was run with `--persist` and you want to undo | Run `./99_cleanup_runtime.sh --remove-persist` or manually remove `/etc/netplan/60-cx8-fabric.yaml` on both DUTs and `sudo netplan apply`. |

## Stop conditions

Stop and inspect before changing anything else if:

- `mlx5_0` or `mlx5_1` is missing
- A rail remains down after the cable is connected
- Jumbo ping fails on either rail
- RDMA tools are missing
- CUDA DMA-BUF/Data Direct is unavailable or Step 8 `--gdr` fails when GPU-memory RDMA is required
- `rdma_topo check` says ACS/Data Direct prerequisites are not met

Add ACS/Data Direct grub configuration with Step 11 only after the basic two-rail setup is proven and GPUDirect/Data Direct validation requires it.