Combined memory and compute over a direct high-speed link
| Symptom | Cause | Fix |
|---|---|---|
mlx5_0 or mlx5_1 is missing in Step 3 output | CX8 driver not loaded, PCIe issue, or incomplete SOT image | Run lspci | grep -i mellanox and ibdev2netdev on the DUT. Confirm DOCA/OFED and the NVIDIA driver are installed. Contact NVIDIA support if the HCA is not visible. |
Required command missing in Step 1 (other than flint / mlxconfig) | SOT software image missing a core RDMA or network tool | Stop setup. Install or restore the missing package from the SOT image (rdma-core, perftest, ethtool, etc.) or contact NVIDIA support. |
| Step 4 reports link down on one or both rails | Cable not seated, wrong QSFP port, or crisscrossed rails | Confirm QSFP0-to-QSFP0 and QSFP1-to-QSFP1 with no crisscross. Reseat both ends on the failed rail and rerun ./04_check_cable_presence.sh. |
Step 4 reports Cable error or High Temperature | QSFP module thermal or cable fault | Stop software setup. Let the module cool, verify CX8/QSFP airflow, reseat the cable. Swap the cable with the other rail or replace it if the condition returns. |
Step 4 reports speed_ok=no or 200G instead of 400G | Cable not 400G-capable, degraded negotiation, or port/module issue | Replace with 400G-capable QSFP cables. Limited-speed validation (Steps 5–8) is possible, but do not claim 800G pass until both rails report 400G. |
| Step 5 or 6 fails with permission errors | Missing sudo access for the SSH user | Ensure the nvidia user (or CX8_OS_USER) can run privileged network commands. Enter the DUT OS password at both the SSH and [sudo] prompts. |
| Step 7 reports missing rail IP or wrong MTU | Step 5 did not complete or interface was reset | Rerun ./05_configure_rails_runtime.sh and then ./07_validate_setup.sh. |
| Step 7 jumbo ping fails on one rail | MTU mismatch, wrong route, or cable/link issue on that rail | Check ip addr and ip route on both DUTs for the failed rail subnet. Rerun Step 4 for link health, then Step 5 for IP/MTU. |
Step 7 reports Link detected: no or Speed: Unknown | Physical link not up despite earlier steps | Rerun ./04_check_cable_presence.sh. Inspect cable, port, and module before retrying Step 7. |
Step 7 reports Link detected: no (Overheat), High Temperature, or Cable error | QSFP/CX8 thermal or cable condition | Stop software setup. Let the module cool, verify airflow, reseat the cable, and swap/replace the failed rail cable if the condition returns. |
| Step 7 yellow warning that rails are below 400G | Degraded link speed from cable, port, or negotiation | Continue only for limited-speed functional validation. Do not treat Step 8 results as an 800G pass until Step 4 reports 400G on both rails. |
Yellow WARN for cma_roce_tos in Step 6 | RoCE ToS tool not available or not applicable on this image | Continue to Step 7 for basic RoCE validation. ToS tuning is optional for the basic fabric path. |
nvidia_peermem is not loaded | Normal on some DGX Station OS images when CUDA DMA-BUF/Data Direct is the GPUDirect path | Continue. Basic rail/RoCE validation is not blocked. Step 8 --gdr can pass through CUDA DMA-BUF/Data Direct if ib_write_bw supports --use_cuda_dmabuf and --use_data_direct. |
nvidia_peermem fails to load (Invalid argument) after CX8_TRY_NVIDIA_PEERMEM=1 | The optional peermem-based GPUDirect path is unavailable on this stack | Do not treat this as a basic setup failure. Use the CUDA DMA-BUF/Data Direct --gdr path first. Collect Step 6/8 logs only if the --gdr path also fails. |
Step 6 or Step 8 says ib_write_bw does not support --use_cuda_dmabuf or --use_data_direct | Installed perftest is too old or was built without CUDA DMA-BUF/Data Direct support | Run host-memory Step 8 first for basic RDMA. If GPU-memory RDMA is required, run ./10_install_perftest_gdr.sh, then rerun Step 6 and Step 8 --gdr. |
Step 8 --gdr reaches CUDA DMA-BUF/Data Direct but fails with mlx5dv_reg_dmabuf_mr / Couldn't allocate MR with error=524 | ACS/Data Direct/IOMMU prerequisites are not configured for the GPU and ConnectX path | Run sudo rdma_topo check or ./11_configure_acs_grub.sh to confirm. If ACS/IOMMU checks fail and GDR is required, run ./11_configure_acs_grub.sh --apply, reboot both DUTs, then rerun Steps 5, 6, 7, and Step 8 --gdr. Step 3 is only needed if OS/packages changed; Step 4 is only needed if cable/link state is uncertain. |
| Step 8 perftest fails or shows very low bandwidth | Wrong rail selected, firewall, or degraded link speed | Confirm Step 7 passed. Run perftest per rail separately. Check Step 4 speed reports. Verify rail IPs and routes match the selected --rail index. |
NCCL shows NET/Socket instead of NET/IB after fabric passes | Container missing RDMA devices, wrong NCCL_IB_HCA, or launcher config | Expose /dev/infiniband to the runtime. Set NCCL_IB_HCA=mlx5_0,mlx5_1. Use NCCL_DEBUG=INFO and confirm both HCAs appear in the log. |
rdma_topo check reports ACS/Data Direct prerequisites not met | PCIe topology or ACS configuration not set for GPUDirect/Data Direct | Complete basic two-rail setup first (Step 7 pass). Run Step 11 dry-run to capture the current state, then use ./11_configure_acs_grub.sh --apply only when GPUDirect/Data Direct validation is required. Reboot before retesting, then restore runtime setup with Steps 5, 6, and 7 before Step 8 --gdr. |
| SSH control-socket reset or repeated connection-reuse failure | SSH multiplexing is incompatible with the control-host environment | Retry with CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail <0-or-1>. This affects only SSH connection reuse, not RDMA, Data Direct, or perftest behavior. Windows Git Bash/MSYS/Cygwin disables multiplexing automatically. |
| Rail settings lost after reboot | Step 5 run without --persist | Expected for runtime setup. Rerun ./05_configure_rails_runtime.sh after reboot, or use --persist if settings must survive reboot. |
| Persistent netplan needs removal | Step 5 was run with --persist and you want to undo | Run ./99_cleanup_runtime.sh --remove-persist or manually remove /etc/netplan/60-cx8-fabric.yaml on both DUTs and sudo netplan apply. |
Stop and inspect before changing anything else if:
mlx5_0 or mlx5_1 is missing--gdr fails when GPU-memory RDMA is requiredrdma_topo check says ACS/Data Direct prerequisites are not metAdd ACS/Data Direct grub configuration with Step 11 only after the basic two-rail setup is proven and GPUDirect/Data Direct validation requires it.