Combined memory and compute over a direct high-speed link
| Symptom | Cause | Fix |
|---|---|---|
mlx5_0 or mlx5_1 is missing in ./03_prereq_check.sh output | CX8 driver not loaded, PCIe issue, or incomplete software image | Run lspci | grep -i mellanox and ibdev2netdev on the DUT. Confirm DOCA/OFED and the NVIDIA driver are installed. Contact NVIDIA support if the HCA is not visible. |
Required command missing in ./01_probe_access.sh output (other than flint / mlxconfig) | Software image missing a core RDMA or network tool | Stop setup. Install or restore the missing package (rdma-core, perftest, ethtool, etc.) or contact NVIDIA support. |
./04_check_cable_presence.sh reports link down on one or both rails | Cable not seated, wrong QSFP port, or crisscrossed rails | Confirm QSFP0-to-QSFP0 and QSFP1-to-QSFP1 with no crisscross. Reseat both ends on the failed rail and rerun the cable/link presence check. |
./04_check_cable_presence.sh reports Cable error or High Temperature | QSFP module thermal or cable fault | Stop software setup. Let the module cool, verify CX8/QSFP airflow, and reseat the cable. Swap the cable with the other rail or replace it if the condition returns. |
./04_check_cable_presence.sh reports speed_ok=no or 200G instead of 400G | Cable not 400G-capable, degraded negotiation, or port/module issue | Replace with 400G-capable QSFP cables. Limited-speed rail/RoCE/perftest validation is possible, but do not claim 800G pass until both rails report 400G. |
| Rail or RoCE runtime configuration fails with permission errors | Missing sudo access for the SSH user | Ensure the nvidia user (or CX8_OS_USER) can run privileged network commands. Enter the DUT OS password at both the SSH and [sudo] prompts. |
./07_validate_setup.sh reports missing rail IP or wrong MTU | ./05_configure_rails_runtime.sh did not complete, or NetworkManager/system restart removed the temporary in-memory rail profile | Rerun ./05_configure_rails_runtime.sh and then ./07_validate_setup.sh. Do not use a raw ip addr assignment on a NetworkManager-managed rail because its DHCP profile can remove that address. |
./07_validate_setup.sh jumbo ping fails on one rail | MTU mismatch, wrong route, or cable/link issue on that rail | Check ip addr and ip route on both DUTs for the failed rail subnet. Rerun ./04_check_cable_presence.sh for link health, then ./05_configure_rails_runtime.sh for IP/MTU. |
./07_validate_setup.sh reports Link detected: no or Speed: Unknown | Physical link not up despite earlier actions | Rerun ./04_check_cable_presence.sh. Inspect cable, port, and module before retrying ./07_validate_setup.sh. |
./07_validate_setup.sh reports Link detected: no (Overheat), High Temperature, or Cable error | QSFP/CX8 thermal or cable condition | Stop software setup. Let the module cool, verify airflow, reseat the cable, and swap/replace the failed rail cable if the condition returns. |
./07_validate_setup.sh warns that rails are below 400G | Degraded link speed from cable, port, or negotiation | Continue only for limited-speed functional validation. Do not treat perftest results as an 800G pass until ./04_check_cable_presence.sh reports 400G on both rails. |
./06_configure_roce_gdr_runtime.sh prints a yellow WARN for cma_roce_tos | RoCE ToS tool not available or not applicable on this image | Continue with ./07_validate_setup.sh for basic RoCE validation. ToS tuning is optional for the basic fabric path. |
nvidia_peermem is not loaded | Normal on some OS images when CUDA DMA-BUF/Data Direct is the GPUDirect path | Continue. Basic rail/RoCE validation is not blocked. Before GDR, require the Script 06 perftest-flag pass and Script 11 ACS/Data Direct pass. |
nvidia_peermem fails to load (Invalid argument) after CX8_TRY_NVIDIA_PEERMEM=1 | The optional peermem-based GPUDirect path is unavailable on this stack | Do not treat this as a basic setup failure. Use the CUDA DMA-BUF/Data Direct path only after its Script 06 and Script 11 readiness gates pass. |
Script 06 or the Script 08 GDR prerequisite gate says ib_write_bw does not support --use_cuda_dmabuf or --use_data_direct | Installed perftest is too old or was built without CUDA DMA-BUF/Data Direct support | Run host-memory Script 08 first for basic RDMA. If GPU-memory RDMA is required, run Script 10, rerun Script 06, and complete the Script 11 ACS readiness gate before either GDR rail test. |
| The Script 08 GDR prerequisite gate stops before starting a server | Perftest GDR flags or ACS/Data Direct topology did not pass on one or both DUTs | Inspect *_gdr_preflight.log. Use Script 10 only for missing flags. Use Script 11 --apply plus reboot only for an ACS-not-ready result; then rerun Scripts 05 through 07 and Script 11 before retrying GDR. |
A GDR test reaches mlx5dv_reg_dmabuf_mr and fails with error 524 despite the prerequisite gate | Prerequisite enforcement was bypassed, topology changed after the check, or a new kernel/driver defect exists | Stop and collect the GDR logs plus current rdma_topo check and kernel messages. Do not treat this as the normal setup path or repeatedly rerun GDR. |
./08_run_perftest_pair.sh fails or shows very low bandwidth | Wrong rail selected, firewall, or degraded link speed | Confirm ./07_validate_setup.sh passed. Run perftest per rail separately. Check ./04_check_cable_presence.sh speed reports. Verify rail IPs and routes match the selected --rail index. |
NCCL shows NET/Socket instead of NET/IB after fabric passes | Container missing RDMA devices, wrong NCCL_IB_HCA, or launcher config | Expose /dev/infiniband to the runtime. Set NCCL_IB_HCA=mlx5_0,mlx5_1. Use NCCL_DEBUG=INFO and confirm both HCAs appear in the log. |
rdma_topo check reports ACS/Data Direct prerequisites not met | PCIe topology or ACS configuration not set for GPUDirect/Data Direct | Complete basic two-rail setup first. Use Script 11 --apply only when GDR validation is required, then reboot. Restore runtime setup with Scripts 05 through 07 and require Script 11 to pass before either GDR rail test. |
| SSH control-socket reset or repeated connection-reuse failure | SSH multiplexing is incompatible with the control-host environment | Retry with CX8_DISABLE_SSH_MUX=1 ./08_run_perftest_pair.sh --rail <0-or-1>. This affects only SSH connection reuse, not RDMA, Data Direct, or perftest behavior. Windows Git Bash/MSYS/Cygwin disables multiplexing automatically. |
| Rail settings lost after reboot | ./05_configure_rails_runtime.sh was run without --persist | Expected for runtime setup. Rerun the command after reboot, or use --persist if settings must survive reboot. |
| Persistent netplan needs removal | ./05_configure_rails_runtime.sh --persist was used and you want to undo it | Run ./99_cleanup_runtime.sh --remove-persist or manually remove /etc/netplan/60-cx8-fabric.yaml on both DUTs and sudo netplan apply. |
Stop and inspect before changing anything else if:
mlx5_0 or mlx5_1 is missingrdma_topo check says ACS/Data Direct prerequisites are not metInspect ACS/Data Direct readiness before the first GDR test. Add the generated GRUB configuration with Script 11 --apply only after the basic two-rail setup is proven, the inspection is not ready, and GPUDirect/Data Direct validation requires it.