Set Up NCCL for Multi-Node GPU Communication

Common issues for multi-node NCCL

Symptom	Cause	Fix
`mpirun` hangs or times out	SSH connectivity issues	1. Test basic SSH connectivity: `ssh <remote_ip>` should work without password prompts 2. Try a simple mpirun test: `mpirun -np 2 -H <IP for Node 1>:1,<IP for Node 2>:1 hostname` 3. Verify SSH keys are set up correctly for all nodes
Network interface not found	Wrong interface name or down status	Check interface status with `ibdev2netdev` and verify IP configuration
NCCL build fails	Missing dependencies such as OpenMPI or incorrect CUDA version	Verify CUDA installation and required libraries are present

For latest known issues, see the documentation linked under Resources for your hardware platform.