Ensure Same Username on Both Systems
On both systems check the username and make sure it's the same:
# Check current username
whoami
If usernames don't match, create a new user (e.g., nvidia) on both systems and login in with the new user:
# Create nvidia user and add to sudo group
sudo useradd -m nvidia
sudo usermod -aG sudo nvidia
# Set password for nvidia user
sudo passwd nvidia
# Switch to nvidia user
su - nvidia
Physical Hardware Connection
Connect the QSFP cable between both DGX Spark systems using any QSFP interface on each device. Make sure to use the same physical port on each device to prevent issues with NCCL tests. This establishes the 200GbE direct connection required for high-speed inter-node communication. Upon connection between the two nodes, you will see an output like the one below: in this example the interfaces showing as 'Up' are enp1s0f1np1 / enP2p1s0f1np1 (each physical port has two logical interface).
Example output:
# Check QSFP interface availability on both nodes
nvidia@dxg-spark-1:~$ ibdev2netdev
roceP2p1s0f0 port 1 ==> enP2p1s0f0np0 (Down)
roceP2p1s0f1 port 1 ==> enP2p1s0f1np1 (Up)
rocep1s0f0 port 1 ==> enp1s0f0np0 (Down)
rocep1s0f1 port 1 ==> enp1s0f1np1 (Up)
NOTE
If none of the interfaces are showing as 'Up', please check the QSFP cable connection, reboot the systems and try again. The interfaces showing as 'Up' depend on which port you are using to connect the two nodes. Each physical port has two logical interfaces, for example, enp1s0f1np1 and enP2p1s0f1np1 refer to the same physical port.
Network Interface Configuration
Choose one option to setup the network interfaces. Option 1 and 2 are mutually exclusive.
NOTE
Full bandwidth can be achieved with just one QSFP cable. When two QSFP cables are connected, all four interfaces must be assigned IP addresses to obtain full bandwidth.
Option 1: Manual IP Assignment with the netplan configure file
On node 1:
# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f1np1:
addresses:
- 192.168.100.10/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.101.10/24
dhcp4: no
EOF
# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
# Apply the configuration
sudo netplan apply
On node 2:
# Create the netplan configuration file
sudo tee /etc/netplan/40-cx7.yaml > /dev/null <<EOF
network:
version: 2
ethernets:
enp1s0f1np1:
addresses:
- 192.168.100.11/24
dhcp4: no
enP2p1s0f1np1:
addresses:
- 192.168.101.11/24
dhcp4: no
EOF
# Set appropriate permissions
sudo chmod 600 /etc/netplan/40-cx7.yaml
# Apply the configuration
sudo netplan apply
Option 2: Manual IP Assignment with command line
NOTE
Using this option, the IPs assigned to the interfaces will change if you reboot the system.
Use the "(Up)" interfaces. In this example, we'll use enp1s0f1np1 and enP2p1s0f1np1.
On Node 1:
# Assign static IP and bring up interface.
sudo ip addr add 192.168.100.10/24 dev enp1s0f1np1
sudo ip link set enp1s0f1np1 up
sudo ip addr add 192.168.101.10/24 dev enP2p1s0f1np1
sudo ip link set enP2p1s0f1np1 up
Repeat the same process for Node 2, but using IP 192.168.100.11/24 and 192.168.101.11/24. Ensure to use the correct interface name using ibdev2netdev command.
# Assign static IP and bring up interface.
sudo ip addr add 192.168.100.11/24 dev enp1s0f1np1
sudo ip link set enp1s0f1np1 up
sudo ip addr add 192.168.101.11/24 dev enP2p1s0f1np1
sudo ip link set enP2p1s0f1np1 up
You can verify the IP assignment on both nodes by running the following command on each node:
# Check the interfaces showing as "(Up)" in your output, eg. enp1s0f1np1 and enP2p1s0f1np1
ip addr show enp1s0f1np1
ip addr show enP2p1s0f1np1
Set up passwordless SSH authentication
Option 1: Automatically configure SSH
Run the DGX Spark discover-sparks.sh script from one of the nodes to automatically discover and configure SSH:
bash ./discover-sparks
Expected output similar to the below, with different IPs and node names. The first time you run the script, you'll be prompted for your password for each node.
Found: 192.168.100.10 (dgx-spark-1.local)
Found: 192.168.100.11 (dgx-spark-2.local)
Setting up bidirectional SSH access (local <-> remote nodes)...
You may be prompted for your password for each node.
SSH setup complete! Both local and remote nodes can now SSH to each other without passwords.
NOTE
If you encounter any errors, please follow Option 2 below to manually configure SSH and debug the issue.
Option 2: Manually discover and configure SSH
You will need to find the IP addresses for the CX-7 interfaces that are up. On both nodes, run the following command to find the IP addresses and take note of them for the next step. You only need the IP address of one of the interfaces for configuring the SSH keys eg. enp1s0f1np1
ip addr show enp1s0f1np1
Example output:
# In this example, we are using interface enp1s0f1np1.
nvidia@dgx-spark-1:~$ ip addr show enp1s0f1np1
4: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 3c:6d:66:cc:b3:b7 brd ff:ff:ff:ff:ff:ff
inet **192.168.100.10**/24 brd 192.168.100.255 scope global noprefixroute enp1s0f1np1
valid_lft forever preferred_lft forever
inet6 fe80::3e6d:66ff:fecc:b3b7/64 scope link
valid_lft forever preferred_lft forever
In this example, the IP address for Node 1 is 192.168.100.10. Repeat the process for Node 2.
On both nodes, run the following commands to enable passwordless SSH:
# Copy your SSH public key to both nodes. Please replace the IP addresses with the ones you found in the previous step.
ssh-copy-id -i ~/.ssh/id_rsa.pub <username>@<IP for Node 1>
ssh-copy-id -i ~/.ssh/id_rsa.pub <username>@<IP for Node 2>
Verify Multi-Node Communication
Test basic multi-node functionality:
# Test hostname resolution across nodes
ssh <IP for Node 1> hostname
ssh <IP for Node 2> hostname
Cleanup and Rollback
WARNING
These steps will reset network configuration.
# Rollback network configuration (if using Option 1) on both nodes.
sudo rm /etc/netplan/40-cx7.yaml
sudo netplan apply
# Rollback network configuration (if using Option 2) on both nodes.
# Node 1:
sudo ip addr del 192.168.100.10/24 dev enp1s0f1np1 # Adjust the interface name to the one you used in step 3.
sudo ip addr del 192.168.101.10/24 dev enP2p1s0f1np1 # Adjust the interface name to the one you used in step 3.
# Node 2:
sudo ip addr del 192.168.100.11/24 dev enp1s0f1np1 # Adjust the interface name to the one you used in step 3.
sudo ip addr del 192.168.101.11/24 dev enP2p1s0f1np1 # Adjust the interface name to the one you used in step 3.