Connect Multiple DGX Sparks for Distributed Workloads

Configure a DGX Spark cluster manually

You will use the setup-cluster.sh helper script from the spark-vllm-docker repository to configure the ConnectX-7 network and SSH access between DGX Sparks.

The same script covers every layout in Cable the Devices: two devices with a direct link, three devices in a ring, four devices in a ring, and devices through a switch.

The full reference for the script is Automated cluster setup in that repository.

Physically connect the devices

Follow the port wiring table for your layout in Cable the Devices.

The cables (and switch) establish the physical links. In the following steps, you will use the script to configure the network so the devices are reachable by other software and workloads over those links.

The cabling tab includes the port-to-port connections and switch preparation. Complete those before you run the script. The script does not configure the switch.

Check the requirements on every device

The script connects to every device over SSH on your management LAN. Each device must meet these requirements:

You do not need passwordless SSH before you start. The script asks for the account password when it first needs it. The script keeps the password in memory only. It does not write the password to a file.

The script does not install packages. If a package is missing, install it before you continue.

Pick a device in the cluster and download the script to it

You do not run the script from your laptop. You must run it from one of the devices (or "nodes") that you connected in Step 1.

First, choose one Spark to be the head node and open a terminal on it.

The script sets up all of the nodes from the head node. The head node is configured as part of the cluster too, so at the end of the process all of the nodes are ready to use. For a ring, use Spark 1 from the wiring table as the head node.

Run the script as your normal login user. Do not run it with sudo.

Then clone the repository to the head node and open its folder.

git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker

Only the head node needs the repository. The script sends its node helper to the other devices over SSH.

Preview the setup with a dry run

Find the management IP address of each device.

List the addresses with the head node first. For a ring, list the devices in cable order: Spark 1, Spark 2, Spark 3, and then Spark 4 if present. In a ring, port 0 of each device connects to port 1 of the next device. This matches the wiring table in Cable the Devices.

Run the command for your layout with --dry-run.

Replace the example addresses with your management IP addresses.

Use --mtu 9000 only for two Sparks with a direct link. Use --mtu 1500 for all other layouts. These values match the MTU that NVIDIA Sync Cluster Assistant sets. If you leave out --mtu, the script uses 9000 for every layout.

The script uses the name mesh for the three-device ring, because three devices in a ring are also a full mesh. The script uses the name ring for four devices in a ring.

If you list the ring devices in the wrong order, the verification step fails. If LLDP is already running on the devices, the script compares your order with the LLDP neighbors and rejects an order that does not match.

If you leave out --topology, the script tries to detect the layout from the active ports. With one active port on each device, the script selects direct for two devices and switch for three or more. The script cannot tell a two-device switch from a direct link, so give --topology for every layout.

The dry run checks SSH and sudo access, inspects the ConnectX-7 interfaces, and validates the planned Netplan configuration in a temporary folder. The dry run does not change the network or SSH settings.

If a check fails, use the error information to fix the issue before you run the script again.

Success: The output shows the planned interface addresses for each device and ends with Dry run complete; no persistent configuration was changed.

Run the setup

Run the same command without --dry-run.

For example, for two Sparks with a direct link:

./setup-cluster.sh 192.168.1.11 192.168.1.12 --topology direct --mtu 9000

The script prints the plan again and asks Apply this cluster setup plan? [y/N]. Review the plan, then type y.

If the script shows an unknown host key fingerprint, verify the fingerprint before you accept it.

netplan apply can briefly interrupt networking. Run the setup when the cluster is idle.

The script makes these changes on every device:

Back up files → Create SSH keys → Add Docker group membership → Apply Netplan → Verify links and SSH → Save .env on the head node

The script also moves existing ConnectX-7 entries out of other files in /etc/netplan into its own file, so Netplan does not combine old addresses with the new ones. The script backs up those files first.

If setup fails or you press Ctrl-C, the script tries to roll back the changes on every device.

The script assigns /24 subnets from 10.20.0.0/16 and gives each device the host numbers .11, .12, and so on, in the order you listed the devices. We recommend /24 subnets for the ConnectX-7 network. NCCL uses /24 as the default subnet size for subnet-aware routing. If you plan the network yourself, also use /24 subnets. Each QSFP port has two Ethernet interfaces, one for each PCIe path, so each link uses two subnets.

See Understand the ConnectX-7 Network for the hardware explanation, subnet design, and addressing details.

For other options, such as another subnet range or port selection, see Automated cluster setup.

Success: The script prints Cluster networking and mutual passwordless SSH are configured and verified.

Before it prints this message, the script checks ping on both interface paths at the full MTU, in both directions. The script also checks passwordless SSH from every device to every other device.

Log in again and verify the result

Open a new login session on each device.

The docker group change applies only to new login sessions. Existing SSH sessions keep their old groups.

Run the doctor check from the head node.

./setup-cluster.sh --doctor --dry-run

The doctor check confirms the saved files, ConnectX-7 addresses, routes, MTU, links, passwordless SSH, the launch configuration, and Docker access. The command returns a nonzero exit status if any issue remains.

Success: The output ends with Doctor: all saved configuration, CX7 links, SSH, and Docker checks passed.

To repair the issues that the doctor check finds, run ./setup-cluster.sh --doctor without --dry-run. The script lists the repairs and asks before it applies them.

Next steps

Open the distributed workload playbook you want to use. It provides the application configuration and performance tests for the network you just established.

Roll back the setup

Run the restore command on the same head node, as the same user.

./setup-cluster.sh --restore

Restore uses the manifest on the head node and the backups on each device. Keep these files until restore is complete. Every device must be reachable over the management LAN.

Restore puts back the original Netplan, SSH, and .env files with their original permissions. Restore also removes the SSH keys and files that the script created, and reapplies the previous network configuration. Restore removes the docker group membership only if the script added it.

If you edited a file that the script manages after setup, restore stops and names that file. Reconcile your edits, then run restore again.

To change the layout, the head node, or the addresses, run restore first. Then run the setup again.

For a switch, also restore any port, bridge or VLAN, DHCP, link speed, or MTU settings you changed on the switch.