Use NVIDIA Sync clustering so larger models and training can span nodes.
You will use the NVIDIA provided cluster helper scripts to configure the ConnectX-7 network and SSH access between DGX Sparks.
You will follow the basic procedure and the choices that depend on your layout, so you can generalize the process to a larger cluster.
The helper supports two to four devices. For more than four devices through a switch, use the same procedure with manual configuration or your own automation.
NOTE
The helper changes network and SSH settings on each device. If NVIDIA Sync has already configured your cluster successfully, continue to your workload playbook.
Follow the port wiring table for your layout in Cable the Devices.
The cables (and switch) establish the physical links. In the following steps, you will use the helper scripts to configure the network so the devices are reachable by other software and workloads over those links.
The cabling tab includes the port-to-port connections and switch preparation. Complete those before checking the interfaces in Step 4.
You do not run the helper script from your laptop. You must run it from one of the devices (or "nodes") that you connected above in Step 1.
First, choose one Spark from which to set up the cluster and open a terminal on it.
The helper script does its work no matter which node you pick. It sets up all of the devices (or "nodes") in the cluster from that one node, so that each node has the correct network configuration relative to the other nodes.
The node you pick is configured as part of the cluster too, so at the end of the process all of the nodes are ready to use.
Then clone the NVIDIA playbook repository to that device, open the helper folder and inspect the files.
git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/playbook-connect-multiple-sparks/assets/spark_cluster_setup
If you inspect the individual files, you can piece together the overall process of how the ConnectX-7 network can be configured in more general situations than those shown here.
| Files | Role | Behavior |
|---|---|---|
spark_cluster_setup.sh | Entry point for running checks or setup | You run it with a configuration file and a check or setup option in Steps 4 and 5 |
spark_cluster_setup.py | Coordinates checks, network setup, and SSH access across the devices | Invoked by spark_cluster_setup.sh |
node_scripts/detect_and_configure_cluster_networking.py | Discovers connections and configures network addresses on each device | Copied to each device and invoked by spark_cluster_setup.py during setup |
config/* | JSON files specifying device SSH information on the LAN | Used by spark_cluster_setup.py for sudo privileges and setting up inter-device SSH |
The helper script inspects all devices to verify the cabled layout and coordinate the interfaces, IP address and network plans for implementation on each device.
In addition, inter-device communications need to be set up with key based SSH access to avoid requiring a password.
Finally, implementing a network configuration requires sudo permissions on each device.
So, you need to give the script the IP addresses, usernames and related passwords.
First, copy the appropriate JSON template for your device count.
<cluster-count.json>config/my-cluster.json| Number of nodes | Sample file |
|---|---|
| Two devices, direct or switch | config/spark_config_b2b.json |
| Three devices, ring or switch | config/spark_config_ring.json |
| Four devices through a switch | config/spark_config_switch.json |
Then, edit the copy to include every device to be clustered, including the head node.
vim| Field | Value to supply |
|---|---|
ip_address | The device's existing management IP address on your LAN |
port | Its SSH port, normally 22 |
user | A login account with sudo access |
password | The password for that account |
WARNING
The JSON file stores passwords as plain text. Delete it when setup completes.
The helper uses the login details in your JSON file to check each device before changing the network:
Check LAN SSH and sudo access → Check active ConnectX-7 interfaces → Check link speeds
spark_cluster_setup.py coordinates these checks from the node where you run the helper. Pre-validation does not assign cluster addresses or change SSH settings.
Run the setup script with the pre-validation flag.
bash spark_cluster_setup.sh -c config/my-cluster.json --pre-validate-only
If a check fails, use the error information to troubleshoot the issue before re-running the script.
On failure, the script will output an error with relevant information for you to troubleshoot.
Success: The output will print Pre-setup validations completed successfully.
The helper repeats the checks from Step 4, then runs detect_and_configure_cluster_networking.py on each device. That script discovers the connected peers and turns the chosen addresses into a saved network configuration:
Discover connections → Choose addresses → Build Netplan YAML → Save the file → Apply it
Netplan is how Linux reads and applies the saved interface and address settings. After the network scripts finish, the coordinator checks cluster connectivity and sets up key-based SSH between devices. It uses the LAN connections to coordinate the process throughout.
Back up existing network configuration on every device before applying changes.
The network script writes /etc/netplan/40-cx7.yaml on each device and replaces that file if it already exists. Preserve the existing file and inspect other Netplan files configuring the same interfaces, since Netplan combines their settings. See Inspect and Verify a ConnectX-7 Cluster Network Plan for how to inspect the saved configuration and compare it with the active network.
Run the helper from its own folder:
bash spark_cluster_setup.sh -c config/my-cluster.json --run-setup
The subnet plan follows the layout.
For these configurations, each connected QSFP port needs two subnets, one for each Ethernet interface representing a PCIe path. Corresponding interfaces on connected peers must share the appropriate subnet.
| Layout | Subnet pattern | What changes as you adapt the setup |
|---|---|---|
| Two Sparks, direct | Two subnets across the single cable | Match each subnet between the corresponding interfaces on both Sparks |
| Three Sparks, ring | Two subnets per cable; six across the ring | Match assignments to the actual cable endpoints; each Spark participates in four subnets |
| Sparks through a switch | Two shared subnets across the cluster | Give each additional Spark a unique address in each subnet |
See Understand the ConnectX-7 Network for the hardware explanation, subnet design, and addressing details.
For a larger switch cluster, extend the address plan across all nodes.
Adding a Spark requires another address in each of the two shared subnets, rather than another subnet pair. Choose subnets with enough available addresses for the planned cluster and room for expansion. Keep them separate from your management LAN and other existing networks.
Apply the matching interface and address settings on every node using Netplan or your own automation. Confirm the switch has enough suitable ports and capacity. Then establish passwordless SSH between the accounts that will run your workloads. A common username simplifies configuration; otherwise specify the destination user explicitly.
For helper setup, the scripts create or reuse ~/.ssh/id_ed25519_shared, distribute that shared key, and update SSH configuration. For your own setup, you can instead authorize each node's public key on its peers with ssh-copy-id. Configure access in every required direction.
Verify the resulting network on every node.
Compare the Netplan configuration with the active interfaces and addresses, then test the peers expected for your topology.
| Check | Command | Expected result |
|---|---|---|
| Active links | ip -br link | Connected interfaces show LOWER_UP; unused ports may show NO-CARRIER |
| Assigned addresses | ip -br addr | Both interfaces of each connected port have addresses in their respective planned subnets |
| Route to a peer | ip route get <peer-cluster-IP> | The route selects the intended ConnectX-7 interface |
| Communication on each path | ping -I <interface> -c 3 <peer-cluster-IP> | The corresponding peer responds through that interface |
| Passwordless SSH | ssh -o BatchMode=yes <user>@<peer-cluster-IP> hostname | The expected hostname prints without a password prompt |
Verify and accept each peer's host key during an initial interactive SSH connection before using BatchMode=yes.
Use cluster addresses for these checks. In a ring, choose the peer address on the subnet shared by the source and that peer; not every address on a peer is reachable through every link.
See Inspect and Verify a ConnectX-7 Cluster Network Plan for how to compare configured and active settings and identify the expected peers. For helper setup, inspect 40-cx7.yaml rather than the NVIDIA Sync filename used in that guide.
Success: The helper prints Spark cluster setup completed successfully., and the checks confirm the subnet plan, communication over both interface paths, and passwordless SSH between the required accounts.
After setup works, delete the JSON file containing the passwords:
rm config/my-cluster.json
If you used your own automation, remove any temporary credential files it created.
Open the distributed workload playbook you want to use. It provides the application configuration and performance tests for the network you just established.
Inspect the active configuration before restoring the previous settings.
Follow Inspect and Verify a ConnectX-7 Cluster Network Plan, using the filename your setup created. The helper uses /etc/netplan/40-cx7.yaml; NVIDIA Sync uses /etc/netplan/99-nvidia-sync-cluster.yaml.
Restore a saved configuration if setup replaced an existing file. If the cluster file was newly created, move it out of /etc/netplan, then regenerate and try the restored configuration as described in the documentation. Verify management access and confirm the cluster addresses have been removed.
Network rollback does not remove SSH changes. Remove only the key authorizations and SSH settings added for this cluster. For a switch, also restore any port, bridge or VLAN, DHCP, link speed, or MTU settings you changed.