---
title: "Connect Multiple DGX Sparks for Distributed Workloads — Configure Manually"
canonical: "https://build.nvidia.com/playbooks/connect-multiple-sparks/advance.md"
---

# Configure a DGX Spark cluster manually

You will use the NVIDIA provided cluster helper scripts to configure the ConnectX-7 network and SSH access between DGX Sparks.

You will follow the basic procedure and the choices that depend on your layout, so you can generalize the process to a larger cluster.

The helper supports two to four devices. For more than four devices through a switch, use the same procedure with manual configuration or your own automation.

> [!NOTE]
> The helper changes network and SSH settings on each device. If NVIDIA Sync has already configured your cluster successfully, continue to your workload playbook.

# Step 1. Physically connect the devices

**Follow the port wiring table for your layout in Cable the Devices.**

The cables (and switch) establish the physical links. 
In the following steps, you will use the helper scripts to configure the network so the devices are reachable by other software and workloads over those links.

The cabling tab includes the port-to-port connections and switch preparation. 
Complete those before checking the interfaces in Step 4.

# Step 2. Pick a device in the cluster and download the helper scripts to it

You do **not** run the helper script from your laptop.
You **must** run it from one of the devices (or "nodes") that you connected above in Step 1. 

**First, choose one Spark from which to set up the cluster and open a terminal on it.**

The helper script does its work no matter which node you pick. It sets up all of the devices (or "nodes") in the cluster from that one node, so that each node has the correct network configuration relative to the other nodes.

The node you pick is configured as part of the cluster too, so at the end of the process all of the nodes are ready to use.

**Then clone the NVIDIA playbook repository to that device, open the helper folder and inspect the files.**

```bash
git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/playbook-connect-multiple-sparks/assets/spark_cluster_setup
```

If you inspect the individual files, you can piece together the overall process of how the ConnectX-7 network can be configured in more general situations than those shown here.

| Files | Role | Behavior |
| --- | --- | --- |
| `spark_cluster_setup.sh` | Entry point for running checks or setup | You run it with a configuration file and a check or setup option in Steps 4 and 5 |
| `spark_cluster_setup.py` | Coordinates checks, network setup, and SSH access across the devices | Invoked by `spark_cluster_setup.sh` |
| `node_scripts/detect_and_configure_cluster_networking.py` | Discovers connections and configures network addresses on each device | Copied to each device and invoked by `spark_cluster_setup.py` during setup |
| `config/*` | JSON files specifying device SSH information on the LAN | Used by `spark_cluster_setup.py` for `sudo` privileges and setting up inter-device SSH |

# Step 3. Edit the appropriate JSON file to enable inter-device SSH 

The helper script inspects all devices to verify the cabled layout and coordinate the interfaces, IP address and network plans for implementation on each device. 
In addition, inter-device communications need to be set up with key based SSH access to avoid requiring a password.
Finally, implementing a network configuration requires `sudo` permissions on each device. 

So, you need to give the script the IP addresses, usernames and related passwords.   

**First, copy the appropriate JSON template for your device count.**

1. Select the appropriate JSON template from the table below, `<cluster-count.json>`
2. Copy the template to a new file, `config/my-cluster.json`
3. Change the permissions on the file so it's visible only to the user on the head node: chmod 600 config/my-cluster.json

| Number of nodes | Sample file |
| --- | --- |
| Two devices, direct or switch | `config/spark_config_b2b.json` |
| Three devices, ring or switch | `config/spark_config_ring.json` |
| Four devices through a switch | `config/spark_config_switch.json` |

**Then, edit the copy to include every device to be clustered, including the head node.**

1. Open the file with a file editor like `vim`
2. Edit the individual entries with the appropriate information from the table below
3. Make sure to have correct entries for each device, including the head node
4. Finally, save the file

| Field | Value to supply |
| --- | --- |
| `ip_address` | The device's existing management IP address on your LAN |
| `port` | Its SSH port, normally `22` |
| `user` | A login account with sudo access |
| `password` | The password for that account |

> [!WARNING]
> The JSON file stores passwords as plain text. Delete it when setup completes.

# Step 4. Run the setup script to inspect the ConnectX-7 interfaces, links and link speeds. 

The helper uses the login details in your JSON file to check each device before changing the network:

```text
Check LAN SSH and sudo access → Check active ConnectX-7 interfaces → Check link speeds
```

`spark_cluster_setup.py` coordinates these checks from the node where you run the helper. Pre-validation does not assign cluster addresses or change SSH settings.

**Run the setup script with the pre-validation flag.** 

```bash
bash spark_cluster_setup.sh -c config/my-cluster.json --pre-validate-only
```

**If a check fails, use the error information to troubleshoot the issue before re-running the script.**

On failure, the script will output an error with relevant information for you to troubleshoot.

**Success:** The output will print `Pre-setup validations completed successfully.`

# Step 5. Back up the existing network implementation and create a new one to include the ConnectX-7 interfaces

The helper repeats the checks from Step 4, then runs `detect_and_configure_cluster_networking.py` on each device. That script discovers the connected peers and turns the chosen addresses into a saved network configuration:

```text
Discover connections → Choose addresses → Build Netplan YAML → Save the file → Apply it
```

Netplan is how Linux reads and applies the saved interface and address settings. After the network scripts finish, the coordinator checks cluster connectivity and sets up key-based SSH between devices. It uses the LAN connections to coordinate the process throughout.

**Back up existing network configuration on every device before applying changes.**

The network script writes `/etc/netplan/40-cx7.yaml` on each device and replaces that file if it already exists. Preserve the existing file and inspect other Netplan files configuring the same interfaces, since Netplan combines their settings. See [Inspect and Verify a ConnectX-7 Cluster Network Plan](https://docs.nvidia.com/sync/latest/cluster-network-inspection.html) for how to inspect the saved configuration and compare it with the active network.

**Run the helper from its own folder:**

```bash
bash spark_cluster_setup.sh -c config/my-cluster.json --run-setup
```

**The subnet plan follows the layout.**

For these configurations, each connected QSFP port needs two subnets, one for each Ethernet interface representing a PCIe path. Corresponding interfaces on connected peers must share the appropriate subnet.

| Layout | Subnet pattern | What changes as you adapt the setup |
| --- | --- | --- |
| Two Sparks, direct | Two subnets across the single cable | Match each subnet between the corresponding interfaces on both Sparks |
| Three Sparks, ring | Two subnets per cable; six across the ring | Match assignments to the actual cable endpoints; each Spark participates in four subnets |
| Sparks through a switch | Two shared subnets across the cluster | Give each additional Spark a unique address in each subnet |

See [Understand the ConnectX-7 Network](https://docs.nvidia.com/dgx/dgx-spark/spark-clustering.html#understand-the-connectx-7-network) for the hardware explanation, subnet design, and addressing details.

**For a larger switch cluster, extend the address plan across all nodes.**

Adding a Spark requires another address in each of the two shared subnets, rather than another subnet pair. Choose subnets with enough available addresses for the planned cluster and room for expansion. Keep them separate from your management LAN and other existing networks.

Apply the matching interface and address settings on every node using Netplan or your own automation. Confirm the switch has enough suitable ports and capacity. Then establish passwordless SSH between the accounts that will run your workloads. A common username simplifies configuration; otherwise specify the destination user explicitly.

For helper setup, the scripts create or reuse `~/.ssh/id_ed25519_shared`, distribute that shared key, and update SSH configuration. For your own setup, you can instead authorize each node's public key on its peers with `ssh-copy-id`. Configure access in every required direction.

**Verify the resulting network on every node.**

Compare the Netplan configuration with the active interfaces and addresses, then test the peers expected for your topology.

| Check | Command | Expected result |
| --- | --- | --- |
| Active links | `ip -br link` | Connected interfaces show `LOWER_UP`; unused ports may show `NO-CARRIER` |
| Assigned addresses | `ip -br addr` | Both interfaces of each connected port have addresses in their respective planned subnets |
| Route to a peer | `ip route get <peer-cluster-IP>` | The route selects the intended ConnectX-7 interface |
| Communication on each path | `ping -I <interface> -c 3 <peer-cluster-IP>` | The corresponding peer responds through that interface |
| Passwordless SSH | `ssh -o BatchMode=yes <user>@<peer-cluster-IP> hostname` | The expected hostname prints without a password prompt |

Verify and accept each peer's host key during an initial interactive SSH connection before using `BatchMode=yes`.

Use cluster addresses for these checks. In a ring, choose the peer address on the subnet shared by the source and that peer; not every address on a peer is reachable through every link.

See [Inspect and Verify a ConnectX-7 Cluster Network Plan](https://docs.nvidia.com/sync/latest/cluster-network-inspection.html) for how to compare configured and active settings and identify the expected peers. For helper setup, inspect `40-cx7.yaml` rather than the NVIDIA Sync filename used in that guide.

**Success:** The helper prints `Spark cluster setup completed successfully.`, and the checks confirm the subnet plan, communication over both interface paths, and passwordless SSH between the required accounts.

# Step 6. Remove the password file

**After setup works, delete the JSON file containing the passwords:**

```bash
rm config/my-cluster.json
```

If you used your own automation, remove any temporary credential files it created.

# Next steps

Open the distributed workload playbook you want to use. It provides the application configuration and performance tests for the network you just established.

# Roll back the manual setup

**Inspect the active configuration before restoring the previous settings.**

Follow [Inspect and Verify a ConnectX-7 Cluster Network Plan](https://docs.nvidia.com/sync/latest/cluster-network-inspection.html#remove-the-connectx-7-netplan-configuration-on-an-individual-node), using the filename your setup created. The helper uses `/etc/netplan/40-cx7.yaml`; NVIDIA Sync uses `/etc/netplan/99-nvidia-sync-cluster.yaml`.

Restore a saved configuration if setup replaced an existing file. If the cluster file was newly created, move it out of `/etc/netplan`, then regenerate and try the restored configuration as described in the documentation. Verify management access and confirm the cluster addresses have been removed.

Network rollback does not remove SSH changes. Remove only the key authorizations and SSH settings added for this cluster. For a switch, also restore any port, bridge or VLAN, DHCP, link speed, or MTU settings you changed.