Use NVIDIA Sync clustering so larger models and training can span nodes.
You can connect the devices directly with the QSFP cables, or you can use a switch and connect each device to the switch with one QSFP cable per device.
In either case, you will plug QSFP cables into the appropriate ports on the devices.
If you are using a switch, do not connect some devices directly and others through a switch.
The procedure changes based on the number of devices and whether you use a switch.
| Devices | Layout | Cables |
|---|---|---|
| Two | Direct | One cable between the devices |
| Three | Direct ring | Three cables; each device links to the other two |
| Two or more | Switch | One cable and one 200 Gbit/s link from each device to the switch |
NVIDIA Sync and the provided helper support up to four devices through a switch. Larger switch clusters need manual configuration or your own automation.
See Plugging in a QSFP Cable for an image.
Viewed from the back of a Spark, port 0 is closest to the ordinary Ethernet port and port 1 is farther away. Use the wiring table for your layout below.
Use the following steps each time you plug in a QSFP cable.
WARNING
Do not force a cable into a port. If it does not slide in, stop and check the pull tab and port alignment.
Connect one device to the other with a single QSFP cable. Using two cables will not increase performance.
Choose one of the port correspondence options below to meet the NVIDIA provided helper's interface requirements used in the Configure Manually tab.
| Port Correspondence | Spark 1 | Spark 2 |
|---|---|---|
| Port 0 on both | Port 0 | Port 0 |
| OR Port 1 on both | Port 1 | Port 1 |
Use three QSFP cables to make a ring. Each Spark will have two QSFP cables plugged into it.
Use the following cabling pattern to meet the NVIDIA provided helper's interface requirements used in the Configure Manually tab.
| Cable | From | To |
|---|---|---|
| 1 | Spark 1, port 0 | Spark 2, port 1 |
| 2 | Spark 2, port 0 | Spark 3, port 1 |
| 3 | Spark 3, port 0 | Spark 1, port 1 |
Set up the switch before you configure the cluster:
NOTE
Changing how a high-speed port is split may restart other links that use the same port group. Set the port mode before you rely on those links.
Connect one cable from each Spark to a separate 200 Gbit/s switch connection. Choose port 0 on every Spark or port 1 on every Spark. The table uses port 1; replace it with port 0 in every row if you choose port 0.
| Device | Spark QSFP port | Switch connection |
|---|---|---|
| Spark 1 | Port 1 | First Spark-facing connection |
| Spark 2 | Port 1 | Second Spark-facing connection |
| Spark 3, if present | Port 1 | Third Spark-facing connection |
| Spark 4, if present | Port 1 | Fourth Spark-facing connection |
| Each additional Spark | Port 1 | Another available 200 Gbit/s connection |
The switch connection names are examples; use the actual port or breakout connection names shown by your switch.
Check the connected devices:
Switch settings and port names differ by maker and model. Follow the switch maker's guide for the exact steps. Do not use commands written for a different switch model.
For MikroTik CRS804 and CRS812 switches, see MikroTik wired interface compatibility and MikroTik bridge configuration.
Continue with tab Configure with NVIDIA Sync for a click-through, managed clustering process.
Or, go under the hood with the Configure Manually tab to use NVIDIA provided helper scripts to set things up.