Serve LLMs with vLLM Across DGX Sparks

Run a vLLM model across two DGX Sparks

You will use NVIDIA Sync to set up a two-Spark cluster, then use a supplied recipe to run one model across both devices.

The recipe launcher starts a container on each Spark and runs vLLM across the pair. You will send requests to the head Spark and use an SSH tunnel to reach its API from your laptop.

Connect and set up the two Sparks

First, connect two Sparks with the same memory size.

Use two 64GB Sparks or two 128GB Sparks. Each recipe below is set for one of those pairs.

Follow the Connect Multiple Sparks playbook to cable the devices and set up the cluster with NVIDIA Sync. Add both Sparks to Sync and allow it to set up the Docker group if prompted.

Then, save the cluster's network details.

Keep the head and worker addresses, SSH aliases, and ConnectX interface names from the cluster setup. The launcher will use the ConnectX link to pass model data between the two Sparks.

Success: Both Sparks are in the cluster, and you can open a terminal on each through Sync.

Set up the launcher on the head Spark

First, choose one Spark as the head and open its terminal through Sync.

The head runs the API and starts the model on both devices. Use this Spark for the rest of the commands unless a step says otherwise.

Then, install uv and get the launcher.

Run:

curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker

These are community scripts maintained outside NVIDIA. The commands below use uv to run the recipe tool with its Python dependency.

Next, find the cluster and check the saved settings.

uv run --with pyyaml python run-recipe.py --discover
uv run --with pyyaml python run-recipe.py --show-env

Check that the saved .env file names exactly two Sparks, with this Spark first as the head. Check the addresses and ConnectX interfaces against the details from Step 1. If the launcher cannot reach the worker through SSH, fix that connection before you go on.

Success: The launcher has saved the intended head, worker, and network settings in .env.

Choose a model and save its recipe

First, choose the row for your pair of Sparks.

Memory in the table is per Spark. These models use both devices to serve one endpoint.

Then, save the recipe on the head Spark.

1. Open the recipe file from your row
2. Copy its full contents into a file in the launcher's recipes folder on the head Spark
3. Use the file name shown in the table

For example, open this file in a text editor on the head Spark for the 64GB pair:

nano recipes/qwen38-flash-next-2x64gb.yaml

For the 128GB pair, use recipes/glm53-flash-2x128gb.yaml instead. The supplied files keep the image, model, and serving settings from the matching NVIDIA recipe together.

Success: Your chosen recipe is saved in the head Spark's recipes folder.

Download the image and launch the model

First, download the container image on the head Spark.

Both recipes use this image:

docker pull eugr/spark-vllm-b12x:nightly-20261015

The recipe uses this tag so the model runs with the version of the backend it calls for.

Then, run the command for your pair.

For two 64GB Sparks:

uv run --with pyyaml python run-recipe.py recipes/qwen38-flash-next-2x64gb.yaml --setup

For two 128GB Sparks:

uv run --with pyyaml python run-recipe.py recipes/glm53-flash-2x128gb.yaml --setup

The setup step copies the prepared image and model files to the worker, then starts vLLM on both devices. Allow time and free disk space on both Sparks for the image and model files. If a model download asks for access, sign in on the head with uvx hf auth login, then run the command again.

Next, keep the launch terminal open while the model runs.

The first start can take a while as the model loads and the backend builds runtime files. Wait for the head's logs to show that the API server has started.

Success: vLLM is running on both Sparks, and the head has started its API on port 8000.

Check the API on the head Spark

First, open a second terminal on the head through Sync.

Leave the launch terminal open and run:

curl --fail-with-body -i http://127.0.0.1:8000/health
curl --fail-with-body -sS http://127.0.0.1:8000/v1/models

The health check should return HTTP 200. Copy the model ID from the model-list reply. If the server is still loading, wait and try again.

Then, send a chat request.

Replace <model-id> with the ID from the model list:

curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'

These models may use some output tokens for reasoning before the final answer. The larger output limit gives that work room to finish.

Success: The API returns a chat response with a final answer.

Reach the model from your laptop

First, open a terminal on your laptop and start an SSH tunnel.

Use the head Spark's management address or an SSH alias that works from your laptop. Replace <user> and <head-host> below. This command works in Windows PowerShell, macOS, and Linux terminals with OpenSSH installed:

ssh -N -L 8000:127.0.0.1:8000 <user>@<head-host>

Keep this terminal open. The tunnel sends requests from port 8000 on your laptop to port 8000 on the head Spark. Sync's cluster setup does not create this API tunnel.

If port 8000 is in use on your laptop, change the first 8000 to 8001 and use 8001 in your laptop's API URLs.

Then, check the API from a second laptop terminal.

For an OpenAI-compatible client on your laptop, use http://localhost:8000/v1 as the base URL and the model ID from the list. Both recipes support automatic tool calling; see the Advanced Tab for a test request.

Success: Your laptop can reach the model through the head Spark.

Stop and restart the model

First, stop the model when you are done.

Press Ctrl+C in the launch terminal on the head Spark. The foreground launcher stops the containers on both Sparks when it exits.

You can also stop them from a second terminal on the head:

cd ~/spark-vllm-docker
./launch-cluster.sh stop
./launch-cluster.sh status

Close the laptop tunnel with Ctrl+C when you no longer need it. The cluster's network settings and downloaded files stay in place.

Then, restart with the same recipe when you need it again.

From the repository on the head, run your Step 4 recipe command without --setup. Keep that terminal open and start the laptop tunnel again if you need it. Use --setup again if you need to restore missing model files or copy the image to the worker. If the image is missing on the head, first run the docker pull command from Step 4 again.

Success: You can start and stop the model on both Sparks from the head.