---
title: "Serve LLMs with vLLM Across DGX Sparks — Intermediate"
canonical: "https://build.nvidia.com/spark/vllm-cluster/intermediate.md"
---

# Run a vLLM model across two DGX Sparks

You will use NVIDIA Sync to set up a two-Spark cluster, then use a supplied recipe to run one model across both devices.

The recipe launcher starts a container on each Spark and runs vLLM across the pair.
You will send requests to the head Spark and use an SSH tunnel to reach its API from your laptop.

## Step 1. Connect and set up the two Sparks

**First, connect two Sparks with the same memory size.**

Use two 64GB Sparks or two 128GB Sparks.
Each recipe below is set for one of those pairs.

Follow the [Connect Multiple Sparks playbook](https://build.nvidia.com/playbooks/connect-multiple-sparks) to cable the devices and set up the cluster with NVIDIA Sync.
Add both Sparks to Sync and allow it to set up the Docker group if prompted.

**Then, save the cluster's network details.**

Keep the head and worker addresses, SSH aliases, and ConnectX interface names from the cluster setup.
The launcher will use the ConnectX link to pass model data between the two Sparks.

**Success:** Both Sparks are in the cluster, and you can open a terminal on each through Sync.

## Step 2. Set up the launcher on the head Spark

**First, choose one Spark as the head and open its terminal through Sync.**

The head runs the API and starts the model on both devices.
Use this Spark for the rest of the commands unless a step says otherwise.

**Then, install `uv` and get the launcher.**

Run:

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
git clone https://github.com/eugr/spark-vllm-docker.git
cd spark-vllm-docker
```

These are [community scripts](https://github.com/eugr/spark-vllm-docker) maintained outside NVIDIA.
The commands below use `uv` to run the recipe tool with its Python dependency.

**Next, find the cluster and check the saved settings.**

```bash
uv run --with pyyaml python run-recipe.py --discover
uv run --with pyyaml python run-recipe.py --show-env
```

Check that the saved `.env` file names exactly two Sparks, with this Spark first as the head.
Check the addresses and ConnectX interfaces against the details from Step 1.
If the launcher cannot reach the worker through SSH, fix that connection before you go on.

**Success:** The launcher has saved the intended head, worker, and network settings in `.env`.

## Step 3. Choose a model and save its recipe

**First, choose the row for your pair of Sparks.**

Memory in the table is **per Spark**.
These models use both devices to serve one endpoint.

| Cluster | Model | Recipe file |
| --- | --- | --- |
| 2 × DGX Spark, 64GB each | [Qwen3.8 Flash-Next](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=qwen%2Fqwen3.8-flash-next&os=linux&backend=vllm&memory=64&precision=fp4-local&memoryConfig=64) | [qwen38-flash-next-2x64gb.yaml](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm-cluster/assets/qwen38-flash-next-2x64gb.yaml) |
| 2 × DGX Spark, 128GB each | [GLM 5.3 Flash](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128) | [glm53-flash-2x128gb.yaml](https://raw.githubusercontent.com/NVIDIA/dgx-spark-playbooks/refs/heads/main/nvidia/playbook-vllm-cluster/assets/glm53-flash-2x128gb.yaml) |

**Then, save the recipe on the head Spark.**

1. Open the recipe file from your row
2. Copy its full contents into a file in the launcher's `recipes` folder on the head Spark
3. Use the file name shown in the table

For example, open this file in a text editor on the head Spark for the 64GB pair:

```bash
nano recipes/qwen38-flash-next-2x64gb.yaml
```

For the 128GB pair, use `recipes/glm53-flash-2x128gb.yaml` instead.
The supplied files keep the image, model, and serving settings from the matching NVIDIA recipe together.

**Success:** Your chosen recipe is saved in the head Spark's `recipes` folder.

## Step 4. Download the image and launch the model

**First, download the container image on the head Spark.**

Both recipes use this image:

```bash
docker pull eugr/spark-vllm-b12x:nightly-20261015
```

The recipe uses this tag so the model runs with the version of the backend it calls for.

**Then, run the command for your pair.**

For two 64GB Sparks:

```bash
uv run --with pyyaml python run-recipe.py recipes/qwen38-flash-next-2x64gb.yaml --setup
```

For two 128GB Sparks:

```bash
uv run --with pyyaml python run-recipe.py recipes/glm53-flash-2x128gb.yaml --setup
```

The setup step copies the prepared image and model files to the worker, then starts vLLM on both devices.
Allow time and free disk space on both Sparks for the image and model files.
If a model download asks for access, sign in on the head with `uvx hf auth login`, then run the command again.

**Next, keep the launch terminal open while the model runs.**

The first start can take a while as the model loads and the backend builds runtime files.
Wait for the head's logs to show that the API server has started.

**Success:** vLLM is running on both Sparks, and the head has started its API on port `8000`.

## Step 5. Check the API on the head Spark

**First, open a second terminal on the head through Sync.**

Leave the launch terminal open and run:

```bash
curl --fail-with-body -i http://127.0.0.1:8000/health
curl --fail-with-body -sS http://127.0.0.1:8000/v1/models
```

The health check should return HTTP `200`.
Copy the model ID from the model-list reply.
If the server is still loading, wait and try again.

**Then, send a chat request.**

Replace `<model-id>` with the ID from the model list:

```bash
curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'
```

These models may use some output tokens for reasoning before the final answer.
The larger output limit gives that work room to finish.

**Success:** The API returns a chat response with a final answer.

## Step 6. Reach the model from your laptop

**First, open a terminal on your laptop and start an SSH tunnel.**

Use the head Spark's management address or an SSH alias that works from your laptop.
Replace `<user>` and `<head-host>` below.
This command works in Windows PowerShell, macOS, and Linux terminals with OpenSSH installed:

```text
ssh -N -L 8000:127.0.0.1:8000 <user>@<head-host>
```

Keep this terminal open.
The tunnel sends requests from port `8000` on your laptop to port `8000` on the head Spark.
Sync's cluster setup does not create this API tunnel.

If port `8000` is in use on your laptop, change the first `8000` to `8001` and use `8001` in your laptop's API URLs.

**Then, check the API from a second laptop terminal.**

| Laptop terminal | Health check | List served models |
| --- | --- | --- |
| Windows PowerShell | `curl.exe --fail-with-body -i http://localhost:8000/health` | `curl.exe --fail-with-body -sS http://localhost:8000/v1/models` |
| macOS or Linux | `curl --fail-with-body -i http://localhost:8000/health` | `curl --fail-with-body -sS http://localhost:8000/v1/models` |

For an OpenAI-compatible client on your laptop, use `http://localhost:8000/v1` as the base URL and the model ID from the list.
Both recipes support automatic tool calling; see the Advanced Tab for a test request.

**Success:** Your laptop can reach the model through the head Spark.

## Step 7. Stop and restart the model

**First, stop the model when you are done.**

Press `Ctrl+C` in the launch terminal on the head Spark.
The foreground launcher stops the containers on both Sparks when it exits.

You can also stop them from a second terminal on the head:

```bash
cd ~/spark-vllm-docker
./launch-cluster.sh stop
./launch-cluster.sh status
```

Close the laptop tunnel with `Ctrl+C` when you no longer need it.
The cluster's network settings and downloaded files stay in place.

**Then, restart with the same recipe when you need it again.**

From the repository on the head, run your Step 4 recipe command without `--setup`.
Keep that terminal open and start the laptop tunnel again if you need it.
Use `--setup` again if you need to restore missing model files or copy the image to the worker.
If the image is missing on the head, first run the `docker pull` command from Step 4 again.

**Success:** You can start and stop the model on both Sparks from the head.