---
title: "Serve LLMs with vLLM Across DGX Sparks — Advanced"
canonical: "https://build.nvidia.com/spark/vllm-cluster/advanced.md"
---

# Run and tune a vLLM recipe across two DGX Sparks

You will run one vLLM service across two Sparks, using the Docker and server commands from an NVIDIA Local AI Inference Recipe.

This path uses SSH and a container on each node.
It covers the settings that split the model across the pair, move data between nodes, and set the capacity of the service.

## Step 1. Choose a model and prepare the cluster

**First, choose the recipe for your pair of Sparks.**

Memory in this table is per device.
Use two Sparks of the same memory size for these configurations.

| Cluster | Model and recipe | Checkpoint |
| --- | --- | --- |
| 2 × DGX Spark, 64GB each | [Qwen3.8 Flash-Next](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=qwen%2Fqwen3.8-flash-next&os=linux&backend=vllm&memory=64&precision=fp4-local&memoryConfig=64) | `nvidia/Qwen3.8-Flash-Next-FP4-Local` |
| 2 × DGX Spark, 128GB each | [GLM 5.3 Flash](https://nvidia.github.io/local-ai-inference-recipes/hardware/dgx-spark-2gpu/index.html?model=z-ai%2Fglm-5.3-flash&os=linux&backend=vllm&memory=128&precision=fp4-local&memoryConfig=128) | `nvidia/GLM-5.3-Flash-FP4-Local` |

Both recipes use `eugr/spark-vllm-b12x:nightly-20261015`, a community image with the B12X backend.
Keep the image, checkpoint, environment settings, and server flags from the same recipe.
The 64GB Qwen configuration has its own memory settings; the settings for a pair of 128GB Sparks are different.

**Then, cable and configure the two nodes.**

Use **Configure Manually** in the [Connect Multiple Sparks playbook](https://build.nvidia.com/playbooks/connect-multiple-sparks) if the cluster is not already set up.
Confirm interdevice SSH and Docker access on both nodes.
Choose a head Spark and a worker Spark, then open an SSH session to each.

Keep these values from the network setup:

| Value | Use |
| --- | --- |
| Head's ConnectX IP address | Rendezvous address shared by both vLLM processes |
| Each node's ConnectX Ethernet interface | Socket and control traffic on that node |
| Each node's RoCE HCA names | GPU data transfers over the cluster link |
| Head's management address or SSH alias | Laptop access to the API |

**Success:** You have a complete recipe and can reach both nodes over the cluster link.

## Step 2. Read the distributed and memory settings

**First, check how the recipe splits work across the pair.**

Each Spark runs one tensor-parallel rank.
The ranks hold parts of the model and exchange data during inference.
The head exposes the API; the worker joins the same service.
This gives you one model endpoint across two GPUs.

| Setting | Why it is in the command |
| --- | --- |
| `--tensor-parallel-size 2` | Splits supported model tensors across the two GPUs. Communication uses the link between the Sparks. |
| `--nnodes 2` | Sets the number of nodes in the distributed executor. |
| `--node-rank 0` or `1` | Gives each node a unique rank. The head is rank `0`. |
| `--master-addr` and `--master-port` | Set the shared rendezvous address and port. Use the head's ConnectX address on both nodes. |
| `--headless` | Runs the worker without an API server. |
| `--pipeline-parallel-size 1` | GLM keeps one pipeline stage; its model split uses tensor parallelism. |

These commands use native multi-node execution.
They do not need a separate Ray cluster.
API port `8000` and rendezvous port `29501` serve different roles.
See the [vLLM multi-node multiprocessing guide](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/#running-vllm-with-multiprocessing) for the rank and rendezvous settings.

**Next, check the memory and scheduling limits.**

Each node must fit its share of the weights, caches, and runtime work.
The pair does not act as one flat pool of memory.

| Setting | Why it is in the recipe |
| --- | --- |
| `--quantization modelopt_mixed` and `--load-format b12x` | Select the weight format and loader supported by the checkpoint and image. |
| `--kv-cache-dtype fp8` | Sets KV-cache precision apart from weight precision. |
| `--gpu-memory-utilization` | Sets the fraction of GPU memory vLLM budgets for its executor on each node. |
| `--kv-cache-memory-bytes 3G` | Gives Qwen an explicit KV-cache budget per GPU. This takes precedence over the utilization setting for KV-cache sizing. |
| `--max-model-len` | Caps the total input and output tokens in one request. |
| `--max-num-seqs` | Caps the number of sequences processed in one iteration. |
| `--max-num-batched-tokens` | Sets the token budget for one scheduler iteration. |
| `--enable-chunked-prefill` | Lets long prompt processing share batches with decode work. |
| `--enable-prefix-caching` | Reuses cached prompt work when requests share a prefix. |

The context limit is a per-request ceiling.
It does not mean every active request can use that full length at once.
Actual capacity also depends on the model's cache layout and the other requests in flight.

**Then, keep the model-specific backend settings together for the first run.**

The B12X kernels, compilation settings, and environment variables are part of these recipes.
For example, the 64GB Qwen recipe uses `VLLM_PLE_TABLE_MEMORY=disk`, limits compile workers, and sets a `3G` KV-cache budget.
GLM uses a different attention backend, block size, and memory budget.

Both recipes use native multi-token prediction (MTP) for speculative decoding.
They do not call for a separate draft checkpoint.
Keep the reasoning and tool-call parsers if the service will be used by an agent.

See the [vLLM serve options](https://docs.vllm.ai/en/latest/cli/serve/), [memory and scheduling guide](https://docs.vllm.ai/en/latest/configuration/optimization/), and [speculative decoding guide](https://docs.vllm.ai/en/latest/features/speculative_decoding/) for the core settings.
The community image supplies the B12X-specific code paths used here.

**Success:** You know which settings define the cluster, which set its capacity, and which belong to the chosen model and backend.

## Step 3. Start a container on each Spark

**First, set the network values in each node's host shell.**

Use the same head address and rendezvous port on both nodes.
Set `ETH_IF` and `IB_IF` to the local node's interfaces; their names may differ between nodes.
Keep both RoCE HCA names called for by the recipe.

```bash
export HEAD_IP="<head-connectx-ip>"
export ETH_IF="<local-connectx-interface>"
export IB_IF="<local-roce-hca-1>,<local-roce-hca-2>"
export MASTER_PORT=29501
export IMAGE="eugr/spark-vllm-b12x:nightly-20261015"
export HF_CACHE_DIR="$HOME/.cache/huggingface"
```

**Then, run this Docker command on each node.**

If checkpoint access needs a Hugging Face token, set `HF_TOKEN` in the host shell first.
The container downloads the image if needed, and vLLM downloads the model when it starts.
Both nodes need the model files and enough free disk space for their local caches.

```bash
mkdir -p "$HF_CACHE_DIR" "$HOME/.cache/vllm" \
"$HOME/.cache/flashinfer" "$HOME/.triton" "$HOME/.cache/b12x"

docker run --rm -it \
--name vllm_cluster \
--privileged --gpus all --ipc host --network host \
--shm-size=16g \
--ulimit nofile=1048576:1048576 \
--ulimit memlock=-1 --ulimit stack=67108864 \
-e HF_TOKEN="${HF_TOKEN:-}" \
-e MN_IF_NAME="$ETH_IF" -e UCX_NET_DEVICES="$ETH_IF" \
-e NCCL_SOCKET_IFNAME="$ETH_IF" -e NCCL_IB_HCA="$IB_IF" \
-e NCCL_IB_DISABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
-e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
-e GLOO_SOCKET_IFNAME="$ETH_IF" -e TP_SOCKET_IFNAME="$ETH_IF" \
-e HEAD_IP="$HEAD_IP" -e MASTER_PORT="$MASTER_PORT" \
-v "$HF_CACHE_DIR:/root/.cache/huggingface" \
-v "$HOME/.cache/vllm:/root/.cache/vllm" \
-v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \
-v "$HOME/.triton:/root/.triton" \
-v "$HOME/.cache/b12x:/root/.cache/b12x" \
--entrypoint /bin/bash "$IMAGE"
```

Host networking and the interface variables let the processes use the ConnectX link.
The mounts keep model and runtime files across container restarts.
The API will bind to the head's host interfaces, so use the service on your trusted network.

**Success:** You have an open Bash shell inside the container on each node.

## Step 4. Run the model on the head and worker

**First, set the rank arguments inside each container.**

On the head, run:

```bash
NODE_ARGS=(--node-rank 0)
```

On the worker, run:

```bash
NODE_ARGS=(--node-rank 1 --headless)
```

**Then, run the full command for your model in both containers.**

Use the same model, environment settings, and serving flags on both nodes.
Only the rank arguments differ.
Start the head command, then start the worker command in its session; the distributed startup waits for both ranks.

### Qwen3.8 Flash-Next on two 64GB Sparks

```bash
export VLLM_PLE_TABLE_MEMORY="disk"
export CUTE_DSL_ARCH="sm_121a"
export SAFETENSORS_FAST_GPU="1"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_SSM_CONV_STATE_LAYOUT="DS"
export VLLM_USE_AOT_COMPILE="1"
export VLLM_USE_MEGA_AOT_ARTIFACT="1"
export VLLM_USE_V2_MODEL_RUNNER="1"
export B12X_POLICY_MODE="auto"
export VLLM_ENABLE_ROCE_ALLREDUCE="1"
export VLLM_ROCE_ALLREDUCE_MAX_SIZE="2MB"
export NCCL_BUFFSIZE="1048576"
export NCCL_MAX_CTAS="8"
export B12X_WEIGHTS_COMPILE_WORKERS="4"
export B12X_STATE_COMPILE_WORKERS="4"
export B12X_BIND_COMPILE_WORKERS="4"
export VLLM_B12X_MOE_FP4_FORCE_A16="1"
export B12X_W4A16_A4_PREFILL="1"

vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--tensor-parallel-size 2 \
--mamba-cache-mode align \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype bfloat16 \
--kv-cache-dtype fp8 \
--quantization modelopt_mixed \
--block-size 16 \
--load-format b12x \
--max-model-len 262144 \
--max-num-seqs 8 \
--max-num-batched-tokens 2048 \
--speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
--gdn-decode-kernel b12x \
--linear-backend b12x \
--moe-backend b12x \
--no-enable-flashinfer-autotune \
--mm-encoder-tp-mode data \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_xml \
--enable-auto-tool-choice \
--compilation-config '{"pass_config":{"fuse_act_quant":true},"max_cudagraph_capture_size":40,"cudagraph_capture_sizes":[1,2,4,5,8,10,16,20,24,32,40]}' \
--mm-processor-cache-gb 0 \
--limit-mm-per-prompt '{"image":1}' \
--gpu-memory-utilization 0.8 \
--kv-cache-memory-bytes 3G \
--nnodes 2 \
--master-addr "$HEAD_IP" \
--master-port "$MASTER_PORT" \
"${NODE_ARGS[@]}"
```

### GLM 5.3 Flash on two 128GB Sparks

```bash
export CUTE_DSL_ARCH="sm_121a"
export SAFETENSORS_FAST_GPU="1"
export VLLM_ENABLE_ROCE_ALLREDUCE="1"
export VLLM_ROCE_ALLREDUCE_MAX_SIZE="2MB"
export NCCL_BUFFSIZE=1048576
export NCCL_MAX_CTAS=8
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_SSM_CONV_STATE_LAYOUT="DS"
export VLLM_USE_AOT_COMPILE="1"
export VLLM_USE_MEGA_AOT_ARTIFACT="1"
export VLLM_USE_V2_MODEL_RUNNER="1"
export VLLM_ENABLE_PCIE_ALLREDUCE="0"
export B12X_POLICY_MODE="auto"
export VLLM_B12X_MOE_FP4_FORCE_A16=1
export B12X_W4A16_A4_PREFILL=1

vllm serve nvidia/GLM-5.3-Flash-FP4-Local \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 2 \
--pipeline-parallel-size 1 \
--decode-context-parallel-size 1 \
--mamba-cache-mode align \
--enable-prefix-caching \
--enable-chunked-prefill \
--dtype bfloat16 \
--kv-cache-dtype fp8 \
--quantization modelopt_mixed \
--attention-backend B12X \
--block-size 256 \
--moe-backend b12x \
--linear-backend b12x \
--no-enable-flashinfer-autotune \
--load-format b12x \
--max-model-len 500000 \
--max-num-seqs 4 \
--max-num-batched-tokens 4096 \
--speculative-config '{"method":"mtp","num_speculative_tokens":5,"moe_backend":"humming","attention_backend":"B12X"}' \
--reasoning-parser glm45 \
--tool-call-parser glm47 \
--enable-auto-tool-choice \
--gpu-memory-utilization 0.87 \
--nnodes 2 \
--master-addr "$HEAD_IP" \
--master-port "$MASTER_PORT" \
"${NODE_ARGS[@]}"
```

**Success:** Both ranks load the model and join the distributed executor, and the head starts the API server.
Keep both container sessions open.

## Step 5. Test the API and tool calling

**First, check readiness from a second host shell on the head.**

```bash
curl --fail-with-body -i http://127.0.0.1:8000/health
curl --fail-with-body -sS http://127.0.0.1:8000/v1/models
```

Wait for HTTP `200`, then copy the model ID from the model list.
If startup fails, check both container sessions: a worker failure can block the head.

**Next, send a chat request with that model ID.**

```bash
curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'
```

Allow enough output tokens for reasoning and a final answer.
Check `finish_reason` if the response stops before the answer is complete.

**Then, test automatic tool calling.**

Replace `<model-id>` and run:

```bash
curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"<model-id>","messages":[{"role":"user","content":"Use get_weather to look up the weather in Boston."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a location.","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],"tool_choice":"auto","max_tokens":4096}'
```

Look for a `tool_calls` entry for `get_weather` with Boston in its arguments, and `finish_reason` set to `tool_calls`.
The request declares the tool; your client would execute it.
If the reply is plain text, check the model's tool support, chat template, and parser settings.
See the [vLLM tool-calling guide](https://docs.vllm.ai/en/latest/features/tool_calling/).

**Success:** Chat and automatic tool calling work through the head's API.

## Step 6. Connect a client and tune the workload

**First, forward the head's API port to your laptop if needed.**

Run this in a local laptop terminal, using the head's management address or a working SSH alias:

```text
ssh -N -L 8000:127.0.0.1:8000 <user>@<head-host>
```

Keep the tunnel open and use `http://localhost:8000/v1` as the client's base URL.
Set the client's model to the ID from `/v1/models`.
The worker does not need a separate client connection.

**Then, measure the service with prompts and concurrency that match your workload.**

Record prompt length, output length, active requests, time to first token, and output tokens per second.
Watch memory and link use on both Sparks.
Two nodes add capacity and communication work; measure the result for the model and request mix you plan to serve.

Change one setting at a time and apply the same change on both ranks.

| Goal | Settings to review |
| --- | --- |
| Fit the workload within memory | Active sequences, context length, and the recipe's KV-cache budget |
| Change the balance between prompt work and active replies | `--max-num-batched-tokens` with chunked prefill |
| Reuse repeated prompt text | Prefix caching with matching system prompts or document prefixes |
| Check the benefit of MTP | Compare the recipe's speculative settings with a supported baseline under the same request mix |

Keep a copy of the working commands, image tag or digest, checkpoint revision, network values, and measurements before you tune.

**Success:** You have a working client and a measured baseline for further changes.

## Step 7. Stop or save the service

**First, stop both containers when you are done.**

Run this from a host shell on **each Spark**:

```bash
docker stop vllm_cluster
```

The containers are removed when they stop.
The mounted model and runtime caches remain for the next start.
Close the laptop tunnel with `Ctrl+C`.

**Then, save the host and server commands if you want a repeatable launch.**

Keep the per-node network values and rank arguments with the full model command.
If you use a wrapper, have it start and stop both ranks and report a failed worker.
The Intermediate Tab includes supplied YAML recipes for the community launcher; those files keep the model environment and serve flags together while the launcher supplies the node ranks and starts both containers.

**Success:** Both nodes have stopped, and you have the commands needed to start the same service again.