Serve LLMs with vLLM Across DGX Sparks

Run and tune a vLLM recipe across two DGX Sparks

You will run one vLLM service across two Sparks, using the Docker and server commands from an NVIDIA Local AI Inference Recipe.

This path uses SSH and a container on each node. It covers the settings that split the model across the pair, move data between nodes, and set the capacity of the service.

Choose a model and prepare the cluster

First, choose the recipe for your pair of Sparks.

Memory in this table is per device. Use two Sparks of the same memory size for these configurations.

Both recipes use eugr/spark-vllm-b12x:nightly-20261015, a community image with the B12X backend. Keep the image, checkpoint, environment settings, and server flags from the same recipe. The 64GB Qwen configuration has its own memory settings; the settings for a pair of 128GB Sparks are different.

Then, cable and configure the two nodes.

Use Configure Manually in the Connect Multiple Sparks playbook if the cluster is not already set up. Confirm interdevice SSH and Docker access on both nodes. Choose a head Spark and a worker Spark, then open an SSH session to each.

Keep these values from the network setup:

Success: You have a complete recipe and can reach both nodes over the cluster link.

Read the distributed and memory settings

First, check how the recipe splits work across the pair.

Each Spark runs one tensor-parallel rank. The ranks hold parts of the model and exchange data during inference. The head exposes the API; the worker joins the same service. This gives you one model endpoint across two GPUs.

These commands use native multi-node execution. They do not need a separate Ray cluster. API port 8000 and rendezvous port 29501 serve different roles. See the vLLM multi-node multiprocessing guide for the rank and rendezvous settings.

Next, check the memory and scheduling limits.

Each node must fit its share of the weights, caches, and runtime work. The pair does not act as one flat pool of memory.

The context limit is a per-request ceiling. It does not mean every active request can use that full length at once. Actual capacity also depends on the model's cache layout and the other requests in flight.

Then, keep the model-specific backend settings together for the first run.

The B12X kernels, compilation settings, and environment variables are part of these recipes. For example, the 64GB Qwen recipe uses VLLM_PLE_TABLE_MEMORY=disk, limits compile workers, and sets a 3G KV-cache budget. GLM uses a different attention backend, block size, and memory budget.

Both recipes use native multi-token prediction (MTP) for speculative decoding. They do not call for a separate draft checkpoint. Keep the reasoning and tool-call parsers if the service will be used by an agent.

See the vLLM serve options, memory and scheduling guide, and speculative decoding guide for the core settings. The community image supplies the B12X-specific code paths used here.

Success: You know which settings define the cluster, which set its capacity, and which belong to the chosen model and backend.

Start a container on each Spark

First, set the network values in each node's host shell.

Use the same head address and rendezvous port on both nodes. Set ETH_IF and IB_IF to the local node's interfaces; their names may differ between nodes. Keep both RoCE HCA names called for by the recipe.

export HEAD_IP="<head-connectx-ip>"
export ETH_IF="<local-connectx-interface>"
export IB_IF="<local-roce-hca-1>,<local-roce-hca-2>"
export MASTER_PORT=29501
export IMAGE="eugr/spark-vllm-b12x:nightly-20261015"
export HF_CACHE_DIR="$HOME/.cache/huggingface"

Then, run this Docker command on each node.

If checkpoint access needs a Hugging Face token, set HF_TOKEN in the host shell first. The container downloads the image if needed, and vLLM downloads the model when it starts. Both nodes need the model files and enough free disk space for their local caches.

mkdir -p "$HF_CACHE_DIR" "$HOME/.cache/vllm" \
  "$HOME/.cache/flashinfer" "$HOME/.triton" "$HOME/.cache/b12x"

docker run --rm -it \
  --name vllm_cluster \
  --privileged --gpus all --ipc host --network host \
  --shm-size=16g \
  --ulimit nofile=1048576:1048576 \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e HF_TOKEN="${HF_TOKEN:-}" \
  -e MN_IF_NAME="$ETH_IF" -e UCX_NET_DEVICES="$ETH_IF" \
  -e NCCL_SOCKET_IFNAME="$ETH_IF" -e NCCL_IB_HCA="$IB_IF" \
  -e NCCL_IB_DISABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 \
  -e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
  -e GLOO_SOCKET_IFNAME="$ETH_IF" -e TP_SOCKET_IFNAME="$ETH_IF" \
  -e HEAD_IP="$HEAD_IP" -e MASTER_PORT="$MASTER_PORT" \
  -v "$HF_CACHE_DIR:/root/.cache/huggingface" \
  -v "$HOME/.cache/vllm:/root/.cache/vllm" \
  -v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \
  -v "$HOME/.triton:/root/.triton" \
  -v "$HOME/.cache/b12x:/root/.cache/b12x" \
  --entrypoint /bin/bash "$IMAGE"

Host networking and the interface variables let the processes use the ConnectX link. The mounts keep model and runtime files across container restarts. The API will bind to the head's host interfaces, so use the service on your trusted network.

Success: You have an open Bash shell inside the container on each node.

Run the model on the head and worker

First, set the rank arguments inside each container.

On the head, run:

NODE_ARGS=(--node-rank 0)

On the worker, run:

NODE_ARGS=(--node-rank 1 --headless)

Then, run the full command for your model in both containers.

Use the same model, environment settings, and serving flags on both nodes. Only the rank arguments differ. Start the head command, then start the worker command in its session; the distributed startup waits for both ranks.

Qwen3.8 Flash-Next on two 64GB Sparks

export VLLM_PLE_TABLE_MEMORY="disk"
export CUTE_DSL_ARCH="sm_121a"
export SAFETENSORS_FAST_GPU="1"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_SSM_CONV_STATE_LAYOUT="DS"
export VLLM_USE_AOT_COMPILE="1"
export VLLM_USE_MEGA_AOT_ARTIFACT="1"
export VLLM_USE_V2_MODEL_RUNNER="1"
export B12X_POLICY_MODE="auto"
export VLLM_ENABLE_ROCE_ALLREDUCE="1"
export VLLM_ROCE_ALLREDUCE_MAX_SIZE="2MB"
export NCCL_BUFFSIZE="1048576"
export NCCL_MAX_CTAS="8"
export B12X_WEIGHTS_COMPILE_WORKERS="4"
export B12X_STATE_COMPILE_WORKERS="4"
export B12X_BIND_COMPILE_WORKERS="4"
export VLLM_B12X_MOE_FP4_FORCE_A16="1"
export B12X_W4A16_A4_PREFILL="1"

vllm serve nvidia/Qwen3.8-Flash-Next-FP4-Local \
  --host 0.0.0.0 \
  --port 8000 \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --quantization modelopt_mixed \
  --block-size 16 \
  --load-format b12x \
  --max-model-len 262144 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":4}' \
  --gdn-decode-kernel b12x \
  --linear-backend b12x \
  --moe-backend b12x \
  --no-enable-flashinfer-autotune \
  --mm-encoder-tp-mode data \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice \
  --compilation-config '{"pass_config":{"fuse_act_quant":true},"max_cudagraph_capture_size":40,"cudagraph_capture_sizes":[1,2,4,5,8,10,16,20,24,32,40]}' \
  --mm-processor-cache-gb 0 \
  --limit-mm-per-prompt '{"image":1}' \
  --gpu-memory-utilization 0.8 \
  --kv-cache-memory-bytes 3G \
  --nnodes 2 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT" \
  "${NODE_ARGS[@]}"

GLM 5.3 Flash on two 128GB Sparks

export CUTE_DSL_ARCH="sm_121a"
export SAFETENSORS_FAST_GPU="1"
export VLLM_ENABLE_ROCE_ALLREDUCE="1"
export VLLM_ROCE_ALLREDUCE_MAX_SIZE="2MB"
export NCCL_BUFFSIZE=1048576
export NCCL_MAX_CTAS=8
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_SSM_CONV_STATE_LAYOUT="DS"
export VLLM_USE_AOT_COMPILE="1"
export VLLM_USE_MEGA_AOT_ARTIFACT="1"
export VLLM_USE_V2_MODEL_RUNNER="1"
export VLLM_ENABLE_PCIE_ALLREDUCE="0"
export B12X_POLICY_MODE="auto"
export VLLM_B12X_MOE_FP4_FORCE_A16=1
export B12X_W4A16_A4_PREFILL=1

vllm serve nvidia/GLM-5.3-Flash-FP4-Local \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 2 \
  --pipeline-parallel-size 1 \
  --decode-context-parallel-size 1 \
  --mamba-cache-mode align \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --dtype bfloat16 \
  --kv-cache-dtype fp8 \
  --quantization modelopt_mixed \
  --attention-backend B12X \
  --block-size 256 \
  --moe-backend b12x \
  --linear-backend b12x \
  --no-enable-flashinfer-autotune \
  --load-format b12x \
  --max-model-len 500000 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 4096 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5,"moe_backend":"humming","attention_backend":"B12X"}' \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice \
  --gpu-memory-utilization 0.87 \
  --nnodes 2 \
  --master-addr "$HEAD_IP" \
  --master-port "$MASTER_PORT" \
  "${NODE_ARGS[@]}"

Success: Both ranks load the model and join the distributed executor, and the head starts the API server. Keep both container sessions open.

Test the API and tool calling

First, check readiness from a second host shell on the head.

curl --fail-with-body -i http://127.0.0.1:8000/health
curl --fail-with-body -sS http://127.0.0.1:8000/v1/models

Wait for HTTP 200, then copy the model ID from the model list. If startup fails, check both container sessions: a worker failure can block the head.

Next, send a chat request with that model ID.

curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Write a haiku about a GPU."}],"max_tokens":4096}'

Allow enough output tokens for reasoning and a final answer. Check finish_reason if the response stops before the answer is complete.

Then, test automatic tool calling.

Replace <model-id> and run:

curl --fail-with-body -sS http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Use get_weather to look up the weather in Boston."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a location.","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],"tool_choice":"auto","max_tokens":4096}'

Look for a tool_calls entry for get_weather with Boston in its arguments, and finish_reason set to tool_calls. The request declares the tool; your client would execute it. If the reply is plain text, check the model's tool support, chat template, and parser settings. See the vLLM tool-calling guide.

Success: Chat and automatic tool calling work through the head's API.

Connect a client and tune the workload

First, forward the head's API port to your laptop if needed.

Run this in a local laptop terminal, using the head's management address or a working SSH alias:

ssh -N -L 8000:127.0.0.1:8000 <user>@<head-host>

Keep the tunnel open and use http://localhost:8000/v1 as the client's base URL. Set the client's model to the ID from /v1/models. The worker does not need a separate client connection.

Then, measure the service with prompts and concurrency that match your workload.

Record prompt length, output length, active requests, time to first token, and output tokens per second. Watch memory and link use on both Sparks. Two nodes add capacity and communication work; measure the result for the model and request mix you plan to serve.

Change one setting at a time and apply the same change on both ranks.

Keep a copy of the working commands, image tag or digest, checkpoint revision, network values, and measurements before you tune.

Success: You have a working client and a measured baseline for further changes.

Stop or save the service

First, stop both containers when you are done.

Run this from a host shell on each Spark:

docker stop vllm_cluster

The containers are removed when they stop. The mounted model and runtime caches remain for the next start. Close the laptop tunnel with Ctrl+C.

Then, save the host and server commands if you want a repeatable launch.

Keep the per-node network values and rank arguments with the full model command. If you use a wrapper, have it start and stop both ranks and report a failed worker. The Intermediate Tab includes supplied YAML recipes for the community launcher; those files keep the model environment and serve flags together while the launcher supplies the node ranks and starts both containers.

Success: Both nodes have stopped, and you have the commands needed to start the same service again.