Serve LLMs with vLLM on One Device

Choose and run a vLLM recipe on DGX Spark or Station

You will choose a recipe for your device, run its commands, and test the model's API and tool calling.

Along the way, you will see how the recipe uses memory, batches requests, and selects the code that runs the model.

Once the endpoint works, you can tune it for your workload, save the launch command, and use it with a client such as Hermes or OpenShell.

Choose a model and recipe

First, open the recipe page for your device and memory size.

Use NVIDIA Local AI Inference Recipes to select a model and vLLM configuration.

The recommended Spark model is also one of the models in the Intermediate tab, so you can use either path. If you pick a different model in the recipe page, check its prerequisites before you download it.

Then, copy the "Start container" and "Launch" commands and save them to a file.

The checkpoint, image, environment settings, and server flags work together. For example, a low-precision checkpoint needs a backend that can load and run its format, and a speculative-decoding variant may need a second checkpoint. Start with the full configuration from one recipe.

Success: You have saved a complete recipe for your device and chosen model.

Read the settings that shape the service

First, check how the recipe divides memory between weights and requests.

The model weights take up part of the device's memory. Serving also needs space for runtime work and the KV cache, which holds attention keys and values from tokens already processed. Longer requests and more requests in flight compete for that space.

See the vLLM serve options and quantized KV-cache guide for details.

Next, check how the recipe schedules work and reuses it.

vLLM first processes the input prompt, called prefill, then generates output tokens, called decode. The best settings depend on prompt length, output length, and the number of clients.

Read more about scheduling and tuning, prefix caching, and speculative decoding.

Then, keep the recipe's model-specific runtime settings together.

Load formats, attention and MoE backends, compilation settings, and environment variables select code paths supported by that image and model. Keep them for the first run so you have a working baseline.

For agent use, also keep the model's tool-call and reasoning parsers. --enable-auto-tool-choice allows the model to choose a tool, while --tool-call-parser reads its tool-call format. --reasoning-parser separates reasoning from the final answer; the model's chat template and request settings control whether it thinks. See tool calling and reasoning outputs.

Success: You know which settings control capacity, scheduling, and the response format your client will receive.

Run the recipe

First, check the setup command in a terminal on the DGX device.

Use a local terminal on the device or an SSH session. Set any host variables called for by the recipe, such as HF_TOKEN.

For Station, check that the command selects the GB300.

If the recipe uses --gpus all and your Station has more than one GPU, get the GB300's ID on the host:

GB300_UUID=$(nvidia-smi --query-gpu=uuid,name --format=csv,noheader | \
  awk -F, 'tolower(\$2) ~ /gb300/ {gsub(/[[:space:]]/, "", \$1); print \$1; exit}')
: "\${GB300_UUID:?No GB300 found}"

Before running the container, replace --gpus all with --gpus "device=${GB300_UUID}". Keep any GPU selection already supplied by the recipe.

Then, run the full setup command on the DGX host.

Docker pulls the image if it is missing. The setup or server command downloads model files as needed; keep any explicit download, revision, and local-path options in the recipe.

The image supplies the vLLM build and libraries used by the configuration. The Hugging Face cache mount keeps model files on the host. Mounts for the vLLM, FlashInfer, and Triton caches also let later containers reuse compiled work. First startup can include downloads, compilation, and kernel tuning.

Next, run the server command inside the container.

A setup command that ends with --entrypoint /bin/bash and an image name opens a shell in that image. In that shell:

1. Apply any listed environment settings
2. Run the complete vllm serve command
3. Leave the terminal open and follow startup output

If the recipe already launches the server in its Docker command, follow that form directly.

Finally, note the API address and port.

With --network host, vLLM listens directly on the DGX host's port. With a Docker -p mapping, use the published host port for checks outside the container. With host networking, a loopback bind is enough for host-local clients and SSH forwarding. With port mapping, keep vLLM bound to 0.0.0.0 inside the container and choose the host-side bind in the mapping. The OpenShell handoff in Step 7 needs the service reachable from its sandbox.

The commands below use DGX host port 8000. Use your recipe's host port if it differs.

Success: The server reports that it is ready to accept requests. Confirm readiness with the API checks in Step 4.

Test inference and tool calling

First, open a second terminal on the DGX device and check the API.

Run:

curl --fail-with-body -i http://localhost:8000/health
curl --fail-with-body -sS http://localhost:8000/v1/models

The health check should return HTTP 200, and the model list should have a non-empty data array. Copy the served model's id for the requests below. It can differ from the checkpoint name when the recipe sets --served-model-name.

Then, send a chat request with that model ID.

Replace <model-id> and run:

curl --fail-with-body -sS http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Reply with a short greeting."}],"max_tokens":4096}'

Success: choices[0].message.content contains an answer. Reasoning can use the output-token budget before an answer appears; follow the recipe's request settings or raise the budget if the response ends with finish_reason: length.

Finally, test a tool call.

Replace <model-id> and run:

curl --fail-with-body -sS http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"<model-id>","messages":[{"role":"user","content":"Use get_weather to look up the weather in Boston."}],"tools":[{"type":"function","function":{"name":"get_weather","description":"Get current weather for a location.","parameters":{"type":"object","properties":{"location":{"type":"string"}},"required":["location"]}}}],"tool_choice":"auto","max_tokens":4096}'

Success: The response contains a tool_calls entry for get_weather with Boston in its arguments, and finish_reason is tool_calls. This checks the model's automatic tool-call path and response format; the request declares the tool, and your client would execute it. If the model replies in plain text, check the recipe's tool-call settings and the chosen model's tool support.

(optional) Tune the service for your workload

First, record a baseline with representative requests.

Use the prompt lengths, output lengths, and client count you expect to serve. Record time to first token, output tokens per second, and memory use. For multiple clients, distinguish each client's generation rate from total throughput.

Then, change one setting and repeat the same workload.

1. Set --max-model-len to cover your actual input and output needs
2. Try --max-num-seqs and --max-num-batched-tokens to balance concurrency and latency
3. Compare repeated-prefix workloads with prefix caching, or test the recipe's speculative-decoding variant
4. Keep the settings that improve your target measure, then rerun the chat and tool tests

Memory capacity, request length, and concurrency form a shared budget. A context limit of 262144 tokens and --max-num-seqs 8 do not guarantee that eight full-length requests fit at once. Read the KV-cache capacity and concurrency estimates in startup output, then test the load you intend to run.

Success: You have measured settings that suit your workload and still pass the functional checks.

(optional) Save a repeatable launch script

First, save the working host setup and server commands.

Keep the image tag, checkpoint revision or path, GPU selection, cache mounts, environment settings, and tested vLLM options together.

Then, combine the commands if you want one launch script.

For a recipe that opens /bin/bash before you run vllm serve:

1. Move the server's environment settings into Docker -e NAME=value options
2. Replace --entrypoint /bin/bash with --entrypoint vllm
3. After the image name, append serve, the checkpoint or path, and the tested server arguments
4. Keep any host preparation before docker run, including recipe-specific downloads or cache setup
5. Add a container name, such as --name vllm-local, and save the result as launch-vllm.sh

Make it executable and run it on the DGX device:

chmod +x launch-vllm.sh
./launch-vllm.sh

For a background service, replace the interactive -it options with -d. Follow its output with docker logs --tail 50 --follow vllm-local and stop it with docker stop vllm-local. If the recipe uses --rm, run the saved script again to create the next container.

Success: The saved script starts the same service and passes the checks in Step 4.

Connect Hermes, OpenShell, or a laptop client

First, use an API address reachable from your client.

Keep the server running and use the exact model ID returned by /v1/models.

The recipe's 0.0.0.0 bind can expose an unauthenticated API on the device's network interfaces. Use a trusted network or suitable access controls for that route.

For a laptop client, open an SSH tunnel from the laptop.

Replace user@dgx-host with your SSH login and device address, then run:

ssh -N -L 8000:127.0.0.1:8000 user@dgx-host

Keep the tunnel open, then run the health and model-list checks from a second laptop terminal. Use curl.exe for those checks in Windows PowerShell. If laptop port 8000 is busy, change the leftmost port to 8001 and use http://localhost:8001/v1 in the client.

For Hermes or OpenShell, continue with the matching playbook.

• In the Hermes playbook, use the host-local API URL and the served model ID. For DeepSeek, follow that playbook's model-specific reasoning settings.
• In the OpenShell playbook, reuse this endpoint and continue at Step 6 after exporting MODEL_HANDLE as in its Step 5. Use the ID from /v1/models; the recipe may give the service an alias.

Success: Your client reaches the existing vLLM service and uses the model you tested.

Next steps

• Use the Troubleshooting Tab for startup, memory, or connection errors.
• Review the vLLM tuning guide for further performance work.
• Use the vLLM cluster playbook for more than one DGX Spark.