Serve LLMs with vLLM
High-throughput serving for 30+ models, with continuous batching and an OpenAI-compatible API
Basic idea
vLLM is an inference engine designed to run large language models efficiently. The key idea is maximizing throughput and minimizing memory waste when serving LLMs.
- PagedAttention handles long sequences without running out of GPU memory.
- Continuous batching keeps GPUs fully utilized by adding new requests to batches already in progress.
- An OpenAI-compatible API lets applications built for the OpenAI API switch to a vLLM backend with little or no modification.
What you'll accomplish
Serve a model with vLLM on your supported hardware platform using a pre-built container and an OpenAI-compatible endpoint.
What to know before starting
Required:
- Basic Docker container usage
- Familiarity with REST APIs
Optional:
- Basic networking and SSH between nodes (multi-node capable hardware only)
Supported hardware platforms
Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base single-node workflow applies across supported hardware platforms. Multi-node serving in this playbook is DGX Spark only (see the Multi-node serving tab).
| Hardware platform | OS | Memory | Multi-node capable hardware |
|---|---|---|---|
| DGX Spark | DGX OS (Linux) | 128 GB Unified Memory | ✅ (QSFP + Ray) |
| DGX Station | DGX OS (Linux) | Large HBM + Grace DRAM | — |
| RTX PRO | Ubuntu 22.04 / 24.04 (Linux) | Dedicated VRAM | — |
Prerequisites
Hardware requirements
- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for your chosen model (see vLLM Recipes for your hardware platform)
- Multi-node capable hardware: QSFP connectivity and passwordless SSH between nodes
Software requirements
- Docker installed:
docker --version - NVIDIA Container Toolkit configured
- HuggingFace account with an access token (for gated / private model downloads)
- Network access to NGC and HuggingFace
- NGC vLLM container image for your hardware platform — see Instructions
Find model recipes
Browse tested vLLM launch settings for your hardware platform on vLLM Recipes. Each recipe includes copyable vllm serve commands, container images, and tuning notes for that model on your hardware.
For more recipes, open the filtered catalogs below:
| Hardware platform | More recipes |
|---|---|
| DGX Spark | recipes.vllm.ai — DGX Spark |
| DGX Station | recipes.vllm.ai — DGX Station |
| RTX PRO | recipes.vllm.ai — RTX PRO |
Use the Instructions tab for container setup and a base docker run workflow. For agentic workloads, see the Agent-ready Models tab.
NOTE
Memory determines what you can run. Large models need substantially more memory and may require CPU offload. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in Instructions.
Time & risk
- Estimated time: 30 MIN (longer on first run due to model download)
- Risk level: Low
- Model download requires HuggingFace authentication
- Some containers require NGC credentials
- Rollback: Stop and remove the container to restore state (non-destructive)
- Last Updated: 08/03/2026
- Step 5 API test uses
max_tokens: 2048so reasoning-enabled recipes return a visible answer - 07/27/2026: Multi-node serving scoped to DGX Spark only; Find model recipes links out to Spark / Station / RTX PRO filtered catalogs
- Step 5 API test uses