Serve LLMs with vLLM

Basic idea

vLLM is an inference engine designed to run large language models efficiently. The key idea is maximizing throughput and minimizing memory waste when serving LLMs.

PagedAttention handles long sequences without running out of GPU memory.
Continuous batching keeps GPUs fully utilized by adding new requests to batches already in progress.
An OpenAI-compatible API lets applications built for the OpenAI API switch to a vLLM backend with little or no modification.

What you'll accomplish

Serve a model with vLLM on your supported hardware platform using a pre-built container and an OpenAI-compatible endpoint.

What to know before starting

Required:

Basic Docker container usage
Familiarity with REST APIs

Optional:

Basic networking and SSH between nodes (multi-node capable hardware only)

Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base single-node workflow applies across supported hardware platforms. Multi-node serving in this playbook is DGX Spark only (see the Multi-node serving tab).

Hardware platform	OS	Memory	Multi-node capable hardware
DGX Spark	DGX OS (Linux)	128 GB Unified Memory	✅ (QSFP + Ray)
DGX Station	DGX OS (Linux)	Large HBM + Grace DRAM	—
RTX PRO	Ubuntu 22.04 / 24.04 (Linux)	Dedicated VRAM	—

Prerequisites

Hardware requirements

Supported hardware platform — see Supported hardware platforms matrix above
Sufficient memory for your chosen model (see vLLM Recipes for your hardware platform)
Multi-node capable hardware: QSFP connectivity and passwordless SSH between nodes

Software requirements

Docker installed: docker --version
NVIDIA Container Toolkit configured
HuggingFace account with an access token (for gated / private model downloads)
Network access to NGC and HuggingFace
NGC vLLM container image for your hardware platform — see Instructions

Find model recipes

Browse tested vLLM launch settings for your hardware platform on vLLM Recipes. Each recipe includes copyable vllm serve commands, container images, and tuning notes for that model on your hardware.

For more recipes, open the filtered catalogs below:

Hardware platform	More recipes
DGX Spark	recipes.vllm.ai — DGX Spark
DGX Station	recipes.vllm.ai — DGX Station
RTX PRO	recipes.vllm.ai — RTX PRO

Use the Instructions tab for container setup and a base docker run workflow. For agentic workloads, see the Agent-ready Models tab.

NOTE

Memory determines what you can run. Large models need substantially more memory and may require CPU offload. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in Instructions.

Time & risk

Estimated time: 30 MIN (longer on first run due to model download)
Risk level: Low
- Model download requires HuggingFace authentication
- Some containers require NGC credentials
Rollback: Stop and remove the container to restore state (non-destructive)
Last Updated: 08/03/2026
- Step 5 API test uses max_tokens: 2048 so reasoning-enabled recipes return a visible answer
- 07/27/2026: Multi-node serving scoped to DGX Spark only; Find model recipes links out to Spark / Station / RTX PRO filtered catalogs