---
title: "Serve LLMs with vLLM"
publisher: "nvidia"
type: "playbook"
updated: "2026-08-06T18:38:21.718Z"
description: "High-throughput serving for 30+ models, with continuous batching and an OpenAI-compatible API"
canonical: "https://build.nvidia.com/playbooks/vllm.md"
---

# Basic idea

vLLM is an inference engine designed to run large language models efficiently. The key idea is **maximizing throughput and minimizing memory waste** when serving LLMs.

- **PagedAttention** handles long sequences without running out of GPU memory.
- **Continuous batching** keeps GPUs fully utilized by adding new requests to batches already in progress.
- An **OpenAI-compatible API** lets applications built for the OpenAI API switch to a vLLM backend with little or no modification.

# What you'll accomplish

Serve a **model** with vLLM on your **supported hardware platform** using a pre-built container and an OpenAI-compatible endpoint.

# What to know before starting

**Required:**

- Basic Docker container usage
- Familiarity with REST APIs

**Optional:**

- Basic networking and SSH between nodes (multi-node capable hardware only)

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base single-node workflow applies across supported hardware platforms. **Multi-node serving in this playbook is DGX Spark only** (see the Multi-node serving tab).

| Hardware platform | OS | Memory  | Multi-node capable hardware |
| :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 128 GB Unified Memory | ✅ (QSFP + Ray) |
| **DGX Station** | DGX OS (Linux) | Large HBM + Grace DRAM | — |
| **RTX PRO** | Ubuntu 22.04 / 24.04 (Linux) | Dedicated VRAM | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for your chosen model (see [vLLM Recipes](https://recipes.vllm.ai/browse) for your hardware platform)
- Multi-node capable hardware: QSFP connectivity and passwordless SSH between nodes

**Software requirements**

- Docker installed: `docker --version`
- NVIDIA Container Toolkit configured
- HuggingFace account with an access token (for gated / private model downloads)
- Network access to NGC and HuggingFace
- NGC vLLM container image for your hardware platform — see Instructions

# Find model recipes

Browse tested vLLM launch settings for your hardware platform on [vLLM Recipes](https://recipes.vllm.ai/browse). Each recipe includes copyable `vllm serve` commands, container images, and tuning notes for that model on your hardware.

For **more recipes**, open the filtered catalogs below:

| Hardware platform | More recipes |
| ----------------- | ------------ |
| **DGX Spark** | [recipes.vllm.ai — DGX Spark](https://recipes.vllm.ai/browse?panel=open&hw=dgx_spark_gb10) |
| **DGX Station** | [recipes.vllm.ai — DGX Station](https://recipes.vllm.ai/browse?panel=open&hw=dgx_station_gb300) |
| **RTX PRO** | [recipes.vllm.ai — RTX PRO](https://recipes.vllm.ai/browse?panel=open&hw=rtx_pro_6000) |

Use the **Instructions** tab for container setup and a base `docker run` workflow. For agentic workloads, see the **Agent-ready Models** tab.

> [!NOTE]
> **Memory determines what you can run.** Large models need substantially more memory and may require CPU offload. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in **Instructions**.

# Time & risk

- **Estimated time:** 30 MIN (longer on first run due to model download)
- **Risk level:** Low
- Model download requires HuggingFace authentication
- Some containers require NGC credentials
- **Rollback:** Stop and remove the container to restore state (non-destructive)
- **Last Updated:** 08/03/2026
- Step 5 API test uses `max_tokens: 2048` so reasoning-enabled recipes return a visible answer
- 07/27/2026: Multi-node serving scoped to DGX Spark only; Find model recipes links out to Spark / Station / RTX PRO filtered catalogs

## More

- [Instructions](/playbooks/vllm/instructions.md)
- [Agent-ready Models](/playbooks/vllm/agent-ready-models.md)
- [Multi-node serving](/playbooks/vllm/multi-node.md)
- [Troubleshooting](/playbooks/vllm/troubleshooting.md)