---
title: "Serve LLMs with SGLang"
publisher: "nvidia"
type: "playbook"
updated: "2026-08-04T16:15:58.908Z"
description: "High-throughput serving with RadixAttention, structured output, and an OpenAI-compatible API"
canonical: "https://build.nvidia.com/station/sglang.md"
---

# Basic idea

SGLang is a high-performance serving framework for large language models and vision-language models. It co-designs the backend runtime and frontend language so interactions are faster and more controllable — especially for workloads that share prefixes.

- **RadixAttention** automatically caches and reuses KV cache entries across requests that share common prefixes (multi-turn chat, RAG, agents), reducing redundant prefill work.
- **Structured output** uses compressed finite-state machine decoding (xGrammar) for JSON, regex, and grammar-constrained generation with low overhead.
- An **OpenAI-compatible API** supports `/v1/chat/completions`, `/v1/completions`, and related endpoints so existing clients can switch backends with little or no change.

# What you'll accomplish

Serve a **model** with SGLang on your **supported hardware platform** using a pre-built CUDA 13 container and an OpenAI-compatible endpoint. You will also exercise prefix-cached multi-turn chat and structured JSON output.

# What to know before starting

**Required:**

- Basic Docker container usage
- Familiarity with REST APIs

**Optional:**

- Python for client scripts and the optional multi-turn benchmark

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base workflow applies across supported hardware platforms.

| Hardware platform | OS | Memory  | Multi-node capable hardware |
| :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 128 GB Unified Memory | — |
| **DGX Station** | DGX OS (Linux) | Large HBM + Grace DRAM | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for your chosen model (see Supported models below)

**Software requirements**

- Docker installed: `docker --version`
- NVIDIA Container Toolkit configured (`nvidia-smi` works inside a GPU container)
- HuggingFace account with an access token (for gated / private model downloads)
- Network access to Docker Hub and HuggingFace
- SGLang container image — see Instructions
- `git` — only for the optional benchmark and offline-inference scripts (see Ancillary files below); Steps 1–7 need no local files

# Ancillary files

Steps 1–7 of the **Instructions** tab run entirely against the container and need nothing on the host. The two optional Python scripts live under `assets/` in this playbook and require cloning the repository (Instructions → Step 8):

| File | Purpose |
|------|---------|
| `assets/benchmark_multiturn.py` | Multi-turn throughput and prefix-cache benchmark (Instructions → Step 8) |
| `assets/offline-inference.py` | In-process SGLang Engine example, no server required (Instructions → Next steps) |

# Supported models

Use the matrices below to pick a model for your hardware platform. Full serve workflow is in the **Instructions** tab.

> [!NOTE]
> **Memory determines what you can run.** Large models need substantially more memory. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in **Instructions**.

## DGX Spark

Models validated with SGLang on DGX Spark:

| Model | Quantization | HF Handle |
|-------|-------------|---------|
| **Nemotron-3-Nano-Omni-30B-A3B-Reasoning** | BF16 | [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16) |
| **GPT-OSS-20B** | MXFP4 | `openai/gpt-oss-20b` |
| **GPT-OSS-120B** | MXFP4 | `openai/gpt-oss-120b` |
| **Llama-3.1-8B-Instruct** | FP8 | `nvidia/Llama-3.1-8B-Instruct-FP8` |
| **Llama-3.1-8B-Instruct** | NVFP4 | `nvidia/Llama-3.1-8B-Instruct-FP4` |
| **Llama-3.3-70B-Instruct** | NVFP4 | `nvidia/Llama-3.3-70B-Instruct-FP4` |
| **Qwen3-8B** | FP8 / NVFP4 | `nvidia/Qwen3-8B-FP8` / `nvidia/Qwen3-8B-FP4` |
| **Qwen3-14B** | FP8 / NVFP4 | `nvidia/Qwen3-14B-FP8` / `nvidia/Qwen3-14B-FP4` |
| **Qwen3-32B** | NVFP4 | `nvidia/Qwen3-32B-FP4` |
| **Phi-4-multimodal-instruct** | FP8 / NVFP4 | `nvidia/Phi-4-multimodal-instruct-FP8` / `nvidia/Phi-4-multimodal-instruct-FP4` |
| **Phi-4-reasoning-plus** | FP8 / NVFP4 | `nvidia/Phi-4-reasoning-plus-FP8` / `nvidia/Phi-4-reasoning-plus-FP4` |

For NVFP4 models, add `--quantization modelopt_fp4` to the serve command. Certain models (for example Nemotron-3-Nano-Omni) may require extra flags from their HuggingFace model card.

## DGX Station

Starting points documented for SGLang on DGX Station (confirm memory headroom and SGLang build support before large downloads):

| Model | Role | HF Handle |
|-------|------|---------|
| **Qwen3-8B** | Default first-run (fast validation) | `Qwen/Qwen3-8B` |
| **Qwen3.6-35B-A3B** | MoE (~3B active); hybrid mamba/SSM — prefix-cache check in Instructions does not apply | `Qwen/Qwen3.6-35B-A3B` |
| **Qwen3.6-27B** | Dense Qwen3.6 | `Qwen/Qwen3.6-27B` |
| **Llama-3.3-70B-Instruct** | Gated on Hugging Face — accept license before download | `meta-llama/Llama-3.3-70B-Instruct` |
| **DeepSeek-V4-Flash** | Large local MoE when memory allows | `deepseek-ai/DeepSeek-V4-Flash` |
| **DeepSeek-V4-Pro** | Larger V4 variant — only with sufficient memory and a supported SGLang build | `deepseek-ai/DeepSeek-V4-Pro` |

You may also use other Hugging Face text-generation or chat checkpoints that your SGLang build supports.

# Time & risk

- **Estimated time:** 30 MIN (longer on first run due to model download and CUDA-graph capture; larger MoE models can take 45–60 MIN)
- **Risk level:** Low
- Model download may require HuggingFace authentication for gated models
- **Rollback:** Stop and remove the container to restore state (non-destructive)
- **Last Updated:** 07/31/2026
- Documented how to obtain the `assets/` scripts and defined the playbook root; cached-model cleanup now uses `sudo`
- Replaced recipe-portal links with Supported models matrices (Spark + Station); removed Agent-ready Models tab

## More

- [Instructions](/station/sglang/instructions.md)
- [Troubleshooting](/station/sglang/troubleshooting.md)