Serve LLMs with SGLang

Basic idea

SGLang is a high-performance serving framework for large language models and vision-language models. It co-designs the backend runtime and frontend language so interactions are faster and more controllable — especially for workloads that share prefixes.

RadixAttention automatically caches and reuses KV cache entries across requests that share common prefixes (multi-turn chat, RAG, agents), reducing redundant prefill work.
Structured output uses compressed finite-state machine decoding (xGrammar) for JSON, regex, and grammar-constrained generation with low overhead.
An OpenAI-compatible API supports /v1/chat/completions, /v1/completions, and related endpoints so existing clients can switch backends with little or no change.

What you'll accomplish

Serve a model with SGLang on your supported hardware platform using a pre-built CUDA 13 container and an OpenAI-compatible endpoint. You will also exercise prefix-cached multi-turn chat and structured JSON output.

What to know before starting

Required:

Basic Docker container usage
Familiarity with REST APIs

Optional:

Python for client scripts and the optional multi-turn benchmark

Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base workflow applies across supported hardware platforms.

Hardware platform	OS	Memory	Multi-node capable hardware
DGX Spark	DGX OS (Linux)	128 GB Unified Memory	—
DGX Station	DGX OS (Linux)	Large HBM + Grace DRAM	—

Prerequisites

Hardware requirements

Supported hardware platform — see Supported hardware platforms matrix above
Sufficient memory for your chosen model (see Supported models below)

Software requirements

Docker installed: docker --version
NVIDIA Container Toolkit configured (nvidia-smi works inside a GPU container)
HuggingFace account with an access token (for gated / private model downloads)
Network access to Docker Hub and HuggingFace
SGLang container image — see Instructions
git — only for the optional benchmark and offline-inference scripts (see Ancillary files below); Steps 1–7 need no local files

Ancillary files

Steps 1–7 of the Instructions tab run entirely against the container and need nothing on the host. The two optional Python scripts live under assets/ in this playbook and require cloning the repository (Instructions → Step 8):

File	Purpose
`assets/benchmark_multiturn.py`	Multi-turn throughput and prefix-cache benchmark (Instructions → Step 8)
`assets/offline-inference.py`	In-process SGLang Engine example, no server required (Instructions → Next steps)

Supported models

Use the matrices below to pick a model for your hardware platform. Full serve workflow is in the Instructions tab.

NOTE

Memory determines what you can run. Large models need substantially more memory. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in Instructions.

DGX Spark

Models validated with SGLang on DGX Spark:

Model	Quantization	HF Handle
Nemotron-3-Nano-Omni-30B-A3B-Reasoning	BF16	`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`
GPT-OSS-20B	MXFP4	`openai/gpt-oss-20b`
GPT-OSS-120B	MXFP4	`openai/gpt-oss-120b`
Llama-3.1-8B-Instruct	FP8	`nvidia/Llama-3.1-8B-Instruct-FP8`
Llama-3.1-8B-Instruct	NVFP4	`nvidia/Llama-3.1-8B-Instruct-FP4`
Llama-3.3-70B-Instruct	NVFP4	`nvidia/Llama-3.3-70B-Instruct-FP4`
Qwen3-8B	FP8 / NVFP4	`nvidia/Qwen3-8B-FP8` / `nvidia/Qwen3-8B-FP4`
Qwen3-14B	FP8 / NVFP4	`nvidia/Qwen3-14B-FP8` / `nvidia/Qwen3-14B-FP4`
Qwen3-32B	NVFP4	`nvidia/Qwen3-32B-FP4`
Phi-4-multimodal-instruct	FP8 / NVFP4	`nvidia/Phi-4-multimodal-instruct-FP8` / `nvidia/Phi-4-multimodal-instruct-FP4`
Phi-4-reasoning-plus	FP8 / NVFP4	`nvidia/Phi-4-reasoning-plus-FP8` / `nvidia/Phi-4-reasoning-plus-FP4`

For NVFP4 models, add --quantization modelopt_fp4 to the serve command. Certain models (for example Nemotron-3-Nano-Omni) may require extra flags from their HuggingFace model card.

DGX Station

Starting points documented for SGLang on DGX Station (confirm memory headroom and SGLang build support before large downloads):

Model	Role	HF Handle
Qwen3-8B	Default first-run (fast validation)	`Qwen/Qwen3-8B`
Qwen3.6-35B-A3B	MoE (~3B active); hybrid mamba/SSM — prefix-cache check in Instructions does not apply	`Qwen/Qwen3.6-35B-A3B`
Qwen3.6-27B	Dense Qwen3.6	`Qwen/Qwen3.6-27B`
Llama-3.3-70B-Instruct	Gated on Hugging Face — accept license before download	`meta-llama/Llama-3.3-70B-Instruct`
DeepSeek-V4-Flash	Large local MoE when memory allows	`deepseek-ai/DeepSeek-V4-Flash`
DeepSeek-V4-Pro	Larger V4 variant — only with sufficient memory and a supported SGLang build	`deepseek-ai/DeepSeek-V4-Pro`

You may also use other Hugging Face text-generation or chat checkpoints that your SGLang build supports.

Time & risk

Estimated time: 30 MIN (longer on first run due to model download and CUDA-graph capture; larger MoE models can take 45–60 MIN)
Risk level: Low
- Model download may require HuggingFace authentication for gated models
Rollback: Stop and remove the container to restore state (non-destructive)
Last Updated: 07/31/2026
- Documented how to obtain the assets/ scripts and defined the playbook root; cached-model cleanup now uses sudo
- Replaced recipe-portal links with Supported models matrices (Spark + Station); removed Agent-ready Models tab