Serve LLMs with SGLang
High-throughput serving with RadixAttention, structured output, and an OpenAI-compatible API
Basic idea
SGLang is a high-performance serving framework for large language models and vision-language models. It co-designs the backend runtime and frontend language so interactions are faster and more controllable — especially for workloads that share prefixes.
- RadixAttention automatically caches and reuses KV cache entries across requests that share common prefixes (multi-turn chat, RAG, agents), reducing redundant prefill work.
- Structured output uses compressed finite-state machine decoding (xGrammar) for JSON, regex, and grammar-constrained generation with low overhead.
- An OpenAI-compatible API supports
/v1/chat/completions,/v1/completions, and related endpoints so existing clients can switch backends with little or no change.
What you'll accomplish
Serve a model with SGLang on your supported hardware platform using a pre-built CUDA 13 container and an OpenAI-compatible endpoint. You will also exercise prefix-cached multi-turn chat and structured JSON output.
What to know before starting
Required:
- Basic Docker container usage
- Familiarity with REST APIs
Optional:
- Python for client scripts and the optional multi-turn benchmark
Supported hardware platforms
Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The same base workflow applies across supported hardware platforms.
| Hardware platform | OS | Memory | Multi-node capable hardware |
|---|---|---|---|
| DGX Spark | DGX OS (Linux) | 128 GB Unified Memory | — |
| DGX Station | DGX OS (Linux) | Large HBM + Grace DRAM | — |
Prerequisites
Hardware requirements
- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for your chosen model (see Supported models below)
Software requirements
- Docker installed:
docker --version - NVIDIA Container Toolkit configured (
nvidia-smiworks inside a GPU container) - HuggingFace account with an access token (for gated / private model downloads)
- Network access to Docker Hub and HuggingFace
- SGLang container image — see Instructions
git— only for the optional benchmark and offline-inference scripts (see Ancillary files below); Steps 1–7 need no local files
Ancillary files
Steps 1–7 of the Instructions tab run entirely against the container and need nothing on the host. The two optional Python scripts live under assets/ in this playbook and require cloning the repository (Instructions → Step 8):
| File | Purpose |
|---|---|
assets/benchmark_multiturn.py | Multi-turn throughput and prefix-cache benchmark (Instructions → Step 8) |
assets/offline-inference.py | In-process SGLang Engine example, no server required (Instructions → Next steps) |
Supported models
Use the matrices below to pick a model for your hardware platform. Full serve workflow is in the Instructions tab.
NOTE
Memory determines what you can run. Large models need substantially more memory. If a model is not listed for your hardware platform, check whether it fits in available memory and try the base configuration in Instructions.
DGX Spark
Models validated with SGLang on DGX Spark:
| Model | Quantization | HF Handle |
|---|---|---|
| Nemotron-3-Nano-Omni-30B-A3B-Reasoning | BF16 | nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 |
| GPT-OSS-20B | MXFP4 | openai/gpt-oss-20b |
| GPT-OSS-120B | MXFP4 | openai/gpt-oss-120b |
| Llama-3.1-8B-Instruct | FP8 | nvidia/Llama-3.1-8B-Instruct-FP8 |
| Llama-3.1-8B-Instruct | NVFP4 | nvidia/Llama-3.1-8B-Instruct-FP4 |
| Llama-3.3-70B-Instruct | NVFP4 | nvidia/Llama-3.3-70B-Instruct-FP4 |
| Qwen3-8B | FP8 / NVFP4 | nvidia/Qwen3-8B-FP8 / nvidia/Qwen3-8B-FP4 |
| Qwen3-14B | FP8 / NVFP4 | nvidia/Qwen3-14B-FP8 / nvidia/Qwen3-14B-FP4 |
| Qwen3-32B | NVFP4 | nvidia/Qwen3-32B-FP4 |
| Phi-4-multimodal-instruct | FP8 / NVFP4 | nvidia/Phi-4-multimodal-instruct-FP8 / nvidia/Phi-4-multimodal-instruct-FP4 |
| Phi-4-reasoning-plus | FP8 / NVFP4 | nvidia/Phi-4-reasoning-plus-FP8 / nvidia/Phi-4-reasoning-plus-FP4 |
For NVFP4 models, add --quantization modelopt_fp4 to the serve command. Certain models (for example Nemotron-3-Nano-Omni) may require extra flags from their HuggingFace model card.
DGX Station
Starting points documented for SGLang on DGX Station (confirm memory headroom and SGLang build support before large downloads):
| Model | Role | HF Handle |
|---|---|---|
| Qwen3-8B | Default first-run (fast validation) | Qwen/Qwen3-8B |
| Qwen3.6-35B-A3B | MoE (~3B active); hybrid mamba/SSM — prefix-cache check in Instructions does not apply | Qwen/Qwen3.6-35B-A3B |
| Qwen3.6-27B | Dense Qwen3.6 | Qwen/Qwen3.6-27B |
| Llama-3.3-70B-Instruct | Gated on Hugging Face — accept license before download | meta-llama/Llama-3.3-70B-Instruct |
| DeepSeek-V4-Flash | Large local MoE when memory allows | deepseek-ai/DeepSeek-V4-Flash |
| DeepSeek-V4-Pro | Larger V4 variant — only with sufficient memory and a supported SGLang build | deepseek-ai/DeepSeek-V4-Pro |
You may also use other Hugging Face text-generation or chat checkpoints that your SGLang build supports.
Time & risk
- Estimated time: 30 MIN (longer on first run due to model download and CUDA-graph capture; larger MoE models can take 45–60 MIN)
- Risk level: Low
- Model download may require HuggingFace authentication for gated models
- Rollback: Stop and remove the container to restore state (non-destructive)
- Last Updated: 07/31/2026
- Documented how to obtain the
assets/scripts and defined the playbook root; cached-model cleanup now usessudo - Replaced recipe-portal links with Supported models matrices (Spark + Station); removed Agent-ready Models tab
- Documented how to obtain the