LLM Inference with SGLang

Basic idea

SGLang is a high-performance serving framework for large language models, optimized for workloads where requests share common prefixes — multi-turn conversations, RAG pipelines, and agentic workflows. Its core innovation, RadixAttention, automatically caches and reuses KV cache entries across requests using a radix tree, eliminating redundant prefill computation. SGLang also provides best-in-class structured output generation (JSON, regex, grammar-constrained decoding) through its xGrammar backend, running up to 3x faster than standard guided decoding.

• RadixAttention — Automatically reuses KV cache across requests sharing common prefixes. Multi-turn conversations and repeated system prompts skip re-computation entirely, reducing first-token latency and increasing throughput.
• Structured output — Compressed finite-state machine decoding with grammar mask generation overlapped with the LLM forward pass. Produces valid JSON, regex-matched, or grammar-constrained output with minimal overhead.
• OpenAI-compatible API — Drop-in replacement for OpenAI and vLLM endpoints. Supports /v1/chat/completions, /v1/completions, and /v1/embeddings.
• Blackwell optimized — SGLang includes optimizations for SM100+ GPUs and CUDA 13, with NVFP4 GEMM support and accelerated softmax kernels.

What you'll accomplish

Launch SGLang on DGX Station to serve an LLM, then exercise its two key differentiators: prefix-cached multi-turn chat and structured JSON output generation. You will also benchmark multi-turn throughput to see RadixAttention's effect.

• Serve Qwen3-8B with SGLang's Blackwell-optimized backend
• Send multi-turn conversations and observe prefix cache hits in server metrics
• Generate structured JSON output using schema-constrained decoding
• Benchmark multi-turn throughput with and without prefix caching

What to know before starting

• Basic Docker container usage
• Familiarity with REST APIs (curl or Python requests)

Prerequisites

• NVIDIA DGX Station with GB300 GPU (Blackwell SM103)
• Docker installed: docker --version
• NVIDIA Container Toolkit configured: nvidia-smi should show the GB300
• HuggingFace account with access token
• Network access to HuggingFace and Docker Hub

Ancillary files

• assets/benchmark_multiturn.py — Python script that benchmarks multi-turn conversation throughput and demonstrates structured output generation

Time & risk

• Duration: 20–25 minutes (including model download)
• Risks: Model download requires HuggingFace authentication
• Rollback: Stop and remove the container to restore state
• Last Updated: 04/06/2026
  • First Publication