Serve LLMs with SGLang
30 MIN
High-throughput serving with RadixAttention, structured output, and an OpenAI-compatible API
Common issues
The Hardware platform column shows where an issue is most relevant. "All hardware platforms" applies to every platform listed in this playbook.
| Symptom | Hardware platform | Cause | Fix |
|---|---|---|---|
| "permission denied" when running docker | All hardware platforms | User not in docker group | Run sudo usermod -aG docker $USER && newgrp docker |
| Container fails to start with GPU error | All hardware platforms | NVIDIA Container Toolkit not configured | Run nvidia-ctk runtime configure --runtime=docker and restart Docker |
| HuggingFace authentication failure, gated model access denied, or model download hangs/fails | All hardware platforms | Missing/invalid token, restricted model access, or network issue | Export HF_TOKEN before running docker; regenerate your HuggingFace token and request access to the gated model if needed |
| CUDA out of memory / server exits with OOM | All hardware platforms | Model or context too large for available memory | Lower --mem-fraction-static (for example 0.7) and/or reduce --context-length |
| Server not responding on port 30000 / connection refused | All hardware platforms | Server still loading, or port already in use | Check docker logs sglang-server; wait for readiness; or use -p 30001:30000 if the port is busy |
json_schema response_format returns error | All hardware platforms | Container image missing schema-constrained decoding support | Use lmsysorg/sglang:latest-cu130 |
| Slow first request after server start | All hardware platforms | Kernel JIT + CUDA-graph capture | Wait for the ready message in logs; subsequent requests are fast |
Med cached prefill / cached_tokens is n/a or 0 unexpectedly | All hardware platforms | Cache report not enabled, or hybrid mamba/SSM model | Add --enable-cache-report; for mamba/SSM models (for example Qwen3.6-35B-A3B), zero cached tokens across requests is expected — validate with a standard-attention model |
python3: can't open file 'assets/benchmark_multiturn.py': [Errno 2] No such file or directory | All hardware platforms | Playbook repository not cloned, or command not run from the playbook root | Clone the repository and cd into nvidia/playbook-sglang (the directory containing assets/) as shown in Instructions → Step 8 |
rm: cannot remove '.../.cache/huggingface/hub/models--...': Permission denied | All hardware platforms | The container downloads weights as root into the mounted hub cache, so cached model files are root-owned | Remove with sudo rm -rf $HOME/.cache/huggingface/hub/"<downloaded model name>" |
device >= 0 && device < num_gpus INTERNAL ASSERT FAILED | DGX Station | --gpus '"device=N"' index does not exist | Re-run nvidia-smi --query-gpu=index,name --format=csv,noheader and use the GB300 index, or --gpus all if there is only one GPU |
RuntimeError: ... buildNdTmaDescriptor ... Check failed: false during CUDA-graph capture | DGX Station | Default trtllm_mha attention backend incompatible with GB300 / SM103 | Pass --attention-backend flashinfer |
AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90 | DGX Station | --attention-backend fa3 on Blackwell SM103 | Use --attention-backend flashinfer |
User lacks permission to set NUMA affinity warning | DGX Station | Docker dropped SYS_NICE | Add --cap-add SYS_NICE to docker run |
| Memory pressure within capacity | DGX Spark | UMA buffer cache not released | See UMA note below |
deepseek-ai/DeepSeek-V4-* fails to load | DGX Station | Unsupported in this SGLang build or insufficient memory | Check SGLang docs for model support; try Flash before Pro; lower --mem-fraction-static and --context-length |
NOTE
Unified memory (UMA). On hardware platforms with unified memory, GPU and CPU share memory dynamically. Some applications have not yet been updated for UMA, so you may hit memory issues even within capacity. If that happens, manually flush the buffer cache:
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
NOTE
Monitoring GPU memory with UMA. Because of unified memory, nvidia-smi --query-gpu memory fields may report N/A. Use plain nvidia-smi instead.
NOTE
On DGX Station the GB300 may be at device 0 or 1 depending on configuration. Always verify with nvidia-smi --query-gpu=index,name --format=csv,noheader before pinning a device.