---
title: "Serve LLMs with SGLang — Troubleshooting"
canonical: "https://build.nvidia.com/station/sglang/troubleshooting.md"
---

# Common issues

The **Hardware platform** column shows where an issue is most relevant. "All hardware platforms" applies to every platform listed in this playbook.

| Symptom | Hardware platform | Cause | Fix |
|---------|-------------------|-------|-----|
| "permission denied" when running docker | All hardware platforms | User not in docker group | Run `sudo usermod -aG docker $USER && newgrp docker` |
| Container fails to start with GPU error | All hardware platforms | NVIDIA Container Toolkit not configured | Run `nvidia-ctk runtime configure --runtime=docker` and restart Docker |
| HuggingFace authentication failure, gated model access denied, or model download hangs/fails | All hardware platforms | Missing/invalid token, restricted model access, or network issue | Export `HF_TOKEN` before running docker; regenerate your [HuggingFace token](https://huggingface.co/docs/hub/en/security-tokens) and request access to the [gated model](https://huggingface.co/docs/hub/en/models-gated) if needed |
| CUDA out of memory / server exits with OOM | All hardware platforms | Model or context too large for available memory | Lower `--mem-fraction-static` (for example `0.7`) and/or reduce `--context-length` |
| Server not responding on port 30000 / connection refused | All hardware platforms | Server still loading, or port already in use | Check `docker logs sglang-server`; wait for readiness; or use `-p 30001:30000` if the port is busy |
| `json_schema` response_format returns error | All hardware platforms | Container image missing schema-constrained decoding support | Use `lmsysorg/sglang:latest-cu130` |
| Slow first request after server start | All hardware platforms | Kernel JIT + CUDA-graph capture | Wait for the ready message in logs; subsequent requests are fast |
| `Med cached prefill` / `cached_tokens` is `n/a` or 0 unexpectedly | All hardware platforms | Cache report not enabled, or hybrid mamba/SSM model | Add `--enable-cache-report`; for mamba/SSM models (for example Qwen3.6-35B-A3B), zero cached tokens across requests is expected — validate with a standard-attention model |
| `python3: can't open file 'assets/benchmark_multiturn.py': [Errno 2] No such file or directory` | All hardware platforms | Playbook repository not cloned, or command not run from the playbook root | Clone the repository and `cd` into `nvidia/playbook-sglang` (the directory containing `assets/`) as shown in Instructions → Step 8 |
| `rm: cannot remove '.../.cache/huggingface/hub/models--...': Permission denied` | All hardware platforms | The container downloads weights as root into the mounted hub cache, so cached model files are root-owned | Remove with `sudo rm -rf $HOME/.cache/huggingface/hub/"<downloaded model name>"` |
| `device >= 0 && device < num_gpus INTERNAL ASSERT FAILED` | DGX Station | `--gpus '"device=N"'` index does not exist | Re-run `nvidia-smi --query-gpu=index,name --format=csv,noheader` and use the GB300 index, or `--gpus all` if there is only one GPU |
| `RuntimeError: ... buildNdTmaDescriptor ... Check failed: false` during CUDA-graph capture | DGX Station | Default `trtllm_mha` attention backend incompatible with GB300 / SM103 | Pass `--attention-backend flashinfer` |
| `AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90` | DGX Station | `--attention-backend fa3` on Blackwell SM103 | Use `--attention-backend flashinfer` |
| `User lacks permission to set NUMA affinity` warning | DGX Station | Docker dropped `SYS_NICE` | Add `--cap-add SYS_NICE` to `docker run` |
| Memory pressure within capacity | DGX Spark | UMA buffer cache not released | See UMA note below |
| `deepseek-ai/DeepSeek-V4-*` fails to load | DGX Station | Unsupported in this SGLang build or insufficient memory | Check [SGLang docs](https://docs.sglang.io/) for model support; try Flash before Pro; lower `--mem-fraction-static` and `--context-length` |

> [!NOTE]
> **Unified memory (UMA).** On hardware platforms with unified memory, GPU and CPU share memory dynamically. Some applications have not yet been updated for UMA, so you may hit memory issues even within capacity. If that happens, manually flush the buffer cache:
> ```bash
> sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
> ```

> [!NOTE]
> **Monitoring GPU memory with UMA.** Because of unified memory, `nvidia-smi --query-gpu` memory fields may report `N/A`. Use plain `nvidia-smi` instead.

> [!NOTE]
> On DGX Station the GB300 may be at device `0` or `1` depending on configuration. Always verify with `nvidia-smi --query-gpu=index,name --format=csv,noheader` before pinning a device.