Serve LLMs with vLLM
30 MIN
High-throughput serving for 30+ models, with continuous batching and an OpenAI-compatible API
Common issues
The Hardware platform column shows where an issue is most relevant. "All hardware platforms" applies to every platform.
| Symptom | Hardware platform | Cause | Fix |
|---|---|---|---|
| "permission denied" when running docker | All hardware platforms | User not in docker group | Run sudo usermod -aG docker $USER && newgrp docker |
| Container fails to start with GPU error | All hardware platforms | NVIDIA Container Toolkit not configured | Run nvidia-ctk runtime configure --runtime=docker and restart Docker |
| HuggingFace authentication failure, gated model access denied, or model download hangs/fails | All hardware platforms | Missing/invalid token, restricted model access, or network issue | Export HF_TOKEN before running docker; regenerate your HuggingFace token and request access to the gated model if needed; check internet connection and verify the token is valid |
| CUDA out of memory | All hardware platforms | Context length too large / model too big | Reduce --max-model-len and --max-num-seqs, or lower --gpu-memory-utilization |
| Server not responding on port 8000 | All hardware platforms | Port already in use | Check with lsof -i :8000; use -p 8001:8000 for a different port |
| NGC authentication fails | All hardware platforms | Invalid or missing credentials | Run docker login nvcr.io with your NGC API key |
| Memory pressure within capacity | DGX Spark | UMA buffer cache not released | See UMA note below |
| Container startup fails / missing ARM64 image | DGX Spark | Image not built for ARM64 | Use the default NGC image for your hardware platform from the Instructions tab |
| Model runs on wrong GPU | DGX Station | Default GPU selection with two GPUs | Use --gpus '"device=N"' to pin the GB300 (N from nvidia-smi) |
| EngineCore failed / FlashInfer "Buffer overflow when allocating memory for batch_prefill_tmp_v" | DGX Station | CUDA graph capture failure during batch prefill | Use the recommended container image: nvcr.io/nvidia/vllm:26.01-py3 |
| Node not visible in Ray cluster | multi-node capable hardware | Network connectivity issue | Verify QSFP cable connection and IP configuration; see Multi-node serving tab |
Chat completion returns content: null with finish_reason: length | max_tokens exhausted on the thinking pass before an answer | Raise max_tokens in the request (Step 5 uses 2048) so reasoning-enabled recipes can finish with a visible answer |
NOTE
Unified memory (UMA). On hardware platforms with unified memory, GPU and CPU share memory dynamically. Some applications have not yet been updated for UMA, so you may hit memory issues even within capacity. If that happens, manually flush the buffer cache:
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
NOTE
Monitoring GPU memory with UMA. Because of unified memory, nvidia-smi --query-gpu memory fields report N/A. Use plain nvidia-smi instead.