LLM Inference with SGLang

Set up Docker permissions

If you haven't already, add your user to the docker group to run Docker without sudo:

sudo usermod -aG docker $USER
newgrp docker

Set up environment variables

# HuggingFace token (only required for gated models such as Llama 3.3).
# Leave empty for public models like Qwen3-8B; for gated models get a token at
# https://huggingface.co/settings/tokens.
export HF_TOKEN=""

# Model to serve (see **Example model IDs** below).
# Default uses Qwen3-8B for fast first-run validation (~10–15 min boot on Station).
# Switch to Qwen3.6-35B-A3B once the workflow is working end-to-end.
export MODEL_HANDLE="Qwen/Qwen3-8B"

# Maximum context length
export MAX_MODEL_LEN=8192

Example model IDs (MODEL_HANDLE)

Use any Hugging Face text-generation or chat checkpoint that your SGLang build supports. The table below lists common starting points on DGX Station; always check the model card for license / gated access, VRAM, and context length.

Heavyweight MoE (very large weights; confirm SGLang version + GPU memory before serving):

Choosing an inference backend (DGX Station)

Several OpenAI-compatible servers run well on NVIDIA hardware. None is universally “best”—pick by workload shape and operational constraints.

This playbook focuses on SGLang; consult each project’s documentation for model support matrices and quantization modes.

Pull the SGLang container

Pull the SGLang container image with CUDA 13.0 support (required for Blackwell SM103):

docker pull lmsysorg/sglang:latest-cu130

Identify the GB300 GPU

Identify the GB300's device index:

nvidia-smi --query-gpu=index,name --format=csv,noheader

Look for the row showing NVIDIA GB300. Note its index — on DGX Station the GB300 may be at index 0 or 1 depending on configuration. If nvidia-smi shows only a single GB300, you can simply use --gpus all in the next step.

Start SGLang server

Launch the SGLang server. The flags below are tuned for GB300 (Blackwell SM103) — see notes after the command:

# Use --gpus all on a single-GPU Station, or --gpus '"device=N"' with the
# index from Step 4 if multiple GPUs are present.
docker run -d \
  --name sglang-server \
  --gpus all \
  --ipc host \
  --cap-add SYS_NICE \
  --ulimit memlock=-1 \
  --ulimit stack=67108864 \
  -p 30000:30000 \
  -e HF_TOKEN="$HF_TOKEN" \
  -v "$HOME/.cache/huggingface/hub:/root/.cache/huggingface/hub" \
  lmsysorg/sglang:latest-cu130 \
  sglang serve --model-path "$MODEL_HANDLE" \
    --host 0.0.0.0 \
    --port 30000 \
    --context-length $MAX_MODEL_LEN \
    --mem-fraction-static 0.85 \
    --attention-backend flashinfer \
    --enable-cache-report

Check the server logs:

docker logs -f sglang-server

Wait for the server to show it is ready:

INFO:     Uvicorn running on http://0.0.0.0:30000

Press Ctrl+C to exit the log view.

Test basic inference

Send a chat completion request using the OpenAI-compatible API:

curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [{"role": "user", "content": "Explain quantum computing in simple terms."}],
    "max_tokens": 256
  }'

The response follows the standard OpenAI format with a choices array containing the model's answer.

Multi-turn conversation with prefix caching

SGLang's RadixAttention automatically caches the KV cache for processed tokens. When follow-up messages share the same conversation prefix, the cached entries are reused — skipping prefill for all previously seen tokens.

Send a multi-turn conversation. The system prompt is deliberately long so the shared prefix exceeds SGLang's page size (64 tokens), which is the minimum unit for cache reuse:

# Turn 1
curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [
      {"role": "system", "content": "You are an expert physics tutor who explains concepts clearly and concisely. You use real-world analogies and everyday examples to make abstract ideas concrete. When answering, first state the key concept in one sentence, then give a short explanation with an example."},
      {"role": "user", "content": "What is the difference between speed and velocity?"}
    ],
    "max_tokens": 256
  }' | python3 -m json.tool

# Turn 2 — extends the same conversation
curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [
      {"role": "system", "content": "You are an expert physics tutor who explains concepts clearly and concisely. You use real-world analogies and everyday examples to make abstract ideas concrete. When answering, first state the key concept in one sentence, then give a short explanation with an example."},
      {"role": "user", "content": "What is the difference between speed and velocity?"},
      {"role": "assistant", "content": "Speed is a scalar quantity that measures how fast an object moves, while velocity is a vector quantity that includes both speed and direction. For example, a car driving at 60 km/h has a speed of 60 km/h regardless of where it is headed. But if that car is driving 60 km/h north, that is its velocity — change direction to south and the velocity changes even though the speed stays the same."},
      {"role": "user", "content": "Can you give me another example that shows why the distinction matters in real physics problems?"}
    ],
    "max_tokens": 256
  }' | python3 -m json.tool

The second request reuses the KV cache for the shared prefix (system message + first user turn + assistant response) via RadixAttention, so repeated prefill work on that prefix is avoided. End-to-end HTTP latency can still go up on later turns: the transcript is longer (more tokens to attend to even with cache hits on the prefix), each assistant reply adds decode work, and the client measures full request time—not prefill alone.

Check cache reuse in the server logs. SGLang logs each prefill batch with the number of cached tokens reused:

docker logs sglang-server 2>&1 | grep "cached-token" | tail -10

Look for #cached-token values greater than 0 on later turns — this confirms RadixAttention is reusing the KV cache from the shared prefix. Treat that as the primary signal of prefix caching; wall-clock curl latency alone can be misleading.

Structured JSON output

SGLang's constrained decoding guarantees valid JSON output matching a schema. This uses the xGrammar backend to overlap grammar mask generation with the model's forward pass, adding minimal latency.

Generate a structured response:

curl -s http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "'"$MODEL_HANDLE"'",
    "messages": [
      {"role": "user", "content": "List three programming languages with their primary use case and year created."}
    ],
    "max_tokens": 512,
    "response_format": {
      "type": "json_schema",
      "json_schema": {
        "name": "languages",
        "schema": {
          "type": "object",
          "properties": {
            "languages": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": {"type": "string"},
                  "primary_use": {"type": "string"},
                  "year_created": {"type": "integer"}
                },
                "required": ["name", "primary_use", "year_created"]
              }
            }
          },
          "required": ["languages"]
        }
      }
    }
  }' | python3 -m json.tool

The response content is guaranteed to be valid JSON matching the provided schema. Parse the choices[0].message.content field — it will contain a well-formed JSON object.

Benchmark multi-turn throughput

This step uses benchmark_multiturn.py from this playbook's assets/ directory. Clone (or download) the playbook repository first so the script is available locally:

git clone https://github.com/NVIDIA/dgx-spark-playbooks
cd dgx-spark-playbooks/nvidia/station-sglang-inference

The benchmark stress-tests the server with parallel conversations (default: 20) and reports per-turn wall time, token counts, and (when the API exposes it) cached prefill tokens.

Install the requests dependency. The virtualenv approach below is the preferred, default installation path — it keeps the script's dependencies isolated from the system Python interpreter so you cannot accidentally damage Ubuntu's own Python packages. Ubuntu 24.04 on DGX Station does not ship python3-venv by default, so install it once before creating the virtualenv:

sudo apt update && sudo apt install -y python3-venv
python3 -m venv .venv && source .venv/bin/activate
pip install requests

If you cannot run sudo apt install python3-venv (for example, a locked-down host), the next safest option is a per-user install that still respects PEP 668:

python3 -m pip install --user requests

python3 -m pip install --user --break-system-packages requests

python3 assets/benchmark_multiturn.py \
  --base-url http://localhost:30000 \
  --model "$MODEL_HANDLE" \
  --num-conversations 20 \
  --turns-per-conversation 5 \
  --cache-detail-file ./sglang_benchmark_cache_details.log

The script prints:

• Median / P90 wall time per turn — often increases as prompts grow and under parallel load; that does not contradict RadixAttention.
• Median prompt tokens per turn — should climb as history lengthens.
• Median cached prefill tokens (when returned in usage) — populated by --enable-cache-report (already set in Step 5); this is the primary cache signal from the OpenAI-style usage payload.
• A short summary of cache-related /server_info or /metrics lines; the full responses are written to --cache-detail-file (default ./sglang_benchmark_cache_details.log) so you are not flooded with an unparsed metrics blob in the terminal.

To isolate prefix-cache behavior from multi-client contention, rerun with a single conversation:

python3 assets/benchmark_multiturn.py \
  --base-url http://localhost:30000 \
  --model "$MODEL_HANDLE" \
  --num-conversations 1 \
  --turns-per-conversation 5

Always correlate behavior with docker logs (#cached-token lines) as in Step 7.

Next steps: heavier models on Station

To stress GPU memory and throughput after completing the steps above, point MODEL_HANDLE at a larger checkpoint (for example deepseek-ai/DeepSeek-V4-Flash), lower --mem-fraction-static if you hit OOM, and reduce --context-length until the server starts cleanly. Confirm your SGLang image version supports the architecture (see SGLang documentation) and accept any gated model licenses on Hugging Face before pulling weights.

Cleanup

Stop and remove the container:

docker stop sglang-server
docker rm sglang-server

Optionally remove the image:

docker rmi lmsysorg/sglang:latest-cu130