SGLang for Inference

Use model specific deployment guide

Certain models require special deployment configurations. Please refer to their respective model cards to run on DGX Spark:

Verify system prerequisites

Check that your NVIDIA Spark device meets all requirements before proceeding. This step runs on your host system and ensures Docker, GPU drivers, and container toolkit are properly configured.

Note: If you experience timeouts or "connection refused" errors while pulling the container image, you may need to use a VPN or a proxy, as some registries may be restricted by your local network or ISP.

# Use the latest CUDA 13.0 SGLang image.
export SGLANG_IMAGE="lmsysorg/sglang:latest-cu130"

# Verify Docker installation
docker --version

# Check NVIDIA GPU drivers
nvidia-smi

# Verify Docker GPU support
docker run --rm --gpus all "$SGLANG_IMAGE" nvidia-smi

# Check available disk space
df -h /

If you see a permission denied error (something like permission denied while trying to connect to the Docker daemon socket), add your user to the docker group so that you don't need to run the command with sudo .

sudo usermod -aG docker $USER
newgrp docker

Pull the SGLang Container

Download the latest SGLang container. This step runs on the host and may take several minutes depending on your network connection.

export SGLANG_IMAGE="lmsysorg/sglang:latest-cu130"

# Pull the SGLang container
docker pull "$SGLANG_IMAGE"

# Verify the image was downloaded
docker images --digests | grep sglang

Launch SGLang container for server mode

Start the SGLang container in server mode to enable HTTP API access. This runs the inference server inside the container, exposing it on port 30000 for client connections.

export SGLANG_IMAGE="lmsysorg/sglang:latest-cu130"

# Launch container with GPU support and port mapping
docker run --gpus all -it --rm \
  -p 30000:30000 \
  -v /tmp:/tmp \
  "$SGLANG_IMAGE" \
  bash

Start the SGLang inference server

Inside the container, launch the HTTP inference server with a supported model. This step runs inside the Docker container and starts the SGLang server daemon.

# Start the inference server with DeepSeek-V2-Lite model
python3 -m sglang.launch_server \
  --model-path deepseek-ai/DeepSeek-V2-Lite \
  --host 0.0.0.0 \
  --port 30000 \
  --trust-remote-code \
  --tp 1 \
  --attention-backend flashinfer \
  --mem-fraction-static 0.75 &

# Wait for the server to become ready
timeout 900 bash -c 'until curl -sf http://localhost:30000/health > /dev/null; do sleep 5; done'

# Check server status
curl -f http://localhost:30000/health

Test client-server inference

From a new terminal on your host system, test the SGLang server API to ensure it's working correctly. This validates that the server is accepting requests and generating responses.

# Test with curl
curl -f -X POST http://localhost:30000/generate \
  -H "Content-Type: application/json" \
  -d '{
      "text": "What does NVIDIA love?",
      "sampling_params": {
          "temperature": 0.7,
          "max_new_tokens": 100
      }
  }'

Test Python client API

Test programmatic access to the SGLang server from the host system. This demonstrates how to integrate SGLang into Python applications.

python3 << 'EOF'
import requests

# Send prompt to server
response = requests.post('http://localhost:30000/generate', json={
  'text': 'What does NVIDIA love?',
  'sampling_params': {
      'temperature': 0.7,
      'max_new_tokens': 100,
  },
})

print(f"Response: {response.json()['text']}")
EOF

Validate installation

Confirm that both server and offline modes are working correctly. This step verifies the complete SGLang setup and ensures reliable operation.

# Check server mode (from host)
curl -f http://localhost:30000/health
curl -f -X POST http://localhost:30000/generate -H "Content-Type: application/json" \
  -d '{"text": "Hello", "sampling_params": {"max_new_tokens": 10}}'

# Check container logs
docker ps
docker logs <CONTAINER_ID>

Cleanup and rollback

Stop and remove containers to clean up resources. This step returns your system to its original state.

# Stop all SGLang containers
docker ps | grep sglang | awk '{print $1}' | xargs docker stop

# Remove stopped containers
docker container prune -f

# Remove SGLang images (optional)
docker rmi lmsysorg/sglang:latest-cu130

Next steps

With SGLang successfully deployed, you can now:

• Integrate the HTTP API into your applications using the /generate endpoint
• Experiment with different models by changing the --model-path parameter
• Scale up using multiple GPUs by adjusting the --tp (tensor parallel) setting
• Deploy production workloads using the container orchestration platform of your choice