Serve LLMs with vLLM

Multi-node serving

Serve models larger than a single node can hold by pooling GPUs across multiple multi-node capable hardware systems with a Ray cluster and tensor parallelism. Two topologies are covered:

Two nodes (direct QSFP cable) — connect two nodes back-to-back.
Four or more nodes through a QSFP switch — scale out over a switch fabric.

This tab applies to multi-node capable hardware only. Other supported hardware platforms serve models on a single node (see the Instructions tab).

Prerequisites

Docker permissions

If docker ps fails with a permission error, complete Step 1 in the Instructions tab on every node in the cluster before continuing.

A. Two nodes (direct QSFP cable)

Step 1. Configure network connectivity

Follow the Connect two nodes for distributed workloads playbook to establish connectivity on multi-node capable hardware: physical QSFP cable, network interface configuration, passwordless SSH, and connectivity verification.

Heads up: the connectivity script from that playbook writes its SSH key to ~/.ssh/ and fails if the directory does not exist. Run mkdir -p ~/.ssh && chmod 700 ~/.ssh on both nodes first if you have never used SSH on them.

Step 2. Download the cluster deployment script

On both nodes, download and patch the Ray cluster script:

Shell

wget https://raw.githubusercontent.com/vllm-project/vllm/51c1ee9b7c8acbba4899a8ebffd390685d171946/examples/ray_serving/run_cluster.sh

sed -i 's|^RAY_START_CMD="ray start|RAY_START_CMD="pip install -q --root-user-action=ignore '\''ray[default]>=2.9'\'' \&\& ray start|' run_cluster.sh

chmod +x run_cluster.sh

Step 3. Pull the NGC vLLM image

Pull the image on both nodes:

Shell

docker pull nvcr.io/nvidia/vllm:26.05-py3
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.05-py3

Step 4. Start the Ray head node (Node 1)

Run inside tmux/screen so an SSH drop doesn't tear down the cluster (run_cluster.sh has an EXIT trap that stops the container).

Set MN_IF_NAME to the QSFP interface name from your connectivity playbook (validated example on multi-node capable hardware: enp1s0f1np1). Substitute if your interface differs.

Shell

export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.05-py3

echo "Using interface $MN_IF_NAME with IP $VLLM_HOST_IP"

bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$VLLM_HOST_IP

Leave this terminal open — closing it stops the head node and tears down the cluster.

Step 5. Start the Ray worker node (Node 2)

Open a second terminal, SSH to Node 2, and join the cluster. Replace <NODE_1_IP_ADDRESS> with Node 1's QSFP IP (run echo $VLLM_HOST_IP on Node 1). Run inside tmux/screen on Node 2 as well. Use the same MN_IF_NAME guidance as Step 4.

Shell

export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
export HEAD_NODE_IP=<NODE_1_IP_ADDRESS>
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.05-py3

echo "Worker IP: $VLLM_HOST_IP, connecting to head node at: $HEAD_NODE_IP"

bash run_cluster.sh $VLLM_IMAGE $HEAD_NODE_IP --worker ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$HEAD_NODE_IP

Step 6. Verify cluster status

Shell

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')
echo "Found container: $VLLM_CONTAINER"
docker exec $VLLM_CONTAINER ray status

Expected output shows 2 nodes with available GPU resources.

Step 7. Download Llama 3.3 70B

Llama 3.3 70B is gated — accept its license at https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct and create an HF token with read permission. Authenticate inside the container so the cache lands at /root/.cache/huggingface:

Shell

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  hf auth login
  hf download meta-llama/Llama-3.3-70B-Instruct'

Step 8. Launch inference server (tensor parallel across both nodes)

Shell

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  vllm serve meta-llama/Llama-3.3-70B-Instruct \
    --tensor-parallel-size 2 --max-model-len 2048 \
    --distributed-executor-backend ray'

Step 9. Test inference

Run on Node 1; from an external client, replace localhost with Node 1's reachable IP.

Shell

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.3-70B-Instruct",
    "prompt": "Write a haiku about a GPU",
    "max_tokens": 32,
    "temperature": 0.7
  }'

Step 10. (Optional) Llama 3.1 405B — two-node topology only

WARNING

Shell

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  hf download hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4'

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  vllm serve hugging-quants/Meta-Llama-3.1-405B-Instruct-AWQ-INT4 \
    --tensor-parallel-size 2 --max-model-len 64 --gpu-memory-utilization 0.9 \
    --max-num-seqs 1 --max-num-batched-tokens 64 \
    --distributed-executor-backend ray'

The server is ready when you see Application startup complete.

B. Four or more nodes through a QSFP switch

Same Ray + tensor-parallel workflow as Section A, scaled to more nodes over a QSFP switch. Set --tensor-parallel-size equal to your node count.

Topology note: the four-or-more-node path uses a different validated container image and run_cluster.sh source than the two-node path above. Follow the steps in this section exactly — do not mix image tags or script versions between topologies.

Step 1. Configure network connectivity

Follow the Connect multiple nodes through a switch playbook for multi-node capable hardware: QSFP cabling between nodes and switch, interface configuration, passwordless SSH, connectivity verification, and the NCCL bandwidth test.

Step 2. Download the cluster deployment script (all nodes)

On every node, download the Ray cluster script:

Shell

wget https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/examples/ray_serving/run_cluster.sh
chmod +x run_cluster.sh

Step 3. Pull the NGC vLLM image (all nodes)

Shell

docker pull nvcr.io/nvidia/vllm:26.02-py3
export VLLM_IMAGE=nvcr.io/nvidia/vllm:26.02-py3

Step 4. Start the Ray head node (Node 1)

Run inside tmux/screen so an SSH drop doesn't tear down the cluster.

Set MN_IF_NAME to the QSFP interface name from your connectivity playbook (validated example on multi-node capable hardware: enp1s0f1np1). Substitute if your interface differs.

Shell

export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')

echo "Using interface $MN_IF_NAME with IP $VLLM_HOST_IP"

bash run_cluster.sh $VLLM_IMAGE $VLLM_HOST_IP --head ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$VLLM_HOST_IP

Leave this terminal open — closing it stops the head node and tears down the cluster.

Step 5. Start the Ray worker nodes (all other nodes)

Repeat the block below on each worker node (Nodes 2 through N). SSH to each node in turn, run inside tmux/screen, and replace <NODE_1_IP_ADDRESS> with Node 1's QSFP interface IP from the switch playbook. Use the same MN_IF_NAME guidance as Step 4.

Shell

export MN_IF_NAME=enp1s0f1np1
export VLLM_HOST_IP=$(ip -4 addr show $MN_IF_NAME | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
export HEAD_NODE_IP=<NODE_1_IP_ADDRESS>

echo "Worker IP: $VLLM_HOST_IP, connecting to head node at: $HEAD_NODE_IP"

bash run_cluster.sh $VLLM_IMAGE $HEAD_NODE_IP --worker ~/.cache/huggingface \
  -e VLLM_HOST_IP=$VLLM_HOST_IP \
  -e UCX_NET_DEVICES=$MN_IF_NAME \
  -e NCCL_SOCKET_IFNAME=$MN_IF_NAME \
  -e OMPI_MCA_btl_tcp_if_include=$MN_IF_NAME \
  -e GLOO_SOCKET_IFNAME=$MN_IF_NAME \
  -e TP_SOCKET_IFNAME=$MN_IF_NAME \
  -e RAY_memory_monitor_refresh_ms=0 \
  -e MASTER_ADDR=$HEAD_NODE_IP

Step 6. Verify cluster status

Shell

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')
docker exec $VLLM_CONTAINER ray status

Expected output shows all nodes with available GPU resources.

Step 7. Download MiniMax M2.5

With four or more nodes you can run this model with tensor parallelism. Authenticate and download inside the head-node container (the cache is shared across the cluster):

Shell

docker exec -it $VLLM_CONTAINER /bin/bash -c '
  hf auth login
  hf download MiniMaxAI/MiniMax-M2.5'

Step 8. Launch inference server (tensor parallel = node count)

Shell

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')
docker exec -it $VLLM_CONTAINER /bin/bash -c '
  vllm serve MiniMaxAI/MiniMax-M2.5 \
    --tensor-parallel-size 4 --max-model-len 129000 --max-num-seqs 4 --trust-remote-code \
    --distributed-executor-backend ray'

Set --tensor-parallel-size to match your node count (example above uses 4).

Step 9. Test inference

Run on Node 1; from an external client, replace localhost with Node 1's reachable IP.

Shell

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MiniMaxAI/MiniMax-M2.5",
    "prompt": "Write a haiku about a GPU",
    "max_tokens": 32,
    "temperature": 0.7
  }'

Validate and monitor (both topologies)

Shell

export VLLM_CONTAINER=$(docker ps --format '{{.Names}}' | grep -E '^node-[0-9]+$')
docker exec $VLLM_CONTAINER ray status

curl http://localhost:8000/health

nvidia-smi

On hardware platforms with unified memory, nvidia-smi --query-gpu memory fields report N/A — use plain nvidia-smi instead.

The Ray dashboard runs on port 8265 of the head node under host networking, so it is only directly reachable from Node 1. Tunnel it from a workstation:

Shell

ssh -L 8265:localhost:8265 nvidia@<NODE_1_IP>
# then open http://localhost:8265

Next steps

Consider for production:

Health checks and automatic restarts
Log rotation for long-running services
Persistent model caching across restarts
Alternative quantization (FP8, NVFP4, INT4) to fit more models on the cluster

Serve LLMs with vLLM

Multi-node serving

Prerequisites

Docker permissions

A. Two nodes (direct QSFP cable)

Step 1. Configure network connectivity

Step 2. Download the cluster deployment script

Step 3. Pull the NGC vLLM image

Step 4. Start the Ray head node (Node 1)

Step 5. Start the Ray worker node (Node 2)

Step 6. Verify cluster status

Step 7. Download Llama 3.3 70B

Step 8. Launch inference server (tensor parallel across both nodes)

Step 9. Test inference

Step 10. (Optional) Llama 3.1 405B — two-node topology only

B. Four or more nodes through a QSFP switch

Step 1. Configure network connectivity

Step 2. Download the cluster deployment script (all nodes)

Step 3. Pull the NGC vLLM image (all nodes)

Step 4. Start the Ray head node (Node 1)

Step 5. Start the Ray worker nodes (all other nodes)

Step 6. Verify cluster status

Step 7. Download MiniMax M2.5

Step 8. Launch inference server (tensor parallel = node count)

Step 9. Test inference

Validate and monitor (both topologies)

Next steps

Resources