---
title: "Run NemoClaw with a Local LLM — Multi-node"
canonical: "https://build.nvidia.com/spark/nemoclaw/multi-node.md"
---

# Multi-node inference for NemoClaw

> [!NOTE]
> **Multi-node currently applies to DGX Station only.** Single-node setup for all supported hardware platforms is covered in the **Instructions** tab.

Deploy **NVIDIA Nemotron 3 Ultra** across two DGX Station nodes, then point NemoClaw on the head node at that shared vLLM endpoint.

# Phase 1: Prepare both nodes

## Step 1. Prerequisites
- Complete the connectivity playbook for your DGX Station pair before starting this tab. Both high-speed fabric rails must be configured and validated.
- Hugging Face [Access Token](https://huggingface.co/docs/hub/en/security-tokens) for downloading the model.
- Docker and [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html) installed on both nodes.
- Both nodes must be connected to each other via the validated fabric.

> [!NOTE]
> The initial model download, weight loading, kernel compilation, autotuning, and CUDA graph capture can take more than an hour. Later starts are faster when the model cache and the existing containers are retained. Recreating the containers can rebuild compilation artifacts.

## Step 2. Verify the fabric

On both nodes, confirm that the high-speed interfaces are active:

```bash
ibdev2netdev
ip -br link show
ip -br address show
show_gids
```

The examples in this guide use the following direct-attach network. Use your configured addresses if they differ.

| Rail | `node-1` | `node-2` |
| --- | --- | --- |
| 0 | `192.168.240.1/30` | `192.168.240.2/30` |
| 1 | `192.168.240.5/30` | `192.168.240.6/30` |

From `node-1`, verify both peers with jumbo packets:

```bash
ping -c 4 -M do -s 8972 192.168.240.2
ping -c 4 -M do -s 8972 192.168.240.6
```

Do not continue until both pings succeed and the two-node fabric playbook's RDMA and NCCL checks pass.

## Step 3. Download and copy the model cache

>[!NOTE]
> This step takes more than an hour depending on the internet speed and requires substantial storage (on the order of hundreds of GB) on both nodes.

Authenticate to Hugging Face on `node-1`. The model is gated, so the account must have access before the download starts.

```bash
ssh node-1
hf auth login
hf auth whoami

export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4

hf download "${HF_MODEL}" --cache-dir "${HOME}/.cache/huggingface"
```

Copy the complete Hugging Face cache layout to `node-2`. Copying only the model shards is not sufficient.

```bash
ssh node-2 'mkdir -p ~/.cache/huggingface'

rsync -aH --info=progress2 "${HOME}/.cache/huggingface/" node-2:.cache/huggingface/
```

Confirm that the model snapshot exists on both nodes:

```bash
find "${HOME}/.cache/huggingface" -maxdepth 1 -type d \
-name 'models--nvidia--NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4'
```

# Phase 2: Launch the multi-node inference server

## Step 4. Start the head container on node-1

Connect to `node-1` and set the deployment variables. Discover fabric devices and the first fabric interface from `ibdev2netdev` (use the values from your validated connectivity playbook if discovery differs):

```bash
ssh node-1

mapfile -t IB_LINES < <(ibdev2netdev)
export HEAD_HCA0=$(awk '{print $1; exit}' <<< "${IB_LINES[0]}")
export HEAD_HCA1=$(awk '{print $1; exit}' <<< "${IB_LINES[1]}")
export HEAD_IFACE=$(awk '{print $5; exit}' <<< "${IB_LINES[0]}")
export NCCL_IB_HCA="${HEAD_HCA0},${HEAD_HCA1}"
export UCX_NET_DEVICES="${HEAD_HCA0}:1,${HEAD_HCA1}:1"
export HEAD_IP=$(ip -4 -o address show dev "${HEAD_IFACE}" \
| awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_HEAD=$(nvidia-smi \
--query-gpu=name,uuid --format=csv,noheader \
| awk -F', ' '/GB300|B300/ {print $2; exit}')
export IMG=vllm/vllm-openai:v0.25.1-aarch64
export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
export SERVED_MODEL=nemotron-ultra
```

Review the discovered values before starting the container:

```bash
printf 'HEAD_IFACE=%s\nHEAD_IP=%s\nGPU_UUID_HEAD=%s\nNCCL_IB_HCA=%s\nUCX_NET_DEVICES=%s\n' \
"${HEAD_IFACE}" "${HEAD_IP}" "${GPU_UUID_HEAD}" "${NCCL_IB_HCA}" "${UCX_NET_DEVICES}"
```

Start the head container. It creates the Ray cluster and waits up to one hour for the worker GPU before launching vLLM.

```bash
sudo docker rm -f nemotron-ultra-head 2>/dev/null || true

sudo docker run -d --name nemotron-ultra-head \
--restart unless-stopped --init \
--network host --shm-size 16g \
--gpus "device=${GPU_UUID_HEAD}" \
--device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/uverbs1 \
--ulimit memlock=-1 \
-e HEAD_IP="${HEAD_IP}" \
-e HF_MODEL="${HF_MODEL}" \
-e SERVED_MODEL="${SERVED_MODEL}" \
-e VLLM_HOST_IP="${HEAD_IP}" \
-e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e NCCL_IB_HCA="${NCCL_IB_HCA}" \
-e NCCL_IB_DISABLE=0 \
-e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ROCE_VERSION_NUM=2 \
-e NCCL_IB_TC=106 \
-e NCCL_NET_GDR_LEVEL=PHB \
-e NCCL_SOCKET_IFNAME="${HEAD_IFACE}" \
-e GLOO_SOCKET_IFNAME="${HEAD_IFACE}" \
-e TP_SOCKET_IFNAME="${HEAD_IFACE}" \
-e NCCL_IB_QPS_PER_CONNECTION=4 \
-e NCCL_IB_PCI_RELAXED_ORDERING=1 \
-e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
-e HF_HOME=/models/huggingface \
-v "${HOME}/.cache/huggingface:/models/huggingface" \
--entrypoint bash "${IMG}" -lc '
set -euo pipefail
python3 -m pip install --break-system-packages "ray==2.56.0"
python3 -m pip install --break-system-packages --ignore-installed "blinker==1.9.0" "aiperf==0.11.0"
ray start --head --node-ip-address="${HEAD_IP}" --port=6379 --num-gpus=1

python3 - <<"PY"
import time
import ray

ray.init(address="auto")
deadline = time.time() + 3600
while ray.cluster_resources().get("GPU", 0) < 2:
if time.time() >= deadline:
raise TimeoutError("node-2 GPU did not join Ray within 3600 seconds")
time.sleep(5)
print(ray.cluster_resources())
PY

exec vllm serve "${HF_MODEL}" \
--served-model-name "${SERVED_MODEL}" \
--host 0.0.0.0 --port 8000 \
--trust-remote-code \
--tensor-parallel-size 1 \
--pipeline-parallel-size 2 \
--distributed-executor-backend ray \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--gpu-memory-utilization 0.9 \
--max-num-seqs 256 \
--distributed-timeout-seconds 7200 \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3
'
```

## Step 5. Start the worker container on node-2

Connect to `node-2` and set the worker variables:

>[!NOTE]
> HEAD_IP value should be fetched from node-1 [Step 4](#step-4-start-the-head-container-on-node-1)

```bash
ssh node-2

mapfile -t IB_LINES < <(ibdev2netdev)
export WORKER_HCA0=$(awk '{print $1; exit}' <<< "${IB_LINES[0]}")
export WORKER_HCA1=$(awk '{print $1; exit}' <<< "${IB_LINES[1]}")
export WORKER_IFACE=$(awk '{print $5; exit}' <<< "${IB_LINES[0]}")
export NCCL_IB_HCA="${WORKER_HCA0},${WORKER_HCA1}"
export UCX_NET_DEVICES="${WORKER_HCA0}:1,${WORKER_HCA1}:1"
export WORKER_IP=$(ip -4 -o address show dev "${WORKER_IFACE}" \
| awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_WORKER=$(nvidia-smi \
--query-gpu=name,uuid --format=csv,noheader \
| awk -F', ' '/GB300|B300/ {print $2; exit}')

export HEAD_IP="<node-1 fabric IP>" # e.g. 192.168.240.1
export IMG=vllm/vllm-openai:v0.25.1-aarch64
```

Review the values and then start the worker:

```bash
printf 'WORKER_IFACE=%s\nWORKER_IP=%s\nGPU_UUID_WORKER=%s\nHEAD_IP=%s\nNCCL_IB_HCA=%s\n' \
"${WORKER_IFACE}" "${WORKER_IP}" "${GPU_UUID_WORKER}" "${HEAD_IP}" "${NCCL_IB_HCA}"

sudo docker rm -f nemotron-ultra-worker 2>/dev/null || true

sudo docker run -d --name nemotron-ultra-worker \
--restart unless-stopped --init \
--network host --shm-size 16g \
--gpus "device=${GPU_UUID_WORKER}" \
--device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/uverbs1 \
--ulimit memlock=-1 \
-e HEAD_IP="${HEAD_IP}" \
-e WORKER_IP="${WORKER_IP}" \
-e VLLM_HOST_IP="${WORKER_IP}" \
-e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e NCCL_IB_HCA="${NCCL_IB_HCA}" \
-e NCCL_IB_DISABLE=0 \
-e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ROCE_VERSION_NUM=2 \
-e NCCL_IB_TC=106 \
-e NCCL_NET_GDR_LEVEL=PHB \
-e NCCL_SOCKET_IFNAME="${WORKER_IFACE}" \
-e GLOO_SOCKET_IFNAME="${WORKER_IFACE}" \
-e TP_SOCKET_IFNAME="${WORKER_IFACE}" \
-e NCCL_IB_QPS_PER_CONNECTION=4 \
-e NCCL_IB_PCI_RELAXED_ORDERING=1 \
-e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
-e HF_HOME=/models/huggingface \
-v "${HOME}/.cache/huggingface:/models/huggingface" \
--entrypoint bash "${IMG}" -lc '
set -euo pipefail
python3 -m pip install --break-system-packages "ray==2.56.0"
exec ray start --address="${HEAD_IP}:6379" --node-ip-address="${WORKER_IP}" --num-gpus=1 --block
'
```

# Phase 3: Verify the multi-node inference server

## Step 6. Monitor startup without interrupting it

Open one terminal for each node to monitor the startup:

```bash
# node-1
sudo docker logs -f nemotron-ultra-head
```

```bash
# node-2
sudo docker logs -f nemotron-ultra-worker
```

During a first start, vLLM loads model shards, compiles kernels, builds the KV cache, autotunes FlashInfer/TRT-LLM kernels, and captures CUDA graphs. The containers can remain `Up` while port 8000 returns HTTP `000`. This is expected while the logs continue to advance.

Do not restart merely because the API is not yet listening. Investigate when a container exits, its restart count increases, an explicit error appears, or the worker logs stop advancing for an extended period.

## Step 7. Validate the API, reasoning, and tool calls

Wait for this message in the head logs:

```text
Application startup complete.
```

Then run all API checks on `node-1`.

Confirm the model alias:

```bash
curl -fsS http://127.0.0.1:8000/v1/models | jq
```

Send a short chat request:

```bash
curl -fsS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "nemotron-ultra",
"messages": [
{"role": "user", "content": "Reply with exactly: READY"}
],
"max_tokens": 16,
"temperature": 0
}' | jq
```

## Step 8. Perform routine health checks

Run these checks from `node-1`:

```bash
sudo docker inspect \
--format 'head={{.State.Status}} restarts={{.RestartCount}}' \
nemotron-ultra-head

ssh node-2 sudo docker inspect \
--format 'worker={{.State.Status}} restarts={{.RestartCount}}' \
nemotron-ultra-worker

curl -fsS http://127.0.0.1:8000/v1/models \
| jq -e '.data[] | select(.id == "nemotron-ultra")'
```

All three checks must succeed before treating the agent service as healthy.

When the endpoint is healthy, install NemoClaw on `node-1` (see the **Instructions** tab) and point it at this local vLLM endpoint with **NVIDIA Nemotron 3 Ultra** as the inference backend.

# Phase 4: Troubleshoot or remove the deployment

## Step 9. Troubleshoot common failures

| Symptom | Likely cause | Corrective action |
| --- | --- | --- |
| API responds, but tool calls appear as text | Tool parser flags are missing or NemoClaw is using the Responses API | Confirm `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder`, `--reasoning-parser nemotron_v3`, and Chat Completions. |
| Direct `curl` works, but NemoClaw inference is unhealthy | OpenShell cannot reach the host endpoint | Confirm vLLM listens on `0.0.0.0`, inspect the OpenShell subnet, review firewall rules, and rerun onboarding. |
| NemoClaw reports the wrong model | The provider route or served-model alias is stale | Verify `/v1/models`, then rerun onboarding or use `nemoclaw inference set` with `nemotron-ultra`. |

Useful diagnostics:

```bash
# node-1
sudo docker logs --tail 200 nemotron-ultra-head
sudo docker exec nemotron-ultra-head \
ray status --address=${HEAD_IP}:6379
ss -ltnp | grep ':8000' || true
```

```bash
# node-2
sudo docker logs --tail 200 nemotron-ultra-worker
nvidia-smi
```

## Step 10. Stop or remove the deployment

Remove the inference containers in this order:

```bash
# node-1
sudo docker rm -f nemotron-ultra-head
```

```bash
# node-2
sudo docker rm -f nemotron-ultra-worker
```

Keep the Hugging Face caches unless reclaiming disk is intentional. Retaining the model cache makes a controlled relaunch substantially faster. Reusing the existing containers also preserves their compilation artifacts.

## Related resources

- [NemoClaw documentation](https://docs.nvidia.com/nemoclaw/latest/index.html)
- [Set up vLLM for NemoClaw](https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/inference/local-inference/set-up-vllm)
- [Verify the sandbox inference route](https://docs.nvidia.com/nemoclaw/latest/user-guide/openclaw/inference/validate-inference/verify-inference-route)
- [NVIDIA Nemotron](https://github.com/NVIDIA-NeMo/Nemotron)