Run NemoClaw with a Local LLM
Build a local AI assistant in an OpenShell sandbox with vLLM inference and optional Telegram
Multi-node inference for NemoClaw
NOTE
Multi-node currently applies to DGX Station only. Single-node setup for all supported hardware platforms is covered in the Instructions tab.
Deploy NVIDIA Nemotron 3 Ultra across two DGX Station nodes, then point NemoClaw on the head node at that shared vLLM endpoint.
Phase 1: Prepare both nodes
Prerequisites
- Complete the connectivity playbook for your DGX Station pair before starting this tab. Both high-speed fabric rails must be configured and validated.
- Hugging Face Access Token for downloading the model.
- Docker and NVIDIA Container Toolkit installed on both nodes.
- Both nodes must be connected to each other via the validated fabric.
NOTE
The initial model download, weight loading, kernel compilation, autotuning, and CUDA graph capture can take more than an hour. Later starts are faster when the model cache and the existing containers are retained. Recreating the containers can rebuild compilation artifacts.
Verify the fabric
On both nodes, confirm that the high-speed interfaces are active:
ibdev2netdev
ip -br link show
ip -br address show
show_gids
The examples in this guide use the following direct-attach network. Use your configured addresses if they differ.
| Rail | node-1 | node-2 |
|---|---|---|
| 0 | 192.168.240.1/30 | 192.168.240.2/30 |
| 1 | 192.168.240.5/30 | 192.168.240.6/30 |
From node-1, verify both peers with jumbo packets:
ping -c 4 -M do -s 8972 192.168.240.2
ping -c 4 -M do -s 8972 192.168.240.6
Do not continue until both pings succeed and the two-node fabric playbook's RDMA and NCCL checks pass.
Download and copy the model cache
NOTE
This step takes more than an hour depending on the internet speed and requires substantial storage (on the order of hundreds of GB) on both nodes.
Authenticate to Hugging Face on node-1. The model is gated, so the account must have access before the download starts.
ssh node-1
hf auth login
hf auth whoami
export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
hf download "${HF_MODEL}" --cache-dir "${HOME}/.cache/huggingface"
Copy the complete Hugging Face cache layout to node-2. Copying only the model shards is not sufficient.
ssh node-2 'mkdir -p ~/.cache/huggingface'
rsync -aH --info=progress2 "${HOME}/.cache/huggingface/" node-2:.cache/huggingface/
Confirm that the model snapshot exists on both nodes:
find "${HOME}/.cache/huggingface" -maxdepth 1 -type d \
-name 'models--nvidia--NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4'
Phase 2: Launch the multi-node inference server
Start the head container on node-1
Connect to node-1 and set the deployment variables. Discover fabric devices and the first fabric interface from ibdev2netdev (use the values from your validated connectivity playbook if discovery differs):
ssh node-1
mapfile -t IB_LINES < <(ibdev2netdev)
export HEAD_HCA0=$(awk '{print \$1; exit}' <<< "\${IB_LINES[0]}")
export HEAD_HCA1=$(awk '{print \$1; exit}' <<< "\${IB_LINES[1]}")
export HEAD_IFACE=$(awk '{print \$5; exit}' <<< "\${IB_LINES[0]}")
export NCCL_IB_HCA="${HEAD_HCA0},${HEAD_HCA1}"
export UCX_NET_DEVICES="${HEAD_HCA0}:1,${HEAD_HCA1}:1"
export HEAD_IP=$(ip -4 -o address show dev "${HEAD_IFACE}" \
| awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_HEAD=$(nvidia-smi \
--query-gpu=name,uuid --format=csv,noheader \
| awk -F', ' '/GB300|B300/ {print $2; exit}')
export IMG=vllm/vllm-openai:v0.25.1-aarch64
export HF_MODEL=nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4
export SERVED_MODEL=nemotron-ultra
Review the discovered values before starting the container:
printf 'HEAD_IFACE=%s\nHEAD_IP=%s\nGPU_UUID_HEAD=%s\nNCCL_IB_HCA=%s\nUCX_NET_DEVICES=%s\n' \
"${HEAD_IFACE}" "${HEAD_IP}" "${GPU_UUID_HEAD}" "${NCCL_IB_HCA}" "${UCX_NET_DEVICES}"
Start the head container. It creates the Ray cluster and waits up to one hour for the worker GPU before launching vLLM.
sudo docker rm -f nemotron-ultra-head 2>/dev/null || true
sudo docker run -d --name nemotron-ultra-head \
--restart unless-stopped --init \
--network host --shm-size 16g \
--gpus "device=${GPU_UUID_HEAD}" \
--device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/uverbs1 \
--ulimit memlock=-1 \
-e HEAD_IP="${HEAD_IP}" \
-e HF_MODEL="${HF_MODEL}" \
-e SERVED_MODEL="${SERVED_MODEL}" \
-e VLLM_HOST_IP="${HEAD_IP}" \
-e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=7200 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e NCCL_IB_HCA="${NCCL_IB_HCA}" \
-e NCCL_IB_DISABLE=0 \
-e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ROCE_VERSION_NUM=2 \
-e NCCL_IB_TC=106 \
-e NCCL_NET_GDR_LEVEL=PHB \
-e NCCL_SOCKET_IFNAME="${HEAD_IFACE}" \
-e GLOO_SOCKET_IFNAME="${HEAD_IFACE}" \
-e TP_SOCKET_IFNAME="${HEAD_IFACE}" \
-e NCCL_IB_QPS_PER_CONNECTION=4 \
-e NCCL_IB_PCI_RELAXED_ORDERING=1 \
-e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
-e HF_HOME=/models/huggingface \
-v "${HOME}/.cache/huggingface:/models/huggingface" \
--entrypoint bash "${IMG}" -lc '
set -euo pipefail
python3 -m pip install --break-system-packages "ray==2.56.0"
python3 -m pip install --break-system-packages --ignore-installed "blinker==1.9.0" "aiperf==0.11.0"
ray start --head --node-ip-address="${HEAD_IP}" --port=6379 --num-gpus=1
python3 - <<"PY"
import time
import ray
ray.init(address="auto")
deadline = time.time() + 3600
while ray.cluster_resources().get("GPU", 0) < 2:
if time.time() >= deadline:
raise TimeoutError("node-2 GPU did not join Ray within 3600 seconds")
time.sleep(5)
print(ray.cluster_resources())
PY
exec vllm serve "${HF_MODEL}" \
--served-model-name "${SERVED_MODEL}" \
--host 0.0.0.0 --port 8000 \
--trust-remote-code \
--tensor-parallel-size 1 \
--pipeline-parallel-size 2 \
--distributed-executor-backend ray \
--kv-cache-dtype fp8 \
--max-model-len 262144 \
--gpu-memory-utilization 0.9 \
--max-num-seqs 256 \
--distributed-timeout-seconds 7200 \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser nemotron_v3
'
Start the worker container on node-2
Connect to node-2 and set the worker variables:
NOTE
HEAD_IP value should be fetched from node-1 Step 4
ssh node-2
mapfile -t IB_LINES < <(ibdev2netdev)
export WORKER_HCA0=$(awk '{print \$1; exit}' <<< "\${IB_LINES[0]}")
export WORKER_HCA1=$(awk '{print \$1; exit}' <<< "\${IB_LINES[1]}")
export WORKER_IFACE=$(awk '{print \$5; exit}' <<< "\${IB_LINES[0]}")
export NCCL_IB_HCA="${WORKER_HCA0},${WORKER_HCA1}"
export UCX_NET_DEVICES="${WORKER_HCA0}:1,${WORKER_HCA1}:1"
export WORKER_IP=$(ip -4 -o address show dev "${WORKER_IFACE}" \
| awk '{split($4, address, "/"); print address[1]; exit}')
export GPU_UUID_WORKER=$(nvidia-smi \
--query-gpu=name,uuid --format=csv,noheader \
| awk -F', ' '/GB300|B300/ {print $2; exit}')
export HEAD_IP="<node-1 fabric IP>" # e.g. 192.168.240.1
export IMG=vllm/vllm-openai:v0.25.1-aarch64
Review the values and then start the worker:
printf 'WORKER_IFACE=%s\nWORKER_IP=%s\nGPU_UUID_WORKER=%s\nHEAD_IP=%s\nNCCL_IB_HCA=%s\n' \
"${WORKER_IFACE}" "${WORKER_IP}" "${GPU_UUID_WORKER}" "${HEAD_IP}" "${NCCL_IB_HCA}"
sudo docker rm -f nemotron-ultra-worker 2>/dev/null || true
sudo docker run -d --name nemotron-ultra-worker \
--restart unless-stopped --init \
--network host --shm-size 16g \
--gpus "device=${GPU_UUID_WORKER}" \
--device=/dev/infiniband/uverbs0 \
--device=/dev/infiniband/uverbs1 \
--ulimit memlock=-1 \
-e HEAD_IP="${HEAD_IP}" \
-e WORKER_IP="${WORKER_IP}" \
-e VLLM_HOST_IP="${WORKER_IP}" \
-e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e NCCL_IB_HCA="${NCCL_IB_HCA}" \
-e NCCL_IB_DISABLE=0 \
-e NCCL_IB_ADDR_FAMILY=AF_INET \
-e NCCL_IB_ROCE_VERSION_NUM=2 \
-e NCCL_IB_TC=106 \
-e NCCL_NET_GDR_LEVEL=PHB \
-e NCCL_SOCKET_IFNAME="${WORKER_IFACE}" \
-e GLOO_SOCKET_IFNAME="${WORKER_IFACE}" \
-e TP_SOCKET_IFNAME="${WORKER_IFACE}" \
-e NCCL_IB_QPS_PER_CONNECTION=4 \
-e NCCL_IB_PCI_RELAXED_ORDERING=1 \
-e UCX_NET_DEVICES="${UCX_NET_DEVICES}" \
-e HF_HOME=/models/huggingface \
-v "${HOME}/.cache/huggingface:/models/huggingface" \
--entrypoint bash "${IMG}" -lc '
set -euo pipefail
python3 -m pip install --break-system-packages "ray==2.56.0"
exec ray start --address="${HEAD_IP}:6379" --node-ip-address="${WORKER_IP}" --num-gpus=1 --block
'
Phase 3: Verify the multi-node inference server
Monitor startup without interrupting it
Open one terminal for each node to monitor the startup:
# node-1
sudo docker logs -f nemotron-ultra-head
# node-2
sudo docker logs -f nemotron-ultra-worker
During a first start, vLLM loads model shards, compiles kernels, builds the KV cache, autotunes FlashInfer/TRT-LLM kernels, and captures CUDA graphs. The containers can remain Up while port 8000 returns HTTP 000. This is expected while the logs continue to advance.
Do not restart merely because the API is not yet listening. Investigate when a container exits, its restart count increases, an explicit error appears, or the worker logs stop advancing for an extended period.
Validate the API, reasoning, and tool calls
Wait for this message in the head logs:
Application startup complete.
Then run all API checks on node-1.
Confirm the model alias:
curl -fsS http://127.0.0.1:8000/v1/models | jq
Send a short chat request:
curl -fsS http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "nemotron-ultra",
"messages": [
{"role": "user", "content": "Reply with exactly: READY"}
],
"max_tokens": 16,
"temperature": 0
}' | jq
Perform routine health checks
Run these checks from node-1:
sudo docker inspect \
--format 'head={{.State.Status}} restarts={{.RestartCount}}' \
nemotron-ultra-head
ssh node-2 sudo docker inspect \
--format 'worker={{.State.Status}} restarts={{.RestartCount}}' \
nemotron-ultra-worker
curl -fsS http://127.0.0.1:8000/v1/models \
| jq -e '.data[] | select(.id == "nemotron-ultra")'
All three checks must succeed before treating the agent service as healthy.
When the endpoint is healthy, install NemoClaw on node-1 (see the Instructions tab) and point it at this local vLLM endpoint with NVIDIA Nemotron 3 Ultra as the inference backend.
Phase 4: Troubleshoot or remove the deployment
Troubleshoot common failures
| Symptom | Likely cause | Corrective action |
|---|---|---|
| API responds, but tool calls appear as text | Tool parser flags are missing or NemoClaw is using the Responses API | Confirm --enable-auto-tool-choice, --tool-call-parser qwen3_coder, --reasoning-parser nemotron_v3, and Chat Completions. |
Direct curl works, but NemoClaw inference is unhealthy | OpenShell cannot reach the host endpoint | Confirm vLLM listens on 0.0.0.0, inspect the OpenShell subnet, review firewall rules, and rerun onboarding. |
| NemoClaw reports the wrong model | The provider route or served-model alias is stale | Verify /v1/models, then rerun onboarding or use nemoclaw inference set with nemotron-ultra. |
Useful diagnostics:
# node-1
sudo docker logs --tail 200 nemotron-ultra-head
sudo docker exec nemotron-ultra-head \
ray status --address=${HEAD_IP}:6379
ss -ltnp | grep ':8000' || true
# node-2
sudo docker logs --tail 200 nemotron-ultra-worker
nvidia-smi
Stop or remove the deployment
Remove the inference containers in this order:
# node-1
sudo docker rm -f nemotron-ultra-head
# node-2
sudo docker rm -f nemotron-ultra-worker
Keep the Hugging Face caches unless reclaiming disk is intentional. Retaining the model cache makes a controlled relaunch substantially faster. Reusing the existing containers also preserves their compilation artifacts.