Nemotron Model Family on DGX Spark

Overview and prerequisites

This tab serves NVIDIA Nemotron-3-Nano on a single DGX Spark with vLLM, using the upstream multi-arch image vllm/vllm-openai:v0.20.0. The server exposes an OpenAI-compatible HTTP API on port 8000. The example uses the multimodal Nemotron-3-Nano Omni weights (text, image, audio, and video), so the launch command installs the optional audio packages and enables the multimodal limits.

DGX Spark ships a single Grace–Blackwell GB10 GPU with 128 GB of unified memory—the batch, context, and cache choices below assume that footprint.

This tab covers the Spark-specific setup. For everything not covered here—API examples, reasoning mode, and video tuning—follow the general Nemotron instructions and the model card.

Requirements

• DGX Spark with GB10 and sufficient disk for weights, caches, and the container image
• NVIDIA Container Toolkit and Docker with GPU access
• Local Nemotron-3-Nano Omni weights on disk (point WEIGHTS at that directory in Step 3)

Pull the vLLM container image

Pull the upstream multi-arch vLLM v0.20.0 image. Docker automatically pulls the arm64 variant on Spark.

docker pull vllm/vllm-openai:v0.20.0

Launch the vLLM server on Spark

Point WEIGHTS at your local Nemotron-3-Nano Omni weights directory, then start the server. The base image does not include audio packages, so the command installs them with pip install vllm[audio] before running vllm serve.

WEIGHTS=/path/to/nemotron-3-nano-omni-weights

docker run --rm -it \
  --gpus all \
  --ipc=host -p 8000:8000 \
  --shm-size=16g \
  --name vllm-nemotron-omni \
  -v "${WEIGHTS}:/model:ro" \
  --entrypoint /bin/bash \
  vllm/vllm-openai:v0.20.0 -c  \
  "pip install vllm[audio] && vllm serve /model \
  --served-model-name=nemotron_3_nano_omni \
  --max-num-seqs 8 \
  --max-model-len 131072 \
  --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.8 \
  --limit-mm-per-prompt '{\"video\": 1, \"image\": 1, \"audio\": 1}' \
  --media-io-kwargs '{\"video\": {\"fps\": 2,  \"num_frames\": 256}}' \
  --allowed-local-media-path=/ \
  --enable-prefix-caching \
  --max-num-batched-tokens 32768 \
  --reasoning-parser nemotron_v3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder"

Flag rationale (summary)

Key Spark-specific flags

Verify the server and test the API

In another terminal, confirm the server is ready and reports the served model:

curl -sS http://localhost:8000/v1/models | python3 -m json.tool

Once the model is listed, send a chat completion. Use the served model name from Step 3:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron_3_nano_omni",
    "messages": [{"role": "user", "content": "New York is a great city because..."}],
    "max_tokens": 100
  }'

For multimodal requests, reasoning-mode prompting, and video tuning, follow the general Nemotron instructions and the model card.

Memory tuning on Spark

Spark uses unified LPDDR5X memory (~128 GB shared between CPU and GPU), not separate system + VRAM pools. If you hit OOM, use these two levers, in order of impact:

1. Lower --gpu-memory-utilization from 0.8 toward 0.70 to free memory back to the OS and re-enable weight prefetch. Cost: a smaller KV cache budget.
2. Lower --max-model-len to reduce KV cache allocation (for example, halving the context window halves the KV cache at --max-num-seqs=1).

Combined override for a tight-memory run:

  --gpu-memory-utilization=0.70 \
  --max-model-len=32768 \

Cleanup

Stop the running container with Ctrl+C, or from another terminal:

docker stop vllm-nemotron-omni

Remove the image only if you want to reclaim disk:

docker rmi vllm/vllm-openai:v0.20.0

Delete the local weights directory only if you no longer need it.

For the larger Nemotron Super deployment on the same hardware, use the Run Nemotron Super tab in this playbook.