Nemotron Model Family on DGX Spark

Step 1
Overview and prerequisites

This tab serves NVIDIA Nemotron-3-Super (NVFP4 checkpoint) on a single DGX Spark. Deploy with vLLM (vllm/vllm-openai:cu130-nightly) or TensorRT-LLM (nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9) using the commands and tuning from the Nemotron Spark deployment guide.

DGX Spark ships a single Grace–Blackwell GPU with 128 GB of unified memory—the guide’s KV, batch, and cache choices assume that footprint.

Architecture refresher of Nemotron 3 Super

LatentMoE — Expert computation runs in a compressed latent dimension (d=4096 → ℓ=1024); all-to-all routing is reduced ~4× versus a standard MoE. On a single GPU, expert parallelism does not apply: use --tensor-parallel-size 1 (vLLM) and --tp_size 1 --ep_size 1 (TensorRT-LLM).
MTP (multi-token prediction) — One MTP layer is baked into the checkpoint for speculative decoding, with minimal extra KV versus an external draft model.
Mamba-2 hybrid — SSM state cache (mamba_ssm_cache) is separate from the KV cache. vLLM uses float32 for that cache in the recipe below; TensorRT-LLM uses FP16 SSM cache plus stochastic rounding in extra-llm-api-config.yml, as in the guide.

Requirements

DGX Spark with GB10 and sufficient disk for weights, caches, and images
NVIDIA Container Toolkit and Docker with GPU access
Hugging Face token (HF_TOKEN) when the model card requires it

How the steps are grouped

Step 2: Download the reasoning parser file—the guide asks for this before starting either server.
Steps 3–6: vLLM (pull image, environment, run, test on port 8000). Skip if you only use TensorRT-LLM.
Steps 7–11: TensorRT-LLM (pull image, checkpoint, YAML, run, test on port 8123). Skip Steps 3–6 if you only use TensorRT-LLM.
Step 12: Cleanup.

The configurations in the upstream guide were contributed by Izzy Putterman, Nave Assaf, Joyjit Daw, and other NVIDIA engineers.

Step 2
Download the Nemotron Super reasoning parser

Download the reasoning parser before starting the server (for both vLLM and TensorRT-LLM):

wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/raw/main/super_v3_reasoning_parser.py

vLLM: Mount this file in Step 5 (docker run uses -v …/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py).
TensorRT-LLM: The documented trtllm-serve line uses --reasoning_parser nano-v3; still keep this file in your project directory so your setup matches the guide.

Step 3
Pull the vLLM container image

vLLM path only.

docker pull vllm/vllm-openai:cu130-nightly

MTP + NVFP4 on DGX Spark requires this cu130 nightly image. Pinned releases such as 0.17.1 do not support this combination on single-GPU Spark for this model.

Step 4
Configure vLLM environment variables

vLLM path only.

On the host, export the vLLM variables (they match the upstream Spark guide) and your Hugging Face token before docker run. Copy and paste:

export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
export VLLM_USE_FLASHINFER_MOE_FP4=0
export HF_TOKEN=<your_huggingface_token>

Why these environment variables:

VLLM_NVFP4_GEMM_BACKEND — Marlin GEMM for NVFP4 on Spark.
VLLM_ALLOW_LONG_MAX_MODEL_LEN — Required for --max-model-len 1000000 on one GPU.
VLLM_FLASHINFER_ALLREDUCE_BACKEND — Allreduce fix for single-GPU topology.
VLLM_USE_FLASHINFER_MOE_FP4 — Disable FlashInfer FP4 MoE (multi-GPU Blackwell); Marlin handles FP4 on Spark.

The Step 5 docker run passes the same values with -e; keeping these exports on the host is optional but makes the flags easy to reuse or inspect.

Step 5
Run Nemotron Super with vLLM

vLLM path only.

From the directory that contains super_v3_reasoning_parser.py, in the same shell where you ran Step 4 (so HF_TOKEN is set):

docker run --rm -it --gpus all \
  -e VLLM_NVFP4_GEMM_BACKEND=marlin \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
  -e VLLM_USE_FLASHINFER_MOE_FP4=0 \
  -e HF_TOKEN=$HF_TOKEN \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v $(pwd)/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py \
  -p 8000:8000 \
  vllm/vllm-openai:cu130-nightly \
    --model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
    --served-model-name nemotron-3-super \
    --host 0.0.0.0 \
    --port 8000 \
    --async-scheduling \
    --dtype auto \
    --kv-cache-dtype fp8 \
    --tensor-parallel-size 1 \
    --pipeline-parallel-size 1 \
    --data-parallel-size 1 \
    --trust-remote-code \
    --gpu-memory-utilization 0.90 \
    --enable-chunked-prefill \
    --max-num-seqs 4 \
    --max-model-len 1000000 \
    --moe-backend marlin \
    --mamba_ssm_cache_dtype float32 \
    --quantization fp4 \
    --speculative_config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
    --reasoning-parser-plugin /app/super_v3_reasoning_parser.py \
    --reasoning-parser super_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder

Flag rationale (summary)

Flag	Role
`--tensor-parallel-size 1`	Single GPU; no expert parallelism on Spark
`--kv-cache-dtype fp8`	Smaller KV footprint for long context in unified memory
`--max-num-seqs 4`	Conservative concurrency for memory headroom
`--moe-backend marlin` / `--quantization fp4`	NVFP4 + Marlin MoE on one GPU
`--speculative_config` … MTP	Baked-in MTP head (3 draft tokens); Triton MoE on speculative path
`--mamba_ssm_cache_dtype float32`	SSM cache separate from KV; TensorRT-LLM can use FP16 + stochastic rounding instead
`--async-scheduling`	Better throughput on single GPU

Step 6
Test the vLLM API

vLLM path only.

The server takes several minutes to download and load model weights on first run (the NVFP4 checkpoint is tens of gigabytes). In another terminal, confirm the server is ready and reports the served model:

curl -sf http://localhost:8000/v1/models

Once the model is listed, send a chat completion:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nemotron-3-super",
    "messages": [{"role": "user", "content": "Summarize what Latent MoE changes for routing traffic."}],
    "max_tokens": 256
  }'

If you enabled API key auth in vLLM, add the matching Authorization header.

Step 7
Pull the TensorRT-LLM container image

TensorRT-LLM path only — skip Steps 3–6 if you use this stack.

docker pull nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9

Step 8
Download the NVFP4 checkpoint for TensorRT-LLM

TensorRT-LLM path only.

trtllm-serve expects a local NVFP4 checkpoint directory. Download the Hugging Face model into a folder you will mount into the container (here ./nemotron-super-nvfp4):

export HF_TOKEN=<your_huggingface_token>
hf download nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
  --local-dir ./nemotron-super-nvfp4

If your TensorRT-LLM build expects engines or another layout, follow TensorRT-LLM documentation for release:1.3.0rc9.

Step 9
Create extra-llm-api-config.yml for TensorRT-LLM

TensorRT-LLM path only.

In the same parent directory as nemotron-super-nvfp4, create extra-llm-api-config.yml:

kv_cache_config:
  dtype: fp8
  enable_block_reuse: false
  free_gpu_memory_fraction: 0.9
  mamba_ssm_cache_dtype: float16
  mamba_ssm_stochastic_rounding: true
  mamba_ssm_philox_rounds: 5
moe_config:
  backend: CUTLASS
cuda_graph_config:
  enable_padding: true
  max_batch_size: 8
enable_attention_dp: false
enable_chunked_prefill: true
stream_interval: 1
print_iter_log: true
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 3
  allow_advanced_sampling: true

Config rationale

Setting	Rationale
`kv_cache_config.dtype: fp8`	FP8 KV cache to maximize context in 128 GB unified memory
`mamba_ssm_cache_dtype: float16`	FP16 SSM cache saves memory vs float32 on one GPU
`mamba_ssm_stochastic_rounding: true`	Mitigates FP16 SSM precision loss (e.g. 5 Philox rounds)
`enable_block_reuse: false`	Mamba recurrent state is not prefix-cacheable
`free_gpu_memory_fraction: 0.9`	Aggressive allocator use on single GPU
`moe_config.backend: CUTLASS`	CUTLASS MoE for single-GPU NVFP4
`max_batch_size: 8`	Conservative for memory headroom
`num_nextn_predict_layers: 3`	MTP with three speculative steps
`max_seq_len: 1048576`	Full 1M token context window (set on `trtllm-serve` in Step 10).

Step 10
Run Nemotron Super with TensorRT-LLM

TensorRT-LLM path only.

From the directory that contains both nemotron-super-nvfp4 and extra-llm-api-config.yml:

export HF_TOKEN=<your_huggingface_token>

docker run --rm -it --gpus all \
  -e HF_TOKEN=$HF_TOKEN \
  -e TLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -v "$(pwd)":/workspace \
  -w /workspace \
  -p 8123:8123 \
  nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9 \
  trtllm-serve nemotron-super-nvfp4 \
    --host 0.0.0.0 \
    --port 8123 \
    --max_batch_size 8 \
    --tp_size 1 --ep_size 1 \
    --max_num_tokens 8192 \
    --trust_remote_code \
    --reasoning_parser nano-v3 \
    --tool_parser qwen3_coder \
    --extra_llm_api_options extra-llm-api-config.yml \
    --max_seq_len 1048576

Change /workspace/nemotron-super-nvfp4 if your checkpoint folder name or path differs.

Step 11
Test the TensorRT-LLM API

TensorRT-LLM path only.

Use the same shape as Step 6 on port 8123. Set "model" to the served name printed in trtllm-serve logs (the example below uses a placeholder you should replace).

curl http://localhost:8123/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "YOUR_SERVED_MODEL_NAME",
    "messages": [{"role": "user", "content": "Summarize what Latent MoE changes for routing traffic."}],
    "max_tokens": 256
  }'

Step 12
Cleanup

Stop the running container with Ctrl+C.

Remove the vLLM reasoning parser if you no longer need it:

rm -f super_v3_reasoning_parser.py

Remove local checkpoint directories or Hugging Face cache only if you want to reclaim disk.

For Nemotron Nano model, use the Run Nemotron Nano tab.

Nemotron Model Family on DGX Spark

Step 1Overview and prerequisites

Step 2Download the Nemotron Super reasoning parser

Step 3Pull the vLLM container image

Step 4Configure vLLM environment variables

Step 5Run Nemotron Super with vLLM

Step 6Test the vLLM API

Step 7Pull the TensorRT-LLM container image

Step 8Download the NVFP4 checkpoint for TensorRT-LLM

Step 9Create extra-llm-api-config.yml for TensorRT-LLM

Step 10Run Nemotron Super with TensorRT-LLM

Step 11Test the TensorRT-LLM API

Step 12Cleanup

Resources

Step 1
Overview and prerequisites

Step 2
Download the Nemotron Super reasoning parser

Step 3
Pull the vLLM container image

Step 4
Configure vLLM environment variables

Step 5
Run Nemotron Super with vLLM

Step 6
Test the vLLM API

Step 7
Pull the TensorRT-LLM container image

Step 8
Download the NVFP4 checkpoint for TensorRT-LLM

Step 9
Create extra-llm-api-config.yml for TensorRT-LLM

Step 10
Run Nemotron Super with TensorRT-LLM

Step 11
Test the TensorRT-LLM API

Step 12
Cleanup