---
title: "Nemotron Model Family on DGX Spark — Run Nemotron Super"
canonical: "https://build.nvidia.com/spark/nemotron/instructions-super.md"
---

# Step 1. Overview and prerequisites

This tab serves **NVIDIA Nemotron-3-Super** (NVFP4 checkpoint) on a single DGX Spark. Deploy with **vLLM** (`vllm/vllm-openai:cu130-nightly`) or **TensorRT-LLM** (`nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9`) using the commands and tuning from the [Nemotron Spark deployment guide](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/usage-cookbook/Nemotron-3-Super/SparkDeploymentGuide).

DGX Spark ships a single Grace–Blackwell GPU with **128 GB of unified memory**—the guide’s KV, batch, and cache choices assume that footprint.

**Architecture refresher of Nemotron 3 Super**

- **LatentMoE** — Expert computation runs in a compressed latent dimension (`d=4096 → ℓ=1024`); all-to-all routing is reduced ~4× versus a standard MoE. On a single GPU, expert parallelism does not apply: use `--tensor-parallel-size 1` (vLLM) and `--tp_size 1 --ep_size 1` (TensorRT-LLM).
- **MTP (multi-token prediction)** — One MTP layer is baked into the checkpoint for speculative decoding, with minimal extra KV versus an external draft model.
- **Mamba-2 hybrid** — SSM state cache (`mamba_ssm_cache`) is separate from the KV cache. vLLM uses `float32` for that cache in the recipe below; TensorRT-LLM uses FP16 SSM cache plus stochastic rounding in `extra-llm-api-config.yml`, as in the guide.

**Requirements**

- DGX Spark with GB10 and sufficient disk for weights, caches, and images
- NVIDIA Container Toolkit and Docker with GPU access
- Hugging Face token (`HF_TOKEN`) when the model card requires it

**How the steps are grouped**

- **Step 2:** Download the reasoning parser file—the guide asks for this before starting **either** server.
- **Steps 3–6:** vLLM (pull image, environment, run, test on port **8000**). Skip if you only use TensorRT-LLM.
- **Steps 7–11:** TensorRT-LLM (pull image, checkpoint, YAML, run, test on port **8123**). Skip Steps 3–6 if you only use TensorRT-LLM.
- **Step 12:** Cleanup.

The configurations in the upstream guide were contributed by Izzy Putterman, Nave Assaf, Joyjit Daw, and other NVIDIA engineers.

---

# Step 2. Download the Nemotron Super reasoning parser

Download the reasoning parser **before** starting the server (for both vLLM and TensorRT-LLM):

```bash
wget https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/raw/main/super_v3_reasoning_parser.py
```

- **vLLM:** Mount this file in Step 5 (`docker run` uses `-v …/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py`).
- **TensorRT-LLM:** The documented `trtllm-serve` line uses `--reasoning_parser nano-v3`; still keep this file in your project directory so your setup matches the guide.

---

# Step 3. Pull the vLLM container image

**vLLM path only.**

```bash
docker pull vllm/vllm-openai:cu130-nightly
```

MTP + NVFP4 on DGX Spark requires this **cu130 nightly** image. Pinned releases such as **0.17.1** do not support this combination on single-GPU Spark for this model.

---

# Step 4. Configure vLLM environment variables

**vLLM path only.**

On the host, export the vLLM variables (they match the upstream Spark guide) and your Hugging Face token before `docker run`. Copy and paste:

```bash
export VLLM_NVFP4_GEMM_BACKEND=marlin
export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
export VLLM_USE_FLASHINFER_MOE_FP4=0
export HF_TOKEN=<your_huggingface_token>
```

Why these environment variables:
- `VLLM_NVFP4_GEMM_BACKEND` — Marlin GEMM for NVFP4 on Spark.
- `VLLM_ALLOW_LONG_MAX_MODEL_LEN` — Required for `--max-model-len 1000000` on one GPU.
- `VLLM_FLASHINFER_ALLREDUCE_BACKEND` — Allreduce fix for single-GPU topology.
- `VLLM_USE_FLASHINFER_MOE_FP4` — Disable FlashInfer FP4 MoE (multi-GPU Blackwell); Marlin handles FP4 on Spark.

The Step 5 `docker run` passes the same values with `-e`; keeping these exports on the host is optional but makes the flags easy to reuse or inspect.

---

# Step 5. Run Nemotron Super with vLLM

**vLLM path only.**

From the directory that contains `super_v3_reasoning_parser.py`, in the **same shell** where you ran Step 4 (so `HF_TOKEN` is set):

```bash
docker run --rm -it --gpus all \
-e VLLM_NVFP4_GEMM_BACKEND=marlin \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e HF_TOKEN=$HF_TOKEN \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v $(pwd)/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py \
-p 8000:8000 \
vllm/vllm-openai:cu130-nightly \
--model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
--served-model-name nemotron-3-super \
--host 0.0.0.0 \
--port 8000 \
--async-scheduling \
--dtype auto \
--kv-cache-dtype fp8 \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--data-parallel-size 1 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--max-num-seqs 4 \
--max-model-len 1000000 \
--moe-backend marlin \
--mamba_ssm_cache_dtype float32 \
--quantization fp4 \
--speculative_config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
--reasoning-parser-plugin /app/super_v3_reasoning_parser.py \
--reasoning-parser super_v3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```

**Flag rationale (summary)**

| Flag | Role |
| ---- | ---- |
| `--tensor-parallel-size 1` | Single GPU; no expert parallelism on Spark |
| `--kv-cache-dtype fp8` | Smaller KV footprint for long context in unified memory |
| `--max-num-seqs 4` | Conservative concurrency for memory headroom |
| `--moe-backend marlin` / `--quantization fp4` | NVFP4 + Marlin MoE on one GPU |
| `--speculative_config` … MTP | Baked-in MTP head (3 draft tokens); Triton MoE on speculative path |
| `--mamba_ssm_cache_dtype float32` | SSM cache separate from KV; TensorRT-LLM can use FP16 + stochastic rounding instead |
| `--async-scheduling` | Better throughput on single GPU |

---

# Step 6. Test the vLLM API

**vLLM path only.**

The server takes several minutes to download and load model weights on first run (the NVFP4 checkpoint is tens of gigabytes). In another terminal, confirm the server is ready and reports the served model:

```bash
curl -sf http://localhost:8000/v1/models
```

Once the model is listed, send a chat completion:

```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "nemotron-3-super",
"messages": [{"role": "user", "content": "Summarize what Latent MoE changes for routing traffic."}],
"max_tokens": 256
}'
```

If you enabled API key auth in vLLM, add the matching `Authorization` header.

---

# Step 7. Pull the TensorRT-LLM container image

**TensorRT-LLM path only** — skip Steps 3–6 if you use this stack.

```bash
docker pull nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9
```

---

# Step 8. Download the NVFP4 checkpoint for TensorRT-LLM

**TensorRT-LLM path only.**

`trtllm-serve` expects a **local NVFP4 checkpoint directory**. Download the Hugging Face model into a folder you will mount into the container (here `./nemotron-super-nvfp4`):

```bash
export HF_TOKEN=<your_huggingface_token>
hf download nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
--local-dir ./nemotron-super-nvfp4
```

If your TensorRT-LLM build expects engines or another layout, follow [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) documentation for `release:1.3.0rc9`.

---

# Step 9. Create extra-llm-api-config.yml for TensorRT-LLM

**TensorRT-LLM path only.**

In the same parent directory as `nemotron-super-nvfp4`, create `extra-llm-api-config.yml`:

```yaml
kv_cache_config:
dtype: fp8
enable_block_reuse: false
free_gpu_memory_fraction: 0.9
mamba_ssm_cache_dtype: float16
mamba_ssm_stochastic_rounding: true
mamba_ssm_philox_rounds: 5
moe_config:
backend: CUTLASS
cuda_graph_config:
enable_padding: true
max_batch_size: 8
enable_attention_dp: false
enable_chunked_prefill: true
stream_interval: 1
print_iter_log: true
speculative_config:
decoding_type: MTP
num_nextn_predict_layers: 3
allow_advanced_sampling: true
```

**Config rationale**

| Setting | Rationale |
| ------- | --------- |
| `kv_cache_config.dtype: fp8` | FP8 KV cache to maximize context in 128 GB unified memory |
| `mamba_ssm_cache_dtype: float16` | FP16 SSM cache saves memory vs float32 on one GPU |
| `mamba_ssm_stochastic_rounding: true` | Mitigates FP16 SSM precision loss (e.g. 5 Philox rounds) |
| `enable_block_reuse: false` | Mamba recurrent state is not prefix-cacheable |
| `free_gpu_memory_fraction: 0.9` | Aggressive allocator use on single GPU |
| `moe_config.backend: CUTLASS` | CUTLASS MoE for single-GPU NVFP4 |
| `max_batch_size: 8` | Conservative for memory headroom |
| `num_nextn_predict_layers: 3` | MTP with three speculative steps |
| `max_seq_len: 1048576` | Full 1M token context window (set on `trtllm-serve` in Step 10). |

---

# Step 10. Run Nemotron Super with TensorRT-LLM

**TensorRT-LLM path only.**

From the directory that contains both `nemotron-super-nvfp4` and `extra-llm-api-config.yml`:

```bash
export HF_TOKEN=<your_huggingface_token>

docker run --rm -it --gpus all \
-e HF_TOKEN=$HF_TOKEN \
-e TLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-v "$(pwd)":/workspace \
-w /workspace \
-p 8123:8123 \
nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc9 \
trtllm-serve nemotron-super-nvfp4 \
--host 0.0.0.0 \
--port 8123 \
--max_batch_size 8 \
--tp_size 1 --ep_size 1 \
--max_num_tokens 8192 \
--trust_remote_code \
--reasoning_parser nano-v3 \
--tool_parser qwen3_coder \
--extra_llm_api_options extra-llm-api-config.yml \
--max_seq_len 1048576
```

Change `/workspace/nemotron-super-nvfp4` if your checkpoint folder name or path differs.

---

# Step 11. Test the TensorRT-LLM API

**TensorRT-LLM path only.**

Use the same shape as Step 6 on port **8123**. Set `"model"` to the served name printed in `trtllm-serve` logs (the example below uses a placeholder you should replace).

```bash
curl http://localhost:8123/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "YOUR_SERVED_MODEL_NAME",
"messages": [{"role": "user", "content": "Summarize what Latent MoE changes for routing traffic."}],
"max_tokens": 256
}'
```

---

# Step 12. Cleanup

Stop the running container with `Ctrl+C`.

Remove the vLLM reasoning parser if you no longer need it:

```bash
rm -f super_v3_reasoning_parser.py
```

Remove local checkpoint directories or Hugging Face cache only if you want to reclaim disk.

For **Nemotron Nano** model, use the **Run Nemotron Nano** tab.