---
title: "Nemotron Model Family on DGX Spark — Troubleshooting"
canonical: "https://build.nvidia.com/spark/nemotron/troubleshooting.md"
---

# Troubleshooting for Nemotron Nano

The **Run Nemotron Nano** tab serves the model on Docker with **vLLM** (`vllm/vllm-openai:v0.20.0`) on port **8000**. Full flags and paths are in that tab.

| Symptom | Cause | Fix |
|---------|-------|-----|
| `docker: could not select device driver` or no GPU in container | NVIDIA Container Toolkit / driver | Install or restart the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html); keep `docker run --gpus all` as in Step 3 |
| Container exits immediately or model fails to load | Wrong or incomplete weights path | Point `WEIGHTS` at the local Nemotron-3-Nano Omni weights directory and keep `-v "${WEIGHTS}:/model:ro"` so the container sees `/model` (Step 3) |
| `ModuleNotFoundError` / audio backend errors | Audio packages not installed | The base image omits audio; keep `pip install vllm[audio] && vllm serve /model …` in the launch command (Step 3) |
| "CUDA out of memory" when starting server | Context or concurrency too large for free memory | Lower `--gpu-memory-utilization` (e.g. `0.70`) first, then `--max-model-len` (e.g. `32768`), per Step 5 |
| Reasoning or tool output malformed | Parser flags mismatch | Keep `--reasoning-parser nemotron_v3`, `--enable-auto-tool-choice`, and `--tool-call-parser qwen3_coder` as in Step 3 |
| `curl` returns model / 404 errors | Name does not match served id | Use `"model": "nemotron_3_nano_omni"` to match `--served-model-name` in Step 3 |
| "Connection refused" on port 8000 | Port or container | Map `-p 8000:8000`, `--port 8000`, and confirm the container is still running (`docker ps`) |

# Troubleshooting for Nemotron Super

The **Run Nemotron Super** tab uses **Step 2** (reasoning parser download) for both stacks, then **vLLM** (Steps **3–6**) or **TensorRT-LLM** (Steps **7–11**). Match the table to the stack you run. Full flags and paths are in that tab; the upstream [Spark deployment guide](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/usage-cookbook/Nemotron-3-Super/SparkDeploymentGuide) is the source of truth for tuned settings.

## Nemotron Super — Docker and GPU (both paths)

| Symptom | Cause | Fix |
|---------|-------|-----|
| `docker: Error response from daemon: could not select device driver` or no GPU in container | NVIDIA Container Toolkit / driver | Install or restart the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html); use `docker run --gpus all` as in the Super tab |
| Image pull failures | Auth or network | For NGC (`nvcr.io/...`), `docker login nvcr.io` with your API key if required; check proxy and registry access |

## Nemotron Super — vLLM (Steps 3–6)

| Symptom | Cause | Fix |
|---------|-------|-----|
| Container exits immediately or model fails to load | Missing HF access or token | Set `HF_TOKEN` (Step 4) in the **same shell** as Step 5 so `-e HF_TOKEN=$HF_TOKEN` is not empty. Confirm your account can use `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` |
| Stuck or repeated downloads inside the container | Cache not persisted | Keep `-v ~/.cache/huggingface:/root/.cache/huggingface` so weights reuse across runs |
| Error loading reasoning parser / plugin | Missing file or bad mount | Run Step 2 `wget`, start Step 5 `docker run` from the directory that contains `super_v3_reasoning_parser.py`, and keep `-v $(pwd)/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py` |
| Errors about max model length | Long context blocked | Export or pass `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` with large `--max-model-len` (e.g. 1000000), per Step 4–5 |
| FP4 / MoE kernel errors on Spark | Wrong image or backend | Use `vllm/vllm-openai:cu130-nightly` (not an older pin such as `0.17.1` for this recipe). Keep `VLLM_NVFP4_GEMM_BACKEND=marlin`, `VLLM_USE_FLASHINFER_MOE_FP4=0`, and `--moe-backend marlin` / `--quantization fp4` |
| Allreduce / distributed warnings on one GPU | Single-GPU topology | Set `VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm` (Steps 4–5) as in the Spark guide; see [vLLM PR #35793](https://github.com/vllm-project/vllm/pull/35793) |
| MTP / speculative decoding errors | Bad JSON or flags | Copy `--speculative_config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'` exactly; fix shell quoting if the JSON was split or escaped wrong |
| OOM or server killed under load | Context × concurrency too high | Lower `--max-num-seqs`, `--max-model-len`, or `--gpu-memory-utilization`; free GPU memory from other jobs. KV uses `--kv-cache-dtype fp8` to save space |
| Reasoning or tool output malformed | Parser flags mismatch | Keep `--reasoning-parser-plugin /app/super_v3_reasoning_parser.py`, `--reasoning-parser super_v3`, `--enable-auto-tool-choice`, `--tool-call-parser qwen3_coder` as in Step 5 |
| `curl` returns model / 404 errors | Name does not match served id | Use `"model": "nemotron-3-super"` to match `--served-model-name` in Step 5 (Step 6 `curl`) |
| "Connection refused" on port 8000 | Port or container | Map `-p 8000:8000`, `--host 0.0.0.0`, `--port 8000`, and confirm the container is still running |

## Nemotron Super — TensorRT-LLM (Steps 7–11)

| Symptom | Cause | Fix |
|---------|-------|-----|
| `trtllm-serve` exits or cannot open model | Wrong checkpoint path or incomplete download | Finish Step 8 `hf download` into `./nemotron-super-nvfp4`. Mount the parent with `-v "$(pwd)":/workspace` and pass the same folder name as in Step 10 (e.g. `/workspace/nemotron-super-nvfp4`) |
| Long context / max seq errors | Long-seq guard | Keep `TLLM_ALLOW_LONG_MAX_MODEL_LEN=1` on the container when using `--max_seq_len 1048576` (Step 10) |
| Config or YAML parse errors | Missing or invalid `extra-llm-api-config.yml` | Place the file next to the checkpoint directory, mount `/workspace`, and use `--extra_llm_api_options /workspace/extra-llm-api-config.yml`. Match YAML indentation to Step 9 |
| OOM or process killed | Batch or sequence too large | Reduce `--max_batch_size`, `--max_num_tokens`, or `--max_seq_len`; in the YAML, try lowering `cuda_graph_config.max_batch_size` or `kv_cache_config.free_gpu_memory_fraction` slightly |
| MoE / NVFP4 backend errors | Backend mismatch for single GPU | Keep `moe_config.backend: CUTLASS` in `extra-llm-api-config.yml` for this Spark recipe |
| Reasoning or tools look wrong | Parser confusion with vLLM | TensorRT-LLM uses `--reasoning_parser nano-v3` and `--tool_parser qwen3_coder` (Step 10), not the vLLM `super_v3` plugin file |
| "Connection refused" on port 8123 | Port mapping | Use `-p 8123:8123`, `--host 0.0.0.0`, `--port 8123` as in Step 10 |
| `curl` fails or wrong model in JSON | Served name differs | Read `trtllm-serve` startup logs for the actual model id and substitute `YOUR_SERVED_MODEL_NAME` in Step 11 |
| Checkpoint layout errors | TRT-LLM expects engines or another format | Your image may need a converted engine or different layout; follow [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM) docs for `release:1.3.0rc9` and this checkpoint |

> [!NOTE]
> DGX Spark uses a Unified Memory Architecture (UMA), which enables dynamic sharing between GPU and CPU. Some workloads can still hit memory pressure while reporting headroom. If you see unexplained OOM or stalls, try flushing the page cache (administrative host only):

```bash
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
```

For platform known issues, see the [DGX Spark known issues](https://docs.nvidia.com/dgx/dgx-spark/known-issues.html) page.