Nemotron Model Family on DGX Spark
45 MIN
Deploy Nemotron 3 model family (Nemotron-3-Nano or Nemotron-3-Super) on DGX Spark
Troubleshooting for Nemotron Nano
The Run Nemotron Nano tab serves the model on Docker with vLLM (vllm/vllm-openai:v0.20.0) on port 8000. Full flags and paths are in that tab.
| Symptom | Cause | Fix |
|---|---|---|
docker: could not select device driver or no GPU in container | NVIDIA Container Toolkit / driver | Install or restart the NVIDIA Container Toolkit; keep docker run --gpus all as in Step 3 |
| Container exits immediately or model fails to load | Wrong or incomplete weights path | Point WEIGHTS at the local Nemotron-3-Nano Omni weights directory and keep -v "${WEIGHTS}:/model:ro" so the container sees /model (Step 3) |
ModuleNotFoundError / audio backend errors | Audio packages not installed | The base image omits audio; keep pip install vllm[audio] && vllm serve /model … in the launch command (Step 3) |
| "CUDA out of memory" when starting server | Context or concurrency too large for free memory | Lower --gpu-memory-utilization (e.g. 0.70) first, then --max-model-len (e.g. 32768), per Step 5 |
| Reasoning or tool output malformed | Parser flags mismatch | Keep --reasoning-parser nemotron_v3, --enable-auto-tool-choice, and --tool-call-parser qwen3_coder as in Step 3 |
curl returns model / 404 errors | Name does not match served id | Use "model": "nemotron_3_nano_omni" to match --served-model-name in Step 3 |
| "Connection refused" on port 8000 | Port or container | Map -p 8000:8000, --port 8000, and confirm the container is still running (docker ps) |
Troubleshooting for Nemotron Super
The Run Nemotron Super tab uses Step 2 (reasoning parser download) for both stacks, then vLLM (Steps 3–6) or TensorRT-LLM (Steps 7–11). Match the table to the stack you run. Full flags and paths are in that tab; the upstream Spark deployment guide is the source of truth for tuned settings.
Nemotron Super — Docker and GPU (both paths)
| Symptom | Cause | Fix |
|---|---|---|
docker: Error response from daemon: could not select device driver or no GPU in container | NVIDIA Container Toolkit / driver | Install or restart the NVIDIA Container Toolkit; use docker run --gpus all as in the Super tab |
| Image pull failures | Auth or network | For NGC (nvcr.io/...), docker login nvcr.io with your API key if required; check proxy and registry access |
Nemotron Super — vLLM (Steps 3–6)
| Symptom | Cause | Fix |
|---|---|---|
| Container exits immediately or model fails to load | Missing HF access or token | Set HF_TOKEN (Step 4) in the same shell as Step 5 so -e HF_TOKEN=$HF_TOKEN is not empty. Confirm your account can use nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 |
| Stuck or repeated downloads inside the container | Cache not persisted | Keep -v ~/.cache/huggingface:/root/.cache/huggingface so weights reuse across runs |
| Error loading reasoning parser / plugin | Missing file or bad mount | Run Step 2 wget, start Step 5 docker run from the directory that contains super_v3_reasoning_parser.py, and keep -v $(pwd)/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py |
| Errors about max model length | Long context blocked | Export or pass VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 with large --max-model-len (e.g. 1000000), per Step 4–5 |
| FP4 / MoE kernel errors on Spark | Wrong image or backend | Use vllm/vllm-openai:cu130-nightly (not an older pin such as 0.17.1 for this recipe). Keep VLLM_NVFP4_GEMM_BACKEND=marlin, VLLM_USE_FLASHINFER_MOE_FP4=0, and --moe-backend marlin / --quantization fp4 |
| Allreduce / distributed warnings on one GPU | Single-GPU topology | Set VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm (Steps 4–5) as in the Spark guide; see vLLM PR #35793 |
| MTP / speculative decoding errors | Bad JSON or flags | Copy --speculative_config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' exactly; fix shell quoting if the JSON was split or escaped wrong |
| OOM or server killed under load | Context × concurrency too high | Lower --max-num-seqs, --max-model-len, or --gpu-memory-utilization; free GPU memory from other jobs. KV uses --kv-cache-dtype fp8 to save space |
| Reasoning or tool output malformed | Parser flags mismatch | Keep --reasoning-parser-plugin /app/super_v3_reasoning_parser.py, --reasoning-parser super_v3, --enable-auto-tool-choice, --tool-call-parser qwen3_coder as in Step 5 |
curl returns model / 404 errors | Name does not match served id | Use "model": "nemotron-3-super" to match --served-model-name in Step 5 (Step 6 curl) |
| "Connection refused" on port 8000 | Port or container | Map -p 8000:8000, --host 0.0.0.0, --port 8000, and confirm the container is still running |
Nemotron Super — TensorRT-LLM (Steps 7–11)
| Symptom | Cause | Fix |
|---|---|---|
trtllm-serve exits or cannot open model | Wrong checkpoint path or incomplete download | Finish Step 8 hf download into ./nemotron-super-nvfp4. Mount the parent with -v "$(pwd)":/workspace and pass the same folder name as in Step 10 (e.g. /workspace/nemotron-super-nvfp4) |
| Long context / max seq errors | Long-seq guard | Keep TLLM_ALLOW_LONG_MAX_MODEL_LEN=1 on the container when using --max_seq_len 1048576 (Step 10) |
| Config or YAML parse errors | Missing or invalid extra-llm-api-config.yml | Place the file next to the checkpoint directory, mount /workspace, and use --extra_llm_api_options /workspace/extra-llm-api-config.yml. Match YAML indentation to Step 9 |
| OOM or process killed | Batch or sequence too large | Reduce --max_batch_size, --max_num_tokens, or --max_seq_len; in the YAML, try lowering cuda_graph_config.max_batch_size or kv_cache_config.free_gpu_memory_fraction slightly |
| MoE / NVFP4 backend errors | Backend mismatch for single GPU | Keep moe_config.backend: CUTLASS in extra-llm-api-config.yml for this Spark recipe |
| Reasoning or tools look wrong | Parser confusion with vLLM | TensorRT-LLM uses --reasoning_parser nano-v3 and --tool_parser qwen3_coder (Step 10), not the vLLM super_v3 plugin file |
| "Connection refused" on port 8123 | Port mapping | Use -p 8123:8123, --host 0.0.0.0, --port 8123 as in Step 10 |
curl fails or wrong model in JSON | Served name differs | Read trtllm-serve startup logs for the actual model id and substitute YOUR_SERVED_MODEL_NAME in Step 11 |
| Checkpoint layout errors | TRT-LLM expects engines or another format | Your image may need a converted engine or different layout; follow TensorRT-LLM docs for release:1.3.0rc9 and this checkpoint |
NOTE
DGX Spark uses a Unified Memory Architecture (UMA), which enables dynamic sharing between GPU and CPU. Some workloads can still hit memory pressure while reporting headroom. If you see unexplained OOM or stalls, try flushing the page cache (administrative host only):
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
For platform known issues, see the DGX Spark known issues page.