---
title: "Quantize Models to NVFP4 with NVIDIA Model Optimizer — Instructions"
canonical: "https://build.nvidia.com/spark/nvfp4-quantization/instructions.md"
---

> [!NOTE]
> These instructions target **Linux** on a supported hardware platform. Use the default container and command block for your hardware platform from the tables below.

# Step 1. Set up Docker permissions

To manage containers without `sudo`, add your user to the `docker` group. Open a terminal and test Docker access:

```bash
docker ps
```

If you see a permission-denied error, add your user to the docker group (skip if it already works):

```bash
sudo usermod -aG docker $USER
newgrp docker
```

# Step 2. Prepare the environment

Create a local output directory for quantized model files. The directory is mounted into the container so results persist after the container exits.

```bash
mkdir -p ./output_models
chmod 755 ./output_models
```

# Step 3. Authenticate with Hugging Face

Export a Hugging Face token so the container can download the model:

```bash
# Get a token from: https://huggingface.co/settings/tokens
export HF_TOKEN="your_token_here"
```

# Step 4. Set hardware platform launch variables

Container image, GPU device selection, and Model Optimizer pin differ by hardware platform. Set the variables that match your hardware platform before running quantization.

| Hardware platform | Container image | GPU device | Model Optimizer |
| ----------------- | --------------- | ---------- | --------------- |
| **DGX Spark** | `nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev` | `--gpus all` | `NVIDIA/Model-Optimizer` @ `0.35.0` |
| **DGX Station** | `nvcr.io/nvidia/vllm:25.12.post1-py3` | `--gpus "device=$GPU_ID"` (GB300) | `NVIDIA/Model-Optimizer` @ `0.41.0` |

## DGX Station only — identify the GB300 GPU

If the system has more than one GPU, identify the GB300 device ID:

```bash
nvidia-smi
```

Example (GB300 is device **1**):

```text
|   0  NVIDIA RTX 6000  ...
|   1  NVIDIA GB300     ...
```

```bash
export GPU_ID=1  # Replace with your GB300 device number
```

On a single-GPU DGX Station (GB300 only), use `GPU_ID=0`.

# Step 5. Run NVFP4 quantization with Model Optimizer

Use the command block for your hardware platform. Both paths quantize `deepseek-ai/DeepSeek-R1-Distill-Llama-8B` to NVFP4 and write artifacts under `./output_models`.

## DGX Spark

```bash
docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
-v "./output_models:/workspace/output_models" \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-e HF_TOKEN=$HF_TOKEN \
nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
bash -c "
git clone -b 0.35.0 --single-branch https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer && pip install -e '.[dev]' && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
/app/Model-Optimizer/examples/llm_ptq/scripts/huggingface_example.sh \
--model 'deepseek-ai/DeepSeek-R1-Distill-Llama-8B' \
--quant nvfp4 \
--tp 1 \
--export_fmt hf
"
```

Expected output directory:

```bash
export MODEL_PATH="./output_models/saved_models_DeepSeek-R1-Distill-Llama-8B_nvfp4_hf/"
```

## DGX Station

```bash
docker run --rm -it --gpus "device=$GPU_ID" --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
-v "./output_models:/workspace/output_models" \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-e HF_TOKEN=$HF_TOKEN \
nvcr.io/nvidia/vllm:25.12.post1-py3 \
bash -c "
git clone -b 0.41.0 --single-branch https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
cd /app/Model-Optimizer && pip install -e '.[dev]' && \
export ROOT_SAVE_PATH='/workspace/output_models' && \
/app/Model-Optimizer/examples/llm_ptq/scripts/huggingface_example.sh \
--model deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
--quant nvfp4 \
--tasks quant
"
```

Expected output directory:

```bash
export MODEL_PATH="./output_models/saved_models_DeepSeek-R1-Distill-Llama-8B_nvfp4/"
```

> [!NOTE]
> - You can safely ignore `No module named 'mpi4py'` during quantization when it appears; it does not block NVFP4 export.
> - `pynvml.NVMLError_NotSupported: Not Supported` can appear in some environments and does not affect results.
> - If the model is too large for available GPU memory, try a smaller model.

What this step does:

- Runs the container with GPU access and shared-memory settings suitable for large models
- Mounts `./output_models` and your Hugging Face cache
- Installs NVIDIA Model Optimizer and runs the NVFP4 quantization script

# Step 6. Monitor the quantization process

Watch for:

- Model download progress from Hugging Face
- Quantization calibration steps
- Model export and validation phases

# Step 7. Validate quantized model files

After the container exits, confirm artifacts exist:

```bash
ls -la ./output_models/

find ./output_models/ \( -name "*.bin" -o -name "*.safetensors" -o -name "*.json" -o -name "*.jinja" \)
```

You should see weight files, configuration, and tokenizer files under the `MODEL_PATH` for your hardware platform.

# Step 8. Serve and test with an OpenAI-compatible API

Post-quantization validation and serving use the stack that matches your hardware platform. Set `MODEL_PATH` from Step 5 if it is not already set.

## DGX Spark (TensorRT-LLM)

Load-test the checkpoint:

```bash
docker run \
-e HF_TOKEN=$HF_TOKEN \
-v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
-v "$MODEL_PATH:/workspace/model" \
--rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
--gpus=all --ipc=host --network host \
nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
bash -c '
python examples/llm-api/quickstart_advanced.py \
--model_dir /workspace/model/ \
--prompt "Paris is great because" \
--max_tokens 64
'
```

Start the OpenAI-compatible server:

```bash
docker run \
-e HF_TOKEN=$HF_TOKEN \
-v "$MODEL_PATH:/workspace/model" \
--rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
--gpus=all --ipc=host --network host \
nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
trtllm-serve /workspace/model \
--backend pytorch \
--max_batch_size 4 \
--port 8000
```

In another terminal:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
"messages": [{"role": "user", "content": "What is artificial intelligence?"}],
"max_tokens": 100,
"temperature": 0.7,
"stream": false
}'
```

## DGX Station (vLLM)

Start the OpenAI-compatible server (also used as the load check):

```bash
docker run \
-e HF_TOKEN=$HF_TOKEN \
-v "$MODEL_PATH:/workspace/model" \
--rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
--gpus "device=$GPU_ID" --ipc=host --network host \
nvcr.io/nvidia/vllm:25.12.post1-py3 \
vllm serve /workspace/model \
--served-model-name DeepSeek-R1-Distill-Llama-8B-NVFP4 \
--max-num-seqs 4 \
--max-model-len 8192 \
--port 8000
```

`--served-model-name` sets the model ID returned by the API. Without it, vLLM defaults to the mount path. Confirm with `curl http://localhost:8000/v1/models`, then:

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "DeepSeek-R1-Distill-Llama-8B-NVFP4",
"messages": [{"role": "user", "content": "What is artificial intelligence?"}],
"max_tokens": 100,
"temperature": 0.7,
"stream": false
}'
```

Adjust knobs such as `--max-model-len` for your workload. Stop the server with **Ctrl+C** when finished.

# Step 9. Cleanup and rollback

> [!WARNING]
> This permanently deletes quantized model files and optional cached data.

> [!NOTE]
> Quantization containers may write `./output_models/` as root. Try without `sudo` first; if you get permission denied, retry with `sudo`.

```bash
# Remove quantized outputs
rm -rf ./output_models
# If permission denied (root-owned files from the container):
# sudo rm -rf ./output_models

# Optional: remove Hugging Face cache
rm -rf ~/.cache/huggingface

# Optional: remove the container image for your hardware platform
# DGX Spark:
# docker rmi nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev
# DGX Station:
# docker rmi nvcr.io/nvidia/vllm:25.12.post1-py3
```

# Step 10. Next steps

The quantized model is ready for further use. Common follow-ups:

- Benchmark inference performance against the original model
- Integrate the checkpoint into your inference pipeline
- Deploy with NVIDIA Triton Inference Server for production serving
- Run additional validation on your target prompts and evaluation sets