Quantize Models to NVFP4 with NVIDIA Model Optimizer

NOTE

These instructions target Linux on a supported hardware platform. Use the default container and command block for your hardware platform from the tables below.

Step 1
Set up Docker permissions

To manage containers without sudo, add your user to the docker group. Open a terminal and test Docker access:

Bash

docker ps

If you see a permission-denied error, add your user to the docker group (skip if it already works):

Bash

sudo usermod -aG docker $USER
newgrp docker

Step 2
Prepare the environment

Create a local output directory for quantized model files. The directory is mounted into the container so results persist after the container exits.

Bash

mkdir -p ./output_models
chmod 755 ./output_models

Step 3
Authenticate with Hugging Face

Export a Hugging Face token so the container can download the model:

Bash

# Get a token from: https://huggingface.co/settings/tokens
export HF_TOKEN="your_token_here"

Step 4
Set hardware platform launch variables

Container image, GPU device selection, and Model Optimizer pin differ by hardware platform. Set the variables that match your hardware platform before running quantization.

Hardware platform	Container image	GPU device	Model Optimizer
DGX Spark	`nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev`	`--gpus all`	`NVIDIA/Model-Optimizer` @ `0.35.0`
DGX Station	`nvcr.io/nvidia/vllm:25.12.post1-py3`	`--gpus "device=$GPU_ID"` (GB300)	`NVIDIA/Model-Optimizer` @ `0.41.0`

DGX Station only — identify the GB300 GPU

If the system has more than one GPU, identify the GB300 device ID:

Bash

nvidia-smi

Example (GB300 is device 1):

Text

|   0  NVIDIA RTX 6000  ...
|   1  NVIDIA GB300     ...

Bash

export GPU_ID=1  # Replace with your GB300 device number

On a single-GPU DGX Station (GB300 only), use GPU_ID=0.

Step 5
Run NVFP4 quantization with Model Optimizer

Use the command block for your hardware platform. Both paths quantize deepseek-ai/DeepSeek-R1-Distill-Llama-8B to NVFP4 and write artifacts under ./output_models.

DGX Spark

Bash

docker run --rm -it --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "./output_models:/workspace/output_models" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN=$HF_TOKEN \
  nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
  bash -c "
    git clone -b 0.35.0 --single-branch https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
    cd /app/Model-Optimizer && pip install -e '.[dev]' && \
    export ROOT_SAVE_PATH='/workspace/output_models' && \
    /app/Model-Optimizer/examples/llm_ptq/scripts/huggingface_example.sh \
    --model 'deepseek-ai/DeepSeek-R1-Distill-Llama-8B' \
    --quant nvfp4 \
    --tp 1 \
    --export_fmt hf
  "

Expected output directory:

Bash

export MODEL_PATH="./output_models/saved_models_DeepSeek-R1-Distill-Llama-8B_nvfp4_hf/"

DGX Station

Bash

docker run --rm -it --gpus "device=$GPU_ID" --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
  -v "./output_models:/workspace/output_models" \
  -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
  -e HF_TOKEN=$HF_TOKEN \
  nvcr.io/nvidia/vllm:25.12.post1-py3 \
  bash -c "
    git clone -b 0.41.0 --single-branch https://github.com/NVIDIA/Model-Optimizer.git /app/Model-Optimizer && \
    cd /app/Model-Optimizer && pip install -e '.[dev]' && \
    export ROOT_SAVE_PATH='/workspace/output_models' && \
    /app/Model-Optimizer/examples/llm_ptq/scripts/huggingface_example.sh \
    --model deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
    --quant nvfp4 \
    --tasks quant
  "

Expected output directory:

Bash

export MODEL_PATH="./output_models/saved_models_DeepSeek-R1-Distill-Llama-8B_nvfp4/"

NOTE

You can safely ignore No module named 'mpi4py' during quantization when it appears; it does not block NVFP4 export.
pynvml.NVMLError_NotSupported: Not Supported can appear in some environments and does not affect results.
If the model is too large for available GPU memory, try a smaller model.

What this step does:

Runs the container with GPU access and shared-memory settings suitable for large models
Mounts ./output_models and your Hugging Face cache
Installs NVIDIA Model Optimizer and runs the NVFP4 quantization script

Step 6
Monitor the quantization process

Watch for:

Model download progress from Hugging Face
Quantization calibration steps
Model export and validation phases

Step 7
Validate quantized model files

After the container exits, confirm artifacts exist:

Bash

ls -la ./output_models/

find ./output_models/ \( -name "*.bin" -o -name "*.safetensors" -o -name "*.json" -o -name "*.jinja" \)

You should see weight files, configuration, and tokenizer files under the MODEL_PATH for your hardware platform.

Step 8
Serve and test with an OpenAI-compatible API

Post-quantization validation and serving use the stack that matches your hardware platform. Set MODEL_PATH from Step 5 if it is not already set.

DGX Spark (TensorRT-LLM)

Load-test the checkpoint:

Bash

docker run \
  -e HF_TOKEN=$HF_TOKEN \
  -v $HOME/.cache/huggingface/:/root/.cache/huggingface/ \
  -v "$MODEL_PATH:/workspace/model" \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
  bash -c '
    python examples/llm-api/quickstart_advanced.py \
      --model_dir /workspace/model/ \
      --prompt "Paris is great because" \
      --max_tokens 64
    '

Start the OpenAI-compatible server:

Bash

docker run \
  -e HF_TOKEN=$HF_TOKEN \
  -v "$MODEL_PATH:/workspace/model" \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus=all --ipc=host --network host \
  nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev \
  trtllm-serve /workspace/model \
    --backend pytorch \
    --max_batch_size 4 \
    --port 8000

In another terminal:

Bash

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-8B",
    "messages": [{"role": "user", "content": "What is artificial intelligence?"}],
    "max_tokens": 100,
    "temperature": 0.7,
    "stream": false
  }'

DGX Station (vLLM)

Start the OpenAI-compatible server (also used as the load check):

Bash

docker run \
  -e HF_TOKEN=$HF_TOKEN \
  -v "$MODEL_PATH:/workspace/model" \
  --rm -it --ulimit memlock=-1 --ulimit stack=67108864 \
  --gpus "device=$GPU_ID" --ipc=host --network host \
  nvcr.io/nvidia/vllm:25.12.post1-py3 \
  vllm serve /workspace/model \
    --served-model-name DeepSeek-R1-Distill-Llama-8B-NVFP4 \
    --max-num-seqs 4 \
    --max-model-len 8192 \
    --port 8000

--served-model-name sets the model ID returned by the API. Without it, vLLM defaults to the mount path. Confirm with curl http://localhost:8000/v1/models, then:

Bash

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "DeepSeek-R1-Distill-Llama-8B-NVFP4",
    "messages": [{"role": "user", "content": "What is artificial intelligence?"}],
    "max_tokens": 100,
    "temperature": 0.7,
    "stream": false
  }'

Adjust knobs such as --max-model-len for your workload. Stop the server with Ctrl+C when finished.

Step 9
Cleanup and rollback

WARNING

This permanently deletes quantized model files and optional cached data.

NOTE

Quantization containers may write ./output_models/ as root. Try without sudo first; if you get permission denied, retry with sudo.

Bash

# Remove quantized outputs
rm -rf ./output_models
# If permission denied (root-owned files from the container):
# sudo rm -rf ./output_models

# Optional: remove Hugging Face cache
rm -rf ~/.cache/huggingface

# Optional: remove the container image for your hardware platform
# DGX Spark:
# docker rmi nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev
# DGX Station:
# docker rmi nvcr.io/nvidia/vllm:25.12.post1-py3

Step 10
Next steps

The quantized model is ready for further use. Common follow-ups:

Benchmark inference performance against the original model
Integrate the checkpoint into your inference pipeline
Deploy with NVIDIA Triton Inference Server for production serving
Run additional validation on your target prompts and evaluation sets

Quantize Models to NVFP4 with NVIDIA Model Optimizer

Step 1Set up Docker permissions

Step 2Prepare the environment

Step 3Authenticate with Hugging Face

Step 4Set hardware platform launch variables

DGX Station only — identify the GB300 GPU

Step 5Run NVFP4 quantization with Model Optimizer

DGX Spark

DGX Station

Step 6Monitor the quantization process

Step 7Validate quantized model files

Step 8Serve and test with an OpenAI-compatible API

DGX Spark (TensorRT-LLM)

DGX Station (vLLM)

Step 9Cleanup and rollback

Step 10Next steps

Resources

Step 1
Set up Docker permissions

Step 2
Prepare the environment

Step 3
Authenticate with Hugging Face

Step 4
Set hardware platform launch variables

Step 5
Run NVFP4 quantization with Model Optimizer

Step 6
Monitor the quantization process

Step 7
Validate quantized model files

Step 8
Serve and test with an OpenAI-compatible API

Step 9
Cleanup and rollback

Step 10
Next steps