Quantize Models to NVFP4 with NVIDIA Model Optimizer

Basic idea

NVFP4 is a 4-bit floating-point format for NVIDIA Blackwell GPUs. It reduces memory bandwidth and storage for inference while keeping accuracy close to higher-precision formats.

Unlike uniform INT4 quantization, NVFP4 keeps floating-point semantics with a shared exponent and a compact mantissa, which improves dynamic range. Blackwell Tensor Cores support mixed-precision execution across FP16, FP8, and FP4, so models can use FP4 for weights and activations while accumulating in higher precision (typically FP16).

Immediate benefits:

• Cut memory use ~3.5× vs FP16 and ~1.8× vs FP8
• Maintain accuracy close to FP8 (usually <1% loss)
• Improve speed and energy efficiency for inference

What you'll accomplish

You'll produce an NVFP4 checkpoint of DeepSeek-R1-Distill-Llama-8B with NVIDIA Model Optimizer inside a GPU container on your supported hardware platform. As a validation check, you can load the checkpoint and call an OpenAI-compatible endpoint.

Quantization can change model quality. Run evaluations for your use case before deploying.

What to know before starting

Required:

• Working with Docker containers and GPU-accelerated workloads
• Basic understanding of model quantization and its impact on inference
• Familiarity with Hugging Face model repositories and authentication

Optional:

• Experience with NVIDIA TensorRT / CUDA toolkit environments

Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The shared workflow is: prepare Docker and Hugging Face auth → run Model Optimizer NVFP4 quantization → validate artifacts → serve an OpenAI-compatible endpoint. Container image and post-quantization serving stack differ by hardware platform (see Instructions).

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• Enough free disk space for model download and quantized outputs (several GB; plan for tens of GB)

Software requirements

• Docker installed with GPU support: docker --version
• NVIDIA Container Toolkit configured
• Hugging Face account with an access token and access to the target model
• Network access to NGC / container registry and Hugging Face

Verify your setup:

# Check GPU visibility on the host
nvidia-smi

# Optional: confirm the default container for your hardware platform can see the GPU
# Set IMAGE to the container image for your hardware platform (see Instructions)
export IMAGE=nvcr.io/nvidia/vllm:25.12.post1-py3   # example — use your hardware platform's image
docker run --rm --gpus all "$IMAGE" nvidia-smi
# On dual-GPU DGX Station, pin the GB300 instead, e.g.:
# docker run --rm --gpus "device=$GPU_ID" "$IMAGE" nvidia-smi

# Verify sufficient disk space
df -h .

Time & risk

• Estimated time: 60 MIN (45–90 MIN depending on network speed and model size)
• Risk level: Medium
  • Model download may fail due to network issues or Hugging Face authentication
  • Quantization is memory-intensive and can fail if GPU memory is insufficient
  • Output files are large and need adequate storage
• Rollback: Remove the output directory and optionally remove pulled Docker images to restore the original state (see Cleanup in Instructions)
• Last Updated: 08/03/2026
  • Updated Model Optimizer documentation and GitHub links after repository rename
  • 07/27/2026: Added supported hardware platforms matrix