---
title: "Quantize Models to NVFP4 with NVIDIA Model Optimizer"
publisher: "nvidia"
type: "playbook"
updated: "2026-08-04T16:15:40.399Z"
description: "Cut memory ~3.5× vs FP16 while keeping accuracy close to FP8, then validate with an OpenAI-compatible endpoint"
canonical: "https://build.nvidia.com/playbooks/nvfp4-quantization.md"
---

# Basic idea

NVFP4 is a 4-bit floating-point format for NVIDIA Blackwell GPUs. It reduces memory bandwidth and storage for inference while keeping accuracy close to higher-precision formats.

Unlike uniform INT4 quantization, NVFP4 keeps floating-point semantics with a shared exponent and a compact mantissa, which improves dynamic range. Blackwell Tensor Cores support mixed-precision execution across FP16, FP8, and FP4, so models can use FP4 for weights and activations while accumulating in higher precision (typically FP16).

Immediate benefits:

- Cut memory use ~3.5× vs FP16 and ~1.8× vs FP8
- Maintain accuracy close to FP8 (usually <1% loss)
- Improve speed and energy efficiency for inference

# What you'll accomplish

You'll produce an **NVFP4 checkpoint** of **DeepSeek-R1-Distill-Llama-8B** with NVIDIA Model Optimizer inside a GPU container on your supported hardware platform. As a validation check, you can load the checkpoint and call an OpenAI-compatible endpoint.

Quantization can change model quality. Run evaluations for your use case before deploying.

# What to know before starting

**Required:**

- Working with Docker containers and GPU-accelerated workloads
- Basic understanding of model quantization and its impact on inference
- Familiarity with Hugging Face model repositories and authentication

**Optional:**

- Experience with NVIDIA TensorRT / CUDA toolkit environments

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, OS, memory, and whether multi-node applies. The shared workflow is: prepare Docker and Hugging Face auth → run Model Optimizer NVFP4 quantization → validate artifacts → serve an OpenAI-compatible endpoint. Container image and post-quantization serving stack differ by hardware platform (see Instructions).

| Hardware platform | OS | Memory  | Multi-node capable hardware |
| :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 128 GB Unified Memory | — |
| **DGX Station** | DGX OS (Linux) | Large HBM + Grace DRAM | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Enough free disk space for model download and quantized outputs (several GB; plan for tens of GB)

**Software requirements**

- Docker installed with GPU support: `docker --version`
- NVIDIA Container Toolkit configured
- Hugging Face account with an access token and access to the target model
- Network access to NGC / container registry and Hugging Face

Verify your setup:

```bash
# Check GPU visibility on the host
nvidia-smi

# Optional: confirm the default container for your hardware platform can see the GPU
# Set IMAGE to the container image for your hardware platform (see Instructions)
export IMAGE=nvcr.io/nvidia/vllm:25.12.post1-py3   # example — use your hardware platform's image
docker run --rm --gpus all "$IMAGE" nvidia-smi
# On dual-GPU DGX Station, pin the GB300 instead, e.g.:
# docker run --rm --gpus "device=$GPU_ID" "$IMAGE" nvidia-smi

# Verify sufficient disk space
df -h .
```

# Time & risk

- **Estimated time:** 60 MIN (45–90 MIN depending on network speed and model size)
- **Risk level:** Medium
- Model download may fail due to network issues or Hugging Face authentication
- Quantization is memory-intensive and can fail if GPU memory is insufficient
- Output files are large and need adequate storage
- **Rollback:** Remove the output directory and optionally remove pulled Docker images to restore the original state (see Cleanup in Instructions)
- **Last Updated:** 08/03/2026
- Updated Model Optimizer documentation and GitHub links after repository rename
- 07/27/2026: Added supported hardware platforms matrix

## More

- [Instructions](/playbooks/nvfp4-quantization/instructions.md)
- [Troubleshooting](/playbooks/nvfp4-quantization/troubleshooting.md)