NVFP4 Quantization

Basic idea

NVFP4 is a 4-bit floating-point format introduced with NVIDIA Blackwell GPUs to maintain model accuracy while reducing memory bandwidth and storage requirements for inference workloads. Unlike uniform INT4 quantization, NVFP4 retains floating-point semantics with a shared exponent and a compact mantissa, allowing higher dynamic range and more stable convergence. NVIDIA Blackwell Tensor Cores natively support mixed-precision execution across FP16, FP8, and FP4, enabling models to use FP4 for weights and activations while accumulating in higher precision (typically FP16). This design minimizes quantization error during matrix multiplications and supports efficient conversion pipelines in TensorRT-LLM for fine-tuned layer-wise quantization.

Immediate benefits are:

• Cut memory use ~3.5x vs FP16 and ~1.8x vs FP8
• Maintain accuracy close to FP8 (usually <1% loss)
• Improve speed and energy efficiency for inference

What you'll accomplish

You'll quantize the Qwen3.6-35B-A3B Mixture-of-Experts model using NVIDIA's TensorRT Model Optimizer inside a TensorRT-LLM container, producing an NVFP4 quantized Hugging Face checkpoint, and then serve it on NVIDIA DGX Spark with vLLM.

Quantization is driven by Model Optimizer recipes — declarative YAML configurations selected with the --recipe flag. The playbook offers two: a Qwen3.5/3.6-MoE dedicated W4A16 recipe (NVFP4 weights on the MoE MLPs, FP8 attention) recommended for interactive use on DGX Spark, and a general-purpose NVFP4 experts-only recipe (W4A4) for compute-bound, higher-concurrency serving.

Depending on the recipe, quantization reduces model size by roughly 3x compared to the BF16 model. This quantization approach aims to preserve accuracy while providing significant throughput improvements. However, it's important to note that quantization can potentially impact model accuracy - we recommend running evaluations to verify if the quantized model maintains acceptable performance for your use case.

What to know before starting

• Working with Docker containers and GPU-accelerated workloads
• Understanding of model quantization concepts and their impact on inference performance
• Experience with NVIDIA TensorRT and CUDA toolkit environments
• Familiarity with Hugging Face model repositories and authentication

Prerequisites

• NVIDIA Spark device with Blackwell architecture GPU
• Docker installed with GPU support
• NVIDIA Container Toolkit configured
• Available storage for model files and outputs (about 100 GB free: the BF16 model download is roughly 70 GB and the quantized output roughly 25 GB)
• Hugging Face account with access to the target model

Verify your setup:

# Check Docker GPU access
docker run --rm --gpus all nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev nvidia-smi

# Verify sufficient disk space
df -h .

Time & risk

• Estimated duration: 1.5-3 hours, usually dominated by the model download
• Risks:
  • Model download may fail due to network issues or Hugging Face authentication problems
  • Quantization process is memory-intensive and may fail on systems with insufficient GPU memory
  • Output files are large (several GB) and require adequate storage space
• Rollback: Remove the output directory and any pulled Docker images to restore original state.
• Last Updated: 07/21/2026
  • Update TensorRT Model Optimizer to the 0.45.0 release
  • Switch the example model from DeepSeek-R1-Distill-Llama-8B to Qwen3.6-35B-A3B (MoE)
  • Use recipe-driven PTQ (hf_ptq.py --recipe) instead of the removed --export_fmt flow, offering the Qwen3.5/3.6-MoE dedicated W4A16 recipe and the general-purpose NVFP4 experts-only recipe
  • Serve the quantized checkpoint with the NGC vLLM container