Deploy NVIDIA NIM for LLM Inference

Basic idea

NVIDIA NIM is containerized software for fast, reliable AI model serving and inference on NVIDIA GPUs. This playbook shows how to run NIM microservices for LLMs on your hardware platform through a simple Docker workflow: authenticate with NVIDIA's registry, launch the NIM inference microservice, and validate the OpenAI-compatible HTTP endpoint.

What you'll accomplish

You'll launch a NIM container on your hardware platform to expose a GPU-accelerated HTTP endpoint for chat completions. These instructions use the Nemotron 3.5 Lightning 30B-A3B NIM as the default example; additional NIM containers are available in the NGC catalog (see Find model recipes).

What to know before starting

Required:

• Working in a terminal environment
• Using Docker commands and GPU-enabled containers
• Basic familiarity with REST APIs and curl commands

Optional:

• Understanding of NVIDIA GPU environments and CUDA

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

For full hardware x NIM support details, see NIM for LLMs support matrix.

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• Sufficient memory for your chosen NIM (varies by model and precision). On DGX Spark, Nemotron 3.5 Lightning needs roughly 30 GB (NVFP4), 32 GB (W4A16), or 66 GB (BF16) of unified memory at TP=1, before KV cache headroom
• At least 19–63 GB available storage for model caching, depending on the precision you pull (NVFP4 is smallest, BF16 largest)

Software requirements

• NVIDIA drivers installed: nvidia-smi
• Docker with NVIDIA Container Toolkit configured:

  docker run -it --gpus=all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi

• NGC account with an API key from NGC API Key setup:

  echo $NGC_API_KEY | grep -E '^[a-zA-Z0-9]{86}=='

• Network access to NGC (nvcr.io) to pull containers and download model assets
• Port 8000 available for the NIM HTTP endpoint

Find model recipes

Browse NIM containers for your hardware platform in the NVIDIA NGC catalog and the NIM for LLMs supported models list. Each container page includes pull and run guidance for that model.

Use the Instructions tab for the base Docker workflow with the default Nemotron 3.5 Lightning 30B-A3B NIM.

Time & risk

• Estimated time: 30 MIN (longer on first run due to model download)
• Risk level: Low
  • Large model downloads may take significant time depending on network speed
  • GPU memory requirements vary by model size
  • Container startup time depends on model loading
• Rollback: Stop and remove the container with docker stop <CONTAINER_NAME> && docker rm <CONTAINER_NAME>. Remove cached models from ~/.cache/nim only if you need the disk space (requires re-download on next run).
• Last Updated: 09/30/2026
  • Deploy NVIDIA NIM for LLM inference on supported hardware platforms with Docker, NGC auth, and OpenAI-compatible endpoint validation