Deploy NVIDIA NIM for LLM Inference

Basic idea

NVIDIA NIM is containerized software for fast, reliable AI model serving and inference on NVIDIA GPUs. This playbook shows how to run NIM microservices for LLMs on your hardware platform through a simple Docker workflow: authenticate with NVIDIA's registry, launch the NIM inference microservice, and validate the OpenAI-compatible HTTP endpoint.

What you'll accomplish

You'll launch a NIM container on your hardware platform to expose a GPU-accelerated HTTP endpoint for chat completions. These instructions use the Llama 3.1 8B Instruct NIM as the default example; additional NIM containers are available in the NGC catalog (see Find model recipes).

What to know before starting

Required:

• Working in a terminal environment
• Using Docker commands and GPU-enabled containers
• Basic familiarity with REST APIs and curl commands

Optional:

• Understanding of NVIDIA GPU environments and CUDA

Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

Prerequisites

Hardware requirements

• Supported hardware platform — see Supported hardware platforms matrix above
• Sufficient memory for your chosen NIM (varies by model)
• At least 10–50 GB available storage for model caching (varies by model)

Software requirements

• NVIDIA drivers installed: nvidia-smi
• Docker with NVIDIA Container Toolkit configured:
  Bash

  CopiedCopy

  docker run -it --gpus=all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi
  

• NGC account with an API key from NGC API Key setup:
  Bash

  CopiedCopy

  echo $NGC_API_KEY | grep -E '^[a-zA-Z0-9]{86}=='
  

• Network access to NGC (nvcr.io) to pull containers and download model assets
• Port 8000 available for the NIM HTTP endpoint

Find model recipes

Browse NIM containers for your hardware platform in the NVIDIA NGC catalog and the NIM for LLMs supported models list. Each container page includes pull and run guidance for that model.

Use the Instructions tab for the base Docker workflow with the default Llama 3.1 8B Instruct NIM.

Time & risk

• Estimated time: 30 MIN (longer on first run due to model download)
• Risk level: Low
  • Large model downloads may take significant time depending on network speed
  • GPU memory requirements vary by model size
  • Container startup time depends on model loading
• Rollback: Stop and remove the container with docker stop <CONTAINER_NAME> && docker rm <CONTAINER_NAME>. Remove cached models from ~/.cache/nim only if you need the disk space (requires re-download on next run).
• Last Updated: 07/31/2026
  • Deploy NVIDIA NIM for LLM inference on supported hardware platforms with Docker, NGC auth, and OpenAI-compatible endpoint validation