---
title: "Deploy NVIDIA NIM for LLM Inference — Overview"
canonical: "https://build.nvidia.com/spark/nim-llm/overview.md"
---

# Basic idea

NVIDIA NIM is containerized software for fast, reliable AI model serving and inference on NVIDIA GPUs. This playbook shows how to run NIM microservices for LLMs on your hardware platform through a simple Docker workflow: authenticate with NVIDIA's registry, launch the NIM inference microservice, and validate the OpenAI-compatible HTTP endpoint.

# What you'll accomplish

You'll launch a NIM container on your **hardware platform** to expose a GPU-accelerated HTTP endpoint for chat completions. These instructions use the Llama 3.1 8B Instruct NIM as the default example; additional NIM containers are available in the NGC catalog (see **Find model recipes**).

# What to know before starting

**Required:**

- Working in a terminal environment
- Using Docker commands and GPU-enabled containers
- Basic familiarity with REST APIs and curl commands

**Optional:**

- Understanding of NVIDIA GPU environments and CUDA

# Supported hardware platforms

Use the matrix below to confirm your hardware platform, recommended default local settings, and whether multi-node applies.

| Hardware platform | OS | Memory | Recommended default local settings | Multi-node capable hardware |
| :---- | :---- | :---- | :---- | :---- |
| **DGX Spark** | DGX OS (Linux) | 128 GB Unified Memory | `nvcr.io/nim/meta/llama-3.1-8b-instruct-dgx-spark:latest` | — |

# Prerequisites

**Hardware requirements**

- Supported hardware platform — see Supported hardware platforms matrix above
- Sufficient memory for your chosen NIM (varies by model)
- At least 10–50 GB available storage for model caching (varies by model)

**Software requirements**

- NVIDIA drivers installed: `nvidia-smi`
- Docker with NVIDIA Container Toolkit configured:
```bash
docker run -it --gpus=all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi
```
- NGC account with an API key from [NGC API Key setup](https://ngc.nvidia.com/setup/api-key):
```bash
echo $NGC_API_KEY | grep -E '^[a-zA-Z0-9]{86}=='
```
- Network access to NGC (`nvcr.io`) to pull containers and download model assets
- Port 8000 available for the NIM HTTP endpoint

# Find model recipes

Browse NIM containers for your hardware platform in the [NVIDIA NGC catalog](https://catalog.ngc.nvidia.com) and the [NIM for LLMs supported models](https://docs.nvidia.com/nim/large-language-models/latest/supported-models.html) list. Each container page includes pull and run guidance for that model.

| Hardware platform | More recipes |
| ----------------- | ------------ |
| **DGX Spark** | [Llama 3.1 8B Instruct NIM for DGX Spark](https://catalog.ngc.nvidia.com/orgs/nim/teams/meta/containers/llama-3.1-8b-instruct-dgx-spark) · [Qwen3-32B NIM for DGX Spark](https://catalog.ngc.nvidia.com/orgs/nim/teams/qwen/containers/qwen3-32b-dgx-spark) · [NIM for LLMs supported models](https://docs.nvidia.com/nim/large-language-models/latest/supported-models.html) |

Use the **Instructions** tab for the base Docker workflow with the default Llama 3.1 8B Instruct NIM.

> [!NOTE]
> **Memory and disk determine what you can run.** Larger NIMs need more unified memory and cache space. If a model is not listed for your hardware platform, check the container page and supported-models list before downloading.

# Time & risk

- **Estimated time:** 30 MIN (longer on first run due to model download)
- **Risk level:** Low
- Large model downloads may take significant time depending on network speed
- GPU memory requirements vary by model size
- Container startup time depends on model loading
- **Rollback:** Stop and remove the container with `docker stop <CONTAINER_NAME> && docker rm <CONTAINER_NAME>`. Remove cached models from `~/.cache/nim` only if you need the disk space (requires re-download on next run).
- **Last Updated:** 07/31/2026
- Deploy NVIDIA NIM for LLM inference on supported hardware platforms with Docker, NGC auth, and OpenAI-compatible endpoint validation