---
title: "Deploy NVIDIA NIM for LLM Inference — Instructions"
canonical: "https://build.nvidia.com/spark/nim-llm/instructions.md"
---

# Step 1. Verify environment prerequisites

Check that your system meets the basic requirements for running GPU-enabled containers.

```bash
nvidia-smi
docker --version
docker run --rm --gpus all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi
```

Expected output should show GPU details from `nvidia-smi` inside the container.

If you see a permission-denied error connecting to the Docker daemon socket, add your user to the `docker` group so you do not need `sudo`:

```bash
sudo usermod -aG docker $USER
newgrp docker
```

# Step 2. Configure NGC authentication

Set up access to NVIDIA's container registry using your NGC API key.

```bash
export NGC_API_KEY="<YOUR_NGC_API_KEY>"
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
```

# Step 3. Select and configure NIM container

Choose a specific LLM NIM from NGC and set up local caching for model assets. The default image below matches the Supported hardware platforms matrix; swap `IMG_NAME` for another NIM from **Find model recipes** when needed.

```bash
export CONTAINER_NAME="nim-llm-demo"
export IMG_NAME="nvcr.io/nim/meta/llama-3.1-8b-instruct-dgx-spark:latest"
export LOCAL_NIM_CACHE=~/.cache/nim
export LOCAL_NIM_WORKSPACE=~/.local/share/nim/workspace
mkdir -p "$LOCAL_NIM_WORKSPACE"
chmod -R a+w "$LOCAL_NIM_WORKSPACE"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R a+w "$LOCAL_NIM_CACHE"
```

# Step 4. Launch NIM container

Start the containerized LLM service with GPU acceleration and shared memory for model loading.

```bash
docker run -it --rm --name=$CONTAINER_NAME \
--gpus all \
--shm-size=16GB \
-e NGC_API_KEY=$NGC_API_KEY \
-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
-v "$LOCAL_NIM_WORKSPACE:/opt/nim/workspace" \
-p 8000:8000 \
$IMG_NAME
```

The container downloads the model on first run and may take several minutes to start. Look for startup messages indicating the service is ready.

# Step 5. Validate inference endpoint

Test the deployed service with a basic chat completion request. Run the following curl command in a **new terminal** while the container is running.

```bash
curl -X 'POST' \
'http://0.0.0.0:8000/v1/chat/completions' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"model": "meta/llama-3.1-8b-instruct",
"messages": [
{
"role":"system",
"content":"detailed thinking on"
},
{
"role":"user",
"content":"Can you write me a song?"
}
],
"top_p": 1,
"n": 1,
"max_tokens": 15,
"frequency_penalty": 1.0,
"stop": ["hello"]
}'
```

Expected output should be a JSON response with a `choices` array containing generated text.

From another device on the same network, replace `0.0.0.0` with your hardware platform's reachable address.

# Step 6. Cleanup

Stop and remove the container when you are done testing. Cleanup is optional rollback — not required to complete the playbook.

> [!WARNING]
> Removing cached models will require re-downloading on the next run.

```bash
docker stop $CONTAINER_NAME
docker rm $CONTAINER_NAME
```

To remove cached models and free disk space:

```bash
rm -rf "$LOCAL_NIM_CACHE"
```

# Step 7. Next steps

With a working NIM deployment, you can:

1. Integrate the API endpoint into your applications using the OpenAI-compatible interface
2. Experiment with different models from the NGC catalog (see **Find model recipes**)
3. Scale the deployment using container orchestration tools
4. Monitor resource usage with `nvidia-smi` and optimize container resource allocation

Test the integration with your preferred HTTP client or SDK to begin building applications.