Deploy NVIDIA NIM for LLM Inference

Verify environment prerequisites

Check that your system meets the basic requirements for running GPU-enabled containers.

nvidia-smi
docker --version
docker run --rm --gpus all nvcr.io/nvidia/cuda:13.0.1-devel-ubuntu24.04 nvidia-smi

Expected output should show GPU details from nvidia-smi inside the container.

If you see a permission-denied error connecting to the Docker daemon socket, add your user to the docker group so you do not need sudo:

sudo usermod -aG docker $USER
newgrp docker

Configure NGC authentication

Set up access to NVIDIA's container registry using your NGC API key.

export NGC_API_KEY="<YOUR_NGC_API_KEY>"
echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin

Select and configure NIM container

Choose a specific LLM NIM from NGC and set up local caching for model assets. The default image below matches the Supported hardware platforms matrix; swap IMG_NAME and MODEL_ID for another NIM from Find model recipes when needed.

export CONTAINER_NAME="nim-llm-demo"
export IMG_NAME="nvcr.io/nim/nvidia/nemotron-3.5-lightning-30b-a3b:latest"
export MODEL_ID="nvidia/nemotron-3.5-lightning-30b-a3b"
export LOCAL_NIM_CACHE=~/.cache/nim
export LOCAL_NIM_WORKSPACE=~/.local/share/nim/workspace
mkdir -p "$LOCAL_NIM_WORKSPACE"
chmod -R a+w "$LOCAL_NIM_WORKSPACE"
mkdir -p "$LOCAL_NIM_CACHE"
chmod -R a+w "$LOCAL_NIM_CACHE"

docker run --rm --gpus=all -e NGC_API_KEY=$NGC_API_KEY $IMG_NAME list-model-profiles

Launch NIM container

Start the containerized LLM service with GPU acceleration and shared memory for model loading.

docker run -it --rm --name=$CONTAINER_NAME \
  --gpus all \
  --shm-size=16GB \
  -e NGC_API_KEY=$NGC_API_KEY \
  -e NIM_MODEL_NAME=$MODEL_ID \
  -e NIM_SERVED_MODEL_NAME=$MODEL_ID \
  -e NIM_PASSTHROUGH_ARGS="--reasoning-parser nemotron_v3" \
  -v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \
  -v "$LOCAL_NIM_WORKSPACE:/opt/nim/workspace" \
  -p 8000:8000 \
  $IMG_NAME

Nemotron 3.5 Lightning is a reasoning model, so --reasoning-parser nemotron_v3 is required for the server to split reasoning traces from the final answer. Without it, the trace and the answer arrive together in choices[].message.content, separated by a literal </think> marker, instead of in a separate choices[].message.reasoning field.

-e NIM_PASSTHROUGH_ARGS="--reasoning-parser nemotron_v3 --gpu-memory-utilization 0.75" \

The container downloads the model on first run and may take several minutes to start. Look for startup messages indicating the service is ready.

Validate inference endpoint

Test the deployed service with a basic chat completion request. Run the following curl command in a new terminal while the container is running.

curl -X 'POST' \
    'http://0.0.0.0:8000/v1/chat/completions' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "nvidia/nemotron-3.5-lightning-30b-a3b",
      "messages": [
        {
          "role":"user",
          "content":"Summarize GPU computing in one sentence."
        }
      ],
      "max_tokens": 128,
      "temperature": 0.0,
      "chat_template_kwargs": {"enable_thinking": false}
    }'

Expected output should be a JSON response with a choices array containing generated text. The model field in the response echoes nvidia/nemotron-3.5-lightning-30b-a3b, matching the NIM_SERVED_MODEL_NAME set at launch.

"enable_thinking": false returns a direct answer. Set it to true (or omit it) to let the model reason first — with the nemotron_v3 parser enabled, the reasoning trace arrives in choices[].message.reasoning and the answer in choices[].message.content.

From another device on the same network, replace 0.0.0.0 with your hardware platform's reachable address.

Cleanup

Stop and remove the container when you are done testing. Cleanup is optional rollback — not required to complete the playbook.

docker stop $CONTAINER_NAME
docker rm $CONTAINER_NAME

To remove cached models and free disk space:

rm -rf "$LOCAL_NIM_CACHE"

Next steps

With a working NIM deployment, you can:

1. Integrate the API endpoint into your applications using the OpenAI-compatible interface
2. Experiment with different models from the NGC catalog (see Find model recipes)
3. Scale the deployment using container orchestration tools
4. Monitor resource usage with nvidia-smi and optimize container resource allocation

Test the integration with your preferred HTTP client or SDK to begin building applications.